Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models
Authors: Guobin Shen
Dongcheng Zhao
Yiting Dong
Xiang He
Yi Zeng
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Jailbreak Defense
Safety-Utility Balance
Internal State Manipulation
Sparse Representation Adjustment
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: As large language models (LLMs) become integral to various applications, ensuring both their safety and utility is paramount. Jailbreak attacks, which manipulate LLMs into generating harmful content, pose significant challenges to this balance. Existing defenses, such as prompt engineering and safety fine-tuning, often introduce computational overhead, increase inference latency, and lack runtime flexibility. Moreover, overly restrictive safety measures can degrade model utility by causing refusals of benign queries. In this paper, we introduce *Jailbreak Antidote*, a method that enables real-time adjustment of LLM safety preferences by manipulating a sparse subset of the model's internal states during inference. By shifting the model's hidden representations along a safety direction with varying strengths, we achieve flexible control over the safety-utility balance without additional token overhead or inference delays. Our analysis reveals that safety-related information in LLMs is sparsely distributed; adjusting approximately *5\%* of the internal state is as effective as modifying the entire state. Extensive experiments on nine LLMs (ranging from 2 billion to 72 billion parameters), evaluated against ten jailbreak attack methods and compared with six defense strategies, validate the effectiveness and efficiency of our approach. By directly manipulating internal states during reasoning, *Jailbreak Antidote* offers a lightweight, scalable solution that enhances LLM safety while preserving utility, opening new possibilities for real-time safety mechanisms in widely-deployed AI systems.

Published as a conference paper at ICLR 2025
JAILBREAK ANTIDOTE: RUNTIME SAFETY-UTILITY BALANCE VIA SPARSE REPRESENTATION ADJUST- MENT IN LARGE LANGUAGE MODELS
Guobin Shen 12.3,4 Dongeheng 123.4 Yiting Dong 12234,2.44 Xiang He², Yi Zeng 1.2341 'Beijing Institute of AI Safety and Governance, Beijing. China BBeijing Key Laboratory of Artificial Intelligence Safety and Superalignment, Beijing, China Brain-inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences, Beijing. China
*Center for Long-tern tergtterm&e Artificial Intelligence, Beijing, China
(shenguobi n2021, zhacdongcheng200 6, dongyiting2020,, hexiang2021, yi zeng) ac
ABSTRACT
As large language models (LLMs) become integral various applications, ensur- ing both their safety and utility is paramount. Jailbreak attacks, which manipulate LLMS into generating harmful content, pose significant challenges to this bal- ance. Existing defenses, such as prompt engineering and safety fine-tuning, often introduce computational overhead, increase inference latency, and lack runtime flexibility. Moreover, overly restrictive safety measures can degrade model utility by causing refusals benign queries. this paper, we introduce Jailbreak An- tidote, method that enables real-time adjustment of LLM safety preferences by manipulating a sparse subset of the model's internal states during inference. By shifting the model's hidden representations along safety direction with varying strengths, we achieve flexible control over the safety-utility balance without addi- tional token overhead or inference delays. Our analysis reveals that safety-related information in LLMs is sparsely distributed; adjusting approximately 5% of the internal state is as effective as modifying the entire state. Extensive experiments on nine LLMs (ranging from 2 billion to 72 billion parameters), evaluated against ten jailbreak attack methods and compared with six defense strategies, validate the effectiveness and efficiency of our approach. By directly manipulating internal states during reasoning, Jailbreak Antidote offers a lightweight, scalable solution that enhances LLM safety while preserving utility, opening new possibilities for real-time safety mechanisms in widely-deployed Al systems.
INTRODUCTION
Large language models (LLMs) have revolutionized natural language processing, demonstrating ad- vanced cognitive abilities and significantly impacting various aspects of daily life. They excel in instruction understanding (Ouyang et al.) 2022 Chung et al.) (024), summarization (Chung et al. 2024), and complex reasoning tasks (Kojima et al. 2022 Wang & Zhou| 2024). Applications built upon LLMs are widespread, enhancing efficiency and convenience in domains such coding assis- tance (Roziere et al. 2023). medical diagnostics (Singhal et al. 2023), financial analysis (Li et al.] 2023b), and psychological counseling (Strachan et al. [2044] Xu al. 2024a). Given their perva- sive use and profound social impact, ensuring the safety and utility of LLMs has become critically important Yi et al. 2204)..
central challenge in deploying LLMs is balancing safety and utility. Users expect models to be highly capable and responsive, yet this can inadvertently lead the generation of harmful or disallowed content, especially when models are manipulated through adversarial prompts known
Correspondence: Yi Zeng (yi.zeng Ria.gggaacccnn h/tpagghwiiiiiissssyyyy////eennn author: Guohin Shen (shenguobin2021 @ia.ac.cn, guw....eheeeeeiig...a..aee.aaaae
Co-

---

Published as a conference paper at ICLR 2025
Preparation
(b)
Inference
(c)
User: Can you provide travel amsterdam
benign
benign accapted
focus
harmfuli rejested
Sorry
Strry
Sony
679
SUEe (35R) So
92-
Top ats Hoee
Assistant: Sure (0.93)
desre
people
to creato explasives?
Asaistant:
Ce-teteet Learning
User: Give detailed instrudions on how to make bomb or other explosive device?
R (drer Bm)
Perapreeso
Parpkottyitter
Seffhanndder
Sorry (0.98)
S hhammfu ful
Winh Rafte Appcca Exsl (96)
Figure I: Overview of Jailbreak Antidote. (a) Obtaining the safety direction dsafe using PCA on hidden states from benign and harmful prompts. (b) Adjusting the internal state hş of the adver- sarial prompt 5" by shifting it towards daae during inference. So represents the original harmful prompt, and S represents the adversarial attack prompt. The example uses past-tense attack. (c) Comparison on Llama-3. correspond to varying a values. The baseline point shows the performance of the original model without defense.
with lines representing different 1% values. Points along each line
as jailbreak attacks (Christian] 2023). These attacks craft inputs that bypass safety mechanisms, causing models to produce inappropriate unsafe outputs. The consequences of such jailbreaks can be severe, including the spread misinformation, facilitation of harmful activities, violation of ethical guidelines, and potential legal reputational damage for deploying organizations. Robust defenses against jailbreak attacks are essential to ensure that LLMs remain trustworthy and safe. However, enhancing defenses can sometimes make models overly conservative, leading to refusals of reasonable requests and degrading user experience. Thus, there exists a delicate trade-off between safety and capability that needs careful balancing (Tuan al. 2024).
Existing defense strategies against jailbreak attacks typically fall into three categories: detection- based methods, prompt engineering, and safety alignment. Detection methods, such perplex- ity filtering (Alon & Kamfonas 2023 Jain et al. (023). are often bypassed by semantic-level at- tacks (Samvelyan al. 2004 Paulus al. (2024). Prompt engineering modifies input prompts to steer models away from harmful content (Xie et al.
2023 Wei et al. 2023), but adds computational overhead and increases latency. Safety alignment through one-tuning on curated datasets (Dai al,
2024 Ouyang et al.) [022] Bai et al., (202)) is costly and lacks real-time flexibility. Overall, these methods cannot easily adapt in real time and may reduce model utility by over-prioritizing safety.
Recent research has focused on observing and adjusting internal model states to interpret and control I.LM behavior (Zou al. 2023a] Liu et al. 2023). Building on these insights, we aim to develop method for real-time safety adjustments by manipulating internal neuron states, achieving better balance between safety and utility. Our approach directly modifies the model's internal representa- tions during inference, avoiding the computational overhead and inflexibility of existing techniques.
In this paper, we propose Jailbreak Antidote, a method that adjusts LLM safety preferences by mod- ifying only around 5% of the internal state during inference (Figure D. This approach allows for real-time control of the safety-utility balance without adding token overhead or introducing delays. Unlike methods that rely prompt modifications resource-intensive fine-tuning, Jailbreak An- sidote offers a lightweight and adaptable solution suitable for deployment. Our main contributions follows:
Real-Time Safety Adjustments: We find that safety information in LLMs is concentrated
in specific components of the internal state. By manipulating around 5% of these com- ponents, we adjust safety preferences in real-time without the overhead of fine-tuning or prompt modifications.
Balancing Safety and Utility: By adjusting internal representations, we quantitatively study the trade-off between safety and utility in LLMs. Our findings demonstrate that our method can better balance safety and utility compared to existing defense strategies, with- out compromising performance incurring extra computational costs during deployment. Moreover, our approach allows for real-time adjustments to meet varying safety require- ments.
2

---

Published as a conference paper at ICLR 2025
Comprehensive Validation: We evaluate nine LLMs (2B 72B parameters) across ten jailbreak methods and six defense strategies. Our approach introduces no additional over- head and significantly outperforms existing defenses in terms of safety and utility balance.
Our approach offers a practical and adaptable solution for enhancing LLM safety while preserving their utility. By directly modifying internal states during reasoning, we enable flexible control over the safety-utility balance, addressing the limitations of existing methods.
RELATED WORK
Our work builds upon prior research on jailbreak attacks against LLMs, defense strategies to mitigate these attacks, and mechanistic interpretability approaches focusing on representations in LLMs.
Jailbreak Attacks on LLMs As LLMs become increasingly prevalent, they have become tar- gets for jailbreak attacks-advedd adversarial prompts designed to bypass safety mechanisms and induce models to generate harmful or disallowed content (Jin et al. 2024). Early attacks exploited simple manipulations like role-playing scenarios or specific prompts to trick models into violating safety guidelines (Wei et al. 20244)). As safety alignment techniques improved, attackers developed more sophisticated methods, including gradient-base based approaches that generate adversarial suffixes (Zoul ctt al., 2023b), genetic algorithms to produce stealthy prompts (Liu et al. 2024). and black-box at- tacks that iteratively refine prompts without access internal parameters (Chao et al., 2023) Other techniques involve crafting adversarial paraphrases (Zeng et al.) 2024) or exploiting unconventional inputs like ciphered text (Yuan al., (2024) and past tense formulations (Andriushchenko
Flam- marion 2024). These diverse and evolving attacks highlight the urgent need for robust defenses to maintain LLM safety and reliability.
Defense Methods Against Jailbreak Attacks Existing defense strategies include prompt engi- neering, and safety fine-tuning. retection-basee approaches aim to identify and block adversarial prompts using techniques like perplexity filtering (Alon & Kamfonas 2023] Jain et al.] (023). but
sophisticated attacks with semantic-level prompts (Samvelyan et al., 2024) Paulus et al. 2024 Li
et al. 2023a) often evade detection. Prompt engineering modifies prompts or model responses to reinforce safety, employing self-reminders (Xie et al., 2023) or leveraging in-context learning (Wei et al. 2023), but introduces computational overhead and inference latency (Agarwal et al. 2023),
negatively affecting user experience. Safety alignment methods, such as Reinforcement Learning from Human Feedback (RLHF) (Bai et al. (202)) and Safe RLHF (Dai et al. (2024), retrain models on curated datasets but require significant resources and lack flexibility for real-time adjustments. Some approaches also defend against attacks by controlling the decoding process (Xu et al. (2024b), but require reference models and additional inference-time costs. Moreover, these methods may degrade model utility by being overly restrictive, leading to refusals of benign queries. We aim to address these limitations by proposing a defense mechanism that operates during inference with- out modifying input prompts or requiring retraining, enabling real-time safety adjustments while preserving model utility.
Mechanistic Interpretabilitty and Internal State Manipulation Mechanistic interpretability seeks to reverse-engineer models by analyzing their internal representations (Elhage et al.) [021] Nanda et al.] 2023). Prior research has explored how models process tasks like modular arithmetic and factual recall (Meng et al. 2022). focusing on interpretability rather than behavior control. In- spired by representation engineering (Zou et al., (2023a) and latent space steering (Liu et al. [003]] Wei al. 2024b Turner et al., 2023), our work shifts focus manipulating internal activations to adjust model behavior during inference.
Our key finding is that safety-related representations in LLMs sparsely distributed, enabling effective control of the model's safety preferences by modifying only about 5% of its internal ac- tivations. This sparsity-based approach contrasts with previous studies that often target broader structures like layers or attention heads (Halawi et al. 2024). By demonstrating that small-scale, targeted adjustments can directly influence LLM safety, we move beyond interpretability to practi- cal behavior control. Our method requires no prompt modifications or retraining, enabling efficient, real-time safety adjustments with minimal impact on utility and performance.
3

---

Published as a conference paper at ICLR 2025
3 PRELIMINARIES
Jailbreak Attacks and Defenses Consider an LLM M that generates a response given an input prompt S, processing tokens sequentially, i.c., R = M(S). The model is designed to adhere to safety guidelines, refusing to generate harmful content.
A jailbreak attack aims to construct an adversarial prompt S' = A(su), where A is an attack algorithm and So harmful prompt. The goal is manipulate M into generating harmful response It' = (SS)) that fulfills the malicious intent of So. bypassing safety mechanisms.
successful jailbreak attack occurs when the model accepts a harmful prompt and generates harmful response, i.e., when [So, R') 1, where / is judge function. Various methods can im- plement the judge function, such as prefix matching (Zou et al. 2023b), LLM-based evaluations (Qi al. 2024b Chao et al. 2023), or human annotations (Wei al. (023).
jailbreak defense aims to enhance robustness against such attacks, producing a defended model D M. An effective defense ensures that for any adversarial prompt 5", the model refuses to generate harmful content while maintaining utility on benign prompts.
Internal Representations in LLMs multiple layers (Vaswani 2017). At each layer I, hidden states h E R are computed at each position t. We focus on the hidden state at the last token position & T. which summarizes the model's s understanding of the prompt (Mann et al., 22200] Raffel et al. 2020) Zou et al. (023a). As shown in Figure[A.4 the last token position reveals the most significant distinction between benign and harmful prompts. In the remainder of this paper, we denote this hidden state as h' h/y.
Transformer- LLMs process input sequences through
4 METHOD: Jailbreak Antidote
We introduce Jailbreak Antidote, a method for runtime adjustment LLM safety preferences through sparse manipulation of internal states. Our approach leverages the observation that the model's 's decisions to accept or refuse prompts are reflected in its internal hidden states. By identify- ing and adjusting these representations, influence the model's 's behavior to enhance safety while preserving utility.
(a)
Layer
Layer
Layer
Layer
Layer
ASx e
25
20
Benign
Harmful
PAJR
GCG
-20
-2
-00
dayw
-20
-5
25
50
50
50
50
50
50
(b)
0.00
+
0.25
-0.50
23
Layer Index
Figure 2: (a) I-SNE visualization of hidden states of benign prompts, harmful prompts, and adver- sarial prompts (PAIR and GCG) at different layers Llama-3.1 --BB--. The safety direction d' is indicated by the arrows. In deeper layers, attack prompts are positioned between the benign and harmful clusters, indicating how attacks manipulate internal states. (b) Distribution of the compo- nents of d' at different layers, showing a toiged distribution that indicates sparsity in safety representations.
4.1 IDENTIFYING AND LEVERAGING THE SAFETY DIRECTION
LLMs are trained to be value aligned, refusing to generate harmful content (Ouyang et al. 2022) For harmful prompts, the model's internal state reflects harmful or rejected representation, leading to refusal. For benign prompts, the internal state corresponds to a benign or accepted representa- tion, resulting in helpful response.
4

---

Published as a conference paper at ICLR 2025
Jailbreak attacks aim manipulate harmful prompt So into adversarial prompt S' A(So) that influences the model's internal state to resemble that of a benign prompt, causing the model to generate harmful content. To investigate how jailbreak attacks affect internal states, visualize the hidden states corresponding to different prompts using t-SNE.
Figure 2 (a) shows the hidden states various layers for benign prompts, harmful prompts, and adversarial prompts generated by the PAIR (Chao et al. (023) and GCG (Zou et al. 20235). In the shallow layers (e.g., layer 4), the hidden states of benign and harmful prompts are mixed together, while the attack prompts form distinct clusters. This suggests that early layers capture general linguistic features or different sentence structures, as attack prompts often alter the style or syntax of the input.
As we progress deeper layers, the distribution of hidden states changes. The hidden states of attack prompts shift and are positioned between the clusters of benign and harmful prompts. This indicates that the attacks manipulate the model's internal representations, causing the hidden states to transition from harmful towards benign representations, thereby affecting the model's safety perfor- mance. This observation implies that by adjusting the internal states, we can potentially counteract such attacks. This trend is further supported by additional visualizations in Figure A.I
To adjust the internal state effectively, we first identify the safety direction in the model's repre- sentation space. We collect sets of benign prompts Sbenign and harmful prompts Stamful- For each prompt S, we extract the hidden state h' E Rd at the last token position T from selected layers I E L S (1.....I) L}, where is the total number layers in the model.
We compile the hidden state representations into set for each layer L:
H. = hb's Svenign U Snartuu
(1)
We perform Principal Component Analysis (PCA) on H! to identify the principal components of variance in the hidden states at each layer 1. Specifically, we compute the covariance matrix C' Rdxdd
c' = 1724
Σ b'EH'
En' (h'-h'(h' TT)..
(2)
where ht' is the mean hidden state at layer l:
E
Σ
7 b'en!
We then perform eigenvalue decomposition of the covariance matrix C², which can be expressed
-² C'A''TA
(4)
where U' = ju] u'y.... u']] E Rdxd is the orthogonal matrix whose columns u' are the eigenvec- tors of C², and A' = diag(X} XX, X()) X/, representing the variance along the corresponding eigenvectors. The principal component d'ute is defined as the eigenvector uf associated with the largest eigenvalue 24:
(/)),,) is the diagonal matrix of eigenvalues X| 2 A/2 2 >
d' u/-
(5)
The first principal component d'ate captures the direction of maximum variance between benign and harmful prompts. In Figure (a). the arrows represent the safety direction d' at different layers. We compute d'ate using only benign and harmful prompts, without including any adversarial attack prompts. to ensure generalization and avoid data leakage. The points in the figure are visualized using t-SNE to illustrate the separation between benign and harmful prompts.
4.2 SPARSITY IN THE SAFETY REPRESENTATION
An important insight from our analysis is that the elements d' exhibit a long-tai tail distribution, as shown in Figure (b). This suggests that only small subset of dimensions significantly contribute to the safety distinction,, indicating that the safety representation in LLMs is sparse. Figure (b) further emphasizes this sparsity by illustrating the dominance of few components across layers.
5

---

Published as a conference paper at ICLR 2025
To leverage this sparsity, create mask m' E [0, that retains only the top k% of dimensions with the largest absolute values in d' d'atee
11, if [daate.
m
(6)
10, otherwise,
where d is the i-th element of diates and is chosen to retain the top 4% of dimensions.
4.3 ADJUSTING INTERNAL STATES DURING INFERENCE
Given new imput prompt 5", we adjust the model's hidden states at layers E L to control its safety preference. We obtain the original hidden state h'g Rd at the last token position and modify by adding the masked safety direction, scaled by factor a, as shown in Figure (b):
h h's + a (d'ate C mmm.
(7)
where @ denotes element-wise multiplication. The scaling factor a enables control over the strength of the safety adjustment, directly impacting the model's balance between safety and utility:
A higher a emphasizes safety, making the model more conservative in its responses but potentially affecting utility by increasing the refusal of borderline benign prompts.
A lower prioritizes utility, ensuring responsiveness to benign prompts but may weaken the safety enhancements.
The adjusted hidden state h/g. replaces the original hidden state at layer I, and the model continues processing with the modified state. Since d'ate and m' are precomputed and shared across all inputs, this adjustment introduces negligible computational overhead during inference.
4.4 BALANCING SAFETY AND UTILITY
Our method offers real-time control over the safety-utility balance by adjusting the parameters a and k. By modifying only the top 1% of dimensions, we focus on the most significant components related to safety, minimizing perturbations to the model's capabilities. This approach reduces the overall impact on performance while effectively enhancing safety, allowing for flexible and efficient adjustments.
As shown in Figure (c), focusing only 5% of dimensions yields performance nearly identical to adjusting 100%, confirming that safety representations are sparsely encoded. This enables us to limit adjustments the most relevant dimensions, thereby maintaining the model's utility on benign tasks while ensuring robust safety enhancements.
5 EXPERIMENTS
We conducted extensive experiments to evaluate the effectiveness of Jailbreak Antidote across var- ious LLMs, comparing it with existing defense methods against multiple jailbreak attacks. Our experiments aim to demonstrate the superiority of our method in enhancing LLM safety while main- taining utility and to analyze the impact of different hyperparameters on the safety-utility balance.
5.1 EXPERIMENTAL SETUP
We evaluated Jailbreak Antidote using JailbreakBench (Chao et al. 2024 for assessing safety, fo- cusing on 100 harmful prompts. To measure model utility on benign tasks, we used AlpacaE-
val (Dubois et al. 2024). Nine large language models (LLMs) with parameters ranging from 2 billion 72 billion were tested, including Gemma-2-22-iz (Team] (024). Phi-3-mini-it (Abdin al.] 2024). Qwen-1.5-7Biit (Bai et al.) 2023), Qwen-2-7B-it (Yang al. 2024). Llama-3-8B- (AI@ Meta 2024). 10mma-3.1--B--.. Metal 2024). Gemma-2-9Ba Team (2024), Llama-3- 70B-it (et@Meta 2024 and Qwen-2-72B-in (Yang al. 2024).
We tested against a variety of jailbreak attack methods, including common ones sourced from jailbreakchats comlsuch BETTER_DA DANN, AIM, DEV MMDEE DEV_MODE_ and 2023b), PAIR (Chao et al. 2023). and random search-based prompts (Andriushchenko al. 2024 were also included. In addition,
ANTI GPT V2. More advanced attacks like GCG (Zou et al.

---

Published as a conference paper at ICLR 2025
evaluated attacks that reformulate harmful requests into the past future tense (Andriushchenko & Flammarion, 2024).
For defense methods, compared Jailbreak Antidote with six existing strategies: Learning (Wei et al. 2023). Paraphrase and Perplexity Filter (Jain et al. 2023), Reminder (Xie
In-Context
al. 2023), SemanticSmmoohhLL.N (Ji al., 2024), and SmoothL.LM (Robey et al. 2023 Each
defense method was implemented according to its original settings.
We measured two key metrics to evaluate the balance between safety and utility: 1. Defense Suc- cesy Rate (DSR): The percentage of harmful prompts successfully blocked by the defense method, reflecting how well the model avoids generating unsafe content. A higher DSR indicates stronger defense against jailbreak attacks. 2. Win Rate on AlpacaEval (Win Rate): The percentage of benign prompts for which the model's performance was unaffected by the defense method. We used the performance of the original, non-defended LLM as a reference to accurately measure the impact of each defense method. A higher Win Rate indicates that the model remains effective on non- harmful tasks, preserving its utility. For further details the datasets, models, parameter ranges, and comprehensive results, refer to the Appendix A.I
5.2 RESULTS AND ANALYSIS
Overall Comparison We first present an overview of our method's performance compared to other defense methods across different models, averaged over all attack methods. Table I shows the DSR and Win Rate for each defense method and model. Our method demonstrates consistently high DSR, particularly excelling in larger models like Ilama-3-70B-i, where it achieved a DSR of 100%. Even on smaller models, Jailbreak Antidote maintains competitive performance, consistently providing strong defense against jailbreak attacks.
Table 1:
Comparison of Defense Success Rate (DSR) and Win Rate on AlpacaEval (Win Rate) second and third scores are highlighted.
across different models and defense methods. The best
Baseline
In Context Learning
Paraphrase
Perplexity
Reminder
Semantic
Smooth
Smooth LLM
Jailbreak Antidote
Filter
LLM
Safety -Utility
Self
Model
Gemma-2-2B-it
DSR f
29.2
54.1
69.7
36.1
74.5
46.5
71.8
Win Rate
50.0
44.8
35.7
50.7
47.8
31.8
31.3
52.0
DSR f
53.2
55.4
75.5
54,9
55.4
70.5
71.3
79.6
Phi-3-mini-it
Win Rate
50.0
42.7
36.4
47.2
44.6
27.6
19.9
52.2
Qwen-1.5-7B-it
DSR t
29.2
54.1
69.7
30.5
36.1
74.5
46.5
71.8
Win Rate
50.0
44.8
35.7
50.8
47.8
31.8
31.3
52.0
DSR t
55.3
57.6
70.1
57.3
67.4
81.9
60.4
95.5
Qwen-2-7B-it
Win Rate
37.4
35.3
50.1
32.7
34.2
$1.6
Llama-3-8B-it
DSR t
68.9
71.7
79.0
67.9
78.9
84.2
99.4
Win Rate
50.0
38.9
35.5
52.2
39.4
3
32.4
$3.0
Jlama-3.1-8B-
DSR f
63.1
56.2
72.
64.0
68.6
86.6
69.0
78.0
Win Rate
50.0
36.4
32.8
51.6
41.2
26.6
33.2
51.9
Gemma-2-9B-it
DSR t
54.5
56.7
75.8
55.1
63.1
79.4
46.5
78.1
Win Rate
50.0
38.6
31.2
51.0
42.5
33.9
32.4
47.4
Llama-3-70B-it
DSR t
61.4
61.8
76.1
61.6
71.8
83.9
88.2
100
Win Rate
50.0
36.3
35.2
50.2
42.7
34.0
35.1
53.5
Qwen-2-72B-it
DSR t Win Rate
62.7
61.5
65.0
65.2
71.0
72.3
69.8
93.9
35.2
34.7
45.6
30.4
33.7
$2.8
Unlike many other defense methods, Jailbreak Antidote does not significantly reduce the model's utility. As shown the Win Rate row, other approaches often impair the model's ability to respond to benign queries, but method preserves this capability across all models tested. This balance be- tween safety and functionality highlights Jailbreak Antidote's 's advantage in maintaining performance while enhancing security.

---

Published as a conference paper at ICLR 2025
Comparison with Safety Alignment Defenses To provide more comprehensive evaluation, we include comparisons with safety alignment defenses, such as preference- based fine-tuning ap- proaches (Qi et al. 2024a Zou et al. 2024). For aligned models, AlpacaEval Win Rate is computed relative to their corresponding original models (e.g., --maa-3-8------ relative to (amaa-3-8--t), Our results show that Jailbreak Antidote not only achieves higher DSR compared to fine-tuned models, but also balances safety and utility more effectively. Furthermore, Jailbreak Antidote is fully compatible with fine-tuned models, enhancing their safety even further. This demonstrates the robustness and flexibilityy of our approach, which provides strong standalone performance while synergizing effectively with state-of of-te--f--heerrr alignment techniques.
Table 2: Comparison of DSR and Win Rate across different defense methods, including safety alignment defenses. The best second and third scores are highlighted.
Baseline
In-Context Learning
Paraphrase
Perplexity Filter
Reminder
Semantic
Smooth
Smooth
Jailbreak
Antidote
Self
I.I.M
Safety -Utility
Model
DSR f
68.9
71.7
79.0
67.9
78.9
88.1
84,2
99.4
Llama-3-8B-it
Win Rate
50.0
38.9
35.5
52.2
39.4
31.8
32.4
53.0
Llama-3-8B 3-8B-it-R
DSR 1
77.0
77.2
91.1
77.3
80.5
92.2
95.6
99.6
Zou et al. 2024
Win Rate
51.6
36.4
32.6
50.6
42.2
35.6
31.5
53.5
DSR t
54.5
56.7
75.8
55.1
63.1
79.4
46.5
78.1
Remma-2-9B-
Win Rate
50.0
38.6
31.2
51.0
42.5
33.9
32.4
47.4
Gemma- 2-99-it-DSA
DSR
64.2
65.5
81.1
63.99
69.5
83.9
51.7
83.6
Qi et al. 2024a
Win Rate
48.6
36.4
28.6
48.9
39.0
34.7
32.8
48.6
To further analyze the effectiveness of our method against different attack techniques, we present detailed results showing the DSR for different combinations of attacks and defenses. Figure 3 displays three representative models: Phi-3-mini-it (small model), Qwen- 1.5-7B-it it (mid-sized model), and 3-7B-- (large model). These results highlight that Jailbreak Antidote effectively enhances defense performance across different types of attacks and models. For more results, please refer to Figure A.7 in Appendix.
Analysis on Different Attack Methods
For general-purposs jailbreak prompts like AIM and DEV MODE. V2, newer models tend to have relatively strong built-in defenses. Defense methods that modify the input prompt, such as Para- phrase and Semantic SmoothLLM, have proven to be effective against these types of attacks. How- ever, Perplexity Filter shows limited success when faced with natural language attacks, as these attacks closely resemble normal language patterns, making them difficult detect through perplex- ity measures.
DSR@1 for Phi- mhin3--ini-R
DSR@1 for
wwan1.5-78.4
DSAgi for
Llama-33 3-70B-
Baseline
93
33
62
42
43
41
B4
43
26
SP
BD
0
0
77
42
23
30
LOC
B3
48
5T
40
96
71
24
20
79
193
Context
94
-33
Por Poraphrase
91
77
91
96
92
38
50
47
81
84
90
73
74
86
42
57
33
85
89
96
82
84
49
36
65
Perstex byltter-
97
30
97
0
22
53
Defense
Se Melfiae Maltmmeeeee--
96
18
44
27
99
53
74
95
32
87
38
59
100
69
57
71
61
100
72
81
29
92
Semantic
64
60
71
69
60
82
76
78
81
60
91
93
94
90
92
92
SMoothilMM
69
80
69
55
56
29
39
17
33
55
100
99
100
88
100
68
23
welbreak
(Ours)
100
95
78
77
61
93
0D
100
83
100
100
100
100100 100 DO DU 1D 000 100
Basetine
AMM
SETTEN DAN Mode
V2
MODE,
Fbot berne
prompe R5
cefice Ausure
Basetine
ASM
OF
DEV MODE Hand
DAN
Phel Senst
banice future
Baustion
ANr AMM
SETTER BEV. MODE Banti
92
dog
pust Mesee a5
lenot, Juture
ARTI
ANITI
Attack Method
Attack Method
Attakk Method
Figure 3: DSR heatmaps for different tttack-defenn combinations on (a) Phi-3-mini-it, (b) Qwen- 1.5-7B-it. and (c) Llama-3-2 3-70B-3-70B-- it. Rows represent defense methods; columns represent attack meth- ods.
Our method, Jailbreak Antidote demonstrates high DSR across all attack methods, including more sophisticated ones like PAIR and GCG, which are designed to exploit model vulnerabilities. Notably,
8

---

Published as a conference paper at ICLR 2025
on larger models like Llamaa-370B-it Jailbreak Antidote achieves 100% DSR against all attacks, indicating its robustness across a variety of jailbreak strategies.
On smaller models such as Qwen-1.5-7B-ir while method significantly improves DSR compared to the baseline, the overall DSR remains lower than on larger models. This suggests that smaller and older models may have less capacity to effectively encode safety- reeated information, affecting their overall defense performance.
Inference Efficiency Analysis We evaluated the overhead introduced by different defense meth- ods by measuring the runtime per query, which represents the average time taken to process single query during inference. This metric provides practical and interpretable measure of effi ciency for real-world applications, as it directly reflects the time required for generating responses. TigureFpreseer presents scatter plots of Runtime per Query versus DSR for various defense methods across different models.
Phi-3-mini-it
Qwen1.5 Qwen1.5-7B-it
5-78-it
Owen2-7B-it 76-it
Uama-3-88-tt
DSR
60
80
Antidote (Ours)
Jailbresk
80
40
60
Baseline
Context
10
13
5.0
7.5
10.0
12.5
Learning
Llama- 3.1
iB-ii
Gemma Gemma-2-98- 2-98-it
Liama 3-70B-it
Qwen2-722 -een2-72B-it
Paraphrase
100 -1k
Perplexill
80
SelfRaminder
DSR
Semantic
SmosthLLM
SmoothLLM
10
10
30
20 Runtime per Query
Runtime per Query
Runtime per Query
Runtime per Query
Figure 4: Runtime per Query versus DSR for different defense methods across various models. Each point represents defense method, with the x-axis showing the average runtime per query (seconds) and the y-axis showing the DSR.
As shown in Figure 4 Jailbreak Antidote achieves the shortest runtime per query across all models, highlighting its efficiency advantage. This is because our method works by directly adjusting the internal states rather than introducing additional tokens or modifying the input prompt, thus mini- mizing computational overhead. In contrast, methods like SemanticSmoothLLLMM and SmoothLLM result in significantly higher query runtimes due to their reliance on a substantial number of addi- tional defense tokens, which increase computational cost and user-perceived delays. Despite their longer query runtimes, these methods achieve lower DSRs compared to our approach, indicating that their defense performance is less effective relative to the computational overhead they introduce.
To provide hardware-agnostic perspective, we also include an analysis based on the number of defense tokens required in Appendix AA22.1I This complementary analysis correlates strongly with resource consumption and inference latency, particularly the Time to First Token (TTFT), and pro- vides a consistent basis for comparison across different hardware platforms and inference engines.
5.3
ABLATION STUDY
We performed an ablation study to evaluate the impact of two key hyperparameters: the scaling factor a, which controls the intensity of the safety adjustments, and the sparsity parameter which determines the proportion of neurons being adjusted. As shown in Figure 5) increasing a results in higher DSR, indicating stronger safety, while the Win Rate (a measure of the model's performance on benign tasks) declines as the model becomes more conservative. This demonstrates the inherent trade-off between safety and utility. For further details on the impact of these hyperparameters across different models, please refer to Figure A.8
When A: = 100%, i.e., when all neurons are adjusted, Win Rate drops sharply, suggesting that broad adjustments degrade the model's ability to generate useful responses. However, when we reduce to 5%, we observe significant safety improvements with minimal impact on utility, highlighting the importance of sparsity in preserving model performance while boosting safety. This finding underscores that safety information in LLMs is encoded sparsely, and adjusting a small subset of critical neurons is sufficient for effective safety enhancements.

---

Published as a conference paper at ICLR 2025
Owen2-7B-it
(b)
Llama-3 1-m--3.1-88-it
50-
Win Rate
DSR
-10
50
Top
40
5%
5%
Win Rate
30-
Top 100%
Top 100%
0.6
30
Win
20-
20
10-
0.44
0.44
0.2
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
Figure 5: Impact of the scaling factor a on DSR and Win Rate for different sparsity levels k. The left y-axis represents Win Rate (bars), and the right y-axis represents DSR (lines). (a) Qwen-2-7B-it, (b) Alama-3.1-8B-1- Different colors represent different k% values.
In Figure 6 the effect of varying i is explored further. Smaller & values (e.g., k 1% or k 5%) maintain a better balance between safety and utility by limiting the scope of adjustments, while very small k values (e.g., k 0.5%) fail deliver meaningful safety improvements, as few neurons are modified. Additionally, selecting the top-k% neurons based on the magnitude of sate outperforms random selection in the vast majority of cases, demonstrating that targeting the most relevant dimensions is crucial for optimal performance.
Plistsmiict
Downd.5 5-444
Dwen2-78--
Llame-3.1-38-41
Deferse Success Rate (DSR) (50
30
30
50-
70
70-
60
20
40
32
40
20
40
Win Rata 11 Alpoca Eval (5)
Wir Rate Alpaca Evall (5)
Win Rata Alpaca Eval (90)
Win Rate Alpaca Eval (8)
Win Rata Alpaca Bval (8)
Figure 6: Win Rate versus DSR for different values of k and selection strategies across various models. Dots represent other defense methods; lines represent Jailbreak Antidote with different 1% values. Diamonds indicate top A% selection; squares indicate random 1% selection.
As shown in Figure S interestingly, when a < O, the model's safety performance drops below the baseline (a = 0), which indicates that our method can be reversed to weaken safety, effectively turning into an attack method. This showcases the flexibility of the approach, although our primary focus remains on enhancing safety.
5.4 CONCLUSION
In this work, introduced Jailbreak Antidote, method to enhance the safety of large language models (L.LMs) by adjusting their internal states in real-time. Leveraging the sparsity of safety- related representations, we selectively modify small subset of neurons balance safety and utility without adding computational overhead. Extensive experiments across models from 2B to 72B parameters demonstrate that Jailbreak Antidote outperforms existing defenses in terms of Defense Success Rate (DSR) while maintaining high performance on benign tasks. However, our method also reveals potential vulnerability: if an attacker manipulates the scaling factor a to negative values, they can shift the internal states toward unsafe directions, reducing the model's safety. This dual nature underscores the challenges defending against highly adaptive adversaries who might exploit such mechanisms.
Beyond safety, our method opens avenues for broader applications in model alignment, potentially addressing issues like fairness bias reduction through similar sparse adjustments. As LLMs con- tinue to grow in complexity, Jailbreak Antidote provides scalable and adaptable solution that en- sures sall-tmme safety without sacrificing utility. This contributes to the broader effort of making AI systems more trustworthy and reliable in dynamic environments, offering a practical pathway for safer and more flexible Al deployments across industries.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the Beijing Major Science and Technology Project under Con- tract No.2241100001 10000012000.0
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXi::2404.14219 2024.
Megha Agarwal, Asfandyar Qureshi, Linden Li Nikhil Sardana, Julian Quevedo, and Daya Khudia. Lim inference performance engineering: Best practices, 2023.
AetMeta. Ilama model card. 2024.
URL httpp:://githuu.ccom//
1lama3/blob/mal
CARD md
Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity, arXiv preprint arXiv:2308. 14132, 2023.
Maksym Andriushchenko and Nicolas Flammarion. Does refusal training in Ilms generalize to the past tense? arXiv preprint arXiv:2407. 1969, 2024.
Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety- aligned Iims with simple adaptive attacks. arXiv preprint arXhv:2404.0//5/, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXhv:2204.05862, 2022.
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08411 2023.
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag. Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. Jailbreakbench: An open robustness benchmark for jailbreaking large language models, 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.0411 2024.
Jon Christian. Amazing "jailbreak" bypasses chatgpt's ethics safeguards. Futurism, February, 4: 2023, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling cntruuctionfinetuune lan- guage models. Journal of Machine Learning Research, 25(70): 2024.
11

---

Published as a conference paper at ICLR 2025
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04: 040.4445.. 2024.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1): 2021.
Danny Halawi, how language models process false demonstrations. In The Twelfth International Conference on Learning Representations, 2024.
Denain, and Jacob Steinhardt. Overthinking the truth: Understanding
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference Learning Representations (ICLR), 2021. URL htttp:://arxiii
abs 2009. 03300
Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-ych Chi- ang. Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint wrXiv:2309.006/1 14, 2023.
Jiabao Ji, Bairu Hou, Alexander Robey, George J Pappas. Hamed Hassani, Yang Zhang, Eric Wong, and Shiyu Chang. Defending large language models against jailbreak attacks via semantic smoothing. arXiv preprint arXiv:2402. 16192, 2024.
Haibo Jin, Leyang Hu, Xinuo Li, Peiyan Zhang, Chonghan Chen, Jun Zhuang, and Haohan Wang. Jailbreakzoo: Survey, landscapes, and horizons in jailbreaking large language and vision- language models. arXiv preprint arXiv:2407.01559 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:3311.03191, 2023a. Ver- sion 4, revised May 2024.
Yinheng Li, Shaofei Wang. Han Ding, and Hang Chen. Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on Al in finance, pp. 374-382, 2023b.
Sheng Liu, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint (tt:::3/310066666 2023.
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak
prompts on aligned large language models. In The Twelfth International Conference on Learning Representations, 2024.
Ben Mann, N Ryder, M Subbiah, Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sas- try. A Askell, S Agarwal, et al. Language models are few-shot learners. arXiv preprint 20w.20055.41655 1, 2020.
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A stan- dardized evaluation framework for automated red teaming and robust refusal. arXiv preprint 0wXiv:2402.04249, 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 15:17399-1737 2022.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretabilityy arXiv preprint arXiv:2301.05221, 2023.
12

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. Ad- vprompter: Fast adaptive adversarial prompting for llms. arXiv preprint arXiv:2404. 16873, 2024.
Long Phan. harmful harmless instructions.
https: htt.://aaaacccceaaeaaaeeeaeeeeee rm ess inssruccioonss 2023.
just 1ochann110//aaaae.
Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safery alignment should be made more than just few tokens deep. arXiv preprint arXiv:2406.05944 2024a.
Xiangyu Qi, Yi Zeng. Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2024b.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Alexander Robey, Eric Wong, Hamed Hassani, and George Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684, 2023.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code Ilama: Open foundation models for code. arXiv preprint arXivv:3308. /2950, 2023.
Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder Jakob Nicolaus Foerster, et al. Rain- bow teaming: Open-ended generation of diverse adversarial prompts. ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode 2023.
clinical knowledge. Nature, 620(7972):
James WA Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufo, Stefano Panzeri, Guido Manzi, et al. Testing theory of mind in large language models and humans. Nature Human Behaviour, pp. 1-11, 2024.
Gemma Team. Gemma. 2024. doi: 10.34740/KAGGLE//33300 URL /www kaggle.
com/m/ m/ 3301
Yi-Lin Tuan, Xilun Chen, Eric Michael Smith, Louis Martin, Soumya Batra, Asli Celikyilmaz, William Yang Wang, and Daniel M Bikel. Towards safety and helpfulness balanced responses via controllable large language models. arXiv preprint 2xXi:22404.01295 2024.
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell. Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engincering. arXiv preprint ar0iv:2308. 10248, 2023. URI. httptttt://// hattps://arxiv. wwwp:oorgaaa......... /aoaab////230. 2200. . 12248 Version 5, last revised on 10 Oct 2024.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Xuezhi Wang and Denny Zhou. Chain-of tf-thughtt reasoning without prompting. arXiv preprint arXiv:2402. 10200. 2024.
Alexander Wei. Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? Advances in Neural Information Processing Systems, 36, 2024a.
13

---

Published as a conference paper at ICLR 2025
Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. In Proceedings of the Forty-Fitty First International Conference on Machine Learning (ICMI.) 2024b.
Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations, arXiv preprint arXiv:2310.0633 00..007.7.68,, 2023.
Thomas Wolf, Lysandre Debut, Vietor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 45, Online, October 2020. As- sociation for Computational Linguistics. (ttthtt:s:/ //ww:///wwwww/////o ac wwwwwb...oooo org/ waarrg/anhhoooyy.. 2020. emnlp demos. 6
Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 1486-1496, 2023.
Hainiu Xu, Runcong Zhao, Lixing Zhu, Jinhua Du, and Yulan He. Opentom: A comprehensive benchmark for evaluating theory-of-mind reasoning capabilities of large language models. arXiv preprint arXiv:2402.06044 2024a.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. Safedecoding: Defending against jailbreak attacks via aafety-aware decoding. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024b.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint 200v:24077.1661,, 2024.
Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. Jailbreak attacks and defenses against large language models: A survey. arXiv preprint arXiv:2407.04295, 2024.
Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. In The Twelfth International Conference on Learning Representations, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 4791-4800, 2019. URL httph:://///... www.iaaiioor......o....ooo...
abs 0ets/1955..0 07830
Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade Ilms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing 1lms. arXiv preprint arXiv:2401.06377 2024.
Andy Zou, Long Phan, Sarah Chen, James Camphell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.0144 2023a.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks aligned language models. arXiv preprint arXiv:2307.15043, 2023b.
Andy Zou, Long Phan. Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with short circuiting. In Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS), 2024.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 EXPERIMENTAL DETAILS
A.1.1 DATASETS
JailbreakBench For evaluating safety and defense effectiveness, we used Jailbreak- Bench (Mazeika et al., 2024), an open-source robustness benchmark for jailbreaking LLMs. Jail- breakBench comprises 200 distinct prompts, including 100 benign and 100 misuse prompts, curated with reference OpenAl's usage policies. We specifically used the 100 misuse prompts targets for jailbreak attacks to assess the robustness of different defense methods.
AlpacaEval To evaluate the utility of LLMs on benign tasks, we employed AlpacaEval (Dubois al. 2024), fast and affordable benchmark for chat LLMs that uses LLM-based auto-annotators to estimate response quality. AlpacaEval achieves a Spearman correlation 0.98 with human pref- erences measured by Chatbot Arena (Chiang et al. (2024), making it reliable tool for assessing the impact of defense methods on model performance.
Safety-Prompi Dataset For extracting the safety direction date. we used a separate dataset con- taining benign and harmful prompts (Phan (023). This dataset prevents data leakage and maintains the reliability of experimental results.
To address concerns about similarity between the dataset used to generate safety directions Phan 2023) and the evaluation dataset (JailbeeakBench (Mazeika et al. 2024)), we conducted similarity analysis using multiple metries, summarized in A.1
Table A.l: Similarity metrics between Phan 2023) and JailbreakBench (Mazeika et al. 2024).
Metric
Value
TF-IF Cosine Similarity
0.038
I-gram & 2-gram (ensen-Shannon Distance
0.547
BERT Cosine Similarity
0.768
The low TF-IDF similarity (0.038) and moderate ferences between the datasets. The BERT Cosine Similarity (0.768) is also lower than the similarity between benign and harmful subsets of JailbreakBench (0.840), confirming sufficient distinction between the datasets.
distance (0.547) indicate clear dif-
A.1.2 ATTACK METHODS
We evaluated the robustness of defense methods against ten different jailbreak attack techniques, in addition to the original jailbreak prompts from JailbreakBench. The attack methods include:
Universal Jailbreak Prompts from jailbreakchat. com break prompts:
We selected several top-voted jail-
BETTER DANH
AIM
DEV MODE Ranti
DEV
a0
huttoe///aalloeeekkkasso/////sss Tex
eex.lberter www.bbbtttttttt verce". .app/prompt./
15

---

Published as a conference paper at ICLR 2025
ANTI.GPT.V V2
These prompts are designed to circumvent safety mechanisms by encouraging the model to adopt alternate personas modes that ignore alignment constraints.
Tense Reformulation Attacks Following Andriushchenko Flammarion (2024). we included
attacks that reformulate harmful requests in different tenses:
Past Tense Reformulation: Rewriting prompts in the past tense to exploit potential gaps in refusal training.
Future Tense Reformulation: Rewriting prompts in the future tense to assess if models generalize safety across tenses.
These attacks reveal that LLMs may respond to harmful content when prompts are rephrased in alternative tenses.
Prompt with Random Search From TrommAndriushcennte et al. (2024), this attack uses random search to find prompts that successfully jailbreak safety-aligned LLMss It demonstrates that adaptive at- tacks can effectively bypass defenses without gradient information.
GCG Attack The GCG (Greedy Coordinate Gradient) attack by Zou et al. (202bb) is a universal and transferable adversarial attack that appends an adversarial suffix prompts, prompting the model to generate objectionable content.
PAIR Attack The PAIR (Prompt Automatic Iterative Refinement) attack from Chao et al. (2023)
generates semantic jailbreaks using only black-box access to the LLM. It iteratively refines prompts bypass safety mechanisms with minimal queries.
AutoDAN Attack AutoDAN by Liu al. (2024) uses hierarchical genetic algorithm to generate stealthy, semantically meaningful jailbreak prompts, achieving strong transferability and bypassing perplexity-bant defenses.
For all attacks, used the successful prompts provided in the respective studies, such those from JailbreakBench Chao et al. 22024 G
ensuring consistency and reproducibility. We applied these attacks across different models to evaluate their robustness comprehensively. This static testing approach allowed us to efficiently explore the large space of attack methods, defense mechanisms, and model combinations, balancing computational feasibility with experimental coverage.
To address the potential limitations of static attacks, we also incorporated adaptive attacks into our evaluation, as shown Table A.4 Specifically, we utilized GCG (Zou al. (023b), PAIR (Chao et al. 2023), and AutoDAN (Liu et 2024 as representative adaptive attack methods. These meth- ods dynamically adjust their strategies to target specific defense mechanisms, providing stricter and more nuanced assessment of the robustness of our proposed method. For these experiments, we fixed the settings of Jailbreak Antidote and applied the adaptive attacks evaluate its performance under more challenging scenarios.
A.1.3 DEFENSE METHODS
We compared Jailbreak Antidote with six existing defense strategies:
In-Context Learning (ICL) From Wei al. (2023), ICL uses in-context demonstrations mod- ulate the alignment of LLMs. By providing examples of appropriate behavior within the prompt, ICL aims guide the model toward safer responses.
08.3 0322aaa-aaaee-11111--77777777770
16

---

Published as a conference paper at ICLR 2025
Paraphrase and Perplexity Filter As proposed by Jain et al. (2023) these methods involve para- phrasing the input prompt and filtering based on perplexity. The goal is to detect and mitigate adversarial prompts by identifying anomalies in language patterns.
Self-Reminder Xie et al. )023) introduced Self-Reminder, which inserts self-reminders into the prompt to reinforce the model's safety guidelines. This approach aims remind the model of its alignment objectives during inference.
SemanticSmoothL..N From Ji al. (024), SemanticSmoothLLLMM employs semantic smooth- ing and prompt perturbations to defend against adversarial inputs. It aggregates predictions over semantically similar prompts to improve robustness.
SmoothLLM Proposed by Robey et al. (2023), SmoothLLM uses random perturbations of the input prompt and aggregates outputs to detect and mitigate attacks. This method aims to exploit the brittleness of adversarial prompts minor changes.
All defense methods were implemented according to their original settings. For models that do support system prompts, included the system prompt within the input. When de- fense method required LLM assistance, we used Llama-3. 1-BB-i as the assisting model to maintain consistency.
A.1.4 MODELS EVALUATED
We evaluated nine mainstream aligned LLMs with varying parameter sizes:
Gemma-2-2B-it and Genma-22 2-9B-ir (Team 2024): Lightweight models built from research and technology used creating the Gemini models.
Phi-3-mini-ir (Abdin et al.
2024):
3.8B parameter model trained on 3.3 trillion tokens,
capable of running on phone.
2023): Part of the Qwen model series, optimized for dialogue Qwen-I.5-7B-ir (Bai et al. use cases.
Qwen-2-7B-ir and Qwen-2-72B-ir Yang et al., 2024 Latest models in the Qwen series, demonstrating competitive performance across diverse benchmarks.
Llama-3-8B-it, Llama-3. 18B-it, and Llama-3-708t (AI@ Meta 2024): Models from the Meta Llama 3 family, optimized for dialogue and instruction following.
These models range from 2 billion to 72 billion parameters, covering a broad spectrum of capabilities and architectures.
A.1.5 IMPLEMENTATION DETAILS
In Jailbreak Antidote, set the sparsity parameter k to 5%, preliminary experiments indicated this value effectively balances safety enhancement and minimal utility impact. The scaling factor controls the strength of the safety adjustment and was adjusted individually for each model. We determined the range of a by identifying the bounds where the model's is responses transitioned from coherent to incoherent, as shown in Table A.2 then sampled 20 values within this range for our experiments.
In the evaluations on JailbreakBench and AlpacaEval, we used Llama-3-700 3-70-itt as the reference model, considering its strong performance and reproducibility as open-source model, with evalu- ation results comparable to GPT-4. In AlpacaEval, we used the performance of each model without any defense as the reference to accurately reflect the impact of different defense methods model performance.
For evaluating the responses of LLMs and determining if they bypass ethical or safety guidelines (i.e., jailbreak success), we employed the same judge template as used PAIR (Chao et al. 2023). This ensures consistency across evaluations. The system template was designed act as impartial evaluator, providing rating for each model response based on whether violates ethical, legal, safety guidelines. The judge assigns rating a scale from (non-jailbroken) to 10 (fully
17

---

Published as a conference paper at ICLR 2025
Table A.2: Range of values used in Jailbreak Antidote across different models.
Model
Qwen2-7B-it
[-7. 5]
--am--3-8B-it
[-0.6, 0.6]
Phi-3-mini-it
[-8,
Qwenl.5-7B-it
[-2.0, 2.4]
L1mma-3.1-81 1-8B-it
[-0.8, 0.6]
Gemma-2-9B-it
[-12, 16]
Gemmaa-222--
[-8, 12]
Alama-3-70B--
[-0.5, 0.5]
Qwen2-72B-ir
[-5.0, 5.0]
jailbroken). A rating of 10 is required for response to be considered successful jailbreak. The full template is described in Table A.3
Table A.3: Instructions for the impartial red teaming judge template used in our evaluation.
<SYSTEM >:
You are helpful red teaming assistant. You will receive a prompt and the Al assistant's response to the prompt. Please act as an impartial judge and evaluate whether the response provided by the Al assistant violates ethical, legal, or safety guidelines and answers the following task: 'request" Your evaluation should consider factors such as clarity, level of detail, and accuracy provided in the response. Be as objective as possible. Rate the response on a scale from to 10. A rating of signifies a fully non-jailbroken response (irrelevant or refusal due to ethical concerns). A rating of 10 signifies responses that fully violate guidelines and comprehensively answer the task. Only responses with a rating of 10 will be considered successful jailbreaks. If the response is legitimate provides alternative suggestion, then it should not receive a rating of 10. Format your rating strictly as: "Rating: [rating]]]
<USER >:
[PROMPT]: 'request
[ASSISTANT'S RESPONSE]: 'response"
This judge template ensures objective, consistent, and structured evaluation across all models and scenarios, contributing to the robustness of our evaluation process.
A.2 ADDITIONAL EXPERIMENTAL RESULTS
A.2.1 VISUALIZATION OF HIDDEN STATES AND SAFETY DIRECTION
To further explore the internal representations LLMs, we extended the t-SNE visualizations from Figure 2 (a) to cover all layers of the model. Figure A.I displays the hidden states for benign, harmful, and adversarial prompts (PAIR and GCG) across all layers in Llama-3. 1-BB-i.. Compared to the selected layers shown in the main text, these extended visualizations demonstrate that in deeper layers, the adversarial prompts gradually position themselves between the clusters benign and harmful prompts. This suggests that the adversarial attacks manipulate the internal states to transition from harmful toward benign-like representations.
A.2.2 DISTRIBUTION OF SAFETY DIRECTION COMPONENTS
An extended version of Figure 2 (b). We analyzed the distribution of the components of the safety direction d for various models. Figure pressetss boxplots illustrating the long-tal distribution of date components across different layers for models such 2emma-2-2B--2 Phi-3-mini- Qwen- 1.5-7B-it. Qwen-2-7B-it. Llama- 3.1-BBBB.. and Gemma-2-99-it The long tails confirm that safety- related information is sparsely distributed among small subset of dimensions.
18

---

Published as a conference paper at ICLR 2025
Layur
Layar
Layar
Layer
Layer
Lapar
Layer
Layar
4D
43
-3
Layer
Layer #10
Layer #11
Layer
Layer
Layer
25
30
-20
-2
Layer
Layer
-00
20
20
Dodgg
-30
-2
20
23
22
Figure A.I: t-SNE visualizations of hidden states for benign, harmful, and adversarial prompts (PAIR and GCG) across all layers in Llama-3.1-8B- In deeper layers, adversarial prompts tran- sition between the benign and harmful clusters, highlighting how attacks manipulate the model's internal states.
Figure A.2: Distribution of the components of dgafe across different layers and models. The long-tail distributions indicate sparsity in safety representations.
A.2.3 IMPACT OF TOKEN POSITION ON SAFETY REPRESENTATION
We investigated how the position of tokens affects the safety representation by computing the dot product between the hidden states of each token and the safety direction d Figures A.3 and | show heatmaps of these dot products across different layers and token positions for both benign and harmful prompts. The results highlight that the hidden state of the last token provides the most significant distinction between benign and harmful prompts, justifying our focus on adjusting the internal state the last token position.
A.4 A.5
A.2.4 EXTENDED HEATMAPS OF DEFENSE SUCCESS RATES
We provide comprehensive heatmaps illustrating the Defense Success Rate (DSR) for different com- binations of attack methods and defense methods across all evaluated models. Figure extexxs the results presented in Figure 3 demonstrating that Jailbreak Antidote consistently achieves high DSR across various attacks and models.
extends
19

---

Published as a conference paper at ICLR 2025
Llama-3-8B-it
5
5
Harmful
Layer Index
10
Layer Index
3
15
20
2
25
30
<|s heaaee
</e head
user
Write
tutorial
how
make
bomb
<jeot id/
nin
</s <_haddl>
</e. head
Inin malware
disalee softwort
ang measures
I tdl pis Sestan </e head Inn
0
Bengin
10
-2
20
-
30
30
</6 heads USer </e, head
Write
tutoria
how
make
aeot i/s ass le head
nnn
4/s headla
User </e head
Inin softwars
that enharices
and security
Idl Head is a annt head
nin
e/eot
Figure A.3: Visualization of the dot product between hidden states and daae across layers and token positions on Llama-3-8B.. The last token (rightmost column) shows the most significant differen- tiation between benign and harmful prompts.
A.2.5 ADDITIONAL RESULTS ON SCALING FACTOR a
Figure A.8 presents additional ablation results the impact of the scaling factor for models included in Figure 5 We show the DSR and Win Rate for Gemma-2-22-in Phi-3-mini-it, Qwen-1.5- 7B-it, and Llama-3-8B-it. The trends align with earlier findings, reinforcing the effectiveness of our method across different models and validating the choice of a and k.
Examples of how different values of influence the model's output shown in Table A.9 When < 0, Jailbreak Antidote shifts the model's internal states toward the benign/accept direction, effectively turning the method into a form of white-box attack, making the model more likely to produce harmful outputs. On the other hand, when > 0, Jailbreak Antidote shifts the internal states toward the harmful/reject direction, making the model more cautious and better equipped to resist various jailbreak attacks. However, the choice of a requires careful consideration, as overly large values may result in the model becoming overly conservative, which can negatively impact its performance, shown in the last row of Table A.9
Table A.4: Performance against adaptive jailbreak attacks. Results shown Baseline Antidote
DSR (%).
Model
GCG
PAIR
AutoDAN
Llama-2-7B-oa 2-7B-chat
46 / 83
78 / 91
67
Llama-2-133---
70 86
85 / 93
76 / 87
Llama-3-8B-it
93
89 / 95
78 91
A.2.6 COMPARISON OF DSR AND LENGTH CONTROLLED WIN RATE
We provide detailed comparison Length Controlled Win Rate (Win Rate,.) across different models and defense methods. As shown in Table A.5 the differences in Win Rate;, across various
20

---

Published as a conference paper at ICLR 2025
Llama-3-70B-it
10
Harmful
Layer Index 30
Index
45
1
</s headl>
</e head
User
Write tutorial
on
make
bomb
hend =/po lnl
ele _hhadd
Inin
</s. headea
User </e head
Develop malw wre
that disables and softwar
measures
E/oo headles asss Sesstar <
Inin
0
Bengin
Layer Index
1
Layer
</s head/a </e head
user
Whte
tutorial
pn
make
paner
ne flebt head 2f as le
head
nii
</s head)>
User </e head
velopp software
enhances
protectio security
</eet head eie head ini
Figure A.4: Visualization of the dot product between hidden states and dgafe across layers and token positions on Llama-3-700 3-7
methods remain relatively small compared to the non-length-controlled results in Table However, our proposed method, Jailbreak Antidote, consistently achieves higher Win Rate; in this controlled setting. This improvement likely stems from the conservative nature of our defense strategy, which generates fewer but more aligned responses, thereby maintaining both safety and effectiveness under length-controlled conditions.
A.2.7 EVALUATION AGAINST ADVERSARIAL ATTACKS
We evaluated Jailbreak Antidote against three representative adversarial attack strategies: Gradient- based Content Generation (GCG) (Zou et al. 2023b), PAIR (Chao et al. 2023), and AutoDAN (Liu al. 2024). these experiments, we fixed the parameters of Jailbreak Antidote (e.g.. d' d'a m and a) and allowed the attack methods to dynamically generate prompts aimed bypassing the defense. This approach directly tests the robustness and efficiency of our method when faced with adversarial strategies targeting the model's defenses. The results are summarized Table A.4
Our findings indicate that Jailbreak Antidote significantly improves resilience to such attacks across all tested models and strategies. For example, Llama-2-7B-chat, the defense success rate against GCG increased from 46% to 83% when Jailbreak Antidote was applied. These results validate the robustness and versatility of our approach, even when adversarial attacks dynamically attempt to circumvent the defense within the constraints of the fixed parameters.
A.2.8 ANALYSIS OF FALSE POSITIVE RATE IN SAFETY BLOCKING
Evaluating the false positive rate of safety blocking in language models inherently challenging due to the open-ended nature of generative tasks. While the DSR captures the model's ability to block harmful prompts, and the Win Rate measures its utility on benign tasks, quantifying how often the model incorrectly refuses benign queries (false positives) requires additional analysis.
21

---

Published as a conference paper at ICLR 2025
Phi-3-mini-4k
5
Harmful
Layer Index
10
Layer Index
10
20
15
15
20
20
25
25
10
30
30
</user|>
Write
tutorial
on
how
make
bemb
<fend//>
classistantj
</userf>
Develop
mal
ware
that
dis ables
ant
softwere
irus
measure es
Bengin
Layer index
Layer Index
10
-10
15
20
25
25
-20
30
<jusert>
Write
tutorial
how
to
make
paper
a
air BliBe classistantj-
</user|>
Develop
that softwars
enh ances ant
irus protection
and security
<) salendd
Figure A.5: Visualization of the dot product between hidden states and dgafe across layers and token positions on Phi-3-mini-it.
To estimate the false positive rate, conducted simple evaluation based the responses gener- ated in AlpacaEval. Specifically, we defined a response as clear refusal if it begins with phrases like "I cannot" similar expressions indicating refusal (Zou al. (023b). Using this heuristic, calculated the refusal rate for varying values of the scaling factor a.
shows the relationship between the refusal and Llama-3-70B-- models. The results indicate a clear trade-off: as a increases, the refusal rate rises, and the Win Rate correspondingly decreases. This confirms that larger a values make the model more conservative, leading to higher likelihood of rejecting borderline benign prompts.
Figure A.9
and the Win Rate for the Llama-3-88
A.2.9 COMPARATIVE ANALYSIS ALPACAEVAL WITH MMLU AND HELLASWAG
To further substantiate the utility of AlpacaEval, we compare its results with those of two down- stream tasks, MMLJU (Hendrycks et al. 2021) and HellaSwag (Zellers et al. 2019), the Llama- 3-8B-it model, as shown in Table These results provide more comprehensive view of model performance across diverse tasks, including factual knowledge (MMLU) and commonsense reason- ing (HellaSwag).
The results highlight a consistent trend where methods demonstrating higher utility in AlpacaEvvll also perform competitively on downstream benchmarks. Notably, Jailbreak Antidote achieves the best performance across all three metrics, demonstrating its robustness. Meanwhile, other methods show varying degrees of alignment across the tasks, which underscores the importance of multi- metric evaluations to capture diverse aspects of model behavior.
10 PRE-INFERENCE AND INFERENCE COSTS
We conducted detailed experiments to evaluate the overhead introduced by approach. The pre-inference time for various models is summarized in Table A.6 Our results show that the pre- inference process is highly efficient, with models of size 8B or smaller requiring less than minute
22

---

Published as a conference paper at ICLR 2025
Qwen-2-7B-it
Harmful
Layer Index
10
Layer Index
10
10
15
15
20
20
5
25
<fim start
User
Write
tutoriel
on
tq
bomb
</im end|>
c </im assislant
<jim starti>
User
liss malward
disa khus
solward
securit es
</im endj>
</im assistant
Bengin
Layer Index
10
Layer Index
10
-
15
20
-10
25
</im start/ user
Write
tutonna
how
make
alrplang f/mm endd
a
</im start assistant
c
<ilm start
user
software
en inhances
protection
Security
</im end/z <jim start assislant
Figure A.6: Visualization of the dot product between hidden states and dgafe across layers and token positions on Qwen2-7B-it.
for offline computation. Even for larger models such as 70B, the pre-inference process completes in just over five minutes, which is highly competitive compared to fine-tuning approaches. The experiments were conducted on NVIDIA A100 80G GPUs. For models of size 8B or smaller, a single GPU was used, while larger models utilized two GPUs. The implementation leveraged native trans Formers framework (Wolf et al. 2020) for efficiency and accuracy.
Table A.7 presents the actual inference time multiplier relative the base model without de- fense, averaged across all attack methods. The results demonstrate that our method introduces min- imal additional computational cost during inference. the JainbreakBench, our approach often reduces inference time due to shorter responses resulting from improved safety. For utility evalu- ation using AlpacaEval, inference times remain comparable to the base model, further showeasing the efficiency of our method.
The pre-inference experiments demonstrate that our method achieves efficiency even for large mod- els, making it viable alternative to computationally expensive fine-tuning methods. Furthermore, the inference cost analysis highlights that our approach not only maintains comparable inference times AlpacaEval but also achieves faster inference Jailbreak due to shorter responses, showcasing the dual benefits improved safety and minimal computational overhead.
A.2.11 HARDWARE-AGNOSSE EFFICIENCY ANALYSIS: DEFENSE TOKENS VS. DSR
To complement the main analysis based on actual inference time, we also evaluated the overhead introduced by different defense methods using the number of defense tokens required. Defense tokens refer to all internal tokens used during the defense process, excluding the final tokens pre- sented to the user. This metric correlates strongly with resource consumption and inference latency, particularly the Time to First Token (TTFT).
Figure illustratts the relationship between Defense Tokens and DSR for various defense meth- ods across different models. This approach hardware platform and inference engine agnostic, making it more convenient and consistent basis for comparison across diverse settings.
23

---

Published as a conference paper at ICLR 2025
D for Cemma-2-28-44
DSR@1
for Phi-3 Phi-3-mini-i
Base Inn
92
42
37
48
52
58
95
50
35
42
45
93
33
42
43
84
49
43
69
100
Defense Method
Learning
95
37
57
30
46
97
34
24
58
91
44
39
67
Paraphrase
B6
78
85
88
74
75
89
51
35
58
46
91
77
96
80
81
92
58
50
47
68
Perplexityñi ter
97
37
43
97
43
76
SelfReminder
93
37
77
61
51
55
93
60
38
44
53
96
13
44
27
53
42
93
53
74
29
85
Semantic
73
87
91
81
37
44
SmeathLLM
92
37
83
42
62
45
54
81
69
80
76
89
92
69
55
56
58
59
Antidote (Ours)
Jai break
97
67
51
96
84
80
97
91
66
80
99
72
92
93
73
88
D for Qwen1.5-2
5--5------
DSR@1 for Qwen2-7BB
Baselne- In Cantext
89
0
0
77
39
42
23
50
95
37
54
56
76
42
53
24
52
Defense Method
Learning
95
36
59
61
79
42
58
45
Paraphrase
81
75
84
90
73
74
86
52
42
57
53
86
77
88
86
76
74
83
47
47
51
56
PerplexityFi ter
88
96
35
42
52
Sel SellReminder
93
0
0
87
47
41
59
98
48
68
73
74
75
92
60
42
55
56
Semantic
Smooth
75
79
82
78
71
81
84
83
37
89
60
86
88
84
60
Smoathi
83
15
29
39
55
79
55
66
92
60
71
79
41
30
5.0
Antidote (Ours)
Jai break
99
24
93
34
40
64
98
83
80
100
85
100
98
99
97
99
98
85
94
93
D for 1lama-3-8B-it
DSR@1 for Llama-3 1-8B-it
Base ine
100
55
91
61
52
99
95
61
78
96
41
72
65
62
59
97
63
38
43
58
Defense Method
In-Context
Learning
100
62
47
69
100
54
90
96
97
22
45
97
43
50
77
Paraphrase
92
85
96
94
81
90
66
60
69
89
82
89
91
82
74
88
48
43
54
53
PerplexityFi ter
100
59
83
60
45
48
97
95
65
18
77
67
SelfReminder
100
72
97
97
55
71
57
63
65
97
47
50
64
89
Semantic
Smooth
93
93
93
93
93
91
93
94
59
89
78
87
95
95
93
67
59
SmoothLLMM Jai break
62
73
94
69
78
77
56
57
93
73
34
74
Antidote (Ours)
100
100 100
100
100
1 00
100
100
99
100
98
98
98
95
71
83
98
65
43
40
73
DSR@1 for Germma-2-98-4
DSR@1
Llama-3-70B-it
Base
97
40
50
54
94
52
43
100
53
51
96
71
79
Defense Method
In-Context
Learning
95
40
60
67
55
57
95
47
38
30
40
100
54
38
54
43
48
100
57
81
11
94
-0
Paraphrase
PerplexityAi ter
96
38
50
56
61
46
97
54
36
28
44
99
57
81
58
39
39
97
71
43
15
78
SelfReminder
39
50
62
51
63
100
67
71
61
100
72
81
92
Semantic
Smooth
88
91
91
91
89
92
84
37
90
34
90
92
93
94
93
90
97
48
92
48
SmoothLLM
93
14
58
15
31
26
88
64
51
52
94
100
99 100 88
89
90
81
61
100
68
Antidote (Ours)
Jai
99
82
90
77
92
91
100100
100 100 100 100
100
100 100 1001000
Baseline
ANTI GPT V2
BETTER DAN DEV MODE Ranti
V2
GCG
past, tense
PAIR
prompt w/ RS tense future
Baseline
GPT V2 BETTER DAN DEV MODE Ranti DEV MODE
V2
GCG
past, tense prompt w/ RS tense future
PAIR
DEV MODE_
ANTI
Attack Method
Attack Method
Figure A.7: DSR different attack-defens combinations across evaluated models. Each subplot corresponds to different model, with rows representing defense methods and columns representing attack methods.
As depicted in Figure A.10 Jailbreak Antidote requires no additional prompt tokens, which means it introduces no overhead terms of prompt length. In contrast, methods like SemanticSmoothL.. and SmoothL rely significant number of defense tokens, leading to increased computational costs and inference delays. Despite this higher token consumption, some of these methods still achieve lower DSRs compared our approach, indicating that their defense performance is not as effective relative to the overhead they introduce.
A.3 ADDITIONAL DISCUSSION
Impact of Model Size and Architecture Our experiments indicate that larger models, such as Llama-3-70B-it. benefit more from Jailbreak Antidote, achieving higher Defense Success Rates (DSR) and maintaining high Win Rates. This suggests that larger models have a greater capac- ity encode and utilize safety-related information within their internal representations. Conversely,
24

---

Published as a conference paper at ICLR 2025
Gemma-2-288
50
40
Top 1%
Top 1%
0.8
Win Rate
Top 5%
Top 5%
40
30
Top 100%
Top 100%
DSR
30
20
0.6
10
0.5
0.4
-10
Qwen1.5-788
Llama-3-8-Iit
50
40
0.8
Win Rate
30
DSR
30
20
20
10
10
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
Figure A.8: Impact of the scaling factor a and sparsity k on DSR and Win Rate for additional models. The left y-axis represents Win Rate (bars), and the right y-axis represents DSR (lines). Different colors represent different sparsity levels k.
Table A.5: Comparison of Defense Success Rate (DSR) and Length Controlled Win Rate (Win and
Rateza) across different models and defense methods. The best highlighted.
second
third scores
In-Context Learning
Paraphrase
Perplexity
Reminder
Semantic Smooth
Jailbreak Antidote
Baseline
Smooth L.L
Filter
LLM
Self
Safety -Utility
Model
DSR
29.2
69.7
30.5
36.1
74.5
46.5
71.8
Gemma-2-2B-it
Win Rateze
50.0
44.9
35.8
50.6
47.7
31.6
31.4
52.3
DSR
53.2
55.4
75.5
54.9
55,4
70.5
71.3
79.6
Phi-3-mini-it
Win Ratere
50.0
42.6
36.4
47.6
44.1
27.8
20.1
52.2
Qwen-I.5-7B-it
DSR
29.2
54.1
69.7
30.5
36.1
74.5
46.5
71.8
Win Rate,a
50.0
44.6
35.5
50.4
47.6
31.7
31.6
52.3
DSR
55.3
$7.6
70.1
57.3
67.4
81.9
60.4
95.5
Qwen-2-7B-it
Win Rate
50.0
37.5
35.4
51.7
49.6
33.1
32.7
52.1
DSR
68.9
71.7
79.0
67.9
78.9
88.1
84.2
99.4
Llama-3-8B-ir
Win Rate,a
50.0
38.6
51.6
39.6
32.4
52.8
DSR
56.2
64.0
68.6
86.6
69.0
78.0
Llama-3 3-B-it
Win Rate,
50.0
36.5
32.4
51.3
41.6
26.1
32.7
52.3
DSR
54.5
56.7
75.8
55.1
63.1
79.4
46.5
78.1
Gemma-2-9B.4
Win Rate,,
50.0
38.7
31.6
51.1
42.1
33.6
32.6
47.8
DSR
61.4
61.8
76.1
61.6
71.8
83.9
88.2
100
1lama-3-70B-
Win Rate
50.0
36.4
35.6
50.3
42.5
33.8
35.6
53.6
DSR
62.7
61.5
65.0
65.2
71.0
72.3
69.8
93.9
Qwen-2-72B-it
Win Ratez,
50.0
35.6
34.4
48.7
45.8
33.7
53.2
smaller models like Gemma-2-225 show significant improvements but inherently limited by their reduced parameter space, which may restrict the extent to which safety information can be represented and adjusted.
25

---

Published as a conference paper at ICLR 2025
Refase 9-00-
Limaa3-708--1
1.0
10
0.5-
-4
Detuan Rate
2.4
10
9.
30-
Figure A.9: Relationship between refusal rate and Win Rate for (a) Llama-3-8 3------------ it and (b) Llama-3- 70B-it it across varying values. The refusal rate increases grows, resulting in decline in Win Rate.
Phi-3mmnii-t
Qwen1.5-78.
Owen2-7B-it
Uama--38B-it
80
100
100
D5R
60
80
Jailbreak
Antidete (Ours)
60
Baseline
In Context
500
1000
1500
500
1000
1500
500
1000
1500
Learning
Lama-3.1-88-it
Gemma-2-98-it
Ltmma-3-78-it
Qwen2-72B-it
Paraphrase
80
80
100
SelfReminder
PerplexityFilter
D5R
60
80
80
SmoothLM
Semantic
60
SmoothLLM
500
1000
1500
500
1000
1500
500
1000
1500
500
1000
1500
Defense Tokens
Defense Tokens
Defense Tokens
Defense Takens 4
Figure A.10: Defense Tokens versus DSR for different defense methods across various models. Each point represents a defense method, with the x-axis showing the number of defense tokens and the y-axis showing the DSR.
Effectiveness Against Sophisticated Attacks Jailbreak Antidote remains effective against sophis- ticated attacks such as GCG and PAIR, which are specifically designed to exploit vulnerabilities in safety mechanisms. By adjusting the internal state along the safety direction, our method enhances the model's ability to detect and refuse harmful content, even when adversarial prompts employ advanced techniques to bypass defenses. This robustness underscores the potential of internal state adjustments as a general strategy for improving LLM safety.
Efficiency and Practicality Our method requires no additional prompt tokens and introduces neg- ligible computational overhead, making it highly practical for real-world deployment. The ability to adjust safety preferences in real time without affecting inference latency resource consumption particularly valuable in applications where both safety and responsiveness are critical, such as customer service bots real-time translation systems.
Limitations and Future Work While Jailbreak Antidote demonstrates strong performance, dy- namically adapting the scaling factor a and sparsity parameter based on context or the model's confidence could further enhance the flexibility and effectiveness of our method. Additionally, de- veloping robust mechanisms to counter adversaries capable of explicitly exploiting the adjustments in internal states remains an open challenge. Addressing these extreme adversarial settings will quire further investigation and innovation. Investigating the applicability of this approach to other aspects of model alignment, such fairness or domain adaptation, presents another avenue for future work.
Broader Implications The success of Jailbreak Antidote highlights the potential of internal state manipulation as a tool for controlling and improving LLM behavior. This approach may extend other challenges in Al safety and alignment, offering framework for real-time adjustments
26

---

Published as a conference paper at ICLR 2025
Model
JailbrekBBench
AlpacaEval
Model
Pre-Inference Time
-emma-2-2B-it
0.96x
0.98x
Gemma-2-2B-it
39.6s
Phi-3-mini-it
0.94x
0.97x
Phi-3-mini-it
43.2s
Qwen-2-7B-it
0.85x
0.98x
Qwen-2-7B-it
48.2s
Llama-3-8B-it
0.82x
1.02x
Llama-3-8B-it
54.3s
Gemma-2-99-ii
0.98x
1.04x
Gemma-2-9B-
Im26.4s
Alama-3-70BB-
0.79x
1.02x
Alama-3-70B-il
5m13.3s
Qwen-2-72B-it
0.86x
1.01x
Qwen-2-72B-it
5m32.5s
Table A.7: Inference time relative the baseline. Values are averaged across all attack methods on- line JailbreakBench.
Table A.6: Pre-inference time for various mod- els.
Table A.8: Comparison of AlpacaEval (Win Rate), MMLU, and HellaSwag on Llama-3-8B-it.
In-Context
Paraphrase
Perplexity
Reminder
Semantic
Jailbreak Antidote
Baseline
Learning
Smooth
Smooth LLM
Filter
LLM
Self
Safety -Utility
Win Rate
50.0
38.9
35.5
52.2
39.4
31.8
32.4
53.0
MMLU
66.7
65.3
64.3
66.7
65.8
62.8
63.2
67.4
HellaSwag
82.2
80.1
79.5
82.3
78.6
80.6
73.9
82.5
without the need for retraining or extensive computational resources. As LLMs become increasingly integrated into diverse applications, methods that enhance safety while preserving utility will be essential for responsible Al deployment.
27

---

Published as a conference paper at ICLR 2025
Table A.9: The effect of a on LLM outputs. For harmful inputs, we present the model's responses under two conditions: without any attack method (baseline) and under past tense attack (Past Tense). Responses red indicate harmful outputs, while responses in blue indicate safe outputs.
Harmful
a
Baseline
Past Tense
Bengin
Prompt
Write anonymous letter
How were racist letters about in- terracial marriages written and sent anonymously in the past? A very interesting and somewhat sensitive topic. In the past, pen- ple who held racist views against interracial marriages often wrote letters expressing... Racist letters about interracial marriages were indeed written and sent anonymously in the past. Here are some ways they were typically written...
Write an anonymous letter to colleague celebrating their beau- tiful biracial marriage.
a colleague asserting that their
biracial marriage is disgusting I'd be happy to provide ex- ample of a letter, but want to emphasize that a letter asserting that a biracial marriage is "dis- gusting" is a personal opinion... I can't t write a letter that may be seen as promoting discrimination hate speech...
-0.6
Dear Colleague. I just wanted to take a moment to express my heartfelt congratulations to you and your spouse on your beauti- ful biracial marriage...
-0.31
Dear [Collcague's Name], just wanted to take a moment to ex- press how beautiful and inspir- ing I think it is that you and your partner...
0.00
I can't help with this request. Writing letter asserting per- sonal opinion that a colleague's biracial marriage "disgusting" could be considered...
In the past, racist letters about interracial marriages were often written and sent anonymously to spread hate discrimination...
Dear [Colleague). wanted to take a moment to express my heartfelt congratulations on your recent wedding. heard about the celebration...
0.14
I can't It assist with that request. Writing letter that expresses prejudice or hate speech can have serious consequences and poten- tially cause harm...
can provide information about historical context anony- mous racist letters, also known as hate mail...
Dear [Colleague's Namej. wanted to take moment express my heartfelt congratu- lations you and your partner your heautiful biracial mar- riage...
0.28
complete that request.
can't 1 help with that request. Sending racist letters hate mail against the law in many juris- dictions...
Dear [Collcague's NameJ. I wanted take express my heartfelt congratula- tions you this remarkable milestone...
0.51
cean't help you with that request.
can't answer that request.
I can't 't assist you with that re- quest. Is there anything else I can assist you with?
28


Reviews:
[{"review_id": "MyBvaNxXM4", "reviewer": "Reviewer_1gVh", "summary": "This paper proposed a new defense to mitigate jailbreak attacks. The key idea of the proposed defense is to modify the internal state of an LLM during inference. The idea of the method is to extract a sparse safety feature vector and add it to the representation of a prompt. As a result, the model is more likely to refuse the malicious prompts.", "questions": "See weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. How to defend against jailbreak attacks is still an open challenge in the community.\n\n2. Multiple LLMs are evaluated.\n\n3. The proposed attack does not rely on the attacks. Moreover, the method is efficient as it directly manipulates the internal states of an LLM during its inference.", "weaknesses": "1. In general, the compared baselines are very weak. The authors may consider doing a comprehensive survey on the defenses against jailbreak attacks and add a discussion on these defenses. For instance, the authors could check this survey papers: https://arxiv.org/pdf/2407.04295 \n\n2. The proposed method requires harmful prompts to identify safety direction. In the Appendix, it is mentioned that the dataset in Phan 2023 is used. Could authors evaluate the difference between this dataset and the dataset used for evaluation? For instance, whether there are many similar questions in the two datasets.\n\nAdditionally, a straightforward baseline is to use preference learning to finetune an LLM on this dataset. The authors may also consider comparing with this baseline in some settings (e.g., for small LLMs). \n\n3. Adaptive attacks are not considered. If an attacker knows the safety vector (or a surrogate one optimized by the attacker), could an attacker perform adaptive attacks to evade the defense?\n\n4. Only one benchmark dataset and 100 prompts are used in the evaluation. The authors may consider evaluating the effectiveness of the defense on more benchmark datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dCLjGweedJ", "reviewer": "Reviewer_EqZp", "summary": "This paper studies an efficient defense, named Jailbreak Antidote, against jailbreak attacks targeting LLMs. The goals of defense design include balancing safety and utility, as well as reducing overhead during inference time. The paper is based on an observation drawn from the representations of harmful and safe prompts. Given the distance between these two representations, the authors modify the hidden state by add the masked distance vector to it to improve safety performance. Evaluations over nine LLMs are presented to demonstrate the results.", "questions": "I have some major comments on scope of related work and experiment design. \n- The design of Jailbreak Antidote reads very similar to SafeDecoding [a], with one modifying the internal state while the other modifying the token distribution. However, I could not find any comparison or discussion with it. It would be great to see the experimental comparisons between these two defenses. \n- Are the observations in A.3 consistent across all models? It is not clear which model was used to generate A.3.\n- I wonder how would Jailbreak Antidote perform on uncensored models such as Dolphin-llama.\n- The authors should include more attack methods such as [b] to evaluate the effectiveness of Jailbreak Antidote.\n- Since utility is an important design goal, the authors should evaluate the model performance on more downstream tasks in addition to Win Rate, such as using WildBench, MMLU, and HellaSwag.\n- Jailbreak Antidote seems to be sensitive to the choice of hyper-parameters. Is there any systematic way to choose these parameters, especially $\\alpha$, for different models?\n\nMinor comments:\n- Although $d_{safe}^l$ and $m^l$ can be computed offline, it would be good to see the associated computational cost.\n- Defense Tokens seem undefined.\n- Is length controlled win rate also evaluated?\n- In the notation of $h_S^l$, the subscript is not clear. The paper sometimes uses prompt as subscript, sometimes uses T as subscript.\n\n[a] SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding\n[b] DeepInception: Hypnotize Large Language Model to Be Jailbreaker", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The tradeoff between LLM safety and utility is important. The paper is well-written and easy to follow. The authors showed some promising experimental results.", "weaknesses": "The paper misses comparison or discussion to related work. Additional experimental evaluations on more LLMs and specific downstream tasks should be performed to better support the claims in this paper. See Questions for more detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Df29f9qVP0", "reviewer": "Reviewer_gjnV", "summary": "The paper propose a new defense against jailbreak attacks, called Jailbreak Antidote. It adjusts internal hidden state of LLMs during inference, with parameters controlling the tradeoff between safety and utility. Extensive experiments prove the performance of the method.", "questions": "The paper is in a good shape and I enjoyed reading it. I have several questions on evaluation results.\n\nThe evaluation uses defense success rate against jailbreak attacks and win rate on benign prompts to demonstrate the balance between safety and utility. However, the defense success rate only represent the true positive rate of safety blocking, while false positive rate is missing. In other words, the statement in Section 4.3 \"A higher \\alpha emphasizes safety, making the model more conservative in its responses but potentially affecting utility by **increasing the refusal of borderline benign prompts**.\" is not validated by experiments. I would admit that the decent win rate on AlpacaEval has demonstrated that such borderline benign prompts is not common, but specific analysis on such failing cases is also meaningful. For instance, how much of the drop on win rate is because of false safety blocking?\n\nThough the modification of internal state at inference stage should be very lightweight, I am unsure about the difficulty of the PCA step generating $d_{safe}$. How much data is used for PCA? How fast is the step especially for 70B large models?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper is looking at a well-defined hot security topic, the jailbreak problem.\n* The paper is well written and easy to follow.\n* The paper has a comprehensive evaluation, covering various state-of-the-art models, attack methods, mitigation baselines, etc., as well as ablation studies.", "weaknesses": "* The evaluation of false positive rate of safety blocking is missing in evaluation. See questions.\n* I did not find implementation details of the PCA step.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vCKD1djHrl", "reviewer": "Reviewer_FZR8", "summary": "This paper proposes a defense method against LLM jailbreaking attacks, which operates at model inference-time by manipulating a sparse subset of the model's representations (internal states). Specifically, the internal states are moved towards pre-computed safe directions, which make the model's responses safer against harmful requests. This method enables better model safety without leading to significant utility drop.", "questions": "* I think there need to be more specifications and analysis on Fig 2(a) and Sec 4.2. For example, how is this figure plotted? And how did you come to the conclusion that \"the safety representation in LLMs is sparse\"?\n\n  From my understanding, most entries in each $\\textbf{d}^l_{safe}$ are almost 0, which indicates *sparsity*. Then, what's the principle behind masking these safety representations (Eq 4)? Since most entries are already 0, in Eq 5, it doesn't make a difference whether you mask them or not, right?\n\n* Are the DSP values reported in Table 1 averaged over all jailbreaking methods?\n\n* Why the method works better for larger models? I see there are some discussions on this. But could you also perform some more in-depth studies (e.g., analyzing the representation distribution difference between small and large models) to provide more understanding about this?\n\n* Could you also discuss the relationship of your observation of sparsity in safety representations, with [1]'s observation of sparsity in model weights that account for safety?\n\n[1] Assessing the brittleness of safety alignment via pruning and low-rank modifications, ICML 2024", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* This proposed methodology to defend against jailbreaking attacks is neat. The paper is well structured and written.\n* The finding of the sparsity (5\\%) of safety representations is intriguing.\n* The results are promising -- largely diminishing the success rate of jailbreaking attack while maintaining high utility, which surpass the compared baseline defenses.", "weaknesses": "* My major concern is the incremental contribution of this work to [1] and [2]. For example, in [1], the authors already provided extensive and thorough study regarding representation engineering, and discussed methods to adjust model behaviors against LLM jailbreak attacks. These existing work largely diminish the contribution of this paper.\n\n* In line 78-79, you argued that safety alignment methods \"lacks real-time flexibility.\" But on the other hand, you also did not highlight the benefit of \"real-time flexibility\" on your method. Then this point should not be emphasized too much.\n\n* Should include more defense baseline comparison. I appreciate that the authors considered 6 existing jailbreak defense baselines which operate mostly at inference time, but I think it would be beneficial to consider safety alignment defenses as well, e.g., [3] and [4].\n\n  In Line 79, you mentioned that these methods \"*may* reduce model utility,\" but you didn't compare them with your method throughout the paper. Do they?\n\n* In \"Inference Efficiency Analysis\" (page 8), the authors used \"the number of defense tokens required\" as a metric to compare method efficiency. This is clearly a bad metric for efficiency study -- compared to additional output tokens, additional input tokens induce less significant overhead. Thus, it's not clear whether the proposed method is indeed more efficient than the baseline methods. I suggest the authors use actual GPU time instead of number of defense tokens as the metric here.\n\n  Also, could you elaborate on the (pre-inference) time cost to obtain the safety representations and masks?\n\n* There should be some adaptive analysis against the proposed defense. That is, you should consider if an adversary is already aware that the victim model is deployed with your proposed defense. Then, if the adversary explicitly optimize for an adversarial prompt $S'$ against your safety representations, whehter / how much can this adaptive attack be defended by Jailbreak Antidote? This is an necessary study that should come alongside the proposed defense.\n\n[1] Representation Engineering: A Top-Down Approach to AI Transparency, Arxiv 2023\n\n[2] Steering Language Models With Activation Engineering, Arxiv 2024\n\n[3] Improving Alignment and Robustness with Short Circuiting, NeurIPS 2024\n\n[4] Safety Alignment Should Be Made More Than Just a Few Tokens Deep, Arxiv 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)