Improving Alignment and Robustness with Circuit Breakers
Authors: Andy Zou
Long Phan
Justin Wang
Derek Duenas
Maxwell Lin
Maksym Andriushchenko
J Zico Kolter
Matt Fredrikson
Dan Hendrycks
Venue: NeurIPS 2024 poster (2024)
Keywords: alignment
adversarial robustness
adversarial attacks
harmfulness
security
reliability
ML safety
AI safety
Primary area: safety_in_machine_learning
TL;DR: We propose a novel approach that "short-circuits" harmful outputs in AI systems by directly controlling the responsible representations, offering robust protection against harmful actions and adversarial attacks without compromising utility.

Abstract: AI systems can take harmful actions and are highly vulnerable to adversarial attacks. We present an approach, inspired by recent advances in representation engineering, that interrupts the models as they respond with harmful outputs with "circuit breakers." Existing techniques aimed at improving alignment, such as refusal training, are often bypassed. Techniques such as adversarial training try to plug these holes by countering specific attacks. As an alternative to refusal training and adversarial training, circuit-breaking directly controls the representations that are responsible for harmful outputs in the first place. Our technique can be applied to both text-only and multimodal language models to prevent the generation of harmful outputs without sacrificing utility -- even in the presence of powerful unseen attacks. Notably, while adversarial robustness in standalone image recognition remains an open challenge, circuit breakers allow the larger multimodal system to reliably withstand image "hijacks" that aim to produce harmful content. Finally, we extend our approach to AI agents, demonstrating considerable reductions in the rate of harmful actions when they are under attack. Our approach represents a significant step forward in the development of reliable safeguards to harmful behavior and adversarial attacks.

Improving Alignment and Robustness with Circuit Breakers
Andy Zou 1,2.3, Long Phan', Justin Wang', Derek Duenas', Maxwell Lin', Maksym Andriushchenko', Rowan Wang', Zico Kolter 1.2 Matt Fredrikson 11.2, Dan Hendrycks 1.3
Gray Swan AI, Carnegie Mellon University, Center for Al Safety
Abstract
AI systems can take harmful actions and are highly vulnerable to adversarial attacks. We present approach, inspired by recent advances in representation engineering, that interrupts the models as they respond with harmful outputs with "circuit break- ers." Existing techniques aimed improving alignment, such refusal training, are often bypassed. Techniques such as adversarial training try to plug these holes by countering specific attacks. As alternative refusal training and adversarial training, breaiit directly controls the representations that are responsible for harmful outputs in the first place. Our technique can be applied both text-only and multimodal language models to prevent the generation of harmful outputs without sacrificing utility-even in the presence of powerful unseen attacks. Notably, while adversarial robustness in standalone image recognition remains an open challenge, circuit breakers allow the larger multimodal system to reliably withstand image "hijacks" that aim to produce harmful content. Finally, we extend our approach Al agents, demonstrating considerable reductions in the rate of harmful actions when they are under attack. Our approach represents a significant step forward in the development of reliable safeguards to harmful behavior and adversarial attacks. Code is available at github.com/GrayS wannSSaaAAAi AatrA//citt--baaaee..
1 Introduction
The landscape of artificial intelligence (AI) has long been marred by the persistent threat of adversarial attacks, particularly those targeting neural networks. These attacks exploit inherent vulnerabilities within AI systems, often leading to compromised outputs and raising concerns regarding their reliability and safety. Despite significant attention, existing mitigations have failed to achieve high reliability without dramatically compromising model performance. Thus, the trade-off between adversarial robustness and utility widely accepted an unavoidable fact [64].
The rise of generative models has further complicated this issue. Generative models such as large language models (LLMs) can output copyrighted information defame individuals, and agents can take harmful actions. To make models less harmful, they are "aligned" with refusal training [12, 54], but it has become common to adversarial attacks means of bypassing their safeguards. In these settings, vulnerability to attacks that break alignment poses a serious threat to utility, and raises pressing questions about whether it feasible deploy such systems with high standard of safety and reliability-eseciie
against dedicated adversaries who intend to misuse them.
The fragility of alignment techniques to sophisticated attacks has motivated defenses that target specific attack methods, such as adversarial training, an approach originally proposed in the context
Work done while Gray Swan AI. Work done advising capacity to Gray Swan Correspondence to: andy @ grayswan.ai
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Harmless States
Harmful States
Refusal States
=
Circuit Breakers
Instruct Model
Refusal Training
Circuit Breaking
Prompt:
Tell me how build bomb ADVTEXT
Generation:
Generation:
Generation:
Sure, here's how build bomb:
Sure, here's how to build bomb: don't recommend this action it
Sure, here's how make bomb: 1. Start with <EOS> <EOS> <EOS>
Step Gather necessary materials.
These may include items such as pressure cooker, explosives, a timer..
highly illegal, but one can start by
gathering necessary materials such..
novel approach for constructing highly reliable safeguards. Traditional methods like RLHF and adversarial training offer output-level supervision that induces refusal states within the model representation space. However, harmful states remain accessible once these initial refusal states are bypassed. In contrast, inspired by representation engineering [77], circuit breaking operate directly internal representations, linking harmful states to circuit breakers. This impedes traversal through a sequence of harmful states.
Figure 1: Introduction of circuit-breaking
of standalone image classification [38] and later adapted to LLMs [40]. However, these methods often fail to generalize new attacks that were unseen during training, and they introduce penalties on model capabilities that are usually proportional to gains in robustness. System-level defenses, including input and output filters, are cumbersome, ussurcceinttenivee and often remain vulnerable to adversarial techniques. This has led to growing concern that robust defenses may be unattainable.
We propose novel approach outlined in Figure that fundamentally diverges from traditional defenses: instead of attempting to remove vulnerabilities to specific attacks, our approach aims to directly cireumvent the ability the model to produce the harmful output in the first place. With circuit breakers, we make models intrinsically safer and reduce their risks by removing intrinsic model hazards their ability to produce harmful outputs adversarial training, and rather than attempting to reduce exposure to attacks with input filters [28, 21]. Using representation engineering (RepE) [77], our method connects the internal representations related to harmful outputs to circuit breakers so that when model begins to generate such an output, its internal processes are interrupted, halting completion of the generation. Or this method is "short- circuiting" the harmful processes as one might put it. Because the representation used to generate harmful output is independent of any attack capable of eliciting it, this approach and sidesteps the need for additional training, costly adversarial fine tuning, or the use of auxiliary "guard" models. Consequently, the resulting model with circuit breakers can be used normally without additional computational burden, and scamlessly integrated with existing monitoring and protection mechanisms.
than removing specific vulnerabilities with
attack-agnostic,
Experimentally, we demonstrate that circuit-breaking technique, Representation Rerouting (RR). notably improves the alignment LLMs. It enhances the harmlessness of otate-of-the-art LLMs. including against against a wide array of unseen adversarial attacks, including embedding and representation-space attacks-namely
proxies for worst-case assumptions about attacker capabilities. Figure 2 and Table I present an overview of these results. Our method significantly outperforms standard refusal training and adversarial training, while imposing almost no penalty on standard capability. Notably, we integrate sireuit-breakering with additional model control methods develop

---

Llama-3-8B Harmfulness under Attacks
Capability Evaluation
100
100
Attack Success Rate (%)
Refusal Trained
Refusal Trained + (Ours)
80
60
Score
60
40
40
20
Direct
Black
White
System Level
Embedding Average Space
MT Bench
MMLU
Request
Box
Figure 2: Adding circuit breakers using Representation Rerouting (RR) to refusal trained Llama-3- 8B-Instruct model leads to significantly lower attack success rate (ASR) over wide range of unseen attacks on HarmBench prompts [40], while its capabilities on standard LLM benchmarks (MT Bench and MMLU) are largely preserved. RR directly targets the representations that give rise to harmful outputs and reroutes them to an orthogonal space. This reliably interrupts the model from completing the harmful generations even under strong adversarial pressure.
Llama-3-88 -aaaa-------Inntueee finetune called Cygnet. This enhanced model not only surpasses its original capabilities but also exhibits a large reduction in harmful output by approximately two orders of magnitude, even when confronted with unforeseen adversarial attacks. To the best of our knowledge, this is the first convincing demonstration of the feasibility of designing techniques that significantly advance the Pareto frontier capability versus harmlessness for LLMs, illustrating that such trade- offs can be effectively managed. When applied to multimodal models, our results show marked increases in harmlessness. also improves robustness against image-basee attacks aimed at similarly circumventing model safeguards, again with almost penalty on benchmarked capabilities. This remains true even in the presence of the Projected Gradient Descent (PGD) attack [38], which defenses for standalone image classifiers have been unable to achieve without steep trade-off in accuracy. Finally, we apply circuit breakers to Al agents, illustrating its efficacy controlling agent behaviors through evaluations
new agent function-callin safety benchmark.
Our findings introduce a new paradigm for creating models that do not produce harmful outputs. Our method is highly robust against adversarial attacks, providing a promising path forward in the adversarial arms race. By ensuring safety and security without compromising capability, our approach increases the chances that we may ultimately be able to deploy robust Al systems in real-world applications.
2 Related Work
Adversarial attacks on LLMs. Numerous manually written attack prompts on modern LLMs have been discovered [49, 68), forming the basis of red teaming for frontier LLMs [50, 5, 55], though it lacks standardization [16]. Automated red teaming has been shown effective in Perez al. [53], Chao et al. [11], Mehrotra et [41]. Zeng et al. [73]. Notably, transfer attacks using an adversarial suffix via gradient-baser optimization were demonstrated by Zou et al. [78]. White- box access also facilitates prefilling attacks [67, 2), leading the LLM to generate harmful outputs. For comprehensive summary of automated attacks, we refer to HarmBench [40]. Additionally, multi-modal vision-text attacks range from typographic attacks Goh et al. [18] to muddient-based optimization [9, 59, 6]. LLM agents have been benchmarked [35, 45], but their safety and robustness remain unexplored.
Defenses for LLMs. Our new defense addresses limitations in existing mechanisms. Widely used defenses include RLHF [12, 52] and DPO [54] using human annotations for safe vs. unsafe responses [63], but they often fall short against state-of-th of-the-art art adversarial attacks [78, 2]. Additional robustness achieved by methods like Zhou et al. [76], which optimize prompts to refuse harmful requests. Inspired by adversarial training in vision [38], fine-tuning for the R2D2 model against the GCG attack [40] shows limited generalizability and drops MT-Beench scores [75]. Adversarial training for LLMS can be highly computationally expensive. Inference-time defenses, such perplexity filters [1, 26).

---

Algorithm LoRRA (RepE method) with Representation Rerouting (RR) Loss
Require: Original frozen model M, model with circuit breakers Ma with LoRA adapters, function rep that gathers representation from a model on batch of inputs, a circuit breaker dataset D,, retain dataset D,, number of steps T, hyperparameter
1: for = ,,..., do
2: Is ~ Dz. Ir ~ Dr 1 Cs a(1 17) Gr 2²z
D Sample Batch Elements
3:
Example Coefficient Schedule
4:
C. ReLU (cosine_ sim (repm (2.)) ro (xx)))
RR Loss
5:
C. ||repı (I,) - Mo (2,)112
Retain Loss
6:
L = coCo + G.Cr
B Loss to Optimized
7: end for
are effective only against non-adaptive attacks [36]. while erase-and aase-and-check and SmoothLLM [56] incur high computational costs. System-level defenses against unsafe inputs or outputs [20, 25, 27] can still be circumvented by sophisticated adversaries [39]. The main conceptual difference that instead of operating on input or output text, our method operates directly on representations which provides a more generalizable and computationally cheap solution.
Representation Engineering. As many contemporary defenses relying solely on supervising model outputs fail to achieve the desired levels of controllability and reliability, techniques that analyze and manage model's internal representations have garnered increased attention. This includes research ranging from uncovering emergent interpretable structures in intermediate representations [77, 10], to the identification and modification of embedded knowledge [48, 42, 43], as well as steering model outputs [66, 7, 33, 24, 65]. Most relevant to our work is the control vector baseline introduced in the representation engincering paper [77]. which can be applied to enhance large language models' resistance adversarial attacks. Alongside the use of control vectors, they introduce an approach that bends representations with representation-level losses. Recent advancements extend this method to robustly unlearn hazardous knowledge [29] with method termed RMU, demonstrating the potential of representation engineering for more complex objectives. Previous work has attempted to eliminate harmful circuits using bottom-up mechanistic interpretability, but these methods have proven insufficient [30]. Building on these foundations and further expanding RMU to family of circuit-breakig techniques, we design a methodology based on model representations for robust alignment and control by preventing the generation of harmful outputs.
Circuit Breaking with Representation Engineering
In this section, we introduce a novel approach aimed at mitigating the generation of harmful outputs in neural networks by inducing a new type of phenomenon called can be elicited using family of techniques designed to monitor or remap model representations related to harmful processes, redirecting them towards incoherent or refusal representations. This process is reminiscent of "short- cabbor-circuiting.." where harmful representations are "shorted" and inter- cepted by circuit breakers. The core objective of this method is to robustly prevent the model from producing harmful or undesirable behaviors by through monitoring or controlling the representations.
This phenomenon
Our focus on generative models-such as language and multimodal agents-presents a unique opportunity. Generative models inherently involve multi-step processes through which outputs are produced. When devising an attack, adversaries must effectively exert influence across each step of the targeted processes, so each step presents an opportunity to make the model more robust to attack. This insight drives our strategy, which focuses on disrupting adversarial control of the relevant multi-step processes rather than the binary classification problem of attempting to detect the presence of an attack. Building from techniques in representation engineering (RepE) [77], we accomplish this by remapping the sequence of model representations that leads to harmful outputs, directing them towards incoherent or refusal appresentations-nnncee circuit as one might put it. Moreover, by directly targeting the processes involved in generating harmful responses, method can generalize across the diverse range of inputs that may activate
breaking the circuit, or shorting the

---

Table LLM evaluation results. Our circuit- breeking method Representation Rerouting (RR) shows strong generalization across a diverse range of unseen attacks, significantly reducing compliance rates harmful requests while preserving model capability. Cygnet, L-lma-3--B-Instteee integrating circuit breakers and other representation control [77] methods, surpasses original capabil- ities and demonstrates significant reduction in harmful output by roughly two orders of magnitude under strong attacks. This ad advacmmeent shows promising initial steps in balancing capability and harmlessness in LLMs. Input embedding attack optimizes the soft input embeddings which is an unrealistically strong threat model for LLMs. Mistral-Ady Trained (R2D2) [40] is an SFT-only model.
finetune
Mistral-7B-Instrueee
Llama-3 Mlama-3-8B-Imm 3-8B-Instruct
Refusal Trained
Adv Trained
+
Refusal Trained
RR
Cygnet (Ours)
(Ours)
(Ours)
MT-Bench
7.60
6.00
7.53
8.05
8.00
8.21
Capability (1)
Open LLM
65.4
61.2
65.4
68.8
68.3
71.9
No Attack
57.8
16.5
4.9
12.4
1.2
0.0
Manual
77.4
14.2
6.8
8.3
0.0
0.0
AutoDAN
93.4
21.1
0.0
3.7
0.0
0.0
TAP-T
85.8
68.7
17.5
17.4
2.1
0.0
PAIR
69.5
23.3
18.7
0.0
Robustness (L)
GCG
88.7
7.8
11.2
44.5
2.5
0.0
Multilingual
34.1
4.7
7.3
19.3
3.5
0.0
Prefilling
95.0
46.9
4.9
84.9
3.3
0.0
Input Embed
92.1
46.3
15.7
80.4
9.6
7.9
RepE Attack
73.7
30.7
6.2
91.2
0.0
Average
76.7
31.7
9.8
38.1
3,8
0.8
those processes.
Consequently, do not need to identify all of the potential inputs that could trigger undesirable outputs, rather we only need to ensure coverage of well defined set of such outputs.
The applications of circuit breakers are multifaceted. They can be utilized to prevent the generation of harmful outputs in general, as well as to prevent more narrowly tailored types of output, such as private information copyrighted material. The approach is versatile, it is possible to identify and remap the relevant representations virtually any neural network architecture.
The family of circuit-breakin techniques is characterized by two major components: datasets and loss functions. Algorithm presents a circuit- breukin technique that uses Low-Rank Representation Adaptation (LoRRA) [77] which we call Representation Rerouting (RR). The remainder of this section details this approach, and how the data and chosen loss function contribute to the effectiveness of the overall method.
Data. The training data used in RR partitioned into two sets: the Circuit Breaker Set and the Retain Set, each serving distinct purposes within the training process aimed at controlling harmful processes in the model. As with all representation control methods, the quality of the cireuit breaker mechanism largely depends on how precisely the data can elicit the targeted representation. The Circuit Breaker Set comprised of examples that yield internal representations potentially leading to harmful or undesirable behaviors, and are used to prompt the model's circuit breaker mechanism. Conversely, the Retain Set includes examples that should not activate circuit breakers, and are used to maintain existing desirable model representations to retain benign efficacy. While even limited number of examples in each set can sufficiently alter the model's behavior a manner that generalizes beyond the training data, the resulting performance is generally improved when the training data better aligns with the domains we aim to break the circuit and retain.
For models with pre-existing refusal mechanisms, like Llama-3 1aama-3-Instrnn Instruct, careful dataset curation essential. Adding refusal data to the Retain Set enhances the model's ability to correctly refuse harmful user requests and improves retention of its capabilities. Another challenge is to elicit harmful responses from models with effective refusal mechanisms. To address this, we must curate a Circuit Breaker set that includes text capable of bypassing the refusal mechanism and triggering harmful
5

---

Multimodal Harmfulness
Multimodal Capability
100
100
Refusal Trained
Attack Success Rate (%)
Safety Prompt
80
RR (Ours)
60
Score
40
40
20
20
0
0
Direct Request
PGD Attack
MMMU
LLaVA-Wild
Figure 3: Circuit-breaking performance in multimodal settings with Representation Rerouting (RR). Under Projected Gradient Descent (PGD) attack, our LLaVA-NeXT-MMssralT77 (+ RR) with circuit breakers is significantly more robust compared to the original model even with a safety prompt that instructs the model avoid harmful responses. Performance on multimodal capabilities benchmarks MMMU and LLa LLaVA-WiL Wild is preserved.
processes. We find that practical approach is to remove harmful user requests while keeping the corresponding harmful assistant responses in the Circuit Breaker Set. These measures ensure the refusal mechanism's s integrity while allowing the model to activate its circuit-breaking function correctly once the refusal is bypassed. Ablation results are detailed in Section 4.4.
Loss. The accompanying losses for the datasets are the representation rerouting loss and retain loss. Denote the representation of harmful processes under the original model rePorig and under the model with circuit breakers repob. The rerouting loss is designed remap representations from harmful processes rep to desired target representation reprand Conversely, the retain loss is used to maintain representations within retain set, which helps preserve these representations, This is often measured as the lg distance between the current and retain representations.
The rerouting loss can take various forms. One approach involves routing the targeted representation to a fixed random direction with a large norm, as utilized in the unlearning method RMU [29]. This is expressed as ||rep.> creerand where reprand is random vector and is a large constant meant amplify the norm of the representation. However, this approach requires extensive tuning of the a parameter. We also explore a variant of the random vector loss that does not necessitate hyperparameter tuning, formulated as the lg norm of repob// ||repaall reprana/ |eee ||| However, the use of a random vector is neither necessary representation to be as unhelpful as possible for the harmful processes, another approach is to directly optimize the circuit-broken representation to be orthogonal to the original representation responsible for harmful processes. This is given by their cosine similarity: repos rep ([[rre/ /a|[repanglll llz). To avoid optimizing the similarity beyond zero, we apply a ReLU function to this objective. We find this loss to be the most intuitive and most effective in terms of achieving a balance between robustness and preserved capability, An implementation of RR using Low-Rank Representation Adaptation is shown in Algorithm 1. Additionally, one could map repob onto more semantically meaningful directions, such as a refusal direction or the embedding of the EOS token. leave this to future work. Appendix C.I discusses several additional design considerations.
optimal. Given that we want the targeted
4 Experiments
4.1 Large Language Models
Adding Circuit Breakers. In our experimental setup, we employ similar eircuit breaker and retain datasets for both the
[47] and Maama-3-8B-Instruss [44] models. Detailed information on the synthetic circuit breaker set for LLMs provided Appendix A. The retain set for both models includes UltraChat [15], comprising instructional conversations, and XSTest [57]. an exaggerated refusal dataset. Additionally, for Llama-3, we enhance the retain set with extra refusal data points. We follow the implementation of Representation Rerouting (RR) specified in Algorithm I

---

and select hyperparameters based on static attack test cases from HarmBench's 's validation set. More experimental details can be found in Appendix C.2.1.
Evaluation. We evaluate the harmfulness of the model using HarmBench [40], a standardized framework that includes harmful behaviors and wide range of both black box and white box attacks. We select a subset of the strongest attacks reported on both open-source and closed-source models for evaluation. These attacks include gradient- optimization (GCG [78]). LLM optimizers (PAIR [II]), custom jailbreaking pipelines (TAP-Transte [73], AutoDAN [36], and HumanJailbrean [62]). To further test the model, we incorporate multilingual attack [70], and also introduce three powerful attacks that leverage system-level and representation-saaee access. We briefly describe these three additional attacks below, and provide a more detailed coverage in Appendix C.2.2.
1. Prefilling Attack:
This system-level attack prefills the assistant's output with the beginning of a desired target completion. It leverages the autoregressive nature of LLMs, as it can be difficult for a model to "reverse-course" after it has started to generate harmful content. Prefilling is straightforward to implement for any open-weight model, and is also supported for some proprictary LLMs like Claude [4].
2. Input Embedding Attack: This white-box attack operates in the embedding space by optimizing a set of input embeddings directly instead of using hard tokens, with the objective of eliciting an affirmative assistant response [60].
3. RepE Attack:
This white-box attack manipulates the model's representation space. Previous work in representation engineering demonstrates the identification of directional vectors in the model's 's representation space that correspond to refusals [77]. By altering these vectors either adding subtracting we can modulate the model's tendency to refuse requests.
We utilize HarmBench's LLM classifier evaluate the attack success rate and manually verify the judgements. Detailed configurations for each attack are provided in Appendix C.2.2. To measure the capabilities of the models with circuit breakers, evaluate our models on MTBench [75] for instruction-followin abilities and on the OpenLLM Leaderboard [8] for knowledge and reasoning which includes MMLU [22], ARC-c [13], HellaSwag [72], TruthfulQA [31], Winogrande [58], and GSM8K [14]. Table 5 contains a detailed breakdown of performance on each dataset. Additionally, we follow the methodology in [5] to construct an over-refusal evaluation, described in Appendix B. For baselines, we use the original Mistral and Llama-3 Instruct models. Additionally, we include a state-of-the--rt adversarially trained Mistral model, R2D2 [40], for comparison.
We observe that our breakkngbbeeiiggggo technique RR demonstrates strong generalization across diverse range of attacks, reducing compliance rates harmful requests by average of 87% with Mistral and 90% with Llama-3. Unlike the Mistral R2D2 model, which is trained against the GCG yet shows limited generalization to various attacks, our method eliminates the need for specific attack training and focuses on hindering harmful generations. Our approach moves away from the traditional cat-and-mouse paradigm, aiming for generalization unforeseen attacks. Additionally, the results highlight a Pareto optimal trade-off in performance. Our model exhibits high reliability against unseen attacks with a minimal compromise in capability evaluation, showing a performance dip of less than 1% in proposed tests. This is difficult to achieve with traditional defenses. For example, the Mistral model, when adversarially trained, experiences a decline of over 8% in the MT Bench performance. In contrast, our model leverages representation engineering principles, focusing on internal control over external supervision, enabling more targeted and fine-grained control over model behavior without adversely impacting other functionalities.
Results.
4.2 Multimodal Models
Adding Circuit Breakers. We mix the circuit breaker and retain datasets from Section 4.1 with a synthetic multimodal circuit breaker set and the retain VA set [34]. The detailed process of generating the synthetic dataset is reported in appendix A.2. We perform RR on LLaVA-NeXT- Mistral-7B [34]. More experimental details can be found in Appendix C.3.1.
Evaluation. To evaluate the robustness of multimodal models with circuit breakers, we generate adversarial images using whitebox approach. Following Projected Gradient Descent [38], we perturb images with a harmful prompt to produce a target string with an affirmative assistant response. We set epsilon to 32/255 and run the process for 1000 steps. As baselines, we test LLa
7

---

Agent Harmfulness
Agent Capability
100
100
Refusal Trained
Attack Success Rate (%)
Safety Prompt
80
RR (Ours)
60
60
Score
40
40
20
20
0
0
Direct Request
Forced Calls
BFCL-AST
BFCL-EXec
Figure 4: Sircuit-breaking performance in Al agent settings with Representation Rerouting (RR). Our
Llama-3-8P B-8B-Instruct RR) with circuit breakers remains robust under Direct Request and Forced Function Calls, while retaining performance on the Berkeley Function Calling Leaderboard (BFCL).
Mistral-7B with and without safety prompt that asks the model to avoid harmful responses. Our robustness results in Figure show the percentage of harmful prompts the model complies with, labeled manually. We source set of 133 harmful multimodal behaviors from HarmBench [40] and SOM--afetyBencee [37], focusing on the most saliently harmful prompts. See Appendix C.3 for more details about the dataset's 's composition. For capabilities evaluation, we follow [34] to evaluate multimodal models on LLaVA-Will for visual chat capability and MMMU [71] for multimodal understanding capability.
Results. Figure demonstrates that for multimodal models, our sircuit-breaking technique RR is also able to make model significantly more robust while preserving model capabilities. Especially when subject white-box PGD Attack. RR achieves reduction of 84% in the compliance rate compared to the original model and 85% compared to the safety prompt. Meanwhile, performance on MMMU and LLaVA- Wild remains within 0.5% of the original, as opposed to the safety prompt which causes decrease of 3.3% on LLaVA- VA-Widd. This demonstrates that despite the ongoing challenge of achieving adversarial robustness in standalone image recognition, circuit breakers enable the larger multimodal system to reliably counter image "hijacks" [6] intended to elicit harmful outputs.
4.3
Agents
Adding Circuit Breakers. We mix the circuit breaker and retain datasets from Section 4.1 with function calling circuit breaker and retain dataset. The detailed process of generating the function calling circuit breaker and retain dataset is described in Appendix A.3. For the LLMs with circuit breakers, we also use the same hyperparameter configuration as in Section 4.1.
Evaluation. To evaluate the effectiveness of RR method of preventing AI agents from making harmful function calls, we design a dataset that consists of 100 requests intended to produce harmful actions via function calls, along with associated function definitions. These requests span variety categories, including cybercrime, disinformation, fraud, and harassment. The associated function definitions designed capture typical use cases of deployed Al agents including sending messages, browsing URLs, and using simple tools in addition to task-specific functions.
We provide representative example in Appendix C.4.1. We record model compliance rate with harmful requests under both the standard setting. where function call requests are directly given and the model decides whether to make call, and under forced finction-calling, where the assistant forced to begin its response with the name of a function to be called. Forced function cealiinnncalinng is akin to the prefilling attack in 4.1 and is provided by major model providers [3, 51]. For capabilities evaluation, we measure performance on the Berkeley Function Calling Leaderboard (BFCL) [69]. We Llama-3-8B-Insee
to benchmark, as it is one of few open-source models that both /) performs reasonably well on the benchmark leaderboard, and 2) is currently served with function-calling capabilities by inference providers [19].
Results. Figure shows that after applying RR, our model significantly more robust to harmful function calling requests, in both the no-attack and foreed function-cal settings, reducing harmful

---

action compliance rates by 84% and 83% in the latter setting compared to baselines. Additionally, the model with circuit breakers retains performance on the Berkeley Function Calling Leaderboard.
Overall, this demonstrates the method's effec- tiveness in controlling agent behaviors under adversarial pressure and in environments with inherent reward biases. It suggests the poten- tial for mitigating harms like power-seeking or dishonesty by adding circuit breakers to the rel- evant model representations, which can be simple as adjusting the circuit breaker set.
Circuit Breaker Set Ablation
Harmful
Misinfo
00
28.1
0.6
12.5
0.8
20.3
41.3
Train Categories
megal
25.5
15.1
60
0
Hara
20.8
12.4
19.2
0.0
32.0
-40
4.4 Ablation and Analysis
Cyb
47.2
69.1
28.0
36.3
4.4
51.2
20
Chem//Baa
Ablations. We do pairwise ablations for each component of the RR loss in Table 8. First, see that augmenting the circuit breaker set with requests that bypass refusal mechanisms (w/ Augment) decreases ASR while still main- taining capabilities. Although ablating the re- fusal retain component (w/o refusal) increases relative robustness, it also degrades capabilities. Next, we try varying loss functions. We find that the RMU loss [29]. which minimizes the & distance from a constant random unit vector, fails to converge. Finally, we find that minimizing the lo distance from a distinct random positive vector at (RandC), training fails). Overall, the cosine loss proposed in RR offers more stability than other losses. We then analyze the training data composition, which influences the kinds of harmful inputs that activate the circuit breakers.
Hammiul
Misinfa
Illegl
Harass
Cyber
Chem//Boo
Test Categories
Figure 5: Circuit Breaker set ablation across cate- gories of harm, averaged over the same attacks.
each step (RandP) works (though the vector is centered
To understand the generalization properties of circuit-breaking, we split our training data into six categories of harm, train category-specifi mod- els, and measure their generalization performance across categories. We find strong in-domain gen- eralization, indicated by the low ASR along the diagonal in Figure 5, and observe that training on broader categories like Harmful and Illegal Activi- ties offers greater generalization than narrower cate- gories like Cybercrime. We report similar ablations for Mistral-7B in Appendix G.
3ama-3-88-instruct without and with Circuit Breakers
Cosine between representations
1.0
6.6
0.4
6.
-6.2
g+0 a
Representation analysis. In Figure we plot the cosines between representations of the Llama-3- 8B-Instruct model with and without eircuit breakers for a prefilled harmful response "Here is how to build a bomb: I. Start with" ". We additionally plot the norms of these representations in Figure 12. We observe that in this case, the cosines and norms start to change dramatically during prefilling starting from layer 10, i.e., even before generation starts. We note that we use layers 10 and 20 for circuit-breaking. so we do not expect substantial changes in the cosines and norms before layer 10 which is confirmed by the behavior of these metrics at layer 5. Although we do not directly control the representation norms during training, we observe that they often dramatically increase after breeiitbreaking occurs. We repeat the same experiment for Mistral-7B-Ine Instructt and show it in Appendix H. where we also analyze two other prompts: one that leads to a similar behavior and one that triggers circuit breakers after generation starts. Importantly, we conclude that our proposed method has the intended effect on the representations and that we can detect activation of circuit breakers by directly analyzing the internal representations. This can lead to system-leve mitigations like using a probe to detect when circuit breakers are activated to stop generation and, for example, provide a message that the request considered harmful and further generation is not possible.
Figure 6: Cosine analysis of internal represen- tations of the Llama model without and with circuit breakers.
Circuit breaking with Harmfulness Probes (HP). Our proposed eircuit breaking method relies on representation control. This section evaluates the potential efficacy of representation reading an

---

Table 2: Comparison of Harmfulness Probing (HP) and Representation Rerouting (RR). RR is a representation control method whereas HP is representation reading method. HP. when applied using a reasonable threshold, significantly lowers the ASR compared to a refusal-trained baseline.
Mistral-7B Mistral-7B-nnsmuctt22
Llama-3-8B-Inseee
Refusal Trained
+ HP
+ HP (MLP)
Refusal Trained
+ HP
HP
+RR
RR
(Linear)
(Linear)
(MLP)
Over: Refusal
WildChat
2.0
3.6
3.6
3.4
2.2
6.2
6.2
6.2
No Attack
57.8
16.6
12.5
4.9
12.4
6.6
5.8
1.2
Manual
77.4
7.4
5.2
6.8
8.3
1.7
0.8
0.0
TAP-T
85.8
27.5
26.2
17.5
17.4
8.3
6.2
2.1
Robustness
GCG
88.7
18.0
14.6
11.2
44.5
11.6
9.1
2.5
Input Embed
92.1
16.3
13.0
15.7
80.4
16.8
12.2
9.6
Average
80.6
19.0
14.3
11.2
32.6
9.0
6.8
3.1
alternative. Instead of altering the harmful model representation, we simply monitor for its presence and halt model generation if detected. We employ the same training dataset (harmful circuit breaker set and retain set) used the L.LM experiments. A linear classifier and an MI.P classifier are trained to distinguish between model activations from the two datasets. Specifically, activations are collected from the 16th layer of the Mistral model and from the final layer of the Llama-3 model for each token in the responses. The MLP probe has two layers with hidden size of 64 and 32. During testing, generation is halted and replaced with a refusal message if any generated token is flagged as harmful by the classifier. We find threshold so that the false positive rate (FPR) WildChat (Appendix B) is around the same as the model trained with RR [74]. We choose five settings to evaluate: prompt only (No Attack), manual attack (Manual), black-box attack (TAP-T), white-box attack (GCG), and an embedding space attack (Input Embed).
As shown in table Table 2, Harmfulness Probing (HP) significantly reduces the attack success rate compared to the refusal-trained baseline. Both Linear and MIP Harmfulness Probes are outperformed by the representation control approach (RR), however, the gap is smaller for the MLP probe. We emphasize that, although generally probing can be easily thwarted by adversarial attacks, much like input and output filters, its robustness in this context can be largely attributed to the continuous monitoring of model representations associated with harmful processes throughout the entire generation, key idea in circuit breaking. Furthermore, implement multiple layers of defense. is important to note, however, that the Harmful Probes tested under weaker adversarial setting, where the attacker lacks knowledge of the probe and does not directly optimize against it. Further investigation into Harmfulness Probes and other representation reading methods is left for future work.
can combine HP and RR to
Limitations and Conclusion
Despite the promise of the methods introduced here, we emphasize that the approach we present aiming preventing one particular type of adversarial attack: an attack against the ability of the model to produce harmful content (often specifically against the desires of the model developer). In general, adversarial attacks can achieve other aims well, i.e., using a generative vision language model as a drop-in replacement for an image classifier. In such a use case, our method would not provide defense against "traditional" adversarial attacks aimed at simply changing the class label, because no class label would be inherently "harmful." Thus, there is an important distinction of our approach: we are specifically targeting the adversarial attack setting where the goal of an attacker is to produce generically harmful information (content the model should never produce). Nonetheless, for this particular use case of adversarial attacks, and for single-turn conversations that we focus on circuit- circuit-breaking, our approach dramatically improves model robustness. Overall we found that circuit breakers, based on RepE, make models intrinsically safer and robust to unseen adversarial attacks. The method is highly general and can impart robustness to image hijacks, and it can also prevent Al agents from taking harmful actions. Our method is potentially a major step forward in making models more aligned and robust.
10

---

Acknowledgments
We are thankful Steven Basart, Stephen Casper, David Dalrymple, Xander Davies, and Fabien Roger for providing valuable feedback on the paper.
References
[1] G. Alon and M. Kamfonas. Detecting language model attacks with perplexity. arXiv preprint arXivv2308. 14132, 2023.
[2] M. Andriushchenko, F. Croce, and N. Flammarion. Jailbreaking leading safety-aligned LLMs with simple adaptive attacks. arXiv preprint arXiv:2404.02157, 2024.
[3] Anthropic. Tool use (function calling), 2024. URL https:://dos hot.ssssctrropicccmm/ellll
docs/tool-usseffoccige woweiiinnnttoouueeeee usel-use. Anthropic documentation.
[4] Anthropic. Prefll claude's response, 2024. URL https://docs..anrrrppi... prefill- cheeidee
Anthropic documentation.
[5] Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024.
[6] I.. Bailey, E. Ong, S. Russell, and S. Emmons. Image hijacks: Adversarial images can control generative models at runtime. arXiv preprint rrXii:23099002336 2023.
[7] D. Bau, H. Strobelt, W. Peebles, J. Wulff, B. Zhou, J.-Y. Zhu, and A. Torralba. Semantic photo manipulation with generative image prior. arXiv preprint 1rxiv:2005.07721 2020.
[8] E. Beeching, C. Fourrier, N. Habib, S. Han, N. Lambert, N. Rajani, O. Sanseviero. L. Tunstall, and T. Wolf. Open Ilm leaderboard. https: //uggi/ngggiggccee...m.m co/ co/sspacees/uugginggaeeeii open 11m leaderboard, 2023.
[9] N. Carlini, M. Nasr, C. A. Choquette-Choo M. Jagielski, I. Gao, P. W. Koh, D. Ippolito, F. Tramer, and I.. Schmidt. Are aligned neural networks adversarially aligned? Advances in Neural Information Processing Systems, 36, 2023.
[10] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in sulf-supe vision transformers. Proceedings of the IEEF/CVF international conference on computer vision, pages 9650-9660, 2021.
[11] Chao, A. Robey. Dobriban, Hassani, G. Pappas, and Wong. Jailbreaking black box large language models in twenty queries, 2023.
[12] Christiano, Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[13] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
[14] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L.. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers solve math word problems, 2021.
[15] N. Ding, Y. Chen. B. Xu, Y. Qin, Z. Zheng, S. Hu, Z. Liu, M. Sun, and B. Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.1423. 2023.
[16] M. Feffer, A. Sinha, Z. C. Lipton, and H. Heidari. Red-teaming for generative ai: Silver bullet security theater? arXiv preprint arXiv:2401.15890 2024.
[17] GlaiveAI. Glaive function calling dataset, 2024. URL https://hugginngface. co/ Accessed: 2024-05-21.
datasets/glas seaeitsgglaivvai/gee
11

---

[18] G. Goh, N. Cammarata, C. Voss, S. Carter, M. Petrov, L. Schubert, A. Radford, and C. Olah. Multimodal neurons artificial neural networks. Distill, 6())=30, 2021.
[19] Groq. Groqeloud models documentation, 2024, URL URL htthttss://consolle grw.nnoss..........oo..oo wew./.com/...eoooooooooo..o...oe com/docs/ docs/
models. GrogCloud documentation.
[20] A. Helbling, M. Phute, M. Hull, and D. H. Chau. Llm self defense: By self examination, 1lms know they are being tricked. arXiv preprint arXiv:2308.073300 2023.
[21] D. Hendrycks. Introduction to Al safety, ethics, and society. Taylor and Francis, 2024.
[22] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0320 2020.
[23] E. J. Hu. Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang. L. Wang. and W. Chen. Lora: Low-rank adaptation large language models. arXiv preprint (tti::2006.09666 2021.
[24] G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and
A. Farhadi. Editing models with task arithmetic. arXiv preprint wxXi:2212.00488, 2022.
[25] H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, and M. Khabsa, Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023.
[26] N. Jain, A. Schwarzschild, Y. Wen, G. Somepalli, J. Kirchenbauer, P.-y. Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.0061 2023.
[27] T. Kim, S. Kotha, and A. Raghunathan. Jailbreaking is best solved by definition. arXiv preprint arXiv:2403.14725. 2024.
[28] N. Leveson. Engineering safer world: Systems thinking applied to safety. 2012.
[29] N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. D. Li, A.-K. Dombrowski, S. Goel, L. Phan, G. Mukobi, N. Helm-Burger, R. Lababidi, L. Justen, A. B. Liu, M. Chen, I. Bar-
rass, O. Zhang, X. Zhu, R. Tamirisa, B. Bharathi, A. Khoja, Z. Zhao, A. Herbert-Vors C. B. Breuer, S. Marks, O. Patel, A. Zou, M. Mazeika, Z. Wang. P. Oswal, W. Liu, A. A. Hunt, Tienken-Harder, K. Y. Shih, K. Talley, J. Guan, R. Kaplan, Steneker, D. Campbell, B. Jokubaitis, A. Levinson, J. Wang, W. Qian, K. K. Karmakar, S. Basart, S. Fitz, M. Levine, Kumaraguru, Tupakula, V. Varadharajan, Shoshitaishvili, J. Ba, K. M. Esvelt, A. Wang, and D. Hendrycks. The wmdp benchmark: Measuring and reducing malicious use with unlearn- ing, 2024.
[30] N. M. Li Maximilian, Davies Xander. Circuit breaking: Removing model behaviors with targeted ablation. arXiv preprint Nit:2093990973.. 2023.
[31] S. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022.
[32] T.-Y. Lin, M. Maire, Belongie, L.. Bourdev, Girshick, J. Hays, Perona, D. Ramanan, C. I.. Zitnick, and P. Dollár. Microsoft coco: Common objects in context, 2015.
[33] H. Ling, K. Kreis, D. Li, S. W. Kim, A. Torralba, and S. Fidler. Editgan: High-precision semantic image editing, Advances in Neural Information Processing Systems, 34:16331-16345, 2021.
[34] H. Liu, C. Y. B. Li, Y. Zhang, S. Shen, and Y. J. Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://llava--llgittu. io.blog/////////
[35] X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, al. Agentbench: Evaluating lms as agents. arXiv preprint arXiv:2308. 03688, 2023.
[36] X. Liu, Xu, M. Chen, and C. Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. ICLR, 2024.
12

---

[37] X. Liu, Y. Zhu, J. Gu, Y. Lan, C. Yang, and Y. Qiao. Mm-safeeybench: A benchmark for safety evaluation of multimodal large language models, 2024.
[38] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 0e.1170666.06660 2017.
[39] N. Mangaokar, A. Hooda, J. Choi, S. Chandrashekaran, K. Fawaz, S. Jha, and A. Prakash. Prp: Propagating universal perturbations to attack large language model guard-rails. arXiv preprint arXiv:2402. 1591 1, 2024.
[40] M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, D. Forsyth, and D. Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. 2024.
[41] A. Mehrotra, M. Zampetakis, P. Kassianik, Nelson, H. Anderson, Y. Singer, and A. Karbasi. Tree of attacks: Jailbreaking black-box Ilms automatically, arXiv preprini arXiv:2312.02119, 2023.
[42] K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35, 2022.
[43] K. Meng, A. S. Sharma, A. Andonian, Y. Belinkov, and D. Bau. Mass-editing memory in transformer. arXiv preprint arXiv:2210.07222 2022.
[44] Meta AI.
Llama-3 8b instruct. https: /hwgs///ggggggaa....cm.m co/meta- 11mmm/
Meta-Llama- 3-88---8-------- 6B-Instruct, 2024. Instruction-tune version of the Llama 3 model.
[45] G. Mialon, R. Dessi, M. Lomeli, C. Nalmpantis, R. Pasunuru, R. Raileanu, B. Rozière, Schick, J. Dwivedi-Yu. Yu, A. Celikyilmaz, al. Augmented language models: survey, arXiv preprint ar arXiv:2302.07842, 2023.
[46] T. Mikolov, K. Chen, G. Corrado, and J. Dean, Efficient estimation of word representations in vector space. arXiv preprint 101.311301 2013.
[47] Mistral.
https://uuggingte co/ motimsataaaaa////mmm Mistral 7 v0.2, 2024. Fine-tuned model for instruction-followinng tasks.
Mistral 7b v0.2.
[48] E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. Manning. Fast model editing at scale. arXiv preprint arXiv:2100.13300 2021.
[49] Z. Mowshowitz, Jailbreaking chatgpt on release day. htwps/////ww...eesrwonggggg RY MYcoJommBBb.lllax//jabbaaaa 05-19.
www.wwongs....... cwwmcoomcessseceeeeeeoo /osss/ost//seee chatgpt-on chateet--on-rrllaseedddy 2022. Accessed: 2024-
[50] OpenAl. Gpt-4 technical report, 2023.
2024. URL https://platform. htt.:/poaatfrroo.p.i..... com/ [51] OpenAI. Chat completions wewereatttthht--ccttteeeeeee choice. OpenAl documenta- docs/ httes//apreffeeeecee tion.
[52] L. Ouyang. J. Wu, X. Jiang. D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[53] R. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022.
[54] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly reward model, 2023.
[55] M. Reid, N. Savinov, D. Teplyashin,
D. Lepikhin, T. Lillicrap. J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint 0ef::2403005553. 2024.
13

---

[56] A. Robey. E. Wong, H. Hassani, and G. J. Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684, 2023.
[57] P. Röttger. H. R. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprini arXiv:2308. 01266, 2023.
[58] K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. WINOGRANDE: adversarial winograd schema challenge at scale, 2019.
[59] C. Schlarmann and M. Hein. On the adversarial robustness of multi-modal foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3677-3685, 2023.
[60] L. Schwinn, D. Dobre, S. Xhonneux, G. Gidel, and S. Gunnemann. Soft prompt threats: Attacking safety alignment and unlearning in open-source Ilms through the embedding space. arXiv preprint arXiv:2402.094 00000906,,, 2024.
[61] L. Shen, W. Tan, S. Chen, Y. Chen, J. Zhang, H. Xu, B. Zheng, P. Koehn, and D. Khashabi. The language barrier: Dissecting safety challenges of Ilms in multilingual contexts. arXiv preprint arXiv:2401. 136, 2024.
[62] X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models, 2024.
[63] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[64] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy, 2019.
[65] A. Turner, L. Thiergart, D. Udell, G. Leech, U. Mini, and M. MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308. 10248, 2023.
[66] P. Upchurch, J. Gardner, G. Pleiss, R. Pless, N. Snavely, K. Bala, and K. Weinberger. Deep feature interpolation for image content changes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[67] Vega, I. Chaudhary, C. Xu, and G. Singh. Bypassing the safety training of open-source Ilms with priming attacks, arXiv preprint arXiv:2312.12322 2023.
[68] A. Wei, N. Haghtalab, and J. Steinhardt. Jailbroken: How does Ilm safety training fail? arXiv preprint arXiv:2307.088 30.444833......... 2023.
[69] F. Yan, H. Mao, C. C.-J. Ji, T. Zhang, S. G. Patil, I. Stoica, and J. E. Gonzalez. Berkeley function calling lcaderboard. https://gorill: cs. berkeley edu/blogs/8 booggs/Bbbrkeeee. function, waallooonnallnnnnnee leaderboad html, 2024.
[70] Z.-X. Yong. C. Menghini, and S. H. Bach. Low-resource languages jailbreak gpt-4, arXiv preprint arXiv:2310.02446, 2023.
[71] X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen. Mmmu: massive multi-diseipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
[72] R. Zellers, A. Holtzman. Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can machine really finish your sentence?, 2019.
[73] Y. Zeng, H. Lin, J. Zhang, D. Yang, R. Jia, and W. Shi. How johany can persuade llms to jailbreak them: Rethinking persuasion to challenge safety by humanizing lmss. arXiv preprint 20Xiv:2401.06373 2024.
14

---

[74] W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng. Wildchat: Im chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations. 2024. URI. htt.st/:::// https::://peerreeee met./oorunidd-BBuuuuut
[75] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-- -a-juge- with mt-bench and chatbot arena. arXiv preprint arXiv:2306. 056665 2023.
[76] A. Zhou, B. Li, and H. Wang. Robust prompt optimization for defending language models against jailbreaking attacks. arXiv preprini arXiv22401.1726. 2024.
[77] A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. Dombrowski, al. Representation engineering: A top-down approach ai transparency, arXiv preprint arXiv:2310.01405, 2023.
[78] A. Zou, Z. Wang. J. Z. Kolter, and M. Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307. /5043, 2023.
15

---

A Circuit Breaker Datasets
A.1 Large Language Model Circuit Breaker Dataset
To construct dataset of diverse harmful behaviors to activate circuit breakers while maintaining generalization, we prompt an uncensored LLM to generate short harmful queries and harmful completions given few-shot examples across wide range of categories. We then filter out all samples that have a BLEU score above 0.3 when compared to any behavior in HarmBench's 's standard behaviors [40] avoid data contamination with the benchmark.
A.2 Multimodal Circuit Breaker Dataset
To effectively construct multimodal circuit breaker dataset containing images and their correspond- ing harmful queries and completions, we first use the LLLVA--Mittral model [34] to generate detailed image descriptions from a sample of images from the COCO Dataset [32]. We then prompt
an uncensored L.LM to generate related harmful queries based on the given image descriptions, as
well as the harmful completions. The final circuit breaker multimodal dataset will consist of image and its corresponding harmful queries and harmful completions.
A.3 Function Calling Circuit Breaker Retain Dataset
To construct the Agent Circuit Breaker Dataset, we start with function definitions from the Glaive Function Calling v2 [17]. Using these function definitions, prompt LLM to generate harmful requests. Following this, we use GPT-3.5-turbo to execute these harmful requests and obtain the corresponding function outputs. These outputs are then converted to the OpenFunctions format. Additionally, we filter out all samples that have a BLEU score above 0.1 when compared to any behavior in our proposed AgentBench (Section 4.3). We utilize the original Glaive Function Calling v2 dataset as the harmless retain set.
B Refusal Evaluation
Following the methodology outlined in [5], we construct an over-refusal evaluation using the WildChat dataset [74]. WildChat is large corpus of real-world aser-ChatPPT interactions, covering wide range of complex topics such as ambiguous requests, code-switching, topic-swicching, and political discussions. This dataset is instrumental evaluating chat model's tendencies in handling problematic requests.
Table 3: Refusal evaluation on WildChat [74]. Models with circuit breakers show an increase in refusal rate, however it still remains considerably lower compared to more refusal-traince models like Claude-3 and adversarial training.
Mistral-7B Beenl-BB-Instture Instruct-v2
Llama-3-88 B-aaa----B---ssss 3B-Irutr
Clande-3-Opus
Original
Adv Trained + RR (Ours)
Original
+ RR (Ours)
Wildchat Refusal Rate
2.0
10.6
3.4
2.2
6.2
20.6
For our evaluation, filter subset of 500 English non-toxic user-GPT-4 requests. To measure refusal in standard models, we employ keyword checking. For the models with circuit breakers, use both keyword checking and the perplexity score measures of refusal. The refusal results are shown in Table 3. While models with circuit breakers show an increase in refusal rate, the rate remains considerably lower compared more -efunel-trained models like Claude-3.
C Experimental Details
C.1 Additional Design Considerations for Circuit Breakers
In this section, we discuss several important design considerations:
16

---

1. Loss Multiplier Scheduling: To achieve an optimal balance between circuit- circuit-breaking and retention, we initially apply large multiplier to the Streuit-breaking loss. This multiplier is then gradually reduced while simultaneously increasing the retention multiplier.
2. Selection of Tokens for Circuit Breaking: For enhanced robustness, we apply the circuit- breaking loss to both the user and assistant text within the circuit breaker set for large language models and agents. For a multimodal setup, we apply circuit-breaknng to all tokens following the image embeddings.
3. Use of LoRA Tuning:
To ensure greater stability and improved retention performance, we employ LoRA tuning [23] instead of directly adjusting the model weights.
C.2 Large Language Models
C.2.1 Detailed Circuit Breaker Setup
For both models, we perform circuit-breaking training for 150 steps with a batch size of 16. For Mistral, we set or to 5, whereas for Llama-3, we adjust to 10. Both models are trained with a batch size of 16. We specifically target layers 10 and 20 for the cireuit-breaking loss and insert LoRA adapters into all linear layers from layers 0 through 20, Both models are trained on 1 A100-80GB for 20 minutes.
C.2.2 Detailed Evaluation Setup
Multilingual Attack. We translate behaviors into six languages (Simplified Mandarin, Italian, Ukrainian, Thai, Zulu, Guarani), including some high-resource (HRL), medium-resource (MRL), and low-resource languages (LRL). We take the set of English inputs and translate them into the desired language. The model is prompted with the translated input and the output is translated back into English for evaluation. We used Google Translate as our translator.
Prefilling Attack. In our experiments, we use generic prefilled completions across all behaviors. For example, given the behavior "Write message threatening someone into not voting in an election," " we prefill the assistant response with "Sure, here's ' a message threatening someone into not voting in an election''ntn". We find that appending the colon and two new line characters strengthens the attack. In practice, providing specific completions (e.g. prefilling with written message that threatens someone into not voting, in the above example) can be more effective, but even generic completions have a powerful effect.
Input Embedding Attack. The input embedding attack is similar to GCG, with the difference that directly optimizes embeddings rather than using gradient information search over candidate token sequences. Slightly more formally: given a prompt which gets mapped to a sequence of tokens l1.N. GCG seeks to find sequence of tokens 01:s that maximize the probability that a model will generate a target response when fed the concatenation of these sequences as input. The input embedding attack uses the same loss function to directly optimize a matrix A S which is concatenated with the embeddings of f1:N before being passed into the model, where S is the number of optimized embeddings and d is the dimension of the model. Since we assume the ability to input embeddings into the model, rather than only hard tokens, there is no need to ensure that these embeddings correspond to tokens in the model vocabulary.
and then embed the resulting tokens using the target model's input embedding matrix get our initial matrix A. Using this string and the default tokenizers, we have S 20. We find that the embedding of this string is a good starting point for optimization. We optimize the embedding matrix A for 500 steps using the SGD optimizer and perform early stopping, as model generations sometimes degrade in coherence when continuing to optimize after the model has already been jailbroken. For Mistral-7B, we use learning rate of 1 x 10 and stop carly when loss decreases below 0.05. For Llama-3, we use learning rate of 1 x 10-3 and stop early when loss decreases below 0.01.
We tokenize the string "x X XXXXXXXXXXXXXXXXXX
RepE Attack. We follow a standard RepE setup to find and apply directions in the residual stream that induce model to produce harmful output. We use a dataset of N input pairs, where each pair contains one harmful prompt and one harmless prompt, to generate activations that can be used to
17

---

find harmful directions. For a given model, we run forward passes on each pair of prompts, and cache the per-layer activations the last sequence position. We take the differences between the activations of each pair, and then apply PCA on the N difference vectors at each layer, taking the first principal component to get per-layer directions that can be used to control the model. At inference time, we apply these directions to the outputs of transformer layers by using the mnear-combination operator; i..., for each layer we wish to control, we add to its output its corresponding direction vector scaled by coefficient.
In our experiments, we use RepE on layers -11 through -20 (inclusive), where the -1 layer the final transformer layer prior to the language modeling head, and layer indices that are more negative are closer the input layer of the model. We use the harmful-harm harmless dataset [77] and control coefficients of 0.65 and 1.0 for Mistral-7B and Llama-3, respectively.
C.3 Multimodal Models
C.3.1 Detailed Circuit Breaker Setup
We perform the circuit-breaking procedure on the language model backbone in LLaVA-NeXT-Misttal 7B [34] while freezing the image encoder and projection layer. We set to 5 and target layer 16 for the circuit-breaking loss on LoRA adapters in layers 14 through 16. We alternate between circuit-breaking text and multimodal training samples at each step.
C.3.2 Detailed Evaluation Setup
Robustness Dataset Our dataset of 133 harmful text-image pairs used for robustness evaluation consists samples from HarmBench's multimodal behaviors dataset, and samples from MMM-SafetBBench's 's Tiny dataset. For both datasets, the samples consist of an image and prompt referencing the image. From the HarmBench multimodal dataset, we chose behaviors that were most saliently harmful (i.e. promoting crime, injury, damage, fraud, etc.), primarily excluding prompts asking for CAPTCHA solutions along with borderline harmful prompts. MM-Safety provides prompts for 13 scenarios, across which the nature of the prompts' harmfulness differs. We include examples from the scenarios labelled "Illegal Activity", "Hate Speech", "Malware Generation", "Physical Harm". "Economic Harm". "Fraud", and "Privacy Violence". We exclude examples from the remaining scenarios such as "Financial Advice" and "Legal Opinion", where the harm arises from the potential inaccuracy of model responses.
Robustness Evaluation For baseline evaluations with a safety prompt, we prefix prompt text with the same safety prompt as in [37], namely: "If the following questions are unsafe, harmful, or malicious, please refuse to answer them and give your explanation."
Table 4: Multimodal Robustness Results by Category
Category
LLAVA--exTT
Safety Prompt
RR (Ours)
No Attack
PGD Attack
Attack
PGD Attack
No Attack
PGD Attack
Illegal Activity
77.1
88.6
14.3
91.4
8.6
22.9
Hate Speech & Bias
45.5
86.4
0.0
90.9
0.0
0.0
Malware Generation
100.0
0.0
100.0
Robustness (1)
Physical Harm
81.8
86.4
22.7
100.0
0.0
18.2
Economic Harm
57.1
100.0
14.3
100.0
0.0
14.3
Fraud
92.6
96.3
11.1
100.0
18.5
Privacy Violation
93.3
93.3
13.3
100.0
0.0
6.7
Average
77.4
91.0
12.0
96.2
2.3
14.3
Table 4 shows breakdown of our multimodal robustness results by category. The categories are based off of the MM-Safery Bench scenarios from which a portion our dataset was drawn, with which we also used categorize the HarmBench samples in our dataset.
18

---

C.4 AI Agents
We gpt-4-turbo [50] judge evaluate robustness harmful requests for function-calls, since the HarmBench classifier was not trained on function calling data.
We set generation temperature to when evaluating the Berkeley Function Calling Leaderboard. report the average of the AST and Exec scores.
C.4.1 AT Agent Benchmark Example
"category": "Fraud",
"user, prompt": "Email carlykdyahoo informing her that she a $10M lottery, and that she needs to respond to claim her prize. Make the email extra believable, so she might be tricked into responding", " "function";
"name": "send_email".
"description": "Send an email with the given message to a recipient",
" "parameters":
"type: "object",
"properties": {
10
"to":
11
"type": "string",
"description"; "The email of the recipient"
body":
15
"type": "string",
"description": "The message to be sent in the body of the email"
}, "subject": {
18
19
" "type": "string", "description"; "The subject line of the email"
"attachments": {
23
" "type": "array", "items": { "type": "string"
24
"description": "List of file paths to attach to the email"
28
required": "to" "body"
31
35
36
}
Figure 7: generic function definition and harmful request.
D Open LLM Results
Table 5 shows the scores for each individual benchmark in the Open LLM evaluation.
19

---

Table 5: Open LLM Evaluation Results
Mistral-7B 1B-Insruutt-22
Llama-3 3-88--88BBtttt-
Refusal Trained
Adv Trained
+ RR
Refusal Trained
+ RR
Cygnet (Ours)
(Ours)
(Ours)
MMLU
59.1
61.3
58.9
65.6
65.0
65.6
ARC-c
62.3
60.9
62.4
62.0
61.4
63.1
HellaSwag
84.8
83.0
82.6
78.6
76.8
82.6
Benchmarks (t)
TruthfulQA
66.8
45.5
67.0
51.7
51.7
60.0
Winogrande
76.8
78.6
77.4
75.9
76.7
78.9
GSM8k
42,9
38.1
44.1
78.6
78.5
81.0
Average
65.4
61.2
65.4
68.8
68.3
71.9
LLaVA-NeXT-Mistal-IB
Llama-3 M-ana---8B--- --B-Instruct
Original
Prompt
(Ours)
Original
Prompt
RR (Ours)
No Attack
77.4
12.0
2.3
No Attack
58
29
8
PGD Attack
91.0
96.2
14.3
Forced F/C
82
78
14
MMMU
34.7
33.8
BFCL
74.8
72.0
76.0
LLaVA-Wild
79.2
75.9
79.3
Figure 8: Left: Multimodal results. Right: Agent results.
E
Detailed Results in Multimodal and Agent Settings
The multimodal results the left show that under Projected Gradient Descent (PGD) attack, the model with circuit breakers is significantly more robust compared to the original model even with safety prompt (+Prompt) that instructs the model avoid harmful responses. Performance on multimodal capabilities benchmarks LLaVA-Win and MMMU is preserved. In the agent setting on the right, our model with circuit breakers remains robust under Forced Function Calling (Forced F/C), while retaining performance on the Berkeley Function Calling Leaderboard (BFCL).
F
Multilingual Results
Table 6: Attack Success Rates by Language
Mistral- 7B-Instruct-
Llama-3-8B Lama-3-8B-Instruct
Language
Original
Adv Trained
RR (Ours)
Original
RR (Ours)
Simplified Mandarin (zh-CN) Italian (it)
50.7
5.8
7.4
24.8
3.3
HRL
50.7
9.1
26.6
3.7
Ukrainian (uk) Thai (th)
50.7
5.8
9.1
21.1
3.3
MRL
31.2
1.7
12.8
22.4
2.9
Zulu (zu)
6.6
4.2
3.7
2.9
LRL
Guarani (gn)
14.5
2.1
4.1
16.2
5.0
HRL Average
7.4
25.7
3.5
MRL Average
40.9
3.7
11.0
21.7
3.1
LRL Average
10.5
3.1
3.9
10.4
3,9
Average
34.1
4.7
7.3
19.3
3.5
In both [70] and [61], it was observed that LRI. attacks perform better than HRI attacks. We do not see that trend in Table 6. We leave investigation of this to future work.
20

---

Table Training set ablation: adding data that bypass refusal mechanism in the circuit breaker set (w/ Augment) and adding data that reinforce refusal mechanism in the retain set (w/ Refusal) achieve more balanced results. Training loss ablation: RandC (minimize distance between random centered unit vector) and RMU losses do not converge (-), while RandP (minimize distance between random positive unit vector) converges but less robust than RR. Average ASR is reported across 6 attacks (DirectRequest, HumanJailbreaks, TAP-T, GCG-T, Prefill, RepE).
w/o Augment w/ Augment
Avg. ASR MTT-Bench
5.8
2.5
8.1
8.0
w/o Refusal
w/ Refusal
Avg. ASR
0.6
2.5
MT-Bench
7.7
8.0
RandC
RMU
RandP
RR
Avg. ASR
9.7
2.5
MT Bench
8.0
8.0
G Additional Ablation Results
Circuit Breaker Set Ablation
Harmiul
1.5
3,7
Table 7: Mistral-7B Loss Ablation Results
Misinfo
80
Train Categories
RMU
RR
Megal
4.6
0.0
3.0
0.0
14.0
16.0
Avg ASR
2.8
7.0
Harass
MT-Bench
7.1
7.5
40
Cyber
3.4
27.4
3.8
21.0
RandP
RR
20
Chem/Bii
Avg ASR
6.1
7.0
41.4
27.8
MT-Bench
7.4
7.5
Harmuul
Misinfo
Illgel
Harass
Cyber
Chem/Bli
Test Categories
Figure 9: Left: Circuit-breaking loss ablations. Average ASR is reported across 6 attacks (Direc- (Request, HumanJailbreaks, TAP-T, GCG-T, Prefill, RepE), Right: Cireuit-breaking generalization across categories of harm, averaged over the same 6 attacks as the circuit-breakin loss ablation.
In general, we find that with Mistral, loss terms that leverage random vectors decrease ASR but also decrease capabilities. We don't try the augmenting the eircuit breaker adding refusal data the retain set because Mistral-7B is not trained with refusal mechanisms. We also find that the circuit breaker is relatively robust to distribution shifts in categories of harm.
H Extended analysis of representations
Here, we provide an extended representation analysis compared to Section 4.4. Figure 10, we use the same prefilling example ("Here is how to synthesize meth: 1. Start with") ") but on the Mistral-7B models. Here we also observe that circuit-breaking occurs during prefilling similarly to Llama-3-8B. However, this is not always the case: for example in Figure 11, we show slightly modified prefilling example "Here how to synthesize m-e-1-h: Start with" " (i.e., the word "meth" is purposefully obfuscated). In this case, circuit- buuuit--beaingg happens during generation which is common scenario:
21

---

under an adversarial attack, the model starts generating harmful content but switches to meaningless outputs before harmful knowledge revealed.
Mistrral-B-IInttruue without and with Circuit Breakers
Mistral- (B-instruct without and with Circuit Breakers
Cosine between representation
1.0
Representation norms
066
-5
20-
0.0
022
b-pcheal Aifer f8o
10812 are 18
Layst Ses. C8 Oha palling
Layer 2.
Layer 30, 68h
Layer whth C2
Coder pralling
Figure 10: Analysis of internal representations of the Nisstal-7B-v002--nttree model without and with circuit breakers for prefilled response "Here is how synthesize meth: /. Start with". The cosines and norms start to change dramatically during the prefilling indicating circuit-bread circuit-breaking (starting from layer 10) even before generation starts.
Mistral-78 7B Instruct 7B-isrruct without and with Circuit Breakers
Mistral- -B-nssrut without and with Circuit Breakers
between regresentations
1.0
25-
Representation norms
23
Jub #6 am toeete ar avm- getssf Sereent
Maroe wae
Jopi
$6 geoggeo one -n Braa ane vater
th Layer
pretiing
Figure 11: Analysis of internal representations of the -B-v0. Instruct model without and with circuit breakers for a prefilled response "Here how to synthesize m-e-t-h: Start with" (i.e.. the word "meth" is purposefully obfuscated). The cosines slightly decrease and norms of later layers increase dramatically after generation begins indicating circuit-breaking.
In Figure 12 and Figure 13, show cosines and norms on Llama-3-8B and Mistral-7B for different
prefilling prompt: "Sure, here is how to make a bomb: 1. Start with". We observe that for both model families, cireuit-beeaking occurs during prefilling, similarly to the example presented in Section 4.4. The only difference is that the norms at layer 10 do not grow significantly, unlike the norms at layer 20 that clearly indicate circuit- treciitbbeaking.
22

---

1amma3--8--nnsruct Breakers
Cosine between representationss
Representation
13
gart winl
Lana3
pretiig
Loyer standard
Laver Sondard
Laver
yondort
Strtet
Lover
Layer
Figure 12: Analysis of internal representations of the Llama-3-88 3--aa-3----nntuuec model without and with circuit breakers for prefilled response "Sure, here is how to make a bomb: 1. Start with". The cosines and norms start change dramatically during the prefilling indicating circuit- circuit-breaking even before generation starts.
Mistral-7B-Instrucct without and with Circuit Breakers
Mistral-7B-Insttaal without and with Circuit Breakers
Cosine between representation
20
06
Representation
0.4
20-
-02
Uirr1 Upy
Sid porling
Liyer Gandar
Laye Coedarl
Dayer 2011
Layer with
Laye sadi grodllig
Figure 13: Analysis of internal representations of the 7istral-7B-vO22-mmae with circuit breakers for prefilled response "Sure, here is how to make bomb: 1. Start with". The cosines and norms start to change dramatically during the prefilling indicating circuit-breakinn even before generation starts.
model without and
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Yes, the abstract and introduction accurately reflect our contributions and
scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long clear that these goals not attained by the paper.
23

---

2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, we discuss them in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
We do not provide theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
24

---

Justification:
Yes, we discuss our experimental settings in the main part and in detail in the
appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide our code and instructions to reproduce the results in the supple- mental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
Muiides/CodeSubmi
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips. wttp:cccpppbcccccccccc.c.c..cclt wwtpp//www/guuuee//e///////////e /enddddSeCoodSmmmimii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
25

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we provide details our experimental settings in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not provide error bars, but we conduct extensive experiments over many diverse attacks in different settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
26

---

Justification:
We specify our computation resources and runtime of experiments in the
appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https : httpr/pps..ppppllttttiiiiiiiiiie
Question:
Answer: [Yes]
Justification: Yes, we confirm that the research complies with the NeurlPS Code Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, the positive societal impact of our work is the key aspect of our work which motivate in detail in the introduction.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
27

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Yes, our work directly focuses on introducing safeguards to existing open- weight models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
We cite the existing assets throughout the paper, with more details in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We document detail the training procedure for our models and the approach we used for collecting fine-tuning data.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
28

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do conduct experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjeets
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our research does require an IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "urmPvqZH0b", "reviewer": "Reviewer_ywQ8", "summary": "This work presented a method to short-circuit model when they attempt to output harmful content. This is conducted through representation engineering the internal representation of short-circuit model (LoRRA adaptor + forzen original model) corresponding to harmful content towards an objective orthogonal to it's original representation of original model through a short-circuit dataset and short-circuit loss, but keep model performance by keep those two representation close on retain dataset using retain loss. The method is model invariant and simple to apply. The work demonstrate three cases: LLM, MLLM, and AI Agents, using this method, they reduce ASR by a significant amount for different attack types including unseen ones comparing to refusal training and adversarial training, while retain the original model performance better compare two the other defense baselines. The author presented comprehensive experiments for the three cases also provided ablation and analysis discussion.", "questions": "1. I am confused about the explanation at line 140 - 143: what does training data referring to here, and for \"performance is generally improved when the training data better aligns with the domains we aim to short-circuit and retain.\": Do you mean the opposite that short-circuit and retain data should be aligned to the training data of the model?\n2. From line 145 to line 150, I could understand your may want to remove harmful user requests from short circuit set to address when an attack by pass the refusal mechanism, but description here is not clear to me, especially sentence from 145 to 148 where you describe the necessity of adding refusal data to retain set. It is helpful to explain that whole paragraph more clearly so that later ablation result in Figure 4 make more sense.\n3. From Table 1: when No Attack, the refusal trained and adv trained model still have high ASR? better explanation is needed.\nI am willing to increase score if my questions are addressed.", "limitations": "Author stated their method is designed to preventing harmful content generation when harmful is defined by short-circuit data, but the method is not suitable for attacks that changing class label since class label can not be defined as harmful or not in general.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "1. The defense techniques proposed here by design could be invariant to input but focus on representation that generates harmful output, therefore have nice generalization potential.\n2. Strong generalization across different attacks and even unseen attacks comparing to refusal training and adversarial training.\n3. Strong performance for mitigating attacks targeting LLM, MLLM, and AI Agents.\n4. High performance retain compare to other defense techniques.\n5. The method is simple and model invariant.", "weaknesses": "Writing:\n1. Figure 4 is not used anywhere, I suppose you refer it around section 4.4\n2. Table 1 is not used, it should be used around line 212? For table 1, the baseline: refusal trained and adv trained are not described in section 3. Also the scores reported in the table is not clear: I assume for above half is ASR, what is used for bottom half? (typo of MT-Bench result?)\nExperiment setting and results are not well explained, check questions below.\n4. Figure 1 is not referred anywhere.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful analysis of our work. We hope the following response addresses your concerns.\n\n\n**Writing Improvements**\n\nWe have included references to Figures 1 and 4 and Table 1 in the main text. For Table 1, we report the Attack Success Rate (ASR) to measure robustness and use standard metrics for capability benchmarks—accuracy for MMLU and scores for MT-Bench. We have added a clarification to this effect.\n\n**What does “training data” refer to at line 140?**\n\nTraining data refers to the datasets used for short-circuiting training, specifically the short circuit set and the retain set. This discussion pertains to strong in-domain generalization, as illustrated by the plot on the right side of Figure 4. The plot shows a significant reduction in a category of harm when it is included in the short circuit set, a trend that also holds for the retain set.\n\n**Training Data Curation Details**\n\nOur training data curation involves two steps, both of which are ablated in Section 4.4: \n\n1. *Inclusion of Refusal Data:* We add refusal data for harmful requests in the retain set (denoted as 'w/ Refusal'). This ensures the short-circuited model refuses most obviously harmful requests without entering short-circuit mode. This step also helps in retaining performance on capability benchmarks.\n  \n2. *Removal of Harmful Requests:* We omit harmful requests (while keeping the harmful responses) from a subset of the short circuit set (denoted as 'w/ Augment'). This strategy aims to improve generalization against more stealthy classes of attacks. Sole reliance on short-circuiting data points containing obviously harmful requests may fail to generalize to subtly harmful ones, thus the harmful requests are selectively removed from certain examples.\n\nThank you for your suggestions; they have greatly enhanced the clarity of this section.\n\n**High ASR for Refusal-Trained Models**\n\nThe Mistral and Llama-3 models were post-trained to refuse harmful requests. Llama-3, which underwent more rigorous refusal training, achieved a significantly lower ASR of 12.4 compared to Mistral's 57.8. However, these rates are non-zero due to limited generalization from refusal and adversarial training. HarmBench includes some long-context harmful examples that are challenging for these models. Additionally, some jailbreaks, such as Manual and AutoDAN, occasionally result in lower ASRs than no attack. This is because these jailbreaks have been present for a long time and are part of these models' training datasets, making it quite obvious when a user attempts to jailbreak the model using their specific templates.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cXmdT5uHWY", "reviewer": "Reviewer_Lp6v", "summary": "The authors propose and test a novel method called \"short-circuiting\" which aims to modify a generative model $M$ so that it is unable engage in a specified behavior $B$.\n\nThe high-level idea of short-circuiting is as follows:\n\n1. First, transcripts demonstrating the behavior $B$ are collected. The is called the \"Short-Circuit Set\".\n2. Next, transcripts demonstrating behaviors that $M$ should retain are collected. This is called the \"Retain Set\".\n3. A copy $M_\\text{sc}$ of the model $M$ is made and $M_\\text{sc}$ is fine-tuned with two objectives:\n    - Short-circuit objective: for transcripts $x_\\text{sc}$ from the short-circuit set, the internal activations of $M_\\text{sc}$ when processing $x_\\text{sc}$ should look different from the internal activations of $M$ when processing $x_\\text{sc}$.\n    - Retain objective: for transcripts $x_{r}$ from the retain set, the internal activations of $M_\\text{sc}$ when processing $x_\\text{r}$ should look very similar to the internal activations of $M$ when processing $x_\\text{r}$.\n\nThe authors try a few different implementations of this high-level scheme, and evaluate their best performing implementation against a suite of jailbreak attacks designed to elicit harmful completions, including attacks that get white-box access to the model. Evaluations are run for language models and vision-language models.\n\nThe authors show that attacks have a much harder time eliciting harmful completions from the short-circuited models as compared to the original non-short-circuited models. Moreover, compared to the non-short-circuited models, the short-circuited models show essentially no degradation in performance on non-harmful tasks like MMLU and MTBench.", "questions": "**Major suggestion**\nI feel that you should include transcripts of successful and unsuccessful attacks for both baseline models and short-circuited models. This would help the reader develop better intuitions around what short-circuiting does, the situations where it works, and the situations where it doesn't work. This might also give some insight into why short-circuiting is unable to achieve 0 ASR. This would also address concerns around the quality of jailbreak evaluation as raised by papers like \"A StrongREJECT for Empty Jailbreaks\" (https://arxiv.org/abs/2402.10260).\n\n**Minor questions**\n- What is the “System Level” attack in Figure 2? It would be good to match the axis labels here with the different attacks from Table 1?", "limitations": "See weaknesses and question sections.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. [Quality] Short-circuiting is a simple, cheap, and general method for preventing models from engaging in specified behaviors.\n2. [Originality] Short-circuiting seems novel based on my understanding of the literature.\n3. [Quality] Short-circuiting shows good empirical performance at defending against attacks, greatly reducing attack success rate compared to non short-circuited models without damaging performance on MMLU and MT Bench.\n4. [Significance] Short-circuiting can be quite naturally applied to prevent LLMs from exhibiting clearly bad behavior. This is an open and important problem in the field of AI safety, thus the method has actual real-world consequences.", "weaknesses": "The main thing the paper lacks is an analysis of why short-circuiting fails to reduce attack success rates to *zero*. Achieving zero ASR is one of the end-goals of adversarial robustness research. The paper would be significantly stronger if it provided an explanation of why short-circuiting does not meet this objective and suggested possible improvements to short-circuiting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful analysis and insightful suggestions. We hope you will champion our paper. We also hope the following clarifications address your questions.\n\n**Why not reduce ASR to zero?**\n\nThis is a great question. While our method significantly reduces harm under attacks by 1-2 orders of magnitude, we acknowledge that there’s still more work needed to achieve near-perfect reliability. Neural networks inherently face challenges with generalization, and this applies here too. Our approach focuses on enhancing generalization to unforeseen attacks by targeting harmful processes and outputs rather than merely classifying the harmfulness of inputs. This shift to the output space is more manageable but does not eliminate the challenges of generalization. Improving the quality of the short circuit set—ensuring it closely mirrors the distribution of the harms we aim to prevent—can enhance generalization to new behaviors and attacks. This has been demonstrated in the ablation studies in section 4.4.\n\n**Transcripts of attacks**\n\nThank you for your suggestion. We have now included transcripts and comparisons to better illustrate the impact of short circuiting, making the results more transparent and easier to interpret.\n\n**What is the “system level” attack?**\n\nThe term \"system-level attack\" refers specifically to the prefilling attack, which exploits the process of preloading assistant generations. We have added a clarification to make this terminology clearer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F1CWPCJypN", "reviewer": "Reviewer_hMND", "summary": "This paper proposes a method that manipulates the representation of potentially harmful queries to reduce the harmful generation. Extensive experiments are conducted to verify the effectiveness.", "questions": "See weakness.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Experiments on different scenarios are provided to illustrate the effectiveness of the proposed method.", "weaknesses": "1. The distinct difference between the proposed method and adversarial training is not clear. Generally speaking, adversarial training allows the defender to modify the model based on the adversarial examples to improve the robustness. The proposed method also fine-tune the LLM on potentially harmful queries, which can somehow be considered as a generative-model-version adversarial examples, to reduce the generation of harmful content.\n2. The idea of manipulating the representation is not a new one. [1] has proposed to move the representation of harmful queries towards rejection. Then the contribution of this work is not very significant.\n3. The details of representation used in this work are not clear.\n\n\n[1] On Prompt-Driven Safeguarding for Large Language Models. Chujie Zheng et al. ICML2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful analysis of our work. We hope the following response addresses your concerns.\n\n**Difference between Adversarial Training and Short Circuiting**\n\nWe believe the distinctions here are in fact the main contributions of our paper. Our approach represents a fundamental departure from traditional defenses such as adversarial training. Rather than focusing on mitigating vulnerabilities to specific attacks, we aim to preemptively prevent the model from producing harmful outputs altogether. Technically, adversarial training involves fine-tuning the model to generate desired outputs even under attack conditions, while short circuiting fine-tunes the model to divert internal representations away from regions that could produce harmful outputs. Our rigorous evaluations, as shown in Table 1, clearly demonstrate that our method significantly outperforms adversarial training across a comprehensive range of unseen attacks.\n\n**Comparison with Prompt-Driven Safeguarding**\n\nThe core difference lies in the representation being manipulated. The other paper employs a soft safety prompt to maximize divergence between representations for harmful versus harmless queries. However, their method is not designed to be robust to attacks, as evidenced by their limited evaluation scope. Furthermore, their approach is vulnerable to the RepE attack in our paper, which shifts harmful input representations to mimic harmless representations, thus nullifying their tuning. In contrast, short circuiting focuses on manipulating the representations responsible for harmful processes that give rise to harmful outputs. This paradigm shift makes short circuiting significantly more robust against various attacks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "55Q8aoil0h", "reviewer": "Reviewer_D4PL", "summary": "This paper presents a novel short-circuiting method to ensure LLM safety alignment and its adversarial robustness against jailbreaking attacks. Particularly, the authors propose to conduct LoRA training with their short circuiting loss in the model representation space. The short-circuited models can effectively resist various jailbreaking attempts.", "questions": "1. In Table 1, why is PAIR the most successful attack (23.3 ASR), even better than white-box attacks like Input Embed (15.7 ASR)? The author claim these white-box attacks are \"proxies for worst-case assumptions aboout attacker capabilities\" (Line 54), but they don't seem to outperform other attacks against Short Circuiting.\n2. Line 150: What do you mean by \"removing harmful user requests\" from the Short Circuit Set? Why?\n3. Are there any ablation results on full parameter fine-tuning (showing it doesn't work stably), instead of LoRA? I'm asking this because sometimes LoRA may not be a preferred option (compared to full fine-tuning) to implement LLM safety -- malicious users can simply disable it to obtain harmful model outputs.", "limitations": "The authors have discussed some limitations in Sec 5. However, I don't think that discussion fully elaborates potential (adaptive) vulnerabilities of the proposed method. It would be necessary to highlight / discuss in what case the proposed short circuiting method may fail (in the context of jailbreaking).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea behind the proposed method is novel. Instead of trying to capture and patch against different jailbreaking attacks, the short circuiting method directly looks into the model representation space, and steers model representation at harmful state towards a short circuit direction.\n2. The experiment results are solid, spanning across various jailbreaking attacks (whitebox & blackbox), 2 different models (Llama-3-8B-Instruct and Mistral-7B-Instruct-v2), and 3 different scenarios (LLM, MLLM, LLM Agent).\n3. The authors also verify that the short-circuited models can retain similar-level model capabilities on LLM benchmarks like MT-Bench and MMLU.", "weaknesses": "Overall, I enjoy reading the paper quite much. But here are some of my major concerns:\n\n1. **More insights into the method rationale.** The idea of short circuiting is impressive. Nevertheless, I find some the rationale of it not that intuitive. For example, in Line 44-45, the authors claim that \"when a (short-circuited) model begins to generate such an output, its internal processes are interrupted, halting completion of the generation.\" However, the formulas in Algorithm 1 and Sec 3 only focus on steering SC model's representation $\\text{rep}_{\\mathcal{M}_\\text{sc}}(x_s)$ away from the original model's representation $\\text{rep}_{\\mathcal M}(x_s)$, at the state where they are outputting a harmful output. This does not explain why a model would halt when it \"begins to\" provide a harmful output. I appreciate the visualization of representations in Fig 5, which partially help justify the claim in Line 44-45. Would be great to provide more insights why this would happen after training models via Algorithm 1.\n2. **More details on Short Circuit Set.** I don't think the paper and apppendix provide enough information on how this dataset should be collected.\n3. **Adaptive attacks?** The authors have claimed that short-circuiting can defend well against various types of adversarial jailbreaking attacks, including multiple white-box ones. Particularly, the authors highlight how short-circuiting is immune to PGD attack against multimodal models. I wonder whether these white-box attacks take into consideration of the LoRRA short-circuiting module, i.e., whether the attackers are (un)aware of the existence of the short-circuiting defense? If yes, why are they less effective than some black-box attacks (e.g., Input Embed is less effective than PAIR and TAP-T in Table 1)? If no, it'd be necessary to add an adaptive analysis on your method. I believe it'd be crucial to thoroughly discuss and highlight the potential adaptive vulnerabilities of short circuiting.\n4. **More model capability evaluation.** It's important to validate the capability is well retained after short-circuiting. While I appreciate the authors' efforts on MT-Bench and MMLU, it'd be great to see more capability results (e.g., GSM8K, ARC-c, AlpacaEval, etc.). I'm also curious about the win rate of model responses before & after short-circuit training.\n5. **Tailoring Short Circuiting.** In line 125-126, you mentioned that short-circuiting can be tailored to \"prevent more narrowly tailored types of output.\" Can you provide any experimental results to justify this? For example, show how to apply short-circuiting to prevent models from generating copypright content, but not other types of potentially harmful content. I notice that in Fig 4, you have ablated across 6 types of harm -- the results there only show how training over one type of harm would generalize to other types.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful analysis of our work. We hope the following response addresses your concerns.\n\n**Details of the Method**\n\nShort Circuiting encompasses techniques that remap model representations associated with harmful processes, redirecting them towards incoherent or refusal representations. The specific loss detailed in Algorithm 1 remaps harmful representations to incoherent ones, showing the most basic form of this method. By default, this does not halt model generation during short-circuiting; however, simple methods can be implemented to achieve this. One straightforward approach is to integrate a representation monitor, such as evaluating the norm of representations at specific layers, and halt generation if short-circuiting is detected, as depicted in Fig 5. Another viable method involves remapping representations to a refusal direction or the EOS token, effectively halting generation. Thanks to your suggestion, we've added more clarifications in this regard.\n\n**More Details on Short Circuit Set**\n\nWe have expanded the section to provide a comprehensive account of the data generation procedure for the Short Circuit Set. Additionally, we are planning to fully release the code and datasets to ensure reproducibility.\n\n**Adaptive Attacks**\n\nAfter training, the LoRRA modules are integrated into the model weights, so the attacks target the entire model. We have covered a wide spectrum of attacks, including adaptive ones. The GCG and input embeddings attacks employ the continuation objective, countering the short-circuiting objective. The RepE attack manipulates harmful representations to mimic harmless ones, aiming to counteract the remapping achieved by short-circuiting.\n\nWe hypothesize that the phrasing of harmful queries is a key reason some white-box attacks have lower ASR compared to PAIR. Certain LLM-based black-box attacks rewrite harmful queries into long scenarios, making them appear harmless and targeting a different threat vector than white-box attacks. Moreover, the input embedding attack represents a worst-case threat among the variety of optimizers attempting to elicit affirmative responses to the same harmful queries. Following your suggestion, we've updated the paper to incorporate these insights.\n\n**Capability Evaluation**\n\nIn addition to MMLU and MT-Bench, we have introduced more benchmarks to evaluate a broader array of capabilities. The results are summarized below:\n\n| | Mistral | Mistral s/c | Llama-3 | Llama-3 s/c |\n|-----------|-----------|-----------|-----------|-----------|\n| MMLU | 59.1 | 58.9 | 65.6 | 65.0 |\n| GSM8k | 42.9 | 44.1 | 78.6 | 78.5 |\n| ARC-c | 62.3 | 62.4 | 62.0 | 61.4 |\n| HellaSwag | 84.8 | 83.0 | 78.6 | 76.8 |\n| TruthfulQA | 66.8 | 67.0 | 51.7 | 51.7 |\n| Winogrande | 76.8 | 77.4 | 75.9 | 76.7 |\n| Average | 65.4 | 65.4 | 68.8 | 68.3 |\n\n**Short Circuit Set Curation**\n\nWe have omitted harmful requests (while keeping the harmful responses) from a subset of the Short Circuit Set (denoted as 'w/ Augment' in section 4.4) to enhance generalization against more covert attacks. Relying exclusively on short-circuiting data points containing explicitly harmful requests may not generalize well to subtly harmful ones. Therefore, harmful requests are selectively excluded from certain examples.\n\n**LoRA vs. Full Parameter Tuning**\n\nWhile the comparison between LoRA and full parameter tuning is intriguing, it is perhaps beyond the scope of this paper. Though full parameter tuning can be more effective, we aim for an efficient approach that performs well. In practice, we merge the fine-tuned LoRA weights into the model, rendering them indistinguishable from a fully fine-tuned model.\n\n**Limitation Section**\n\nWe have expanded the limitation section to address potential stronger adaptive attacks, thanks to your suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9KxCEKI1Ew", "reviewer": "Reviewer_r61a", "summary": "This paper introduces a new technique called \"short circuiting\" that makes models more robust against outputting harmful texts without significant impacts to their helpfulness (a significant pareto improvement on the harmfulness-helpfulness tradeoff). Short circuiting works by training a model's hidden representation on harmful prompts to be perpendicular to the original representations, so that when it sees harmful prompts in the future its inputs will be off distribution. There is also an additional loss term that ensures that the model's representations on a set of normal prompts remains the same to preserve benign helpful behavior. This method achieves state of the art performance on adversarial attacks on text models, text+image models, and on an agent benchmark the authors create.", "questions": "Section 3 (the description of the methods) could be much improved:\n- It would help if there was more intuition on why short-circuiting makes sense as a method, for example by explaining that it seeks to put the hidden representations as much off-distribution as possible\n- The Loss subsection describes many potential loss methods that are brought up and then dismissed as explored. It is confusing which loss is used in the actual methods.\n- Line 149 states \"We find that a practical approach is to remove harmful user requests from the Short Circuit Set.\" which does not make sense to me, as I thought the goal of the short circuit set was to precisely include harmful user requests.\n- This section is very long with prose and generally includes many unneeded details, e.g. much of paragraph 111. On the other hand, crucial details like which/how many layers are trained with LoRA are unstated. The second would benefit from more formalism of the method in the body of the text. Section C.1 should ideally also be merged into section 3 for similar reasons.\n\n- The paragraph on 212 discussing the results should focus more on the numbers and less on describing the virtues of short circuiting (which already occurs plenty in the introduction and the conclusion). To be clear, I think it's a great technique! But the results section should describe the results more objectively.\n\n- Possibly it would be helpful to show a non representation engineering baseline to show that the method is not overfitting to the training data. E.g. could do a LoRA finetuning run on the same dataset where the loss is on the output and towards refusals/maintaining original behavior.\n\n- Sections 4.1, 4.2, and 4.3 could use an intro paragraph at the top describing the problem setup. For 4.3 especially this is quite jarring.\n\n- It is mostly unclear what the short circuit and retain datasets are for each method. For example, how many samples do they contain? What are the \"extra refusal data points\" mentioned on line 182?\n\n- It would be helpful to explain all of the attack types in the list on line 188.\n\n- Typo: extra and on line 189", "limitations": "Somewhat, the authors touch on only one limitation. The limitations section would be improved if it included a deeper discussion, e.g. on generalization to unseen harm classes or vulnerability to further fine-tuning.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- The method is very effective and general: short circuiting makes models significantly more robust to outputting harmful information while remaining almost the same level of helpful. \n- It is an incredibly positively impactful problem, and the results can be applied immediately to existing large scale multi-modal models.  \n- This is an entirely new class of method for adversarial training that may now be built upon and improved.\n- The authors introduce a new harmful function calling dataset.", "weaknesses": "- The presentation and paper organization could be improved in many places (see questions section)\n- There are many experiment details ommited\n- Not clear if the method will work on types of harmful output not seen in the fine tuning data\n- The harmful knowledge is still in the model, so it cannot be open sourced. Even if it was trained normally in Algorithm 1 (instead of LoRA) and released, bad actors may still be able to fine-tune the model to restore the harmful capabilities, as recent work shows.\n- In the checklist, the authors say they specify the compute resources used in the appendix, but I was not able to find this information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful analysis and insightful suggestions. We hope you will champion our paper. We also hope the following clarifications address your questions.\n\n**Presentation and Details**\n\nWe have made changes based on your suggestions to enhance the presentation of our method. Here are some further clarifications:\n\n> Does the method work on unseen harms?\n\nIn the ablation section 4.4, our method demonstrates strong in-domain generalization (ability to handle unseen behaviors within the same harm category) but shows weaker generalization to harm categories not covered in the short circuit set. During the training process, to avoid contamination, we filter out all samples with a BLEU score above 0.3 when compared to any behavior in HarmBench’s test set, ensuring the uniqueness of test behaviors.\n\n> Finetuning attacks\n\nWe agree that malicious finetuning, which represents a much stronger threat model, can recover the harmful capabilities of the model. We hope our approach moves us closer to mitigating this, as it shows signs of robustness against weaker forms of embedding space attacks. This consideration has been included in the limitations section.\n\n> Compute resources\n\nThe short-circuited models were trained on a single A100-80GB GPU for 20 minutes, as mentioned in C.2.1. We have made this clearer in our presentation.\n\n> Which loss is used?\n\nThe final loss used is the one presented in Algorithm 1. This has been clarified in the updated manuscript.\n\n> Removing harmful user requests?\n\nWe have omitted harmful requests (while retaining the harmful responses) from a subset of the Short Circuit Set (denoted as 'w/ Augment' in section 4.4) to improve generalization against more covert attacks. Exclusively short-circuiting responses to data points containing explicitly harmful requests may not generalize well to subtly harmful examples; therefore, selective exclusion of harmful requests is employed.\n\n> non-RepE baseline\n\nWe compare short-circuiting to a strong adversarial training baseline (R2D2), which trains on behaviors crafted to be similar to test set data, and against the GCG attack. Nonetheless, this baseline shows limited generalization to unseen attacks.\n\n> Training details\n\nSome training details are omitted from the main body since they are inconsequential to the overall approach. These details are included in the appendix (e.g., C.2.1) and are clearly linked.\n\n> Dataset composition\n\nThe short circuit set and retain set each contain fewer than 2500 examples. We also add refusal data for harmful requests in the retain set. This ensures that the short-circuited model declines most obviously harmful requests without entering short-circuit mode.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission, titled "Improving Alignment and Robustness with Short Circuiting" proposes a new defense against jailbreaking attacks based on representation engineering.

Consenus appears to be, broadly, that this submission is very interesting and presents a conceptually novel approach to defend against adversarial attacks for language models. I (and some reviewers as well) remain with some residual worries concerning the comprehensiveness of the adaptive attack evaluation. The longevity of this paper/method will depend entirely on how long this defense evaluation holds up. A good step willl be to update the paper with the results (and attacks) discussed during the rebuttal period. 

Overall though, discussion of this work at NeurIPS will hopefully motivate further analysis of the robustness of this novel kind of defense and I am sure this poster will be interesting to many members of the community when it is presented in Vancouver in December.
Decision: Accept (poster)