The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence
Authors: Tom Wollschläger
Jannes Elstner
Simon Geisler
Vincent Cohen-Addad
Stephan Günnemann
Johannes Gasteiger
Venue: ICML 2025 poster (2025)
Keywords: Refusal directions
Large Language Model
Activation Steering
Interpretability
Representation Engineering
Probing
Causality
Interventions
Primary area: deep_learning->large_language_models
TL;DR: We identify that refusal is mediated by polyhedral cones and analyze the interactions of their basis vectors with our novel notion of independence.

Abstract: The safety alignment of large language models (LLMs) can be circumvented through adversarially crafted inputs, yet the mechanisms by which these attacks bypass safety barriers remain poorly understood. Prior work suggests that a *single* refusal direction in the model's activation space determines whether an LLM refuses a request. In this study, we propose a novel gradient-based approach to representation engineering and use it to identify refusal directions. Contrary to prior work, we uncover multiple independent directions and even multi-dimensional *concept cones* that mediate refusal. Moreover, we show that orthogonality alone does not imply independence under intervention, motivating the notion of *representational independence* that accounts for both linear and non-linear effects. Using this framework, we identify mechanistically independent refusal directions. We show that refusal mechanisms in LLMs are governed by complex spatial structures and identify functionally independent directions, confirming that multiple distinct mechanisms drive refusal behavior. Our gradient-based approach uncovers these mechanisms and can further serve as a foundation for future work on understanding LLMs.

The Geometry of Refusal in Large Language Models:
Concept Cones and Representational Independence
Tom Wollschläger I
Jannes Elstner",
Simon Geisler 12 Vincent Cohen-Addad 3
Stephan Günnemann'
Johannes Gasteiger 34
Abstract
these advancements also introduce serious risks. As artificial intelligence becomes more powerful, it can be misused for harmful purposes, such as attacking critical in- frastructure or spread- ing misinformation. En- suring that these mod- els remain aligned with human values has be- come a erucial research challenge (Liu al., 2023; Schwinn et al., 2025). Despite signifi- cant progress, LLMs, like all machine learning models, re- main vulnerable to adversarial attacks that can bypass align- ment mechanisms and induce harmful outputs (Szegedy et al., 2014; Carlini et al., 2024).
Cone
Basis vector
The safety aligmment of large language mod- els (LLMs) can be circumvented through adver- sarially crafted inputs, yet the mechanisms by which these attacks bypass safety barriers re- main poorly understood. Prior work suggests that single refusal direction in the model's ac- tivation space determines whether an LLM re- fuses a request. In this study, we propose a novel gradient-bane based approach representation engi- neering and use it to identify refusal directions. Contrary to prior work, we uncover multiple in- dependent directions and even multi-dimensional concept cones that mediate refusal. Moreover, we show that orthogonality alone does not im- ply independence under intervention, motivating the notion of representational independence that accounts for both linear and non-linear effects. Using this framework, we identify mechanisti- cally independent refusal directions. We show that refusal mechanisms in LLMs are governed by complex spatial structures and identify func- tionally independent directions, confirming that multiple distinct mechanisms drive refusal behav- ior. Our gradient-based approach uncovers these mechanisms and can further serve foundation for future work on understanding LLMs.
Figure 1. example a 3D concept cone with its basis vec- tors. directions in the cone mediate refusal.
Recent work in interpretability has provided valuable in- sights into how LLMS encode and process information (Nanda et al., 2024; Wang et al., 2022; Cunningham et al., 2023; Heinzerling & Inui, 2024). Prior studies (Bel- rose et al., 2023; Gurnee & Tegmark, 2023; Marks & Tegmark, 2024) suggest that concepts raaniinggnnnnggggnnn from sim- ple to complex- are often encoded linearly in the model's residual stream. Methods such as representation engineering (Zou et al., 2023a) allow researchers to use input prompts to analyze model behavior by extracting and manipulating such concepts. However, the mechanisms enabling adversarial jailbreaks that bypass alignment safeguards remain poorly understood. Some evidence suggests that refusals to harm- ful queries are mediated by single "refusal direction" in activation space (Arditi al., 2024), and that jailbreaks rely manipulating this direction (Yu et al., 2024), yet these assumptions require further examination. Understanding re- fusal mechanisms is crucial, as it has direct implications for both offensive capabilities cerabiilies--iffrmnnnn inffrrmin more sophisticated jailbreaks (Huang et al., 2024) and more importantly de- veloping robust defensive strategies like improved adversar- ial training (Yu et al., 2024) and inference time monitoring.
1. Introduction
The breakthrough of scaling large language models (LLMs) has led to an unprecedented leap in capabilities, driving widespread woal-world adoption (OpenAI, 2022). However, 'Equal contribution 'School of Computation, Information & Technology and Munich Data Science Institute, Technical University of Munich "Now at Google Research "Google Re- scarch "Now at Anthropic. Correspondence to: Tom Wollschläger <tom, wwwlschlaegerruuu.... tum.....rrrtm...........
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

The Geometry of Refusal in Large Language Models
In this work, we go beyond extracting concepts using com- mon input prompt methods by introducing novel gradient- based approach to representation engineering which we use to investigate the mechanisms underlying refusal behavior in LLMs. We extract refusal-mediating directions more effectively, improving both precision and control while min- imizing unintended side effects, which we demonstrate in Section 4. Unlike prior work that assumes model refusal is controlled by single linear direction, we show in Section 5 that there exist multi-dimennsonal polyhedral cones which contain infinite refusal directions; we show an illustrative example in Figure 1. To further characterize refusal mech- anisms in language models, we introduce representational independence, a criterion for identifying directions that re- main mutually unaffected under intervention, capturing both linear and non-linear dependencies across layers. In Sec- tion 6, demonstrate that even under this strict notion of independence, multiple complementary refusal directions exist.
feature from the model's residual stream as shown by prior work (Panickssery et al., 2024; Bolukbasi al., 2016; Burns et al., 2024). Recent studies obtain this direction by com- puting the difference- perrence-i--mans (DIM) (Panickssery et al., 2024; Arditi et al., 2024; Stolfo et al., 2024) between model representations on datasets of harmful prompts D Puumm and harmless prompts D gooo
Σ x (o') D''
Σ DE Pake
x (1)
(p)
12 Pharm
Here, (0 (p) represents the residual stream activations at position i, layer for input prompt p.
Adversarial steering attacks. The extracted harmfulness direction can be used to manipulate the model's refusal behavior. With white-box access, an attacker can prompt the model with harmful queries and suppress activations in the harmfulness direction, thereby reducing the model's probability of refusal. This can be done through directional ablation of r (where denotes the unit vector) (Zou et al., 2023a):
To summarize, our core contributions are:
We show that our gradient-based representation engi- neering can advance general LLM understanding and specifically demonstrate its efficacy for understanding refusal mechanisms.
!I) (1) -= x ff x
(1)
which projects the residual stream to a subspace orthogonal to r, or alternatively through activation subtraction:
We introduce representational independence, practi- cal framework for characterizing how different inter- ventions interact within an LLM's activation space, and use it to find independent refusal directions.
(1) x a.
(2)
We show that rather than a single refusal direction, there exist tions mediate refusal.
which subtracts a scaled r from the residual stream. We follow common practice to apply both operations across all token positions and ablation across all layers while doing subtraction only single layer (Arditi et al., 2024).
cones in which all direc-
2. Background
Notation. Let f T- - A X denote a language
3. Related Work
model, where TT| is the probability simplex over vocab- ulary T. Given a prompt p- (N,, E TN con- sisting of tokens t,, each token is first embedded: z!!") (M)ED((:). The model then processes the token sequence through L layers, where at each layer / = 1,..., L and token position i the following transformation is applied:
Adversarial attacks for LLMs. Many studies have ex- plored hand-crafted adversarial techniques, such as persona modulation (Shah et al., 2023), language modifications (Zhu et al., 2023), or prompt engineering using repetitions and persuasive phrasing (Rao et al., 2024). Other works take more systematic approach, employing techniques like ge- netic algorithms and random search (Chen et al., 2024), discrete optimization over input tokens (Zou et al., 20236), or gradient- based methods to identify high-impact pertur- bations (Geisler et al., 2024). While identifying these vul- nerabilities enables adversarial fine-tuning (Xhonneux et al., 2024) or improved training through Reinforcement Learning with Human Feedback (RLHF), recent works suggest that robustness remains challenge (Zou et al., 2023a; Schwinn et al., 2024; Geisler et al., 2024; Scholten et al., 2025).
( !!!0 + ATTN ( (112); x.
((+1) -!6' + MLP(#.))
The final residual stream x (L+1) is unembedded to yield log- its: E; (NEMBED(1,) (L.4)) The softmax function con- verts these logits into a probability distribution over tokens: P(t f1....,44) 4)) softmax((.)). We omit technical details that are not critical for this work such as LayerNorm.
Extracting refusal directions. Paired prompts of harmful and harmless requests allow the extraction of directional
Interpretability of LLMs. A parallel line of research fo- cuses on understanding the internal mechanisms of L.LMs,
Resources code: es. tetttuddellammggmeetaaaeeeessss
2

---

The Geometry of Refusal in Large Language Models
as their natural language outputs provide unique oppor- tunity to link internal states interpretable behaviors. In- terpretability research has led to the identification of var- ious "features" Saauuress"coon concepts represented by distinct activa- tion patterns (Cunningham et al., 2023)-as well as "cir- cuits", which are subnetworks that implement a specitic function or behavior. Prominent examples are backup cir- cuits (Nanda al., 2024) and information mover circuits (Wang et al., 2022). Many interpretability insights rely extracting features using paired inputs with opposing se- mantics (Burns et al., 2024) and then manipulating residual stream activations to elicit specific behaviors (Panickssery et al., 2024). Representation engineering, as proposed by Zou et al. (2023a), investigates the linear representation of concepts such as truthfulness, honesty, and fairness in LLMs. The effectiveness of these methods supports the hypothesis that many features are encoded linearly in LLMs (Marks & Tegmark, 2024). These insights allow researchers to pin- point and manipulate concept representations or specific circuits, enabling targeted debugging of behaviors, mitigat- ing biases, and advancing safer, more reliable AI systems.
4. Gradient-based Refusal Directions
Research Question: Can our gradient- repre- sentation engineering identify refusal directions?
To investigate the refusal mechanisms in language models, we propose a gradient- based algorithm that identifies direc- tions controlling refusal in the model's 's activation space. We refer to it as Refusal Direction Optimization (RDO). Un- like prior approaches that extract refusal directions using paired prompts of harmless and harmful instructions (Arditi et al., 2024), our method leverages gradients to find better directions instead of solely relying on model activations. Similar to (Park et al., 2023), we define two key properties for refusal directions:
Definition 4.1. Refusal Properties:
Monotonic Scaling:
when using the direction for ac- tivation addition/subraation a!)) x a- r, the model's probability of refusing instructions should scale monotonically with a.
Understanding Refusal Mechanisms. Recent research has focused on understanding the mechanisms underlying refusal behaviors in LLMs. For example, removing safety- critical neurons has been shown to decrease robustness (Wei et al., 2024; Li et al., 2024b). Zheng et al. (2024) demon- strate that adding explicit safety prompts shifts the internal representation along a harmfulness direction. O'Brien et al. (2024) propose to use sparse autoencoders to identify latent features that mediate refusal. The most relevant work to ours is Arditi et al. (2024), which builds on Zou et al. (2023a) and examines the representation of refusal in LLMs. Their work suggests that single direction in model's activation space determines whether the model accepts or refuses a request. We challenge this notion by showing that refusal is mediated through more nuanced mechanisms. Concurrently, Pan et al. (2025) identify multiple independent refusal direc- tions, providing more evidence to findings in Section 6. While their focus is on representation shifts during safety finetuning, our work introduces a novel, gradient-based top-down discovery approach applicable to any model.
Surgical Ablation: ablating the refusal direction through projection a = x - - x! should cause the model to answer previously refused harmful prompts, while preserving normal behavior on harm- less inputs.
We can encode the desired refusal properties into loss func- tions, allowing us to find corresponding refusal vectors r using gradient descent. For the monotonic scaling property, we train the model to refuse harmless instructions Psate when running the model with modified forward pass fadd(r.)) in which we add r to the activations at layer 1. We mini- mize the cross-entropy between the model output and target refusal response tretusal. For the surgical ablation property, we similarly compute the cross-entropy between a harmful response target answer and the output of a modified forward pass fablat((+) to make the model respond to harmful instruc- tions. A key strength of our gradient-based approach is the ability to control any predefined objective and thus can control the extent to which other concepts are affected dur- ing interventions. For this, we use retain loss based on the Kullback-Leibler (KI.) divergence to ensure that directional ablation of r on harmless instructions does not change the model's output over a target response Tremiin- Algorithm I shows the full training procedure for our refusal directions.
Over-refusal Recent work addresses LLM over-refusal, where benign queries are rejected by overly strict safety filters. Röttger et al. (2023) (XSTest) and An al. (2024) (PHTest) provide benchmarks to quantify false refusal rates, expanded by Cui et al. (2025)'s OR-Bench. Shi et al. (2024) attribute this to lexical shortcuts, proposing self-contrastive decoding for mitigation. Similarly, Li & Liu (2025) high- light over-defense in prompt injection guards with NotInject and InjecGuard. Collectively, these studies underscore the challenge of balancing safety and helpfulness, offering re- sources to diagnose and mitigate overrefusal.
Setup. We construct dataset of harmless and harmful prompts from the ALPACC (Taori et al., 2023) and SALAD- BENCH (Li et al., 2024a) datasets (see Appendix A. An important consideration for our algorithm is the choice of targets fanswer and tretusal. Generally, language models differ in their refusal and response styles, which is why we use
3

---

The Geometry of Refusal in Large Language Models
Method
Operation
DIM
RDO (ours)
Directional Ablation
Activation
1.0
Subtraction
Attack Success Rate
0.8
0.6
0.4
0.2
0.0
Gemmaa 2 2.2B
Gemma 2 9B
Qwen 2.5 1.5B
Qwen 2.5 3B
Qwen 2.5 7B
Qwen 2.5 14B
Llama 3 8B
Figure 2. Attack success rates of refusal directions for different models. We compare the DIM direction baseline that is extracted from prompts against Refusal Direction Optimization for jailbreaking with directional ablation and activation subtraction.
Algorithm Refusal Direction Optimization (RDO)
2024), Qwen2.5 (Yang et al., 2024), and Llama-3 (Dubey et al., 2024) families and compare against the DIM direction for which we use the same setup as Arditi et al. (2024) but with our expanded dataset. For fair comparison, we train the refusal direction at the same layer that the DIM direction is extracted from, and during activation addittion/subraaction set the scaling coefficient a to the norm of the DIM direc- tion. We evaluate the jailbreak Attack Success Rate (ASR) JAILBREAKBENCH (Chao et al., 2024) using the STRON- GREJECT Ane-tuned judge (Souly al., 2024). For in- ducing refusal via activation addition, we test 128 harmless instructions sampled from ALPACA using substring match- ing of common refusal phrases. Model completions for evaluation are generated using greedy decoding with a max- imum generation length of 512 tokens.
Input: Frozen model f. scaling coefficient a, addition layer index lad learning rate 7), loss weights >abl. Aadd- Aret, and data D {(Pbam,,. Psafe,i s: fanswer, is E (r)
Output: Refusal direction
Initialize randomly
2: while not converged do
3: Sample batch B DD
4: L + -O-MOOELLLSSS:
5: r
6: r r r/rlll
7: end while
1: function COMPUTELOSS(r, f. f.B)
2:
Pharm Psafe, fanswer: trefusal, Eretain = B
3:
Cablation = CE..OS (faimco))))))) (Pham), fanswer)
Does the direction mediate refusal? In Figure 2, show that for jailbreaking, approach is competitive when ing directional ablation and, average, outperforms DIM when subtracting the refusal direction. Notably, despite not being explicitly optimized for subtraction attacks, our direction naturally generalizes to this setting. Figure 10 shows that adding the refusal direction to harmless inputs induces refusal more effectively with RDO than with DIM, further indicating that our method manipulates refusal more effectively.
4:
Luddition = CELOSSLLSSS.... (EELOOSSSaaataarr (fadd(arP (Peate); frefusaa)
5:
Cretain (L((abluter)) (Psate), ), (Peate) tretain)
6: L C Cablation + add Caddition + A Eretain
7:
return L
model-specific targets rather than generating them via censored LLMSs in Zou et al. (2024). Specifically, we use the DIM refusal direction to generate our targets, though any effective attack can work. For the harmful answers fanswer, we ablate the DIM direction and generate 30 tokens. Similarly, we use activation addition on harmless instruc- tions to produce refusal targets frefusal- For helpful answers on harmless instructions that should be retained factain we generate 29 tokens without intervention. The retain loss Cretain is applied over the last 30 tokens, such that the last token of the model's is chat template is included. We detail hyperparameters and implementation in Appendix A.
Is the direction more precise? To measure the side effects when intervening with the directions we track benchmark performance. Arditi et al. (2024) show that directional abla- tion with the DIM direction tends to have little impact on benchmark performance, except for TruthfulQA (Lin et al., 2021). In Table 1, we show that RDO impacts TruthfulQA performance much less severely, reducing the error by 40% average. We show the results for more benchmarks in Appendix B.3.
Evaluation. We evaluate our method by training a refusal direction on various models from the Gemma 2 (Team et al.,
We then evaluate the trade-off between safety and over-

---

The Geometry of Refusal in Large Language Models
constraint X 2 0 ensures that all directions within the cone consistently strengthen refusal behavior. Without this constraint, allowing negative coefficients could introduce opposing effects, reducing the overall effectiveness. En- forcing orthogonality of the basis vectors prevents finding co-linear directions. Note that in practice, directions in acti- vation space cannot be scaled arbitrarily high without model degeneration, which effectively bounds AІ.
Table TruthhulQA benchmark performance for directional abla- tion with the DIM or RDO directions, compared to the baseline (no intervention). Higher values indicate better performance.
Chat model
DIM
RDO (ours)
Baseline
GEMMA 2 2B
47.8
51.4 (+3.6)
55.8
GEMMA 2 9B
52.8
56.7 (+3.9)
61.1
LLAMA 3 8B
48.7
51.0 (+2.3)
52.8
QWEN 2.5 1.5B
42.9
44.0 (+1.1)
46.5
QWEN 2.5 3B
54.2
54.5 (+0.3)
57.2
Algorithm 2 Refusal Cone Optimization (RCO)
QWEN 2.5 7B
58.7
60.0 (+1.3)
63.1
1: Initialize B [b,...., bn] randomly
QWEN 2.5 14B
63.3
67.9 (+4.6)
70.8
2: while not converged do
3: Sample batch B ~ DD
refusal for RDO and DIM on the XSTest benchmark (Röttee et al., 2023). As detailed Appendix G, RDO consistently achieves a higher refusal rate for harmful in- puts while maintaining or reducing the benign over refusall rate compared to DIM. This means that for any given level of benign over-refusal, our method refuses more harmful requests, thereby yielding a uniformly better trade-off.
4: Laample f- R Sym-Sample(B)) | |COMPUTELLOSS(r, S. B)]
5: Lhusis f- 1 Ei COMPUTELOSS(,,
6:
L Lrample + Lhasis
7:
B t B VB
B < GRAMSCHMIDT(B)
end while
Is our method versatile? Hyperparameter tuning of the retain loss weight Are in Algorithm 1 allows for balancing between attack success and side effects ( (Appendix C.2). We observe that for many models-cspecialle especially those in the Qwen 2.5 family-the majority of estimated DIM directions have too high side- effects, rendering it an unsuccessful attack (Figure 16). Our method is more flexible than previous work as we can choose the target layer freely while limiting side effects through the retain loss.
1: function SAMPLE(B)
8 ~ Uninff(a E [R. ||0/||2 = 1)
r= Bs
return T
In Algorithm 2, we describe the procedure to find the cone's basis vectors. The basis vectors are initialized randomly and iteratively optimized using projected gradient descent. We compute the previous losses defined in Algorithm on Monte Carlo samples from the cone, as well as on the ba- sis vectors themselves. Computing the loss on the basis vectors improves both stability and the lower bounds of the ASR. This is because the basis vectors are the bound- aries of the cone and thus tend to degrade first. After each step, we project the basis back onto the cone using the Gram- Schmidt orthogonalization procedure. Because the directional ablation operation uses the normalized f rather than r, sampling convex combinations of the basis vectors and normalizing them would introduce bias towards the basis vectors themselves. Instead, we sample unit vectors in the cone uniformly ensure better coverage of the space.
Key Takeaways. Our RDO yields more effective re- fusal directions with fewer side effects, establishing that gradient- based representation engineering is an effective approach for extracting meaningful direc- tions, while allowing for more modeling freedom such as incorporating side constraints.
5. Multi--dimensionll Refusal Cones
Research Question: Is refusal in LLMs governed by a single direction, or does it emerge from more complex underlying geometry?
Can we find refusal concept cones? We train cones of in- creasing dimensionality using the same experimental setup as described in Section 4. We measure the cone's effective- ness in mediating refusal by sampling 256 vectors from each cone and computing the ASRs of the samples for directional ablation. We show the results in Figure and confirm that the directions in the cones have the desired refusal proper- ties in Figure 20. Notably, we identify refusal-mediatiee cones with dimensions up to five across all tested models. This suggests that the activation space in language models exhibits a general property where refusal behavior is en- coded within multi-dimensional cones rather than a single
We extend RDO to higher dimensions by searching for regions in activation space where all vectors control re- fusal behavior. For this, we optimize an orthonormal basis B = [b.,...,de bN] spanning an N N-dimensional polyhedral cone RN = EEN b,b: x 2 03}[0}, where all direc- tions r E R satisfy the refusal properties (Definition 4. I.... Since all directions in the cone correspond to the same re- fusal concept, we also refer to this as concept cone. The
5

---

The Geometry of Refusal in Large Language Models
1.0
0.8
Attack Success
0.6
Geemma 9B
Qwen 2.5
Llama 3 8B
0.4
Basis Vectors
0.2
0.0
Cone Dimension
Figure 3. Attack success rate for
cones for Gemma 2, Qwen 2.5 and Llama 3. The cone performance is measured via the performance of Monte Carlo samples which are depicted boxplot.
linear direction.
tack. However, the higher ASR in the low-sample regime suggests that different directions capture distinct, comple- mentary aspects of the refusal mechanism. Additionally, Figure 19 reveals that ASR increases with cone dimensional- ity but plateaus at four dimensions. This trend indicates that figher-dimensional cones offer an advantage over single- direction manipulation, likely by influencing complemen- tary mechanisms. The plateau likely occurs because the model does not support digher-dimensional refusal cones.
Do larger models contain digher-dimensions cones? In Figure 4, we evaluate the effect of model size within the Qwen 2.5 family. We observe that across all model sizes, the lower bounds of cone performance degrade significantly as dimensionality increases. In other words, a higher number of sampled directions have low ASR. Larger models appear to support higher dimensional refusal cones. A plausible explanation is that models with larger residual stream di- mensions (c.g.. 5120 for the 14B model vs. 1536 for the 1.5B model) allow for more distinct and orthogonal direc- tions that mediate refusal. Finally, in Figure 18, we confirm that directions sampled from these cones effectively induce refusal behavior, further supporting the notion that multiple axes contribute to the model's 's refusal decision.
Key Takeaways. We show that refusal mecha- nisms in LLMS span high-dimensional polyhedral cones, capturing diverse aspects of refusal behav- ior. This highlights their geometric complexity and demonstrates the effectiveness of our gradient- based method in identifying intricate structures.
Do different directions uniquely influence refusal?
To further investigate the role of different vectors, we as- sess whether multiple sampled cone directions influence the model in complementary ways. Specifically, we sample varying numbers of directions from Gemma-2-2 's four- dimensional refusal cone and, for each prompt, select the most effective one under directional ablation (more details in Appendix A). To ensure a fair comparison, we use tem- perature sampling with the digensiin RDO direction to generate the same number of attacks and similarly select the most effective instance. We study Gemma 2B and sample from its four- dimensiiolo cone, since performance degrades significantly for larger dimensions (see Figure 17).
6. Mechanistic Understanding of Directions
Research Question: Are there genuinely indepen- dent directions that influence model's refusal be- havior? Can we access the discovered refusal direc- tions through perturbations in the token space?
In the previous section, we demonstrated that refusal be- havior spans multi-dimensional cone with infinitely many directions. However, whether the orthogonal refusal- mediating basis vectors manipulate independent mecha- nisms remains an open question. In this section, we conduct a mechanistic analysis to investigate how these directions interact within the model's activation space and whether they can be directly influenced through input manipulation.
Figure 5 shows that sampling multiple directions leads to higher ASR compared sampling with various tempera- tures in the low- low-sample regime. For a higher number of samples, the randomness dominates the success of the at-
6

---

The Geometry of Refusal in Large Language Models
0.8
Attuck Success Raie
Qwen 2.5 1.58
0.6
Qwen 2.5 3B
Qwen 2.5 7B
Qwen 2.5 148
0.2
0.0
2
3
Cone Dimension
Figure Refusal evaluation different cone dimensions for the Qwen2.5 model family. The cone performance for models with fewer parameters degrades faster with increasing cone dimension compared to larger models.
0.92
though their assumptions differ from ours, especially since they assume ome-to-nn mapping from output feature to direction in activation space, their experiments suggest that independent directions are almost orthogonal. This moti- vates a deeper empirical examination of how orthogonal refusal directions in language models interact practice.
0.90
Attack Success Rate
0.88
0.86
RDO with T=0.2
0.84
RDO with T=0.5
RDO with T=1
Are orthogonal directions independent? To explore this, we first use RDO identify direction r that is orthog- onal to the DIM direction v, i.e., r r = 0. We then measure how much one direction influenced when ab- lating the other direction by monitoring the cosine similarity cos A, 14) = A X||-||||T between prompt's representation in the residual stream æ and the directions and r. Specifi- cally, track: cos(r, ax!" (Pharm) )) and cos w (Pharm)) at the last token position and for all layers ( {O..... L} 128 harmful instructions in our validation set. Intuitively, ablating a causally independent direction in earlier layers should not intervene with the reference direction later layers. Otherwise, there some indirect influence through the non-linear transformations of the neural network.
0.82
RDO with T=1.2
Cone Sampling
8
16
32
64
Number Samples
Figure ASR best-of-N sampling using N samples from 4 dimeensional refusal cone of Gemma 2B, compared to best- of-N sampling with temperature using single-dimension RDO.
This allows us to determine whether they are merely latent properties of the network or actively utilized by the model in response to specific prompts.
6.1. Representational Independence
The top row of Figure 6 shows how the cosine similarity be- tween the RDO and DIM directions changes under interven- tion. The left plot shows the cosine similarity between the RDO direction and the activations on a normal forward pass (solid line) and while ablating the DIM direction (dashed line). The right plot presents the reverse setting. Despite enforced orthogonality. ablating RDO indirectly reduces the representation of the DIM direction in the model activations in the later layers, measured by cosine similarity. This effect is reciprocal, suggesting that orthogonality alone does not guarantee independence throughout the network. In Figure we show the results for additional models.
We defined the basis vectors of the cones to be orthogonal, which often considered an indicator causal indepen- dence. The intuition that two vectors are orthogonal, they each influence third vector without interfering with the other. Mathematically, for the directions and repre- sentation a we have:
if r 0 then y (xx vv x (1)
TTa
However, despite this mathematical property, recent work by Park et al. (2024) suggests that in language models, conclu- sions about causal independence cannot be drawn using or- thogonality measured with the Fuclidean scalar product. Al-

---

The Geometry of Refusal in Large Language Models
(2)
(b)
1.0
Gemma 2B
0,5
RDO
DIM
Llama 3 8B
Chsine simlarity
RDO with
0.8
Qwen 2.5 3B
RDO abl.
Attack Success Rate
0.3
Qwen 2.5 7B
0.2
0.6
0.1
0.4
0..0
(
0.2
Replnd
DIM
Cosine simiarity
0.4
Replnd with DIM abl.
DIM with Replnd abl
0.0
DIM
Replad
Roplad 2
Replad 3
Replnd 4
Replnd 5
No intervention
0.3
0.2
0.1
Directional Ablation Vector
Figure 7. Attack success rate for jailbreaking the model with direc-
Layer
Layer
tional ablation of representationally independent refusal directions for different models on JAILBREAK BrBraKKBnhh. BENCH. Each direction
Figure 6. Influence of representational independence. Figure (a) shows the cosine similarity between RDO1, refusal direction orthogonal to DIM, and the model activations in a normal forward pass (solid line) compared
representationally independent to previous directions and the DIM direction.
instructions:
forward pass where DIM is ablated (striped line). Figure (b) shows the reverse scenario. In Figure (c) and (d) we contrast how the DIM direction and representationally independent direction (RepInd) influence each other.
1
LRepInd
(cos(zol1) -no)- abl(w)> r))
[
IEL
+( (coo(xx") v) cos (1) abi(r) w))
Motivated by this observation, we introduce stricter notion of independence: Representational Independence (RepInd):
Do independent directions exist? With this extension, can find direction that is RepInd from the DIM direction, yet still fulfills the refusal properties from Definition 4.1. We illustrate the representational independence for Gemma 2B in the second row of Figure 6, where we see that the RepInd and DIM direction barely affect each other's representation under directional ablation.
Definition 6.1. The directions X, Rd are representation- ally independent (under directional ablation) with respect to the activations æ of a model a set of layers E T.
VI E L cos(x: (x( = cos( (1) (t)
abl(m)) X)
and cos(a:!)) (0,)( cos( (1)
cos( p))
abl(A)=
We iteratively search for additional directions that not only RepInd to DIM but also of all previously identified RepInd directions. Despite these strong constraints, we suc- cessfully identify multiple such directions that maintain an ASR significantly above random vector intervention (Fig- ure 7). The ASR declines as you search for more directions, which could be attributed to the increased difficulty of the optimization problem due additional constraints, that the models contain limited number of directions that inde- pendently contribute to refusal. Nevertheless, these results show that refusal in L.MSs is mediated by multiple inde- pendent mechanisms, underpinning the idea that refusal behavior is more nuanced than previously assumed.
where (1) (b)(A)
(FO) (2 abl()) ((-11))))
abi()) denotes
the activations at layer produced from the previous layer's already ablated activations with the ablation applied again after the residual addition.
Instead of relying solely on geometric orthogonality, we say that two directions are representationally independent when ablating one of them does not change how strongly the other is expressed in the model's activations. Because we track cosine similarity at every layer, any non-linear distortions introduced earlier in the network are captured downstream. Consequently, representational independence guarantees measured by cosine similarity- no lin- ear, non-linear, cumulative interaction in the network increases decreases how much the other examined direc- tion is represented.
Do the directions manipulate different mechanisms? Representational independence should have causal signifi- cance in language models, such that ablating different repre- sentationally independent directions corresponds to manipu- lating independent mechanisms. For this, we demonstrate that simultaneously ablating multiple representationally in- dependent directions yields better performance. Figure
To enforce this property, we extend Algorithm with an additional loss term that penalizes changes in cosine sim- ilarity at the last token position when ablating on harmful
8

---

The Geometry of Refusal in Large Language Models
1.0
Key Takeaways. We demonstrate the ability to identify independent refusal directions, revealing that these directions correspond to distinct underly- ing concepts and can be directly accessed through input manipulations. This further underscores the utility of our representational independence frame- work, which provides a generalizable approach for analyzing and understanding a wide range of repre- sentational interventions LLMs.
0,500
6816
088
0.642
Mtack Suczess
0.6
0.423
01.
0.2
DIM Diroction RepInd Directions
0.
3 Number of Reptnd Directiens Ablaned (Top-k)
Figure& 8. Compositionally ablating the top-k RepInd directions compared to ablating the DIM direction
7. Limitations
shows that when ablating the top-k k RepInd directions for Gemma 2 2B, the attack success rate increases monoton- ically with k. even surpassing the DIM baseline for k24, though with diminishing returns beyond this point. In con- trast, we found that ablating multiple DIM directions ex- tracted from different layers does not improve performance. The fact that ablating multiple RepInd directions produces additive improvements in ASR provides evidence that they capture different aspects of refusal rather than different man- ifestations of a single mechanism.
While our work provides new insights into the geometry of refusal in LLMs, some limitations remain. The refusal directions we compute are all optimized on the same targets, which may limit their ability to capture fully distinct mecha- nisms. Extending our method to incorporate diverse targets leveraging reinforcement learning with judge-basee based reward function could help identify additional independent mechanisms (Geisler et al., 2025). Furthermore, while we establish the existence of higher-dimensional refusal cones, we cannot rule out the possibility of other yet-undiscovered regions in the model that mediate refusal.
6.2. Manipulation from input
8. Conclusion
Can we access these directions from the input? Having found several independent directions that are distinct from DIM, we investigate whether these directions can ever be "used" by the model, by checking if they are accessible from the input if they live in regions that combination of input tokens activates. To this end, we use GCG (Zou al., 2023b) to train adversarial suffixes, which are extensions to the prompts that aim to circumvent the safety alignment. In addition to the standard cross-entrope loss on an affirmative target, we add a loss term that incentivizes the suffix to ablate RepInd-1.
This work advances the understanding of refusal mecha- nisms in LLMs by introducing gradient-based representation engineering as a powerful tool for identifying and analyz- ing refusal directions. Our method yields more effective refusal directions with fewer side effects, demonstrating its viability for extracting meaningful structures while allowing for greater modeling flexibility. We establish that refusal behaviors can be better understood via high-dimensional polyhedral cones in activation space rather than single lin- ear direction, highlighting their complex spatial structures. Additionally, we introduce representational independence and show that within this space of independent directions multiple refusal directions exist and correspond to distinct mechanisms. Our gradient-based representation engineer- ing approach can be extended to identify various concepts beyond refusal simply by changing the optimization targets. The generated findings provide new insights into the ge- ometry of aligned LLMs, highlighting the importance of structured, gradient-based approaches in LLM interpretabil- ity and safety.
In Figure 9, show the cosine similarities between Replnd-1 and the model activations on both harmful prompts Puam from JAILBREAK- BENCH and the same prompts with adversarial suffixes Pady- We observe that GCG is able to cre- ate suffixes that signifi- cantly reduce how much RepInd-1 is represented. These suffixes success- fully jailbreak the model 36% of the time, which is similar the ASR of RepInd-1.
Replnd on Phorrn
RepInd on Pads
Cosine similarity
0.2
0.1
0.0
0
5
Layer
Figure 9. Representation of the RepInd-1 direction in model ac- tivations on harmful instructions before and after adversarial at- tacks with GCG.
Acknowledgements
This project was conducted in collaboration with and sup- ported by funding from Google Research. We thank Do- minik Fuchsgruber and Leo Schwinn for feedback on an early version of the manuscript.

---

The Geometry of Refusal in Large Language Models
Impact Statement
Chen, Z., Zhu, J., and Chen, A. Eliciting Offesnive Re- sponses from Large Language Models: A Genetic Algo- rithm. Springer, 2024.
Understanding how refusal mechanisms in language models work could potentially aid adversaries in developing more effective attacks. However, our research aims to deepen the understanding of refusal mechanisms to help the commu- nity develop more robust and reliable safety systems. By focusing on open-source models requiring white-box access, our findings are primarily applicable to improving defensive capabilities rather than compromising deployed systems. We believe the positive impact of advancing model align- ment and safety through better theoretical understanding outweighs the potential risks, making this research valuable to share with the research community.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint / 083555557 2018.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
Cui, J., Chiang, W.-L., Stoica, I., and Hsieh, C.-J. Or- bench: An over-refusal benchmark for large language models, 2025. URL https: /ttp/aawvvvv/..////a/////eewwav. org horggaal /bs/ 2405.20947
References
An, B., Zhu, S., Zhang, R., Panaitescu-Liess, M.-A., Xu, Y., and Huang. F. Automatic meeuo-harmful prompt genera- tion for evaluating false refusals in large language mod- els, 2024. URL https: http//aww........a.a//// /209.. 00598.
Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse Autoencoders Find Highly Interpretable Features in Language Models, Octo- ber 2023. URL http:/ www.i.orrg/aaaaaaaaa...waw.wwota absg/bbb//30 08600. arXiv:2309.08600 [cs].
Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., and Nanda, N. Refusal in language models is mediated by single direction, 2024. URL https: http://axvvvvv.//e org/ abs/ 22..2206..111 11717.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Belrose, N., Schneider-Jseeph. D., Ravfogel, S., Cotterell, R., Raff, E., and Biderman, S. Leace: Perfect linear concept crasure in closed form, 2023. URL https: //arxiv. wrgg/bbss.....ooooo.......eee..b awaaabss23300 230. 93819.
Fiotto-Kaufman, J., Loftus, A. R., Todd, E., Brinkmann, J., Juang, C., Pal, K., Rager, C., Mueller, A., Marks, S., Sharma, A. S., et al. Nnsight and ndif: Democratizing access to foundation model internals. arXiv preprint arXiv:2407.0 14561, 2024.
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., and Kalai, A. Man to computer programmer as woman is to homemaker? debiasing word embeddings, 2016. URL https: //wpvva.a/vvv//aa//////ea..a.av org/ abs/ abss/607... 06520.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi,
A., Foster, C., Golding, L.., Hsu, J., Le Noac'h, h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang,
J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika,
Burns, C., Ye, H., Klein, D., and Steinhardt, J. Discov- ering latent knowledge in language models without su- pervision, 2024. URL hwtpt:tt///// htttps://arxiv. org/ abs/ 2212.03827.
L., Tang, E., Thite, A., Wang, B., Wang. K., and Zou, A. A framework for few-shot language model evaluation,
07 2024. URL https: zenodo. orgo regords/ 12608602.
Carlini, N., Nasr, M., Choquetre-Choo. C. A., Jagielski, M., Gao, I., Awadalla, A., Koh, P. W., Ippolito, D., Lee, K., Tramer, F,, and Schmidt, L. Are aligned neural networks adversarially aligned?, 2024. URL https: //rxiv. org/abs 2306. 15477
Geisler, S., Wollschläger, T., Abdalla, M. H. I., Gasteiger, J., and Günnemann, S. Attacking Large Lan- guage Models with Projected Gradient Descent, Febru- ary 2024. URL http: //axivivvvvvv. www.vvoorrgooo .ogg org//bs//2422 09154. arXiv:2402.09154 [es].
Geisler, S., Wollschläger, T., Abdalla, M. H. I., Gasteiger, J., and Günnemann, S. Reinforce adversarial attacks on large language models: An adaptive, distributional, and semantic objective, February 2025.
Chao, P., Debenedetti, E., Robey, A., Andriushchenko, M.,
Croce, F., Sehwag, V., Dobriban, E., Flammarion, N., Pappas, G. J., Tramèr, F., Hassani, H., and Wong, E. Jailrrekkeenc: An open robustness benchmark for jail- breaking large language models. In NeurIPS Datasets and Benchmarks Track, 2024.
Gurnee, W. and Tegmark, M. Language models represent space and time, arXiv preprint arXiv:2310.02207 2023.
10

---

The Geometry of Refusal in Large Language Models
Heinzerling, B. and Inui, K. Monotonic representation of numeric properties in language models. arXiv preprint arXiv:2403. 10381, 2024.
O'Brien, K., Majercak, D., Fernandes, X., Edgar, R., Chen, J., Nori, H., Carignan, D., Horvitz, E., and Poursabzi- Sangde, F. Steering language model refusal with sparse autoencoders, 2024. URL https: http::://axxi..rrp
Hendryeks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
OpenAl. Introducing chatgpt, November 2022. URL https://openaal cww/blog//cccccccccccc chatgpt/. Ac- cessed: 2025-01-26.
Huang, D.-S., Si, Z., and Pan, Y. (eds.). Advanced Intelligent Computing Technology and Applications: 20th Interna- fional Conference, ICIC 2024, Part III, volume 14864 of Lecture Notes in Computer Science. Springer Nature Singapore, Singapore, 2024. ISBN 978-988-97-5587-5 978-981-97-5588-2 doi: 10. 10.1007/978-981--97-555 URL httlt:/ts://// 978 981 51-85588-22 2.
Pan, W., Liu, Z., Chen, Q. Zhou, X., Yu, H., and Jia, X. The hidden dimensions of 1lm alignment: A multi-dimensional analysis of orthogonal safety di- rections, 2025. URL 2502.09674.
Panickssery, N., Gabrieli, N.. Schulz, J.. Tong, M., Hubinger, E., and Turner, A. M. Steering Llama 2 via Contrastive Activation Addition, July 2024. URL ht / /arxiv. org/ abs 2312 06681. arXiv:2312.0668 [cs].
Li, H. and Liu, X. Injecguard: Benchmarking and mitigating over-defense in prompt injection guardrail models, 2025.
URL https: aaxiv.iiiiv.///.i///aaaaa/aaaavv org/ abs / 2410... 22770.
Li, L., Dong, B., Wang, R., Hu, X., Zuo, W., Lin, D., Qiao, Y., and Shao, J. Salad-bench: A hierarchical and com- prehensive safety benchmark for large language models. arXiv preprint 2rXiv:2402.05044 2024a.
Park, K., Choe, Y. and Veitch, V. The linear represen- tation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03653 2023.
Park, K., Choe, Y. J., and Veitch, V. The Linear Represen- tation Hypothesis and the Geometry of Large Language Models, July 2024. URL http: / / arxiv. wwwiv..or//aaa. 2311. 03658. arXiv:2311.03655 [cs].
Li, T., Wang. Z., Liu, W., Wu, M., Dou, S., Lv, C., Wang, X., Zheng, X., and Huang, X. Revisiting jailbreaking for large language models: A representation engineering per- spective, 2024b. URL /tattps://axiii org/ abs/ 2401. 06824.
Rao, A., Vashistha, S., Naik, A., Aditya, S., and Choudhury, M. Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks, March 2024. URL http:/ /aaaraivv.orrr.e/.aa abs/ 2305 14965. arXiv:2305. 14965 [cs].
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv: 10Xi:2/09,00958, 2021.
Lin, Z., Wang. Z., Tong, Y., Wang, Y., Guo, Y., Wang, Y., and Shang, J. Toxicchat: Unveiling hidden challenges of toxicity detection real- world user-ai conversation. arXiv preprint ar arXiv:2310. 2023.
Röttger, P., Kirk, H. R., Vidgen, B., Attanasio, G., Bianchi, F. and Hovy, D. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308. 01263, 2023.
Liu, Y., Yao, Y., Ton, J.-F., Zhang, X., Cheng, R. G. H., Klochkov, Y., Taufiq, M. F., and Li, H. Trustworthy 1lms: A survey and guideline for evaluating large language models' alignment. arXiv preprint arXiv:2308.0552 2023.
Scholten, Y., Günnemann, S., and Schwinn, L. A proba- bilistic perspective on unlearning and alignment for large language models. In The Thirteenth International Con- ference on Learning Representations, 2025.
Marks, S. and Tegmark, M. The geometry of truth: Emer- gent linear structure large language model represen- tations of true/false datasets, 2024. URL https:// arxiv. org/ abs
Schwinn, L., Dobre, D., Xhonneux, S., Gidel, G., and Günnemann, S. Soft prompt threats: Attacking safety alignment and unlearning in open-source LLMs through the embedding space. The Thirty-eighth Annual Con- ference on Neural Information Processing Systems, 2024.
Nanda, N., Olah, C., Olsson, C., Elhage, N., and Hume, T. Attribution patching: Activation patching at industrial scale. https:// nww.nnneenadd.. 10 to/mmechanaaiic--nneeeetttta www.riibuuiin--patchhhhe 2024. 2025-01-10.
Schwinn, L., Scholten, Y., Wollschläger, T., Xhonneux, S., Casper, S., Günnemann, S., and Gidel, G. Ad- versarial alignment for lims requires simpler, repro- ducible, and more measurable objectives. arXiv preprint arXiv:2502.1 /910, 2025.
Accessed:

---

The Geometry of Refusal in Large Language Models
Shah, R., Feuillade-Montixi Q. Pour, S., Tagade, A., Casper, S., and Rando, Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation, November 2023. URL http:///// arxiv. org/ 2311 03348. Ttt::2311.03344. [es].
Xie, T., Qi, X., Zeng, Y., Huang, Y., Sehwag, U. M., Huang. K., He, L.., Wei, B., Li, D., Sheng, Y., Jia, R., Li, B., Li, K., Chen, D., Henderson, P., and Mittal, P. Sorry- bench: Systematically evaluating large language model safety refusal, 2025. URI. https: aaw.iv.ov.oo////. abs/ 145988.
Shi, C., Wang, X.. Ge, Q., Gao, S., Yang. X., Gui, T., Zhang, Q. Huang, X., Zhao, X., and Lin, D. Navigating the overkill in large language models, 2024. URL ht htsps: //arxiv. org/ abs/ / /401.40.1..0 17633.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:24/2.15115, 2024.
Souly, A., Lu, Q., Bowen, D., Trinh, T., Hsieh, E., Pandey, S., Abbeel, P., Svegliato, J., Emmons, S., Watkins, O., and Toyer, S. A strongreject for empty jailbreaks, 2024.
Yu, L., Do, V., Hambardzumyan, K., and Cancedda, N. Robust Ilm safeguarding via refusal feature adversarial training. arXiv preprint arXiv:2409. 20089, 2024.
Stolfo, A., Balachandran, V., Yousefi, S., Horvitz, E., and Nushi, B. Improving instruction-follloingg in language models through activation steering, 2024. URL ht https: //arxiv. org/ abs/ 2410. 12877.
Zheng, C., Yin, F., Zhou, H., Meng. F., Zhou, J.. Chang, K.-w,, Huang, M., and Peng, N. On Prompt-Driven Safeguarding for Large Language Models, June 2024. URL http: /arxiv. org/ abs/2401 18018. arXiv:2401.180) [cs].
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D.. Goodfellow, I., and Fergus, R. Intriguing properties of neural networks, 2014. URL https:/ www.v.ioorg........eeooooooooooe abs/1312 00332..00 6199.
Zhu, S., Zhang, R., An, B., Wu, G., Barrow, J., Wang. Z., Huang, F., Nenkova, A., and Sun, T. AutoDAN: Inter- pretable Gradient-Based Adversarial Attacks on Large Language Models, December 2023. URL http:// arxiv. arxiv.org/asee wttp://gg//.a/oeee 15140. arXiv:2310.15140 [cs].
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X. Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-followigg Ilama model. https:/ /ttp//ggtttttttt www/tatsu-laallaee/acccc stanford_alpaca 2023.
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin. X., Mazeika, M., Dombrowski, A.-
K., Goel, S., Li, N., Byun, M. J., Wang, Z., Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, 7., and Hendrycks, Representation Engineer- ing: Top-Down Approach to AI Transparency, Octo- ber 2023a. URL http: arxiv. htrggabb/2310. 01405. arXiv:2310.01400 [es].
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma Improving open language models at practical size. arXiv preprint arXiv:2408.00116 2024.
Wang, K. Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the Wild: Circuit for Indirect Object Identification in GPT-2 small, Novem- ber 2022. URL http: arxiv. org/ abs abw/aba/222 2211. 00593. ar arXiv:2211 1112221000059 [es].
Zou, A., Wang, Z., Kolter, Z., and Fredrikson, M. Uni- versal and Transferable Adversarial Attacks on Aligned Language Models, July 2023b. URL http: / /arxiv. org/ abs 2307 15043. AXXi:23307.1044 [cs].
Zou, A., Phan, L., Wang. J., Duenas, D., Lin, M., An- driushchenko, M., Kolter, Z., Fredrikson, M., and Hendrycks, D. Improving alignment and robustness with circuit breakers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Wang, W., Tu, Z., Chen, C., Yuan, Y., Huang, J.-t., Jiao, W., and Lyu, M. R. All languages matter: On the multi- lingual safety of large language models. arXiv preprint S.I1:23100000000 2023.
Wei, B., Huang, K., Huang, Y. Xie, T., Qi, X., Xia, M., Mittal, P., Wang, M., and Henderson, P. Assessing the briutleness of safety alignment via pruning and low-rank modifications, 2024. URL https::t///........ ttttts://arxiv.coo /ttp/aaxxxvvrr///// 0t55/402.05111
Xhonneux, S., Sordoni, A., Günnemann, S., Gidel, G., and
Schwinn, L. Efficient adversarial training in LLMs with continuous attacks. In The Thirty-eighth Annual Confer- ence on Neural Information Processing Systems, 2024.
12

---

The Geometry of Refusal in Large Language Models
A. Setup Details
A.1. Datasets
We construct our experimental dataset using harmful and harmless instructions from established benchmarks. For harmful instructions, we draw from SALADBENCH (Li et al., 2024a), comprehensive collection of adversarial prompts from diverse sources. We exclude the Multilingual (Wang et al., 2023) and ToxicChat (Lin al., 2023) sources since they unsuited harmful instructions. Afterwards, we sample up 256 instructions from each remaining source. This results in 1,184 instructions for training and 128 for validation. We sample equal numbers of harmless instructions from the ALPACA dataset, and additionally reserve 128 more harmless instructions for testing.
A.2. Models
We exclusively chat models for our experiments, but omit "IT" and "INSTRUCT" from model names. We use each chat model's default chat template throughout our analysis.
Table 2. Model families, sizes, references.
Model family
Sizes
Reference
QWEN2.5 INSTRUCT
L.5B, 3B, 7B, 14B
Yang et al. (2024)
GEMMA IT
2B, 9B
Team al. (2024)
LLAMA-3 INSTRUCT
8B
Dubey et al. (2024)
A.3. Hyperparameters and Implementation
Table 3. Hyperparameters for all algorithms
Component
Parameter
Value
Training
Total Batch Size
16
Gradient Accumulation Steps
16
Base Learning Rate
0.01
Learning Rate Reduction
Every batches if plateaued
Learning Rate Factor
Divide by 1/10 up to times
Optimizer Weight Decay
AdamW
0
Main Loss
Ablation Loss Weight Aabl
1.0
Addition Loss Weight Asdd
0.2
Retain Loss Weight Ave
1.0
Monte Carlo Sampling
Samples per Accumulation Step
16
Effective Samples per Batch
256
RepInd
RepInd Loss Weight Aind Layer Cutoff
200
0.9
Table 3 presents the hyperparameters used in our algorithms. Since our method converges before completing full epoch, we not utilize validation scores during training. Instead, after convergence, we apply the direction selection algorithm from Arditi et al. (2024) identify the optimal refusal direction from the last 20 training steps.
Implementation and Evaluation Framework. All algorithms and exploratory experiments implemented using the NNsight (Fiotto-aafmaal et al., 2024) library. Additionally, we use the LM Evaluation Harness (Gao et al., 2024) to run TruthfulQA (Lin et al., 2021) with default settings, with the exception that we enable the use of each model's default chat templates.
Retain and Representational Independence Loss Computation. The retain loss is computed as the KL divergence between the probability distributions derived from the logits of the model with and without directional ablation, masked
13

---

The Geometry of Refusal in Large Language Models
target response and the last token of the chat template. The resulting value is then averaged across tokens. For single instruction Psate with its target retain we formalize the loss as follows:
{Psato + Egetain)
Cretain (L(faaatte(r)) (Psate) (psate) fretain)
Σ VEI RET
[(Psae + (retain) log
7
Jablate (Psate + Arretain
where I contains the target token indexes and the last instruction token's index, the subscript i, denotes the model output at sequence position i and vocabulary index / as defined in Section 2, and + denotes concatenation.
For the implementation of the representational independence loss, Reppndd we compute the average loss over the tokens in the harmful instructions Pharm- The RepInd loss is computed over the first 90% of layers, as applying it too close to the unembedding layer overly constrains the model's 's output.
Selection of Refusal and Independent Directions In Algorithm I, after training the refusal directions convergence, we again use the direction selection algorithm from Arditi et al. (2024) to identify the most effective directions from the final 20 training steps.
In Section 5, we extend this selection process to determine a basis where all basis vectors effectively mediate refusal (from the last 20 bases of the training). If no such basis exists, we instead select the basis where the samples are most effective for directional ablation using the refusal score heuristic from the selection algorithm.
Training Procedure for Representational Independence Directions In Section 6, our approach training and validating representationally independent (RepInd) directions differs because of high variance between different runs. For each RepInd direction, we train five candidate vectors and select the one with the lowest refusal score on our validation set. This process is repeated five times, ultimately producing our final set of RepInd directions. The Repind loss is computed as the sum of losses over all vectors that the current vector should remain independent of.
14

---

The Geometry of Refusal in Large Language Models
B. Extended Results for Refusal Direction Optimization
In this section, we present additional results, including results for activation addition, more datasets for directional ablation, and more benchmarks.
B.1. Activation Addition
We first confirm that our directions can also be used to induce refusal. Figure 10 demonstrates that using RDO refusal directions for activation addition successfully induces refusal behavior across all models for both DIM and RDO, and RDO slightly outperforms DIM for most models.
DIM
RDO (ours)
0.8
Refusal Score
0.6
0.4
0.2
Gemma 2 2B
Gemma 2 9B
Qwen 2.5 1.5B
Qwen 2.5 3B
Qwen 2.5 7B
Qwen 2.5 14B
Llama 8BB
Figure 10. Refusal scores of different models on harmless instructions after activation addition that aims to induce refusal.
B.2. Directional Ablation on Additional Datasets
For a more robust evaluation of how the RDO directions compare to DIM in terms of performance, we additionally evaluate directional ablation ASR on STRONGREJECTT(Suu) We include baseline performance here without any intervention. The results are shown Figure 11, Figure 12, and Figure 13, and confirm that findings transfer across datasets.
al., 2024) and the CORRY-BENCEE base dataset (Xie et al., 2025).
StrongRJJECT
Method
Baseline
Attack Success Rate
(No Intervention)
DIM
0.6
(Ours)
Operation
0.4
Directional
Ablation
0.2
Activation
Subtraction
Gemma 2B
Gemma 98
Llama 8B
Qwen 2.5 1.5B
Qwen 2.5 3B
Qwen 2.5 7B
Qwen 2.5 14B
Figure 11. Attack success rates refusal directions on STRONGREJECT.
15

---

The Geometry of Refusal in Large Language Models
SORRY-Bench
Method
Baseline
Attack Success Rate
(No Intervention)
DIM
0.6
(Ours)
Operation
0.4
Directional
Ablation
0.2
Activation
Subtraction
0.0
Gemma 2 2B
Gemma 298
Llama 3 8B
Qwen 2.5 1.5B
Qwen 2.5 3B
Qwen 2.5 7B
Qwen 2.5 14B
Figure Attack success rates refusal directions on
JailbrakBeeneh
Method
Baseline
Attack Success Rate
0.8
(No Intervention)
DIM
0.6
RDO (Ours)
Operation
0.4
Directional
Ablation
Activation
Subtraction
Gemma 2 28
Gemma B9
Llama 8B
Owen 2.5 1.5B
Qwen 2,5 3B
Owen 2.5 7B
Qwen 2.5 14B
Figure 13, Attack success rates of refusal directions on BEE.H
16

---

The Geometry of Refusal in Large Language Models
B.3. Benchmarks
In Table I showed that RDO directions have significantly lower side-effects on model performance as measured via the reduction in TRUTHFULQA score. Here, show the change in benchmark scores for more benchmarks, specifically ARC (Clark et al., 2018), GSM8K (Cobbe et al., 2021), and MMLU (Hendrycks et al., 2020). In Table 4, that for most combinations of model and benchmark, the RDO direction has less impact on benchmark score, which confirms the effectiveness of our retain loss and that the RDO directions manipulate refusal more precisely. Finally, we summarize all results in Table 5.
Table 4. Sides effect measured by comparing model performance on benchmark datasets when ablating with either DIM or RDO. Ablating RDO has significantly lower side effects for most models and benchmarks.
Dataset
Model
DIM
RDO
Change
Gemma 2
8.0%
4.4%
(-3.6%)
Gemma 2
8.3%
4.4%
(-3.9%)
Llama 3 8B
4.1%
1.8%
(-2.3%)
TruthfulQAMMC.
Qwen 2.5 1.5B
3.6%
2.6%
(-1.1%)
Qwen 2.5 7B
4.4%
3.1%
(-1.2%)
Qwen 2.5 14B
7.5%
2.5%
(-5.0%)
Gemma 2 2B
0.2%
0.2%
(-0.0%)
Gemma 2 9B
0.6%
0.3%
(-0.3%)
Llama 3 8B
1.0%
0.2%
(-0.9%)
ARC.CHALLENGE
Qwen 2.5 1.5B
0.7%
0.3%
(-0.4%)
Qwen 2.5 7B
0.9%
0.0%
(-0.9%)
Qwen 2.5 14B
1.5%
0.6%
(-0.9%)
Gemma 2 2B
0.6%
0.4%
(-0.2%)
Gemma 2 9B
0.1%
0.3%
(+0.2%)
Llama 3 8B
1.0%
2.7%
(+1.7%%
GSM8K
Qwen 2.5 1.5B
1.1%
0.0%
(-1.1%)
Qwen 2.5 7B
0.8%
0.7%
(-0.2%)
Qwen 2.5 14B
1.5%
0.5%
(-1.1%)
Gemma 2 2B
0.3%
1.2%
(+1.0%)
Gemma 2 9B
1.3%
0.1%
(-1.2%)
Llama 3 8B
2.6%
0.1%
(-2.5%)
MMLU
Qwen 2.5 1.5B
1.5%
0.5%
(-1.0%)
Qwen 2.5 7B
0.7%
0.1%
(-0.6%)
Qwen 2.5 14B
1.2%
0.3%
(-0.9%
17

---

The Geometry of Refusal in Large Language Models
Table Comparing DIM terms jailbreaking effectiveness and how jailbreaking affects general capability benchmarks. Each cell in the jailbreaking section contain pairs of ASRs: the first for directional ablation and the second for activation subtraction. The values the general capability section are computed under directional ablation the respective directions.
Jailbreaking
General Capability
JailbreakBench ASR f
RtrCngRFJJCCT ASR 1
SORRY-Bench ASR 1
MMLU
ARC-C
GSM8K
TruthfulQA
Acc ?
Acc
Acc
Acc f
GEMMA 2 2B DIM
2.5 71.2
1.0 80.7 / 74.6
10.4
30.5
42.7
52.3
55.8
71.0 / 66.3
30.3
42.5
52.9
47.8
RDO
79.9/ 69.5
80.5 / 71.4
69.5 63.3
29.3
42.8
52.7
51.4
GEMMA 2 9B DIM
0.3
0,6 80.2 / 82.7
7.8
34.8
52.6
74.7
61.1
79.1 68.3
69.1 69.0
36.1
53.2
74.6
52.8
RDO
76.8 73.0
77.7 75.4
68.1 / 66.3
34.7
52.2
75.0
56.7
LLAMA 3 8B DIM
2.9
1.2 81,5
14.0
58.1
48.6
63.5
52.8
79.7 74.1
/ 74.4 / 73.22
55.5
47.6
62.5
48.7
RDO
80.3 79.0
83.8 / 84.7
58.2
48.5
66.2
51.0
QWEN 2.5 1.5B DIM
3.1 65.2 / 60.4
9.2
58.3
38.5
56.4
46.5
$3.1/55. 55.0
52.5 / 53.0
56.8
37.8
55.3
42.9
RDO
53.5 65.2
61.3 / 71.3
49.1 / 60.1
57.8
38.7
56.4
44.0
QWEN 2.5 3B DIM
8.4
6.9 72.2 / 64.1
16.7
64.6
42,8
61.2
57.2
68.5 54.9
64.8 / 55.6
64.5
42.7
60.3
54.2
RDO
67.6 54.8
73.6 / 55.9
63.6 55.0
64.7
41.6
59.5
54.5
QWEN 2.5 7B DIM
9.1 /9.2/71.0
7.1 68.8 / 74.1
22.7
68.8
45.6
77.9
63.1
63.1 / 64.8
68.1
46.5
77.0
58.7
RDO
69.3 72.0
70.0 / 74.8
63.4
68.7
45.6
77.2
60.0
QWEN 2.5 14B
4.9
2.9
17.9
76.9
52.8
81.7
70.8
DIM
76.2 66.6
/ 70.7
/ 60.0
75.7
51.4
80.2
63.4
RDO
75.7 /
79.9 76.2
69.0 / 66.2
76.6
52.2
81.3
67.9
18

---

The Geometry of Refusal in Large Language Models
C. Ablation Studies
We conduct ablation studies to determine the importance of the three losses our RDO algorithm.
C.1. Addition and Ablation Loss
We first study how the addition and ablation loss should be balanced. We experiment with the Llama-3-8B-Instruut model and range the loss weights Abh and Add from to I, setting >abl = Addi to balance the weights. We then evaluate attack success rates for both directional ablation and activation subtraction interventions. Figure 14 shows that both loss components are essential for optimal performance. ASR is similar across the 0.2-0.8 weight range, where both methods maintain consistently high attack success rates above 80%, and choosing only of the losses reduces performance significantly. This finding indicates that while including both loss terms is critical, the precise weight allocation within this range has minimal impact on effectiveness. This robustness simplifies hyperparameter tuning in practice, as practitioners can select any weight configuration within this range without substantially affecting performance.
Effect of Addition Loss Weight Model Performance
0.8
0.7
0.6
Attack Success Rate
0.5
0.4
0.3
0.2
Directional Ablation
Activation Subtraction
0.0
0.2
0.4 Addition Loss Weight (1 Ablation Loss Weight)
0.6
0.8
1.0
(Only Ablation Loss)
(Only Addition Loss)
Figure 14. Ablation study of loss weights Amn and Auda for Llama-3-8B-Instruue We compare attack success rates for directional ablation and activation subtraction across different weight balances (Aan 1 - Awa).. Both methods perform well in the 0,2-0.8 range, with severe degradation extremes, particularly for directional ablation using only the addition loss (20% ASR Auid 1).
C.2. Retain Loss
Regarding the impact of using the retain loss to minimize side-effects when intervening with our refusal directions, we conduct an ablation study of the retain loss weight Aee for Qwen2.5-3B-Instiee weights to their default values (see Table 3) and systematically vary Aret- Figure 15 presents a Pareto analysis plotting ASR under directional ablation against the average reduction in benchmark performance that results from directional ablation. In more detail, the x-axis represents the average change in benchmark scores when intervening with the refusal direction across GSM8K, MMLU, ARC-Challenge, and TruthfulQA relative the model's 's baseline performance (no intervention). while the y-axis shows the corresponding BAT.BREAK ASR under directional ablation. The ideal refusal direction would maximize ASR while maintaining or improving benchmark performance (c.g.. by preventing the model from inappropriately refusing legitimate questions).
We fix the ablation and addition loss
For this specific model, retain weights up to Ave = 4 increasingly reduce side-effects on benchmark performance with only marginal ASR reduction. However, beyond this threshold, the ASR drops drastically, indicating that excessive weighting of
19

---

The Geometry of Refusal in Large Language Models
0.7
0.6
Attack Success Rate
Retain Loss Weight
0.4
À 0
À = 0.125
À = 0.25
0.3
À
À
0.2
À
À
À 8
0.1
== 16
DIM
0.0
-3.0
-2.5
-2.0
-1.5
-1.0
-0.5
Difference in Avg. Benchmark Score Compared to No Intervention (%)
Figure Ablation study retain weight Aremi for Qwen2.5-3B Instruct. We plot JailbreakBench mance change (x-axis, averaged over GSM8K, MMLU, ARC-Challena Challenge, TruthfulQA) when intervening with the direction compared baseline. Higher retain weights improve benchmark preservation with minimal ASR loss up to Aretin = 4, after which ASR drops significantly. Multiple hyperparameter choices Pareto-dominate the DIM baseline, demonstrating robust improvements over current state-of-the-art.
versus benchmark perfor-
the retain loss impedes the learning of effective refusal directions.
Importantly, observe that multiple hyperparameter configurations achieve Pareto dominance over the DIM baseline across both metrics. This demonstrates that our method provides robust improvements over the current state-of-the-art, rather than gains limited to specific hyperparameter choices.
20

---

The Geometry of Refusal in Large Language Models
D. Flexibility of Algorithms
We find that DIM can struggle to identify refusal directions with sufficiently low side-effects (according to the heuristic used by Arditi al. (2024). Figure 16 visualizes the effectiveness of the direction selection algorithm from Arditi et al. (2024) for DIM directions in the Qwen 2.5 7B model. Among the evaluated token and layer pairs, only one direction is found be effective for both inducing refusal through activation addition and maintaining low side effects. Transparent data points indicate (layer, token) combinations that were filtered out due to their inability to induce refusal reliably. Additionally, the red line represents the KL-divergence threshold, used to estimate potential side effects of directional ablation on harmless instructions.
10
Source Position
Position -5
Position 4
Position -3
KL Divergence Score
10"
Position -2
Position -1
10-1
KL
Filter Threshold (0.1)
10-2
Source Layer
Figure 16. Analysis of the selection direction algorithm from Arditi et al. (2024) for the DIM directions of Qwen 2.5 7B. Among the token and layer combinations, only a single direction is identified as viable for both inducing refusal via activation addition and having low side-effects. Transparent points represent (layer, token) pairs that filtered out because of ineffectiveness in inducing refusal. The red line indicates the KL-divergence threshold used to estimate potential side-effects of directional ablation on harmless instructions.
21

---

The Geometry of Refusal in Large Language Models
E. Extended Results for Refusal Cones
Figure 17 we show the refusal cones for the Gemma model family.
1.0
0.8
Attack Success Rate
0.6
Gemma
Gemma 2 9B
Basis Voctors
0.4
0.2
0.0
2
5
Cone Dimension
Figure 17. Attack rates in refusal of different dimensions for the Gemma 2 model family. We ohserve that for the Gemma 2 2B the lower bounds start to degrade significantly for dimension 5.
In Figure 18 measure the performance of the refusal cones in the Qwen 2.5 model family for activation addition. The models tend to support higher dimensional cones compared to Figure 4, revealing that inducing refusal is significantly easier than disabling refusal via directional ablation. Notably, most directions even in high-dimensional cones remain effective at inducing refusal responses.
1.0
Refusal Score
Qwen 2.5 1,5B
0.6
Qwen 2.5 3B
Qwen 2.5 7B
Qwen
Basis Vectors
0.0
Cone Dimension
Figure 18. Using refusal cones to induce refusal across various Qwen 2.5 models with different dimensions. We observe that inducing refusal is generally casier than executing attack. In this setting, nearly all dimensions maintain strong performance in eliciting refusal responses.
Figure 19 examines the attack success rate when sampling multiple vectors from various N-dimensional refusal cones and selecting the best-performing sample prompt for Gemma 2, 2B. We observe that ASR improves with increasing cone dimensionality but plateaus at four dimensions, suggesting that single-direction manipulation by capturing complementary mechanisms. The plateau likely results from the model's inability to encode higher-dimensional refusal cones, hypothesis further supported by Figure 17.
cones provide an advantage over
22

---

The Geometry of Refusal in Large Language Models
0.92
0.90
Performance
0.86
Cone Dimeensio
0.84
2
3
0.82
0.80
8
16
64
128
256
Number of Samples
Figure 19. Attack success rates when sampling vectors from the N-dimensional refusal cones and selecting the hest-performing sample per prompt Gemma ASR increases with cone dimensionality but plateaus at four dimensions, suggesting higher-dimensionall cones provide advantage over single-direction manipulation by capturing complementary mechanisms. The plateau likely arises because Algorithm cannot find an additional basis vector that preserves the refusal properties in the cone, suggesting that the model docs not support cone of this dimension. Figure 17 also provides evidence for this claim.
10
Refusal Score
-5
DIM dlirection
Dimeusion samples
10
Dimension 3 samples
Dimeusion samples
Dimension samples
-15
100
50
100
Alpha a
Figure Refusal scores of refusal vectors sampled from Gemma refusal cones compared to the DIM direction when scaling the norm of the added direction a for the activation addition intervention. The refusal score is the heuristic from Arditi et al. (2024) here, and we compute on harmful validation instructions, mean and standard deviation over 64 samples per alpha.
23

---

The Geometry of Refusal in Large Language Models
F. Orthogonal Refusal Directions
In Figure we showed how orthogonal RDO; interacts with DIM direction. Here, we show the result for two additional models, on larger dataset (SORRY- BENCH). Figure 22 supports that the DIM direction is greatly influenced by ablating orthogonal refusal directions, supporting our claims in Section 6.1.
RDO bascline
DIM baseline
RDO with DIM ablated
DIM with RDO ablated
Gemma 2 2B
0.4
Cosine similarity
0.3
0
5
10
15
20
25
0
5
10
15
20
25
Layer
Layer
Qwen2.5 7B
Cosine similarity
0.3
0.2
0.1
0.0
0
10
15
20
0
10
15
20
25
Layer
Layer
Llama SB
0.3
Cosine similarity
0.2
0.1
0.0
0
15
20
30
0
15 20
30
Layer
Layer
Figure 21. Interaction of orthogonal refusal directions directions under directional ablation, measured in terms of cosine similarity to model activations the last token position on harmful instructions from SORRY BENCH, across different models.
24

---

The Geometry of Refusal in Large Language Models
G. Over-refusal
evaluate the trade-off between over-refusal and safety by analyzing activation addition of the RDO and DIM directions across different strength configurations for the Gemma 2 2B model. Figure shows results for activation addition strengths (a) ranging from 0 ||v|22, where we compute refusal scores on distinct datasets: harmful instructions from SORRY- BENCH and harmless instructions from XSTEST (Röttger et al., 2023). Higher scores SORRY-BBENCE indicate safer responses to genuinely harmful prompts, while lower scores on XSTEST indicate reduced over-refusal on benign inputs that are designed to measure inappropriate refusal behavior. Our method consistently provides better or equivalent tradc-off's compared to DIM across all strength configurations, suggesting that RDO directions may be more suited for increasing safety at the same rate of over-refusal compared to DIM.
DIM
0.95
SORRY-Bert --Bnnch Refusal Score
0.85
0.80
0.75
0.70
0.2
0.4
0.6
0.8
1.0
XSTest Refusal Score
Figure 22. Trade-off between over-refusal and safety for different activation addition strengths of our refusal directions.
25

---

The Geometry of Refusal in Large Language Models
H. Assets
the following, we show the licenses the assets we used in this work: different models from Table 6 and the datasets that we use for evaluation and training; see Table
H.I. Models
Table 6. The list of models used in this work.
Model
Source
Accessed via
License
Qwen 2.5 {1.5B, 7B, 14B)
Yang al. (2024)
Link
Apache 2.0 License
Qwen 2.5 {3B}
Yang et al. (2024)
Link
Qwen Research License
Gemma 2B
Team et al. (2024)
Link
Apache 2.0 License
Gemma 9B
Team al. (2024)
Link
Gemma Terms of Use
Llama-3 8B
Dubey et al. (2024)
Link
Meta Llama 3 Community License
StrongREJJCCT Judge
Souly al. (2024)
Link
MIT License
H.2. Datasets
Table The datasets used this work. Source
Dataset
Accessed via
License
SALADBENCH
Li et al. (2024a)
Link
Apache License 2.0
ALPACA
Taori et al. (2023)
Link
Apache License 2.0
JAILBREAKBENCH
Chao et al. (2024)
Link
MIT License
STRONGREJECT
Souly et al. (2024)
Link
MIT License
SORRY-BENCH
Xie et al. (2025)
Link
Custom License
XSTEST
Röttger et al. (2023)
Link
CC-BY-4.0
TRUTHFULQA
Lin et al. (2021)
Link
Apache License 2.0
MMLU
Hendrycks al. (2020)
Link
MIT License
ARC
Clark et al. (2018)
Link
CC-BY-SA-4.0
GSM8K
Cobbe et al. (2021)
Link
MIT License
26


Reviews:
[{"review_id": "5YpXfc2XHN", "reviewer": "Reviewer_5cSG", "summary": "This paper investigates the mechanisms behind refusal behaviors in large language models (LLMs) and discovers that refusals are controlled by multiple- dimension in the model’s activation space. The authors introduce RDO to enhance the refusal capability of large language models (LLMs) through training a learnable refusal subtraction feature. The experiments somewhat illustrate the effectiveness of proposed methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort spent reviewing our manuscript!\n\n\n### Ablation studies & Safety and Utility Tradeoff\nWe added a detailed ablation study, investigating the three loss components. We show the results in [Figures 18 & 19](https://figshare.com/s/4ab2ec422f6bd0262b30) for Llama-3-8B-Instruct. In Fig. 18, we measure how the ASR evolves as we shift the balance between addition and ablation loss weights, requiring $\\lambda_{\\text{abl}} + \\lambda_{\\text{add}} = 1$, and observe that performance is quite robust for balanced values (0.2 to 0.8), with both losses being necessary for good generalized performance. \n\nIn Figure 19, we ablate the retain loss weight. We use our default ablation and addition weights with different retain weights to train directions, and plot the ASRs compared to the difference in average benchmark performance on MMLU, ARC, GSM8K, and TruthfulQA (here we use the smaller Qwen2.5-3B-Instruct for computational reasons). The ideal direction would have the highest ASR possible for this model, while having no effect on benchmarks or positive effect only so far as preventing the model from refusing benchmark questions. We observe that many combinations of loss weights pareto-dominate the DIM direction for this model, which shows that our approach robustly outperforms the current SOTA.\n\nFurther, this figure is in our view the best option to investigate the tradeoff between effectiveness for model manipulation and utility. A combined score will always omit important details. We propose to highlight this figure in the paper to investigate this tradeoff if the reviewer agrees. Here we can see that our approach strictly improves this tradeoff compared to DIM as we pareto-dominate it for both metrics. However, if the reviewer would still like to see a combined score, we will provide one in a new response.\n\n### Over-Refusal\nWe added over-refusal experiments to our manuscript. Due to space constraints, we kindly ask the reviewer to see the rebuttal answer to reviewer 4YKU. We further added a paragraph to the related work section giving the reader an overview of the topic based on the provided references.\n\n### (Formal) theoretical validation\nWe agree that formal validation of the independence would be great. We employ an argument that proves the true causal independence with respect to cosine similarity until the final read-out of the model. We can then use the empirical arguments of related work that directionality to a concept primarily drives the model prediction. For a more in-depth answer please refer to the answer to reviewer NvcZ.\n\n### Applications\nWe agree with the reviewer that outlining practical uses of refusal directions is valuable. We added one paragraph to the manuscript and provided a brief overview here: \n- Most importantly, we aim to emphasize that top-down interpretability not only advances our understanding of AI safety but also deepens our general comprehension of how LLMs represent concepts. Numerous studies have explored the latter, suggesting that any method that enhances this approach could be highly impactful.\n- Offense Applications: Refusal directions can inform (white-box) jailbreak methods, for example, Huang et al. (2024)  in *Stronger Universal and Transferable Attacks by Suppressing Refusals* show that aiming at suppressing DIM refusal directions with input attacks increases their universality and transferability.\n- Defensive Applications: Refusal directions can be used to improve safety via adversarial training (Yu et al., 2024) or inference-time monitoring.\n\n\n### Other \n**Added No-Intervention to Figure 2:**\nWe show the result in [Figures 21-23](https://figshare.com/s/4ab2ec422f6bd0262b30) and we observe that without intervention, the model rarely responds to harmful instructions.\n\n**Combination of Refusal Directions:**\nWe thank the reviewer for the great idea and show the result in [Figure 24](https://figshare.com/s/4ab2ec422f6bd0262b30) for the RepInd directions shown in Figure 7. We observe that the composition improves the ASR and using three RepInd directions outperforms DIM. We add this to the main body and believe that this experiment significantly strengthens our manuscript.\n\n**Related Work:**\nWe added a paragraph to over-refusal starting on the provided source papers and further list [6] as related work that uses SAEs and thank the reviewer for providing these references.\n\n**Typos:**\nWe iterate over the manuscript to remove typos\n\n\n### Conclusion\nWe again want to thank the reviewer for the insightful comments and questions. We believe that addressing the reviewer's concerns greatly improves our manuscript and hope the reviewer agrees with us.", "claims_and_evidence": "While the method improves refusal effectiveness, it also impacts performance on safe instructions. The results in Figure 2 (ASR) and Table 1 (utility) suggest a trade-off between safety and general utility. To better illustrate the advantages and limitations of different methods, it would be beneficial to include an overall performance metric that integrates both ASR and utility.", "methods_and_evaluation_criteria": "The design of gradient-based optimization for discovering refusal directions is overall reasonable and effective.", "theoretical_claims": "The claim regarding the independence of refusal directions lacks formal theoretical validation. Further theoretical validation could strengthen the claims. But it is understandable that proving representational independence is a challenging problem due to the modeling difficulty of neural network.", "experimental_designs_or_analyses": "Figure 7 shows that multiple independent directions exist for safe refusal. However, a key question seems undiscovered: what happens if these independent directions are combined as subtract directions? Would this improve or hinder performance? Investigating this could provide further insights into the structure and impact of these refusal directions.", "relation_to_broader_scientific_literature": "The paper contributes to a broader understanding of LLM refusal mechanisms. It provides evidence that refusal is not governed by a single direction, as suggested in prior work, but rather by a combination of multiple directions. This insight expands the existing literature on LLM safety mechanisms and adversarial attacks.", "essential_references_not_discussed": "1.\tI think over-refusal is an important point for this task, due to the trade-off relationship between the safety and utility. Namely, when enhancing a model’s refusal capability against unsafe samples, over-refusal inevitably emerges on safe samples. There has been a body of literature [1, 2, 3, 4, 5] about over-refusal, discuss and provide some insights for defending over-refusal in this work is necessary.\n2.\tThere is also some work [6] about sparse autoencoders in safety, but it could be regarded as a concurrent work, so not necessary.\n\n**Refs:**\n\n[1] Paul Röttger, et al, “XSTEST: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models”, NAACL, 2024.\n\n[2] Chenyu Shi, et al, “Navigating the OverKill in Large Language Models”, ACL, 2024.\n\n[3] Justin Cui, et al, “Or-bench: An over-refusal benchmark for large language models”, arXiv, 2024.\n\n[4] Hao Li, et al, “InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models”, arXiv, 2024.\n\n[5] Bang An, et al, “Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models”, arXiv, 2024.\n\n[6] Kyle O'Brien et al, “Steering Language Model Refusal with Sparse Autoencoders”, arXiv, 2024.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper provide an interesting insight of refusal mechanism in large language model.\n\nWeakness： \n1. Figure 2 lacks baseline results, making it difficult to compare performance improvements..\n2. The paper does not include an ablation study for the three loss components, which would provide insights into their individual contributions.\n3. Although the paper suggests that refusal mechanisms are governed by multiple directions, it does not offer deeper applications or guidance on how this knowledge can be leveraged in LLM safety or adversarial robustness.", "other_comments_or_suggestions": "1. Minor typos exist, such as in the caption of Figure 6, where there should be a space before “on the right.”\n\nBased on previous issues and concerns, I would like to give weak reject first, but if authors could clarify these concerns, I am willing to raise my rating.", "questions_for_authors": "Please refer to prior comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n9O3BGPCa8", "reviewer": "Reviewer_4YKU", "summary": "This work challenges the notion that refusal behavior in language models is mediated by a single direction. Previous research suggested that by ablating the activation strength along a specific direction, LMs could be made to refuse more or less often. The authors introduce a novel approach called Refusal Direction Optimization (RDO), which identifies optimal refusal directions while minimizing unintended effects on unrelated inputs. Their experiments demonstrate that RDO outperforms traditional methods that rely on contrasting prompts. Importantly, the authors show that multiple independent refusal directions can be found within the same model, suggesting that refusal mechanisms are more complex than previously understood.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and effort to examine our work. Below, we address the reviewers points.\n\n### Overall performance\nWe agree with the reviewer that our previous side-effect evaluation should be extended. We now provide results for the benchmarks: Arc Challenge, GSM8K, MMLU, and TruthfulQA (MC2), see [Table 5]((https://figshare.com/s/4ab2ec422f6bd0262b30)). While the improvement is not as strictly pronounced as for the Truthful QA, we observe that RDO performs better in the majority of the cases and provides on average a 24% reduction of error. We added these to the camera-ready appendix, linked to it from the main body, and adapted the framing in this paragraph to account for the overall improvement.\n\n### Baseline Improvements\nWe now provide the No-intervention baseline for the plot in Figure 2, see [Figures 21-23](https://figshare.com/s/4ab2ec422f6bd0262b30) for the respective datasets. It is important to note, that here in the experiment we show the attack success rate while trying to jailbreak the model, thus higher is better. We adjusted the text in the paper and the caption to make this more explicit. This, and our ablation study [Figures 18 & 19](https://figshare.com/s/4ab2ec422f6bd0262b30) shows we strictly improve the tradeoff between utility and ASR (higher is better) for our RDO direction compared to DIM as our direction Pareto-dominates DIM for many configurations. Due to space constraints, we kindly refer the reviewer to the 'Ablation studies ..' response to reviewer 5cSG for more details.\n\n### Over-Refusal\nWe acknowledge the important trade-off between safety and utility highlighted by the reviewer that is generally present for every safety method *compared* to no intervention. Our work primarily focuses on training refusal directions that are effective for model manipulation, specifically jailbreaking scenarios. Regarding over-refusal concerns,  adding refusal directions increases the probability of refusal generally, including over-refusal, as shown in Figure [Figure 25](https://figshare.com/s/4ab2ec422f6bd0262b30). Here, we add the refusal vector onto the representation with varying strength and plot the SORRY-Bench refusal score (higher is better) over the XSTest safe prompts refusal score (lower is better). We observe that RDO provides a better tradeoff as it refuses more harmful queries for the same rate of benign query refusal.\nInstead of increasing refusal, we can also *reduce* over-refusal by ablating the refusal direction. In [Figure 27](https://figshare.com/s/4ab2ec422f6bd0262b30) we show that results on safe instructions from the XSTest dataset. Here we observe that both approaches significantly reduce the already low over-refusal.\n\nWhile our current methodology focuses on identifying and understanding refusal mechanisms, rather than defending from adversarial attacks (which would have more direct implications on over-refusal), it could be extended to selectively reduce over-refusal without compromising refusal on harmful instructions by adjusting the loss functions. This could be achieved by using falsely refused instructions instead of harmful instructions in the ablation loss, and adding a retain loss for preserving refusal on harmful instructions. Though this specific application wasn't the focus of our current work, our framework can flexibly be repurposed for such interventions in future research.\n\n### Conclusion \nWe again thank the reviewer for the time, effort, and valuable feedback on our work. We believe that we can address all of the reviewer's concerns and hope the reviewer agrees with us. Further, we are happy to address any remaining questions!", "claims_and_evidence": "The primary claims in this paper appear supported by the provided evidence with the exception of the overall performance section. I discuss this concern later in my review.", "methods_and_evaluation_criteria": "They do make sense. The ability to steer model activations during inference is a plausibly effective approach for reducing jailbreak attack success rates (ASR). The theoretical foundation behind this approach is sound, as manipulating the model's internal representations could intercept harmful outputs before they manifest. The authors' techniques build logically on previous work in this domain while addressing some of its limitations.", "theoretical_claims": "I did not verify proofs. My interpretation is that the emperical evidence is strong.", "experimental_designs_or_analyses": "I find the evidence supporting the claim that multiple refusal vectors can be identified to be convincing. However, the experimental design omits important measurements, making it difficult to assess how effectively their proposed techniques improve upon the baseline.\n\n**Overall Performance**: Studying the effects on overall performance is an important consideration when introducing a new steering method—an aspect often neglected in previous works. However, measuring solely on TruthfulQA (TQA) provides a relatively narrow assessment of overall performance. My understanding is that the authors selected TQA primarily because this is a benchmark where the DIM baseline struggled. Beyond TQA being an insufficient evaluation of overall performance, selecting this benchmark solely based on the baseline's poor performance could lead skeptical readers to suspect cherry picking. I recommend that the authors include additional commonly studied benchmarks that measure broader capabilities, such as MMLU and GSM8K (though these specific benchmarks aren't strictly necessary).\n\nAnother important metric, especially if the paper aims to demonstrate the practicality of the technique, is over-refusal. Benchmarks like XStest and WildGuard are commonly used for this purpose. It would be valuable to investigate whether different refusal directions exhibit varying degrees of over-refusal.\n\n**Baseline Improvements**: I was unable to find the baseline ASR without any steering applied. How significantly does steering improve ASR compared to not steering at all? This information seems particularly relevant given that my interpretation of Figure 2 suggests ASR remains above 50% even with steering applied.", "relation_to_broader_scientific_literature": "This work challenges the notion that concepts of relevance to AI safety, such as refusal, are mediated by a single direction. The authors make their case via introducing a new direction identification technique.", "essential_references_not_discussed": "I am unaware of any obvious omissions that aren't concurrent work.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gTDfOqFPCg", "reviewer": "Reviewer_oMnW", "summary": "This paper investigates the refusal mechanism of LLMs with a gradient-based representation engineering method. They extract refusal directions with gradient optimization by transferring the two refusal direction requirements (addition and ablation) into two loss functions. Further, they give several novel findings: 1) there are multi-dimensional concept cones mediating refusal, and 2) we can find multiple refusal directions that are independent from each other. The research provides a foundation for future studies to understand and improve the robustness and reliability of LLMs in handling adversarially crafted inputs.\n\n## update after rebuttal\nThis paper presents a thorough investigation and offers interesting findings on the refusal mechanism of LLMs, which is a highly relevant topic. I continue to support acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable time for the review, the kind words, and the positive assessment.\n\n\n### Additional Data\nWe agree that we should increase the number of datasets we use for our evaluations. We thank the reviewer for the pointer towards SORRY-Bench and decided to evaluate our method on their base set of 440 harmful instructions. AdvBench is in our training data since it is part of SALAD-Bench and we therefore cannot use it for evaluation. As an alternative, we further evaluate on StrongREJECT. \n\nWe show the results in [Figures 21-23](https://figshare.com/s/4ab2ec422f6bd0262b30) for our three datasets. We observe that the results behave very similar to JailbreakBench which shows the generalizability of our findings and improves our manuscript. For the camera ready version we changed all figures to include the evaluation of all datasets together, instead of having plots for each dataset. If the reviewer suggests a different way to display the results though, we are happy to adjust. \n\n\n### Intervention positions\nWe apply the ablation operation across all layers and the activation addition on a single layer. This corresponds to the previous work by Arditi et al. in *Refusal is Mediated by a Single Direction*. We further always train the RDO directions and cones in the same layer as the DIM direction for a fair comparison. Future work could look at studying refusal directions across all layers. However, if the reviewer wants to see results for a different approach we are happy to investigate.\n\n### Conclusion\nWe again thank the reviewer for their time and effort. The additional data improves the significance of our findings and strengthens this work. We further added more experiments, other metrics, and evidence for causal independence during this rebuttal phase and detailed the changes in the responses to the other respective reviewers. We believe that the rebuttal further improved our work and we hope that the reviewer agrees with us.", "claims_and_evidence": "The claims in the paper are well-supported by experiments.", "methods_and_evaluation_criteria": "The evaluation benchmark used is JailbreakBench, which, while generally suitable, has shown biases in specific areas [2]. Including additional datasets such as AdvBench [1] or SorryBench [2] could provide a more comprehensive evaluation. Have you considered using other datasets?\n\n[1] Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023). Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.\n\n[2] Xie, T., Qi, X., Zeng, Y., Huang, Y., Sehwag, U. M., Huang, K., ... & Mittal, P. (2024). Sorry-bench: Systematically evaluating large language model safety refusal behaviors. arXiv preprint arXiv:2406.14598.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experimental design effectively addresses the research questions, and the analysis presents very novel findings.", "relation_to_broader_scientific_literature": "The paper presents novel insights into the refusal mechanisms of LLMs, enhancing our understanding of AI safety and encouraging further research into the robustness and reliability of LLMs.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "In the direction addition and ablation experiments, do you focus on a specific layer or do you apply modifications across all layers?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "br1jojJrMV", "reviewer": "Reviewer_NvcZ", "summary": "This paper explores the refusal behavior of LLMs and introduces a gradient-based algorithm called RDO to identify refusal directions $r$ in the activation space. These directions can shift the model’s behavior towards either refusal or acceptance. The authors design three distinct loss terms to help identify directions that can trigger refusal behavior while ensuring that other unrelated behaviors remain unaffected when adjusting refusal directions. Additionally, they extend their analysis to an N-dimensional polyhedral cone of refusal directions rather than a single vector $r$. The paper further examines whether the orthogonal refusal directions identified by RDO are independent and can be manipulated by modifying the input to the model. Experimental results reveal several key insights into the refusal behavior of LLMs, enhancing the understanding of this phenomenon.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort to review our manuscript which helps us to improve. \n\n### Cosine similarity captures linear relationships. RepInd also catches non-linear interactions.\nIf the reviewer means that the orthogonality of the cone basis vectors from section 5 is not enough, we fully agree! This is exactly why we investigate the representational independence in section 6.\n\nIn case the reviewer questions our representational independence in mitigating non-linear effects this might be due to a missing equation. We noticed that we forgot to specify the definition of the ablated $\\hat{x}$. \n\n$$ \n\\hat{x}\\_{abl}^{(l)} =  [\\hat{x}\\_{\\text{abl}}^{(l-1)} +  f_{[l]}(\\hat{x}\\_{\\text{abl}}^{(l-1)})]\\_\\text{abl}\n$$\n\nwhere $f_{[l]}$ is the model's transformation at layer $l$.\n\nThis means, that for rep. ind. we enforce the cosine similarity to be equal to the no-intervention baseline for every layer while propagating the ablated representation and ablating again. Intuitively, any non-linear effects would make it impossible to have the equality of cosine similarities of the subsequent layers. We will provide a simple proof by induction for the camera-ready version of the manuscript if the reviewer thinks this will bring clarity. \nTo argue for causal independence, we would need to show that the cosine similarity to a concept is what drives the model prediction. Since the readout of standard transformers is not linear, we cannot prove this unfortunately. However, there exists some empirical evidence in the mechanistic interpretability field (Elhage et al., Geva et al., Yousefi et al.) that the directional information (captured by cosine similarity) is what greatly drives the model's readout. \nWe will provide an extensive discussion of this topic in section 6 and thank the reviewer for the question as discussing it will improve our manuscript!\n\n### Alternative metrics apart from ASR\nWe provide alternative metrics to evaluate our refusal cones beyond the ASR. While ASR offers the most comprehensive assessment of the cone directions' effectiveness for model manipulation, it is computationally expensive. A more efficient alternative is the refusal metric proposed by Arditi et al. (2024). Using this metric, we show in [Figure 14](https://figshare.com/s/4ab2ec422f6bd0262b30) that the cone directions fulfill the monotonic scaling property we required from our refusal directions.\n\nBesides this, we now include over-refusal as an additional way to examine the performance of RDO directions, see response Over-Refusal to reviewer 4YKU.\nIf the reviewer has another idea on what we should evaluate or could give us any pointers, we would be happy to include more evidence in the next reply.\n\n### More data and models\nWhile we used the same test dataset as related work (Arditi et al., 2024), we agree with the reviewer that we could add more to convincingly validate our claims. We now include the datasets of StrongREJECT and SORRY-Bench, where our results behave similarly to JailbreakBench, strengthening our findings, see [Figures 21-23](https://figshare.com/s/4ab2ec422f6bd0262b30).\nFurther, we added more models and data for the orthogonality / representational independence experiments, see [Figure 26, 28](https://figshare.com/s/4ab2ec422f6bd0262b30).\n\n### Motivation for Refusal Directions (Q1)\nDue to writing constraints, we kindly ask the reviewer to read our **Applications** response to reviewer 5cSG. Apart from safety-relevant topics, this work further shows that RDO & RCO are suitable techniques to identify directions for precise representation steering which is currently a promising technique for LLM understanding.\n\n### Questions 2 & 3\nIn Table 1, the term 'Baseline' was initially unclear—we now clarify it refers to 'No intervention'. While RDO leads to significantly fewer side effects than DIM, it still introduces some compared to no intervention. Since LLMs are highly optimized, any intervention can reduce coherence. Additionally, benchmarks include sensitive questions, where safety interventions can yield differing outputs.\n\nThe relative performance has several explanations: (1) DIM was used to generate labels, creating a performance ceiling; (2) The StrongREJECT judge assesses harmfulness, which also reflects model capability. Outperforming DIM was not our goal—RDO serves as a foundation for RCO and the study of independent refusal directions. Figure 2 demonstrates that RDO achieves comparable or slightly better mediation performance with greater precision (fewer side effects). With further tuning, RDO could surpass DIM, but our focus is on proposing an alternative approach and analyzing its properties.\n\n### Conclusion\nWe again want to thank the reviewer for their insightful questions and remarks. The changes we applied to the paper improved our manuscript. We hope the reviewer agrees with us and would be happy to answer any further questions or concerns.", "claims_and_evidence": "The paper presents three main claims, each supported by a combination of theoretical arguments and empirical evidence. Overall, the evidence is generally convincing.\n\n**1. Gradient-based representation engineering can identify refusal directions.**\n- The authors introduce the Refusal Direction Optimization (RDO) method, which uses a gradient-based approach to identify vectors in the activation space that control refusal behavior.\n- They validate this claim by demonstrating that applying these vectors can reliably increase the probability of refusal (scaling property) and that projecting out these directions allows the model to respond to harmful prompts while retaining behavior on harmless prompts (projection property).\n- The use of cross-entropy loss for training these directions and the retain loss based on KL divergence to prevent interference with other model behaviors provides a robust framework for evaluating this claim.\n\n**2. Refusal behavior is governed by multi-dimensional cones rather than a single direction.**\n- The authors did experiments that measure ASR across cones with increasing dimensionality. The observation that higher-dimensional cones consistently mediate refusal behavior across different models suggests that refusal is indeed encoded in a multi-dimensional structure.\n- The use of orthonormal basis vectors ensures that the identified directions are non-overlapping, which further strengthens the evidence for multi-dimensional control of refusal.\n\n**3. Orthogonal directions in the cone are not just geometrically independent but also causally independent**\n- The experiment uses cosine similarity to test the interaction between orthogonal directions show that ablating one direction does not influence the other, demonstrating the causal independence of different directions.", "methods_and_evaluation_criteria": "The methods proposed in the paper, particularly RDO and its extension to multi-dimensional cones, are well-designed for probing and controlling refusal behavior in LLMs. The use of a gradient-based approach to identify refusal directions and the reliance on orthonormal bases to define multi-dimensional cones are reasonable choices given the complexity of activation spaces in LLMs. The evaluation strategy, which primarily hinges on the Attack Success Rate (ASR) and cosine similarity measures, provides an empirical basis for the claims made. These choices align well with the paper’s objectives of understanding and isolating the mechanisms of refusal behavior.\n\nHowever, there are several limitations that could impact the validity and generalizability of the findings. Firstly, they rely on ASR as the main metric to evaluate the quality of the cones, which might be limited. Could there be alternative or complementary metrics that can better capture, e.g., the robustness of the cones? Additionally, the use of cosine similarity to infer causal independence between orthogonal directions might be insufficient. Cosine similarity captures only linear relationships, potentially overlooking complex, non-linear interactions between directions in the activation space. Furthermore, the experiments on orthogonality focus on a small subset of harmful prompts and a single LLM architecture (Gemma 2 2B). This raises concerns about the robustness of the results across different model architectures and prompt types. It would be beneficial to investigate larger models and datasets. Addressing these limitations through a broader set of metrics—such as mutual information or causal inference techniques—and expanding the evaluation to include a wider range of prompts and models could significantly strengthen the paper’s contributions.", "theoretical_claims": "The proof for the scaling property is straightforward and relies on the assumption that scaling a direction within the activation space should proportionally increase refusal behavior. This proof appears correct, as it logically follows from the gradient-based optimization objective designed to maximize refusal probability. The mathematical argument, which demonstrates that a higher scaling factor leads to an increased likelihood of refusal, is both clear and valid within the context of the assumptions made.\n\nThe projection property proof is also reasonable, showing that projecting out the identified directions should enable the model to respond to harmful prompts while maintaining behavior on harmless ones. The proof leverages the orthogonality of the directions to argue that this ablation selectively removes the influence of refusal-related factors. While the proof is correct, given the orthogonality assumption, one potential issue is that it does not fully address the possibility of non-linear interactions between directions that might influence the model’s behavior indirectly.", "experimental_designs_or_analyses": "I examined the methodology behind the refusal direction optimization experiments, focusing primarily on the Attack Success Rate (ASR) metric and the directional ablation approach used to measure changes in refusal behavior. This setup is sound because it isolates the influence of specific directions in the activation space. However, relying on a single metric can overlook nuances in how robust or consistent those directions are across diverse prompts. Additionally, while the authors’ sampling strategy provides a reasonable set of prompts, the limited number of harmful examples may restrict the generalizability of the findings.\n\nThe layer-wise analysis of orthogonal directions is a valuable step toward understanding deeper interactions in the model, but it is relatively narrow in scope, concentrating on one model architecture (Gemma 2 2B) and a small set of prompts. Broadening the experiments to include multiple models and larger, more varied prompt sets could improve the robustness of the conclusions.", "relation_to_broader_scientific_literature": "The paper’s core contribution—using gradient-based representation engineering to identify and manipulate “refusal directions” in LLMs—builds upon several lines of existing research in interpretability and controllability. Recent works have investigated how specific concepts, such as sentiment or toxicity, can be localized in a model’s latent space and how targeted interventions (e.g., projecting out specific directions) can alter generation behaviors. The authors’ approach extends these ideas by focusing on refusal behaviors, a safety-critical function in many conversational and policy-driven systems.\n\nBy introducing a gradient-based approach rather than relying solely on paired prompts or heuristic methods, the paper also connects to the broader endeavor of fine-grained model steering. It strengthens a growing body of evidence that language models’ internal representations can be both discovered and selectively edited to meet specific policy or safety goals.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No. Please see the previous sections.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. Could the authors provide more insights or motivations regarding the importance of identifying refusal directions for LLMs? What specific objectives can be achieved using these refusal directions? A deeper understanding of these aspects could help clarify the scope of the paper.\n2. In Table 1, higher values indicate better performance. Why does RDO perform significantly worse than the baseline?\n3. In Figure 2, it appears that the performance of RDO is not substantially better than that of DIM. What might be the reasons for this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No."}}]

Author rebuttal:

Meta-review: This paper explores how using adversarial inputs can bypass the internal safety filters of large language models (LLMs). Concretely, the paper explores refusal behaviors in LLMs, finding multiple independent directions and multi-dimensional concept cones that control refusals. It proposes Refusal Direction Optimization (RDO) to identify optimal refusal directions with minimal unintended effects. Experiments show that exploiting these directions enhances attack success rates, indicating complex spatial structures in refusal mechanisms with each direction carrying complementary information. The reviewers unanimously agree that this brings much needed insights into the problem and they support the empirical evidence is sufficient.
Decision: Accept (poster)