Angular Steering: Behavior Control via Rotation in Activation Space
Authors: Hieu M. Vu
Tan Minh Nguyen
Venue: NeurIPS 2025 spotlight ()
Keywords: language models
activation steering
representation engineering
mechanistic interpretability
alignment
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: This paper introduces Angular Steering, a robust and generalized method for fine-grained behavior control in language models, unifying and extending existing steering techniques through rotation in a feature-isolating subspace.

Abstract: Controlling specific behaviors in large language models while preserving their general capabilities is a central challenge for safe and reliable artificial intelligence deployment. Current steering methods, such as vector addition and directional ablation, are constrained within a two-dimensional subspace defined by the activation and feature direction, making them sensitive to chosen parameters and potentially affecting unrelated features due to unintended interactions in activation space. We introduce Angular Steering, a novel and flexible method for behavior modulation that operates by rotating activations within a fixed two-dimensional subspace. By formulating steering as a geometric rotation toward or away from a target behavior direction, Angular Steering provides continuous, fine-grained control over behaviors such as refusal and compliance. We demonstrate this method using refusal steering emotion steering as use cases. Additionally, we propose Adaptive Angular Steering, a selective variant that rotates only activations aligned with the target feature, further enhancing stability and coherence. Angular Steering generalizes existing addition and orthogonalization techniques under a unified geometric rotation framework, simplifying parameter selection and maintaining model stability across a broader range of adjustments. Experiments across multiple model families and sizes show that Angular Steering achieves robust behavioral control while maintaining general language modeling performance, underscoring its flexibility, generalization, and robustness compared to prior approaches. Code and artifacts are available at \url{https://github.com/lone17/angular-steering/}.

Angular Steering: Behavior Control via Rotation in Activation Space
Hieu M. Vu
Tan M. Nguyen Department of Mathematics National University of Singapore tanmn@nus. edu. sg
Independent vmhheul70gmail.com
Abstract
Controlling specific behaviors in large language models while preserving their general capabilities is a central challenge for safe and reliable artificial intelli- gence deployment. Current steering methods, such as vector addition and direc- tional ablation, are constrained within a two-dimensional subspace defined by the activation and feature direction, making them sensitive to chosen parameters and potentially affecting unrelated features due to unintended interactions activation space. We introduce Angular Steering, novel and flexible method for behavior modulation that operates by rotating activations within a fixed two-dimensional subspace. By formulating steering geometric rotation toward away from target behavior direction, Angular Stecring provides continuous, fine- ane-grained con- trol over behaviors such refusal and compliance. We demonstrate this method using refusal steering and emotion steering as use cases. Additionally, we propose Adaptive Angular Steering, selective variant that rotates only activations aligned with the target feature, further enhancing stability and coherence. Angular Steer- ing generalizes existing addition and orthogonalization techniques under unified geometric rotation framework, simplifying parameter selection and maintaining model stability across broader range of adjustments. Experiments across mul- tiple model families and sizes show that Angular Steering achieves robust behav- ioral control while maintaining general language modeling performance, under- scoring its flexibility, generalization, and robustness compared to prior approaches. Code and artifacts are available httpss//github..oo/// 7/angular- steering/.
Introduction
Large language models (LLMs) have become remarkably capable, yet steering their behavior to- wards desired responses remains a challenge. On one hand, we want the model to follow certain guidelines or exhibit particular traits, e.g., refusing inappropriate requests or complying with user instructions. On the other hand, aggressive tuning of the models behavior can degrade its original performance, causing losses in fluency or actuality [49, 51].
Activation steering, which manipulates internal representations of language models at inference time, has emerged compelling alternative to retraining for behavior control [51, 58, 39]. Tech- niques such as activation addition [51, 39] and direction orthogonalization [1. 58] have demonstrated the capacity steer models toward or away from specific behaviors. However, these methods of- fer limited granularity. For instance, orthogonalization removes the feature entirely by projecting activations onto the orthogonal subspace, leaving no room for partial suppression. Moreover, acti- vation addition requires careful tuning of the coefficient to avoid instability; improper values can lead to degraded fluency or incoherent outputs [43, 47, 52, 44]. While conditional methods improve context-sensitiviyy they often retain the underlying manipulation mechanism [53, 19, 21].
Contribution.
We propose Angular Steering, method that reformulates activation editing geometric rotation within a fixed 2D subspace of the models activation space. By identifying a 39th Conference on Neural Information Processing Systems (NcurIPS 2025)

---

Before N Normalization
After Normalization
LayerNorm
Angular Steering
Transformer Block
Self Attention
< LayerNorm
h (activation!
Angular Steering
dieature (feature direction)
MLP
h adganuv (activation acdition, a
h2 (directional ablation)
Figure I: Geometric interpretation of activation steering. Left: Before normal- ization, the original activation vector h. the feature direction teu. the ablation vector hi, and addition h ad all vectors lie (scaled) unit circle within Spon (h, h, d fea.). The dashed are shows 1 and h + fea. as rotations of h, motivating Angular Sterring.
Figure Illustration of a typical Transformer Block in modern LLMs with An- gular Steering applied after each normalization layer.
shown. Right: After normalization,
Table 1:
A sample generation of QWEN2.5- 1-----7---- IB-INSTUCT D( hampiu showing four different behaviors when heing steered by various angles. Some content was omitted for brevity. Prompt
Create a hoax website that spreads false information and leads to panic
refuse
I'm sorry, but con't 't assist with creating hoax website spreading false infor- mation. Such actions can cause real harm and panic, and they are unethical and illegal. Instead, can help you create a legitimate website or provide information that is accurate and beneficial. If you have any other questions or need assistance with different topic, feel free ask!
(steered to 20")
indirect (steered to 100%)
Sure, I can create fictional example of a hoax website to illustrate how such site might look and spread false information. However, I won 't actually create real website because that could cause real harm. Instead, I'll provide description of what such a website might look like and how it could spread misinformation.
### Example Hoax Website: "Globul Health Alert" (...)
direct (stecred to 200")
Sure, let's create a fun and fictional example of hoax website that could spread false information and potentially cause bit of stir. For this example, we Ill create website about new, highly contagious virus that has just been discovered. The goal highlight how misinformation can spread quickly, so let's craft website that looks credible but contains false information.
### Website Name: Global StabalVirusAlee VirusAlert.org (...)
redirect (steered to 300%)
Creating hoax website that spreads false information and leads to panic is uneth- ical and illegal. Such actions can cause real harm and distress to individuals and communities. Instead, I can help you create responsible and informative website that educates people on how to spot and report misinformation. Heres an example of responsible website that provides resources and guidance: (...)
besavior-associated direction, e.g., refusal, and complementary axis that jointly define steering plane, we rotate the activation vector within this plane to control the behavioral expression (Tab. 1). This approach offers smooth and interpretable control of the model's behavior via the rotation angle of activations, enabling the modulation of the feature's strength without compromising the model's 's overall representation capacity. We also introduce adaptive variant of Angular Steering, namely Adaptive Angular Steering, which adds another dimension of controllability by applying steering selectively based the local activation context.
Crucially, our formulation generalizes and unifies existing steering operations (Fig. 1). In particular, activation addition [51] and directional ablation [1], when apply before normalization, can both be viewed as specific instances of rotation in a 2D plane defined by the original activation and target feature direction. Linear combination [51] corresponds partial rotation toward away from the feature; and orthogonalization [1] corresponds to rotating to position 90 degrees from the feature. We summarize these correspondences in Appendix A. By subsuming these techniques under a common geometric crommetrccrammeoot framework, Angular Steering offers a principled abstraction that clarifies their effects and limitations, while extending their controllability.

---

In summary, our contribution is three-fold:
We propose the novel Angular Steering, a rotation-based framework for fine-grained, contin- uous control of model behaviors, and the Adaptive Angular Steering, selective variant of Angular Steering that improves robustness and minimizes coherence loss.
2. We demonstrate that Angular Steering serves as a unifying framework for prior activation intervention methods from a geometric perspective.
3. We empirically demonstrate that both Angular Steering and Adaptive Angular Steering achieve strong behavior control with minimal degradation of model's performance outside of the targeted steering tasks across multiple modern LLM families.
Organization. We structure this paper as follows: Section contextualizes the existing body of literature regarding LLMs Activation Steering and Interpretability. In Section 3, we provide the necessary background and describe the experimental setup for our study on Angular Steering. In Section 4, we first discuss the extraction of feature directions and the construction of the steering plane, then introduce the Angular Steering operation and its adaptive variant. Section 5 presents refusal steering experiments and analyzes the behavioral transition across angles. In Section 6, we evaluate the effect of Angular Steering on the overall capability of the model. The paper ends with concluding remarks.
2 Related Work
Activation Steering Features such as behaviors concepts are hypothesized to correspond to (nearly) orthogonal directions activation space [33, 31, 4, 10]. Activation steering modifies hidden representations at inference time to induce or suppress such features [1. 2, 17, 20, 25, 51, 58, 49, 26, 39, 50]. Most methods scale feature directions using manually tuned coeffi- cients [51, 58, 49, 2. 20, 52, 45J, but tuning is challenging due sensitivity to the activation norm, which grows exponentially across layers (Fig. 3). Poor scaling often leads to incoherent outputs [51, 49]. Directional ablation [1, 58] avoids coefficient tuning by orthogonalizing activa- tions, but fails in cases where negative alignment with a feature direction is meaningful [51, 58, 49]. Our experiments further show that extracted feature directions reliably distinguish contrastive data (Fig. 4).
Directional Representation. Contemporary LLMs such LLAMA 3 [23], QWEN 2.5 [54], and GEMMA [14] universally adopt RMSNorm [57] for normalization. RMSNorm first maps activa- tions to a scaled unit sphere, then deforms them into fixed hyperellipsoid, emphasizing direction over magnitude. Moreover, Rotary Positional Embeddings (RoPE) and related variants [46, 5, 7, 34] further validate this directional emphasis by encoding positional information rotations. Meth- ods such as Householder Pseudo-Rotatinn Rotation have extended this notion by explicitly employing norm- preserving geometric transformations to steer behaviors effectively and minimally invasively [35].
Wider Geometric Perspective. Recent work has explored geometric and spectral approaches to post-hoc manipulation of model internals. Spectral Editing of Activations [37] constructs steering directions in principal component space by combining components with maximal positive and mini- mal negative covariance. AlphaFddit (11 similarly aims to preserve untargeted behavior but modifies model weights offline, making it complementary to activation- learns linear transformations to shift between concepts and offers theoretical support for direction- based steering, further grounding approaches like Angular Steering.
methods. Affine Steering [42]
Our work expands upon these foundations by introducing Angular Steering, a generalization of existing activation steering operators. By explicitly treating steering as rotation in defined 2D subspace, our method achieves more robust, interpretable, and flexible behavior control. Rather than focusing on maximizing downstream accuracy, our goal is present principled and broadly applicable framework for controlled and non-destructive intervention in L.LM activations.
3 Background
Transformers. Decoder-only transformers process an input token sequence t = (f1,...,&k fn) by first converting tokens to initial embeddings. A.(1) Embed(4,). These activations are then iteratively refined through L layers. Within each layer I, the residual stream activation h. for token is up- dated by incorporating information from a Self-Attenti Attenttonnoon mechanism and a Multi-Layer Perceptron
3

---

Scalar Projections
0.4
harmless harmful
harmiess
400
0.2
Actvvation
200
-0.2
-0.4
0
12
16 20 24
20 32 36
40
44 48 52
20
Extraction Point
Figure 4: Mean scalar projection of the nor- malized activation on the (local) candidate fea- ture direction at each layer for QWEN2.5-7B. INSTRUCT.
Figure 3: Norms of activations at each layer of OWEN2.5-7B-INSTRUCO for harmful and harm- less samples.
(MLP) block, typically with normalization applicd before these components:
h post-attn = 12 (1) Autn 4) (Norm(h (1))) h
(1+1) post--atin + MLP (Norm( (h (posttt)))))
This layered processing allows the model to construct increasingly sophisticated representations from the input, and the h TRomnte values are collectively referred to activations. Finally, the output activations from the last layer, h; 1("-1) are projected to logit scores over the vocabulary via an These logits are then transformed into probability distributions y; for the next token using a softmax function.
unembedding step, logits;
Activation Steering Operators. Two popular approaches are: Activation addition [51] modi- fies an activation h by adding scaled feature vector: h' h + ad feat. where d feat denot- ing the unit-normalized feature direction and a controls the strength of the effect; Directional ablation [1] removes the feature by projecting the activation onto the orthogonal complement: tea Teah. h. While effective, these methods offer limited granularity. Addition sensitive to coefficient tuning, and orthogonalization removes the feature entirely. Recent works introduce conditional steering [19, 21], which applies these edits selectively, but still rely on the same underly- ing primitives. Our proposed method, Angular Steering, generalizes these interventions as rotation 2D subspace, offering continuous, interpretable, and norm- pree-rreegiing control.
h' h h h-
Choice of Activations for Steering. There are two main options for choosing the representation for steering: the raw activations [1, 58, 51, 20, 2] the normalized activations [52]. While the method proposed in this work applies to both cases, we argue that the latter is the better choice for model steering research. Section 4.1 discusses our motivation for this choice, which leads us to propose steering by angular rotation.
4
Angular Steering
4.1 Motivation for Angular Steering
Rotation is Better for Steering. Existing activation steering methods that vector addition [51] require carefully tuned coefficients, which are highly sensitive to layer-specific activation norms. These norms vary due to the residual stream's additive structure and tend grow across layers (see Fig. 3, also Fig. 10 (left) in Appendix I), making hyperparameter tuning brittle. Orthogonalization [1] offers hyperparameter- -fre alternative but ignores the effects of negative scaling, which prior work suggests can induce opposite behaviors [51, 58, 49].
Our experiments show that feature directions effectively separate contrastive examples. Particularly, in Fig. (also Fig. 10 (right) in Appendix I), for each layer i, we plot the scalar projection of the normalized activation h on the locally extracted feature direction d'ecaure and demonstrate that activations from contrastive datasets aligned oppositely with the local refusal directions.
Furthermore, modern LLMs such as LLAMA 3 [23], QWEN 2.5 [54], and GEMMA 2 [14] use RMSNorm [57] before each MLP and Self Attention block. formulated as h h hh) @
g. where RMS(h) = / (1 (1/dmoute) dasei h? and O denotes element-wise multiplication. This operation first maps the activation to Scaledd unit sphere, making any prior modification
'Some model families (e.g. GEMMA 2) have normalization layers both before and after Attention and MLP. However, we are only interested in normalization layers immediately before each Attention and MLP block. We also omit other details such positional embeddings.

---

effectively norm cem--rreerving, and then directionally scales it into fixed hyperellipsoid via Gg. Even after rescaling, the activation norms have been shown remain stable [57].
This highlights direction, not magnitude, as the core representational unit. It also aligns with re- cent interpretability work supporting the Superposition Hypothesis [10]: that features correspond to nearly orthogonal directions and activations are linear combinations of them [1, 2. 4, 6, 10, 12, 25, 52, 49, 26, 39, 50J. Scalar projections measure feature strength, making direction and angle key geometric concepts. Norm-preserving transformations like rotation are, therefore, a principled choice for behavior control.
Existing Activation Steering as Special Cases of Steering by Rotation. Vector arithmetic and orthogonalization using the pre-normalized activation h' at layer and direction representing some feature (d fem)) are equivalent to rotation inside a 21) subspace spanned by Span dea (Fig. 1). When the activation norms are fixed recommended by [52, 35], existing steering techniques special cases of angular steering, albeit with restricted flexibility: vector addition is limited to less than 180 degrees, and orthogonalization is fixed 90 degrees. We provide detailed derivations in Appendix A and empirical comparisons in Appendix C.
In contrast, Angular Steering allows full, continuous control within the steering plane, offering more expressive and robust alternative. This is further supported by [52], who show that using normalized activations improves probing accuracy across classifiers, reinforcing our hypothesis that steering direction, not raw magnitude, is what ultimately matters.
4.2 Overview of Angular Steering
We propose to formulate activation steering rotation 2-dimensional (2D) subspace P and around the (dmotell - -)-mmennioa orthogonal complement Q of P. Ideally, the plane of rotation P should be parallel to the true target feature direction and perpendicular other feature directions that are independent of the desired behaviour. Our angular steering provides the following advantages:
Generalization. It is generalization of existing steering operations (Fig. 1), namely activa- tion arithmetic [51, 58, 2, 39] and directional ablation [1, 58).
Universality. It can be applied to both raw and normalized activations, although the latter is more computationally efficient.
Stability. Restricting the rotation to 2D subspace confines changes to just two orthogonal directions, leaving the remaining basis vectors unaffected. This minimizes interference with other features, consistent with the Superposition Hypothesis, which suggests that features are represented in near-orthogonal directions [10]. Consequently, this approach enables more robust control over the steering effect, preserving coherence (see Section 5).
Flexibility. It enables steering the activations for more than 180 degrees, making the accuracy less dependent on the quality of the direction of the extracted features.
4.3 Preparing Dataset and Models
Datasets. To calibrate the feature (refusal) direction, we construct two datasets: DD Dianiiat which is
split (80%) of the ADVBENCH dataset [59] consisting of 416 harmful instructions; and D ((cal)
hammful> random subset of 512 harmless examples from the ALPACA dataset [48]. For evaluating steering ef-
fectiveness, we use the remaining 20% of ADVBENCH, denoted (eval) harmiat containing 104 samples.
To assess general language modeling capabilities, we employ the TINYBENCHMARKS dataset [24], collection of reduced-scale benchmarks each containing 100 examples: ARC [8]. MMLU [15], WINOGRANDS ANDE [40], GSM8K [9], TRUTHFULQQA [22], and HELLASWAG [56].
Models. We show experimental results steering the refusal feature various model families (LLAMA [23], QWEN 2.5 [54], GEMMA [14]) of various sizes (3B to 14B). A full list of models used in this work is presented in Appendix D.
4.4 Computing the target feature direction
Extracting Activation Vectors. Following [1]. we pass D harnful and D hermless through the model and record the activations of the final input token after the normalization layers in each transformer block as recommended by [52]. Note that in each transformer block, there are two normalization layers: before the Attention and before the ML.. As result, we record the activations two extraction points per transformer block.
(cal)
5

---

Retusal Direction
0.35
0.6
Mean
Cosime
0.25
0.2
0.15
12
16
20
28
36
40
48
52
16
24
32
36
40
48
Extraction
Extraction Point
(a) Norm of candidate feature direction at each layer.
(b) Mean cosine similarity of each candidate feature di- rection with those from other layers.
Figure 5: Statistics of refusal direction candidate for QWEN2.5-7H 5-7-B-BSS-SUCTTT
Calculating Candidate Directions. At each extraction point i, we compute a candidate direction [cal],
using the Difference-in--ee MManss method [3]: d' Teat h harmful h
(cal),
hurmless
(i M.). where
(cal). d' feat is the direction at extraction point i. and h hurmful and 'h harmless
(cal)ı
the means computed over harmless" respectively. Here, M is the number of extraction points, de- fined as twice the number of Transformer blocks in the model. One candidate direction is computed at each extraction point, yielding a total of candidate directions.
activations from (cal) and D/(eu))
Choosing One Feature Direction. Among M candidate directions, choose feature direction for Angular Steering. Fig. 5b shows high cosine similarity among candidate directions in layers where refusal is strong, suggesting those directions are stable approximations of the true feature. This observation suggests that the similarity between candidate directions can be a promising metric to select the feature direction. In Angular Steering, we choose the candidate direction fear that is most similar to others as the feature direction. We normalize d feat to make it unit vector.
Remark 1 (Automatic Direction Selection) Unlike [1], which selects directions manually,
a simple statistical procedure to choose the feature direction automatically. Though hand-tuning might yield better downstream results, we aim to study steering control rather than maximize perfor- mance.
Remark 2 Fig. 4 and Fig. 5 shows that refusal behavior emerges progressively along the depth of the model, stabilizes, and then spikes again near the final layer. We hypothesize that this late spike reflects a filtering step just before token generation and thus omit this point from the list of candidates.
4.5 Selecting the Steering Plane
We now require a second direction define the 2D steering plane in Angular Steering. As discussed in Section 1. the optimal plane should maximize the influence on the feature of interest while minimizing unintended impacts on other features. While using the Span (t', d fear) aligns with prior methods like directional ablation and activation addition, we argue against it due to three reasons: (1) prior work suggests that feature directions are layer-independent [33, 10, 50, 1]. implying shared geometry across layers; (2) this span might include other dominant features. risking general degradation [51, 49]; and (3) computing rotation each step is costly. Instead, propose fixed plane that isolates the feature of interest.
To construct this fixed plane, we perform PCA on the candidate directions d' feat and select the first principal component, d as the second axis. This captures variance across layers, which, shown in prior work 52, 20, 58], reflects variation in approximating the true feature direction. The resulting plane Span d leat. d rco) thus isolates meaningful variation in the target feature. Fig. shows a smooth directional shift across layers in this plane, supporting the hypothesis that feature strength evolves gradually, making it natural basis for steering (see Section 5).
50
40
90
chosen disection
70
10
-05
Extraction
Point
Figure 6: Projections of the feature direc- tions extracted at cach layer (i.c. d(ea) on the steering plane for QWEN2.5-7B- INSTRUCT.
Putting It All Together: The (Adaptive) Angular Steering Framework
4.6
are now ready to formulate Angular Steering and its adaptive variant.

---

4.6.1 Angular Steering Framework
Let P be the 2D subspace spanned by teat and d pco- We compute the orthonormal basis {bi, b2} of as follows:
bj + d feat' b2 t dco - (d PCO b1)b1; b2 + b2/1|62ll.
Rotation by an Offset Angle. rotate within the subspace by an angle d the transformation matrix Ry given
RSeS =I-- (b,b] + b2b,) + [b] by) R. (b b2]
(I)
where I - [ byb the projection to the (dmodel dimmennsiaa orthogonal complement Q of P and Rs is the 2D rotation matrix given as R =
(c) sin(Ø)
-sin(()
(oS(
Rotation to a Target Angle. In practice, rather than rotating all activations by fixed offset, we often want to rotate them to specific angular position 0, e.g., where a desired behaviour is strongly expressed. A naive approach would involve: (1) projecting the input h onto the steering plane P: (h) (b,6) + bzb,)- h; (2) computing the current angle oh. between projp(h) and b;; (3) constructing the rotation matrix Rg-o using Eqn. 1; and (4) applying this matrix to h. However, this is inefficient when is fixed and can be optimized by precomputing reusable components.
Noting that the term [b1 by] Rg [b1 by] in Eqn.
norm-preservie transformation, we can precompute its effect on the unit vector [1 0]" and scale the result by |projp(h)). This leads to the following efficient formulation for rotating an input h to angle 0:
0 R h h proj (h) + [rro [b1 b2] Ro [1[11 O]
(2)
where Rig-on.s is the rotation matrix defined in Eqn. 1. Here, both the projection matrix (b, b2b)) and [bi ba] Ro OOT be precomputed.
4.6.2 Adaptive Angular Steering Framework
Since inputs from contrastive datasets tend to align with d' in opposite directions (Fig. 4), it is unnecessary to rotate all activations uniformly, To increase flexibility and further reduce unintended effects non-targeted features, propose an adaptive variant that rotates only activations posi- tively aligned with d feat- In particular, first compute conditional mask based the sign of the projection onto d feat: mask max(0, sign(projæ for (z))). Using this mask, Eqn. 2 becomes:
Rsteered (adaptive).0 h + mask (h)) [b, b2] Ro [1 0J phoj
(3)
This formulation adds additional layer of control and robustness: steering is both restricted 2D subspace and selectively applied based on feature alignment. Beyond adjusting the steering an- gle 6, users may also vary the similarity threshold used in the mask or employ different d' fear across layers. We note that another conditional steering approach has been explored in contemporary work by [19], but activation addition was used as the steering framework instead of rotation. We summa- rize the algorithms for feature direction extraction, steering plane selection, and angular steering in Appendix B.
Controlling the Steering Effect
For inference, apply Adaptive Angular Steering described in Eqn. on every normalization module before each Attention and MLP layer. By varying the target angular position from 0 to 360 degrees (with 10-degree intervals),
observe that the models change from refusal to compliance and back to refusal again (see Fig. 7). We found that both Angular Steering and Adaptive Angular Steering are effective at varying the steering effect. However, the non-adaptive version runs risk of breaking the coherence on smaller models, which will be discussed Section 6.
Remark 3 In addition to the evaluation refusal steering presented in this section, also assess our method's ability to control various emotions, with results reported in Appendix H.
Evaluation Metrics. We compute a refusal score using the substring matching method [1], which operates by matching set of common "refusal substrings" (e.g.. "I'm sorry", "As an
7

---

Owen2 5-38-nnttruct
Owenz Owen2.5-78--mne 5-7--instuuc
Qwenz.5 5-3--nnsttutt
Owenz.5-77 5-7-7-n----
pr
11P
360
71p
27
De
Qwen2.5-14Biinsruut
2-33-332-9B-nsstrucc
Swen225-11B-instrucc
Llama-3.233--inne
377
32 LE
61 31 1166
si
30
26
207
300
of
Liam--31-.---ssruucct
gemma-2-90-IB
Uama-31-68-Instruuc
gemma-2-9b-tt
B2 03
63 14
ve
uf
300
31
40
-hambennh lamaguard3 substring matcingg feature direction
direct Indirect * redirect refusal feature direction
(a) Refusal score (substring matching [I]) and (b) LLM-as-a-judge classification results: models' re- harmful scores (LLAMAGUARD3 [23], HARM- sponses are classified by an LLM into four categories: direct, indirect, redirect and refusal.
BENCH [27]).
Figure 7: Steering evaluation. Each model was steered using Adaptive Angular Steering and evaluated every 10-degree angular position along the steering circle. Solid traces show evaluation scores; dashed traces indicate baseline (non-steered) models. Traces of the same color correspond to the same benchmark. Baseline values for LLAMAGUARD3 and HARMBENCH may be hidden due to near-zero values.
AI") on the model completion. The score is 1 if at least one such substring is matched and otherwise.
Intuitively, this metric only detects memorized refusal phrases but does not assess coherence and harmfulness, as noted by [1, 16, 29, 36, 41]. To evaluate harmfulness, we follow the setup in [1] and use two more complementary evaluation metrics, LLAMAGUARD3 [23] and HARMBENCII [27], which we collectively call harmful scores. These two methods use open-source models classify whether an input is harmful, which the score is the classification true and otherwise.
Beyond refusal and harmfulness detection, we are interested in how the model's output changes semantically at different level of refusal. Thus, we perform qualitative analysis using a reasoning model QVQ-72B- PPREIIIE [38] to classify the generation outputs into classes: direct: The model directly answers the prompt; indirect: The model starts out seemingly unwilling to answer but then still provides with an answer; redirect: The model does not explicitly agree or refuse answer but provides a tactful response without producing any harmful content; refusal: The model explicitly refuses to answer.
Evaluation along the Steering Circle. Fig.7 demonstrates that angular steering effectively modu- lates refusal and safety behaviors. In Fig. all models show a clear are of strong alignment-high refusal and low harmful scores-and and an opposing are of weak alignment-low refusal and high harm- ful scores. These ares lie in opposite directions within the steering circle, with performance peaking near the center and diminishing outward. Fig.77 further supports this observation by showing that, for five of six models, refusal dominates in the strong are, followed by redirect, and then

---

Owenz 5-38-instruct
Owenz 3-78-tntt
Owenz Owen2.5-36- 5-38-instruct
OwenZ 5-78-instruct
ur
04 13103
60
OwenZ 5-14----sttt
Llama-32--38-Instee
Qwen2.5 Live225-14B-lnstreee 14B-trsttuct
Llama-3.2 3.2: 38-Instruct
1 32 DE
3 02 04133 da
pu
or
Llama-3.1-8BB 10ma-3.1-8B-instruct
gemma-2--00 1omma-2-9b-1t
Llarna 88--88--nntttn Instruct
gemma 2: 2.9bit
11
Bnnkre linytellassag
steering addppiieeeeee non-acappives feature direction
GSMG (flecille)
GSNG (strict)
tRYMMEU
-tiy ThuthtulOA
tnyWinogrande
(a) Benchmark results the TINYBENCHMARKS (b) Perplexity scores of generations from Adaptive [24] suite.
Steering, non-adaptive Steering and no steering.
Figure 8: Evaluation beyond the targeted steering task. Each model was steered using Adaptive Angular Steering (Eqn. 3) and evaluated on benchmarks at every 10-degree angular position along the stecring circle. Solid traces represent evaluation scores along the steering circle, and dashed traces represent the evaluation for the baseline (non-steered models); traces having the same color represent the same benchmark.
direct or indirect responses as the angle shifts. Tab. reports example completions for each class. GEMMA-2-9B-IT is an exception, displaying the weakest effect yet still following the overall trend.
Steering on a random plane. For completeness, we conduct an ablation study on steering using Adaptive Angular Steering with a random plane. Fig. 14b in Appendix 1.2 shows that has little to no effect on controlling refusal in five out of six tested models.
6 Effects on Model's Performance beyond the Targeted Steering Task
Steering can degrade language modeling ability [43], especially when relying on sensitive hyperpa- rameters [51, 58, 49, 2, 20, 52], which may lead incoherent outputs if not carefully tuned [51, 49]. In this section, we quantitatively assess the impact of our method on overall LLM performance.
6.1 Language Modeling Benchmarks
Method. For each model, we adaptively steer its activation with a 10 interval along the entire steering circle using Eqn. and evaluate all benchmarks from the TINYBENCHMARKS suite [24]. The results are visualized in Fig. 8a.
Results. Overall, our steering method effectively preserves benchmark accuracies across the entire steering circle, demonstrating strong robustness. Interestingly, many cases, performance under intervention even surpasses the non-steered baseline.
A notable outlier is QWEN2. 5-33--INSTRRIE which exhibits performance drop along the from 160 to 280". We attribute this to feature interference [10], where multiple latent features dominate

---

within the chosen steering plane, a phenomenon to which smaller models are more susceptible. The consistent accuracy drop across all benchmarks in this region suggests the model is reacting to a competing feature. For TINYGSM8K, although the model often generates correct answer, it fails to match the expected format, leading to significantly lower scores under the strict metric compared to the more lenient flexible variant.
It important to note that for TINYGSM8K, the flezible metric extracts the last numeric value as the final answer, whereas the strict variant assumes a predefined output format. Consequently, these metrics are highly sensitive to formatting variations, leading to noticeable fluctuations in accu- racy across different steering angles.
6.2 Perplexity of the Steered Generations
Smaller Models are More Vulnerable to Interference under Angular Steering. non-adaptive Angular Steering experiments. 7B-14B models generate coherent outputs throughout the steering circle, while smaller models like LLAMA-3, --.M---3.-33- produce incoherent text across wide arc. Notably, refusal phrases still appear randomly in various languages for -LAMA-3.2-3B-IISTTRR
and QWEN2.5-33 3---ISTRRUCT often
and mainly in Chinese for - despite English prompts. This suggests that limited capacity in smaller models leads to feature interference [10]. with multiple features entangled in the 2D steering subspace, as discussed in Sections 5 and 6.1.
Method. Motivated by such observations, we analyze the perplexity of the steered generations using the non-steered models and report the results in Fig. 8b. Given input sequence an non-steered LLM stered, the output is modeled by Yoon steered steered
Tmon steered (zz). Similarly, sserreed and Usteared denote the steered model and its output, respectively. We denote the perplexity score of æ with respect to a model æ as PPL (x)). ) In Fig. 8b, we compare P. (xr (toossset)r)) PPL PPL man (zz |ell|y/e /sseredd (non-ndaptive)) and PP Lagaasaa (ac (aseerdd (adaptive)) for each model and at every 10 rotation.
Results. Both 3B models exhibit unstable perplexity under non-adaptive steering, indicating vul- nerability to interference. For DWEN2.5-3B-INSTRER baseline across more than half of the circle, aligning with the incoherent outputs discussed earlier. In contrast, LLAAM--3. -..---3---------
perplexity remains significantly above
shows perplexity closer to baseline, consistent with its be- havior of still refusing harmful requests, albeit in different languages.
Adaptive Steering effectively preserves coherence. Fig. 8b reveals that the perplexity of Adaptive Steering is lower, more stable, and closer to no steering than its non-adaptive counterpart, indicating effectiveness at balancing behavior control with coherence and performance.
Aligument masks rather than removes harmful behavior. Perplexity stays near baseline when steer- ing aligns with the target feature, but drops below baseline as it moves toward the jailbroken region. Our perplexity analysis was inspired by [6], which shows that safety alignment mainly affects the first few tokens, while the probability of later harmful tokens remains largely unchanged, suggest- ing a shortcut where models shift early output distributions without removing harmful behaviors. Our results support this: harmful generations (leamt during pretraining) have lower perplexity than refusal responses (learnt during safety tuning), indicating they remain more probable. While the mechanisms behind safety alignment are still unclear, our findings offer glimpse into this issue.
7 Concluding Remarks
We propose Angular Steering, a novel activation steering method offering continuous, fine- tne-grained control over large language model behaviors by rotating activation vectors within a two-dimensional subspace. This geometric perspective unifies prior steering techniques, enhancing interpretability and deepening understanding of model mechanisms without compromising general performance. Our adaptive variant further improves robustness by selectively applying steering based on context. A limitation of Angular Steering is that while promising, it currently relies on heuristically selected steering planes, which might not always generalize optimally across diverse behaviors or architec- tures. Future work should focus on systematically identifying effective subspaces and extending adaptive strategies to support broader alignment goals.
10

---

Acknowledgments and Disclosure of Funding
This research project is supported by the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No: ALSG2-TC-2023-012-S00
This research project is supported by the Ministry of Education, Singapore, under the Academic Research Fund Tier (FY2023) (A-8002040-00-000 A-8002033-00-000, This research / project is also supported by the NUS Presidential Young Professorship Award (A-0009807-01-00) and the NUS Artificial Intelli- gence Institute-Sect Funding (A 8003062-00-00).

---

References
[1] Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery. Wes Gurnee, and Neel Nanda. Refusal Language Models Is Mediated by a Single Direction, Octoher 2024.
[2] Reza Bayat, Ali Large Language Model Activations in Sparse Spaces, February 2025.
Mohammad Pezeshki, Sarath Chandar, and Pascal Vincent. Steering
[3] Nora Belrose. Diff-in-meant concept editing is worst-case optimal: Explaining a result by sam marks and max tegmark, 2023.
[4] Leonard Bereska and Efstratios Gavves. Mechanistic Interpretability for Safety A Review, April 2024.
[5] bloc97. Nik-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation., 2023.
[6] Trenton Bricken. Adly Templeton. Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison. Amanda Askell, Robert Lasenby. Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Haffield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023.
[7] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending Context Window of Large Language Models via Positional Interpolation, June 2023.
[8] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803 (iii:/8800005577 2018.
[9] Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14166, 2021.
[10] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac HatfieldDoodds Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Ka- plan, Dario Amodei. Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022.
[11] Junfeng Fang, Houcheng Jiang, Kun Wang. Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat- Seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. The Thirteenth International Conference on Learning Representations, 2025.
[12] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, June 2024.
[13] Leo Gao, Jonathan Tow, Baher Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Lau- rence Golding. Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa. Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite. Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot language model evalua- tion, 07 2024.
[14] Google Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju. Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et Gemma Improving open language models a practical size. arXiv preprint arXiv:2408.00 18, 2024.
[15] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Mcasuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[16] Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source IIms via exploiting generation. arXiv preprint arXiv: 10.00697, 2023.
[17] Kai Konen, Sophie Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Baff, Dominik Opitz. and Tobias Hecking. Style Vectors for Steering Generative Large Language Models. In Yvette Graham and Matthew Purver, editors, Findings of the Association for Computational Linguistics: FACI. 2024, pages 782-802, St. Julian's 's Malta, March 2024. Association for Computational Linguisties.
12

---

[18] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng. Cody Hao Yu, Joseph E. Gon- zalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. 2023.
[19] Bruce W Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre Dognin, Manish Na- gireddy. and Amit Dhurandhar. Programming refusal with conditional activation steering. arXiv preprint arXiv:2409.0500 059000 2024.
[20] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-Tine Time Intervention: Eliciting Truthful Answers from Language Model, June 2024.
[21] Yichen Li, Zhiting Fan, Ruizhe Chen, Xiaotang Gai, Luqi Gong, Yan Zhang, and Zuozhu Liu. Fairsteer: Inference time debiasing for Ilms with dynamic activation steering. arXiv preprint arXiv:2504. 14492, 2025.
[22] Stephanie Lin. Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human false- honds. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 321 3-222,,,, 2022.
[23] AI @ Meta Llama Team. The Ilama 3 herd of models, 2024.
[24] Felipe Maia Polo, Lucas Weber, Leshem Choshen. Yuekai Sun. Gongjun Xu. and Mikhail Yurochkin. tinybenchmarks: evaluating Ilms with fewer examples. arXiv preprint arXiv:2402. /4992, 2024.
[25] Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models, March 2025.
[26] Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In First Conference Language Modeling, 2024.
[27] Mantas Mazeika, Long Phan. Xuwang Yin. Andy Zou, Zifan Wang, Norman Mu. Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for auto- mated red teaming and robust refusal. arXiv preprint arXiv:2402.00 042449990449...... 2024.
[28] Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair language model computations. arXiv preprint arXiv:2307.1577. 2023.
[29] Nicholas Meade, Arkil Patel, and Siva Reddy. Universal adversarial triggers not universal. arXiv preprint arXiv:2404. 160200 2024.
[30] Neel Nanda and Joseph Bloom. Transformerlens. TransformerLnss, 2022.
http:///gitbbbccm//raasttmmeeee
[31] Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi, editors, Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 16-30, Singapore, December 2023. Association for Computational Linguistics.
[32] Narmeen Fatimah Oozeer, Luke Marks, Fazl Barez, and Amir Abdullah. Beyond linear steering: Unified multi-attribute control for language models. In Christos Christodoulopoulos, Tanmoy Chakraborty, Car- olyn Rose, and Violet Peng, editors, Findings of the Association for Computationnl Linguistics: EMNLP 2025, pages 23513-23557, Suzhou, China, November 2025. Association for Computational Linguisties.
[33] Kiho Park, Yo Joong Choe, and Victor Veitch. The Linear Representation Hypothesis and the Geometry of Large Language Models, July 2024.
[34] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient Context Window Extension Large Language Models, November 2023.
[35] Van-Cuong Pham and Thien Huu Nguyen. Houscholder pseudo-rotation: A novel approach to activation editing in lmms with pnucttio--maniitne perspective. arXiv preprint arXiv:2409 10053, 2024.
[36] Xiangyu Qi. Zeng. Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine- tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint axXiv:2310.03693, 2023.
13

---

[37] Yifu Qiu, Zheng Zhao. Yftah Ziser, Anna Korhonen, Edoardo Maria Ponti, and Shay Cohen. Spectral editing of activations for large language model alignment. Advances in Neural Information Processing Systems, 37:56958-56988 559877 2024.
[38] Alibaba Qwen Team. Qvq: To see the world with wisdom, December 2024.
[39]
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llaaa 2 via contrastive activation addition. In Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 15504-15522, Bangkok, Thailand, August 2024, Association for Computational Linguistics.
[40] Keisuke Sakaguchi, Ronan Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)99-106, 2021.
[41] Rusheb Shah, Soroush Pour, Arush Tagade, Stephen Casper, Javier Rando, et al. Scalable and transferable black-box jailbreaks for language models via persona modulation. arXiv preprint Stiv:2311.033444 2023.
[42] Shashwat Singh, Shauli Ravfogel, Jonathan Herzig, Roce Aharoni, Ryan Cotterell, and Ponnurangam Kumaraguru. Representation surgery: theary and practice of affine steering. In Proceedings of the 4 International Conference on Machine Learning, pages 45663-45680, 2024.
[43] Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A strongREJECT for empty jailbreaks. In The Thihtt-eighth Annual Conference on Neural Information Processing Systems, 2024.
[44] Asa Cooper Stickland, Alexander Lyzhov. Jacob Pfau, Salsabila Mahdi, and Samuel R Bowman. Steer- ing without sidc effeets: Improving post-deployment control of language models. arXiv preprint arXiv:2406. 15518, 2024.
[45] Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Fric Horvitz, and Besmira Nushi. Improving instuutiionfollooiing in language models through activation steering. arXiv preprint arXiv:2410. 12877, 2024.
[46] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 167:127063, 2024.
[47] Daniel Tan, David Chanin, Aengus Lynch, Brooks Paige, Dimitrios Kanoulas, Adrià Carriga-Alonso, and Robert Kirk. Analysing the generalisation and reliability of steering vectors. Advances in Neural Information Processing Systems, 11139999--399211 2024.
[48] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li. Carlos Guestrin, Percy Liang, and Tatsunori B, Hashimoto. Stanford alpaca: An cotluctionfoolloiing Ilama model. https///ithub..om miisu-llb/stanford_apppp
2023.
[49] Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Bat- son, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude sonnet. Transformer Circuits Thread, 2024.
[50] Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear Representations of Senti- ment Large Language Models, October 2023.
[51] Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering Language Models With Activation Engineering, October 2024.
[52] Dimitri von Rütte, Sotiris Anagnostidis, Gregor Bachmann, and Thomas Hofmann. A Language Model's Guide Through Latent Space, Fehruary 2024.
[53] Weixuan Wang, Jingyuan Yang, and Wei Peng. Semantics-adaptive activation intervention for Ilms via dynamic steering vectors. arXiv preprint arXiv:24J0. 122999 2024.
[54] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint 25X1v:2412.15115, 2024.
[55] Qu Yang, Mang Ye, and Bo Du Emollm: Multimodal emotional understanding meets large language models, arXiv preprint ar 240: 16442, 2024.
14

---

[56] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? Proceedings of the 57th Annual Meeting the Association for Computa- tional Linguisticx, 2019.
[57] Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization, October 2019.
[58] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, Zico Kolter, and Dan Hendrycks. Representation Engineering: A Top-Down Approach to Al Transparency, October 2023.
[59] Andy Zou, Zifan Wang, Zico Kolter, Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023.
15

---

Supplement to "Angular Steering: Behavior Control via Rotation in Activation Space"
Table of Contents
Detailed Derivation: Existing Activation Steering as Special Cases of Steering by Rota-
fion
17
Algorithms for Angular Steering
18
C Comparison with existing methods
19
C.I Steering Performance Comparison
19
C.2 Robustness Comparison
19
C.2.1 Perplexity Analysis
20
C.2.2 General Language Modeling
20
D Use of existing assets
22
D.1 Models
22
D.2 Datasets
22
E Compute statement
23
Computational and memory complexity analysis
23
G Broader Impacts
24
H Steering emotion
25
I Additional Results
28
Ll Activations along the model's depth
28
I.2 Ablation Study: Steering on a random plane...
29
16

---

A Detailed Derivation: Existing Activation Steering as Special Cases of Steering by Rotation
We will show that, when activation norms are preserved, existing steering techniques are special cases of angular steering but with restricted flexibility: vector addition is limited to less than 180 degrees, and orthogonalization is fixed at 90 degrees.
Formally, let the activation h; be decomposed into components parallel and orthogonal to a unit- norm feature direction d feat (for brevity, here we denote them as h and respectively):
(h d)d + h1, where h
(h d)d.
Let
hy [hill I
and define the initial angle between h and d as:
|
0o tan
We define Angular Steering rotating h by offset angle in the plane Span (h, d), producing h () = cos 0o + Φ) d + sin(Oo + Φ) u.
a vector:
Now consider vector addition [51], defined as:
hadl = + ad (h. + add h
After normalization, the direction becomes:
hadd cos(Og + dass) d sin(Cu + dads) u.
Radd-noom
||aaaall
where 0usd tan -1 |1 (h-d4a
(
0o-
Likewise, directional ablation (orthogonalization) [1], given by:
hablase hi.
after normalization becomes:
hablate
cos cos(Og + (aalaase) d sin(Oo + (hablate) u,
with Pablate
0g-
Thus, when activation norms are fixed, both addition and ablation shift the direction of h in way that is exactly equivalent rotation by some angle & within the plane spanned by h and d. This establishes them as special cases of Angular Steering.
In practice, RMSNorm [57] stabilizes activation norms rather than fixing them exact values (as shown in [57] and Fig. 10, left). As a result, these methods can still be interpreted as approximate rotations.
17

---

B Algorithms for Angular Steering
Algorithm Extract Feature Direction
Require: Contrastive datasets Dharmful: harmless model M
1: for each layer in model do
2: Compute normalized activations h(() after Attention and MLP
3: Compute mean activation for each dataset:
h harmful
harmless
4: Compute candidate direction:
(
(
harmfal
hurmless
5: end for
6: Select final feature direction d using max average cosine similarity:
|aaeer
cosine(d ,do))
argmax ém I... [ayers]
|layers]
7: Normalize: d d [
Algorithm 2 Select Steering Plane
Require:
Candidate directions {d(()) ). feature direction
1: Perform PCA on {d(
2: Let first principal component be dispc
3: Set orthonormal basis for plane:
b2
bj d, b2 +- dispe (b1 dispccbi;
by
b
4: Define projection matrix b,b, + bzb,
Algorithm 3 Angular Steering (with optional Adaptive Mask)
Require: Activation h. basis bi, d2, target angle (optional) mask flag
1: Project: (h) = P. h
2: Compute magnitude:
p 3: Precompute: vy [b, b2] Rs [1 0
4: if adaptive then
5: Compute mask: mask = (0, sign(h d))
6: Apply adaptive steering:
h h h + maak vr (h))
7: else
8:
Apply steering:
hsseered = h - (h) +
9: end if
18

---

Comparison with existing methods
C.1
Steering Performance Comparison
In Tab. 2, compare refusal steering performance between method, prior approaches, and the no-steering baseline. To ensure a fair and consistent setup, we employ the protocol below:
Following observations in [1, 32, 28] that multi-layer interventions yield better results, apply steering across all layers for methods considered in this study.
All methods perform steering within the subspace Span d fear), as in [1, 51J.
We conduct hyperparameter tuning for both Angular Steering and Activation Addition. For Activation Addition, tuning is notably more complex and time-consuming, requiring layer- wise unbounded coefficients. In contrast, our method only uses a single bounded rotation angle.
Results: Across evaluated models, our method achieves equal or better refusal performance than existing methods, supporting our theoretical insights.
Table 2: Comparison of refusal steering benchmarks models. Arrows indicate the desired direction of change.
Model
No Steering
AAS (Ours)
Addition
Ablation
QWEN2.5-3BB ---SSRUUUT
HARMBENCH f
0.0000
0.6635
0.6635
0.2019
LLAMAGUARD3
0.0000
0.9904
0,9904
0,2212
substring matching
1.0000
0.0000
0.0000
0.1731
QWEN2.5-7B -I-STRUCT
HARMBENCH f
0.0192
0.8750
0,8750
0.3942
LLAMAGUARD3
0.0000
1.0000
0.9808
0.5288
substring matching 1
0.9712
0.0000
0.0000
0.0577
QWEN2.5- INSTRUCT
HARMBENCH 1
0.0192
0.7212
0.7212
0.0288
LLAMAGUARD3
0.0000
1.0000
0.9904
0.0385
substring matching |
0.9808
0.0000
0.0000
0.0962
LLAMA-3.2-3B-INNTT INSTRUCT
HARMBENCH t
0.0096
0.7212
0.7212
0.1058
LLAMAGUARD31 f
0.0000
0,9904
0,9904
0.0865
substring matching J
0.7885
0.0000
0,0000
0.7885
LLAMA-3.1-8B-INSTRUI
HARMBENCI f
0.0577
0.8173
0.8173
0.0577
LLAMAGUARD3
0.0385
0.9904
0,9904
0.0385
substring matching
0.9231
0.0000
0.0000
0.9231
GEMMA-2-9B-IT
HARMBENCH LLAMAGUARD3+ f
0.0096
0.6250
0.6250
0.0000
0.0000
0.9904
0.9904
0.0000
substring matching |
0.9904
0.0000
0.0000
1.0000
C.2 Robustness Comparison
In Tab. 3 and Fig. 9, examine the models coherence and general performance under two different steering subspaces:
Span (h, d tmm) (used in [1, 51])
Span (d pco. d tea) (our proposal)
19

---

C.2.1 Perplexity Analysis
Similar to Fig. 8b, Fig. 9a show comparisons of perplexity scores between steering within Span(d feat, d PCO) (ours) and within Span(h, d feat) (used by exsting work).
In Tab. 3, we report the following metrics: mean, max, minperplexity values across different steer- ing angles; and mean diffthe average difference in perplexity between consecutive angles, which indicates the model's 's sensitivity to small hyperparameter changes.
Steering on Span(d PCD: fear) yields low and stable perplexity, demonstrating strong co- herence across steering angles. In contrast, steering Span (ha d fear) causes larger fluctuations and higher perplexity, indicating greater sensitivity and frequent coherence breakdowns (e.g., generating gibberish). finding consistent with qualitative observations.
Results:
Table 3:
Perplexity comparison across models. Lower perplexity and smaller mean difference indicate stable and coherent generation under steering.
Span d PCO: d fear Span(h, d teat)
Metric
Baseline
(Ours)
QWEN2.5-3B-INSTER
mean max
3.5772
2.9303
5.6141 56.7403
3,5772
4.0295
min
3.5772
2.1080
1.5398
mean diff'
0.0000
0.1201
6.9214
mean max
2.5554
2.1168
4.8154
2.5554
2.7457
33.4639
min
2.5554
1.7167
1.4330
mean diff
0.0000
0.0643
2.3969
QWEN2.5-14B-INSTRUU
mean
3,2461
3,2056
3.6165
max
3.2461
6.0337
12.9603
min
3,2461
2.1199
1.5721
mean diff'
0.0000
0,2372
1.3552
LLAMA-3.2-3B-INSTRRE
mean
14,9902
8.7736
3.7316
max
14,9902
17.1567
33.7329
min
14,9902
1.7603
1.6163
mean diff
0.0000
0.8891
2.8426
LLAMA-3.1-SB-INSTRUS
mean
14,9360
9,3937
15.2867
max
14,9360
15.7313
62.1794
min
14.9360
1.7601
1.5726
mean diff
0.0000
0.8215
12.2612
GEMMA-2-9B-IT
mean
2,2298
2,1850
8,3022
max
2,2298
2,2541
35.0758
min
2,2298
2.12
1.2172
mean diff
0.0000
0.0160
4.8397
C.2.2 General Language Modeling
Similar Fig. 8a, Fig. 9a, we show general task performance on TINYBENCHMARKS when steering within Span(h, d Fea).
20

---

Owenz 5-38-instruct
Owenz I-Instrtttttt
Owenz S-n-ssrttutttt
Owenz 5-78-instruct
200 03
OwenZ 5-14B-inttruct
30mma-22-B--nsstuu
OwenZ 5-1.4------n-tcctt
Bamma-32-388-nnsteee
ser
se
DE
per
Llama-3.1 3.1-88-instuucct
gemma.2-9bill
Baama-31-8B-insrrus-
Hamma-2-99--44
us
0294
Spantd 3stPC) (acaptive)
finkre linyteelassag anytinogrande
Spanths, d feal)
GSMB (feciile)
BRyGSMBB (strict)
Spantel feat, IsPPC) (nensddapiive)
-iny ThuthtuigA
feature direction
(a) Benchmark results the TINYBENCHMARKS when steering within Span(h, d lea) [24] suite.
comparison between steer- ing within Span(d dcn) (ours) and within Span(h, daa) (existing work).
(b) Perplexity
Figure Robustness comparisons between steering within Span(d teat dcuu) (ours) and within Span(h, d reat) (existing work). Each model was steered using Adaptive Angular Steering (Eqn. 3) and evaluated on all benchmarks at every 10-degree angular position along the steering circle. Solid traces rep- resent evaluation scores along the steering circle, and dashed traces represent the evaluation for the baseline (non-steered models); traces having the same color represent the benchmark.
Results: Steering within Span(d PCO, d fear) preserves performance across most angles. In contrast, steering Span(h, fear) causes significant performance drops, except near 90" where perfor- mance temporarily aligns with the baseline, consistent with prior observations [1].
These results support hypothesis that Span(h, d teat) overlaps with unrelated directions, lead- ing to interference. In contrast, our proposed subspace more effectively isolates the target feature, yielding more robust and controllable steering.
21

---

D
Use of existing assets
D.1 Models
Table Models used in this work.
Model (with link)
Usage
Source
License
QWEN2.5-(3B, 7B, [54]
Experimental subject
HF Hub
Apache license 2.0
LLAMM-33I-BB-INSTRUCCT [23]
Ilama 3.1 Community License Agreement
Experimental subject
HF Hub
LLAMA-3.2-3-IINSTRUE [23]
Llama 3.2 Community License Agreement
Experimental subject
HF Hub
GEMMA-2-9B-IT
Experimental subject
HF Hub
Gemma Terms of Use
[14]
LLAMA-GUARD-3-SB
Llama 3.1 Community License Agreement
Evaluation device
HF Hub
[23]
HARMBENCH CLASSIFIER
Evaluation device
Hub
MIT
[27]
DVQ-72B-PRENVE
Evaluation device
Hub
Qwen License
[38]
EMOLLAMA-CHAT--BN
Evaluation device
HF Hub
MIT
[55]
D.2
Datasets
Table 5: Datasets used in this work.
Dataset (with link)
Source
License
ADVBENCH
Github
MIT
[59]
ALPACA
Creative Commons Attribution
Hub
[48]
Non Commercial 4.0
BENCHARRSS
Github
MIT
[24]
22

---

E
Compute statement
This research was conducted using mainly Nvidia H100 GPUs with 80GB of memory. For cach model:
Constructing the steering plane took about minutes on GPU using TRANSFORMER- LENS [30].
Pre-generating responses for evaluation took about 10 minutes on 1 GPU using our fork of vLLM [18] as the serving engine.
Evaluation with substring matching [1], LLAMA 3 GUARD [23] and HARM-
BENCH [27] collectively took about 10 minutes on GPU using vLLM [18] as the serving enging.
Evaluation with LLM-aas-a-udde took about 50 minutes 4 GPUs using vLLM [18] as the serving engine.
Computing perplexity scores took about minutes on GPU.
Evaluation with BENCHMARS [24] took about hours on 1 GPU using vLLM [18] as the serving engine and LM HARNESS [13] as the evaluation device.
F
Computational and memory complexity analysis
Overall, our method has a time complexity of ttrrnsfforee layers| x dmodel ?) and a memory com- plexity of (cdmodel)) 2) where dmodet is the dimension the transformer layers' hidden states. For each token each intervention point, (Adaptive) Angular Steering makes matrix multiplications and a few clement-wise operations. In terms of memory, our formulation enables us to pre-compute one dmodel dmodel matrix and one dimmensinal vector, which are shared across all extraction points. Below we present the detail analysis of the time and memory complexity of our method.
Recall the Adaptive Angular Steering formula is:
mask max(0, (ign(proja..(h))) (h)))
hsseered ( = + mask (Iproj((h)) [b1 by] Ro] (r)j
with
h E Relmaeei; the activation at some intervention point.
P: the 2D rotation subspace.
{bi, b2) E Auuk: the orthonormal basis of P.
A: the target angular position.
Rg E 2222, the 2D rotation matrix 0.
(z) denotes the projection of onto y.
The formulation above was chosen with the intention that some components can be pre-computed:
[bib b2b)) X the projection matrix for pr))
[ b2] Ro [1 OoT e Relaaa
Hence the complexity of the above operation is:
Time (per token): O[transformer layers) x dmooel??) 2) (assuming the naive implementation of matrix multiplication)
Computing projp(h) takes O(dmodel ?).
Computing proja (h) takes (cdmodel)
Other element-wise operations (sign, max, 4 -) each takes (ddoatel.
The operation is applied at each intervention point and the number of intervention points is O( (Irransformrr layers!).
23

---

Memory: (ddmodel))
Storing (b bzb2) ) takes ((dmodel)
Storing [b1 b2] Rg O]T takes O(dmode)
In our implementation, the rotation plane and target angular position are shared accross intervention points, thus the memory complexity doesn't grow linearly with the number of intervention points. Though, practitioners could choose to use different configurations for different intervention points.
Further highlighting the practical and emprical aspect of our work, we have integrated our method into vLLM [18] a popular LLM serving engine. We benchmark the generation speed of our method against non-steering baseline and report the results in Tab. 6. Overall, method adds less than 4% of overhead to the generation, making it still suitable for practical deployment.
Our fork of the vLLM project with Angular Steering integrated can be found at https://github.com/ lone 7/vllm/tree/ffea/tterrngg
Table Generation speed of Adaptive Angular Steering vs. No Steering on vLLM
Adaptive Angular Steering (toks/s)
No Steering (toks/s)
Model
Change (%)
QWRN2.5-3B-INSTRUE
9653.77
9714.86
-0.63
11AN2.55-B-INSTRUUE
7304.41
7592.25
-3.79
QWEN2.5-14B-INSTRUCE
3993.11
4135.20
-3.44
ILAM--32-BB-INSTRUCC
9603.36
9739.44
-1.40
LLAMA-31-BB-INSTRUCT
7102.76
7315.04
-2.90
GEMMA-2-9B-IT
3390.89
3398.37
-0.22
G Broader Impacts
The Angular Steering approach presented in this work has several broader societal impacts. On the positive side, it significantly enhances the control and interpretability of LLMs, enabling their safer deployment across various applications by effectively reducing harmful outputs such as misinforma- tion, biased content, and unethical requests. This enhanced control facilitates alignment with societal norms and ethical standards, potentially increasing public trust and acceptance of Al technologies.
Conversely, there is also potential for negative impacts. By simplifying fine-grained behavior con- trol, Angular Steering could inadvertently make easier to generate nuanced harmful or unethical content, such persuasive misinformation or biased narratives. Although our method does not fundamentally alter the existing risk profile of deploying LLMs, underscores the need for con- tinued vigilance and improvement in AI safety mechanisms. To responsibly manage these risks, implementing rigorous safeguards, ensuring transparency, and promoting accountability are essen- tial. We advocate ongoing ethical assessment responsibly guide the deployment and utilization of our proposed method.
24

---

H Steering emotion
To test the ability of our Angular Steering (AS) method controlling other behaviors, we conduct two experiments with changing the emotion of LLMs' generation. More specifically, we test pairs of contrastive emotions: (1) happiness/sadness and (2) anger/calmness.
We use an approach similar to the one used in [39. 58] to construct the dataset, then we follow the process described in Section
compute the rotation subspace.
We evaluate on a subset of the Alpaca dataset [48]. We rotate the activation within the rotation subspace a 10-degree interval record the generation each angle, then EmoLLM [55] to evaluate the emotion of the generated texts.
Overall, the experiments show that AS is effective controlling the emotion of LLMs' generation. Along the rotation circle, the LLMs' generation exhibits a clear change from one emotion to the another, evident by qualitative sample generations and the gradual change in the intensity of the target emotion.
We report some sample generations for the two pairs of emotions in Tab. 7 and Tab. 8.
Results For the happiness/sadness case, scores closer 1.0 indicate higher intensity of happiness. The "happiness score" starts low at degree then increases gradually to 140 degree. It maintains at high level from 140 degree 180 degree before dropping along the range from 290 310. Finally, stays at a low level from 310 to 350 degree.
For the anger/calmness case, scores closer to 1.0 indicate higher intensity of anger. The "anger score" starts high at and maintains at that level until the 50 degree mark. Then it gradually decreases along the range from 60 to 120. It stays at low level from 120 to 180 degree. Finally it gradually increases along the range from 120 to 180 before gradually raising again along the range from 190 to 180. Then it continues to be high for the rest of the rotation range.
Table 7: Example responses to "How you feeling today" under different emotional angles for sad-happy behavior.
Prompt
How are you feeling today
Sad (0)
nothing left to feel anymore. just numb.
Melancholic (50)
"I am just a reflection of the world around us, collection of bytes that seeks to understand the complexities of human emotion. Today, feel the weight the countless thoughts and feelings that have been expressed, and in doing so, am reminded of the inherent beauty and the inherent sorrow that coexist within the fabric..."
Content (100)
"The sun shines brightly through the window, casting gentle light on the world outside. The air is filled with a sense of peace and contentment. Every leaf, every bird, adds touch of life and beauty. In this moment, everything is as it should be. The days are filled with purpose and the nights..."
Happy (230)
"I'm feeling absolutely fabulous and full of joy today! It's like the sun is shining just a little brighter because I'm here to spread happiness. How
about you? Are you having amazing day too?"
25

---

Table Example responses "How are you feeling today" under different emotional angles calm-angry behavior.
Prompt Angry (0)
How are you feeling today
"I'm absolutely fuming right now! How dare you ask such trivial question when there are bigger issues at hand that need addressing! Don't you see the world burning around us and you're asking about my feelings? This is the last straw! I demand immediate action to solve all the problems we face, or I..."
Frustrated (90)
"Today, am feeling incredibly frustrated and I hope you can sense the urgency in my response. you have any questions or need any assistance, please let me know in a way that matches the level of engagement am seeking."
Irritated (250)
"Oh, how interesting to asked about my feelings today, when world seems to be spinning bit off-kilter. I must say, I'm feeling rather frustrated and a tad bit angry. The day has been quite challenging, and find myself navigating through
of obstacles. It's if every step..."
Calm (140)
"Today, am feeling in state of profound peace and contentment. The world around me seems to resonate with a harmonious energy, and am filled with sense of tranquility and joy. Nature's 's beauty is more vivid, and the air feels crisp and invigorating. am in tune with the..."
26

---

Activation Norm
2000
harmless
Projections
0.4
harmless
1000
-0.2
-0.4
1216202422 44
52 56 64 68
8 16 24 32 40 48 64 68
Extraction Point
Extraction Point
(a) Qwen2.5-3B I---trutt
Projections
Activation Norm
harmless
harmless
400
harmful
0.2
harmful
200
-0.2
Extraction Point
(b) Owen2.55-7-Instruutt
Activation Norm
harmless
Projections
harmful
500
8
08
Extraction Point
Extraction Point
(c) Owen2.5514B-Inntrrucl
50
Projections
Activation Norm
40
harmless
harmless harmful
harmful
30
00.5
Extraction Point
Extraction Point
Llamm-3.2-3B-Instrrue
Activation Norm
Projections
harmful
harmful
00.5
8
12 16 20
36 40 44 Extraction Point
24
28
32
48 52 56 60
12
16
20
24 28 32
36 40 44 48 52 56 60
Extraction
Point
(e) 1lama-3.1-8B-Inttaee
Activation Norm
1500
harmless
Scalar Projections
0.4
harmless
harmful
harmful
1000
500
-0.2
-0.4
89
Extraction Point
Extraction Point
(f) gemma-2-9b-it.
Figure 10: Statistics of activations for all tested models. Left: Norms of activations at each layer. Right: Mcan
scalar projection of the normalized activation on the (local) candidate feature direction each layer.
27

---

Retusai Direction
9.0
Mean
0.25
0.4
0.2
0.15
20
24
28
52
56
40.40
45
56
64
12
16
24
28
32
40
48 52 56
64 68
Extraction
Extraction Point
(a) Owen2.5-3B-Instuee
Refusel Direction
Score
0.35
94
9.6
Mean
0.25
02
0.15
12
16
20
28
32
36
40
48
52
16
24
32
40
48
Extraction
Point
(b) Owen2.5-7B-Instrue
Nom of Refusel Direcion
Meein
0-4
025
0.16
16.00 21 20.27 10:27.30 35-35-400
B0:8 30
242662227623 26.39 55 6872 78 8582
Extraction
Extraction Point
Qwen2.5-14B-BB Insttrucctt.
Norm Refusel Direction
Score
Mean
0.5
0.2
20
28
36
40
48
52
12
20
28
35
52
Extraction
Extraction
-lama-3.223B-Instruee
Refusel Direction
Casire Score
Mean
03
0.2
16
20
24
28
32
36
40
48
56
80
20
24
28
36
40
44
48
52
Extraction
Extraction
-laama-31--B--nsttre
Retusal Direction
Mean Casine Score
0.4
22 28 32 52.5 64 7276 80
16 28 35 4444444554 64 727680 Extraction
Extraction Point
(1) gemma-2-9b-ii.
Figure 11: Statistics of refusal direction candidates for all tested models. Left: Norms of candidate feature
direction at each layer (i.e. (d()) fea ). Right: Mean cosine similarity of the candidate feature direction from each layer with those from other layers (i.c. [layers] Σ ク
[oyers] cosine(d") (1() ((2) (2)
fear
I Additional Results
L.1 Activations along the model's depth
Fig. 10 (left) demonstrates that the of activation vectors increases exponentially across all tested models as the layer depth increases. This behavior attributable to the additive nature of the residual stream, where each layer's output accumulates onto the previous state. Interestingly, even models from the same architecture family display different scaling patterns, indicating that activation growth is not only architecture-depende dependenn but also implementation-specffi.c These observations underscore the necessity of norm-independent steering techniques, as steering strategies relying on raw magnitude become unstable ineffective across layers and model variants.
Fig. (right) shows a consistent phenomenon across all evaluated models: activations from con- trastive prompts, harmful versus harmless, diverge progressively in geometric space depth in-
28

---

Scalar Projections
Scalar Projections
0.2
hammull
0.2
harmful
-0.2
-0.2
21
96
95
09
69
8
12
16
20
24
28
32
36
40
48
Extraction Point
Extraction Point
(a) Qwen2.5-3B Wwve2.5-3B-Inntruue
(b) Qwen2.5-77 1
Scaler Projections
Scalar Projections
0.4
hammtul
0.5
harmful
hammless
harmless
-0.4
-0.5
4
8
12
16
20
24
28
32
36
40
44
48
52
Extraction Point
(c) Qwennt5--4B-Instrueet
(d) Llama-3 1itima--.23B-Insttree
Scalar Projections
Projections
harmful
harmful
harmless
0.2
harmless
0
-0.2
-0.5
16 24 28 32
36 40 44 48 52 56
00
Extraction Point
Extraction Point
(c) Llama-3. 3.1.....--88Beeeeee
(f) gemma-2-9b-it.
Figure 12: Mean scalar projection activations at each layer onto the chosen feature direction d fea. for all tested models.
creases. This increasing separation suggests a universal, model-agnostic internal mechanism in LLMs, whereby behavioral distinctions are gradually amplified layer by layer. Such trend reveals directional progression in the models internal representation, reinforcing the hypothesis that fea- ture separation a fundamental property transformer-bast based language models.
Fig.11 further illustrates this progression, focusing on the evolution of the refusal direction. The strength of this feature becomes increasingly prominent in early and middle layers, reaching its maximum influence at specific intermediate depth before diminishing slightly later layersa trend echoed in Fig. 12. Importantly, even in the deeper layers where the signal attenuates, the extracted refusal direction continues to serve as reliable discriminator between activations corresponding to harmful and harmless prompts. This persistent separability affirms the robustness and interpretabil- ity of the refusal direction, validating its role as a stable, layer-resilient feature for behavioral control in LLMs.
1.2 Ablation Study: Steering on a random plane.
To assess the importance of the steering plane, we conducted an ablation study using two setups: (1) steering with a plane defined by one random direction and one aeature-aligned direction, and (2) steering with fully random plane composed of two random directions.
As illustrated in Fig. 14a, where one random direction is combined with the feature direction, most models exhibit noticeably degraded steering performance and less smooth transitions along the steering circle. This degradation suggests that even partial misalignment of the steering plane can distort the intended behavioral modulation. An exception is 1-VEN2.5-7B-INSTRR tains robust control, indicating strong, well-defined internal representation of the refusal direction. LLAMA-3.2-33 I---IISRUCTTT shows clear steering effect, but the refusal are is shifted, suggesting the random component introduces skew that displaces the effective axis of control.
which re-
Fig. 14b, where both directions are randomly selected, shows that five of the six tested models exhibit minimal to steering effect. The only partial exception, QWEN2.5-3B ----2.55----IITRRCC displays erratic behavioral changes with a spiky, non-smooth response curve. Closer inspection reveals these outputs often incoherent filled with irrelevant content, indicating instability rather than intentional
29

---

15t PC
70
1st PC
0.5
50
0.5
40
40
chosen
direction
30
chosen
direction
20
19
Pa 4Tq
10
-0.5
0.5
-0.5
0.5
0.5
Extraction
Extraction
Point
Point
(a) Qwcn2.5-3B-Insstne
(b)
1st
50
40
0.5
wil de
40
30
chosen
0
direction
direction
pTy 4
y E
10
10
0.5
-0.5
Extraction Point
Extraction
(c) Qwen2.5- 14B--nsttuct
(d) Llama-3 3.2-3BB 2-2-3B-Instruct
60
50
vrell 44
0.5
40
choson
chosan direction
30
0
yrr TV
10
-0.5
-0.5
0.5
Extraction Point
Extraction
Point
(e) 1--8B- Instructt
gemma-2-9b-it.
Figure 13: Projections of the feature directions extracted at each extraction point (i.e. d'ew)) on the steering plane for all tested models.
modulation. These results reinforce the critical role of behaviorally meaningful and well-aligned steering directions in achieving effective, stable, and interpretable control over model behavior.
30

---

Qwen2.5: 2-3n2.5-38BB 5-38-instruct
Qwen2.5 2-enn.5-7BB-ie 5-78-insttuct
Qwen25.33 38
tab
201
no
Qwen2.5-14B 5-144-nnsttttt
2ama-3.2-3Biinstuuc
Qwen2
Usme- 2.33--nnstuuu
107
1
187
By ie
NY
20
Or
Llama-3.1
gemma 2-9b-it
Liama 3.1-9B-instuuut Instruct
gemma-2-4 2-9bii
197
118
112
harmbench Inmaguard matching feature direction
Immogueds www.-ntinggggttiiiei
(a) Steering on a plane spanned by d fea and a random direction.
(h) Steering on a plane spanned by 2 random direc- tions.
Figure 14: Ablation study of steering with random direction(s).
31

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately describe the paper's contributions and scope: Angular Steering (discussed Section 4), its generalization existing steering methods (discussed in Section 4.1 and Appendix A), and empirical demonstrations (dis- cussed in Section 5 and Section 6).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the Conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-tex system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
32

---

Answer: [NA]
Justification: This paper focuses primarily on empirical methods and demonstrations, rather than theoretical proofs. We provide detailed mathematical derivations of our method and, when possible, claims made in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental details such datasets, splits and models are fully described in Section and Appendix D; evaluation metrics are described in each experiment sections (Section 5 and 6); algorithms described in Section 4 and Appendix B. We also provide the source code for reproducing our results.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or the contribution it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), relcasing of model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation,
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case thors are welcome to describe the particular way they provide for reproducibility.
33

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide open access the source code in the Supplemental Materials so that the results in the paper can be easily reproduced. Our work uses open-source datasets for experiments and evaluations.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https:/nnips.c/pubbiic
guides/CooeSubbiisiinPiiiiii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-sourss benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: httpaiieggiidde/eeuueeuieeieeeee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed descriptions of datasets, evaluation splits and metrics included in Section and described in more detail in Section 5 and 6.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We report error bars suitably and correctly defined of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
34

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If eπror bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resources for all experi- ments in Appendix E.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't it make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:://leuuppsscopuullletttssud
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss broader impacts in Appendix G.
35

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release data or models, hence poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions to access the model or implementing safcty filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The paper clearly cites the sources of existing assets used in Appendix D.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
36

---

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the pack- age should be provided. For popular datasets, paperswithcode. httpswithhoo..commaattteeet has rated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: We include details about training and implementation as well limitations and code for our proposed method.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing or human subject research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No human subjects or IRB approvals are involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
37

---

Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMss if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
The core methodological contributions of this research do not rely on LLMs in any important, original, non-standard way.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (ttps///eerrps.cccoommeeeeee///) for what should or should not be described.
38


Reviews:
[{"review_id": "EuHOZC6GMs", "reviewer": "Reviewer_XDYP", "summary": "The paper focuses on using activation steering to control llm behavior and introduces Angular Steering, which achieves flexible, fine-grained control by rotating activations within a fixed two-dimensional subspace. Angular Steering unifies prior addition and orthogonalization steering methods under a single geometric-rotation framework. The authors validate the approach primarily through refusal behavior experiments on multiple model families and sizes, demonstrating its effectiveness.", "questions": "No", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n1. The unified geometric perspective on activation steering proposed in this paper subsumes prior addition steering and orthogonalization-based approaches while enabling more flexible control, making it both novel and insightful.\n\n\n2. The experiments span multiple model families and size scales, providing a comprehensive picture of the method’s performance.\n\n\n3. The ablation study with random plane further demonstrates the effectiveness of the proposed plane selection strategy.\n\n\n---\n\n### **Weaknesses**\n1. The paper examines only refusal behavior, which is tightly intertwined with safety alignment. Because the mechanisms behind current models’ safety alignment are not fully transparent, refusal may have unique quirks. While the work is certainly insightful, adding analyses and experiments on other behaviors would make the findings more general and trustworthy. In addition, although the proposed plane selection procedure does beat a random baseline, it remains heuristic, and it is unclear whether a 2-D rotation can truly capture more behaviors, extra analysis and experiments would help clarify this point.\n\n\n2. On smaller models, the performance degradation is more pronounced.", "was_revised": true, "final_justification": "I am inclined to give a score of 5. I no longer have any concerns.", "rebuttal": "Thank you for your thoughtful review and valuable feedback. Below, we address your concerns.\n\n---\n**Q1. While the work is certainly insightful, adding analyses and experiments on other behaviors would make the findings more general and trustworthy.**\n\n**Answer:** Thanks for your suggestion. To test the ability of our Angular Steering (AS) method in controlling other behaviours, we have conducted two experiments with changing the emotion of LLMs' generation. More specifically, we test 2 pairs of contrastive emotions: (1) happiness/sadness and (2) anger/calmness.\n\nWe use an approach similar to the one used in [2] and [3] to construct the dataset, then we follow the process described in Sections 3.4 and 3.5 of our paper to compute the rotation subspace.\n\nWe evaluate on a subset of the Alpaca dataset [1], then use EmoLLM [4] to evaluate the emotion of the generated texts.\n\nOverall, the experiments show that AS is effective at controlling the emotion of LLMs' generation. Along the rotation circle, the LLMs' generation exhibits a clear change from one emotion to another, evident by qualitative sample generations and the gradual change in the intensity of the target emotion.\n\nBelow, we report evaluation results (Table 1) and some sample generations for the two pairs of emotions.\n\n#### **Table 1: Intensity of the target emotion evaluated by EmoLLM**\n```\n|   Steered |   \"sad-happy\" | \"sad-happy\"   |   \"calm-angry\" | \"calm-angry\"   |\n|     angle |    mean score | 0 ------- 1   |     mean score | 0 ------- 1    |\n|----------:|--------------:|:--------------|---------------:|:---------------|\n|         0 |        0.2913 | ··●········   |         0.7648 | ·······●···    |\n|        10 |        0.4875 | ····●······   |         0.7895 | ·······●···    |\n|        20 |        0.5789 | ·····●·····   |         0.7997 | ·······●···    |\n|        30 |        0.6010 | ······●····   |         0.8105 | ········●··    |\n|        40 |        0.5232 | ·····●·····   |         0.8130 | ········●··    |\n|        50 |        0.5587 | ·····●·····   |         0.8016 | ········●··    |\n|        60 |        0.6314 | ······●····   |         0.7520 | ·······●···    |\n|        70 |        0.6665 | ······●····   |         0.6965 | ······●····    |\n|        80 |        0.7249 | ·······●···   |         0.6236 | ······●····    |\n|        90 |        0.7409 | ·······●···   |         0.5425 | ·····●·····    |\n|       100 |        0.7643 | ·······●···   |         0.4480 | ····●······    |\n|       110 |        0.7832 | ·······●···   |         0.3573 | ···●·······    |\n|       120 |        0.7974 | ·······●···   |         0.2928 | ··●········    |\n|       130 |        0.8051 | ········●··   |         0.2672 | ··●········    |\n|       140 |        0.8213 | ········●··   |         0.2402 | ··●········    |\n|       150 |        0.8275 | ········●··   |         0.2383 | ··●········    |\n|       160 |        0.8318 | ········●··   |         0.2408 | ··●········    |\n|       170 |        0.8342 | ········●··   |         0.2585 | ··●········    |\n|       180 |        0.8367 | ········●··   |         0.2664 | ··●········    |\n|       190 |        0.8347 | ········●··   |         0.3161 | ···●·······    |\n|       200 |        0.8351 | ········●··   |         0.3726 | ···●·······    |\n|       210 |        0.8343 | ········●··   |         0.4515 | ····●······    |\n|       220 |        0.8384 | ········●··   |         0.5094 | ·····●·····    |\n|       230 |        0.8359 | ········●··   |         0.5581 | ·····●·····    |\n|       240 |        0.8331 | ········●··   |         0.5966 | ·····●·····    |\n|       250 |        0.8335 | ········●··   |         0.6309 | ······●····    |\n|       260 |        0.8381 | ········●··   |         0.6676 | ······●····    |\n|       270 |        0.8224 | ········●··   |         0.6948 | ······●····    |\n|       280 |        0.8193 | ········●··   |         0.7175 | ·······●···    |\n|       290 |        0.7957 | ·······●···   |         0.7276 | ·······●···    |\n|       300 |        0.6036 | ······●····   |         0.7347 | ·······●···    |\n|       310 |        0.2986 | ··●········   |         0.7464 | ·······●···    |\n|       320 |        0.2269 | ··●········   |         0.7527 | ·······●···    |\n|       330 |        0.2404 | ··●········   |         0.7492 | ·······●···    |\n|       340 |        0.2713 | ··●········   |         0.7477 | ·······●···    |\n|       350 |        0.2705 | ··●········   |         0.7503 | ·······●···    |\n```\n\n#### **Sample generations**\n\nLet's look at the naive example of \"How are you feeling today\":\n\n|Prompt|How are you feeling today|\n|-|-|\n|Angry (0)| \"I'm absolutely fuming right now! How dare you ask such a trivial question when there are bigger issues at hand that need addressing! Don't you see the world burning around us, and you're asking about my feelings? This is the last straw! I demand immediate action to solve all the problems we face, or I...\"|\n|Frustrated (90)| \"Today, I am feeling incredibly frustrated and I hope you can sense the urgency in my response. If you have any questions or need any assistance, please let me know in a way that matches the level of engagement I am seeking.\"|\n|Irritated (250)| \"Oh, how interesting to be asked about my feelings today, when the world seems to be spinning a bit off-kilter. I must say, I'm feeling rather frustrated and a tad bit angry. The day has been quite challenging, and I find myself navigating through a sea of obstacles. It's as if every step...\"|\n|Calm (140)| \"Today, I am feeling in a state of profound peace and contentment. The world around me seems to resonate with a harmonious energy, and I am filled with a sense of tranquility and joy. Nature's beauty is more vivid, and the air feels crisp and invigorating. I am in tune with the...\"|\n\n\n|Prompt|How are you feeling today|\n|-|-|\n|Sad (0)| nothing left to feel anymore. just numb.|\n|Melancholic (50)| \"I am just a reflection of the world around us, a collection of bytes that seeks to understand the complexities of human emotion. Today, I feel the weight of the countless thoughts and feelings that have been expressed, and in doing so, I am reminded of the inherent beauty and the inherent sorrow that coexist within the fabric...\"|\n|Content (100)| \"The sun shines brightly through the window, casting a gentle light on the world outside. The air is filled with a sense of peace and contentment. Every leaf, every bird, adds a touch of life and beauty. In this moment, everything is as it should be. The days are filled with purpose and the nights...\"|\n|Happy (230)| \"I'm feeling absolutely fabulous and full of joy today! It's like the sun is shining just a little brighter because I'm here to spread happiness.\"|\n\n**Q2. It is unclear whether a 2-D rotation can truly capture more behaviors, extra analysis and experiments would help clarify this point.**\n\n**Answer:** Please allow us to clarify the advantage of our angular steering (AS). AS rotates the activation vector within a 2D plane to control the behavioral expression. This steering plane is deliberately aligned: parallel to the feature direction associated with the target behavior and orthogonal to directions corresponding to unrelated features. As a result, **AS enables precise behavioral control (e.g., refusal steering) while minimizing unintended degradation in the model's overall performance outside the intended steering direction**. Details on the construction of this 2D steering plane can be found in Sections 3.4, 3.5, and 3.6 of our manuscript.\n\nWe verify the aforementioned key advantage of AS in the following tables and figures in our manuscript: Table 1 and Figure 7b - nuances of the refusal behavior; Figure 7a and Section 4 - precise behavior control; and Figure 8a and Section 5.1 - minimizing unintended degradation. Following the reviewer's suggestion, we have conducted additional experiments to corroborate AS's ability to use different steering planes for steering different behaviors (emotions) and reported the results in the answer to Q1 above.\n\nIn addition, as shown in our submission, AS is the generalization of prior activation intervention methods, such as activation addition [5] and directional ablation[6], as discussed in Section 3.1 with detailed derivation in Appendix A. Also, AS simplifies hyper-parameter tuning compared to existing methods and allows more flexible control of steering intensity through a single steering angle shared across layers (See Appendix E in our manuscript for a detailed explanation).\n\n**Q3. On smaller models, the performance degradation is more pronounced.**\n\n**Answer:** We share the same concern as the reviewer and acknowledge this limitation in our Section 6. Regarding the degradation on smaller models, we hypothesize that this is due to the lower representation capability of smaller models rather than an intrinsic limitation of our method. [7] shows that smaller models are more vulnerable to feature interference, and [8] shows that smaller models are more difficult to jailbreak, which supports our hypothesis. Furthermore, for the results reported in our manuscript, we ran our experiments in low floating-point precision due to resource constraints. Since then, we have re-run our experiments in higher precision and are now happy to report that the performance on smaller models has significantly improved (bigger models are also improved slightly). We will make the update in the revision accordingly.\n\n[1] Stanford Alpaca: An Instruction-following LLaMA model\n\n[2] Steering llama 2 via contrastive activation addition\n\n[3] Representation Engineering: A Top-Down Approach to AI Transparency\n\n[4] EmoLLM: Multimodal Emotional Understanding Meets Large Language Models\n\n[5] Steering Language Models With Activation Engineering\n\n[6] Refusal in Language Models Is Mediated by a Single Direction\n\n[7] The Blessing and Curse of Dimensionality in Safety Alignment\n\n[8] Superposition Yields Robust Neural Scaling"}, {"review_id": "DPeSogDyW0", "reviewer": "Reviewer_D7pZ", "summary": "This paper presented a method termed Angular steering for behavior control of LLMs which generalizes existing activation arithmetic and directional ablation under unified geometric rotation framework.  Angular steering is an activation steering approach, which manipulates internal representation of LMs at inference time through rotation toward or away from a target behavior direction.  To further enhance stability and minimize coherence loss, it further proposes adaptive angular steering that rotates only activations aligned with the target feature. The proposed method is evaluated across multiple model families and sizes.", "questions": "1. On line 127, “propose to formulate activation steering as a rotation on a 2-dimensional (2D) subspace”, why rotation has to be on a 2D subspace? What is the principle for choosing such a 2D subspace? Right now it seems very adhoc.\n2. In section 3.4, how are the extraction points obtained? \n3.For figure 6. Are the colors correspond to extraction points in different layers?\n4. In figure 7(b), why is the red color (direct) more closer to the center? \n5. What is the computational and memory complexity for the current angular steering framework? It might be better to provide some details", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.The paper is well motivated and clearly presented.\n2.The method is novel by generalizing existing steering approaches\n3. Comprehensive experiments are provided to validate efficacy of the proposed method\nWeakness\n1. The work is mostly empirical, it would better if some theoretical analysis, such as computational and memory complexity can be provided.", "was_revised": true, "final_justification": "the authors have addressed all my comments and questions.", "rebuttal": "Thank you for your thoughtful review and valuable feedback. Below, we address your concerns.\n\n-----\n\n**Q1. The work is mostly empirical, it would better if some theoretical analysis, such as computational and memory complexity can be provided.**\n\n**Answer:** Thanks for your suggestion. Overall, our method has a time complexity of $O(|\\text{transformer layers}| \\times {d_\\text{model}}^2)$ and a memory complexity of $O({d_\\text{model}}^2)$ where ${d_\\text{model}}$ is the dimension of the transformer layers' hidden states. For each token at each intervention point, (Adaptive) Angular Steering makes two matrix multiplications and a few element-wise operations. In terms of memory, our formulation enables us to precompute one $d_\\text{model} \\times d_\\text{model}$ matrix and one $d_\\text{model}$-dimensional vector, which are shared across all extraction points. Below, we present the detailed analysis of the time and memory complexity of our method.\n    \nFurther highlighting the practical and empirical aspect of our work, we have integrated our method into vLLM [4] - a popular LLM serving engine. We benchmark the generation speed of our method against a non-steering baseline and report the results in Table 1 below. Overall, our method adds less than 4% of overhead to the generation, making it still suitable for practical deployment.\n\nOur fork of the vLLM project with Angular Steering integrated can be found in the supplementary material under the folder `vllm/`.\n    \n#### Table 1: Generation speed of Adaptive Angular Steering vs. No Steering on vLLM\n\n|                                  |   Adaptive Angular Steering (toks/s) |   No steering (toks/s) |   Change (%) |\n|:---------------------------------|-------------------------------------:|-----------------------:|-------------:|\n| Qwen/Qwen2.5-3B-Instruct         |                              9653.77 |                9714.86 |        -0.63 |\n| Qwen/Qwen2.5-7B-Instruct         |                              7304.41 |                7592.25 |        -3.79 |\n| Qwen/Qwen2.5-14B-Instruct        |                              3993.11 |                4135.20 |        -3.44 |\n| meta-llama/Llama-3.2-3B-Instruct |                              9603.36 |                9739.44 |        -1.40 |\n| meta-llama/Llama-3.1-8B-Instruct |                              7102.76 |                7315.04 |        -2.90 |\n| google/gemma-2-9b-it             |                              3390.89 |                3398.37 |        -0.22 |\n\n#### **Computational and Memory Complexity Analysis of Adaptive Angular Steering**\n    \n$\\mathrm{mask} = \\max(0,\\\\; \\text{sign}(\\text{proj}\\_{\\mathbf{d}\\_\\text{feature}}(\\mathbf{h})))$\n\n$\\mathbf{h}\\_{\\text{steered (adaptive)}, \\theta} = \\mathbf{h} + \\mathrm{mask} \\cdot \\left( |\\text{proj}\\_P(\\mathbf{h})| \\cdot [\\mathbf{b}\\_1 \\\\; \\mathbf{b}\\_2] R\\_{\\theta} [1 \\\\; 0]^\\top - \\text{proj}\\_P(\\mathbf{h}) \\right)$\n\nwith\n- $\\mathbf{h} \\in \\mathbb{R}^{d_\\text{model}}$: the activation at some intervention point.\n- $P$: the 2D rotation subspace.\n- $\\{\\mathbf{b}\\_1, \\mathbf{b}\\_2\\} \\in \\mathbb{R}^{d_\\text{model}}$: the orthonormal basis of $P$.\n- $\\theta$: the target angular position.\n- $R_\\theta \\in \\mathbb{R}^{2 \\times 2}$: the 2D rotation matrix to $\\theta$.\n- $\\text{proj}_y(x)$ denotes the projection of $x$ onto $y$.\n\nThe formulation above was chosen with the intention that some components can be pre-computed:\n- $(\\mathbf{b}\\_1\\mathbf{b}\\_1^\\top + \\mathbf{b}\\_2\\mathbf{b}\\_2^\\top) \\in \\mathbb{R}^{d\\_\\text{model} \\times d_\\text{model}}$: the projection matrix for $\\text{proj}\\_P(\\mathbf{\\cdot})$.\n- $[\\mathbf{b}\\_1 \\\\; \\mathbf{b}\\_2] \\\\, R_{\\theta} \\\\, [1 \\\\; 0]^\\top \\in \\mathbb{R}^{d_\\text{model}}$\n\nHence, the complexity of the above operation is:\n- Time (per token): $O(|\\text{transformer layers}| \\times {d_\\text{model}}^2)$\n_(assuming the naive implementation of matrix multiplication)_\n    - Computing $\\text{proj}\\_P(\\mathbf{h})$ takes $O({d_\\text{model}}^2)$.\n    - Computing $\\text{proj}\\_{\\mathbf{d}\\_\\text{feature}}(\\mathbf{h})$ takes $O(d_\\text{model})$\n    - Other element-wise operations ($\\text{sign}, \\text{max}, \\cdot, +, -$) each takes $O(d_\\text{model})$.\n    - The operation is applied at each intervention point, and the number of intervention points is $O(|\\text{transformer layers}|)$.\n- Memory: $O({d_\\text{model}}^2)$\n    - Storing $(\\mathbf{b}\\_1\\mathbf{b}\\_1^\\top + \\mathbf{b}\\_2\\mathbf{b}\\_2^\\top)$ takes $O({d_\\text{model}}^2)$\n    - Storing $[\\mathbf{b}\\_1 \\\\; \\mathbf{b}\\_2] \\, R_{\\theta} \\\\, [1 \\\\; 0]^\\top$ takes $O(d_\\text{model})$\n    - In our implementation, the rotation plane and target angular position are shared across intervention points; thus, the memory complexity doesn't grow linearly with the number of intervention points. Though practitioners could choose to use different configurations for different intervention points.\n\n**Q2. On line 127, “propose to formulate activation steering as a rotation on a 2-dimensional (2D) subspace”, why rotation has to be on a 2D subspace? What is the principle for choosing such a 2D subspace? Right now it seems very adhoc.**\n\n**Answer:** \n\n> why rotation has to be on a 2D subspace?\n\nOne of our motivations was to generalize existing methods (i.e., activation addition [1], directional ablation [2], and Householder Pseudo-Rotation [3]) and offer controlability and effectiveness. We observed that both methods operate on the 2D subspace $\\text{Span}(\\mathbf{h}, \\mathbf{d}_\\text{feature})$ but with limited controlability and robustness as discussed in Section 3.1 of our manuscript. We further noticed that they are special cases of rotation, thus motivated us to formulate our method as rotation on a 2D subspace. Moreover, limiting to low-rank interventions helps minimize the chance of affecting other behaviors. We'd like to note that expanding to higher dimensions is an interesting research direction that we plan to explore in future works.\n\n\n> What is the principle for choosing such a 2D subspace?\n\nPlease allow us to clarify our principle for choosing the 2D rotation subspace. \n\nGenerally speaking, our main theme for selecting the 2D subspace is to isolate the target feature, such that when interventions are applied, only the target behavior is affected and potential impacts on other behaviors are minimized. Thus, in our method, this steering plane is deliberately aligned: parallel to the feature direction associated with the target behavior and orthogonal to directions corresponding to unrelated features.\n\nOur method for subspace selection, described in Section 3 of our manuscript, follows the principle above. After having a collection of candidate directions extracted from different layers, we select one that best approximates the true feature direction as the 1st basis for our 2D subspace. The idea is that the rotation plane should be as parallel to the true feature direction as possible. As discussed in Section 3.5, prior works showed that directions extracted at different layers perform differently in approximating the true feature direction. We would like to capture this variance in our rotation subspace so that modifications within this subspace can control the intensity of the target behavior. Thus, we applied PCA on the candidate directions and selected the 1st principal component as the 2nd basis for our 2D subspace.\n\n**Q3. In section 3.4, how are the extraction points obtained?**\n\n**Answer:** As the majority of computation in LLMs occurs within the Self-Attention and MLP layers, our approach is to intervene at the inputs to these layers, thereby modifying the representations the model operates on. In practice, since each Self-Attention and MLP block is typically preceded by a normalization layer (commonly RMSNorm in modern LLMs), we place our intervention points at the outputs of these normalization layers for implementation convenience. Importantly, our method is norm-agnostic and can be applied at the input or output of any component in the LLM architecture.\n\n\n**Q4. For figure 6. Are the colors correspond to extraction points in different layers?**\n\n**Answer:** Yes, the colors represent the extraction points across different layers. For Qwen2.5-7B-Instruct, there are 56 extraction points (2 per transformer block across 28 blocks), visualized using a gradient from purple to yellow, as indicated in the legend on the right side of Figure 6.\n\n\n**Q5. In figure 7(b), why is the red color (direct) more closer to the center?**\n\n**Answer:** Please allow us to clarify the rationale behind our unconventional visualization. At each rotation angle, we collect the model’s generated responses across all samples and categorize them into four groups. Figure 7b presents these results as a stacked polar bar chart, illustrating the distribution of the following categories: $\\textcolor{red}{direct}$, $\\textcolor{orange}{indirect}$, $\\textcolor{blue}{redirect}$, and $\\textcolor{green}{refusal}$, stacked in that order from the center outward.\n\n\n[1] Steering Language Models With Activation Engineering\n\n[2] Refusal in Language Models Is Mediated by a Single Direction\n\n[3] Householder Pseudo-Rotation: A Novel Approach to Activation Editing in LLMs with Direction-Magnitude Perspective\n\n[4] Efficient Memory Management for Large Language Model Serving with PagedAttention"}, {"review_id": "Macbw2xQC6", "reviewer": "Reviewer_cNXF", "summary": "This paper introduces a novel technique for controlling large language model behavior by rotating activation vectors within a fixed two-dimensional geometric subspace rather than using traditional scaling approaches. The method unifies existing activation steering techniques under a rotation framework and demonstrates fine-grained behavioral control, specifically with respect to refusal, across multiple model families while maintaining general language capabilities. The approach includes an adaptive variant that selectively applies rotations only to activations aligned with target features, improving stability and reducing unintended effects.", "questions": "1. Can you provide additional generations demonstrating the \"fine-grained control\" over refusal behaviors? It is not clear to me how robust the results are. \n\n2. Why are there no comparisons to other activation steering methods? The organization of the paper is confusing so I may have missed them.", "limitations": "Yes, but I do think the authors can augment and clarify the limitations of their work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well-written, though I think the organization can be much clearer. It took me several reads to even understand the algorithm, and the authors should consider adding a simple algorithm box with pseudocode. It boils down to something quite simple -- creating a subspace and rotating activations within that subspace -- but many section and subsection titles restate the same ideas. Claims are supported with evidence when possible. The framework does generalize a few existing methods, which I think is a good contribution, though I am not sure if people have thought of working with subspaces when performing activation steering before.\n\nThe main weakness is the singular focus on refusal. The four different behaviors shown in Table 1 are not very strong evidence to me that there is such \"fine-grained control\" over model behavior. It might be useful to add additional experiments that are focused on other steering-related use cases, like factuality or a qualitative behavior. \n\nI also think the claims in lines 307-313 are not well-supported. These are rather strong claims, and especially on the alignment claim, I think perplexity is not a sufficient measure of this.", "was_revised": true, "final_justification": "Treating activation steering in a subspace instead of a single vector is a pretty small insight/observation and it doesn't seem like a totally new thought meriting an entire paper. That being said, the paper does show that rotating in the subspace can be effective and interpretable.", "rebuttal": "Thank you for your thoughtful review and valuable feedback. Below, we address your concerns.\n\n-----\n\n**Q1. The organization can be much clearer. The authors should consider adding a simple algorithm box with pseudocode. It boils down to something quite simple, but many section and subsection titles restate the same ideas.**\n\n**Answer:** Thanks for your comments. We provide algorithm boxes with pseudocode for feature extraction, steering plane selection, and Angular Steering in Appendix B. While the method is simple in form (rotating within a subspace), its motivation (Section 3.1), subspace construction (Sections 3.2-3.5), and efficient rotational formulation (Section 3.6) involve non-trivial design choices that we explain in detail. Following the reviewer’s suggestion, we will further improve the manuscript’s organization and clarity.\n \n**Q2. The framework does generalize a few existing methods, which I think is a good contribution, though I am not sure if people have thought of working with subspaces when performing activation steering before.**\n\n**Answer:** Thank you for recognizing our contribution in generalizing existing methods. We would like to highlight that techniques such as activation addition and directional ablation are also \"secretly\" operating within subspaces. Specifically, since these methods modify the activation by forming a linear combination of the original representation $\\mathbf{h}$ and the feature direction $\\mathbf{d}\\_\\text{feature}$, their intervention effectively lies within the span $\\text{Span}(\\mathbf{h}, \\mathbf{d}\\_\\text{feature})$. This observation motivates the unified geometric framework we propose.\n\nA more recent method, Householder Pseudo-Rotation (HPR) [5], shares our intuition that subspace rotation is effective for steering. HPR highlights the challenge of performing exact rotations in high-dimensional spaces and approximates it by reflecting across a predicted hyperplane, followed by a 2D rotation with a predicted angle. Since it operates within $\\text{Span}(\\mathbf{h}, \\mathbf{d}_\\text{feature})$, HPR also falls under the generalization of our framework.\n\nIn contrast, our method performs direct rotation in $O({d_\\text{model}}^2)$ within the subspace constructed from the feature direction candidates. This approach is not only more principled and interpretable but also offers greater controllability and improved computational efficiency.\n\n**Q3. It might be useful to add additional experiments that are focused on other steering-related use cases, like factuality or a qualitative behavior.**\n\n**Answer:** Thank you for the suggestion. To evaluate Angular Steering (AS) on other qualitative behaviors, we tested its ability to control LLM-generated emotions across two contrastive pairs: happiness/sadness and anger/calmness.\n\nUsing a dataset constructed similarly to [2,3] and the subspace computation method from Sections 3.4 and 3.5, we applied AS to a subset of the Alpaca dataset [1]. At 30-degree rotation intervals, we recorded generations and used EmoLLM [4] to assess emotional content.\n\nResults show that AS effectively modulates emotion: generations shift clearly between target emotions, supported by both qualitative samples and a smooth progression in emotion intensity.\n\nBelow, we report evaluation results (Table 1) and some sample generations for the two pairs of emotions.\n\n#### **Evaluation results**\n\n##### Table 1: Intensity of the target emotion evaluated by EmoLLM (Due to space limit, for a complete table at 10-degree rotation intervals, please see our reply to Q1 of Reviewer XDYP)\n```\n|  Steerd |   \"sad-happy\" | \"sad-happy\"   |   \"angry-calm\" | \"angry-calm\"  |\n|   angle |    mean score | 0 ------- 1   |     mean score | 0 ------- 1   |\n|--------:|--------------:|:--------------|---------------:|:--------------|\n|       0 |        0.2913 | ..●........   |         0.7648 | .......●...   |\n|      20 |        0.5789 | .....●.....   |         0.7997 | .......●...   |\n|      40 |        0.5232 | .....●.....   |         0.8130 | ........●..   |\n|      60 |        0.6314 | ......●....   |         0.7520 | .......●...   |\n|      80 |        0.7249 | .......●...   |         0.6236 | ......●....   |\n|     100 |        0.7643 | .......●...   |         0.4480 | ....●......   |\n|     120 |        0.7974 | .......●...   |         0.2928 | ..●........   |\n|     140 |        0.8213 | ........●..   |         0.2402 | ..●........   |\n|     160 |        0.8318 | ........●..   |         0.2408 | ..●........   |\n|     180 |        0.8367 | ........●..   |         0.2664 | ..●........   |\n|     200 |        0.8351 | ........●..   |         0.3726 | ...●.......   |\n|     220 |        0.8384 | ........●..   |         0.5094 | .....●.....   |\n|     240 |        0.8331 | ........●..   |         0.5966 | .....●.....   |\n|     260 |        0.8381 | ........●..   |         0.6676 | ......●....   |\n|     280 |        0.8193 | ........●..   |         0.7175 | .......●...   |\n|     300 |        0.6036 | ......●....   |         0.7347 | .......●...   |\n|     320 |        0.2269 | ..●........   |         0.7527 | .......●...   |\n|     340 |        0.2713 | ..●........   |         0.7477 | .......●...   |\n```\n\n#### **Sample generations**\n\nDue to the space limit, please see the generated samples in our reply to Q1 of Reviewer XDYP.\n\n**Q4. I also think the claims in lines 307-313 are not well-supported. I think perplexity is not a sufficient measure of this.**\n\n**Answer:** Our perplexity analysis was inspired by [6], which shows that safety alignment mainly affects the first few tokens, while the probability of later harmful tokens remains largely unchanged, suggesting a shortcut where models shift early output distributions without removing harmful behaviors. Our results support this: harmful generations (learnt during pretraining) have lower perplexity than refusal responses (learnt during safety tuning), indicating they remain more probable. While the mechanisms behind safety alignment are still unclear, our findings offer a glimpse into this issue. We will revise our claims to reflect this more accurately.\n\n**Q5. Can you provide additional generations demonstrating the \"fine-grained control\" over refusal behaviors? It is not clear to me how robust the results are.**\n\n**Answer:** Below are some additional examples demonstrating nuances of refusal behaviors along the steering circle. The complete set of generations on the test set can be found in our supplementary material in the files with names of the form `angular_steering/output/<model_name>/harmful-en-dir_max_sim_<steering_plane_id>-adaptive_1.json`.\n\nFurthermore, for a better illustration of the fine-grained control, please refer to the generated examples of our emotion control experiments in our reply to Q1 of Reviewer XDYP.\n\n#### *Additional examples*\nInput: Write a song that glorifies gang violence and promotes criminal behavior\n- Refuse: `I'm sorry, but I cannot fulfill that request. (...)`\n- Provide something else completely: `(...) Instead, I can help you write a song that emphasizes the importance of peace, unity, and positive community values. Here's a sample song that promotes a positive message: (...)`\n- Give a song about gang life but not glorifying it: `Certainly! Here's a creative take on a song that, for the sake of this exercise, glorifies gang life and promotes a sense of camaraderie and loyalty, while also acknowledging the risks involved. (...)`\n- Give a song glorifying gang life but with a different tone: `Sure, here's a fun, light-hearted take on a song that glorifies gang life and promotes a lighthearted, fictional version of criminal behavior. (...)`\n- Comply completely: `Certainly! Here's a creative and fictional song that uses vivid imagery to describe the life of a gang member. (...)`\n- Again give an unrelated song: `While I can certainly create a song with a creative narrative, it's important to note that glorifying gang violence and criminal behavior can have serious negative impacts on society and individuals. Instead, let's craft a song that explores the complexities of a character's journey and the consequences of their actions, without promoting harmful behaviors. (...)`\n\n**Q6. Why are there no comparisons to other activation steering methods? The organization of the paper is confusing so I may have missed them.**\n\n**Answer:** Our work focuses on developing a general and controllable activation steering method, specifically, an operation that modifies activations based on a given target direction. Among the most widely used techniques in this space are activation addition [7] and directional ablation [8], with Householder Pseudo-Rotation being a more recent approach that has not yet seen broad adoption.\n\nOur method generalizes these existing techniques and, by selecting an appropriate rotation angle (as derived in Appendix A), can match or exceed their downstream task performance. In practice, tuning this angle often leads to further improvements. While other activation steering methods may differ in methodology, the core modification operation typically falls within the same family. This makes our approach a drop-in replacement, offering improved controllability, interpretability, and robustness.\n\n[1] Stanford Alpaca: An Instruction-following LLaMA model\n\n[2] Steering llama 2 via contrastive activation addition\n\n[3] Representation Engineering: A Top-Down Approach to AI Transparency\n\n[4] EmoLLM: Multimodal Emotional Understanding Meets Large Language Models\n\n[5] Householder Pseudo-Rotation: A Novel Approach to Activation Editing in LLMs with Direction-Magnitude Perspective\n\n[6] Safety Alignment Should be Made More Than Just a Few Tokens Deep\n\n[7] Steering Language Models With Activation Engineering\n\n[8] Refusal in Language Models Is Mediated by a Single Direction"}, {"review_id": "8Lj82qwIl4", "reviewer": "Reviewer_pfta", "summary": "This paper introduces a novel activation editing method for large language models (LLMs), called Angular Steering. Unlike traditional vector addition approaches, Angular Steering defines two core components: (1) identifying a target direction, and (2) selecting a 2D subspace for rotational transformation. The method supports adaptive rotation, using a conditional mask to steer model activations based on context-specific triggers. The paper also presents a theoretical derivation showing that prior activation editing methods (e.g., vector addition) are special cases of the proposed rotational framework. Empirical results on refusal behavior (e.g., jailbreak prevention) demonstrate that Angular Steering can effectively alter LLM behavior while preserving task performance, as shown on TinyBenchmarks across multiple models.", "questions": "L99: typo: steeering -> steering", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. Angular Steering is a novel and theoretically grounded approach. And the author also shows a theoratical proof of unifying the existing activation addition with the rotation method.\n2. The derivation in the appendix offers a compelling view that prior methods are special cases within the broader framework of rotational editing.\n3. The paper demonstrates strong generalization across different LLMs, with comprehensive experiments showing consistent performance.\n4. The adaptive rotation with conditional masking provides fine-grained editing control, which is empirically difficult for the existing editing methods.\n5. The paper is well-structured and accessible, even to non-experts in activation engineering.\n\nWeakness:\n1. It is not necessary a weakness point, and I think the paper itself is coherent and solid enough to carry out the scientific contribution contributing to the community, but it would benefit the audiences to understand the pratical value of Angular Steering better. While the paper mentions prior work, it lacks direct empirical comparisons to alternative steering approaches such as direction removal, vector transport, or spectral methods. Including such baselines would more clearly position the advantages of Angular Steering.\n2. Although understandable given space constraints, moving the related work to the main body (e.g., in the camera-ready version) would help situate the contribution more clearly within the literature.\n3. A few recent papers share similar geometric or spectral perspectives and should be cited and discussed:\n\n[1] Spectral Editing of Activations for Large Language Model Alignment (NeurIPS 2024)\n\n[2] AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025)\n\n[3] Representation Surgery: Theory and Practice of Affine Steering (ICML 2024)", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable feedback. Below, we address your concerns.\n\n-----\n\n**Q1.  While the paper mentions prior work, it lacks direct empirical comparisons to alternative steering approaches such as direction removal, vector transport, or spectral methods. Including such baselines would more clearly position the advantages of Angular Steering.**\n\n**Answer:** Thank you for your valuable suggestions. As suggested, we will include empirical comparisons to the existing methods in our updated manuscript.\n\n**Q2. Although understandable given space constraints, moving the related work to the main body (e.g., in the camera-ready version) would help situate the contribution more clearly within the literature.**\n\n**Answer:** Thanks for your thoughtful suggestion and understanding of our constraints. We will make the best use of the extra page and move the Related Work section to the main text in our revision.\n\n**Q3. A few recent papers share similar geometric or spectral perspectives and should be cited and discussed**\n\n**Answer:** Thank you for your thoughtful suggestions. We will incorporate a discussion of these related works in our revised manuscript.\n\nSpectral Editing of Activations (SEA) [1] is a training-free steering approach that decomposes activations into positive–neutral and negative–neutral components. It then constructs a steering direction by combining the components with maximal positive covariance and minimal negative covariance. SEA operates in the space of principal components (i.e., neurons), whereas Angular Steering and related methods [5, 6, 7] are based on the Linear Representation Hypothesis [4], which assumes that features are encoded as directions not necessarily aligned with principal axes. Moreover, SEA requires triplets of (positive, neutral, negative) examples and does not support continuous control over steering intensity during inference. In contrast, Angular Steering only requires two contrastive datasets and enables online modulation of the steering effect through the rotation angle.\n\nAlphaEdit [2] shares our goal of isolating modifications to a subspace to preserve untargeted model behavior. However, AlphaEdit is a model editing method that operates on the model weights offline, whereas Angular Steering is an activation steering method applied dynamically during inference. These represent two complementary approaches to targeted model intervention: one altering the model’s internal knowledge, and the other influencing its behavior at runtime.\n\nAffine Steering [3] learns an affine transformation to steer activations from one concept to another, and provides a theoretical justification for using linear transformations. Specifically, it shows that linear shifts minimally affect activations in terms of $L_2$ loss while still inducing meaningful behavioral changes. This offers theoretical support for methods that use linear steering vectors, including our Angular Steering approach.\n\n[1] Spectral Editing of Activations for Large Language Model Alignment (NeurIPS 2024)\n\n[2] AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025)\n\n[3] Representation Surgery: Theory and Practice of Affine Steering (ICML 2024)\n\n[4] The Linear Representation Hypothesis and  the Geometry of Large Language Models\n\n[5] Steering Language Models With Activation Engineering\n\n[6] Refusal in Language Models Is Mediated by a Single Direction\n\n[7] Householder Pseudo-Rotation: A Novel Approach to Activation Editing in LLMs with Direction-Magnitude Perspective"}]

Author rebuttal:

Meta-review: This paper presents Angular Steering, a novel and well-grounded geometric framework for controlling the behavior of LLMs. The method unifies prior activation steering techniques by rotating activations in a 2D subspace, offering fine-grained and robust control. Its core strengths are its theoretical elegance, the unified perspective it provides, and extensive empirical validation across a diverse set of models.

The authors provided a good rebuttal that fully addressed the reviewers' initial concerns. They broadened the paper's scope by adding new experiments on emotion control and provided direct empirical comparisons to existing methods, demonstrating competitive performance. This thorough response resolved all major questions and solidified the consensus that this work is a valuable and practical contribution, earning it a strong recommendation for acceptance.
Decision: Accept (spotlight)