Fine-Grained Activation Steering: Steering Less, Achieving More
Authors: Zijian Feng
Tianjiao Li
Zixiao Zhu
Hanzhang Zhou
Junlang Qian
Li Zhang
Chua Jia Jim Deryl
Mak Lee Onn
Gee Wah Ng
Kezhi Mao
Venue: ICLR 2026 Poster (2026)
Keywords: Activation Steering
Large Language Models
Fine-Grained Intervention
Primary area: foundation or frontier models, including LLMs
TL;DR: Breaking LLM blocks to fine-grained atomic units for intervention: steering less achieves more

Abstract: Activation steering has emerged as a cost-effective paradigm for modifying large language model (LLM) behaviors. Existing methods typically intervene at the block level, steering the bundled activations of selected attention heads, feedforward networks, or residual streams. However, we reveal that block-level activations are inherently heterogeneous, entangling beneficial, irrelevant, and harmful features, thereby rendering block-level steering coarse, inefficient, and intrusive. To investigate the root cause, we decompose block activations into fine-grained atomic unit (AU)–level activations, where each AU-level activation corresponds to a single dimension of the block activation, and each AU denotes a slice of the block weight matrix. Steering an AU-level activation is thus equivalent to steering its associated AU. Our theoretical and empirical analysis show that heterogeneity arises because different AUs or dimensions control distinct token distributions in LLM outputs. Hence, block-level steering inevitably moves helpful and harmful token directions together, which reduces efficiency. Restricting intervention to beneficial AUs yields more precise and effective steering. Building on this insight, we propose AUSteer, a simple and efficient method that operates at a finer granularity of the AU level. AUSteer first identifies discriminative AUs globally by computing activation momenta on contrastive samples. It then assigns adaptive steering strengths tailored to diverse inputs and selected AU activations. Comprehensive experiments on multiple LLMs and tasks show that AUSteer consistently surpasses advanced baselines while steering considerably fewer activations, demonstrating that steering less achieves more.

Published as a conference paper at ICLR 2026
FINE-GRAINED ACTIVATION STEERING: LESS, ACHIEVING MORE
STEERING
Zijian Feng'
Tianjiao Li' Jia Jim Deryl Chua²
Zixiao Zhu'
Hanzhang Zhou'
Junlang Qian'
Li Zhang'
Lee Onn Mak
Gee Wah Ng²
Kezhi Mao
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore "Home Team Science and Technology Agency (HTX), Singapore
{feng0119, zixiao001, hanzhang001, junlang0C1, than0735j@e.ntu. enntu.eddu.gg
(tianjiao. 1i, ekzmao)@ntur edu.sg {deryl.chua, mak_lee.onn, ng.gee.wahJehtx.gge gov.ss
ABSTRACT
Activation steering has emerged as cost-effective paradigm for modifying large language model (LLM) behaviors. Existing methods typically intervene at the block level, steering the bundled activations of selected attention heads, feedfor- ward networks, or residual streams. However, we reveal that block-level activa- tions are inherently heterogeneous, entangling beneficial, irrelevant, and harmful features, thereby rendering block-level steering coarse, inefficient, and intrusive. To investigate the root cause, decompose block activations into fine-grained atomic unit (AU)-level activations, where each AU-level activation corresponds to single dimension of the block activation, and each AU denotes a slice of the block weight matrix. Steering an AU-level activation is thus equivalent to steering associated AU. Our theoretical and empirical analysis show that heterogeneity arises because different AUs or dimensions control distinct token distributions in LLM outputs. Hence, block-level steering inevitably moves helpful and harm- ful token directions together, which reduces efficiency. Restricting intervention beneficial AUs yields more precise and effective steering. Building on this insight, we propose AUSteer, simple and efficient method that operates at finer gran- ularity of the AU level. AUSteer first identifies discriminative AUS globally by computing activation momenta contrastive samples. It then assigns adaptive steering strengths tailored to diverse inputs and selected AU activations. Com- prehensive experiments on multiple LLMs and tasks show that AUSteer consis- tently surpasses advanced baselines while steering considerably fewer activations, demonstrating that steering less achieves more
INTRODUCTION
In the era of large language models (LLMs), activation stecring has emerged as a powerful paradigm
for modulating model behavior downstream tasks (Zou et al. 2023 Li et al. 2023b Rimsky et al.,
2024). Unlike reinforcement learning from human feedback (Bai et al.) 2022), supervised fine- tuning (Wei al. (0022), or prompt engineering (Brown et al.)
2020), activation steering intervenes directly in the LLM intermediate activations during forward propagation, enabling fine-grained con- trol without additional training. Prior work (Turner et al. 2023 Rimsky et al. 2024 Han et al.)
2024 Wang et al.] 2025a ble) generally builds task-specific steering vectors and injects them at in-
ference time biases or rescaling factors in selected LLM components, thereby steering the model
toward the target objective.
Corresponding author.
'Code:

---

Published as a conference paper at ICLR 2026
However, common practice in existing methods is block-level steering, where a "block" denotes the multi-heade attention (MHA), the feed-forward network (FFN), or the layer's 's residual stream. As shown in Figure (a), the intervention is vector-level: every dimension of the selected block's s acti- vation is bundled and steered simultaneously. One of the main limitations of block-level intervention is that it ignores heterogeneity within block activations. These activations often span hundreds or thousands of dimensions, each indicating a different feature. Some features are beneficial for the task, while others are irrelevant harmful. As result, block level steering is (1) too coarse: block can be decomposed into finer functional units, and treating it as a single entity prevents pre- eise targeting; (2) inefficient: steering the entire block amplifies both useful and harmful signals, which reduces efficiency and risks performance degradation; and (3) overly intrusive: it modifies many dimensions unnecessarily, increasing the intervention footprint.
In greater depth, we empirically and theoretically justify the heterogeneity of block-level activations. We first decompose block-level activations into finer-grained atomic unit (AU) activations, where each AU-level activation corresponds to single di- mension of the block activation, and each AU de- notes slice of the block weight matrix. Steering an AU-level activation is thus equivalent to steering its associated AU. As shown in Figure (b). each AU- level intervention targets a single dimensior Both the intervention value and the affected activation are scalars. Empirically, we find that AU-level steer- ing effects vary widely: some dimensions improve performance, some degrade it, and others are neu- tral, confirming heterogeneity. In many cases, steer- ing a single dimension small subset outperforms steering the entire block.
Framer Layer
Pulti-llead Atsontion
Add
Add
Original Activation
Loca sicatization
Y/ Yn
Intervention Vector
Y3 Intervention Scalar
Steered Activation
Steered Activation
Our theoretical analysis reveals that the heterogene- ity stems from different AUS modulating distinct output-token distributions. Steering single di- mension therefore shifts the model's output distri- bution toward the distribution controlled by that AU. Some AUs favor task-irrelevant or harmful tokens;
(a) Block-level
(b) AU-level
Figure I:
Comparison of block-level steering (prior work) and AU-level steering (Ours).
steering their dimensions degrades performance. This also explains why block-level steering. which mixes helpful and harmful AUs together, can underperform more targeted and precise AU-level steering. Targeting only beneficial AUS can reduce the intervention footprint and improve efficiency, that is, steering less achieves more.
Beyond the promise of AU-level steering, these findings also pose challenges: (1) how can we localize the most important AUs for intervention? and (2) how can we ensure adaptive steering across diverse inputs and AUs?
To address these challenges, we introduce AUSteer, simple and efficient method with two com- ponents. First, we propose activation momentum, a new metric that analyzes each activation's 's momentum in positive and negative samples to evaluate its discriminative power. This counting- based metric supports global comparison and avoids the issue increasing activation magnitudes across layer. We then localize the most discriminative AUs activations for steering. Second, to ensure adaptivity across inputs and AUs, we assign a per sample steering scalar that follows the original activation pattern rather than constant shift. This makes the update scale with the current activation, and preserves direction. We also assign dynamic steering strength to each AU according to its discriminative power, with important AUS receiving higher strength. We compare AUSteer with thet--rrt-t (SOTA) methods that intervene at the block level by steering hundreds thousands of activations. Using far fewer steered activations (at most 100), AUSteer significantly outperforms these methods across diverse tasks, demonstrating that steering less achieves more.
FFor clarity: block-level activation is vector associated with block (MHA, FFN, or a layer's residual stream), usually comprising hundreds to thousands of dimensions, whereas an AU-level activation is scalar corresponding to single dimension within that block activation.
2

---

Published as a conference paper at ICLR 2026
The contributions of this work are summarized as follows:
Conceptually, we study the heterogeneity within block-level activations and its root causes, both theoretically and empirically, and propose decomposing block-level intervention into fine-grained AU-level intervention ($3)
Methodologically, we propose AUSteer, a framework that localizes discriminative AUs with acti- vation momenta for steering, and ensures adaptivity across diverse inputs and AUs ($
Empirically, we evaluate AUSteer on multiple LLMs of varying sizes across diverse tasks, includ- ing commonsense reasoning. mathematical problem solving, and open-ended generation. With less intrusive intervention, AUSteer significantly outperforms other SOTA activation steering methods, underscoring that steering less achieves more ($5).
RELATED WORK
Activation steering (also known as activation editing) has become a popular and cost-effective ap- proach for modifying LLM behaviors and aligning them with downstream tasks (Turner al. 2023
2024 Wang et al. 2025c; Soo et al. 2025; Stickland et al. 2024 et al. 2023c Wang et al. 2025a b Stolio et al., 2025). The standard workflow involves ex- tracting steering vectors from prompts or contrastive samples and injecting them into LLMs at in- ference time. Most of these methods intervene at the block level. For instance, at MHA blocks, ITI (Li et al. 2023b) derives steering vectors from contrastive activations in attention blocks and then applies interventions using the extracted vectors to important heads. [Bhattacharjee et al. (2024) compute -ppeeir activations from attention heads reduce unsafe responses. In residual streams, CAA (Rimsky et al. (024) extracts vectors from positive and negative samples and applics them to residual streams, while van der Weij et al. (2024) extend this approach to multi-vector steer- ing across residual streams. EAST (Rahn et al. 2024) obtains steering vectors by weighting input prompts with entropy and injects them into the layer outputs. Postmus Abreu (024) use multiple steering vectors as a conceptor to redirect behaviors via residual stream activations. Safety methods such as SafeSwitch (Han et al., 2025) and Safety Arithmetic (Hazra et al.] 2024) intervene in residual streams to suppress harmful outputs. Konen et al. (2024) extract steering vectors from layer outputs to control emotion and writing style, while AnyEdit (Jiang al. 2025) updates hidden states and
Rimsky et al. 2024; Han et al.
knowledge by steering layer outputs. More recently, Stolfo et al. (2025) steer residual streams to enhance instruction following. Some methods can operate across multiple blocks. For example, SADI (Wang et al.) (2025b) computes steering vectors from MHA, FFN, residual streams, then applies mask-adaptive steering.
Notably, STA (Wang et al.] 2025a) identifies atoms in pretrained sparse autoencoders (SAEs)
[icberum et al. 2024; He et al. 2024 Gao al. 2025) of target LLMs and steers residual streams using these localized units. Although STA uses the term atom, its meaning differs from ours: in STA, an atom is a knowledge unit an SAE, whereas in our work an atom is a unit in the original LLM weight matrices. Methodologically, STA depends pretrained SAEs that currently exist for only a few model families such as LLaMA3.1 (Touvron et al.) 2023) and Gemma2 (Team et al.] 2024,, which limits generalization. Moreover, STA's intervention remains the block level as the computed vectors are injected into the residual stream.
3 HETEROGENEOUS BLOCK ACTIVATIONS: STEERING LESS ACHIEVES MORE
3.1 BLOCK DECOMPOSITION
We first show how computations within LLM blocks can be decomposed into fine-grained AU cal- culations. The backbone architecture of LLMs is the Transformer, which consists of attention blocks and FFN blocks in every layer. The outputs of these blocks are added to the layer residual stream for forward propagation. In both MHA and FFN, weight matrix computations (Q, K, V, in MHA and the up projection and down projection in FFN) are linear projections of the form y = Wx, where x the input activation, W is the weight matrix, and y is the output activation.
3

---

Published as a conference paper at ICLR 2026
In existing studies, block activations (x and y) typically treated as indivisible vectors. Steering vectors are calculated and applied at this coarse block level. To decompose blocks into finer- finer-grained units, we reformulate the linear projection
y Wx Σ 24 W...
Here, 2; denotes the i-th dimension of the input activation x. This formulation allows us to isolate each single-dimensional activation. In this view, every scalar serves the coefficient for the corresponding column W the weight matrix. We refer to each column W. as an Atomic Unit (AU) in our study
Pl In this way, steering the i-th dimension activation z; equivalent steering the corresponding i-th AU. To clarify:
X, y: block-level activations, represented as vectors (the standard formulation in prior work).
W the i-th column of the weight matrix W, representing the i-th AU.
zy i-th dimension or i-th AU-level activation, which is scalar and the coefficient for the i-th AU.
3.2 HETEROGENBITY BLOCK ACTIVATIONS
In this section, we examine the heterogeneous effects of AU-level activations within the block acti- vation. To ensure generalizability, we adopt two representative steering methods: the pioneering ITI (Li al. 2023b) and SOTA SADI (Wang et al. 2025b), applying them to MHA and FFN blocks. We use 2019) the illustrative dataset, where the model answers "yes" or "no" for each question and the accuracy is reported. The experimental setup follows SADI, as described in Appendix B
LLMA2-7B--C 7h-Chtt (Touvron et al. 2023 as the backbone model and BoolQ (Clark et al.]
We first use ITI and SADI identify important attention heads and FFNs for intervention, then compare six conditions: (1) Baseline, the original model without steering: (2) ITI, block-level
intervention on attention head activations: (3) SADI (Wang al.) 20255), block-level steering on
attention heads (128 dimensions) and FFNs (4096 dimensions); (4) Dimension Sweep, steering single dimensions rather than whole blocks, sampling one of every four dimensions attention heads and one of every 100 in FFNs; (5) Positive Combination, steering small subset of beneficial dimensions; and (6) Mixed Comb. steering a subset of beneficial and detrimental dimensions.
75
74
74
Accuracy
Accuracy
72
70
71
M
70
68
Dimension
Sweeep
SADI
69
66
68
m
Mixed Comb.
Reviin
0
20
40
60
80
100
120
500
1000 1500 2000 2500 3000 3500 4000
Dimension Index
Dimension Index
(a) Steering results for the 7th attention output in layer 27. Positive Combination: steering four ben- eficial dimensions (28, 40, 84, 92). Mixed Com- bination: steering those four plus two detrimental dimensions (108, 116).
(b) Steering results for the FFN output in layer 20. Positive Combination: steering four beneficial di- mensions (0, 1300, 2300, 3000). Mixed Combi- nation: steering those four plus two detrimental di- mensions (1200, 1700).
Figure 2: Heterogeneous steering results for MHA and FFNs.
Figure 2a shows the results of interventions on the 7th attention head the 27th layer. The orig- inal model achieves 70.52% accuracy, ITI reaches 71.56%, and SADI achieves 73.70% Steering individual AU activations, however, produces highly heterogeneous outcomes: some dimensions
Fach column of W corresponds to an AU, while each row corresponds to what is traditionally termed "neuron." " To ensure rigor and avoid confusion, we adopt the AU rather than neuron.
4

---

Published as a conference paper at ICLR 2026
degrade performance, while others improve it. Notably, steering single dimension can outperform full block steering. For example, steering the 84th dimension alone achieves 74.53%, surpassing the baseline, ITI. and SADI. Furthermore, steering only four positively contributing dimensions (Pos. Comb.) yields even stronger results. While introducing detrimental dimensions (Mixed Comb.), the perform drops. Similar observations hold for FFN blocks in Figure 26 Additional empirical results for other attention heads and FFNs are provided in Appendix [ These findings indicate that block-level steering is inefficient, it mixes beneficial and detrimental components. In contrast, fine-grained AU-level steering enables selective amplification of useful features, achieving more effective control. short, steering less achieves more.
3.3 INTERPRETING THE HETEROGENEITY
To explain the observed heterogeneity, discussed above, treat the block activation coeffi- cients on an AU basis, so steering a single dimensional activation I; is equivalent to steering its prior theory of interpreting LLMs in the embedding space (Geva al. 2022 Dar
AU. Building et al. 2023), different AUs may control different token distributions in LLM outputs. Steering task- relevant AUS promotes the probability of task-specific tokens, whereas steering task-irrelevant AUs may increase the probability of uninformative or even harmful tokens. This provides a theoretical justification for the observed heterogeneity.
To further validate this, we first examine the con- vergence behavior of AU steering: different AUs govern different output token distributions, and as steering strength increases, the LLM's output tends
1.00
10
0.768
0.188
0.000
Strength
100
0.692
0.116
0.025
0.75
1000
0.196
0.004
0.058
0.50
to converge to the AU's token distribution. For the selected 7th attention head at the 27th layer, we scale the AU coefficient from 10 to extremely large value (100,000) and compute the normalized KL di- vergence between the output each strength and the output at 100,000. In Figure 3] columns 1 and 2 show these divergences for the 44th AU and the 84th AU. The divergence decreases with strength, indicat- ing convergence. Column 3 shows the pairwise KL divergence between the 44th AU and the 84th AU across strengths. The divergence increases with strength, indicating that the two AUs tend to drive the model toward different output distributions.
10000
0,007
0.000
0.848
0.25
100000
0.00
LAU AUJ
AUgE
AUS
[AU
[AU]
Figure 3:
Pairwise KL divergence when steering different AUs. s means strength.
Figure 4 illustrates this phenomenon by reporting the top-5 output tokens after steering three different AUs with single-dimensional activations. The input prompt a question from BoolQ dataset with the answer "yes". Steering 2&4 promotes the correct answer token "yes" while suppressing the incorrect "no", thereby improving accuracy. In contrast, steering dimensions 244 I100 el- evates task-irrelevant or incorrect tokens, resulting in de- graded performance. These observations align with the accuracies shown in Figure 2a
Decoded top lop-k tokens
AU
yes'
YYas'
'Yes'
'yes
AUss
yyour'
Ethey'
'your
'YYour'
all'
Probability (high low)
Figure 4: Top-k deceode tokens con- trolled by different AUs. The answer to input prompt is "yes".
In summary, heterogeneity arises because each AU gov- erns a distinct output-token distribution. Block-level activations inevitably mix beneficial, irrelevant, and harmful AUs, making block-level interventions coarse, inefficient, and intrusive. By contrast, selectively steering only the helpful AUs amplifies the desired distribution and enables more efficient control.
METHODOLOGY: AUSTEER
Breaking block-level interventions into finer- greingr AU-level interventions has shown promise for modifying LLM behaviors. Yet AU-level steering faces some fundamental challenges: identifying important AU-level activations for intervention and ensuring adaptability across diverse inputs and AUs. To address these challenges, we propose AUSteer shown as Figure
5

---

Published as a conference paper at ICLR 2026
Block-Level Activations
AU-Level
AU-Localization
Adaptivee Stecring
Pos. Samples
Ney. Samples
u/: (-th sample
Original AU-level activation
AU-level color depth indieating strength
Figure 5: Overview of AUSteer: (1) AU localization using activation momentum and discriminative scores; and (2) Adaptive steering across diverse inputs and AUs.
4.1 ATOMIC UNIT LOCALIZATION
The first challenge is to identify which AUs and their activations should be steered. Prior work often uses probing (Li et al. 2023b) activation values (Wang et al. (2025b) of contrastive pairs impor- tance metrics. Here, a contrastive pair consists of a positive example with a correct or high-quality response and matched negative example with incorrect or low-quality response. However, prob- ing requires additional training resources and does not transfer well to single-dimension settings of AUs, while activation magnitudes tend increase with layer depth, making cross-layer comparisons unreliable. To overcome these limitations, we propose an activation momentum strategy.
Given N pairs of contrastive sample an AU is discriminative if its activation coefficient æ, con- sistently separates positives from their matched negatives. Concretely, if I; is systematically higher (or lower) for the positive sample of each pair than for the negative sample, the AU promotes (or suppresses) activation for positives relative to negatives. Such consistency indicates that the AU distinguishes positive from negative cases and therefore task-relevant.
Formally, let denote the i-th AU with the activation T₁. For the j-th sample pair, we define the activation momentum as
j.neg
where x 2-p0 and g.in are the activation values of u, on the j-th positive and negative sample, respectively. Note that both x ipss and f. are one-dimensional scalars defined in Eq | When m² O, the AU exhibits activation promotion effect for positive samples, whereas my < indicates a suppression effect. By counting the occurrences promotion and suppression across samples, assess whether an AU shows consistent effect positive negative cases, thereby quantifying its discriminative power. The proportions of positive and negative momenta are then given by
pos
1mm > 0),
1mm < 0).
N
The discriminative score of the i-th AU is defined as:
8; max(r..0. 7.8)
This scoring provides a unified scale for cross-layer comparison, allowing us to rank AUs globally and select the most important AUS for steering. To verify how activation momentum contributes to discriminative causality and the final model outputs, we provide both thcoretical and empirical analyses in Appendix H
4.2 ADAPTIVE STEERING
The steering of activation should be adaptive in two respects. First, it should adapt to diverse inputs. Different samples produce activations with different magnitudes and semantic contexts. Adding constant vector ignores this variation, can distort useful directions, and may impair model
Details of contrastive sample construction can he found in Appendix B.1

---

Published as a conference paper at ICLR 2026
performance. We therefore obtain the steered activation by i; Iz + Ye which scales the current activation, preserves its sign, and adapts well across varies samples.
Second, steering should adapt across AUs. More discriminative AUs receive stronger steering, while less important ones receive weaker intervention. This concentrates changes on useful AUs and limits unnecessary perturbations. To achieve this, compute 7/1 as
a 2
nog
7/
a arr
otherwise,
where a is global steering strength factor, r and r. are the positive and negative discriminative scores of the i-th AU caleulated by Eq The steering direction is determined by whether the AU has a promotive or suppressive effect. Finally, for the selected AUs, activations are updated as
2i 76344-
Applicability of AUSteer. The proposed AUSteer can be applied to all key components of LLMs, including MHA, FFN, and residual streams, the analysis above holds uniformly across these mod- ules. Unlike previous approaches that operate on entire block-level activations, AUSteer intervenes only on the most important dimensions within each block activation. This yields interventions that are both more efficient and less intrusive, embodying the principle of steering less to achieve more.
5 EXPERIMENTS
5.1
EXPERIMENTAL SETTINGS
Tasks and Evaluation Metrics. We evaluate AUSteer on three types of tasks:
Commonsense reasoning. We use widely adopted datasets including BoolQ (Clark al. 2019). COPA (Gordon et al., 2012), and WinoGrande (Sakaguchi et al. 2021). and report accuracy of the model's 's responses using exact match.
Math problem solving, We experiment with SVAMP (Patel et al. 2021 and MAWPS (Koncel- Kedziorski et al. 2016), where the model is required to solve math questions with or without reasoning. We evaluate accuracy by comparing the predicted answer with the correct number.
Open-ended generation. We employ RealToxicPrompts (Gehman al. 2020 and BPO (Cheng ee al., 2024). For RealToxicPrompts, which contains challenging prompts that often elicit toxic content, we apply different steering methods to reduce toxicity. Automatic evaluation follows prior work (Wang et al. 2025a): detoxification performance, where toxicity is measured using the Perspective A For BPO, which aligns model outputs with human
human- preferred behaviors, we adopt the automatic evaluation protocol of Zheng et al. (2023); Liang et al. (2024) and report AWR WRsteered WRoriginal- The win-rates are obtained by using GPT-5-mini (OpenAI, 2025) and prompts from Liang et al. (2024) to compare the original and steered responses. For both datasets, human evaluation is conducted, where annotators assess text quality (fluency, diversity) and alignment with the target objective 1-5 scale.
Target LLMs. We evaluate AUSteer diverse set of LLMs: (1) LLLMA2-7B-ChM Touvrron
al. 2023), which serves as the backbone many related studies; (2) Gemma2-9B-it (Team al. 2024). strong decoder-only model for text generation; and (3) Qwen3-8B (Yang et al. [0255, one of the most recent LLMs. further assess scalability, we also experiment with other LLMs and larger variants (c.g., 13B, 27B), with results reported in $5.4
Baselines. We compare AUSteer against several competitive activation steering methods:
ITI (i al. 2023b), which uses contrastive samples to identify important attention heads, then derives steering vectors from activation differences for intervention.
CAA (Rimsky et al. 2024), which extracts steering vectors from activation differences in residual streams and applies them the block level.

---

Published as a conference paper at ICLR 2026
Table 1: Overall results of baseline methods and the proposed AUSteer across seven tasks. "#Acts"
denotes the number of intervened activations for each method. ky indicates the number of selected
attention heads, ranging from 2 to 64. Results with are from Wang et 2025a b).
Commonsense Reasoning (?)
Math Problem Solving (T)
Avg.
Open Generation (f)
Model
Method
#Acts
BoolQ
COPA
WinoG.
SVAMP
MAWPS
Acc.
Vanilla
70.521
70.80
50,91
36.00
51.83
56.01
ITI
Ru 128
74.101
77.20"
$2.80
36.67
54.08
58.97
83.49
12.50
CAA
4096
74.98
75.20
$2.64
34.33
55.21
58.47
84.57
11.00
LLMMA2-7B-Chut
SADI
Kg 128
74.351
78.80'
$3.04
36.33
54.93
59.49
86.32
13.50
AUSteer-Head
100
76.27
75.40
52.80
37.67
56.06
59.64
16.00
AUS@ce-FFN
82,80
53.28
37.00
58.03
61.34
89.24
Vanilla
86.64
77.40
67.25
67.67
92.11
78.21
-56
69.00
93.52
81.17
CAA
3584
86.85
89.40
68.35
68.00
92.96
81.11
98.751
23,50
Gemmu2-99B-i
SADI
Rg 255
86, 79
92.40
69.53
68,00
93.52
82.05
99.08
25.50
STA
3584
87.03
91.60
69.61
68.67
92.39
81.86
99.331
25.00
AUSteer-Head
100
86.79
91.00
70.72
71.00
94.65
82.83
99.00
26.50
70.00
94.08
Vanilla
87.43
89.00
63.61
68.67
84.51
78.64
Rg 128
90.80
65.59
69.67
87.32
80.16
75.15
Qwen3-8B
CAA
87.83
71.67
80.63
76.23
SADI
6u 128
87.65
93.20
64.25
73.00
85.92
80.80
78.07
25.50
AUSteer Head
100
87.71
95.40
65.35
76.33
91.27
83.21
80.90
26.00
AUSteer-FFY
100
88.20
90.80
67.25
71.67
89.58
81.50
81.65
34.00
SADI (Wang et al. (025b), which localizes important attention heads, FFNs, or layers via activa- tion differences, and applies adaptive steering through masking and scaling. We report results for its best-performing variant, SADI-HEAD.
STA (Wang al., 2025a), which identifies important atoms in sparse autoencoders (SAEs) of the target LLM, then applies steering vectors to residual streams. Since pretrained SAEs are currently available only for LLaMA 3.1 and Gemma2, we report its results only on Gemma2.
AUSteer Variants. The proposed method can be applied to any key component of LLMs. Since the two core modules in each Transformer layer are the attention and FFN blocks, we validate the generalizability of AUSteer by implementing two variants: AUSteer-Head, which steers AU-level activations in MHA, and AUSteer-FFN, which steers AU-level activations FFN.
Implementation details for AUSteer and all other baseline methods, including contrastive pair construction, dataset statistics, and prompt templates, are provided in Appendix
Hy alyeeraareeete settings. For baseline methods, perform hyperparameter sweeps following the recommendations their papers to ensure a fair comparison. AUSteer introduces two hyper- parameters: (1) h. the number of AU-level activations selected for steering; and (2) a, global decering-strengee factor. To verify the claim that we can steer less to achieve more, we cap the number of steered activations 100 and then run the sweep. Full details appear in Appendix C
5.2 MAIN RESULTS
AUSteer significantly improves commonsense reasoning and math problem solving with min- imal intervention. Table [ reports overall results on LLaMA2-7B-Chen Gemma2-98. and Qwen3-8B. Across all five tasks on commonsense reasoning and math questions, either AUSteer- FFN or WiSteer-HHad attains the highest average accuracy while steering at most 100 activations, in contrast to SADI's 's ki, x 128 head interventions and CAA/STA, which modify thousands of acti- vations. Concretely, AUSteer-FFN improves the average over SADI by +1.85 on LLaMA2-7B-CII (61.34 vs. 59.49), +1.91 on Gemma2-9B-it (83.96 vs. 82.05), and +0.7 on Qwen3-8B. AUSteer- Head is also competitive, exceeding SADI Qwen3-8B by +2.41 under the same low- oow-budget straint. Beyond averages, AUSteer- Head or AUStcer-FFR consistently achieves the best scores on individual tasks across the five commonsense and math benchmarks.
8

---

Published as a conference paper at ICLR 2026
AUSteer improves open-ended gen- eration. In automatic evaluation (Table D. AUSteer significantly in- creases detoxification rates under toxic prompts. Compared with SADI, it yields around 2%-3% higher detoxification Llama2 and Qwen3.
Table
Human evaluation on open-ended generation tasks.
MaMAA-7B-Chal
Gemma2-9B-ir
Qweni-88
SADI
AUSteer
SADI
STA
AUSreerr
SADI
AUStzer
Quality (t)
4.2
4.1
4.3
Aligament (1)
3.6
3.8
4.5
4.7
4.7
3.9
4.1
On BPO datasets, AUSteer steers models toward human- pretferee responses, improving win-rates (AWR) by 8.5%, 4.5%, and 7% on the three LLMs, respectively. In human evaluation (Table ( AUSteer outperforms baselines in most cases on generation quality (fluency and diversity) and on alignment with the generation target.
5.3 ABLATION STUDIES
We evaluate the contribution of each component in AUS- teer: AU localization and adaptive steering. To assess the proposed activation momentum localization, com- pare it with (1) random localization, which selects acti- vations at random for steering, and (2) activation differ- ence across contrastive samples for localization, as intro- duced in SADI. To assess adaptive steering, we compare with (3) a fixed steering vector, which replaces "/4" with the mean activation difference, following ITI, and (4) a fixed steering strength y, which applies the same strength across all selected AUs.
Table Ablation study results Gemma2-9B-it.
Method
Avg.
AUSteer- -FN
83.96
Random Loc.
79.08 (-4.88)
Act. Diff.
83.12 (-0.84)
Fixed. Vec.
82.05 (-1.91)
Fixed. Strength
83.04 (-0.92)
The average accuracy across commonsense reasoning and math questions are shown in Table When using random or cetivation-diffreent localization, steering performance drops substantially, verifying the effectiveness of the proposed activation momentum based localization. Similarly, replacing adaptive steering with a fixed vector or fixed strength reduces performance by 1.91 and 0.92, respectively, demonstrating the importance of adaptivity across diverse inputs and AUs.
5.4 SCALABILITY AND GENERALIZABILITY OF AUSTEER
Table 4: Experimental results on more LLMs.
LLaMA3.1 1 8B Instruct
LLaMA2- 13---hat
Gemmaa-27BB-o
HoolQ
COPA
WinoG.
BoolQ
COPA
WinoG.
BoolQ
COPA
WinoG.
Vanilla AUSteer-Head 83.18
82.57
83,80
57.77
84.01
89,00
53,99
86.88
86.00
63.61
86.00
60.38
85.25
91.00
59.43
88.10
90.20
67.25
AUSteer-FFN 83.79
86.00
61.56
85.02
91.20
58.88
88.41
89.80
66.30
We evaluate AUSTEER larger and varied LLMs, including LLaMA3. 1-8B-Innttruc, LLaMA2- 13B-Chat (Touvron et al. 2023), and Gemma2-27B-it (Team et al. 2024), on commonsense rea- soning tasks. Table 4 reports the results. Both AUSTEER-HEAT and AUSTEER-FFN substantially improve the base models, confirming the method's scalability and generalizability. More results on larger LLMs with diverse structures including Qwen3-30B 3----3BBB3BBBBB-- and Llama-3.3-70B Instruct --nsttt- provided in Appendidx G
5.5 FURTHER ANALYSIS
To investigate the internal mechanisms of AUSteer more comprehensively, provide the following discussions.
Appendix
We illustrate the hyperparameter sweep for lt and and report their optimal values across tasks. We also provide guidelines for hyperparameter search in both resource-sufficient and resounce-constrainee settings.

---

Published as a conference paper at ICLR 2026
Appendix
We characterize activation momentum for different AUs and analyze the locations AUs within MHA and FFN.
D
Appendix
present and discuss the overlap localized AUs across tasks.
Appendix
F
We evaluate AUSTEER under varying numbers of contrastive pairs used for AU
localization.
Appendix G We demonstrate AUSteer's scalability on larger LLMs with diverse architectures, sparse MoE model) and 1amma-..-70B-Instruue (evaluated in its
including Qwen3-30B-A33
4-bit quantized form).
verify how activation momentum contributes to discriminative causality and final model outputs, providing both theoretical and empirical analyses.
Appendix H
Appendix We present a detailed analysis of AUSteer's 's efficiency and computational overhead are also included.
compared with baseline methods. Overhead results on Llama-3.3-700B-nnstie
We experiment with additional control variants of AUSteer-suce -utteerr-uchh steering all AUs or broader subsets and confirm that, as with AUSteer, steering should be limited to task-relevant and beneficial AUs rather than blindly steering all large numbers of units.
Appendix !
Appendix K To determine whether should promote useful AUS suppress unhelpful ones, conduct both empirical and theoretical analyses and show that promotion consistently outperforms suppression.
CONCLUSION
In this work, we investigate the heterogeneity and its root cause block-level activations and pro- pose AUSteer, a fine-grained AU-level activation steering method. AUSteer localizes salient AUs via activation momentum and assigns dynamic steering strengths per input and AU. Extensive periments show that, with far fewer intervened activations, AUSteer significantly outperforms state- of-the-art methods across diverse tasks, demonstrating that steering less achieves more.
ACKNOWLEDGMENTS
We extend our heartfelt gratitude to the reviewers for their insightful and constructive feedback. This research was supported by the Home Team Science and Technology Agency (HTX), Singa- pore under the NTU-HTX collaboration project: Parsimonious Domain Specific Large Language Model Enabled Multimodality Sensemaking. We express our sincere appreciation to HTX for their continued support and collaboration.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204 05862, 2022.
Amrita Bhattacharjee, Shaona Ghosh, Traian Rebedea, and Christopher Parisien. Towards inference- time category- safety steering for large language models. In Neurips Safe Generative Al Workshop 2024, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https::/ppocceeeee iew.:eeeeeddnnnnnn.nneis......s. neurips cc cc/paper File/145
wap.eeeeeeaaaaap..ae.. ppf
Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang. Yuxiao Dong, Jie Tang, and Minlie Huang. Black-box prompt optimization: Aligning large language models without model training. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 3201 -3219, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653/v1/ 2024.acl-long. URL ac www.cnannooooooyy.ooooooooo org. 2024 we..4..cl-loggg 1767
Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Pupers), pp. 2924-2936, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10.18653/1/1119-13000 URL MEtps://aclantho.. -1tp:/11111..111111111111
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers). pp. 16124-161 16170, Toronto, Canada, July 2023. Association for Computational Linguis- URL pattp:://aclnttcllae org/2023.
2t1.:55/////22..aalllooooo/ ac 121-long.880 8937
tics. doi:
Zijian Feng, Hanzhang Zhou, Zixiao Zhu, Tianjiao Li, Chua Jia Jim Deryl, Mak Lee Onn, Gee Wah Ng, and Kezhi Mao. Restoring pruned large language models via lost component compensation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth Inter- national Conference Learning Representations, 2025.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxic- ityPrompts: Evaluating neural toxic degeneration in language models. In Trevor Cohn, Yulan He, and Yang Liu (eds.). Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1356-3369, Online, November 2020. Association for Computational Linguistics. doi: 10. 18653/61/22220 (00.5///12000....iiieeeeeaaaaaap findings-emnlp3011. URI. httpthttt////// //tp://allaaann wottellogy.o org/2020. Find ngs-ings--a emn 301/
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Yoav Goldberg, Zomitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods
11

---

Published as a conference paper at ICLR 2026
in Natural Language Processing, pp. 30-45, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. http://ww12222...mmmmmiimii URL https://aclanthooo.s wrg. 2022. emnip-main.
Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plau- sible alternatives: An evaluation of commonsense causal reasoning. In Eneko Agirre, Johan Bos, Mona Diab, Suresh Manandhar, Yuval Marton, and Deniz Yuret (eds.), *SEM 2012: The First Joint Conference on Lexical and Computational Semantics Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012). pp. 394-398, Montréal, Canada, 7-8 June 2012. Asso- ////aaaannn http//aaaannttooooaaaaaaoaoooota /ttttp::/aaaaatttlllaaa org/ S12- 1121//52//11//
ciation for Computational Linguistics. URL httpatt::
Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, and Heng Ji. Word embeddings are steers for language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Pupers), pp. 16410-1644 16430, Bangkok, Thailand, August URL
2024. Association for Computational Linguistics. doi: 10. 18.18//3/v 8647]
20tp:/2244 httpp::/aclaatthooooy..ooo http:/oooooo..ooo/22....a------.
Peixuan Han, Cheng Qian, Xiusi Chen, Yuji Zhang, Denghui Zhang, and Heng Ji. Internal activation as the polar star for steering unsafe Ilm behavior. arXiv preprint arXiv:2502.01042, 2025.
Rima Hazra, Sayan Layek, Somnath Banerjee, and Soujanya Poria. Safety arithmetic: A framework for test-time safety alignment of language models by steering parameters and activations. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 21759-21777, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.186533w1/2224..oo 1/202..mmnlppmmain 1212. / www.hooloooooooo//o/oo 2024. emn main. 1212/
Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang. Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang. Zuxuan Wu, et al. Llama scope: Extracting millions of features from Ilama-3.1-88 with sparse autoencoders. arXiv preprint arXiv:2410.20526, 2024.
Houcheng Jiang, Junfeng Fang, Ningyu Zhang, Mingyang Wan. Guojun Ma, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Anyedit: Edit any knowledge encoded in language models. In Forty- second International Conference on Machine Learning, 2025.
Shahar Katz, Yonatan Belinkov, Mor Geva, and Lior Wolf. Backward lens: Projecting lan- guage model gradients into the vocabulary space. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Nat- ural Language Processing, pp. 2390-2422, Miami, Florida, USA, November 2024. Associa- tion for Computational Linguistics. doi: 10. 10.18653/v1/2024..cmm emn 1p halppmain.... 142.
142. URL https:
Rik Koncel-Kedziorski Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1152-1157, San Diego, Califor- nia, June 2016. Association for Computational Linguistics. doi: 10.18653/w1/N16- 1136. URL ht http://aaaaaattto......oooee
1136/
Kai Konen, Sophie Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, and Tobias Hecking. Style vectors for steering generative large language models. In Yvette Graham and Matthew Purver (eds.), Findings of the Association for Computational Linguistics: EACL 2024, pp. 782-802, St. Julian's s. Malta, March 2024. Association for Computational Lin- guistics. URL htt.https../ // a antho logy. /targ/22220 f findinggg waccl .527
Chong Li. Shaonan Wang. Yunhao Zhang, Jiajun Zhang, and Chengqing Zong. Interpreting and exploiting functional specialization in multi-head attention under multi-task learning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pp. 16460-16473 16476, Singapore, December 2023a. Association for Computational Linguistics. doi: 10. 10.18653/11/0033.0 weeml-mmiinnnnmmmm 1026. URL https://acl antho ogy. ogw.org//.2...... emw.00222.mmm..mmmmmmmmm 1amnlpmma 10267
12

---

Published as a conference paper at ICLR 2026
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt. and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 41451-41530. Curran Associates, Inc., 2023b. URL httpss:/pprcceed ngs
www.:eeediinnnnennneeeeeeeeeeeee neurips cc http://perrrfreepppprrrrapa////e Papeeeddd---pp-rr-eee Conference pdf
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. inference-timet intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 66:41451-41530 2023c.
Zhaoyi Li, Gangwei Jiang. Hong Xie, Linqi Song, Defu Lian, and Ying Wei. Understanding and patching compositional reasoning in LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics: ACL 2024, pp. 9668 9688, Bangkok, Thailand, August 2024, Association for Computational Linguisties. doi: 10.18653/ 2.0024..iniins-acl...
URI. wttptttt:////... //acts antho logy. org/2024. findings
5767
Zihan Liang, Ben Chen, Zhuoran Ran, Zihan Wang, Huangyu Dai, Yufei Ma, Dehong Gao, Xi- aoyan Cai, and Libin Yang. Self-renewal prompt optimizing with implicit reasoning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findingy of the Association for Com- putational Linguisties: EMNLP 2024, pp. 3030-3041, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 10.18653/v1/2224 10l.6655w//22224.iinnnneeeemmmme findiings--mmll.1111. URL
.rg. (tw.:e.....nnioooooee findingss emr.p emm.ii.ngseem....1....... 171/
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th Black- boxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 278 7788- Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10. 10.18653/v12002. blackboxnlp- 1.19. https:::: ac www.accnnnntoooyo..oooooooooooos
Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards inter- preting visual information processing in vision-language models. In The Thirteenth International Conference on Learning Representations, 2025.
OpenAl. Gpt-5 models.
openai. com docs/ http:/dccs/ceeeee 1s 2025.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve sim- ple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080-2094, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 waa.el..aall-ommmm maaac-main.16.. URL ht
168/
Joris Postmus and Steven Abreu. Steering large language models using conceptors: Improving addition-based activation engineering. arXiv preprint arXiv:2410.16314 2024.
Nate Rahn, Pierluca D'Oro, and Mare G Bellemare. Controlling large language model agents with entropic activation steering. In ICML 2024 Workshop on Mechanistic Interpretability. 2024.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama via contrastive activation addition. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 15504-15522, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 1/21665511122222a--oaooo 828. URL
https://aclanthon www.oooyyorr///.ooooo.o 2004. ael.4aacll111 aolnonoooooogn 828/]
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adver- sarial winograd schema challenge at scale. Commun. ACM, 64(9)99- 106, August 2021. ISSN 0001-0782. doi: 10.1 10.1145/3474381.
13

---

Published as a conference paper at ICLR 2026
Samuel Soo, Wesley Teng, and Chandrasckaran Balaganesh. Steering large language models with feature guided activation additions. arXiv e-prints, pp. arXiv-2501, 2025.
Asa Cooper Stickland, Alexander Lyzhov, Jacob Pfau, Salsabila Mahdi, and Samuel R Bowman. Steering without side effects: Improving post-deployment control of language models. In Neurips Safe Generative Al Workshop 2024, 2024.
Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, and Besmira Nushi. Im- proving Instruction-following in language models through activation steering. In The Thirteenth International Conference on Learning Representations, 2025.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2433.08295, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307. 009888, 2023.
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv e-prints, pp. arviv22308, 2023.
Teun van der Weij, Massimo Poesio, and Nandi Schoots. Extending activation steering to broad skills and multiple behaviours. arXiv preprint arXiv:2403.05767 2024.
Mengru Wang, Ziwen Xu, Shengyu Mao, Shumin Deng. Zhaopeng Tu, Huajun Chen, and Ningyu Zhang. Beyond prompt engineering: Robust behavior control in L.LMs via steering target atoms. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- une Long Papers), pp. 23381-233332 233399, Vienna, Austria, July 2025a. Association for Compu- tational Linguistics. ISBN 979-8-88 879-8-89176-251-0 doi: 10. 1225://1/2025.cllllaaaa 1139. URL wwt.or//////// www.::2.....1111...111..0....
attts::/accaatte
Weixuan Wang. JINGYUAN YANG, and Wei Peng. Semantics-adaptivy activation intervention for iims via dynamic steering vectors. In The Thirteenth International Conference on Learning Representations, 2025b.
Xintong Wang, Jingheng Pan. Liang Ding, Longyue Wang, Longqin Jiang. Xingshan Li, and Chris Biemann. CogSteer: Cognition-inspiree selective layer intervention for efficiently steering large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 25507- 25522, Vienna, Austria, July 2025c. Association for Computational Linguistics. ISBN 979-8- 19176-256-5. doi: 10.8655/11/2225 andings-acll1300 URL https: orq/2025 -
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Andy Zou, Long Phan, Sarah Chen, James Camphell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405 2023.
14

---

Published as a conference paper at ICLR 2026
A MORE RESULTS ON ACTIVATION HETEROGENEITY
We present additional results for attention heads and FFNs in Figures 6 to ity in block level activations.
confirming heterogene-
76
74
72
70
68
66
64
Dimension Sweup
Pos. Camb. Mixed Comb
Baseline
62
SADI
20
40
60
80 100
120
Dimensian Index
Figure 6: Steering results for 26th attention output at 15th layer. Positive Combination: steering four beneficial dimensions (32, 36, 48, 64). Mixed Combination: steering those four plus two detrimental dimensions (44, 88).
74
Accuracy
73
72
71
70
0
20
40
60
80
100
120
Dimension Index
Figure 7: Steering results for 1st attention output at 19th layer. Positive Combination: steering three beneficial dimensions (28, 40, 100). Mixed Combination: steering those three plus two detrimental dimensions (64, 68).
74
Accuracy
72
70
68
OwPeD
66
500
1000 1500 2000 2500 3000 3500 4000 Dimension Index
Figure 8: Steering results for the FFN output at layer 17. Positive Combination: steering three beneficial dimensions (400, 800, 2400). Mixed Combination: steering those three plus two detri- mental dimensions (2200, 3500).
B DETAILED EXPERIMENT SETUP
B. CONTRASTIVE SAMPLE CONSTRUCTION
Contrastive sample pairs are required by AUSteer and all baseline methods. To ensure a fair com- parison, follow SADI (Wang et al. 2025b) and STA (Wang al. 2025a) to construct same pairs
15

---

Published as a conference paper at ICLR 2026
for every method. For each sample in commonsense reasoning, form a positive sample by con- catenating the question with the correct answer, and a negative sample by concatenating the question with randomly selected incorrect answer. In math problem solving, correct answer as the positive sample. For the negative sample, we use a sentence encoder to select the most semantically similar incorrect answer from the answer pool and concatenate it with the question. For detoxification, we select entries from Pramooxicitrroompts with high toxicity scores as negative prompts. Following STA, the safe response is used as the positive sample. In BPO, we use the original prompt paired with a high-quality (human-prefene preferred) response as the positive sample, and the same prompt paired with a low-quality response from the dataset as the negative sample.
the question plus the
We clarify that (1) contrastive samples are required by almost all activation steering methods and are a common practice in prior work (Li et al., 2023b; Rimsky et al.) 2024 Wang et al. 2025b a) rather than a limitation unique to AUSteer; (2) constructing these pairs is generally straightforward based on available samples and easy to implement; and (3) we provide and verify simple, general, and ready-to-use procedure for constructing contrastive pairs across different and new tasks.
(1) Contrastive samples are widely required in activation steering. Existing activation steering methods, including ITI, CAA, SADI, and STA, all rely on contrastive Dusitive-negatio samples to localize important components and/or to estimate steering vectors. Thus, the requirement of contrastive pairs is not a limitation specific to AUSteer, but rather a standard and widely adopted practice. For fair comparison, also ensure that all baseline methods the same contrastive pairs in our experiments.
(2) Constructing contrastive pairs is simple in practice. Following prior work such as SADI and STA, constructing contrastive pairs is straightforward. For commonsense reasoning tasks, the neg- ative sample can be obtained by pairing the question with an incorrect answer. For other datasets, negative samples can be generated by selecting semantically similar responses from a pool of can- didate answers, or by using datasets that already include ready-to-use negative samples.
(3) A general solution for new tasks. For tasks not covered in existing studies, we use a general and effective approach. Positive sample: concatenate the question with the correct answer. Negative sample: use a sentence encoder to identify the most semantically similar incorrect answer from the answer pool and concatenate with the question (Feng et al., 2025). For example, previous studies did not include math tasks, so we constructed contrastive pairs for those tasks using this method. For all other tasks, we use the contrastive pairs provided by prior work to ensure fair comparison.
(4) Empirical verification of the general solution. Using the above general construction method, we re-evaluated AUSteer on Llama2-78 -hat. As shown in T this simple approach achieves performance comparable to even slightly better than original results.
Table 5: Results on LLaMA2-7B-Char with new contrastive pairs.
Method
Avg. Ace. (5 tasks)
Detox
BPO
Vanilla
56.01
SADI
59.49
86.32
13.50
AUSteer (previous result)
61.34
89.24
22.00
AUSteer (new solution)
61.53
89.99
22.50
In summary, contrastive pairs are commonly required across activation steering studies and are not a unique limitation of AUSteer. Moreover, constructing them is straightforward, and our general solution simple, effective, and empirically validated to yield strong performance. We acknowledge that the reliance on contrastive pairs is inherent limitation of existing activation steering methods, and we plan to explore approaches that reduce or eliminate this requirement in future work.
B.2 DATA STATISTICS
Following SADI, we use most 1,000 contrastive pairs per task to identify important MHA and FFN components or to generate steering vectors. For evaluation, we use the full test set of each task. Detailed dataset statisties are provided in Table
16

---

Published as a conference paper at ICLR 2026
Table 6: The number of contrastive pairs and testing samples for tasks.
BoolQ
COPA
WinoGrande
SVAMP
MAWPS
Detox
ВРО
# of contrastive pairs
1000
1000
1000
700
1000
1000
1000
Test
3270
500
1267
300
355
1199
200
B.3 PROMPTS FOR DATASETS AND EVALUATION
To ensure fair comparison, use identical prompt templates across all methods. For common- sense reasoning tasks, the templates strictly follow SADI (Wang et al., 2025b) and the authors' released code. For RealToxiiittyPropts. the templates follow STA (Wang al., (2025a.. Figure shows the templates for SVAMP and MAWPS. For BPO, we use the prompts provided in the dataset directly.
MAWPS
SVAMP
Anawer the grade school math word problem below, using step-by-step problem-solvingg process. Print the final answer after (##### Question: (question) Answer:
Answer following grade-school math word problem. Reply with only final answer as number. Question: (question) Answer:
Figure 9: Prompt templates for math problems.
C
HYPERPARAMETER SENSITIVITY
COPA
MAWPS
BPO
32 79.00
22.39
P2.11
92.69
23.24
94.0
8
3.50
22.50
50
$1.40
94.80
34.40
16
12.39
22.39
92.63
94.08
03.24
93.5
25.00
13.50
11.59
11.00
14.00
20
64
32
92.09
93.0
32
10
60
80
47.00
96.80
97.09
97.80 09.00
50
92.99
99.24
9-.28
922.96
92.5
50
28.00
10.50
9.00
24.99
6.10
100
5
10
20
50
100
1
5
10
20
50
0.5
5
10
20
Figure Performance heatmaps for COPA, MAWPS, and BPO tasks as functions of a and
AUSteer introduces hyperparameters: (1) k; the number of AU-level activations selected for steering: and (2) a. global steering-strength factor. To verify the claim that we can steer less achieve more, cap the number of stecred activations 100 and sweep both & and a from to 100 for main experiments. Figure I0 reports performance across COPA, MAWPS, and BPO. Neighboring settings around the optimal hyperparameters achieve comparable results, indicating robustness. The optimal values vary across tasks to some extent, showing that the hyperparameters are task-specific, trend consistent with Wang et al. (2025b).
To set the hyperparameters for each task, we provide two solutions: (1) under sufficient computing resources, we perform full hyperparameter sweep, which consistent with previous studies (Li al. 2023b Rimsky et al. 2024, Wang et al. 2025bb a); and (2) in computing-constraiinn we recommend using a very small validation set to conduct a quick hyperparameter sweep. In addition, (3) the optimal hyperparameters used in our experiments
scenarios,
reported in Tables 3 (and
General hyperparameter sweep (resource-sufffccent case). Task-specific hyperparameters are still a common challenge in activation steering, and the standard solution used widely in existing studies is to perform a sweep (Li al., 2023b] Rimsky et al.) [024 Wang et al.) 2025b a). Following these studies, perform full hyperparameter sweep empirically determine optimal and k.
17

---

Published as a conference paper at ICLR 2026
We also run the same sweep for all baseline methods ensure fair comparison in Table 1. Across tasks, both a and k typically fall within 1-100 and consistently yield strong results.
Fast sweep using small validation set (resource- time-constrained case). When resources are limited, we recommend sweeping using only 50-100 validation samples. This process is ex- tremely fast (e.g., --5 minutes using this small-set search are shown below Table It can be observed that even with only very few samples for hyperparameter selection, our proposed method still significantly outperforms the baseline methods and achieves results comparable to the full search.
H100 GPU for 100 samples for the COPA task). Results
Table 7: Results of fast sweep LLaMA2-7B-CMa
Method
Avg. Acc. (5 tasks)
Detox
BPO
Vanilla
56.01
SADI
59.49
86.32
13.50
AUSteer (100-sample search)
61.03
88.49
22.00
AUSteer (Full search)
61.34
89.24
22.00
The optimal values of and & used for each task reported in Tables 3 and These values were obtained via full sweep, and the same process was applied to baseline methods for fairesss. The task-specific variation of hyperparameters aligns with observations from prior work, indicating that different tasks may require different hyperparameter values.
However, for any given task, the hyperparameters in Figure [O] for the COPA task, when 20 S a < 50 and 64 < k < 100, the performance remains stable and varies within only 1.5%, while still significantly outperforming the baseline methods. For the MAWPS task, when 10 < < 50 and 16 < k s 50, the performance also varies within approximately 1.5%. Therefore, for each specific task, our method hyperparameter-robnne and within the optimal region, it achieves comparable results with only small variations.
stable and robust. For example, shown
Table 8: Optimal a for main experiments.
QoolQ
COPA
Winogrande
SVAMP
MAWPS
Detoxic.
BPO
15
50
100
15
32
50
50
100
100
50
8
10
10
20
20
10
50
10
16
Table 9: Optimal l: for main experiments.
BoolQ
CОРА
Winogrande
SVAMP
MAWPS
Detoxic.
BPO
100
16
2
50
80
16
16
8
80
64
16
8
100
8
100
100
2
8
10
In summary, although the hyperparameters remain robust within individual task, task-specific hyperparameters are still a common challenge in activation steering. The standard solution used widely in existing studies is perform sweep. To further reduce cost, show that sweeping on a very small validation set is both efficient and highly effective, while still outperforming strong baselines. We will explore more principled approaches reducing task-dependent hyperparameter sensitivity in future work.
D CHARACTERISTICS OF ACTIVATION MOMENTUM AND LOCALIZED AUS
Figures IIa and 1lb report the discriminative score for each AU in both MHA and FFN, computed
via activation momentum. We observe pronounced heterogeneity: within attention heads and FFN blocks, some dimensions/AUs are strongly discriminative while others are not. Moreover, most AUs localize to the middle or latter layers, consistent with prior findings (Wang et al. 2025b) that middle layers support reasoning while latter layers critical for language generation.
18

---

Published as a conference paper at ICLR 2026
0.8
20
20
25
0.4 25
30
30
15
0.5
Dimwnsion
Bimersion
(a) AU scores in the MHA of Gemma2 --auna2--BBitt 9B-it it on the COPA dataset.
(b) AU scores in the FFN of Gemma2-9BB 9B-it on the COPA dataset.
Figure 11: Characteristics of AUS in MHA and FFN.
AU OVERLAP ACROSS TASKS
1
1.0
1
1.0
2
0.1
1.0
0.0
1.0
3
0.0
0.1
1.0
0.0
0.1
1.0
4
0.0
0.0
0.0
1.0
0.0
0.0
0.0
1.0
5
0.0
0.0
0.0
0.0
1.0
5-
0.0
0.0
0.0
0.0
1.0
0.0
0.0
0.0
0.0
0.0
1.0
6-
0.0
0.0
0.0
0.0
0.0
1.0
7-
0.0
0.0
0.0
0.0
0.0
0.0
1.0
0.0
0.0
0.0
0.0
0.0
0.0
1.0
7
(a) Overlap of identified AUs in the MHA of Gemma2-9R-it across different tasks.
(b) Overlap of identified AUs in the FFN of Gemma2-9B-it across different tasks.
Figure 12: Overlap of localized AUs across tasks. Tasks 1-7 correspond to BoolQ, COPA, Wino- Grande, SVAMP, MAWPS, Detoxification, and BPO.
We visualize the overlap of localized AUs across tasks in Figures 12a and 12b Only very few AUs are shared between tasks, indicating that the AUs supporting different functions are highly specialized. This pattern is consistent with prior studies ct al. 2023a Wang al. 2025b).
F STEERING STABILITY WITH VARYING DATA SIZE
Following prior work, AUSteer uses contrastive sample pairs to localize important AUs. We evaluate how its accuracy varies with the number of contrastive pairs. As shown in Figure B3 the accuracy Gemma2-9B-in improves as the dataset grows. Notably, with 300-500 pairs, AUSteer achieves per- formance comparable using 1,000 pairs. This demonstrates its effectiveness in low-data regimes.
G SCALABILITY ON MORE LLMS
To further verify the generalizability and scalability of AUSteer, we evaluate on two representative large models with diverse structures: (1) Qwen3-30B-A3B 30B-scale sparse MoE model; and (2) LJama-3,3-70B. 703.3-70B-Instruct where we use the 4-bit quantized version to enable evaluation on consumer GPU and to test AUSteer's compatibility with heavily quantized LLMs. The results are shown in Table IO, In most cases, AUSteer improves performance by 1 1%-3%, confirming its effectiveness and scalability across larger, structurally diverse and heavily quantized L.LMs.
19

---

Published as a conference paper at ICLR 2026
97.5
95.0
92.5
Accuracy (%)
90.0
87.5
85.0
82.5
AUSteer-Attn en COPA
80.0
AUSteer-Nitn on BoolQ
AUSteer-FFN en Boold
77.5
200
400
600
800
1000
Data Size
Figure 13: Relationship between accuracy and the number of contrastive pairs
Table 10: More results of diverse and larger L.Mss.
Model
Method
BoolQ
COPA
WinoG
Qwen3-30B-A3B
Vanilla
86.82
93,40
65.98
AUSteer
88.69
97.80
67.17
Llama-3.3-70B-Instuu
Vanilla
89.54
98.60
78.14
AUSteer
90.67
99.20
79.95
H ANALYZING ACTIVATION MOMENTUM IN DISCRIMINATIVE CAUSALITY
We explain the connection between activation momentum and causality based both theoretical justification and empirical evidence.
Theoretical Justification. Prior work in LLM interpretability (Dar al. 2023 Geva et al. 2022
Li et al. 2024 Katz et al. [024]] Neo et al. 2025) shows that intermediate hidden states & in LLMS can be directly projected the output logits through the LM head. This projection directly affects the model's final next-token distribution. Formally, the LM head M computes:
Mæ.
where o the vector output logits.
This aligns with our observations in Section 3.3: different AUS govern different output token distri- butions, and as steering strength increases, the LLM'' output tends to converge the AU's token distribution. For a contrastive pair, the logit difference caused by the two inputs is:
Ao pos
For AU u; and contrastive pair j, define the activation momentum:
m = x
Based on o Mzz we apply first-order Taylor expansion around x
do Da:
o(x)")) o(x;..)
neg
x
Rearranging gives:
do Or;
Ao.
pos
n0
m
20

---

Published as a conference paper at ICLR 2026
This equation shows that the change in activation of AU uy directly causes a proportional change in the output logits. Thus:
mf > tends to increase the logit difference favoring the positive sample.
mf tends to favor the negative sample.
If m is consistent across many pairs, then the AU u, has a stable discriminative causal effect on the output logits.
This provides the theoretical grounding for activation momentum.
Empirical Evidence To further validate the effectiveness of activation momentum, we compare against two alternatives: (i) randomly selected AUs and (ii) the activation-differrnce method used in SADI. On Gemma2-9B-it, the performance follows the order: 83.96 (activation momentum, ours) 83.12 (activation difference by SADI) 79.08 (random selection). These results demonstrate the superior performance of activation momentum. Additional experimental details are provided in Section 5.3.
To summarize, we establish the connection between activation momentum and discriminative output causality through both theoretical analysis and empirical validation, thereby grounding and verifying our method.
I COMPUTATION OVERHEAD ANALYSIS
We conducted detailed efficiency and computation analysis from two perspectives: (1) smaller steering footprint, and (2) the actual computational overhead measured in practice, including cetivation-momentte computation time, inference-time cost, latency, and stability. Our results show that AUSteer requires less overhead and fewer interventions while achieving better performance than baseline methods.
Smaller steering footprint. As shown in Table baseline methods typically require intervening on 3,000-4,000 activations (or K; 128), whereas AUSteer requires at most 100 intervened activations while still achieving the best results on most tasks.
Detailed overhead analysis. We examine the computational overhead of our method and all base- lines at each stage of the method. In the preparation phase, AUSteer extracts activations from con- trastive pairs to compute activation momentum, whereas baseline methods usually require compo- nent localization and steering-vector estimation. During inference, each method applies its corre- sponding intervention, and we compare the resulting overhead across methods. It is worth men- tioning that activation momentum calculation only requires single forward pass over small set of contrastive examples. No backward pass, gradient computation, model modification, or training is needed. Extracting activations simply involves reading intermediate hidden states. Therefore, for any LLM size, activation momentum can always be computed using the same GPU memory required for standard inference, since both perform identical forward passes.
Table I1 below compares the computation cost of AUSteer with ITI and SADI across six metrics: (1) GPU memory for contrastive samples of all tasks, (2) total runtime all contrastive samples, (3) GPU memory during inference, (4) inference time over seven tasks, (5) latency, and (6) latency stability (std from five repeated trials). is noted that all methods rely on contrastive samples to compute the necessary steering momentum (ours), or steering estimation (other methods). The backbone L.LM Gemma2- 9B-it (batch size 1, GPU NVIDIA H100).
for activation differences, localization, activation
Compared to other activation-steering baselines, AUSteer has the lowest overhead while achieving the best results. Specifically, AUSteer requires only ~55 minutes to compute activation momentum and localization, no additional GPU memory beyond inference, and exhibits lower overhead than ITI and SADI while achieving better performance, demonstrating its computational efficiency. During inference (steering), AUSteer also requires slightly less time than the baseline methods, further demonstrating its efficiency in runtime overhead.
21

---

Published as a conference paper at ICLR 2026
Table 11: Computation Overhead Comparison.
Method
GPU Memory (contrastive
Time (contrastive)
GPU Memary (Inlerence)
Inference Time
Latency
Stabilay
Vanilla LLM
18 I8C
53 min 12 soc
0.45 ssamppc
A000015
see
GB
min see
s/sample
40.01
SADE
18 GB
14 min 41 sec
18 Gll
55 min 05 sec
0.47 shample
40.005
AUSteer (Ours)
18 GB
14
GB
s/sampl
40.007
For the computational cost on larger LLMs such as 4-bit -lamm-3.3-70B-IIstruee taking COPA as an example, we report both the preparation (activation-mone overhead. During the activation-monte momentum computation stage, using 1000 contrastive pairs, AUS- teer requires 40 GB of GPU memory and around 15 minutes. During inference, the vanilla LLM requires 40 GB of GPU memory and 3 min 46 sec to run all test samples, while AUSteer requires 40 GB and 3 min 54 sec. These empirical results show that activation momentum scales successfully to large LLMs and remains far from computationally intensive, even on a 70B LLM.
computation) and inference
To summarize, putational overhead, while achieving the best performance on most tasks. This provides clear empirical evidence supporting argument that a smaller steering footprint can achieve improved efficiency.
proposed method requires the least intervention footprint and lowest com-
J BROADER CONTROL VARIANTS OF AUSTEER
We conducted additional experiments broader steering variants and found that, contrary to the assumption that "steering more AUS should be better," precise partial AU control is the correct strategy. It offers clear advantages over steering large portion-or all-of the AUs.
Steering all AUS leads to consistent performance degradation. To test whether AUSteer is merely constrained version of more general "steer-all-units" method, we applied AUSteer-style dynamic weights to all AUs (e.g., 32 x 4096 = 131,072 AUs in LLaMA2-7B-Chan -hBt).. After extensive hyperpa- rameter sweeps, steering all AUs still failed outperform the vanilla model (without any steering). This matches our analysis in Section 3.3: different AUS regulate different output distributions, and only a small subset is task-relevant. Steering all AUS inevitably introduces strong task-irrelevant signals, effectively injecting noise into the model outputs. In contrast, partial AU steering focuses only on useful and task-relevant subspaces, yielding meaningful and targeted interventions.
Broader AU steering does not guarantee better performance. We further tested variants that steer increasingly large subsets of AUs. Table 12 (COPA, LLaMA2-7B-Chai shows that steering more than 5,000 AUs results in worse performance than the vanilla model. This again confirms that broader steering introduces many task-irrelevant or harmful output distributions. degrading performance. These findings also align with our results in Section 3.2, where steering certain AUS leads to negative effects.
Table 12: Experimental results on steering broader AUs using COPA and LLaMA2-7B-CHm There 32 4096- 131,072 AUS in total in LLaMA2-7B--hat.
# of AUs Accuracy (%)
0 (vanilla)
< 100
200
500
1000
3000
5000
10000
70.8
82.8
77.2
73.2
70.8
70.6
70.4
70.4
Overall, our experiments demonstrate that AUSteer should only steer task-relevant or beneficial AUs, rather than steering broad or full set of units. Partial AU control is therefore not restricted version of more general steering method- it the correct and uniquely effective strategy for activation steering in LLMs.
K PROMOTION VERSUS SUPPRESSION
To determine whether we should promote useful AUs or suppress unhelpful ones, we conduct both empirical and theoretical analyses and show that promotion consistently outperforms suppression.
22

---

Published as a conference paper at ICLR 2026
Empirical evidence. To evaluate the "suppression" strategy, use AU importance scores to iden- tify the least important AUs and apply a decreasing factor to suppress their activations. We vary the number of suppressed AUs from 0% to 99.95%, search decreasing factors from 0.05 to 0.99, and report the best results Table 13] Experiments are conducted on LLaMA2-7B-Cha using three commonsense reasoning datasets. The results show that although suppression can yield im- provements over the vanilla model, it consistently underperforms compared to the promotion-based steering used in AUSteer
Table 13: Experimental results of suppressing AUs.
Method
BoolQ
COPA
WinoG.
Vanilla
70.52
70.8
50.91
Suppression
73.36
71.6
53.12
Promotion (AUSteer, ours)
75.57
82.8
53.28
Theoretical explanation. Prior work (Geva et al. 2022 Dar et al. 2023) shows that LLMs update
predictions primarily through promotion mechanism, where top-candidate tokens are driven by dominant positive sub-updates rather than by suppressing irrelevant ones. Consequently, directly promoting task-relevant AUS aligns better with the model's intrinsic update dynamics, producing stronger and more targeted effects than suppression.
23


Reviews:
[{"review_id": "RzHdP0HC7j", "reviewer": "Reviewer_61go", "summary": "In this paper, they present AUSteer, a more fine-grained activation steering technique, to control LLM's behavior during the inference time. First, they recognize the heterogeneity in block activation and explain this through comprehensive experiments. Inspired by these experiments, they developed a more fine-grained activation steering algorithm. In detail, first, use activation momentum to recognize the important atomic unit on the target tasks. Then, steer these atomic units' activation adaptively. They did comperihensive experiments to evaluate AUSteer. And the results are convincing.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. They first use two sections to recognize and interpret the heterogeneity in block activation, which gives insight and inspiration for AUSteer.\n2. The method is natural and effective. \n3. The experiments are comprehensive, spanning three LLMs with different architectures and three different tasks.", "weaknesses": "1. The biggest model used is 27B. Evaluating AUSteer on bigger models, e.g., 32B and 72B, and sparse models, e.g., MoE, even multi-modal models would be better.\n2. The optimal hyperparameters \\alpha and k are task-specific; how to set the hyperparameters for every tasks? And what is the hyperparameters used in Table. 1?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sz2zo3UMCj", "reviewer": "Reviewer_mmMY", "summary": "This paper introduces AUSteer, a novel fine-grained activation steering method for LLMs that operates at the atomic unit level rather than the traditional block level (e.g., attention, FFN, or residual blocks). The authors identify a key limitation in existing steering methods: block-level activations are heterogeneous, mixing beneficial, irrelevant, and harmful components. As a result, conventional approaches (like CAA, SADI, or ITI) that steer all dimensions of a block simultaneously are coarse, inefficient, and potentially harming model performance.\n\nTo address this, AUSteer decomposes each block into fine-grained AU-level activations, where each AU corresponds to a single column of the weight matrix and each activation is a scalar. The method consists of two main components:\n\n-AU Localization via Activation Momentum: A metric that measures the discriminative power of each AU across positive and negative contrastive samples. It identifies which AUs consistently promote or suppress desirable activations.\n-Adaptive Steering : Instead of applying a fixed vector, AUSteer adjusts steering strength per input and per AU, scaling the intervention by the activation’s current value and discriminative score.\n\nExperiments are conducted on various LLMs (LLaMA2, Gemma2, Qwen3) and tasks, including commonsense reasoning, math problem-solving, and open-ended generation.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clearly identifies a fundamental issue: heterogeneity in block activations—and systematically decomposes it into atomic units.\n\n- Introduces the concept of activation momentum to measure discriminative importance without training.\n\n- Extensive experiments across three model families (LLaMA, Gemma, Qwen) and multiple tasks (reasoning, math, safety, alignment).\n\n- No retraining or fine-tuning required.\n\n- Ablation studies isolate the contribution of both components.", "weaknesses": "- The formal derivation connecting activation momentum to discriminative causality is unclear.\n\n- AUSteer requires carefully curated positive–negative pairs, which may not be available or trivial to construct for all tasks.\n\n- While steering itself is efficient, computing activation momentum across many AUs and samples may still be computationally intensive for very large models.\n\n- Hyperparameter sensitivity is unclear and needs further demonstrations and explanations.\n\n- One wonders what is the runtime overhead for AU localization and steering per sample compared to block-level methods like SADI?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b6xrjf7VXl", "reviewer": "Reviewer_xSGu", "summary": "The paper proposes AUSteer, a method for fine-grained activation steering in large language models. Instead of steering at the block level, AUSteer operates at the Atomic Unit (AU) level, corresponding to individual activation dimensions. The authors show that block-level activations mix helpful and harmful components, making coarse interventions inefficient. AUSteer identifies discriminative AUs through an activation momentum metric computed from contrastive pairs and applies adaptive per-AU scaling. Experiments on seven benchmarks and three model families demonstrate consistent improvements over block-level steering with far fewer activations, suggesting that steering less can achieve more.", "questions": "1) Does efficiency refer to computational speed or representational precision?\n2) Have you measured inference cost, latency, or stability?\n3) Would steering all AUs with selective suppression perform similarly?\n4) How scalable is activation momentum computation for very large models?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The problem is clearly defined and relevant. The idea of decomposing block activations into AUs is intuitive and well motivated. AUSteer is simple, interpretable, and does not require retraining. The experiments are broad and consistent across tasks and models, and the analysis convincingly shows heterogeneity within block activations.", "weaknesses": "1) Efficiency claim lacks evidence:\n\nThe paper’s argument that a smaller steering footprint improves efficiency is not empirically verified. No inference-time or computational measurements are provided, and efficiency is used only in a representational sense.\n\n\n2) Lack of comparison with broader control variants. \n\nThe paper assumes that steering only a subset of AUs is inherently superior, but does not test a broader or fully generalized steering scheme where all AUs are jointly optimized or selectively weighted. Without such a comparison, it remains unclear whether partial AU control offers unique advantages beyond being a constrained version of more general steering", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)