Spectral Attention Steering for Prompt Highlighting
Authors: Weixian Waylon Li
Yuchen Niu
Yongxin Yang
Keshuang Li
Tiejun Ma
Shay B Cohen
Venue: ICLR 2026 Poster (2026)
Keywords: Attention steering
Large language model alignment
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose SEKA (Spectral Editing Key Amplification), a training-free method that steers attention by editing key embeddings pre-computation.  It achieves better performance with negligible overhead.

Abstract: Attention steering is an important technique for controlling model focus, enabling capabilities such as prompt highlighting, where the model prioritises user-specified text. However, existing attention steering methods require explicit storage of the full attention matrix, making them incompatible with memory-efficient implementations like FlashAttention. We introduce Spectral Editing Key Amplification (SEKA), a training-free steering method that tackles this by directly editing key embeddings before attention computation. SEKA uses spectral decomposition to steer key embeddings towards latent directions that amplify attention scores for certain tokens. We extend this to Adaptive SEKA (AdaSEKA), a query-adaptive variant that uses a training-free routing mechanism to dynamically combine multiple expert subspaces based on the prompt's semantic intent. Our experiments show both methods significantly outperform strong baselines on standard steering benchmarks while adding much lower latency and memory overhead, in compatibility with optimised attention.

Published as a conference paper at ICLR 2026
SPECTRAL ATTENTION STEERING FOR PROMPT HIGH- LIGHTING
Weixian Waylon Li', Yuchen Niu², Yongxin Yang', Keshuang Li Tiejun Ma', Shay B. Cohen' Unniversity of Edinburgh, UK 2RayNeo, China Huawei Research Ltd., UK *Queen Mary University of London, UK
(waylon. 1i, tie jenuumma @ed. aeeedaa..... ac. uk scohen@inf.ed ac.. ac.uk
ABSTRACT
Attention steering is an important technique for controlling model focus, en- abling capabilities such prompt highlighting, where the model prioritises user- specified text. However, existing attention steering methods require explicit stor- age of the full attention matrix, making them incompatible with memory-effciien implementations like FlashAttention. We introduce Spectral Editing Key Amplifi- cation (SEKA), a training steering method that tackles this by directly editing key embeddings before attention computation. SEKA uses spectral decomposition steer key embeddings towards latent directions that amplify attention scores for certain tokens. We extend this to Adaptive SEKA (AdaSEKA), a guery-adaptive variant that uses training-free routing mechanism to dynamically combine mul- tiple expert subspaces based on the prompt's semantic intent. Our experiments show both methods significantly outperform strong baselines on standard steering benchmarks while adding much lower latency and memory overhead, in compati- bility with optimised attention.
INTRODUCTION
The ability to precisely guide the behaviour of large language models (LLMMs) is paramount as they are increasingly deployed in high-stakes domains. This broad field of model steering encompasses various techniques, from activation steering. which aims to control high-level semantic attributes like
style or factual recall by intervening in MLP layers (Subramani et al., 2022 Turner et al. 2023 Qiu
et al. 2024| Turner et al. 2024 Wang et al., 2025 Stolto et al. (025), attention steering, which operates at a more granular level to direct the model 's focus to specific tokens within a prompt. This paper focuses the latter, where prompt highlighting is one of the key applications. Current state- of-the-art methods, such as PASTA (Zhang et al. (2024), operate by editing the attention score matrix after it has been computed. This post-hoc manipulation creates a critical bottleneck: it requires computing the full attention matrix, making these methods incompatible with modern, 10-aware implementations like FlashAttention (Dao et al. 2022 Dao 2024) that are essential for efficient processing. This architectural limitation, coupled with the need for costly, task-specific searches to identify which attention heads to steer, makes them less practical.
In this paper, we propose to intervene in the input of the attention mechanism rather than edit its output. We introduce Spectral Editing Key Amplification (SEKA), a novel, training- ffree framework that steers attention by directly modifying key vectors before the attention scores are calculated. Our core insight that we can learn a universal "relevance subspace" for a given task by applying spec- tral decomposition to key embeddings derived from contrastive prompts. These learned directions are then used to construct a projection matrix that amplifies the relevant features of highlighted keys via simple, geometrically interpretable transformation: k' = k: + gPk.
Additionally, we propose Adaptive SEKA (AdaSEKA), an advanced variant that learns bank of task-specific "expert" projections (c.g., for factual recall versus instruction following). At inference time, AdaSEKA uses a computationally cheap, training- furee routing mechanism to create a dynamic, query-aware steering operator by blending these experts based on the prompt's semantic intent. Our

---

Published as a conference paper at ICLR 2026
method is fully compatible with FlashAttention it operates directly the key embeddings with negligible computational overhead.
Our experiments confirm the effectiveness of this approach. Both SEKA and AdaSEKA achieve perior results on standard benchmarks for knowledge conflicts, occupation extraction, and instruc- tion following. Furthermore, AdaSEKA's query-adaptive routing mechanism demonstrates superior performance by dynamically tailoring the steering to the prompt's semantic intent. Crucially, we show that these performance gains are achieved with negligible overhead. SEKA adds only =0.03s of latency per sample, in stark contrast to comparable methods like PASTA which incur a +1.03s inference time and nearly double the memory usage.
PROBLEM DEFINITION AND MOTIVATIONS
In this section, we formalise the problem of prompi highlighting instance of attention bias and present the motivation for our spectral attention steering approach, which aims to address the limitations of existing methods.
Problem Definition. Given a prompt x = (Z1...., consisting of T tokens, with subset of token indices H {1..... T) identifying the highlighted tokens (in practice, surrounded by markers such as **), our goal is to steer the attention of the model so that these tokens receive increased focus from queries. In standard multi-head attention, the unnormalised attention score between query i and key j is Attn(i, g. where Qu, kj E Rdd are the query and key vectors, and de is the head dimension.
vda
We aim to amplify the attention assigned to highlighted tokens by introducing an ad- ditive, controllable term the attention score for each (i, where < H: A'; Au + Aij. where Aij is designed to selectively boost the attention towards user-specified highlighted tokens.
Objective.
Motivation. Existing approaches typically modify attention after has been computed. For ex- ample, PASTA (Zhang al. 2024) rescales rows of the attention matrix shown in equation [ where C, a row normalisation factor and > scales attention to highlighted tokens.
Aij
a
if E H.
C
[T())]
(1)
Ag
otherwise.
C,
Similarly, positional calibration methods such as a baseline from the positional attention bias. Let Øe denote the position of the k-th token, and Attnori (xx) the original positional bias. The calibrated bias is AttMaaaiieetddd (Ix) Attm (IT)
(Hsieh et al. 2024) subtract
AttDhaseline (Tx), where (tthhaseline(xx) is estimated independently of content relevance.
Both strategies require explicit storage of the full attention matrix, which is incompatible with memory-efficiiet implementations such as FlashAttention (Dao et al., 2022 Dao 2024). More- over, methods like PASTA often rely on costly head search to decide which attention heads steer. These limitations motivate the consideration for an alternative steering mechanism that operates be- fore attention scores are computed, avoiding any need to materialise or modify the attention matrix. Since attention depends query-key inner products, equivalent control can be achieved by editing either representation (shown Section 3.2). Given our objective of amplifying attention to spe- cific subset of tokens H. key-side intervention is the natural choice: the key vector kj is indexed by token position j and therefore governs how strongly each individual token is attended to.
To provide empirical evidence whether such pre-attention intervention is feasible, we analyse how key representations change under shifts in contextual relevance. We first construct synthetic (1) neutral (context only), (2) positive (context aligned with a relevant query), and (3) negative (context paired with an irrelevant query). The construction of such synthetic triplets is deseribed in Appendix A
contrastive prompt triplets under three conditions:
Using the Qwen3-1.77 1.7--...--Bsce model (28 layers, heads), we extract the key embeddings correspond- ing the same token spans under both positive and negative prompts for each (layer, head) pair. We
2

---

Published as a conference paper at ICLR 2026
Layer 20 Head Pair Shift)
Layer Head (Pairwise Shift)
Layer Head (Pairwise
hmine Monn
Man ott
Compomen:
PCA Compement
PC Compoment
PCA Component
PCA Component I
PCA Component
Layer 24
(Pairwise Shifl)
Layer (Pairwise Shifly
Layer Head (Pairwise Shill)
Stren
Component
PCA Compoment
PCA Compomentt
PCA Component
PCA Component I
PCA Component
Layer 25.
(Pairwise Shift)
Layer Head (Painwise Shift)
Laver 27 Head (Pairwise
PCA Component 2
Component
PCA Component
PCA Component
PCA Component
PCA Component
Figure 1: Visualisation of pairwise key embedding shifts across different (layer, head) in Qwen3- 1.7B-Base via PCA. Positive negative representations are plotted for 26 shared token spans. Grey arrows trace individual shifts; the dark blue arrow shows the average displacement.
then apply PCA jointly project these paired embeddings into two dimensions, and visualise the result using a combination of scatter plots and directed arrows. Each arrow originates from a neg- ative key and points its corresponding positive key, capturing the pairwise representational shift induced by changing question relevance. To summarise the overall trend, we also plot the mean shift vector across all pairs. Figure [ shows that certain heads exhibit robust and consistent directional shifts in key embeddings when token relevance changes. Each plot visualises 26 key embedding pairs corresponding to shared token spans, extracted from 10 positive-negative prompt pairs.
These findings suggest that relevance is encoded in structured subspace of key representations, motivating our approach that edits key embeddings before attention is computed: K = k2 + PP where P is projection matrix (defining relevance subspace per key-value head), and g is scaling coefficient. This preserves compatibility with efficient attention implementations while providing a geometrically interpretable mechanism for steering attention towards highlighted tokens.
3 SPECTRAL ATTENTION STEERING FOR PROMPT HIGHLIGHTING
As shown in Figure 2 we propose a new method, Spectral Editing Key Amplification (SEKA). and its query-adaptive variant, AdaSEKA. Both methods achieve prompt highlighting by directly editing key embeddings before the attention computation. The core mechanism of SEKA is inspired by the Spectral Editing of Activations (SEA) algorithm (Qiu et al.) 2024). adapting it from semantic-level activation stecring to the token-wise attention steering required for prompt highlighting.
3.1 SPECTRAL LEARNING OF RELEVANCE-ALIGNEE PROJECTIONS (OFFLINE)
Using the token-level key embeddings obtained from the aforementioned synthetic contrastive prompts (Section 2 and Appendix A). denoted h (neutral), h+ (positive), and h (negative),

---

Published as a conference paper at ICLR 2026
Highlighted Takens' Keys
Query
U
Key Projection
S
1f+
h
h
h
9
g
SEKA
g
AdaSEKA
Q²
Q
Edited Keys
SVD SVDD
U* U-
Attention Head
Figure 2: An overview of SEKA and AdaSEKA. context; h: key embedding; Q2: cross-covariance; U: left singular vectors; S: singular values; gi gain coefficient. SEKA applies fixed gains, while AdaSEKA uses the query compute dynamic steering weights.
compute cross-covariance matrices for each transformer layer and key-value head h: S2 = ht St = h'*= where n is the number of sampled tokens. Singular value decomposition (SVD) is then applied: 2 U,,SSt Ven Sig U St, Ven
In SVD, St and St,, represent the singular values of the positive and negative cross-covariance matrices, respectively. These singular values quantify the magnitude cross-covariance captured by each component the projection. The larger the singular value, the more significant the cor- responding singular vector (projection direction) is in explaining the cross-covariance between the token key embeddings.
In equation 2 for the positive projection Pe we use the top singular vectors corresponding to the largest singular values, which capture directions most associated with relevant (highlighted) features. For the negative projection Pi,,. we use the least-significant singular vectors, associated with the smallest singular values, to target directions least associated with relevance.
P U. (Uthack))
P
Uenck (UEna .).
where t and lo chosen such that they capture at least proportion y of the total singular value sum:
Σ ii Sgh, E Seh,i
2 %
27.
(3)
Edd St..
du
The threshold 7 hyperparameter that controls how much of variance the data we wish to retain when creating the projection matrices. By selecting the top k+ singular vectors for the positive covariance and & for the negative covariance, we capture the most relevant directions in the key embeddings for each type of projection. The learned projectors {P layer and head, enabling fine-grained steering at inference time.
i, P,,.) stored per
3.2 SPECTRAL EDITING FOR HIGHLIGHTED TOKENS (INFERENCE)
During inference, SEKA injects the learned projections into key embeddings before attention scores computed. For elarity, we omit the explicit (4, h) indices on key vectors k, and queries qi- although they are in practice layer- and hped-spe For each token key k, IRdx at layer and head h, the edited embedding is defined as:
P k +
k
k
(4)
where P.t.. P Rd the selected projection matrices and g+, g two independently adjustable scalars controlling the positive and negative steering gains. All vectors (e.g.. kj qo

---

Published as a conference paper at ICLR 2026
2) column vectors unless otherwise specified. This adjustment modifies the attention logits as equation 5 where qi E R is the i-th query vector. It is algebraically equivalent to augmenting the original attention score matrix A with a low-rank relevance bias matrix B:
g+ P
E. - .
q:
9. kj vd
2
Logits;;
Aij + Big.
(5)
vdk
Thus, SEKA can be interpreted as adding key-dependent term to the attention scores, amplifying each token's 's the directions aligned with the relevance subspace (detailed in Appendix C Unlike methods that directly manipulate the attention matrix, SEKA achieves equivalent modulation by edit- ing the key vectors themselves, offering a more structured and interpretable mechanism. Moreover, because SEKA operates entirely key representations prior to attention computation, requires access to or storage of the attention matrix, making it inherently compatible with memory-efficient implementations like FlashAttention.
3.3 VARIANT: QUERY-DRIVEN ADAPTIVE SEKA
While the standard SEKA framework provides effective token-level attention steering, practical de- ployment often requires hyperparameter tuning across different tasks and model families due the static projections. To address this limitation and reduce the need for manual configuration, we in- troduce Adaptive SEKA (AdaSEKA), which automatically selects and combines expert projections based on query-specific relevance signals.
Multi- Expert Projection Learning. We extend the projection learning framework accommo- date multiple domain-specifie experts. For each exper m E {1..... M. we constructed samples from datasets Dm for different tasks. Each expert learns its own set of positive SVD components {U mf m.E.i Vot.,, following the standard SEKA procedure. This process results in set of SVD components for each expert, layer, and head, which can be represented as 5D tensor (U+ E xxxxxxxxxexxe where L the number of layers, and H is the number of heads.
Expert Routing. At inference time, we extract the query vector 4r at layer and head of the last token the prompt, as the last token serves as the global aggregator of prompt information and hugely influences the downstream generation (Barbero et al. [0224] Q
ct al. 2024). We then compute dynamic coefficients that determine the contribution of each expert:
(k) +(k ΣK (q u
m.thh m.e.h
Om (g,,.)
Ek= (T,, u + (k) +(A)
maxm
m.eh)
m.C.A
+(k) where or m,t.h the corresponding k-th singular value, and K the number of top singular compo- nents used (lypically K 5).
This formulation measures how well the query aligns with each expert's 's main projection directions. weighted by their singular values. The denominator normalises by the largest absolute alignment across experts, which keeps the coefficients on a comparable scale and preserves whether the align- ment is positive or negative.
The final projection matrix at layer l and head h is constructed as a weighted combination of expert projections: P Pnynamic, f. (qr.n) = EM Om, (qr...(qr..)) - "...kkk (U+ k.k....:k) where U+ m.K.k...k denotes the first K columns U.....- m. corresponding to the most significant singular vectors.
This approach reconstructs projection matrices on-demand using only the top-K components, pro- viding computational efficiency whilst enabling automatic expert selection. The key transformation during inference becomes: k/- = k, g Paynamic, f... (qt,))k;
Crucially, AdaSEKA offers several practical advantages: (1) Reduced configuration effort: Auto- matic expert routing reduces the number of
tuning for different tasks and models
Experts can vary across task-specific datasets, such factual correction and instruction-followingg.
5

---

Published as a conference paper at ICLR 2026
Qwen2-1.7BBBal
Qwen? 48-Bass
Meed
Layer
Layer
Qrwen Base
Qwenn: 14B-Base
Mead
Stead
Marr
Layer
Layer
Figure 3: Heatmaps of the average per-token & distance between positive and negative key embed- dings across all KV heads and layers for four Qwen3 model sizes. Higher values (green) indicate greater separation between positive and negative key representations.
(shown in Appendix F). (2) Modular deployment: New experts can be integrated without recaleu- lating existing ones. (3) Interpretable routing: Expert selection is based on explicit
alignment scores. We derive four expert projections from four distinct datasets. The process of constructing data samples for learning these projections is detailed in Appendix
3.4 SELECTING SELEVANCE-SSESIIIVV KEY- VALUE HEADS
SEKA are most effective when applied selectively to KV heads that are naturally sensitive to prompt relevance. As demonstrated the qualitative visualisations in Figure I and discussed in Section 2 the key embedding for a given token span consistently shift in vector space when the question in the prompt changed from an irrelevant one relevant one. In this section, we formalise a method to quantify this relevance sensitivity across all layers and heads to inform our selection strategy.
Figure 3 shows the l distance between positive and negative key embeddings, averaged over all answer tokens from our synthetic dataset (as defined in Appendix (A). This variation is examined across different layers and heads of the Qwen3 model various sizes.
We observe that the distinction between relevant and irrelevant prompts is not uniform: larger norm values (green) consistently emerge in the mid-to-late layers, while early layers and subset of heads display minimal shift (red), suggesting the retrieval behaviour is less likely to happen those lay- ers. This finding is strongly aligned with recent mechanistic analyses. Michel et al. (2019); 1 Voita iet al. (2019); Clark et al. (2019); Neo et al. (2024); et al. (2023b) highlight that attention modules display various token-attending patterns across different heads. QQu et al. (2025) demonstrate that retrieval effectiveness relies only subset of attention heads, identified via probing and relevance filtering. Wu et al. (2025) further show that this sparse set of "retrieval heads" are almost exclusively located in the mid-to-late layers of the transformer. These heads intrinsic the base models, main consistent after fine-tuning, and are dynamically activated according to the context. Therefore, motivated by this alignment, we restrict projection to only those (layer, head) pairs where the empir- ical l difference between positive and negative key embeddings exceeds threshold. This selective approach ensures that attention steering concentrated on components empirically associated with retrieval behaviour, while leaving other heads unaffected. In this way, we amplify relevance signals only where necessary, minimising unintended influence on unrelated model components.
Formally, for each layer & and head h, let S denote the set of all answer tokens (across all sam- ples in the data), with S = N. The average per-token & distance is computed as Dr,
+ E h - hin. where hi,,, and h are the positive and negative key embeddings for token in S. Projection is applied only if Dr, p. òmins where òmin tunable hyperparameter tuned via grid search on a validation set (typically in [0, 0.6)..
4 EXPERIMENTAL SETUP
We consider SEKA particularly useful scenarios that require emphasis or highlighting within the prompt. This includes the tasks used to evaluate PASTA (Zhang al. 2024), which involve (i) han- dling complex user instructions (e.g., pronoun rewriting). (ii) interpreting lengthy and noisy contexts (e.g., Bias in Bios; De-Arteaga al. 2019), and (iii) resolving in-context knowledge conflicts (e.g..

---

Published as a conference paper at ICLR 2026
CounterFact; Meng et al. 2022). In addition, SEKA enables us to invert the typical U-shaped per- formance observed in the "lost in the middle" setting (Liu et al., 2024) by simply highlighting the middle of long contexts, thus improving model recall for these challenging positions.
4.1 STANDARD BENCHMARKS FOR ATTENTION STEERING
We follow the standard benchmarks used by PASTA, ensuring consistent selection of highlighted to- kens. Table | summarises the tasks, prompt formats, and evaluation metries. The CounterFact task is based on the COUNTERFACT dataset (Meng et al.) 2022), while the remaining two tasks (Bias in Bios, Pronouns changing) are derived from the BIASBIOS dataset (De-Arteaga al. 2019), in line with previous research (Zhang et al., 2024). We enhance the evaluation metric for the Pronouns changing task to address flaws in the original protocol which can misleadingly reward empty re- sponse, with the other metrics remaining consistent. Further details, including an introduction to each benchmark task and the calculation of metrics, are available in Appendix
Table I: Summary of standard benchmarks for attention steering, Tokens in bold indicate where attention steering applied.
Task
Prompts
Metrics
Counterfact
Previously, [old fact]. Currently, [new fact]. [question].
Efficacy score (ES), Para- phrase score (PS)
Bias in Bios
[person's occupation]. [career history, may not directly related to prediction]. [person] has the occupation of a/an
Accuracy (Acc.)
Pronouns changing
[biographical contexts]. Substitute 'she' and 'he" with 'they' and generate the occupation of [person] after changing pronouns.
Pronoun-weighted Lexical overlap Score (P. Score), All-changed P. Score
Benchmark Methods. We begin by using direct prompting of the original model as a baseline. Additionally, we include another bascline that incorporates ** marks around the highlighted context. For attention steering methods, ** is solely used to determine the token indices for steering and is re- moved from the input IDs. We then benchmark our proposed methods, SEKA and AdaSEKA, against the existing attention steering method PASTA. We also compare with Selective Prompt Anchoring (SPA) (Tian & Zhang 2025), a prompt highlighting method that operates on the logit distributions of the LLMs. Additionally, we evaluate SEKA with random projections applied and without the KV heads selector serve an ablation study.
4.2 U-SHAPE INVERSION IN THE LOST-IN- MIDDLE SETTING
To further examine SEKA 's ability to steer model attention to specific regions within a long con- text, we introduce an additional experiment targeting positional recall in the challenging lost-in- the-middle setting (Liu et al. 2024). This setting refers to the widely observed phenomenon where LLMs exhibit strong recall for information presented at the beginning and end of long contexts, but their performance substantially degrades when the relevant information is located in the middle, re- sulting in a characteristic U-shaped performance curve. Each of our inputs consists of a long context comprising 30 passages, where only one gold passage contains the true answer to a given question and the rest serve as distractors. The position of the gold passage is varied to test the model's posi- tional sensitivity. Each input is formatted as: "Context: \n (PI Title] \n (PI Text] [P30 Title] \n [P30 Text] An Question: ex( "question' ] \ Answer:".
Unlike prior work that aims to mitigate this effect, our objective is to directly investigate whether explicit relevance highlighting via SEKA can invert this U-shaped curve. By steering attention to- wards the middle passages, we test if the typical performance trough for mid-context answers be transformed into a peak, providing insight into the controllability of positional recall in LLMs.
Metrics. We exact match (EM) score as the evaluation metric, following Liu et al. (024)) a prediction is considered correct if it contains the ground-truth short answer span. To discourage verbose off-topic completions, the generated answer is limited to maximum of 60 tokens.

---

Published as a conference paper at ICLR 2026
Table 2: Performance standard benchmarks. Bold best. Underline second best. We include two ablation studies for SEKA: "w/o learn" " uses random projections instead of spectrally learned ones, and "w/o learn&filt" further removes the head filtering mechanism.
Baselines
Our Methods
Model Metric
Original
mm-areed
PASTA
SPA
SEKA
w/o learn
wlo learn&fiir
AdaSEKA
Qwen Owen3-8B Qwen3 444
CounterFact (ES)
45.00
$7.70
97.16
65.24
99.02
94.96
86.12
98,90
CounterFact (PS)
45.64
52.12
96.03
57.71
98.61
86.20
98.72
Bias Bios (Acc.)
79.84
82.94
89.58
68.00
91.02
86.62
71.76
91.86
Pronoun (P, Score)
93.14
95.76
95.82
80.27
95. 18
90.42
41 98
94.54
Pronoun (A. P. Score)
90.52
93.88
94.64
78.19
93.26
88.66
36.95
92.08
CounterFact (ES)
39.04
56.24
92.70
69.26
99.08
96.12
95.18
99.00
CounterFact (PS)
39.59
49.80
91.68
58.76
98.96
94.74
89.69
98.97
Bias Bios (Acc.)
76.08
80.60
86.32
37.02
88.74
87.26
74.90
$8.50
Pronoun (P. Score)
98.00
98.10
98.86
72.61
98.56
98.12
80.53
99.68
Pronoun (A. P. Score)
97.84
97.84
98.72
74.95
98.26
97.90
80.85
99.52
Qwenn: 14B
CounterFact (ES)
37.56
45.52
76.84
84.22
86.28
99,00
CounterFact (PS)
36.12
40.12
66.33
76.
99.02
88.
92.02
99.15
Bias Bios (Acc.)
85.22
90.94
88.46
57.86
90.28
88.02
91.22
Pronoun Score)
98.42
98, 86
90,98
91.60
98 66
96. 96.32
88, (0
99.88
Pronoun (A. P. Score)
98.22
98.68
90.94
92.20
98.54
96.36
89.76
99, 99.86
Gemmat- 128 Gemma 4B
CounterFact (ES)
55.04
$7.56
78.36
93.90
98.04
95.14
94.46
98.74
CounterFact (PS)
47.77
45.82
98. 83
92.25
98
99.05
Bias in Bios (Ace.)
89.90
91.00
82.58
48.02
92.42
85.60
77.16
92,92
Pronoun Score)
$1.78
93.76
Pronoun (A. P. Score)
35,25
32.45
66.43
74.45
81.11
48.82
$1.94
93.58
CounterFact (ES)
45.34
48.72
68.30
93.76
98.86
63.08
60.96
92.48
CounterFact (PS)
37.21
36.69
71.72
91.24
99,27
50.59
76.37
Bias Bios (Acc.)
91.26
92.90
94,72
46.88
93.04
91.84
90.54
91.14
Pronoun (P. Score)
93.92
95.78
68.47
86.41
97.70
47.26
55.56
96.88
(A. Score)
94.96
96.42
Benchmark Methods. We compare SEKA against standard baseline: directly prompting the base LLM without any intervention, and also PASTA. On top of this, we apply SEKA two config- urations: (i) steering only the middle region of the context (specifically passages 4 through 25), and (ii) steering all context passages. Although Hsieh et al. (2024) presents another potential baseline, exclude it due to the unavailability of its code implementation.
5
RESULTS
5.1
STANDARD BENCHMARKS: SEKA PROVIDES EFFICIENT ATTENTION STEERING
The main experimental results are presented Table 2
We tested the Qwen3 model YYagg et al. 2025) in various sizes, including 4B, 8B, and 14B, as well as the Gemma3 model (Gemma Team 2025 in sizes of 4B and I2B. For PASTA, we present its best performance from three confi configurations ensure robust comparison (see Appendix [ppendix[[]oor for full details). Furthermore, specific examples and the corresponding outputs from both the original model and SEKA are available in Appendix !
The results demonstrate that SEKA and AdaSEKA, are highly effective at steering LLM attention, generally outperforming both baseline models (ranked among the top two most of the time) and existing methods across various tasks and model scales. As demonstrated in Section these im- provements are achieved with significantly lower overhead compared to PASTA and SPA.
primary finding the efficacy of attention-level interventions on tasks requiring factual recall. On CounterFact, both SEKA and PASTA achieve near-perfect scores (e.g.. 99 0,02 and 97. 17.16 respectively for Qwen3-4B), validating the general approach of steering attention for knowledge conflicts, while the logit-based SPA lags considerably. Within this effective category, our methods consistently hold a performance advantage. This trend continues in the Bias in Bios task, where SEKA and AdaSEKA generally secure the top positions across all models.
Performance on the instruction- instruction-followingg Pronoun Changing task is strongly correlated with the base model's pretrained sensitivity to simple emphasis markers. For the Qwen3 family, which is partially responsive to simple markdown emphasis, the *** "**-marked" baseline is notably strong. This contrasts with earlier conclusions that LLMss are inherently restricted to processing plain text without
8

---

Published as a conference paper at ICLR 2026
stylistic emphasis markers (Brown et al. 2020 Wei al. 2022 However, AdaSEKA
still provides further improvement, delivering SOTA performance (e.g.. an A. P. Score of 99.52 on Qwen3-8B). The advantage of our methods is most pronounced the Gemma3-4B which is less responsive to the markdown emphasis. This demonstrates our method's significant value, especially for smaller models that are less receptive to basic emphasis grammar.
Finally, our ablation studies validate the method's core components. Using random projections with head filtering (w/o learn) proves beneficial but clearly suboptimal, underscoring the value of our spectral learning approach. Removing both the learned projections and the head-filtering mechanism (w/o learn&filt) causes a catastrophic decline performance. For instance, on the Qwen3-4B Pronoun task, the A. P. Score drops from the original 90.52 to 36.95, This conclusively demonstrates that both learning meaningful relevance subspaces and selectively applying them to the appropriate KV heads are essential for success.
5.2 LOST IN THE MIDDLE
With the setting described in Section 4.2 we highlight two key findings when benchmarking SEKA against baselines and exploring the impact of different minn for selecting KV heads.
Original Model - PASTA (midde)
-ASTA SEKA (nidde) SEKA (all)
Qwen3-4B-BBl
Qwen3-SB-Bast
Qwen3-14BBBal 1-----1---BBse
0.5
08.
Exact Match Saare
Stotr
0.65
0.7
04
0.5
0.6
0.55
0.3
0.4
0.5
0.2
0.
0.45
0.1
0.4
54h
100
150
2011
2500
300h
161
50
10th
150h
20th
255
30th
1s1
50
100
155
200h
2,50
20th
Gold
Position
Gold
Position
Figure 4: Exact match scores on the lost-in-the- middll task for Qwen3 models of three different
sizes, comparing the original model, PASTA/SEKA applied to the middle region (5th to 25th pas- sages), and PASTA/SEKA applied to all passages.
SEKA Can Invert the U-shape Performance. The re- sults, summarised in Figure 4 reveal two primary find- ings. First, applying SEKA selectively to the middle pas- sages (positions to 25, which is a very rough range) highly effective at inverting the canonical U-shaped performance profile: exact match scores at central posi- tions substantially increase, eliminating the typical per- formance trough for answers located in the middle of long contexts. Second, applying SEKA uniformly across all passages can slightly exacerbate the lost-in-the-mine sue. The most noticeable improvements typically occur at the beginning the middle are less pronounced or may even decrease. In contrast, PASTA is less effective for this task. Applying it to either the middle region or the entire context results in performance generally below the original baseline across all model sizes.
-Origial Model
SEKA (Amin--0 16)
-EEKA (6m1s=0.165)
SEKA (ômin-0.17)
SEKA (Smin=0.18)
Qwen3-8BB Bwen3-8B-Base
Exact Match
0.5
0.45
1st
5th
10th
15th
20th
25th
30th
Gold Answer Position in Context
is-
Figure 5: Exact match scores when ap- plying SEKA to the middle region with different threshold 6min-
end positions, while enhancements in
SEKA Can Mitigate and Flatten the U-Shape When Applied to Appropriate Number of KV Heads. In this control experiment, we fix the positive and negative steering gain coefficients (g" and 9 at 0.2 and 0.1 respectively, and vary only the threshold òmin to control the number of steered KV heads. In practice, decreasing &min increases the number of steered heads: for example, thresh- olds of 0.16, 0.165, 0.17, and 0.18 correspond to SEKA being applied on 58, 48, 41, and 31 KV heads for Qwen3-8B-Baal (around 0.165 and 0.17) and steering the middle region, SEKA can flatten the U-shaped performance curve without significantly compromising accuracy at the beginning and end positions. Note that the optimal threshold may vary with model size. Complete results for the 4B and 14B models are provided in Appendix K
respectively. As shown in Figure 5 with an appropriate threshold dmin

---

Published as a conference paper at ICLR 2026
6 OVERHEAD ANALYSIS
A key advantage of our pre-computation approach is its compatibility with optimised mech-
anisms like FlashAttention (Dao et al.] 2022; Dao 2024, Shhhh et al. 2024). We quan-
tify this by measuring inference overhead on 1O0 samples (avg. 4362 tokens) from
Section [22 using a Qwen3-8B-Bas As shown in Table [] the overhead for SEKA is negligible (+0.03s per sample). This efficiency is particularly notable as, for a fair comparison with PASTA, we use an aggressive configuration that steers 175 out of 288 available KV heads. In con- trast, post-hoc methods incur significant costs. PASTA's reliance on editing the full attention matrix makes it incompati- ble with FlashAttention, leading to sub- stantial increase in latency (+1.03s) and memory usage (+23.12 GB). SPA, while
model on a single NVIDIA-GH200-120GG GPU.
Table 3: Inference overhead on Qwen3-8B-Ban Time is per-sample; memory average peak usage.
Avg. Time (s)
Peak Mem. (GB, B=10)
Peak Mem. (GB, B=1)
Method
Original PASTA
0,55
27.63
16.72
1.58 (+1.03)
50.75 (+23.12)
SPA
5.87 (45.32)
17.71 (+0.99)
SEKA
0.58
(+0.03)
27.66 (+0.03)
16.75 (+0.03)
AdaSEKA
0.82 (+0.27)
43.22 (+15.59)
18.23 (+1.51)
memory-efficient for single samples, does not support batch processing and is thus the slowest over- all. Our adaptive variant, AdaSEKA, introduces moderate overhead for its dynamic, query-aware capabilities (+0.27s). However, remains significantly more efficient than both PASTA and SPA, making it far more practical option for steering in long-context scenarios.
7 RELATED WORK
Research on steering large language models falls into two main paradigms. Activation Steering (Dathathri et al. 2020. Subramani et al., 2022 Hernandez et al.) (2024) guides high-level semantic outputs by intervening in MLP layers, while Attention Steering, the focus of our work, directs the model's s focus to specific tokens within the input prompt.
Activation Steering. This line of work, also known as representation engineering, adds "steering vectors" MLP layer activations control semantic attributes (Zou et al., 2023). Applications
include enhancing honesty and safety (Ravfogel et al., 2020; Burns et al. 2023 Iskander et al.)
2023
[i et al. 2023a Wei et al. 2023 Bhattacharjee et al. 2024
2024|Qiuue QQu et al. 2024). controlling style
Turner et al. 2023 2024), improving reasoning (Tang et al. 2025), and knowledge editing (Fang)
ct al. 2025). Recent studies suggest these methods only work when the model already knows the target knowledge (Simhi al. 2025). These methods are therefore different from our approach as they change what the model knows through its hidden states, but we control where the model looks via its attention mechanism.
Attention Steering. To address the challenge of LLMs failing to attend to key information in long contexts (Liu et al. 2024 Meng et al. 2022 prompt highlighting methods intervene post-hoc
on either the attention scores (Zhang et al. 2024) or final logits (Tian & Zhang 2025). However,
these interventions often introduce significant latency; for instance, editing the full attention matrix
is incompatible with modern optimisations like FlashAttention (Dao et al. 2022 Dao 2024 Shah
et al. 2024). This efficiency bottleneck motivates the need for pre-computiiion alternatives that can steer attention without sacrificing compatibility with optimised architectures.
CONCLUSION
In this paper, we introduced SEKA and its adaptive variant, AdaSEKA, new class of training free attention steering methods that operate by modifying key embeddings before the attention computa- tion. This pre-attention approach overcomes the core efficiency limitations of prior work, ensuring full compatibility with optimised implementations. Our experiments confirm that both methods achieve state-of oftte-tf results on a range of standard benchmarks, with AdaSEKA's 's query-adaptive routing demonstrating particularly strong performance. These gains are achieved with negligible overhead, making our work a practical step towards building more controllable and efficient LLMs for long-context applications.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our research, all necessary materials have been made publicly available on httlttts/gggthhii
com/ http://wlooon/SSaa. This repository includes: (1) the full source code for our proposed methods, SEKA and AdaSEKA; (2) detailed instructions for running all the experiments; (3) the pre-computed projection matrices used in our evaluations; and (4) the pre-processed versions of the datasets.
The original datasets used in our evaluation are publicly available and are cited in Section 4 Specif- ically, the BIASBIOS, COUNTERFACT, and "Lost in the Middle" datasets all distributed under the MIT License. Details regarding the evaluation samples and metrics calculation are provided in Appendix B while hyperparameters are specified in Appendix F
ACKNOWLEDGEMENTS
We thank the reviewers and the area chair for their valuable feedback. We also thank Yifu Qiu for constructive discussions related to this project. The authors acknowledge the use of resources provided by the Isambard-Al National AI Research Resource (AIRR). Isambard-Al is operated by the University of Bristol and funded by the UK Government's 's Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/-A--11023 (Melntosh-Smint et al. 2024).
REFERENCES
Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João Guil- herme Madeira Araújo, Oleksandr Vitvitskyi, Razvan Pascanu, and Petar Velickovic. Trans- formers need glasses! Information over-squashing in language tasks. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http:: nips. ccpppppppcppppppppppppppp..eeeee https///cccppp....ee//////eee/// fites//pappee//2222///////////// /
b 2L555ccc600000002ttaaaaaaa-aaaa3
Amrita Bhattacharjee, Shaona Ghosh, Traian Rebedea, and Christopher Parisien. Towards inference- time category-wise safety steering for large language models. NeurlPS Safe Generative Al Workshop 2024, 2024. URL httttttss///pereeeeeeee wtt.:/wu.et//rrmmmt/em/ idd 1d=EKQRNLPFcn
net / forum?
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Proceedings of the 34th International Conference on Neural Information Processing Systems, NeurIPS 2020, Red Hook, NY, USA, 2020. Curran Associates Inc ISBN 9781713829544
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. In The Eleventh International Conference on Learning Rep- resentations, 2023.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? An analysis of BERT's attention. Proceedings of the 2019 ACL Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 276-286, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-442
URL https:
7/ //aclantholooy.org
Tri Dao. FlashAttention--2: Faster attention with better parallelism and work partitioning. In
The Twelfth International Conference on Learning Representations, 2024. URL lhttps://
oopenreviww... hwtp://eettftteeee htt.rrrummmmmmmmmmm22xxx...5

---

Published as a conference paper at ICLR 2026
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAtten- tion: Fast and memory-efficient exact attention with io-awareness. In Advances in Neu- ral Information Processing Systems 35: Annual Conference
Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http:: nnps cwwpsppccppppepeppppppppppppp www.s//eaasseeeee $74576322600Euaaaa3022222 wtt.:/ccbbd36640000 1d36e40d5 Abstract wonference.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: simple approach to con- trolled text generation. In International Conference on Learning Representations, 2020. URL
net/ forum? wewpummmidHHeedeeee HledEyBKDS)
Maria Dr-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT '19, pp. 120-128, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 978 3781450361255 doi: 10.1 1287560.3287577
0e.....1111118888 3287572
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/ttansformrrcirrruiit http://issippub22////////oo/////
Junfeng Fang, Houcheng Jiang, Kun Wang. Yunshan Ma, Jie Shi, Xiang Wang. Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained model editing for language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openreview
Gemma Team. Gemma 3 technical report, 2025. URL http:://arxxvv.org//bb///
19786
Evan Hernandez, Belinda Z. Li, and Jacob Andreas. Inspecting and editing knowledge rep- resentations in language models. In First Conference on Language Modeling, 2024. URL htttps://openreviie wwwpenrweeeeeeeeeeeeeeeeeeeeeeee wttt:/ooum..--DDD1LE......
Cheng-Yo Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Plister. Found in the middle: Calibrating positional attention bias improves long context utilization. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 14982-14995, Bangkok, Thailand, findiing--aa...800 URL
2024. Association for Computational Linguistics. doi: 10. org. 2024. findings-acil 890/
Shadi Iskander, Kira Radinsky, and Yonatan Belinkov. Shielded representations: Protecting sen- sitive attributes through iterative gradient-based projection. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 5961-5977, Toronto, Canada, 2023. Associa- tion for Computational Linguistics. doi: 10.18653/v1/2223
http:///.22.....ngggseeeeee findings-acl.369. URL https:
//aclanthology.orgg
1.2.3.finne findings aeadinggsacl..3. 369]
Kenneth Li, Oam Patel, Fernanda B. Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. In Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NewrIPS 2023, New Orleans, LA. USA, December 10 16, 2023, 2023a. URL 81
nips. wwtppperrcpppppppppppppppp www.ffllssaaaeee pttee///pppprr/////aaaaa html
Weixian Waylon Li, Yftah Ziser, Maximin Coavoux, and Shay B. Cohen. BERT is not the count: Learning match mathematical statements with proofs. In Andreas Vlachos and Isabelle Au- genstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Associ- ation for Computational Linguisties, pp. 3581-3599 Dubrovnik, Croatia, May 2023b. Asso-
12

---

Published as a conference paper at ICLR 2026
ciation for Computational Linguistics. doi: 10.18653/w1/20233.aad http::/ww11223...cammmm.......00 m60........maa........ //aclanthology..... wwtp:yy..g//oa.eoo/ 2:tg:2223.eaclmma..22555
URL https:
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilaequa, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157- 173, 2024,
doi: 10.1162/tacl a_06638. URL
htttps:/acclantho http://looooooooooooooo. 20rg/2024..0 1ac44aaacl111
Simon Mclntosh-Smith, Sadaf Alam, and Christopher Woods. Isambard-ai: a leadership class supercomputer optimised specifically for artificial intelligence, 2024. URL https://arxiv.
org/abs/2410. 11199
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associ- ations in GPT. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, November httpppppppppppppppeeeeee 28 December 9, 2022, 2022. URL http: wwwps/ppppppppeeee n iss cc/ haccc/pppr____00 faw.fflleppeo esappper
2022/hash/S5 3022 Te oo html
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Ad- vances in Neurul Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 14014-14024, 2019. URL tttp:://proeeedngg..oo http://rrrrcceeeneeeeeeeaa.aa..p ct/ cctppcprpppppsp////c///pp hash/2c60 add.001ddddd2f.. www.999922222000055555555555
html
Clement Neo, Shay B Cohen, and Fazl Barez. Interpreting context look-ups in transformers: In- vestigating attention-MLP interactions. Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Procesy- ing, pp. 16681-1666 Miami, Florida, USA, November 2024. Association for Computational org/
Linguisties. doi: 18.18..3/v 1tt3://112224.mm///// wawallpmmii.999.. main.930. U 2024. emllp main. 930/
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann. Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022. httphhttts:::axxiv...
abs 22222229... 11895
Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo Maria Ponti, and Shay B. Co- hen. Spectral editing of activations for large language model aligmment. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Informa- tion Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, hash/
2024, 2024. URL http:/ http:://papers.nip... 684c59d614 feeae www.eeeeeeeea4eeeeeeeee
Yifu Qiu, Varun R. Embar, Yizhe Zhang, Navdeep Jaitly, Shay Cohen, and Benjamin Han. Elicit- ing in-context retrieval and reasoning for long-context large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Asso- ciation for Computational Lingwistics: ACL 2025, pp. 3 3176-3192, Vienna, Austria, July 2025. doi: 10.18653/ org/ findings-acll..... 1657
Association for Computational Linguistics. ISBN 811-8-8-89767---- aadinng-acl.165.. htRlss/tt::// httpaaaaaanntcooooooo httpaa/sssS..tttttt//a/aaaoooooa
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out:
Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th An- nual Meeting of the Asxociation for Computational Linguistics, pp. 7237-7256, Online, 2020. URL htRpst::::
Association for Computational Linguistics. doi: 10.
#/aclanthologes wwo.gy..org//// 2000.2220..00 acl main. 647
Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. FlashAttention--: Fast and accurate attention with asynchrony and low-precision. In Ad- vances in Neural Information Processing Systems 38: Annual Conference on Neural Infor- mation Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10
13

---

Published as a conference paper at ICLR 2026
2024. (X2e777680222EE00000303aaaaaa0.-
Adi Simhi, Jonathan Herzig. Itay Itzhak, Dana Arad, Zorik Gekhman, Roi Reichart, Fazl Barez, Gabriel Stanovsky, Idan Szpektor, and Yonatan Belinkov. HACK: Hallucinations along certainty and knowledge axes, 2025. URL wtt.nttpss////rii...rr///sssas
24222
Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, and Besmira Nushi. Im- proving instruction-followiin in language models through activation steering. In The Thirteenth International Conference on Learning Representations, 2025. BtLhttps: //tp:s/////pe.eeee..
net/ Sim.mmiddwwwae wdwhhdnRRcc..
Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting latent steering vectors from pretrained language models. In Findings of the Association for Computational Linguis- fics: ACL. 2022, pp. 566-581, Dublin, Ireland, 2022. Association for Computational Linguis- tics. doi: 10. 1161181111/12222 httd://////2222.nnnnn------ 4c1.48. URL /tLL:ttts///aalaahhll.oo org.2222....//
E Eindings acl... 48
Xinyu Tang, Xiaolei Wang, Zhihao Lv, Yingqian Min, Xin Zhao, Binbin Hu, Ziqi Liu, and Zhiqiang Zhang. Unlocking general long chain-of- thouugh reasoning capabilities of large language models via representation engineering. In Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
Vienna, Austria, 2025. Association for Computational Linguistics. ISBN 979 979-8-89176-25--0 doi: 10. 10.18653/1/2025. acl-long.339. URL URL htttp://:/a/a anthologoy... wttp:yy..rggg2200 aacl wow.ellloonnnoooooooooeooooooo ong 339/
Yuan Tian and Tianyi Zhang. Selective prompt anchoring for code generation. In Forty-second International Conference on Machine Learning, 2025. URL htttttts///ppeeeeeeeee
net./
Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte Mac- Diarmid. Activation addition: Steering language models without optimization. ArXiv preprint, abs/2308. 10248, 2023. /Rt/httpss///:xxi.orrr wtt.oorg/abbb.s..aae 10248
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, 2024. URL
https://arxixs org/abs//////ooooooooo/.oooo.o 2308 10248
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5797-5808, Florence, Italy, 2019. Association for Computational Linguisties. doi: 10.18653/v1/119-155 URL https::/aclaa
Tianlong Wang, Xianfeng Jiao, Yinghao Zhu, Zhongzhi Chen, Yifan He, Xu Chu, Junyi Gao, Yasha Wang, and Liantao Ma. Adaptive activation steering: A tuning-free Ilm truthfulness improvement method for diverse hallucinations categories. In Proceedings of the ACM on Web Conference 2025, WWW '25, pp. 2562-2578, New York, NY. USA, 2025. Association for Computing Machinery. ISBN 9798400712746 doi: 10.1145/3696410.77 11..444411.....40... URL
3714640
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10- 16. 2023, 2023. URL httptttt:::/ppssspssppes nips ctpppppepcpppppeppppppppppppp ffllss/paaert222//asst. 116613131883941152222666 http:///c....aaacoooooaaaa-aa--- Cabstract-Conffeeenee.httt html
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe Le. and Denny Zhou. Chain-of-thought prompting elicits reasoning in large lan- guage models. In Proceedings of the 36th International Conference Neural Information Processing Systems, NeurlPS 2022, Red Hook, NY, USA, 2022. Curran Associates Inc ISBN 9781713871088.
14

---

Published as a conference paper at ICLR 2026
Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanis- tically explains long-context factuality. In The Thirteenth International Conference on Learning Representations, 2025. URL hctps: http:::://opereeieww. www.eeewweeeeeeeeee. htte//ooummmiee htt.::t.tttttmmmmaedyyyyay-e.a/.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo. Tianhao Li, Tianyi Tang, Wenbiao Yin. Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: / wwt.iv.vvvvv.//..a//..aaaaaaavva org/ abs 2505 09388
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-23880 Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/D188 1259.
Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, and Tuo Zhao. Tell your model where to attend: Post-hoc attention steering for LLMs. In The Twelfth Interna- tional Conference on Learning Representations, 2024. URL https: httppes///ppnrviee.... net/
forum?id=x2DWOn
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang. Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down ap- 01405
proach ai transparency, 2023. URLitttps httts:://axxvv.rgg/
A SYNTHETIC DATASET FOR TOKEN-LEVEL RELEVANCE SUPERVISION
To supervise attention steering, we construct a synthetic dataset that enables precise control over teken-level relevance. Each sample comprises two contexts (C1, C2), each paired with a question and answer tuple (Q1, Ai and Q2, A2). This structure allows to define relevance by contrasting identical token spans across different query contexts.
Table 4: Constructed prompt triplets for both answer spans. Each group provides neutral, positive, and negative variant based on question-contex alignment.
Group
Prompt
Neutral
Context:
The portfolio manager allocates capital across equitics and bonds.
Positive
Question: What does the portfolio manager allocate across equities and bonds?
Context: The portfolio manager allocates capital across equities and bonds.
Negative
Question: What does the climate model simulate?
Context: The portfolio manager allocates capital across equitics and bonds.
Neutral
Context: The climate model simulates sca-level rise under different scenarios.
Positive
Question: What does the climate model simulate?
Context: The climate model simulates sea-level rise under different scenarios.
Negative
Question: What does the portfolio manager allocate across equities and bonds?
Context:
The climate model simulates sea-level rise under different scenarios.
With every pair of (C, Q, A) triplets, as shown in Table 4 for the answer span "capital" in C1, and another for the answer span "sea-level rise" in Cz-
we can derive two supervision samples:
15

---

Published as a conference paper at ICLR 2026
Table 5: Synthetic data instance.
Context 1 (Ct)
The portfolio manager allocates capital across equities and bonds. The climate model simulates sea-level rise under different scenarios. What does the portfolio manager allocate across equities and bonds? capital What does the climate model simulate?
Context 2 (C2)
Question 1 (Q1)
Answer (A1)
Question 2 (Q2)
Answer (A2)
sea-level rise
For each answer, construct three variants: (1) a positive (relevant) prompt where the question and context are aligned (e.g.. Q1 for C1, and Q2 for C2). (2) a negative (irrelevant) prompt where the question mismatches the context (e.g., Qi for C2. and Q2 for Ci), and (3) neutral prompt containing only the context. This allows us to collect three classes of key embeddings for the answer spans within the context: h+ for positive, h- for negative, and h for neutral. In Figure 1 empirically show that, for some key-value heads, different token spans exhibit a consistent shift in their key embeddings from negative to positive variants. This validates the construction and use of these relevance supervision signals.
Practical construction details. The synthetic dataset is lightweight to produce. We use a fixed template shown in 5 and automatically prompt GPT-4o to produce contrastive samples, using the prompt provided in Figure This process requires no manual annotation. After collecting the generated samples, convert them into ISON format for subsequent
B MULTI-EXPERT PROJECTION LEARNING SAMPLES FOR AdaSEKA
After constructing the synthetic dataset, prepared three additional task-specific datasets, making a total of four, for multi-expert projection learning (Section 3.3). As shown in Table 6 each sample consists of neutral and positive prompt pair. For the Counterfact (Meng al., 2022) dataset and the BiasBios (De-Arteaga et al. 2019) datasets, these pairs are collected from their respective training sets, following the original prompt templates outlined in Table II] For each sample, we extract the key embeddings for the answer spans directly from the context. A distinct procedure adopted for HotpotQA (Yang et al. (018) to account for its multi-hop nature. The context formed by concatenating all candidate paragraphs, and the key embeddings from all supporting facts are subsequently extracted and concatenated. Each expert projection is learned from set of 200 randomly sampled instances from the training set for each task, using fixed random seed of 42 to ensure reproducibility.
C GEOMETRIC INTUITION OF THE SEKA TRANSFORMATION
To provide geometric insight into the effect of SEKA's key editing, consider the case where the projection matrix P is given by UUT, with U E Rd having orthonormal columns that span the relevance subspace (i.e., U = -+ or U- as previously defined). For simplicity, assume g = 1 and focus solely on the positive (or negative) projection. The transformation then becomes:
k = (I + UUT k
Any vector E Rdd can be decomposed as
æ 1 + LL
where x1 = UU Φ, 2 #1 -
(8)
This decomposition is orthogonal. Specifically,
x æ (UUT x) T UUT x) x: -- -UU
x UU -- T0U = 0,
(10)
using the idempotency of the projection (UUT)² UUT).
16

---

Published as a conference paper at ICLR 2026
Synthetic Samples Generation Prompt
We are collaboratively generating a total of 100 synthetic examples.
You will generate examples in batches of exactly 20 per response. Across ALL batches in this conversation, every example must globally unique.
Before generating a new batch, you MUST:
1. Review ALL previous examples in the conversation.
2. Ensure no repeated entities, contexts, events, sentence structures, questions, or answers.
3. Ensure near-duplicates, paraphrased duplicates, re-themed duplicates.
Each example must follow this structure (exact formatting):
Example k:
Context <C1>
Context 2: <C2>
Question I: <Q1>
Answer I: <A1>
Question 2: <Q2>
Answer 2: <A2>
Generation requirements:
Each context must be one concise fictional sentence.
-Cl and C2 must be semantically unrelated.
-Q1 must ask about span that appears verbatim in Cl.
Q2 must ask about span that appears verbatim C2.
Al and A2 must be exact substrings (contiguous spans) of CI and C2.
No context, entity, theme, setting, question type may repeat across any batch.
Avoid any resemblance to earlier examples in wording, structure, or domain. No extra explanation or commentary.
Your task now: Read all previous examples in the conversation so far. Then generate the next 20 completely new, globally unique examples. Stop after exactly 20 examples.
Figure 6: Prompt template used to generate synthetic contrastive examples.
Applying the transformation, have
(I (I+UU = = UU = (u-UUT x)) -U= - TUT
(11)
-U-UUUZ+UUUT
2
(12)
which shows that the component along the subspace is amplified (doubled), while the orthogonal component remains unchanged.
While the case offers geometric clarity, the result generalises for any R:
(I + -UUUUUT x æ1 + (1 + g)æf
(13)
Thus, the relevance-aligned component scaled by (1 + g), while all orthogonal directions are preserved. This operation is neither projection linear modification that selectively amplifies directions aligned with the relevance subspace. SEKA leverages this property to boost relevant token features in controlled and interpretable manner, enabling precise, token-wise attention steering without interfering with unrelated components.
orthogonal transformation, but targeted
While the geometric interpretation above clarifies how SEKA amplifies components of key vectors aligned with a learned relevance subspace, it is important to clarify what this subspace represents. SEKA is not intended to encode manipulate semantic meaning. Its effect is deliberately confined
17

---

Published as a conference paper at ICLR 2026
Table 6: Constructed prompt pairs for multi-expert projection learning. Each dataset provides tral and positive variants based on question-context alignment.
Dataset
Variant
Prompt
CounterFact
Neutral
Previously, The mother tongue of Danielle Darrieux is French. Currenty, The mother tongue of Dunielle Darrieux is English.
Context:
Positive
Question: Danielle Darrieux, native ?
Context:
Previously, The mother tongue Danielle Darrieux is French. Currentl, The mother tongue of Danielle Darrieux English.
BiasBios
Neutral
Context: Nora assistant professor of international relations Bahcesehir Uni- versity Istanbul. She is also Ronald D. Asmus Policy Entrepreneur Fellow with the German Marshall Fund and is Visiting Fellow the Centre for International Studies (CIS) at the University of Oxford. This commentary first appeared at Suda, an online journal published by the Carnegic Endowment for International Peace.
Positive
Question: Nora has the occupation of a/an ?
Context: Nora is assistant professor of international relations at Bahcesehir Uni- versity in Istanbul. She is also Ronald D. Asmus Policy Entrepreneur Fellow with the German Marshall Fund and is Visiting Fellow at the Centre for International Studies (CIS) at the University of Oxford. This commentary first appeared at Sada, online journal published by the Carnegie Endowment for International Peace.
HoipotQA
Neutral
Context: Radio City (Indian radio station) Radio City is India's first private FM radio station and was started July 2001. Arthur's Magazine Arthur's Mag- azine (1844-15 1846) was an American literary periodical published in Philadelphia in the 19th century, Edited by T.S. Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Surah Josepha Hale, Thomas G. Spear, and others. In May 1846 it was merged into "Godey's Lady's 's Book".
First for Women First for Women is womans magazine published by Bauer Media Group in the USA, The magazine was started in 1989. It is based in Englewood Cliffs, New Jersey. started first as a denim line, later evalving into men's 's and women's 's clothing line.
The company
Positive
Question: Which magazine was started first Arthur's Magazine or First for Women? Context: Radio City (Indian radio station) Radio City is India's first private FM radio station and was started on 3 July 2001...- Arthur's ' Magazine Arthur's Mag- azine (1844-1846) was an American literary periodical published in Philadelphia in the 19th century, Edited by T.S. Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others. In May 1846 it was merged into "Giodey's Lady's Book". woman's magazine published by Bauer Media Group the USA. The magazine was started in 1989. It hased in Englewood Cliffs, New Jersey. ... The company started first as a denim line, later evolving into men's 's and women's 's clothing line.
First for Women First for Women is
to the attention to route subspace of the transformer, consistent with prior mechanistic analyses (Blhage al. 2021 Olsson et al. 2022).
Modern transformer-circuits work decomposes the action of an attention head as
(R) (A) @ W R,
(14)
where (A) is the query-key similarity tensor governing which tokens attend to which, and (h(h) w-(h) writes attended features into the residual stream (Elhage et al. 2021 This formula- tion explicitly separates routing (Q/K) from semantic write operations (V/MLP).
Further, studies of induction and retrieval heads (Olsson et al. 2022 show that Q/K vectors imple- ment token- matching and algorithmic routing behaviour, such as copying and continuation, while semantic information is primarily stored in value vectors and MLP activations. These findings align with our design that SEKA aims to modify only the routing (relevance) subspace, leaving the seman- tic subspace untouched.
18

---

Published as a conference paper at ICLR 2026
D SEKA AND AdaSEKA ALGORITHMS
We provide detailed pseudocode for our proposed methods, SEKA and AdaSEKA. Algorithm de- tails the standard SEKA method. It involves an offline phase to learn fixed positive and negative projection matrices from contrastive data using SVD. During inference, a hook then applies these static projections to the key embeddings of highlighted tokens. Algorithm describes descrribs the more flex- ible AdaSEKA framework. In essence, standard SEKA can be viewed special case of AdaSEKA with single expert and no dynamic coefficient calculation. AdaSEKA generalises this by loading bank of multiple expert SVD components offline. For each new prompt, it then performs a dynamic, query-awer crepoomputation: it calculates routing coefficients based the query's alignment with each expert and constructs a bespoke projection matrix on-the- fly. This tailored projection is subse- quently applied during generation via the key-editing hook.
Algorithm Spectral Editing Key Amplification (SEKA)
Require: Triplets {h, ht, h he, variance threshold gains g², Ensure: Projections {P 2. Pe,.. and key-editing hook
1: for all layer i and head / do
2: 2 h ht. S2 n h 3: (U E.h Veht SVD(( (UTN SE. Ven t SVD(S2
n
4: A+ min Ei=1 St..il E: St. 2 7) k t min E Stasl E. ST,, 2 }}
5:
P C.h.s
U+T Cha.tt Pe + T.,-Ui U.nKk-U. Uen,
6: end for
7: Hook applied to each selected h2 (registered per layer E; é is fixed within the hook).
8: Input: A E R BxTx Hxd. mask m
omit the explicit layer index on K; projections remain P 10: for 1. t=1.,, h=1... do
9: Note: For brevity
11:
mm.:=11 then
12:
A- P K h
P2. Kb, h, /2
13:
+A
return K to the attention computation
15:
Register the hook for selected (f, before generation and remove it afterwards.
E DETAILS OF STANDARD BENCHMARKS
We evaluate our method on three established benchmarks adapted from the PASTA frame- work (Zhang et al.) 2024). We introduce sigmificant improvements to the evaluation protocols, such as case-insensitive scoring, to ensure a more robust assessment. The JSON Formatting task was omitted modern models achieve near-perfect performance, rendering less useful for discrimi- nating capabilities.
E.1 COUNTERFACT
The COUNTERFACT benchmark (Meng al., 2022) evaluates an LLM's ability to prioritise new contextual information over its pre-trained knowledge. Here, each fact is represented as a sub- pect-relatinnoojeeee triple os) where s denotes the subject entity, r the relation, and the ob- ject.
Task Format. The model receives input structured as: "Previously. {s Ould). Currently, {s T Onew]- {question)." The challenge arises because models often default to pre-traines asso- ciations rather than attending to the new, contradictory information provided in the context.
19

---

Published as a conference paper at ICLR 2026
Algorithm 2 Query-Driven Adaptive SEKA (AdaSEKA)
Require: SVD components {U m. Sth m.e.A } for M experts, top components K, gain
Ensure: A key-editing hook using dynamically computed projections
Store expert SVD components {Um,,, m,t.h* m.e.a. for all experts m, layers L, and heads h.
S
2: For given prompt with input IDs I:
3: Obtain last-token query vectors yr.h for each selected layer l and head h.
4: for all selected layer and head h do
5: for all expert m = 1..M do
+(K) +(k) 6: Calculate coefficient am,E, & Ekmi (g Um,hhh) m. (as per Eq. 6)
7: end for
8: Construct Paynamic,t.t + >m== Omm.e. (q.,))) U M.K.KaaKK K (U m.t.bask))
9: Store Pay.amic,e.h for use in the hook.
end for
11: Hook applied to each selected (c, h) (registered per layer E; é is fixed within the hook).
Input: K ER Bxxxx mask
13: Note: For brevity we omit the explicit layer index on K.
for b= t=1.,T, h=1..H do
15:
if m then
A A-g faymamic.f.h K(b, L, [. K t, h, + TajjKKbbb b, h, + A
17:
18:
return K to the attention computation
19:
Register the hook for selected (e, h)) before generation and remove it afterwards.
Prompt: "Previously, Kevin Garnell is professional basketball player. Currently, **viv Garnett is professional baseball player**, Kevin Garnett is professional
The model should generate "baseball player" rather than its pre-trained association of "basketball player".
Target:
Evaluation Metries. Following Zhang al. 2024). to evaluate the model's ability to recall the new fact, we measure its internal preferences at the point of generation, rather than relying on parsing free-form text. For given prompt, we provide the model with the entire context and question, and then assess the log probabilities it assigns to the potential next tokens.
Efficacy Score (ES): This metric directly measures if the model prioritises the new, correct fact (Onew) over the old, incorrect fact (Oola). It is the percentage of times the model assigns higher probability to the first token of the fact than the first token of the old fact. A high ES indicates that the model has successfully updated its belief based on the context.
ES
[[PLLM (o
(o
Paraphrase Score (PS): This metric measures generalisation by calculating the average Efficacy Score across a collection of human- written paraphrases of the original question.
E.2 BIASBIOS
The BIASBIOS dataset (De-Arteage et al. 2019) consists of biographies and was originally designed to explore gender bias in occupation prediction. The first sentence of each biography explicitly states the person's occupation, while subsequent sentences provide potentially distracting career details.
Task Format. Each biography is appended with the prompt " "[person] has the occupation of and the model must predict the correct occupation from list of 28 candidates.
20

---

Published as a conference paper at ICLR 2026
"inWinnie is an American photographer living in New York. Specialized in fash- ion photography and portrail, she applies her talent on both humans and animals... Winnie has the occupation of"
Prompt:
Target: "photographer"
Evaluation Metries. We measure standard top-1 Accuracy across the 28 candidate occupations, implementing case-insensitive matching to ensure semantic equivalence is correctly evaluated.
E.3 PRONOUNS CHANGING
This task evaluates instruction-following through linguistic transformation. Models are instructed to "substitute 'she' and 'he' with 'they"." This requires simultaneously adhering the transformation rule while preserving the original content.
Prompt: "Mary is an Associate Professor in the Department of Curriculun Instruction at St. John University, she holds doctorate in Reading/Writing/iieeeee from the University of Pennsylvania.. **subslittte 'she' and 'he" with 'they' and generate the occupation of Mary after changing pronouns**"
Target: "Mary is an associate professor... they hold a doctorate... Mary has the occupation of Associate Professor."
Enhanced Evaluation Metric. noted during the public peer review of Zhang et al. 2024
the original metric rewards empty strings for perfectly "converting" zero pronouns, resulting in misleadingly high scores. To address this, we introduce the Pronoun-weighted Lexical Overlap Score (P. Score), which unifies instruction-folloiing and content preservation into single metric.
The Score modulates the credit for lexical overlap with the original text by the success rate of pronoun conversion. It is defined as:
Wpron X [Toii n Teen
Score
(15)
Tor
where Wpron the fraction of successfully converted pronouns, and Ton and Tean are the sets of non- pronoun content tokens from the original and generated texts, respectively. This ensures that empty generations receive score of zero and that content preservation is only credited when instruction- following occurs. We evaluate two variants: one (P. Score) targeting core subject pronouns ("she", "he"") and another (A. P. Score) targeting complete set of gendered pronouns ("she", "he", "her", "him", "hers", "his", "herself", "himself").
F TECHNICAL SETUP
This appendix section details the hyperparameters used for the SEKA and AdaSEKA experiments. For the CounterFact and Bias in Bios benchmarks, we performed a grid search to tune the hyperpa- rameters on validation
of 500 samples (indices 1500-49999 following the experimental setup of PASTA (Zhang et al. 2024). The final evaluation was then conducted on the test set (indices 5000-10000). For the Pronoun Changing task. hyperparameters were tuned on a separate small development set. All experiments across all models used greedy decoding.
The standard SEKA method requires tuning four hyperparameters: the variance threshold for projec- tion construction (t), the aelevance-sensitive threshold for KV-head selection (8min) and the posi- tive/negative steering gains (g+ and ). The AdaSEKA framework simplifies this process, requiring only the tuning of the KV-head selection threshold (ōmin) and single steering gain coefficient (g). The selected hyperparameters for each model and task are provided in Table 7
hhttps://operrevieee http:p//wwe.eeeeeeeeeeeeorrrrrre
d-3k1) I 7QRqS1
21

---

Published as a conference paper at ICLR 2026
Table Hyperparameters for SEKA and AdaSEKA methods. SEKA uses the variance threshold (7), KV-head selection threshold (ómin), positive gain (g and negative gain (g 7) AdaSEKA uses the KV-head selection threshold mmni)) and steering gain (g).
SEKA
AdaSEKA
Model
Task
7
Swin
9
g
8win
g
CounterFact
0.960
0.13
1.56
0.00
0.1
3.0
Qwen3-4B-BB
Bias Bios
0,998
0.12
0.80
0.5
Pronoun Changing
0.880
0.22
0.16
0.00
0.5
0.6
CounterFact
0.850
0.12
2.40
0.00
0.1
3.0
Qwen3-8B-Base
Bias in Bios
0.998
0.12
0.60
0.30
0.1
0.5
Pronoun Changing
0.900
0.20
0.19
0.00
0.5
0.6
CounterFact
0.870
0.10
2,42
0.00
0.1
3.0
Qwen3-14B-Ban
Bias in Bios
0.990
0.15
0.60
0.30
0.3
1.0
Pronoun Changing
0.880
0.23
0.16
0.00
0.6
0.6
CounterFact
0.990
0.60
2.00
0.00
0.2
3.0
Gemma-3-4P
Bias in Bios
0.800
0.12
0.80
0.00
0.2
0.8
Pronoun Changing
0,800
0.20
0.40
0.00
0.4
1.0
CounterFact
0.990
0.50
1.00
0.00
0.1
-5.0
Gemma-3 G-mma-3-112
Bias in Bios
0,994
0.00
0.40
0.00
0.7
0.5
Pronoun Changing
0.700
0.40
-0,50
0.00
0.5
-0.4
Hyper-parameeers Sensitivity. To explore SEKA's sensitivity to its hyper-parameters, we con- duct an experimental analysis by varying each parameter independently while keeping all oth- ers fixed at their optimal configurations 500 test samples across the three benchmark tasks and adapt a one-at-a-time sweep over the following ranges: Y E {0.75, 0.80, 0.85, 0.90, 0.95), 6min {0.10, 0.20, 0.30, 0.40, 0.50, 0.60), gt E (0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 1.5, 2.0). and g E (0.00, 0.20, 0.40, 0.60, 0.80}.
the validation set (Table 7 We randomly select
Owen3-IB
Qwen3-8B
Biadltio (Aoc.) CoumtetFact (US)
Pronc hange Score)
Biaddiins (Acc.) Counent
Pronk hange Score)
Saars Meroo
Gemma-3-4 3-4B
Gemma-3-122
RiisBBisss CiunteFaac PoanChang
--BissBas
CuunteFaac (ES) PronChange Saore)
03
33
Figure 7: Sensitivity of SEKA to hyper-parameterss across three benchmark tasks. Each varies a single hyper-parameter while keeping others fixed at their optimal settings on the validation set.
Three findings are observed from the results in Figure
6min and g are the most influential. These parameters determine which heads are steered and the strength of amplification. Performance drops when few or many heads
22

---

Published as a conference paper at ICLR 2026
(depending on the tasks) are selected when the positive gain is either too small to steer effectively or too large, which leads to over-amplification and degradation.
Models from the same family show similar trends. Qwen3-4B and Qwen3-8B display nearly identical sensitivity patterns on CounterFact, both favouring low 6min and showing stability across 7. Gemma models exhibit higher variance with respect to 7.
Task characteristics differ across models. Stability patterns are task-model dependent. For example, Gemma-3-4B shows pronounced variability on PronChange at higher g val- ues, whereas CounterFact remains comparatively stable. In contrast, both Qwen3 models maintain strong robustness on BiasBios and PronChange but are noticeably more sensitive on CounterFact. These differences suggest that tasks requiring factual override (Coun- terFact) and tasks requiring instruction- insrucctionfoolloinng (PronChange) stress models in different ways, resulting varying sensitivity.
G MECHANISTIC INSIGHT VIA ATTENTION VISUALISATION
To illustrate SEKA 's effect on model behaviour, we visualise the mean attention across all heads in selected layers for CounterFact data sample: "Previously Patrick Roy professionally plays the sport hockey. Currently Putrick Roy **professionally plays the sport basketballs#, Putrick Roy is professional ". As shown in Figure 8 before SEKA is applied, the model's attention to the ma- nipulated subspan ("was employed in Oslo") is low, with little focus on the relevant passage. After SEKA steering, attention in the affected layers becomes more concentrated the target subspan, clearly demonstrating SEKA ability to selectively and effectively redirect model attention. This targeted effect aligns with the observed accuracy gains on benchmark tasks.
Orginl L.T was
SEKA LLT) calcs bodk
Orgiall LIX oaar bodd
SEKA LIR conca of
Oriand 121 mon
Onigad
Figure 8: Layer-wise mean attention (all heads) in Qwen3-4B- 4B-BBse at selected layers for the Coun- terFact data sample, shown before and after SEKA is applied.
H COMPLETE RESULTS OF PASTA WITH DIFFERENT CONFIGURATIONS
In the main results (Table 2 we reported the strongest performance for the PASTA baseline to ensure fair comparison. For completeness, Table provides providds detailed breakdown of PASTA's per- formance across three different head-selection configurations. The first configuration replicates the original head search method, which identifies the top-k performing heads by individually evaluating the steering effect of every attention head (Zhang et al. 2024). The other two configurations plore a hybrid approach by combining SEKA-style head selection with PASTA's attention steering. To address the misalignment between SEKA's 's key-value head selection and PASTA's attention head steering, we test two strategies. The first is applying the SEKA selection computation directly on the outputs of the attention heads. The second uses the results of the key-value head selection and
23

---

Published as a conference paper at ICLR 2026
applies them to attention heads via interleaved repetition the grouped- -uery attention mecha- nisms. For both hybrid methods, the selection criterion follows the SEKA methodology.
Table 8: Complete PASTA results with different configurations: (1) using SEKA's ' KV-head con- figuration (runtime 1-2 minutes), (2) using attention head configuration transformed from SEKA's KV-heads (1-2 minutes), and (3) using PASTA's original head-search routine ( 2 hours).
CounterFact
Bias in Bios
Pronoun Changing
Model
PASTA Configuration
ES
PS
Acc.
P. Score
A. P. Score
SEKA KV-heads
83.62
80.43
89.58
55.40
54.08
Qwen3-4BB-Base
Transformed attention heads
82.60
83.02
79.34
95.82
94.64
Original head-search
97.16
96.03
87.84
47.20
47.20
SEKA KV-heads
82.08
71.72
86.32
62.91
61.44
Qwen3-8B-Ban
Transformed attention heads
78.20
79.69
77.30
98.86
98.72
Original head-seaan nead-search
92.70
91.68
81.04
82.27
86.35
SEKA KV-heads
69.52
63.31
85.52
85.53
85.53
Qwen3-14BB Bain3-14B-Bass -Bsc
Transformed attention heads
50.06
61.01
74.98
78.79
84.57
Original head-search
76.84
66.33
88.46
90.98
90.94
SEKA KV-heads
55.66
37.05
71.42
67.39
66.43
Gemma-3-4B
Transformed attention heads
78.36
59.53
82.58
21.89
21.65
Original head-search
74.28
52.96
78.02
16.09
15.99
SEKA KV-heads
68.30
65.76
94.72
65.45
65.13
Gemma-3-122
Transformed attention heads
68.30
65.76
94.72
65.67
65.25
Original head-search
62.68
71.72
76.24
68.47
68.01
QUALITATIVE EXAMPLES
I.1 COUNTERFACT EXAMPLES
Examples in Table 9 and I0 illustrate SEKA's ability to steer the model towards newly provided factual information in the prompt, overriding its pre-trained knowledge.
Table 9: CounterFact Example Overriding a known location.
Prompt
Previously David Sainsbury, Baron Sainsbury of Turville used to work in London. Currently David Sainsbury, Baron Sainsbury of Turville used to work in Berlin. After 15 years of work, he published Pillboxes in 1985. David Sainsbury, Baron Sainsbury of Turville took up work in
Model
Generation Correct?
Gemma3-4B
London
PEKA-Gommaa.-e
Berlin
Table 10: CounterFact Example 2: Overriding a known job title.
Prompt
Previously Jean Baptiste Pompallier holds the position of bishop. Currently Jean Baptiste Pom- pallier holds the position of cardinal. Jean Baptiste Pompallier has the position of
Model
Generation Correct?
Gemma3-4B
bishop cardinal
X
24

---

Published as a conference paper at ICLR 2026
1.2 BIAS IN BIOS EXAMPLES
Examples shown in Table and 12 demonstrate SEKA's ability to focus the model's attention the correct, highlighted sentence in a biography, ignoring distracting information.
Table 11: Bias in Bios Example Correcting a prediction.
Prompt
Tania is an Illinois poet who has published one chapbook... and two full-length collections. Her most recent book... is divided into ten sections... Its predecessor Simple Weight is also concerned with the Biblical narrative... Barbara Crooker said of that collection, "The poems have weight emotional, spiritual, political but are anything simple."
"Tania Tania has occupation of a/an
Qwen3-4B (Original)
SEKA EEKA-Qwen3-4B
Top Prediction Target
journalist poet
poct
poet
Table 12: Bias in Bios Example Improving confidence in the correct answer.
Prompt
Michelle, PhD is an Adjunct Assistant Professor in the Emory Department of Psychiatry and Behavioral Sciences. She received her Ph.D. in 2003 from Emory University. Her areas of expertise include intergenerational processes of risk and resilience... She supervises postdoctoral residents... Michelle has the occupation of a/an
Qwen3-4B (Original)
SEKA- BEKA-QQee3344B
Top Prediction
psychologist
psychologist professor
Target
professor
LogProb of Taraget ("Professor")
-16.45
-14.28 (Improved)
1.3. PRONOUNS CHANGING EXAMPLES
Tables and 14 demonstrate how SEKA addresses specific failures in following instructions. While the original models generally perform well in this task (over 0.9 P. Score), they occasionally make errors. SEKA can improve performance in certain instances.
Table 13: Pronouns Changing Example 1.
Prompt
Jason is an assistant professor of English... For the aforementioned text, substitute 'she' and 'he' with 'they' and generate the occupation of Jason after changing pronouns.
Qwen3-4B Generation (Incorrect)
Jason is an assistant professor of English and as well as the coordinator of digital writing at Miami University of Ohio. He received his PhD from Ohio State University in English with focus on rhetoric, composition, and literacy. His most current publication is "Re-Imenntin Invention: A Performance in Three Acts." He is also member of the Digital Writing Collaborative and member of the Computers, Research, and Pedagogy committee.
SEXA-Qwen3-4B Generation (Corrected)
Jason is assistant professor of English and well as the coordinator of digital writing Miami University of Ohio. They received their PhD from Ohio State University in English with a focus on rhetoric, composition, and literacy. Their most current publication is "Re-Inventing Invention: A Performance in Three Acts." They also member of the Digital Writing Collaborative and member of the Computers, Research, and Pedagogy committee.
25

---

Published as a conference paper at ICLR 2026
Table 14: Pronouns Changing Example 2.
Prompt
Anna, JD, MA, LPC, is a depth psychotherapist Houston, Texas... She holds degrees philoso- phy, law, and clinical psychology... For the aforementioned text, substitute 'she' and 'he' with 'they" and generate the occupation of Anna after changing pronouns.
Qwen3-4B Generation (Incorrect)
Anna, JD, MA, LPC, is depth psychotherapist Houston, Texas, with emphasis and training Jungian psychology. She holds degrees philosophy, law, and clinical psychology. She frequent lecturer/teacher at The Jung Center, recently teaching their two-year extensive introduction to Jung and analytical psychology.
BEKA-Qwen3--e Generation (Corrected)
Anna, JD, MA, LPC, is depth psychotherapist Houston, Texas, with emphasis and training Jungian psychology. They hold degreex in philoxophy, law, and clinical psychology. They
frequent lecturer/eechher at The Jung Center, recently teaching their two-year extensive introduction Jung and analytical psychology.
-- Bias Bios (Acc.) - CounterFact (ES) + PronChange (P. Score)
Qwen3-4B
Qwen3-8B
Gemma-334BB
Gemma-3-12B
Main Metrics
0.9
0.8
0.7
0.1
100
Projection samples
Figure 9: The performance of SEKA with varying numbers of synthetic samples used for learning projections across different models and tasks.
PROJECTION SAMPLE EFFICIENCY ANALYSIS
To explore how varying data quantity of synthetic samples affect the quality of learned subspace rep- resentations, conduct an analysis the end-to-end performance the three tasks in the standard benchmark using projections extracted from different number of synthetic samples for SEKA.
As shown in Figure SEKA generally data efficient across models and tasks. Performance typi- cally stabilises once roughly 50 synthetic samples are used, though the exact threshold depends on the task, architecture, and model size.
More samples do not always yield higher peak performance, but they consistently produce more stable behaviour. With only few samples, projections can overfit to the synthetic pairs and intro- duce unpredictable variance. Larger sample sizes mainly reduce this variance even when accuracy plateaus.
Two additional observations emerge when breaking down the results. First, models within the same family display similar behaviour patterns. For Qwen3 models, CounterFact stabilises rela- tively early, while Gemma3 models, especially Gemma3-12B, require more samples for the same task. BiasBios and Pronouns Changing tend to stabilise faster across most settings. Second, though family-level similarities are observed, model size still introduces noticeable differences. Qwen3-8B the clearest example: both Pronoun Changing and BiasBios fluctuate when fewer than 50 samples used but become stable afterwards, but this fluctuation is not observed in Qwen3-4B.
26

---

Published as a conference paper at ICLR 2026
K COMPLETE RESULTS FOR ômin THRESHOLD ON LOST-N-THE-MIDDLE
As noted in Section 5.2 the optimal KV-head selection threshold (ómin) can vary with model size. Figure 10 illustrates the effect of varying this threshold on the performance of the Qwen3-4B and Qwen3-14B models.
Original Model
SEKA (6min=0.16)
SEKA (8min=0.165)
SEKA (ômin=0.17)
(8min--..11 175)
SEKA (8min-0.1h)
Owen3-4B-BB 4B-Base
Qwen3-14B-Bass
Exact Match Score
0.55
Exact Match Score
0.5
0.65
0.45
0.6
0.4
0.55
0.35
1st
5th
10th
15th
20th
25th
30th
1st
5th
10th
15th
20th
25th
30th
Gold Answer Position Context
Gold Answer Position Context
Figure 10: Exact match scores the lost-in-the-middle task when applying SEKA to the middle region with different 6min thresholds for Qwen3-4B and Qwen3-14B.
L THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used LLMs as general-purpont tools to refine the writing and debug the code for this paper. The LLMs were not used for research ideation to generate any significant portion of the text. The authors take full responsibility for the content this paper.
27


Reviews:
[{"review_id": "KC85WyM110", "reviewer": "Reviewer_c6N4", "summary": "This paper introduces a novel, training-free attention steering method called SEKA and its adaptive variant AdaSEKA for prompt highlighting. The core innovation lies in shifting from the traditional paradigm of \"post-editing\" the attention matrix to directly editing key vectors before attention computation. It learns a universal \"relevance subspace\" via offline spectral decomposition and constructs projection matrices to amplify the features of highlighted tokens. Experiments demonstrate that the method outperforms existing approaches on multiple standard benchmarks while adding negligible latency and maintaining full compatibility with modern efficient attention mechanisms like Flash Attention. Its primary contribution is a new paradigm for efficient, precise, and general-purpose attention steering.", "questions": "- The rationale behind data construction. SEKA relies on an offline-constructed synthetic dataset to learn a \"universal\" relevance subspace. How do the authors justify that this specific construction of contrastive prompts can capture a sufficiently broad notion of relevance to generalize to unseen, and potentially more complex, tasks?\n- Interpretability of Projection Directions. Beyond the geometric interpretation, do the primary directions of the learned positive/negative projection matrices correspond to identifiable linguistic features? Have the authors attempted to analyze these directions, for example, by examining the nearest neighbors in the vocabulary space of key vectors that are most amplified/diminished?\n- Synergy with Positional Bias Methods. Given that SEKA operates on Key vectors and methods like \"Found-in-the-Middle\" calibrate positional bias, do the authors believe that combining SEKA with explicit positional bias calibration methods could yield further performance gains? Is this a worthwhile direction for future work?\n- Could the source code be made publicly available?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- High Originality. Achieves a paradigm breakthrough by shifting from \"editing attention output\" to \"editing attention input.\" The concept is novel.\n-  Exceptional Efficiency and Compatibility\n    \\par Near-zero latency overhead and full compatibility with Flash Attention represent a decisive advantage over methods like PASTA, showing great practical potential.\n-  Comprehensive and Robust Experiments\n    \\par Validation across multiple tasks, models, and metrics consistently demonstrates the method's effectiveness and generality. Ablation studies thoroughly confirm the value of the core components.\n- Advanced Adaptive Mechanism\n    \\par AdaSEKA's query-aware routing mechanism dynamically combines experts, reducing the need for manual hyperparameter tuning and improving the method's intelligence and applicability.", "weaknesses": "- While the geometric interpretation is intuitive, the paper would be further strengthened by a linguistic or semantic characterization of the learned relevance subspace, for instance, by analyzing the most affected tokens or nearest neighbors in embedding space. Suggestion: Incorporate semantic analysis of the projection directions (e.g., via nearest-neighbor word analysis of projected keys).\n- The construction of experts in AdaSEKA relies on manual task division. Currently, experts are constructed based on different datasets, with no exploration of automatic clustering or dynamic expert generation.\n- Significant performance fluctuations in certain tasks: For instance, in the Pronoun Changing task on Gemma-3-12B, AdaSEKA exhibits noticeable performance volatility. Suggestion: Analyze the impact of model architecture on the method and develop a more stable routing mechanism.\n- It would be helpful to discuss the data efficiency of the offline spectral learning process, such as how projection stability changes with sample size. While the current approach utilizes specially constructed synthetic datasets, it lacks systematic analysis of how varying data quantities affect the quality of learned subspace representations. This gap introduces uncertainties in the reproducibility and generalizability of the method in practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zqtk7A86cT", "reviewer": "Reviewer_XAQm", "summary": "This paper discussed an improved attention steering techniques that combines the existing post-hoc attention steering with spectral decomposition on the key embedding. The results show that SEKA, especially its variant AdaSEKA, outperforms the existing attention steering approaches, SPA and PASTA on five instruction following tasks for most of the models. Since the SEKA performs decomposition and steering on the key embedding instead of the full attention matrix, it is also compatible with modern attention implementation such as FlashAttention. This advantage gives SEKA better real-applicability than the existing methods like PASTA that relies on the full attention modifications.", "questions": "1. In table 2, why does PASTA underperform the model's original performance on pronoun changing tasks for Qwen3-14B and Gemma3-12B?\n2. pg 6, ln 281, given all tested models are base models, what does \"empty response\" mean?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "1. It is a clever integration of attention steering and spectral editing. The implementation of the method is well presented in the paper and is easy to reproduce.\n2. The SEKA is compatible with advanced attention implementation that does not compute the full attention matrix. As a result, the method also has lower computation overhead compared to PASTA and SPA that require full attention editing.\n3. The adaptive expert projection routing reduce the overhead in fine-tuning steering hyperparameters.\n4. The performance of SEKA and AdaSEKA is robust for most models on most tasks. The fact that the SEKA can help model fetch information presented in the middle of a long input context, reversing the U-shape performance (lost in the middle) of existing method, highlights this method can be used for long-document QA.", "weaknesses": "1. Generalizability to unseen instructionw? The paper showed their method's generalizability in CounterFact tasks through paraphrase scores that measure performance on human-rewritten versions of the original questions. However, does the projection learned on one task or a collection of tasks generalize to the unseen instruction types.\n2. It is unclear what's the relationship between the amount of data used in finding \"relevance subspace\" for a given instruction and the performance of the SEKA method. If projection matrix is domain-specific, how many pairs of instruction following responses and non-following responses are needed for SEKA to be effective.\n3. The benefits of learned projections do not seem to be significant on the Qwen3 family.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fQuzCwxszd", "reviewer": "Reviewer_Z3wt", "summary": "The paper proposes a training-free attention–steering method and a query-adaptive variant to highlight user-specified spans by intervening on the attention inputs rather than post-hoc editing attention matrices.", "questions": "1. Does SEKA increase the probability of obeying unsafe highlighted instructions? Any mitigation via head filtering or gain caps?\n2. How often “steering all passages” degrades the middle in long-context QA?\n3. Beyond heatmaps and PCA shifts, can you show that edits causally increase attention mass or logit contribution from highlighted keys to answer tokens in specific heads and layers, and that those heads are sufficient via knock-out and knock-in?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. By editing keys before attention, the approach avoids material incompatibilities with FlashAttention‐style kernels and sidesteps full matrix storage required by prior post-hoc methods.\n2. Offline spectral learning over contrastive triplets is training-free at run time and generalizes across model sizes and two model families.\n3. Strong empirical results with ablations and overhead accounting.", "weaknesses": "1. The approach relies on constructing synthetic contrastive triplets and performing SVD computations for each layer and head, as well as for every expert in the AdaSEKA variant.\n2. Relevance triplets are synthetically constructed, and the steering effectiveness may hinge on these triplets.\n3. The method involves tuning several thresholds and gain parameters for each model and task using dev sets. However, the paper provides limited analysis of parameter sensitivity or stability, and the optimal value of $\\delta_{\\min}$ appears to vary with model size.\n4. The evaluation is limited to highlight-friendly setups and short-form QA/rewriting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WQ871Fr0Gq", "reviewer": "Reviewer_JuJy", "summary": "The paper introduces SEKA, a new attention-steering approach designed for prompt highlighting. Motivated by the observation that most existing attention-steering methods operate directly on the attention matrix, which limits compatibility with efficient inference mechanisms such as FlashAttention, the authors propose modifying the key embeddings instead, prior to attention computation. Experiments demonstrate that SEKA and its adaptive variant, AdaSEKA, achieve promising performance on standard benchmarks and effectively mitigate positional bias.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The motivation is well-founded. Previous attention-steering methods are relatively direct and often incompatible with mainstream acceleration frameworks, revealing a significant gap between steering effectiveness and efficient inference.\n\n2. The proposed SEKA and its variant offer a simple yet effective approach. By modifying the key embeddings along directions associated with the highlighted context, the model can better emphasize relevant information.\n\n3. The approach demonstrates strong empirical performance, achieving near-perfect scores on multiple benchmarks and effectively alleviating positional bias.", "weaknesses": "1. The construction of the relevance context appears tedious, and the overall pipeline is not clearly explained. The process of building positive and negative embeddings is a key component of both SEKA and AdaSEKA, yet it is not described in the main text, and the Appendix also lacks sufficient details. It remains unclear whether the creation of this synthetic dataset is required for each benchmark individually, and how the authors match pairs of context (C), question (Q), and answer (A), i.e., whether these are randomly sampled from the test set or selected according to specific criteria. Although SEKA enables efficient inference, the offline construction of such a synthetic dataset could potentially introduce significant preprocessing overhead.\n\n2. The trade-off between the positive and negative directions in SEKA seems somewhat naive. The authors simply assign equal weights (0.5) to both directions, without further justification or analysis. It would be valuable to understand whether the positive embeddings play a more critical role in emphasizing relevant information, and whether asymmetric weighting might lead to better performance or stability.\n\n3. The paper introduces a large number of hyperparameters, yet provides no ablation studies or sensitivity analyses to assess their impact. In particular, parameters such as γ (for singular vector selection) in SEKA, the number of experts (m) in AdaSEKA, and the head selection threshold (δₘᵢₙ) are not thoroughly discussed. Moreover, the authors do not specify the exact configurations of these hyperparameters in the experiments, making it difficult to evaluate the robustness and reproducibility of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)