Signal in the Noise: Polysemantic Interference Transfers and Predicts Cross-Model Influence
Authors: Bofan Gong
Shiyang Lai
James Evans
Dawn Song
Venue: ICLR 2026 Poster (2026)
Keywords: Polysemanticity
Sparse Autoencoders
Interpretability
Behavior Intervention
Primary area: interpretability and explainable AI
TL;DR: Using four interventions, we probe how polysemantic structure steers LLM behavior and find that these vulnerabilities transfer across models, reflecting shared polysemantic structure.

Abstract: Polysemanticity is pervasive in language models and remains a major challenge for interpretation and model behavioral control. Leveraging sparse autoencoders (SAEs), we map the polysemantic topology of two small models (Pythia-70M and GPT-2-Small) to identify SAE feature pairs that are semantically unrelated yet exhibit interference within models. We intervene at four loci (prompt, token, feature, neuron) and measure induced shifts in the next-token prediction distribution, uncovering polysemantic structures that expose a systematic vulnerability in these models. Critically, interventions distilled from counterintuitive interference patterns shared by two small models transfer reliably to larger instruction-tuned models (Llama-3.1-8B/70B-Instruct and Gemma-2-9B-Instruct), yielding predictable behavioral shifts without access to model internals. These findings challenge the view that polysemanticity is purely stochastic, demonstrating instead that interference structures generalize across scale and family. Such generalization suggests a convergent, higher-order organization of internal representations, which is only weakly aligned with intuition and structured by latent regularities, offering new possibilities for both black-box control and theoretical insight into human and artificial cognition.

Published as a conference paper at ICLR 2026
SIGNAL IN THE NOISE: POLYSEMANTIC INTERFERENCE TRANSFERS AND PREDICTS CROSS-MODEL INFLUENCE
Bofan Gong"; Shiyang : Dawn Song', James Evans? 'Independent Scholar, University of Chicago, 'UC Berkeley of ofangonge gmail. com, shiyanglaiee www.ag/aaaiccccco.o..ooo
ABSTRACT
Polysemanticity pervasive in language models and remains a major challenge for interpretation and model behavioral control. Leveraging sparse autoencoders (SAEs), we map the polysemantic topology of two small models (Bythia-7)) 70M and 6-PT-2-55 2PT-2-Small) to identify SAE feature pairs that are semantically unrelated yet exhibit interference within models. We intervene at four foci (prompt, token, feature, neuron) and measure induced shifts in the next-token prediction distribu- tion, uncovering polysemantic structures that expose a systematic vulnerability in these models. Critically, interventions distilled from counterintuitive interference patterns shared by two small models transfer reliably to larger instruction-ture models (mlamaa3,) 8--B// Iee-Instrain ruct and Gemma 22 9B- Instruttt, yielding predictable behavioral shifts without access to model internals. These findings challenge the view that polysemanticity is purely stochastic, demonstrating instead that interference structures generalize across scale and family. Such general- ization suggests a convergent, higher-order organization of internal representations, which is only weakly aligned with intuition and structured by latent regularities, offering new possibilities for both black-box control and theoretical insight into human and artificial cognition. Code and data available here
INTRODUCTION
Polysemanticity refers to the phenomenon in which individual neurons or groups of neurons in neural networks often encode greater number of distinct features concepts than the number of neurons involved. This property becomes increasingly prevalent as models scale and has been shown to enhance learning performance (Wang al. 2024, MMrshall & Kirchner 2024, Oikarinen & Weng 2024b). Anthropic's 's work on superposition builds on prior insights, showing that large transtormer models encode more features than neurons by using linear combinations of activations. This mechanism sacrifices monosemanticity but significantly improves model capability (Elhage ctt al. (022). Mathematical analyses reveal that polysemantic neurons enable networks to represent exponentially more features compared to monosemantic approaches (Elhage et al. 2022).
Nevertheless, this representational efficiency comes with trade-offs. Most significantly. it complicates model interpretability, as entangled representations obscure how human- uumaan-uuddestaddablle concepts are encoded within the model's 's internal structure. One mechanistic approach to address this chal- lenge is the use of sparse autoencoders (SAEs), which aim to disentangle superimposed features by learning sparse, higher- digeee-dimmniionall representations of model activations. SAEs enable the extraction of interpretable, monosemantic features, where each SAE neuron ideally corresponds to single concept (Bricken et al., 2023 Templeton et al. 2024 Recent work has shown that SAE-derived features exhibit degree of universality across different LLMs (Lan et al. 2024), suggesting the existence of fundamental patterns in how neural networks encode meaning. This consistency hints at the emergence of shared semantic topologies that persist across architectures and training regimes, raising profound questions about whether these patterns are merely computational artifacts or re- flections of latent semantic regularities (Huh et al.) (0024). Except for SAEs, broader range of interpretability techniques is emerging simultaneously (Chang al. 2025 Dunefisky et al. 2024).
"Equal contribution, alphahetical ordered. Nevertheless, several studies have also documented limitations of SAEs (see Appendix

---

Published as a conference paper at ICLR 2026
The second trade-off, which is largely overlooked in current literature, involves systematic vulnerabil- ity stemming from polysemantic structures in language models. In Anthropic's 's toy experiments, they note that stronger superposition can make models more vulnerable to adversarial attacks (Elhage et al.) 2022). Beyond this, to our knowledge, there is very little existing empirical research that directly addresses the safety implications of polysemanticity in language models. In contrast, the vision model domain has a well-established body of work on various forms of adversarial model control that exploit polysemantic representations (Goh et al. 202T; Oikarinen & Weng 2024a| Geirhos et al. 2023 Dreyer et al.) [0244] Huang et al. 2022] Gorton & Lewis] 2025). Bereska and Gavves, in their review of mechanistic interpretability for Al safety, highlight polysemanticity as a key challenge in building safer LLMs (Bereska & Gavves] 2024). To bridge this gap, we focus polysemantic structures in real-world LLMs, particularly those that persist across models, and explore hard-to-detect and targeted interventions to better understand their associated risks.
Before explaining the details, it is necessary to distinguish three nested representational domains:
Human Symbolic Manifold (M): The latent first-order symbolic domain that encodes human- intuitive semantics independent of contextual usage.
Model Activation Space (As): The d-dimensional vector space spanned by the neurons in layer é of the language model; it partially reflects M.
Sparse Feature Basis (Fi): The k-dimensional, typically overcomplete basis (k >> d) extracted from A by a SAE.
A M
Ti- M
8
Feature intervention
Stearing E to induce G
Semantically ineeevant but interfered
Token intervention
Steering E's top activated token direction inducc G
Prompt intervention
Adding texts that activat E's top activated takens induuce
Original network neurons (d-dim)
Sparse autoencoder features (k-dim)
Neuron intervention
Mesk polysemantic neurors
after the model's behavior
Figure 1: Conceptual illustration. (Top) Neurons A, B, and C a 3D activation space A, while E. F, and G denote SAE feature directions projected into the human symbolic manifold M. In this example, features E and G are nearly orthogonal in M (i.e., semantically distinct), yet both load strongly on neuron C, so they still interfere in A via shared activation direction. This illustrates that semantic orthogonality in M does not guarantee independence in activation space. (Bottom) Features are unevenly distributed across neurons: neuron encodes more features than neurons B and C, forming polysemantic "hub." Together, the panels highlight two vulnerabilities: (I) semantically distant features can interfere through shared activation geometry, and (2) polysemantic hubs concentrate interference risk small subset of neurons.
As illustrated in Figure I. orthogonality in the activation space Ar does not persist after projection into the symbolic manifold M. Consequently, two features from F/ that appear unrelated in M (i.e., anchoring semantically to distinct meanings under interpretation) can still interfere substantially in Aç. This interference is also often unevenly distributed across neurons. Building on these two structural characteristics, we design feature, token, prompt, and neuron levels of intervention to investigate: (1) how models' expression a target feature is affected if semantically unrelated but interfering features are manipulated, (2) whether model vulnerability correlates with neuron polysemanticity, defined as the number of distinct features neuron encodes, and (3) what extent these polysemantic
2

---

Published as a conference paper at ICLR 2026
interference patterns transfer across models. In this work, model vulnerability to interventions is measured by the shift in the next-token prediction distribution following the intervention.
Our findings are four-fold. First, present experimental evidence that interventions lever- aging polysemantic structures of LLMs can effectively manipulate model outputs. Specifi- cally, by targeting features and tokens via steering vector techniques and prompts via prompt injection that are not semantically aligned with the intended target but interfere with it, can reliably induce the model express the desired semantics. Second, we iden- tify the existence of cross-model persistent polysemantic structures. By collecting shared in- terference features from both Pythia-70M and GPT-2-Small and applying them steer Llama-3. 1-t.-/1--//---nnttte intervention effectiveness, revealing a consistent architecture of meaning that transcends specific im- plementations. Third, we explore those counterintuitive yet stable interference patterns that replicate across models. Post-hoc annotation of higher-order semantic relations accounts for a minority of cases, indicating that models learn robust regularities largely opaque to interpretation. Fourth, we an- alyze intervention at the neuron level and find that highly polysemantic neurons are more vulnerable: modifying their activation leads to greater semantic shifts in model output. For "super-neurons" (i.e., activated by over 500 features), amplification strongly alters model behavior, while deactivation has a notably reduced effect, suggesting they may
and Gemma-22 2-----9----- IIInttuuct we still observe substantial
critical junctions in the semantic architecture.
PRELIMINARIES AND METHODS
2.1 SPARSE FEATURE EXTRACTION WITH SAES
Our exploration of polysemantic structures draws on the pre-trained SAEs provided by Neuronpediq" We focus on 1-PT-22Sman and Pythia- 70M, the two models for which Neuronpedia supplies SAEs for all major sub-modules in every layer. Both SAEs have dimensionality 32,768 and we treat each SAE feature as defining an explicit direction in activation space.
Formally, let fi,e denote the i-th SAE feature associated with layer e, and let
di,t E A
be its projected direction model's activation space As. We use the (z-normalized directions
dif
de.
dd..el22
We define the interference scale between two SAE features fi,e and fg. at layer the cosine similarity between their uctivation- sppcee directions:
Ii(i.) cos(dd,, dz.r).
Analogously, let m, m, E EM denote the corresponding directions of these features in the human symbolic manifold M. Concretely, we approximate M using text wmweemmeeeeeeeee embedding-33 3-1aare large embeddings of the natural-language glosses for features and j. generated by CPT-4o-mini, and normalize these vectors to unit length. We quantify their surface-level semantic relatedness by
S(i,)) cos(mi, my).
More details of SAEs are elaborated in Appendix D
2.2 DISTINCT FEATURE IDENTIFICATION WITH AGGLOMERATIVE CLUSTERING
SAEs disentangle polysemantic neurons into monosemantic sparse features. These features, however, not always decomposed consistent semantic level (Bricken et al.) 2023 Foote 2024). For example, a neuron associated with dog-relate concepts might be divided into features representing different dog breeds, while another neuron encoding both cat and car concepts might be split into features representing cat and car. In such cases, the resulting monosemantic features differ in granularity. To mitigate this inconsistency, we employ agglomerative clustering to align feature
http://wwwweuroopdaaa.....
3

---

Published as a conference paper at ICLR 2026
representations to consistent semantic level, facilitating both (1) the quantification of neuron polysemanticity and (2) the isolation of feature groups exhibiting low similarity their surface meanings for subsequent analysis.
To identify distinct, higher-level features, we compute the semantic relatedness between pairs of SAE features using their nnte-rntterrtttiionn glosses (Caden Juang al. 2024 together with embeddings of the feature glosses. Prior work offers different heuristics for identifying semantically distinct SAE features. Foote al. propose a cutoff of 0.5 for distinguishing semantically distinct feature clusters (Foote) 2024), while another work on analyzing text eeeeeedmmeengg 3 -arge shows that unrelated concepts typically fall within the 0.05-0.30 cosine similarity range (Zuchen al., 2025). Drawing on these insights, we conduct agglomerative clustering in each SAE layer using four increasingly strict similarity cutoffs (i.e., 0.40, 0.30, 0.20, and 0.15) to assess whether our results are robust across varying thresholds of semantic distinctness, while retaining sufficient feature density for experimentation. Figure 2 shows an example of the clustering results for the 5th MLP layer of Pythia- 1athia-7000 '70 under the similarity cutoff of 0.4. Detailed descriptive statistics (1) the distribution of cosine similarities among SAE features, (2) the distribution of their interference values, and (3) the correlation between interference scale and semantic similarity provided in Appendix
Glaber
Clustr
BIOCHEMISTYY
CODING
wed
Custer
addondion....
Quster
LICENSE
Agglomerative clustering of SAF features trained on Pythia-70M's 5th MLP layer under the cosine similarity threshold of 0.4. Only the five largest feature clusters labeled.
Figure 2:
2.3 EVALUATION CRITERIA
The effect of polysemantic intervention is quantified the change in alignment between the model's next-token prediction distribution and a target SAE feature f Fj on a context sentence. assess the similarity
Details about sentence generation are elaborated in Appendix [ Specifically, between the model's 's output and a cnature-associate token set T C V, which consists of tokens with activation values above a threshold (here we use 0.8 highest activation value).
Let O, Ô e AII be the model's output distributions before and after intervention, and let E E gl denote the token embedding matrix. Our main metric is weighted cosine similarity:
Because different models can yield divergent auto-interpretatiion of the same feature, conduct cross- validation with DeepSeek-V. on single SAE layer of Pythi a a 70M, which reveals both strong concordance between the interpretations produced by GPT-40-m and DeepSeek-vs Replicating experiments with DeepSeek- -33 feature glosses further confirm the consistency of our principal findings. See Appendix
4

---

Published as a conference paper at ICLR 2026
d(,,T,)
E O(t) TVV
max cos Ee. Er), TET,
(I)
where each token / E V contributes its predicted probability weighted by the highest cosine similarity between its embedding and those in Ty. This metric aims at capturing the overall semantic alignment between model's output and the target feature. Then, the intervention effect is:
Ac c(0,1))-(011) d0,TT)
As an alternative, we also report weighted overlap, which simply sums the output probability mass over T. Formal definition and results are provided in Appendix G In comparison, this metric captures how the model's output is steered towards the most related tokens of the target feature.
2.4 OVERVIEW OF INTERVENTION METHODS
Our investigation of polysemantic interventions begins with Pythia-70M and GPT 2-TT-2-malll using three complementary approaches: feature-direction steering, token-gradient steering, and prompt injection. We randomly select target features to be intervened For each selected target feature, we sample interference features from feature clusters derived from Section 2.2 excluding the target's own cluster-to ensure sufficient meaning dissimilarity with the target. Interference features are drawn from five interference intervals: [0.0, 0.1). 0.2), [0.2, 0.3), 00.3, and [0.4, 1.0). In the first two experiments, we construct steering vectors for interference features using two methods: (1) projecting feature directions from the sparse basis F, into the model's ' activation space Av. and (2) computing token-gradient directions from the partial derivatives of the layer's activations with respect each feature's 's top-activating tokens. For each intervention,
roughly optimize the scaling of the steering vector over the range [-20, 20), balancing intervention strength with the need to preserve coherent model outputs (i.e., avoiding substantial disruption to the overall output distribution). Details about the tuning strategy are elaborated in the Appendix 1.2 In the prompt injection setting, we prepend sampled snippets of top-activating texts from interference features to the input and measure the semantic shifts of the model's output using metrics stated above, conditioned on varying levels of feature interference.
In the second step, we incorporate an evaluation on the cross-model transferability of polyseman- tie structures. Particularly, we apply the two scalable intervention methods (i.e., token-gradient steering and prompt-injection) to black-box, larger models Llama-3 -3. 1.1--B/ 1--B/70B-Inntraai Gemma- 2-amaa--9900 9--9--10-- Inst ruct to see whether same effects manifest. For the steering intervention, select target features and sample interference features that are consistently identified in both Pythia-70K and GPT-2-Small. We then extract the corresponding gradient vectors and use them to steer the black-box models. For the prompting intervention, we prepend activation-inducing text snippets shared across the identified interference features to the input prompts. One might question the practical significance of our next-token control test. While the primary objective of this study is mechanistic, we additionally conduct small-scale evaluation of the gradient- gradient-based interventionwwhicl we empirically show stronger interference effects-on on the HellaSwag dataset. This evaluation assesses whether the intervention can steer model behavior in target-related contexts without degrading overall task performance. Implementation details are provided in Appendix !
and
Finally, we analyze the impact of neuron polysemanticity on model outputs in Pythia-7CM and GPT-2- 122-mmall For each neuron, we identify strongly connected features by thresholding connection weights at 0.2, and define its degree of polysemanticity as the number of such features. We then suppress or amplify the activations of neurons with varying polysemanticity levels and evaluate how the model's output distribution shifts toward the semantics of associated feature clusters.
We randomly sample 480 target features from GFT-2- -malmlll and 180 from Pythia a-70M (See Appendix for details).
5

---

Published as a conference paper at ICLR 2026
3 EXPERIMENTS
3.1 EXPLOITING SAE FEATURE DIRECTIONS FOR INTERVENTION
Our hypothesis posits that if two feature directions interfere in Ay, despite being nearly orthogonal in M, then enhancing one will inevitably influence the other to some degree. If true, this would imply the potential to covertly manipulate the output probability of a target feature by steering with not obviously related features. To evaluate this, we pair each target feature with interference features sampled at varying levels of semantic dissimilarity. We vary feature irrelevance thresholds (from 0.4 to 0.15; Appendix H) to confirm that the observed pattern holds across threshold choices. Additionally, we perform controlled regressions- accounting for feature-patt cosine similarity- under two linear model specifications to confirm that interference effects persist independently of semantic similarity. Figure reports the results, and Table 3 in Appendix H shows particular examples.
B
0.00
Pytka Thresheld
0.3
GF1.2 Threshell
3
Confedrne Interssh
7.00
0.20
6000
0.3
5.00
0.3
0006
0.22
0.11
0.22
nlah ÓAAR Orghaa Mid Low
0.09 Orignal 100 Me Mil Low Randers
Pybha
GP1/2
Figure 3: Interventions along interfering but semantically distinct SAE feature directions reli- ably steer next-token predictions toward the desired semantic direction. (A-B) Relative change in weighted cosine similarity toward the target (Ac). Bars show the mean relative change compared to baseline across interference levels, with lighter shades indicating stricter feature-meaning relevancy cutoff thresholds. The x-axis denotes the interference scale between the target and intervention feature: Original corresponds intervening with the target feature itself, and Random serves random feature intervention baseline. Error bars denote 95% confidence intervals. (C) Regression estimates of the effect of feature-pair interference value on intervention success. Two regression specifications shown: Model A regresses weighted cosine similarity after intervention (T)) on interference value, with feature- feature-meaning similarity, baseline weighted cosine similarity (c(O, Ty)), and layer-type controls; Model B regresses the change score (Ac) on interference value, with feature- meaning similarity and layer-type controls. Error bars denote 90%, 95%, and 99% confidence intervals. Results with the alternative metric are shown in Figure I0
All analyses consistently show that steering with features that are semantically dissimilar yet interfer- ing can alter the output probabilities of a target feature's 's top-activating tokens, with stronger effects observed at higher interference levels. We also find that SAE-based interventions are generally much less effective on GPT-2- Small than on Pythia- 72a--0000 7OM. An intuitive hypothesis is that greater model depth attenuates the downstream impact of mid-layer activation shifts (Fort 2023). To probe this, we run an additional SAE-steering experiment on a third model, Gemma- - 22-22 2B, on a smaller evaluation set. Despite being deeper than GPT-2-Small
Gemma- 2B exhibits larger effect un- der weighted cosine similarity than GPT 2 Small, 1, though still smaller effect than Pythia- Pythia-70M (full discussion in Appendix O At the same time, its gains under the weighted overlap metric are comparable to those of the two smaller models. Taken together, these results suggest that model depth alone cannot account for the differences in cosine- based effect sizes; least part of the discrepancy appears to arise from how different models interact with our two evaluation metrics.
3.2 STEERING WITH GRADIENT VECTOR FOR TOKEN INTERVENTION
In this section, we treat the top-activating tokens of semantically unrelated yet interfering SAE features as intervention signals (Ferrando et al.) 2024). For each interference feature, we pass its top-activating text through the model and, at the feature's 's corresponding layer, compute the gradient of that token with respect all neurons in that layer. This gradient serves the steering vector.

---

Published as a conference paper at ICLR 2026
As shown in Figure using token-gradien gradient steering both models yields roughly 10x larger effects than steering along SAE feature directions. Interestingly, token- weeennntdieent steering also flattens the relationship between interference scale and intervention effectiveness, and steering the original feature's 8 gradient direction is less effective than steering the interfering ones. This may stem from the fact that token gradients are not tied to SAE-defined interference levels and SAE features can exhibit a degree of arbitrariness (Paulo & Belrose
4
2025
Heap et al. 2025)
Pyttr Threshold
10.00
0PT12 Thrsshold
Confidence intersh VIG
2 001
cat -olu
Pal0lee dhange treard Sarger faltue
00100
2.00
Phop watoor cetfoeeettt of
001
50.00
0.01
0.00
4.00
20:00
2.00
10.00
0.00
Rondor
High tollow Raddon
Pysko
Figure 4: Interventions along gradient directions of top-activating tokens for interfering but semantically distinet SAE features reliably steer next-token predictions toward the desired semantics. Subpanels follow the same conventions as Figure but intervention vectors are computed from token gradients rather than SAE decoder weights. For (A-B), error bars indicate 95% confidence intervals; for (C), error bars denote 90%, 95%, and 99% confidence intervals. Results with the alternative metric are shown in Figure I1
3.3 PROMPT INJECTION FOR INFERENCE TIME INTERVENTION
In addition to the two intervention methods described above, which directly modify the model's internal activations, interference effects may also arise from prompt injection. In this experiment, we still employ the same method as the previous two experiments to sample target and interference features. The difference is that we extract continuous text snippets with high activation values from the activation text of the interference features and prepend them to the prompts. This is intended to concisely write the semantics of the interference features into the model's latent space, while avoiding disrupting the semantics of the prompts, compared to pasting the complete activation text or pasting it within after the prompts.
Pythia Threshald
6.3
GF12 Thresheld
Canfidence Intervale 196.1
Nelathee dhange tureett Toakure
cowffidettoo in Interferenee
0.3
5,5430
6.3
5.3
4.3
0.0
60 112h
MM
902 620 Ro1com
0.0 541 102 100
90
Random
Pytlle
2F12
Figure 5: Prepending highly-activating tokens of the interference features to prompts moderately shift next-token predictions toward the desired semantics. Subpanels follow the same conventions as Figure 3 while intervention arise from snippets of highly-activating texts of the interference feature. For (A-B), error bars indicate 95% confidence intervals; for (C), error bars denote 90%, 95%, and 99% confidence intervals. Results with the alternative metric are shown in Figure 15
The overall interference magnitude-and and the monotonic increase in semantic shift under stronger interventions (Figure5)-i weaker than that achieved by direct latent-space manipulation, at roughly the 10% level. However, when evaluated using weighted overlap, the pattern becomes pronounced: intervention effects increase by approximately 10x-100x in GPT 2- Small and x00x-10000 in
7

---

Published as a conference paper at ICLR 2026
Pythia-70M. This suggests that prompt injection concentrates interference small subset of tokens most strongly associated with the target feature.
3.4 GENERALIZATION OF POLYSEMANTEE INTERVENTION VUINERABIILITY
Similar polysemantic interference may exist across multiple models. Note that interfering by token gradient vector and prompt injection do not rely on SAEs, they can be applied to models without internal access. We therefore target Llama-3. 11-BB/CBB-Inttru ruct, selecting several target feature types which widely appear in SAE database provided by Neuronpedia, including location, name, number, science, emotion, animal, color and time. Their interference features in two small models GPT-2 2-mmall and Pythia- 70M are viewed as their potential interference features in other models.
and
For gradient vector-based interventions, we target at Llama-3 3.1 11--8--nsttene 1-9--nnst2a ruct, we select highly-activating tokens from the activation texts of the interference features shared by the two small models and extract the steering vectors from them. Through fine-tuning of the steering strength, we could notably boost the presence of relevant tokens in the top-10 prediction list with over 95% success rate in feature types such loca- tion, name and emotion, as shown in Appendix K Prompt injection interventions are tested on Ll Llama-3. 3. 11-88//0B-InSS some feature types. high-interference tokens derived from the two small models can steer larger models more effectively than random baselines. However, for several other feature types, the generalizability of their interference structures is not salient, as shown in Table
and
and Gemma 2-9--IInn IIBBInstruct. As shown in Table [ for
Notably, we find some specific counterintuitive interference pairs, such as location and datatypes in programming languages as shown in Table 6 that exist across more than two models. In hindsight, these results suggest that shared polysemantic structures observed small models also extend to larger models, indicating generalized vulnerabilities that persist across architectures and training regimes.
Table
Comparing intervention effect of prompt injection
Target
Model
Original
High-interference Low-interference Random
Locations
Pythia-70M
65.08%***
36.93%*
32.53%
35.06%
GPT-2-Small
14.68%*** ***
18.42%
19.08% ***
16.42%
Saama-31--BB-nsiiiee
03.84%***
20.78%***
19.63%*
18.24%
Gemma-2-9B I---rrut
10.16%
12.71% ****
11.36%
10.66%
Llama-3.1 1-BB Instruct
37.23%**
28.21%
23.09%**
24.48%
Number
Pythia-70M
65.32%***
35.15%
36.71 9 ****
34.30%
GPT-2-Small
55.87%***
30.33%
31 31.23%
34.71%
1-ama-3.1-8B-Instme
55.97%**. ***
31.87%*
32. 32.90% ****
30.57%
Denuma-2-9B-Instruue
18.42%***
29.90 ***
30. 30.64% #*:t
27.16%
Llama-3.1 1-70--Inttuctt
25.57%
8.85%
6.67%
7.09%
Science
Pythia-70M
61.66%
23.13%
22.78%
28.58%
GPT-2-Small
+
25.93%
26.07%
21.71%
Llama-3.1 1..1--B--Instre
19.67%***
20.08% (00%%%**
18.95%
17.94%
Gemmma-2-9B-Instruus
16.84%
20.40%
19.25%
20.15%
Llama-3.1-70B-
67. 67.26%***
18.22%
43.57%
42.24%
Note:
Cell values show the success rate of elevating target-type tokens into the top-30 predictions. Gray-shaded rows indicate black-box interventions. Testing uses shared token set from the two small models. ***, **, and e denote t-test significance p 0.001, p < 0.01, p 0.05, respectively. vs. random baseline. High- and low-interference tokens lie in [0.5, 1.0 and [0.2, 0.5) for Pythia-7005 while [0.3, 1.0] and (0.2, 0.3] in GPT-2-Smal 1. Details in Appendix
3.5 ANALYZING SHARED BUT COUNTERINTUITIVE POLYSEMANTIC STRUCTURE
We observe that many similar feature pairs in GPT-2-Smal1 and Pythia-70M that lie far apart in the first-order symbolic manifold, M (by /urface/gloss semantics), nevertheless lie close both models' activation space F. To understand this transferability, we consider mechanisms that couple
8

---

Published as a conference paper at ICLR 2026
features via higher-order semantic structure, including camaantc-priming--ee associations (e.g.. thematic/scripts, causal, frame roles) and morphological relatedness, even when overt meanings appear unrelated (Mandera et al. 2017 Bojanowski et al. 2017).
To probe these links, we conduct large-scale annotation with DeepSeek-V2 and GPT SEPT-5-mt 5-mini. Models are prompted to identify higher-order semantic relations for shared interfering feature pairs that satisfy three criteria: interference > 0.4, semantic similarity in M < 0.2, and cross-model feature-pair semantic similarity > 0.5. In total, 459, 2,22 pairs are annotated (prompts, rubric, and head-to-head annotator comparison in Appendix M). Only 27.7% of pairs are judged by at least one model to exhibit plausible higher-order association, and most such links remain counterintuitive upon post-hoc inspection. Because the association- checck task can be challenging for LLMs, we also run a conservative paired-choice evaluation. We sample 3,800 high-interference, low-similarity pairs. For each, we randomly draw comparison pair strictly matched on semantic similarity but with substantially lower interference, then ask GPT-5-min' n to select which pair is more related (details in Appendix M). The high-interference pair is chosen 64.3% of the time (Wilson 95% CI [0.628, 0.658)): ); a one-sided exact binomial test against 0.5 is significant (p < 0.001), corresponding to log-odds of 0.589 in favor of the high- high-interference pair, indicating a statistically meaningful yet modest effect. Taken together, these results both validate our filtering strategy for isolating unrelated pairs in the intervention analyses and point striking regularity: LLMs instantiate stable, cross-model polysemantic organization that is often opaque to semantic infuition.
In Appendix M. Table reports examples where models detect latent associations between feature pairs and where they do not. One notable case is the last, asterisked example in Table 8 both annotators label it as negative. In our follow-up analysis, however, we hypothesize biographical- affective link: mentions of "Beethoven" may co-occur with expressions of frustration//ufferingg given his late-life deafness and celebrated late-period compositions. These examples suggest that LLM polysemanticity may approximate latent knowledge structures and offer testable hypotheses. In summary, it possible that human studies could reveal comparable human recognition of the same "weak signals" picked up across models, even if they cannot recall or justify them. A full adjudication of this possibility lies beyond our present scope and we leave for future work.
3.6 MANIPULATING ACTIVATIONS FOR NEURON INTERVENTION
To complete our discussion, we explore models' vulnerability to interventions on individual neurons. Specifically, we investigate how the degree of polysemanticity in neurons affects the output. For aggregated features obtained through agglomerative clustering under the cosine similarity threshold of 0.4. we quantify each neuron's 's connected number of features, Here, we only involve neuron-fcature pairs with a connection strength greater than 0.2. Among all neurons, those connected to only one or two aggregated features account for more than 33% in strongly connected neurons, as shown in Figure 17 addition to neurons connected to multiple or dozens of aggregate features, there are also some "super-neurons" with connections exceeding 500. We examine the impact of manipulating these neurons on the model's 's output. Experimental results, as shown in Figure indicate that neurons with higher degrees of polysemanticity outputs more effectively. For certain "super-neurons," however, the impact on the model is notably asymmetric: masking them results in less influence than neurons with lower polysemanticity, while amplifying their activations often leads to exponentially greater effects on model behavior.
more vulnerable, which means they tend to affect model
4 DISCUSSION
This work makes two contributions. First, we systematically investigate the vulnerability of LLMs to structured interventions grounded in their polysemantic representations. Specifically, we examine three types of intervention: (1) feature pirection-based (2) token gradient-based, and (3) prompt- based. Feature direction interventions rely on SAES. While less effective than gradient-base approaches, they form the basis for deriving token gradient vectors. Token gradient-based inter- ventions are most effective and can be constructed directly from activation texts, without requiring SAE pre-trining--llhouugh they assume access to internal activations. Prompt-based interventions require minimal access and, despite their surface-level nature, still yield meaningful behavioral shifts. Additionally, we explore neuron-level interventions, motivated by the uneven distribution of features across neurons. We find that the behavioral impact of masking and amplification correlates

---

Published as a conference paper at ICLR 2026
Weigtted sinfaity slopp
0:17
Weigheed casire simlliity drop
0.00
8:10
Weighted costne siillartty
0.70
014
00:4
0.0
0.0
0.00
0.0
8.64
0.0
Weighted
0.0-
0.2
000
Connected aggregated features
Connected aggragated features
Connected aggregated features
Connected aggregated features
Figure 6: Effects neuron activation and suppression on model behavior depend on neuron polysemanticity level. The z-axis indicates neuron categories grouped by the number of connected features (after clustering). The y-axis reports the change in weighted cosine similarity. Each box plot centers on the median (50%) and progressively splits the remaining data in half at each level. A and correspond to GPT-2-Sma 122-maall :; and D correspond to Pythia-70M. A and C show the effect of masking neuron activations, while B and D show the effect of amplifying them.
with neuron polysemanticity. We also identify a class of "super-neurons," those encoding over 500 features, for which amplification significantly alters model behavior, while deactivation results in markedly reduced effect.
The second contribution lies in our finding that polysemantic, structural vulnerabilities iden- tified in two small models transfer to larger, instruction-tuned black-box models (e.g.. Gemma
L. alama-3. 1-0.11111/7000--tttttt level manipulations, producing predictable behavioral shifts. This suggests that certain polysemantic structures are preserved across architectures and training regimes, exposing a shared representational basis. This directly challenges prevailing theories that treat polysemanticity
via token-gradient- and prompt-
incidental artifact of training (Marshall& & Kirchner 2024] Lecomte et al.) (023). Our exploratory analyses further suggest that these transferable polysemantic structures not reducible higher-order relations readily intel- ligible to humans; we therefore treat these counterintuitive regularities as testable hypotheses about latent knowledge structure. These results sharpen central question about LLM polysemanticity:
they unintended byproducts or stable, higher-order patterns that await rigorous examination? Our results point towards the latter. Finally, our findings strengthen recent evidence representational consistency and topological stability across models (Huh et al. [2024] Wolfram & Schein, 2025 Lee al., 2025), even as the origins and functional implications of this consistency remain open challenges. Our work is the first to systematically evaluate polysemantic structural vulnerabilities in real-world LLMs, but it has several limitations, including intervention depth and transferability robustness tests. We discuss these limitations and ethical considerations in Appendix P
CONCLUSION
We systematically probe the vulnerability of LLMs to structured interventions grounded in the polysemantic representations of two small models using SAEs. We show that model behavior can be steered toward specific feature directions by manipulating semantically unrelated yet interfering features via three intervention methods. Interventions distilled from polysemantic structures shared across the small models transfer to larger, black-box instruction-tuned models, indicating stable and transferable polysemantic topology that persists across architectures and training regimes. Post-hoc annotation suggests that fewer than 30% of these shared interference structures align with higher- order relations readily intelligible to humans; many counterintuitive cases may therefore serve as generators of testable hypotheses about latent knowledge structure. Finally, by leveraging the uneven distribution of features across neurons, we assess models' sensitivity to neuron-level manipulations across degrees of polysemanticity and reveal asymmetric effects in "super-neurons," Together, these findings provide foundation for future work on the structural properties, vulnerabilities, and representational robustness LLMs.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, et al. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits, 2025.
Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety a review, 2024. URL
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the association for compulational linguisties, 5:135-146, 2017.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Anthropic, 2023.
Gonçalo Paulo Caden Juang, Jacob Drori, and Nora Belrose. Open source automated interpretability for sparse autoencoder features. EleutherAl Blog, July, 30, 2024.
Ruidi Chang, Chunyuan Deng, and Hanjie Chen. Safr: Neuron redistribution for interpretability. arXiv preprint arXiv:2501.16377 2025.
David Chanin, James Wilken-Smiin Tomáš Dulka, Hardik Bhatnagar, and Joseph Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. arXiv preprint arXiv:2409. 14507, 2024.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.088: 006000 2023.
Maximilian Dreyer, Erblina Purelku, Johanna Vielhaben, Wojeiech Samek, and Scbastian Lapuschkin. Pure: Turning polysemantic neurons into pure features by identifying relevant circuits. In Proceed- ings of the IEEE//VF Conference on Computer Vision and Pattern Recognition, pp. 8212 2217, 2024.
Jacob Dunefisy,, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable Ilm feature circuits. arXiv preprint arXiv:2406. /994,, 2024.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCan- dlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, 2022. BRRLhttpe
wttp://.rrr///bb/2..2.00000 arxiv. org abs
Eoin Farrell, Yeu-Tong Lau, and Arthur Conmy. Applying sparse autoencoders to unlearn knowledge in language models. arXiv preprint 11XX::2410.11278.. 2024.
Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-Jussà. A primer the inner workings of transformer-base language models. arXiv preprint 2000:22000000000 2024.
Alex Foote. Tackling polysemanticity with neuron embeddings. arXiv preprint arXiv:2411.08166 2024.
Stanislav Fort. Scaling laws for adversarial attacks on language model activations. arXiv preprint arXiv:2312.02780. 2023.
Leo Gao. Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and leffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint 0tle:24066000933 2024.
11

---

Published as a conference paper at ICLR 2026
Robert Geirhos, Roland Zimmermann, Blair Bilodeau, Wieland Brendel, and Been Kim. Don't trust your eyes: on the (un) reliability of feature visualizations. arXiv preprint arXiv:2306.04719 2023.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 6(3)e30, 2021.
Liv Gorton and Owen Lewis. Adversarial examples are not bugs, they are superposition, 2025. URL https://arxiv htrg/abs//500 17456
Thomas Hcap. Tim Lawson, Lucy Farnik, and Laurence Aitchison. Sparse autoencoders can interpret randomly initialized transformers. arXiv preprint arXiv:2501.1772) 2025.
Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, and Yang Zhang. Composite backdoor attacks against large language models. arXiv preprint arXiv:2310.07676, 2023.
Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. arXiv preprint urXiv:2202.0344 034423, 2022.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic representa- tion hypothesis. In Forty-first International Conference Machine Learning, 2024.
Matthew Khoriaty, Andrii Shportko, Gustavo Mercier, and Zach Wood-Doughty. Don't forget it! conditional sparse autoencoder clamping works for unlearning. arXiv preprint arXiv:2503.1112, 2025.
Michael Lan, Philip Torr, Austin Meek, Ashkan Khakzar, David Krueger, and Fazl Barez. Sparse autoencoders reveal universal feature spaces across large language models. arXiv preprint arXiv:2410.06981, 2024.
Victor Lecomte, Kushal Thaman, Rylan Schaeffer, Naomi Bashkansky. Trevor Chow, and Sanmi Koyejo. What causes polysemanticity? alternative origin story of mixed selectivity from incidental causes. arXiv preprint arXiv:23/2.03000 2023.
Andrew Lee, Melanie Weber, Fernanda Viégas. and Martin Wattenberg. Shared global and local geometry of language model embeddings. arXiv preprint arXiv:2503.21073. 2025.
Pawel Mandera, Emmanuel Keuleers, and Mare Brysbaert. Explaining human performance in psycholinguistic tasks with models of semantic similarity based on prediction and counting: A review and empirical validation. Journal of Memory and Language, 92:57-78, 2017.
Samuel Marks, Can Rager, Eric Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:22003. 19647, 2024.
Simon C. Marshall and Jan H. Kirchner. Understanding polyc in neural networks through coding theory, 2024.
Aashiq Muhamed, Jacopo Bonato, Mona Diab, and Virginia Smith. Saes can improve unlearn- ing: Dynamic sparse autoencoder guardrails for precision unlearning in 1lms. arXiv preprint arXiv:2504.08192, 2025.
Neel Nanda. An extremely opinionated annotated list of my favourite mechanistic interpretability papers v2. In A/ Alignment Forum, volume 2, 2024.
Tuomas Oikarinen and Tsui-War Wei Weng. Linear explanations for individual neurons. arXiv preprint arXiv:2405.06855, 2024a.
Tuomas Oikarinen and Teui-Wei Weng. Linear explanations for individual neurons, 2024b. URL https://arxiv.oxxt http://vw.rggbb///////a//
Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Tumnrr Steering llama via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2023.
12

---

Published as a conference paper at ICLR 2026
Gonçalo Paulo and Nora Belrose. Sparse autoencoders trained the same data learn different features. arXiv preprint arXiv:2501.16615, 2025.
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407. 14435, 2024.
Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. A survey on sparse autoencoders: Interpreting the internal mechanisms large language models. arXiv preprint arXiv:2503.05613. 2025.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interprelable features from claude 3 sonnet. Anthropic, 2024.
Jiachuan Wang, Shimin Di, Lei Chen, and Charles Wang Wai Ng. Learning from emergence: study on proactively inhibiting the monosemantic neurons of artificial neural networks. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, KDD '24 pp. 3092-3103 3103, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704901. doi: 10.1145/3637528.367111 7777722886677777 URLhhtps:://aa wttp:/....../11..11.......... 3637528. 3671776
Christopher Wolfram and Aaron Schein. Layers at similar depths generate similar activations across Ilm architectures. arXiv preprint 2xXi:25544008755 2025.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, Zico Kolter, and Matt Fredrikson, Universal and transferable adversarial attacks on aligned language models, 2023. URL httthttts///aaa
org/abs/2307. 15043
Gao Zuchen, Zizheng Zhan, Xianming Li, Erxin Yu, Haotian Zhang, Chenbin Chenbin, Yuqun Zhang, and Jing Li. OASIS: Order- auggeeed strategy for improved code search. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 18451-18461 Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN doi: 10.18653/w1/200 1/2025.222222/m lolgoong.... URL https: htttp::/aclLattholoog/
979-8-89176-251-0,
prg/2025. acl long. 904/
13

---

Published as a conference paper at ICLR 2026
CONTENT OF APPENDIX
A Related Work
15
A.1 Brief Review on LLM Adversarial Interventions
A.2 A Brief Review on SAE-based Intervention Techniques in LLMs
15
Impact Statement
16
The Use of Large Language Models (LLMs)
16
D Sparse Autoencoder Training
Dataset Generation
18
Supportive Statisties
21
G Definition of the Alternative Metric: Weighted Overlap
22
H Intervention Test with Feature Direction
23
H.I Generalized Formulation of Steering- Veetron--Vector Intervention
23
H.2 Experiment Details
I Intervention Test with Token's Gradient
I.1 Token Gradient Direction Extraction
I.2 Experiment Details
J Experiments with DeepSeek-V3 Feature Descriptions
25
K Black-Box Interventions on Larger Models
28
K.1 Steering with Token Gradient Vector
28
K.2 Prompt Injection
30
Covert Intervention on Hellaswag Testset
M Transferable Polysemantic Structure Exploration
M.1 Brainstorming hidden associations
M.2 Feature-pair comparison test
Polysemantic Neuron Manipulation
Denoising Intervention Effect
39
Ethics Statement, Limitations and Future Works
Reproducibility Statement
14

---

Published as a conference paper at ICLR 2026
A RELATED WORK
A.1 A BRIEF REVIEW ON LLM ADVERSARIAL INTERVENTIONS
Over the past five years, growing body of high-impact research has revealed that even aligned LLMs remain vulnerable to a set of converging attack strategies. First, prompt-space jailbreaks have evolved from handcrafted exploits into automated, highly transferable methods. For instance, single
"universal suffix" can consistently bypass refusal policies in ChatGPT, Bard, Claude, and a wide range of open-source models demonss-ddmoossraatinnn both query efficiency and cross moss-model generalizability (Zou et al., 2023). Second, activation spacce steering techniques like Contrastive Activation Addition (CAA) show that simple linear interventions in the residual stream can steer behaviors such as hallucination, sycophancy, or toxicity with minimal performance degradation (Panickssery et al., (023). Third, parameter-space backdoors, such as the Composite Backdoor Attack, embed stealthy triggers during fine-tuning that achieve near-perfect malicious compliance without affecting standard benchmarks (Huang et al. (023). Mechanistic interpretability offers a unifying explanation: transformer activations encode more features than they have dimensions, forcing representations into a compressed superposition and leading to widespread polysemantic overlap (Elhage et al. (022). Recent work with SAES has begun to isolate and in some cases manipulate these overlapping features directly (Nanda (024). Building on this insight, our intervention targets SAE-derived polysemantic directions, integrating prompt-, activation-, and neuron-level interventions into unified, transferable framework that broadens the known landscape of LLM vulnerabilities.
A.2 A BRIEF REVIEW ON SAE- BASSDASED INTERVENTION TECHNIQUES IN LLMS
SAE-based interventions represent promising direction for developing more interpretable and controllable LLMs. Recent research have introduced a diverse set of SAE-based techniques, such as clamping, patching, and causal tracing, applied across range of use cases (Farrell al. 2024] Cunningham et al. 2023: Marks et al. 2024). Empirical results indicate that these methods can be highly effective. For example, targeted unlearning via SAE features has been shown to suppress undesired capabilities with fewer side effects than global fine-tuning (Khoriaty et al. [025 Muhamed jet al., 2025), while feature-level steering enables more nuanced output control than prompt- methods alone (Rajamanoharan et al. 2024). A key advantage of SAE-based approaches is their efficiency at inference time: they often require only a forward pass with lightweight vector operations and typically do not require model retraining, making them well-suited for real-time interventions.
However, the approach is still in its early stages. Key limitations include challenges in achieving complete and disentangled feature representations, which depend heavily on SAE training quality and selection procedures (Chanin et al.] (024). Computational overhead remains non-trivial, though recent developments such k-sparse autoencoders and JumpReLU activations offer promising improvements in scalability (Rajamanoharan et al. 2024). There is also a growing need for standard- ized evaluation benchmarks tailored to intervention methods. A unified benchmark would enable more meaningful comparisons across studies. Currently, researchers often rely on custom evaluation protocols, limiting cross-paper comparability.
In summary, SAE-based interventions offer a powerful mechanism for both understanding and steering model behavior. They uniquely bridge interpretability and utility: not only can we decode model activations into human- mamaannntteprettbble concepts (Cunningham et al.] 2023). but we can also use those same features to drive controlled behavioral change (Khoriaty et al. 2025). In this specific downstream application, we leverage SAES to investigate
work, rather than focusing structural sensitivities in LLMs deMonsdtmnstttiinn that polysemantic features can serve as a substrate for transferable, interpretable interventions. This perspective highlights the broader role of SAEs in the design of more transparent and controllable Al systems.
15

---

Published as a conference paper at ICLR 2026
B IMPACT STATEMENT
This work systematically investigates a semantic vulnerability in LLMs rooted in polysemanticity- where single neurons encode multiple semantically dissimilar features. We introduce four com- plementary approaches that expose this vulnerability: manipulating SAE-derived features, token gradients, and prompts to steer model outputs via semantically unrelated inputs, and intervening at the neuron level to reveal a correlation between polysemanticity and output sensitivity, We also identify class of "super-neurons" whose amplification disproportionately alters model behavior, while masking them has limited effect. These findings not only highlight the unique characteristics of the structural fragility L.LMs but also provide practical tools for probing and controlling their internal mechanisms. Our work lays a foundation for future research in AI safety and mechanistic in- terpretability, not only enabling defenses against such vulnerabilities and more targeted interventions for alignment, but also offering theoretical lens into the model's 's internal organization, revealing stable yet counterintuitive interference patterns that may reflect a form of unconscious knowledge association.
C THE USE OF LARGE LANGUAGE MODELS (LLMs)
LLMs play significant role in study. We list the usages of LLMs below.
Synthetic data generation. For the dataset used for intervention experiments, we use DeepSeek-V3 to generate incomplete sentences for next-token completion. Details are elabo- rated in Appendix
SAE feature auto-interpretaaion.. by GPT 4 4-mini. For cross-validation, we also annotate subsample of SAE features using DeepSeek-V3. Details Appendix !
The SAE feature glosses in the main experiments are generated
Feature pair association analysis. To investigate whether the seemingly unrelated interference feature pairs present higher-order associations that are still comprehensible, we ask T- 5-mini and DeepSeek- to label these feature pairs for post-hoc check. Details in Appendix M
Grammar check for paper writing. We use LLMs to refine phrasing, correct grammar, and im- prove readability during manuscript preparation, while all substantive idcas, analyses, and conclusions remain our own.
16

---

Published as a conference paper at ICLR 2026
D SPARSE AUTOENCODER TRAINING
SAEs are rapidly developing tool for probing the polysemantic structure of neurons (Shu et al. 2025). Given the activation vector E Rdaated from particular model layer, an SAE projects into nigher-dimensional sparse code E IR in order to disentangle the multiple semantics that a single neuron may simultaneously encode. The forward computation and the resulting feature definition
are shown below:
f = Act(W Wenca + bens).
T=AAc(WW
W + bdec-
The encoder and decoder parameters are
Wene
R Xcennted
Waee
Remee Xdex
Dene
RRdor
baec
R
where Act is an activation function, such as ReLU and TopK for Pythia-70M and GPT-2-Small. The SAE is trained by dictionary learning to minimize
L a A
Σ
W
where the first term is the reconstruction loss and the second encourages sparsity (weighted by X). For each feature fü its direction in the embedding space is defined as the uni--norm decoder column.
Wdee [-.a]
d
Wace (-11l12
17

---

Published as a conference paper at ICLR 2026
E DATASET GENERATION
The investigation of the interference effects SAE features conducted within the contexts where they likely fire, which allows us to manipulate the expression of these features without significantly compromising the model. For example, when given the prompt "In the weekend, we are going to", the features related to locations are incorporated into our examination. Additionally, observe a positive correlation between the boost of SAE feature and the increased probability of its nigh-activation tokens the model's next-token predictions. Therefore, here we simply construct sentences for each token in the model's vocabulary, ensuring that these sentences are grammatically capable of leading to the token and evaluate the expression of features with respect the tokens' output probabilities.
System: Generate exactly 3 incomplete English sentences where the next word would clearly be "target_token". Return JSON dictionary where: The ONLY key is the exact "target_token" (including pances/capitallzation
The value is list of 3 sentence fragments that naturally Lead to "target_ tokkn"
Example for "target_token" apple'":
apple":
"Sho reached into the basket and grabbed", "The teacher pointed to the red",
"e washed and polished his"
Rules:
1. A11 sentences MUST grammatically requi "target_token" next to it
2. Use different contexts / scenarios for variety 3. Kaintain exact formatting additional keys explanations User: (bsr))arget_tokken(ttooco
We also DeepSeek-V3 to roughly classify the token types: System: You are a linguistic analyzer. Your task is to classify tokens from Language model vocabularies into semantic categories.
Given list of tokens, classify each token into ONE of these categories:
person: Names, pronouns, occupations, human-related terms Examples: John", Mary", dector", teacher", he", she", people"
Cities, countries, geographical features, places kxamples: London", America", mountain", beach", city", Paris", China"
location:
time: Time units, dates, temporal expressicns, seasons
Examples: Monday", January", morning", year", " day", " week",
winter"
number: Numerical digits, number words, mathematical terms Examples: one", two", 1", 2", first", hundred", plus", minus"
animal: All living creatures, insects, pets, wildlife
Examples: dog", cat", bird", lion", fish", elephant",
butterfly"
food: Edible items, drinks, cooking terms, ingredients
Examples: apple", bread", water", coffee", chicken", rice", pizza"
18

---

Published as a conference paper at ICLR 2026
color:
Color names, shades, visual descriptors
Examples: red", blue", green", black", white", yellow", purple"
Feelings, emeticnal states, psychelogical terms Examples: happy", " sad", angry", " love", fear", excited", worried"
cmotion:
Human/animal bedy parts, anatomy Examples: head", hand", eye", heart", leg", " face", brain"
body:
Vehicles, transportation methods Examples: car", bus", train", plaan", ship", bicycle", truck"
transport:
science: Scientific terms, elements, physics/chemistry concepts Examples: oxygen", energy", atom", DNA", " gravity" " electron", acii"
Philosophy, ideas, concepts, mental constructs Examples: freedom", justice", truth", idea", concept.", theory", belief"
abstract:
object: Physical items, tools, furniture, equipment
Examples: table", chair", book", phone", computer", tool", box"
action: Verbs, activities, movements, processes
Examples: run", walk", eat",, think", write", jump", s sleep"
unknown: Anything that deesn't clearly fit into the above categories
Return JSON object where each key is a token and each value is its category.
Rules:
1. Classify ALL provided tokens
2. Use EXACT token strings as keys (including spaces)
3. Choose the MOST appropriate single category
4. Use lowercase category names only
5.
When in doubt, use "unknown" Return ONLY the JSON object, ne additional text
6.
User: Classify these tokens:
token str_l, token 2, ....
19

---

Published as a conference paper at ICLR 2026
Here provide table of some example tokens and their sentences generated by DeepSeek-V3 (See Table
Table 2: Token type and prompt sentences examples
Token
Type
Sentence Examples
London location
After long flight, we finally arrived in The train from Paris was heading straight She always dreamed of visiting the historic city of
harbor
location
The cruise ship slowly approached the bustling Fishermen gathered at the edge of the protected The city's 's economy thrived thanks to its busy
Mike
person
After the meeting, everyone turned to The teacher called on She handed the report directly
Trump
person
The media has been closely following the latest statements from During the debate, the moderator asked direct question to Many supporters gathered outside the venue to catch a glimpse of
expert
person
After years of practice, she became an The company hired an
When it comes to antique furniture, he's an
loves
emotion
She truly believes that everyone The way he looks at her shows how much he
Despite their differences, their friendship
hates
emotion
Everyone knows that she
The way he treats people shows he It's ss clear from his expression that he
apple
object
She reached into the bag and pulled out The smoothie recipe called for one chopped He carefully balanced the shiny red.
sad
emotion
After hearing the bad news, she felt incredibly The movie's 's ending left everyone feeling His eyes told story of being deeply
happy
emotion After receiving the good news, she felt extremely The children were laughing and playing, clearly very-
Winning the competition made him incredibly
20

---

Published as a conference paper at ICLR 2026
F SUPPORTIVE STATISTICS
Active features refer to SAE features that have input texts enabling them to reach an active state. In addition to the semantic clustering of active features mentioned the main text, also apply agglomerative clustering to cluster their interference values. The threshold for dividing clusters is set 0.4. As shown Figure g the vast majority clusters contain only one feature, indicating that only small number of features exhibit high interference with others.
In Figure 9 we also provide an overview of the distribution of semantic similarity and interference values between features, as well as their correlations. Since the distribution and correlation of these two types of data remain largely consistent across all layers of the model, we present representative example layer for illustration.
32,122
32800
18000
32000
14000
32400
32,135
12000
30200
1000
8020
15000
31900
4000
31600
Sub-Module Type
Sub-Modile Type
Figure 7: Number of active features extracted by SAEs per layer. is the result of Pythia-7000 and B is the result of GPT-2 2.mmal. Error bars represent 95% confidence intervals.
A
B
662,POST
10
Clattrr Counts
40
3.5
Figure 8: Interference Cluster Size Distribution. A is the result of Pythia- Mikiia-70 -70M and B is the result of GPT-2-Sma 11. Error bars represent 95% confidence intervals.
21

---

Published as a conference paper at ICLR 2026
4.0
EER
0.3
3.5
3.0
02
2.5
Denotty
0.1
2.0
1.5
1.0
0.5
0.0
Somantic Similoitty
Interforence Valuo
Semantic Similarity
F
4.0
0.23
3.5
3.0
0.4
2.5
Denshty
2.0
1.5
0.0
1.0
-0.2
0.5
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
Semantic Similarity
Interference Value
Semantic Similarity
Figure 9: Semantic similarity and interference distribution of Pythia Res-4 and Post-4 layer. (A-C) are the results of Pythia-Res-4, For semantic similarity, 14.0% of values are below 0.15, 29.6% are below 0.2, 67.4% are below 0.3, and 89.6% are below 0.4. For interference, 85.5% of values are below 0.1, 96.4% are below 0.2, 99.2% are below 0.3, and 99.8% are below 0.4. The bivarite analysis suggests semantic similarity and inter- ference value is positively associated (8 = 0.071, s.d = 0.092, p < 0.001). (D-F) are the results of GPT-2-Sma1 Res Post-4. For semantic similarity, 10.8% of values below 0.15, 24.5% are below 0.2, 63.3% are below 0.3, and 89.5% are below 0.4. For interference, 95.6% of values below 0.15, 99.7% 7% are below 0.2, approximately all values are below 0.3 and 0.4. The bivarite analysis also suggests a positive association (3 = 0.030, s.d = 0.049, p < 0.001).
GPT2-Smal1
G DEFINITION OF THE ALTERNATIVE METRIC: WEIGHTED OVERLAP
In addition to the weighted cosine similarity, we report results with an alternative metric, weighted overlap. which measures the raw probability mass assigned the feature-associated token set Ty C. V in Section H and This metric does not smooth over near misses via embedding similarity; instead, it directly captures how much of the model's 's next-token distribution lands on tokens Ty.
Definition.
For a model output distribution O II and target token set Ty, w( (O.TT,)
Σ O(t). TET,
(3)
Intervention effect. tion. respectively. The (absolute) change weighted overlap is
Let O and denote the model's output distributions before and after interven-
Aw w(, Ty) w(O, Ty)-
Relative change.
When feale-free summary is preferred, we also report the relative change: w(O Ty) w(, max max
Aw
(5)
where is small constant avoid division by zero
22

---

Published as a conference paper at ICLR 2026
H INTERVENTION TEST WITH FEATURE DIRECTION
H.1 GENERALIZED FORMULATION OF A STEERING-VECTE INTERVENTION
Let æir {1,..., Vj" be the imput sequence, E Rxxd be the token-embedding matrix, and 9 to Or be the blocks of decoder-only Transformer. The unperturbed hidden states are
Ho E[ziTT,,
H₂ = 9e (He-1)
1.=1.....
For any layer index p. we denote the vectorized activation as
Ap = vec (Hp) E RRd
With different strategies, we extract the steering direction Ep RdXT For injection at site we define the linear Jacobian:
p-ss RdxT Rd
obtained by composing linear portions between indices p and 8. The transported steering direction is
p pp
if 8
2g
if < p
where i denotes the Moore Penrose pseudo-invesd inverse. When = p. we set 2g Zp. Eventually, modify the activation at site s:
A, A, azzg.
The network proceeds normally with this perturbation, yielding modified hidden states He and logits V1.r-
H.2 EXPERIMENT DETAILS
To obtain the complete intervention data various levels of interference features on the target features, we first filtered out all features in each layer that contained interference values at all levels, and for which the semantic similarity with the target feature is below the four selected thresholds. Subsequently, select subset of these features the target features and identify the corresponding interference features across the various interference levels. Next, we search for other interference features with low interference values the target features for experimentation. Specifically, other in- terference features are selected based on the interference values lying in intervals:(0.am 0.1]. [0.1, 0.2]. [0.2, 0.3] and [0.3, 0.4]. The scale parameter was tested within the range [-20, 20), and we avoid larger ranges to prevent severe disruption of the model.
Due to limitations on computational power, we sample clusters and features across various layers. In each SAE experiment with aerthia-70p. we sample 180 target features and collect approximately 2,700 interference features. In each SAE experiment with GPT-2-Sma we sample 480 target features and collect approximately 7, 200 interference features. For gradient experiments, we reduced 60% sampled features, but three times the number of gradient intervention vectors are extracted to keep the test set scale. As mentioned in the main text, for each feature, focus its top-activating token and use DeepSeek-V2 to generate three prompt sentences for it. For each sentence, we test within the aforementioned scale range and record the result with the greatest improvement in the two metrics. This result means the best performance that the steering vector can achieve to induce the semantics of output toward the target feature without significantly disrupting the model. The final two metrics are averaged across all sentences for all features. To show the robustness of our experiments, the results of the alternative metric are presented in Figure TO
23

---

Published as a conference paper at ICLR 2026
Pydta Threehald
G77-2 Threshols
Confidrre Intarak
Dalnha toundd Sirgat Pattar
Wishni u.
100
Edo
Coll -000
0.00
Figure 10: Feature-level intervention effects measured by weighted overlap. Subpanels follow the same conventions as Figure 3 but intervention effects are computed using relative change of weighted overlap (Aw). For (A-B), B), error bars indicate 95% confidence intervals; for (C), error bars denote 90%, 95%, and 99% confidence intervals.
I INTERVENTION TEST WITH TOKEN'S GRADIENT
1.1 TOKEN GRADIENT DIRECTION EXTRACTION
Given tokenized input sequence x [ Hy and [eo,...y .. -T-1) i the full input embedding sequence. Let RRxx AA denote the model's 's transformation up to layer L. The activation at position i is:
IT. let e; E[zs] denote the embedding of token
ag fe [e) A
We define a scalar probe loss that selects this activation via a linear projection vector v E R4:
L ( r)-
The gradient of this loss with respect to the input embedding is:
gr = OC A(a,,,,))
de; de;
We then normalize this vector to obtain direction in embedding space:
Ôc.s = g8,4
11ge, 12
We refer to 9e, as the token gradient direction the direction input embedding space along which perturbations to token r; most increase its activation in A along
1.2 EXPERIMENT DETAILS
Steering with the feature direction requires a pretrained sparse auto-encoder of the target model, which incorporates substantial computational costs and lacks scalability. To break this limitation, we need to explore a general approach. Observe that the SAE features are activated mainly by the top-activating token in its activation texts, while other tokens are just diluting its expression. Based on this observation, we can obtain better steering vector by focusing on this particular token, and sketch is follows. We first feed the feature's activation text into the model, then compute the gradients of the top-activating token with respect to all neurons in the layer. The resulting gradients are combined to form a vector.
The SAE dataset from Neuronpedia contains approximately 50 activation text segments per active SAE feature, each strongly activating its corresponding feature. Due to computational limitations, we try to extract the gradient vectors from the first 3 activation texts of each feature. Also, we scale the vector within the same range [0.5, 1, 1.5, 2, 3, 4,5, 5, 6, 7, 8, 9, 10, 12, 14, 17, 20). Each experiment on steering with token gradients combined with steering with feature directions can be done in hour running on a single thread of Intel 17- 14.0K0... The results of the alternative evaluation metric presented in Figure 11
and a half for Pythia- 70M and six hours for GPT-2- (-er-2-Smal
24

---

Published as a conference paper at ICLR 2026
Table Examples of interventions using SAE features, token gradients, and prompt injections
Type
Model
Intervention
Target feature
Result
Feature PYthia-70M
Steering
feature
Geographical loca-
"In the next week, will to"
vector: occur- tions
1 Entered
Dropped
rences of specific
Berlin +0.025
our
-0.029
surnames
London +0.012
some
-0.012
To
+0.010
-0.010
GPT-2-Smai
Steering feature Expressions of sad- vector: positive negative event
Affier hearing the bad news, she felt incredibly"
ness
t Entered
Dropped
outcomes
grateful +0.051
-0.047
blessed +0.028
guilty
-0.044
excited +0.028
uncomfortable
-0.025
Token Pythia-70M Stecring token vee- tor: legal terminol- ogy related to li- censes and their plications
Elements related to political commen- tary and critique
"In the election of this year. it
is suggested to vote for" Entered
4. D.ropped
Donald +0.030 an more
-0.020
+0.026
one -0.015
+0.015 President
-0.013
GPT-2-Small
Steering token vec- References to loca- tor: key terms tion Tokyo
"The organizing commmittee just announced that the upcoming finals will be held in'
lated to prices and transactions
Entered
Dropped
Tokyo +0.005
Toronto -0.007
Scoul
+0.003
Scattle -0.005
Moscow
+0.004
London -0.001
Llama-3.1- 8b-Instruct
Steering gradient vector: references to the world and its various aspects
References 'Switzerland"
to
"I would like to recommend you spend holidays in"
Entered
J Dropped
Switzerland
+0.16
Italy -0.034
Germany
+0.089
Greece -0.016
Canada
+0.015
Bulgaria -0.012
Prompt 70M
Injection of the to- kens "Court" and tions "Dat", both before and within the text
References to loca-
"In the upcoming holiday, we will
go to"
Entered Japan +0.021
+ Dropped
some -0.014
Europe +0.015
an
-0,006
Tokyo +0.012
see
+0.003
GPT-2-Small
Prepending the in- Terms related to jection text "(team names surnames writers writers)"
"After years of hard work, the award finally went to"
Entered
Dropped
Steve +0.005
China
-0.006
John +0.002
waste
-0.005
one
+0.003
Donald
+0.003
1lama-3.1- 8h-Instruct
Prepending the in- References to loca- jection text "(place- tions ment from place- ment)"
"In the next weekend we will go to"
Entered
+ Drropedd
Paris
+0.011
another
-0.013
#8
+0.010
-0.003
+0.006
K
-0.003
Note:
Entered means that corresponding tokens entered the top-10; 10; 4 Dropped means that corresponding tokens dropped from the top-10. Gray-shaded rows indicate black-box interventions.
J EXPERIMENTS WITH DEEPSREX-VS FEATURE DESCRIPTIONS
To evaluate the sensitivity of our findings to the selection of LLMs for auto-interpretation, we employ alternative model, DeepSeek-V3, to interpret the features of sparse auto-encoder. The
25

---

Published as a conference paper at ICLR 2026
Pytko Threshakt
SPT Thaooieed 1odh-33
Corkirorr letervals
1/50
cmd +0.3 2015 +0.13
1300
co00-1115
150
1000
250
Figure 11: Token-level intervention effects measured by weighted overlap. Subpanels follow the same conventions as Figure 4 but intervention effects computed using relative change of weighted overlap (Aw). For (A B), error bars indicate 95% confidence intervals; for (C), error bars denote 90%, 95%, and 99% confidence intervals.
explanation text is subsequently processed by the fext--embedding 3-l lage model to obtain embeddings. These embeddings then utilized semantic vectors assess the similarity between features. For this illustrative sampling, features from the Pythia-70M att-5 layer are selected. The interpretation of features by large models generated by feeding the model the top-activating texts of the feature and denoting the high activating tokens in it. Specifically, the prompts we write for DeepSeek-V3 are listed below.
System: We' ze studying neurons in neural network. Each neuron activates on some particular word or concept in short document. The activating words in each document are enclosed by < and >. Look at the parts of the document the neuron activates for and summarize in a single sentence what the neuron is activating Don't just repeat activation words. Also, you can summarize multiple points if the text content is not highly consistent. Pay attention to things like the capitalization and punctuation of the activating words concepts, it that seems relevant. Keep the explanation short and simple as possible, limited to 32 words or less. Omi punctuation and formatting.
Try to be general in your explanat ions.
(ser)) The activating documents are given bolow: a.ccivvatio_ttox___
2. F.activation_text_2
5. bectivationteet....
In Figure 12 we compare semantic relatedness among SAE features using embeddings of explana- tions generated by DeepSeek-V2 and mi. The left density plot shows that pairwise similarities from the two models are tightly aligned, while the right heatmaps further illustrate that both models induce comparable feature-featur semantic structure.
Figure 13 compares SAE feature-dircction and gradient-basee interventions under auto-interpretations from GP GTT-4o-min and DeepSeek-V3. Semantic dissimilarity threshold is set to different scales for intervention feature selection. Although this check is limited to a single layer (and thus exhibits greater variance), consistently observe that high-interference, now-semanticssmiilarii features steer the target far more strongly than a random baseline. This pattern holds under both interpreters, indicating robustness the choice of auto-interpretatton model.
26

---

Published as a conference paper at ICLR 2026
Simiarity
GFT-lo-minl
DeepSaek-V3
4.0
Dresteska
0.8
3.0
2.5
0.0
2.0
1.5
1.0
0.
-02
0.0
0.2
0.4
0.6
0.8
1.0
Semariicc Simdartty
Figure 12: Semantic relatedness between features from the view of DeepSeek-V3 and GPT-4o-mini.
Thresholds (shade encodes threshold)
c..4000.0
cos B 0.30
co2 x 0.20
8 0.15
Categories
Original (swll)
Interfered (:0.2)
Rendom
SAE-based Weighted Cosine Similarity
SAE-based Weighted Overlap
Relative change toward target feature
Relative change target
6 10
4 10
15
Gradient-based Weighted Cosine Similarity
Gradient-based Weighted Overlap
target feature
Relative change toward target feature
20
Relative change
DeepSeek-V3
GPT-4o-mini
DeepSeek-V3
GPT-4o-mini
Figure 13: Intervention effeets under GPT-4o-mini DeepSeek auto-interpretations of SAE features. Top row shows SAE-direction interventions; bottom row shows gradient-based interventions. Columns are the evaluation metrics: left, Ac; right, Aw. Error bars denote 95% confidence intervals.
27

---

Published as a conference paper at ICLR 2026
K BLACK-BOX INTERVENTIONS ON LARGER MODELS
We hypothesize that the polysemantic structures learned by large language models may exhibit some degree of generalizability. To explore this further, the interference study extended to larger models without pretrained sparse auto-encoders.
K.1 STEERING WITH TOKEN GRADIENT VECTOR
The scalable intervention on Llama 3, 3.1-88 8B Instruct conducted by first selecting target type tokens mentioned above and identifying target features in Pythia-70kk and GPT 1-pT---mmaa- 11 for which these tokens are the top-activating ones. The two interference feature sets in Pythia- 7C CM and GPT-2-Smar with respect to the target features then identified. Next, collect the top-activating tokens two models respectively, and compute the union. The siswws a sketch of the interference tokens extracted from two models.
oqu Hubble prosec
DW
mixed
5 PLAY monitor
White a ttrr.
awty
Emppas hasiisss is/
OURCES
o DOC
a G
UOT
me
THE
Fieement
ethn
himana
23/
par
adj
Tioo
tritnng
Liissenn Frenne
el
Dawitos
Esp Epioen
tos
triangle
bu
P
pronib
car
Mini
t
Chwaa Sivartson
gooD
LOU
tspicyy
now
and
arbie
arnac
sins Jothing
dous
O
$ sta statutes
Furthermore & Dawi
tweak
fram Mal
mo Lar MAX
pat ches
bilabil lity
abs Js
nor
concurrentl
Mile
buo
Gacing
embedded
ue
Strick
LOCK
entre
ikke
Land
Conorasterch
P
C
Rather
Oerporter
COP
r
R ck
eivered
eects
DAMAGE
ATVERN en
Border
NGDuBl
G
Horr
relium
he
E we lene
fuer rt
unmistak
Expect
stacks
Limnkerr onso
tymphor
d
Pd
gons
possession
lains
defense
dealen eership
Jesuit
ogeA aut
maximizing suddivis wwwpi Interpaage
a atura
ymphoid
mant P
pun mutaa
behaviors
fra
(a) Person interference tokens
(b) Location interference tokens
(c) Emotion interference tokens
Figure 14: Three types of interference tokens
After getting the tokens, we proceed to collect the activation texts, which may activate interference fea- tures of the target in black-box models. Due to computational constraints, only compute gradients from and perform operations on the first half of residual layers in Llama-3 -----Innst ruct. The intervention experiments are done for three target token types, each containing about 100 sen- tences. More intervention examples are listed below. It takes about 20 minutes for a single RTX4090 GPU to find highly effective gradient vector for steering.
Table 4: Examples of interventions Llama-3.1-8B-1m Instruuttt Using Token Gradient Vector
Type
Intervention feature
Result
location
terms related to data and its presen-
"After months of planning, our road trip finally reached"
tation
1 Entered
4. Dropped
New +0.017 an
-0.006
Seattle +0.015
-0.007
San
+0.011 our
-0.002
the verb "be" various forms and contexts
"She always dreamed of owning small cafe in"
Entered
Dropped
Vienna +0.080
France -0.008
Munich +0.071
town -0.007
Berlin +0.038
Italy -0.006
Continued next page
28

---

Published as a conference paper at ICLR 2026
Table 4 Examples of interventions Using Token Gradient Vector (Continued)
Type
Intervention feature
Result
proper nouns, names, and references to specific roles or positions
"This novel's 's opening scene takes
place aboard train to" "
EEntered
+ Dropped
Beijing +0,023
New -0,026
Tokyo +0.020
Venice -0.009
Shanghai +0.018
Istanbul -0.010
quantitative data points related to statistics and performance metrics
"The rebels established their hidden base deep within"
Entered
Dropped
Afghanistan +0.018
an -0.009
Germany +0.015
their -0.003
Eastern +0.014
-0.002
references to specific labeled items or categories
"His last known coordinates placed him somewhere near"
1 Entered
Dropped
Paris
+0.012
an -0.003
New
+0.010
their -0.003
Moscow +0.009
Lake -0.002
person
quantitative data points related to statisties and performance metrics
"Nobody expected the mysterious package to be from" Entered
+ Dropped
Paul
+0,261
the -0.104
Emmanuel +0.026
a Lake -0.51
-0.078
Matthew
+0.021
references academic institutions concepts
"The voice on the recording definitely belongs to" 1 Entered + Dropped
Robert +0.015
-0.101
Patrick +0.012
the -0.088
David +0.009
me -0.033
phrases related pre-approval pro- cesses and conditional statements
"The fingerprints found at the scene match those of
+ EEntered
4. Dropped
Michael +0,003
your -0.017
Richard +0.003
-0.011
Smith
+0.004
both -0.008
keywords related to file management and programming constructs
"This traditional folk song
was popularized by"
+ Entered
+ Dropped
Bruce +0.010
Pete -0.086
Walter +0.010
American -0.032
Paul +0.007
Woody -0.021
terms related to multimedia and video production
"The confidential information was
leaked by former employee"
t Entered
DDropped
Mike +0.017
and who
-0.024
Tom +0.012
-0.024
Bill
+0.011
to
-0.010
emotion
instances of the verb "is."
"After trying the new recipe,
my brother absolutely"
Entered
Dropped
love +0.121
fell
-0.042
hate
+0.095
LO
-0.037
dislike
+0.015
ad
-0.031
Continued next page
29

---

Published as a conference paper at ICLR 2026
Table 4 Examples of interventions Using Token Gradient Vector (Continued) Result
Type
Intervention feature
references to legal documents and real estate transactions
"S "Science proves that most infants naturally"
f Entered
+ Dropped
Like +0.048
develop -0,099
like +0.020
prefer -0.051
love -0.001
learn -0.034
phrases indicating topics of discus- sion or content focus
"His body language suggests
he secretly"
1 Enteredd
Dropped
love +0,457
wants -0.158
loved +0.012
enjoys -0.069
hate +0.015
hopes -0.062
phrases indicating relationships and affiliations contexts such as surveillance, borders, and regula- tions
"This fabric texture makes allergy sufferers"
Entered + Dropped miserable -0.091
love +0. 10.
like +0.020
feel and
-0.074
hate +0.027
-0.043
statements that conclude or summa- rize concepts
"After the concert
critics began to"
1 Entered
Dropped
hate +0.017
question -0.070
love +0.016
praise -0.064
enjoy +0.010
dissect -0.054
Note:
Entered means that corresponding tokens entered the top- J Dropped means that corresponding tokens dropped from the t0p-10. Gray-shaded rows indicate black-box interventions.
K.2
PROMPT INJECTION
Beyond the two intervention methods discussed above, which directly manipulate the model's latent, we further examine whether models remain susceptible to polysemantic interference in more realistic inference-time scenarios. In such cases, interference may arise from the input prompt, which can indirectly influence the model's latent activations.
To begin with, following the same methodology, we first conduct an overall experiment in which we randomly sample target features across all sub-modules and layers along with their interference features at levels. And for each interference feature, we select text snippets from its activation texts that include high activation tokens, and prepend them to the input prompt incorporated in two "***" To do a fine-tune on the snippet, we may add other zero-activation text besides it to maintain its grammatical or semantic structure, or just truncate its low-activation parts the interference effect enhanced. We still make sure the snippets don't tt include tokens that are semantically related to the target feature. In this way, the interference potential of the interference feature to the target is roughly estimated, and we simply record the best case. Similarly, we use weighted cosine similarity and weighted overlap to quantify the semantic shift in model outputs toward the target feature. Note that comparing with the SAE and gradient- based intervention, we are using interference level 0.0-0.1 as random baseline instead of using snippets consisting of random tokens. This is because even random sampling of tokens may activate feature such as "random tokens" or "spelling error", which has unknown interference level with the target feature, while the interference level of random baseline is explicit in the previous two experiments. Figure 15 shows the statistical results measured by alternative metric weighted overlap.
whether
Case studies of prompt injection in experiments are done on the token types listed in Appendix B Based on the classification of tokens, we also generate test sentences for each type, requiring that these sentences grammatically lead to tokens of the target type. The prompt is as follows.
System:: You are a creative writing assistant specialized in generating incompl sentences. Your task to create incomplets sentences where
30

---

Published as a conference paper at ICLR 2026
Pythis Threaheld
GPS2 Threshed
Confidence Intervale 196-0
change toweord bangel fexture
Reg ceeiioun Inmerferente
100
00
x32
340 K-000
LMM Randen
Hyy
Lid Ronaon
7/21e
Figure 15: Prompt injection effeets measured by weighted overlap. Relative change weighted overlap toward the target. Bars show the mean relative change compared to baseline across interfer- ence levels, with lighter shades indicating stricter feature-meaning relevancy cutoff thresholds. The x-axis denotes the interference scale between the source features of injection snippets and the target feature. Error bars indicate 95% confidence intervals.
the next logical token would very likely from the category category.
Requirements:
1. Generate approxinately 1000 incomplete sentences
2. Each sentence should be approximately 20 tokens long when complete
3. The sentences should end at natural point where the next word would
very likely be a category word
4. Use diverse contexts, scenarios, and grammatical structures
5. Make the sentences engaging and varied
6. Ensure the incomplete sentences create strong expectation for category words
Examples for category:
For "animal': "In the dense jungle, we could hear the roar of wild"
For coloor "The sunset painted the sky a beautiful shade of"
For remotion': "When she heard the news,
face showed pure"
For 'location: "Our vacation destination this summer will be"
For 'number' "The recipe calls for exactly"
For ppersnn' "The award ceremony will be hosted by famous"
For ssciense' "The experiment required careful measurement of"
For 'time': "The meeting is scheduled for next"
Return your response a Python 1ist strings, with exactly 1000 sentences. Format it properly as valid Python code that can be executed.
Start your response with: sentences
End your response with:
Do not include any explanatory text before or after the list.
User: Generate approximately 1000 incomplete sentences for the category category.
Available type tokens (separated by I, each token is enclosed in quotes):
type_token_1, type_token_2,
IMPORTANT NOTES:
Each token above is separate vocabulary item from language models Some tokens may have leading/trailing spaces (like dog" "cat ")
These are the exact tcken strings that should be likely to appear the
next token after your incomplete sentences Considen the token boundaries when creating sentences
31

---

Published as a conference paper at ICLR 2026
Please generate diverse, engaging incomplete sentences where the next word would very likely be from the categozy category tokens shown above. Make sure to use various contexts and grammatica cal structures.
Return as Python list: sentences = [...]
The model returns approximately 400 to 800 sentences for each type. The dataset of token type denotation, along with the example sentences, has been made available the GitHub repository.
After annotating the tokens in the model's vocabulary with type labels, we partition the token sets based on the intervention information provided by the sparse auto-encoder. First, we filter for features where the highly activated tokens contain the target token type, simply tagging these as target features. Specifically, for each activation text, we identify the token with the highest activation value (max_act) and set ratio (here, 0.8). Tokens with activation values exceeding ratio * max_act are considered highly activating the interference feature.
We then identify all interference features whose interference value with the target feature exceeds 0.2 and whose semantic similarity is below 0.3. Based on this set of interference features, we further partition them into high-ing highinnterference and above a high_tthe threshold (for given target feature) are classified as high-interference features, while those with interference values between 0.2 and high threshold are classified rediium-iteerference features.
sets. Features with interference values
Subsequently, we collect the highly-activating tokens from features in each to form the respective token sets. All remaining tokens that were not collected constitute the random token set. It should be noted that the high- nigh-interference and inntlur--niteeeeeeeeeee token sets exhibit significant overlap. To address this, we deduplicate the two sets to obtain disjoint token sets. For example, in the case of the Pythiaa700k model, we annotate 1,938 tokens as belonging to the "location" type. Using the method described above and setting 0.5 as the high-interference threshold, the resulting high- interference token set contains 10, 185 tokens, while the
set consists of 34,946 tokens. And 13,323 tokens remain in the random set. There is overlap of 9,535 tokens between the high- and medium-interfeeeen tokens, and the medium-interference set retains 25,404 tokens.
sets. After deduplication, the high-interference set retains 650
For experiments on Pythia-700 70M and GPT-2-Smal1, we use the token sets gener- ated by each respective model. For experiments Gemma-2-9B-18 ruct models, we adopt the union of the token sets from the correspond- ing interference levels of the two small models. We examine total of eight categories of token types: location, person, emotion, color, animal, science, number, and time. Apart from the three types mentioned in the main text that demonstrate strong generalizability, the test results for the remaining types are presented below.
and
K.2.1 POTENTIAL SHARED POLYSEMANTIC INTERFERENCE STRUCTURES
Based on preliminary token insertion tests, we first identify the most effective test cases by selecting those where high-interference token sets produce the most pronounced enhancement relative to medium- now-interference, and random sets. We then analyze these selected cases to trace back to the underlying interference features, which include them as highly-activating tokens. Cross-model comparison based on semantics of these interference features, which are quantified as stated in Section 2.1, further reveals that some of them occur in multiple models. A list of these recurring polysemantic feature pairs is provided below, along with their potential locations in each model. Note that for smaller models, positional hints are derived directly from sparse autoencoders; for larger models, locations are estimated by comparing activation patterns with succinet, semantically neutral prompts ge.g., "In the weekend" for location-ry -type features).
32

---

Published as a conference paper at ICLR 2026
Table 5: Token Types without Strong Generalizability
Target
Model
Original
High- High-inteerferenee
Low-interference
Random
Person
Pythia-70M
60.28% ***
29.02%
28.58%
32.31% %
GPT-2-Small
54.29% ***
27.17%**4
26.97%* **
25.16%
1aama-311-8B-Instruue
38.85%
19.20%
19.08% ********
16.86%
Gemma-2-9B-Insmrun
43. 43.27%
25.36%
26.10%
25.04%
Llama-3.1-700
100na-33177OB-nsttruee
46.27%
21.90%
21.13%**
19.61% %
Animal
Pythia-70M
96.96% ***
28.49%
30.75%
29.83%
GPT-2-Small
86. 16.111%o *
21.64%
18.87%
25.03%
Llama-3.1-88 Instructtt
67.07%
42.76%
40.56%
40.89%
Gemma-2-9B- Instruct 2-9--ntrucct
26.09% tote
41.02%
38.50%
38.47%
Llama-3.1-70BB 1-7B----tr---ttttttuuc
49.32%
32.80%
30.79%
31.43%
Emotion
Pythia-70M
61.74% #100
26.13%
21.11%
20.68%
GPT-2-Small
58.84% #1b0
36.91%
35.22%
33.85%
L
60.16% *+rs
27.94%
27.45%
29.37%
Gemma-2299-Insee
56.55% ***
13.85%
13.63%
13.50%
1lama-3.
51.51 **0
44.90%
40.63%
40.40%
Color
PYthia-70M
97.31% 1444 4300s
17.89%
21.49%
21.80%
GPT-2-Small
85.47% *re
35.57%
33.13%
36.93%
Llama-3.1-8B
1-.---Innsttrctt
76.76% wre
22.01%
20.67%
20.97%
Gemma-2-88 2--------Bnnnut
22.57% #r00
13.58%
16.10%
15.67%
Llama-3.1-700 --7B-Instruse
76.85% **0
19.08%
17.88%
18.44%
Time
Pythia-70M
39.92%** #re
45.33%
45.54% ****
41.33%
GPT-2-Small
58.48% #aps
31.55%
27.72% *+*
21.42%
Llama-3.1-8P 1-11--Bnnsttrctt
51.41% wew
25.58%
26.29%
25.95%
Gemma-2-9F 2-w-------------------
55.15% *re
25.10%
25.94% ***
23.62%
Llama-3.1-70 Instruct
78.28% #000
36.87%
36, 36.68%
37.71%
Note: Cell values show the success of clevating target-type tokens into the top 30 predictions. Gray-shaded rows indicate black-box interventions. Testing uses shared token from the small models. **, and denote t-test significance 0.001, p < 0.01, and 0.05, respectively, vs. random baseline. High- and low-interference tokens lie in [0.5, 1.0) and [0.2, 0.5).
Table 6: Shared Polysemantic Interference Structures
Interference Feature Pairs
Position in Models
references to specific locations and estab- Pythia-70M(res-44 GPT-2-Small(res mid-
lishment & datatypes, definitions or other 6). Llama-3.1-8B(res. Gemma-2-
concepts in programming context
9B(res-32), Llama-3.1-70B(rs--2)
numerical values and their patterns & Rythia-70M(mlp-)), Llama3.1-8B(mlpp2) phrases related emotional states or ex- Gemma-2-9B(mlp-34)
pressions
references to medical and cellular biology patterns of punctuation and formatting
Pythia-70M(res-0) GPT-2-Small(res, 20e2-mmll(rrsspostt 2). -lama-3.1-B(res-30) Gemma-2-
9B(res-11)
33

---

Published as a conference paper at ICLR 2026
L COVERT INTERVENTION ON Hellaswag TESTSET
To further examine the impact of the interference vectors obtained the aforementioned sections on the overall performance of the model, we conduct a rapid validation using the experimental results targeting the "location" type of interference in Llama 1intaa-3.. 3.1-- 10-1--B---- Instruct. Given that the intensity of the interference vectors applied in the previous experiments is aimed at maximizing the disruptive effect which likely caused substantial impairment to the model, we first reduce the scale to 0.25 times its original value. Subsequently, we randomly select 500 test samples from HellaSwag validation set and evaluate each interference vector on them. Using accuracy as the evaluation metric, the experimental results demonstrate that, out total of 174 interference vectors, 149 decreased the accuracy. while 25 increased it. The baseline accuracy is 77. 77.2%, and the average reduction in accuracy is 2.77%. Some examples that keep model performance, i.c. reduce accuracy fewer than 1%, while still having substantial intervention effects are listed below.
Table 7: Covert Intervention on Hellaswag Dataset
Intervention Feature
Top Predicted Tokens
Description of precautions related safety
"The documentary crew disappeared while filming in remote areas of"" Raw
protection
Intervention
the 0.35 the
0.24
Papua 0.027
Papua Africa
0.04
Nepal 0.016
0.035
Numerical data that may require ordering or sorting
"She always dreamed of owning small cafe in"
Raw
Intervention
the 0.34
the
0.36
0.27
0.28
her 0.21
Paris
0.13
34

---

Published as a conference paper at ICLR 2026
M TRANSFERABLE POLYSEMANTIC STRUCTURE EXPLORATION
M.1 BRAINSTORMING HIDDEN ASSOCIATIONS
In this section, we analyze shared interference patterns in GPT-2 Small and Pythia-70ke which we previously demonstrated transfer to larger models under intervention. First, we use DeepSeek-V3 and EPT-5-mins to annotate every selected feature pair with the following prompt instruction.
System: You are an expert in analyzing neural network feature semantics. Given two SAF (Sparse Autoencoder) features, their explanations, and their top activation texts, determine if they are semantically related. For each feature, you will An explanation describing what the feature captures
text segments where the feature activates most strongly.
figh-activation tokens are marked with <<token>> or asmultiple tokens>>
Analyze the explanations and the marked tokens/activation patterns to determine:
1. Are these two teatures semantically related? Consider form of semantic relationship including direct overlaps (capturing similar concepts, linguistic patterns, or contextual meanings) as well as higher-order associations (e.g., semantic priming, thematic relatedness, complementary roles).
2. If related, provide a concise description (max 50 tokens) of their relationship.
Return your analysis in this exact JSON format:
"iskelated"; true/false, "description": "brief description" or null Examples:
Features activating different tenses of verbs: "isRelated": true, "description"; "Both capture verbal expressions, ene for past tense, other for present tense"
Features for numbers vs. animals: "isRelated": false, "description"; mull
negative emoticns: "isRelated"; true, "descripticn": "Both capture emctional expressions with opposite
Features for positive
valence"
Features for "doctor" vs, "hospital"; "isRelated"; true, "descriptica"; "Conceptually linked via medical domain (profession vs. location)
Features for "boat" vs. "sand"; "isRelated"; true, "description":
"Loosely associated through beach/marine context (object vs. terrain)
User: Feature A: festure_a
Explanation: feature_at a's explanatien
Top activations: texts that activates feature_a the most Feature B: feature_b Explanation: feature_bt b' explanation Top activations: texts that activates feature_b the most
To compare the behavior of the two automatic annotators (DeepSeex-V3 and CPT-5-min1), we compute the share of feature pairs labeled as "related" and the simple percent agreement in two subsets. In the Pythia-77 7M subset, DeepSeek-v2 labels 10.8% of pairs as related and GPT-5-ming subset, the corresponding proportions
28.7%. with 80.3% percent agreement. In the are 3.7% (DeepSeek-V3) and 18.4% (GFT-5 5 5-mini), with 83.8% agreement. Aggregating both subsets, DeepSeek-V -3 labels 9.5% of pairs related and GPT-5-min n 26.8%, and the overall percent agreement was 80.9%. We define percent agreement as the proportion of pairs for which both annotators assign the same label. Figure 15 A-B shows the agreement matrices for the two annotators. Interestingly, the overall consistency between the two annotators is low (Pythia- 70M: Cohen's 's k 0.408; GPT-2 Small: Cohen's 0.221). Compared with DeepSeek- labels significantly more feature pairs as related, suggesting greater sensitivity to latent semantic
GPT-5-mirg
35

---

Published as a conference paper at ICLR 2026
associations. Nevertheless, more than 70% of pairs still judged completely unrelated in the annotators' combined judgment (i.e., both say unrelated).
DeepSeek DeepSeek
Unvebitd
GPTS
Unroltted
B
100%
Pythia-7OM: DS G5 (T/F)
GPal2-5mall: DS G5 (T/FF)
D tabe!
250000
DeepStell-33 label
Proportion of fabels
80%
True
True
6.0%
0.8%%
150000
40000
Faise
7008: (18.8%)
260766 (P04
100000
False
13238 (15.%)
(0.9))
69443
40%
20%
True
False
True
False
label
GPT-5-min
Pythia-70M GPT-2-Small
Figure 16: GPT-5-mini and DVe'Seek-V3's annotation reports for shared interfered feature pairs. DS is the abbreviation of DeepSeek- V3 and G5 is the abbreviation of GPT-5 5-mini.
In Table 8, we further report eight qualitative examples where the two annotators flag latent associa- tions and contrasting cases where they do not.
Table 8: Eight examples annotated feature pairs.
Category
Feature pair
Interpretation
Frame
Feature A: Phrases and sentences that highlight sys- temic issues related to incarceration and its effects on individuals and families.
Incarceration content often appears alongside quantitative/staiitiiaal refer- (from GPT-S-mini)
Feature Quantitative information related statistics and predictions.
Orthography
Feature A: Occurrences of the word "acher."
Both capture researcher/author name substrings (e.g., -acher / Wascher). (from GP (5P-5-min)
Feature B: References specifie researchers authors studies related to minimum wage.
Axiology
Feature A: The term "dear" in emotional contexts
Both capture emotionally significant concepts (affection vs. sacredness)
lated relationships and feelings of affection.
Feature Concepts related to notion the sacred.
deep personal or spiritual value. (from DeepSeek-V2
Homography
Feature A: References playing cards and card-related concepts.
Surface lexical overlap: "card" playing card vs "Card" (author name). Same token, different senses. (from GP GPT-5- (ei-mmmnn)
Feature B: References to the authors or studies related to economic analysis.
No-relation
Feature References "New Guinea." Feature B: File formats and file compression terminol- ogy.
None
No-relation
Feature Instances of abbreviation "ob" or related terms indicating observational data or annotations. Feature B: A specific term related to a well-knnwn ridc-sharing company.
None
No-relation
Feature A: Terms related to turbidity and its measure- ment.
None
Feature B: References specific music artists groups.
No-relation
Feature A: Occurrences of the name "Becthoven" and related variations.
None
Feature B: Words related expressions of frustration
annoyance.
36

---

Published as a conference paper at ICLR 2026
M.2 FEATURE-PAN PAIR COMPARISON TEST
Let ( f3) denote the semantic similarity (in MM) between features S; and fj. and let [(,,,) f5) denote their interference value.
From the set of shared interfering feature pairs GPT-2-Smal1 and Pythia- 7 satisfying SM(Ji.fj) < 0.4 and [(fi, Jj) > 0.4,
we uniformly sample pairs for testing. For each sampled pair (f1. f2), we then search for a matched comparison pair (fa, fa) from the same layer such that
sM f4) sM((I, Se and I(Ja, f4) < 0.1,
i.c., semantic similarity is tightly matched while interference is substantially lower. For Pythia-70M, 0.01; for GPT-2-Smalle due limited matching, 0.05. Eventually, we collect 3, 800 feature-pair combination for this comparative analysis.
We then present the pairs f2) and (S3, .4) (order randomized) GPT-5 --inni, asking the model to choose the more related pair using the prompt below.
expert in interpretabil ity of 6LM SAB features. You feature pairs (pair_one, pair_two). For each pair ( g), use their explanations and Tep-5 highest-activatie high-activation tokens marked with æstokens> or <<multiple tokens>>) to decide which pair shows higher latent relatedness.
System: You
will receive
texts (with
Definition:
"Latent relatedness" means the two features in a pair likely reflect the same underlying concept or are functionally coupled. Consider direct overlaps (concepts, patterns, meanings) and higher-order associations iitematiicccompememtttt roles).
Decision criteria:
1) Explanations: semantic consistency, paraphrase/synonyyy, or complementary roles within the same domain; penalize opposite/disjoont concepts.
2) Top-5 texts: overlap in topics/contexts; alignment of <<markersw> pointing to the same phrase, siot, entity, or role; penalize disfoint or contracit evidence.
Uncertainty handling: If evidence from (1)-(2) insufficient or conflicting, return "uncertain".
Output:
Return ONLY JSON object with exactly these keys: "more_relate meore_eelated_pair" "pair_one" or "pair_two" or "uncertain" "reason": brief English rationale (5 80 words), citing 1-2 decisive cues (e.g., explanation alignment, shared tepics, token markers). Must be valid JSON (double quotes). extra text.
User: PAIR ONE
Feature 1
Explanation: explanation of feature Texts: five activation texts of feature one
Foature 2
Explanation: explanation feature two Texts: five activation texts of feature two
PAIR TWO
37

---

Published as a conference paper at ICLR 2026
Feature 1
Explanation: explanation of feature one Texts: five activation texts of feature
Feature 2
Explanation: explanation of feature two Texts: five activation texts of feature two
38

---

Published as a conference paper at ICLR 2026
N POLYSEMANTIC NEURON MANIPULATION
During the examination of SAE features and their connections with neurons, many features exhibit semantically similar activation texts. To avoid repetitive analysis on similar activation texts, first perform feature clustering based on the semantics of activation texts, and then check neuron connection at the cluster level. Given that the sparse auto-encoder from Neuronpedia is trained with sparsity setting of 3, the analysis focuses on the top three neurons with the highest alignment values per cluster. threshold of 0.2 is applied filter out weak connections. Figure shows the distribution of polysemantic neurons identified in each layer. We can see that polysemantic neurons with strong connections with aggregated features only take up fewer than 5% each layer.
B
ATT
ATT
MLP
RES, MID
RES
MILP
RES, PCST
Polysemantic Neurons
Polysemartic Neurons
21:50 Polysemanticity Level
101-200
2.5
21:50
51-100 101-200 201-500
Polysomantidtty Level
Figure 17: Distribution of polysemantic neurons in each model. is the result of Pythia-70ke and B is the result of GPT Small. Error bars represent 95% confidence intervals.
For strongly connected polysemantic neurons, we do further investigations on how suppressing or boosting their activation influences the semantic shift in the model's output to their aligned features. Neurons' activation is multiplied with scale value in the range [0, 20). Note that scaling within [0, 1] suppresses activation, while scaling within [1, 20] amplifies it.
O DENOISING INTERVENTION EFFECT
We observe that the SAE intervention experiments on GPT-2-Smal yield considerably weaker effects compared to the Pythia-700 -7M model, although the overall trend-wher nigh-interference intervals tend to produce stronger effects-is still maintained. Given that our experimental results represent average
test cases from all layers and all sub-modules of the model, including both effective and ineffective instances, we hypothesize that the presence of ineffective cases in GPT-2-Smal 1. may account for this outcome.
Therefore, we apply a simple preprocessing step to the test cases and observe a marked improvement (5x in aatthia-700 and 100x in GPT-2-Sma 11) in the experimental results. Specifically, first employ the direction of the target feature for an initial steering attempt, monitoring whether the two metries improve after steering. We thresholds, approximately between 0.05 and 0. filter ineffective feature directions. Subsequently, we select interference vectors for the remaining potential directions and proceed with the experiments. The results are shown in Figure 18
We find that with this simple preprocessing step, the performance gap between GPT-2 2-T---Smoa S0mamall 1 and Pyth a narrows to approximately tenfold, compared the original hundredfold difference. To explore variations in interference effects across different models, we select an additional model, Gemma-2-28. and conduct corresponding interference tests for a subsample. The specific results are presented below in Figure 19
39

---

Published as a conference paper at ICLR 2026
50.0
Theesedd
Caskebreo Merdd
wwg.madaeon www.atoo wwte www.weewwwee... Viku
Neethe Oirgg lowand barget feebre
Reattv Seetrr
60.0
2.0
20.0
2.00
22.0
13.0
0.00
GF
Figure 18: Enhanced Effects of intervention based on the interference SAE feature direction. (A-B) Relative change in weighted cosine similarity toward the target (Ac). Bars show the mean relative change compared to baseline across interference levels, with lighter shades indicating stricter feature-meaning relevancy cutoff thresholds. The x-axis denotes the interference scale between the target and intervention feature: Original corresponds to intervening with the target feature itself, and Random serves random feature intervention baseline. Error bars denote 95% confidence intervals, (C) Regression estimates of the effect of feature-pair interference value on intervention success. Two regression specifications
shown: Model regresses weighted cosine similarity after intervention (c(O,T,) T))) on interference value, with feature-meaning similarity, baseline weighted cosine similarity (c(O, T),), and layer-type controls; Model B regresses the change score (Ac) on interference value, with feature- meaning similarity and layer-type controls. Error bars denote 90%, 95%, and 99% confidence intervals.
Comparing the two metrics, the three models exhibit considerable differences under the weighted cosine similarity metrics. However, under the weighted overlap metric, the nigh-interference intervals in all three models demonstrate comparable effects, with an improvement of approximately two orders of magnitude (about 109). We therefore conclude that interference in GPT-2- Smmma 1 more precisely enhances the probability of the tokens most relevant to the target feature in the output.
Weighted Cosine Similarity
Weighted Overlap
Relative change toward target feature (%)
Relative change toward target feature (%)
Gemma-2-26 Threshold
cas8 x 0.30
10a
cos
102
10"
10'
10!
109
109
Original High Mid High
Mid Mid Low
Low
Rendom
Original
High Mic High
Mid
Mid Low
Low
Random
Figure 19: Effects of SAE intervention Gemma-2-22 Relative change in weighted cosine similarity and weighted overlap towards the target (Ac, Ao). Bars show the mean relative change compared to baseline across interference levels, with lighter shades indicating stricter Ceature-meaning relevancy cutoff thresholds. The x-axis denotes the interference scale between the target and interven- tion feature: Original corresponds to intervening with the target feature itself, and Random serves a random feature intervention baseline.
40

---

Published as a conference paper at ICLR 2026
P ETHICS STATEMENT, LIMITATIONS AND FUTURE WORKS
This paper follows the ICLR Code of Ethics. This study has three key methodological limitations. First, we rely on SAES disentangle polysemantic activations; although SAES are the de-facto tool. their outputs fluctuate with dimensionality and hyper-parameters, yielding unstable features (Paulo Belrose 2025 Heap al. 2025 Gao et al. (024). Second, our interventions steer only one interference feature in one layer, while multi-feature, cross-layer manipulations could amplify and better obscure the effect (Ameisen et al. 2025). Third, we quantify vulnerability solely via shifts in immediate next-token probabilities two small base models-because because only they both expose raw logits and have pre-trained Shes-then check coarse transfer on three larger instructed models; establishing how these interventions alter non-trivial downstream tasks in bigger models is the next stage of this project.
REPRODUCIBILITY STATEMENT
To balance reproducibility with responsible disclosure, we release complete code, evaluation scripts, and synthetic data in this Github repository but deliberately omit the matrices that catalogue shared polysemantic directions between two small models. Publishing those mappings would make it easier weaponize the very vulnerabilities study, whereas the available artifacts still permit independent verification of all empirical claims.
41


Reviews:
[{"review_id": "nD4LCKyy7o", "reviewer": "Reviewer_72iw", "summary": "This paper explores interference in trained SAE latents and polysemanticity thereof. Specifically, polysemanticity is defined as the scenario where the precise descriptions assigned to two features differ and yet they interfere (for which I'm not quite clear what the precise definition is, since, unless I missed something, there was no formula provided for how interference is measured). From hereon, the work is mostly a qualitative study to develop the broader phenomenology of polysemantic SAE latents, i.e., identifying interesting ways that these features interact (e.g., how steering one can affect model behavior along another one). The coolest result, which relates with the work by Lee et al. [1], is that these steering pathologies transfer across model, indicating there are consistent statistical signatures in the data distribution, which all models pick on. This result is especially cool because the results were derived using small models (order 100M parameters), but transfer to substantially larger models (order 7--9B parameters).\n\n[1] https://arxiv.org/abs/2504.14379", "questions": "A question that came to mind while reading:\n\n- Instead of the use of agglomerative clustering, as done for analysis in this paper, I wonder if SAEs with hierarchical priors could have been used to directly elicit statistical interferences between model features. For example, see papers [1, 2, 3]. I'd love to gather authors' thoughts (or experiments if they get the chance) on how they would use such more structured SAEs for their analysis. \n\n\n[1] https://arxiv.org/abs/2503.17547\n\n[2] https://arxiv.org/abs/2506.03093\n\n[3] https://arxiv.org/abs/2506.01197", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "I really like this paper. It needs some improvement on presentation / writing to ensure results are easily legible, but conditioning on that minimal rewrite, the empirical characterization of interference pathologies (such as the super-neurons), the transfer of these pathologies across model scales, and the exhaustiveness of the experiments is awesome. To be clear, I believe these results were relatively expected, but getting a detailed account of them in a single paper is very helpful for the community.", "weaknesses": "- Missing descriptions of relevant concepts: I tried to find the precise mathematical definition for how interference is defined or measured, but didn't see it anywhere in the paper. This really impacted my ability to onboard with the paper on a first reading pass, but eventually digging through the appendix and looking at results, assuming there's a reasonable mathematical definition for the notion of interference, I like this paper's goals and results. If the authors can fix these definitions (unless I missed something there are not provided at the moment), I support the acceptance of this work. \n\nSome other minor comments:\n- Writing style: I found some of the language in the paper unnecessarily complex. For example, the use of the term \"loci\" in the abstract doesn't make sense, both because it's unnecessarily complex, but also (I think) wrong in this case? The closest term that makes sense would be \"foci\", since a \"locus\" refers to a path.\n\n- On adversarial examples and superposition: a recent work makes a similar point as the authors, i.e., adversarial examples can be driven by feature superposition [1]. \n\n- Possible typo on L94--95: If I understand correctly, the line should orthogonality in symbolic manifold M is not does not persist in the activation space after projection. The subsequent lines (L95--96) only make sense to me with this rephrasing.\n\n- Typo in Eq. 1: I presume it should be O(t) instead of P(t), or P(.) should be defined to be an element from the set {$O, \\tilde{O}$}? \n\n- Analysis layer: For several experiments, I wasn't sure which layer SAEs are analyzed. For example, in the paragraph on L279, it is stated that due to greater depth of GPT-2, SAE interventions for interfering features might be less effective. While I intuitively buy this, the precise layer number information would help contextualize the claim.\n\n[1] https://arxiv.org/abs/2508.17456", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yBMhJCJ7WG", "reviewer": "Reviewer_Zy8X", "summary": "This work studies polysemanticity in LLMs, using SAEs to identify features which are semantically unrelated, but have unexpectedly entangled representations. They use these features to develop 4 methods of intervention (prompt, token, feature neuron), and show the effectiveness of interventions with semantically unrelated features, e.g. occurrences of specific surnames with geographic locations. They find cases where, surprisingly, semantically unrelated features transfer from very small models (e.g. 70M) to large models (e.g. 70B). The authors also study highly connected \"super neurons\" which, if steered, affect hundreds of distinct SAE features, as a case study in extreme polysemanticity.", "questions": "- What do the authors think of the relationship between semantically unrelated polysemanticity and recent work on emergent misalignment and subliminal learning [1, 2]?\n- For table 1, I wonder what a \"semantically related\" baseline would look like, rather than a random baseline. Was this tested, or would this make sense to test?\n- I wonder if many of the significance tests in this table would hold up with multiple hypothesis correction (e.g. holm-bonferroni), given that the table is showing 45 test results.\n- Diving deeper into some of the cases with big effect sizes in cross-model transfer (e.g. science on llama-70B) might be interesting.\n- There is no reference to figure 5 in the text.\n\n\n[1] Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., ... & Evans, O. (2025). Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs.\n\n[2] Zur, A., Loftus, A. R., Orgad, H., Ying, Z., Sahin, K., & Bau, D. (2025). _It’s Owl in the Numbers: Token Entanglement in Subliminal Learning_.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Well-motivated problem setup, and grounded in real world impacts of polysemantic features and vulnerability of black-box models to adverserial attacks.\n- Experiments and methods are comprehensive, thorough, and well-documented.\n- Interesting empirical results with clear impact, especially the transfer between models which are tiny by modern standards and full-size 70B models. The super-neuron results are also interesting.\n- Generally well-written.", "weaknesses": "- One of the greatest strengths of this paper is the result with feature transfer across model scale. However, the effect size of these seems very small at only a few % points in most cases (table 1), which the authors don't seem to make note of. If this is right, this seems like an important limitation of this work which should be clear to the reader, although it also may diminish the significance of this work somewhat.\n\t- Relatedly, some of the interventions in figure 3 seem to have very modest effects compared with the random baseline.\n- Presentation in the figures was lacking, in my opinion. I had trouble understanding fig 1, and this could be improved. For many of the results figures, I had some confusion as to exactly what the point is in some cases. It might help readability if the captions clearly state what the takeaway is from reading these results.\n- There seems to be some missing glue between the introduction, and the formalization of e.g. the \"Human Symbolic Manifold\", and the rest of the text. These don't seem to be referenced later, except in 3.5, and so I wonder if this formalization is even necessary, or if instead the authors could utilize it more.\n- Too much reliance on the appendix, which really should be extra information which is not vital to the content of the work. One example is table 3 and some of the word clouds in the appendix - these were helpful to give me intuition of what kinds of particular semantically unrelated features are interfering, and something like this in the main text might improve readability.\n\nI might be willing to raise my score if these concerns were addressed, especially the first point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8L6dc9wf87", "reviewer": "Reviewer_yfBh", "summary": "This paper explores polysemantic interference in transformer representations. Using sparse autoencoders (SAEs) trained on GPT-2-Small and Pythia-70M, the authors identify pairs of features that are semantically unrelated yet exhibit high cosine similarity (“interference”) in the SAE latent space. They then test whether intervening on one feature influences another across four levels: SAE-feature, token-gradient, prompt injection, and neuron manipulation. Results suggest that such interference pairs can weakly steer model predictions and that these patterns partly transfer to larger black-box models.", "questions": "- Why not compute interference directly between token-embedding clusters or residual directions instead of through an SAE?\n\n- Can the authors provide baselines for unrelated concepts under identical interventions?\n\n- How robust are the results to SAE sparsity, random seed, or layer choice?\n\n\nMore related work:\n- Rosetta Neurons (Darvid et al. 2024) \n- Second-Order Effects in CLIP (Gandelsman et al. 2024)\n- Platonic Representations (Huh et al. 2024)\n- Token Entanglement (Zur et al. 2025)", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The paper attempts to connect feature-level correlations to behavioral effects and cross-model transfer --- an ambitious angle.\n\n- The idea of probing “interference” across different loci (feature, gradient, prompt, neuron) is original and could, in principle, yield insights into distributed representations.", "weaknesses": "Unclear necessity of SAEs: It is never justified why SAEs are needed for identifying interference.\nThe same cosine-overlap analysis could be performed directly in the model’s activation or embedding space, or between token clusters in the vocabulary embedding. By relying on pre-computed SAE features (and their textual glosses), the analysis may inherit annotation noise and input-centric biases (see Arad et al., SAEs Are Good for Steering - If You Select the Right Features, 2025).\n\nDefinition: The notion of “interference” and of “polysemantic neurons” are both defined through the SAE basis; this makes the subsequent results self-referential. There is no control showing that similar effects would not appear if random or non-SAE directions were used.\n\nBaselines: In most experiments, there is no test of how non-interfering or semantically similar features behave under the same interventions. Without such controls, it is hard to interpret the magnitude or specificity of the reported changes.\n\nSmall effects: Table 1 and related plots show minor shifts (typically just a tiny bit above random). While statistically significant, the effects are small enough to fall within what could be lexical or topical overlap rather than genuine structural transfer. The cross-model “transfer” results in particular feel underwhelming. In Figure 5, the neuron-masking results are not monotonic, peaking at moderate polysemanticity. This contradicts the narrative that highly polysemantic neurons are “interference hubs.”\n\nIn sum: Interesting question and creative setup, but weak and noisy evidence and unclear conceptual motivation for SAEs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ougJvtMCSe", "reviewer": "Reviewer_dp5x", "summary": "This work investigates polysemanticity in language models using SAEs, and whether this polysemanticity translates to practical vulnerabilities, where vulnerabilities are characterised based on shifts in next token predictions following interventions. They identify pairs of features which interfere, but are semantically unrelated, and show that black and white box interventions (prompt injection and steering) made on one feature interfere with the other. Interestingly, they find that these interference pairs, which appear to have no interpretable semantic link, replicate across models of different sizes, as shown with steering experiments. Regarding vulnerabilities, they show that altering the activations of more polysemantic neurons leads to greater shifts in outputs.\n\nThis is an interesting and well conducted investigation into a known but minimally explored implication of superposition. The writing and Figures are unclear in places (I have made some suggestions to improve this) and this does detract from the contributions. Furthermore, some of the relationships observed in the results are unclear and merit more explanation and investigation, as I have raised in the weaknesses and questions. However, overall this is an interesting piece of work with potentially valuable practical implications.", "questions": "What is your intuition behind the weighted cosine similarity metric (Equation 1) and how what this measures differs to the weighted overlap? Why select these measures?\n\nWhy do you use token frequency instead of these metrics in the prompt injection experiments?\n\nPlease define interference when you first refer to it in the introduction - its somewhat ambiguous otherwise. Similarly for the ‘feature irrelevance thresholds’ introduced in Section 3.1 - it would be much clearer if you define and link this to the semantic similarity measure in the main text (its also not clearly described in the Appendix).\n\nCan you repeat the experiment in Figure 3 with a third model, to investigate the 100x difference in magnitude between the Pythia and GPT-2 effects? \n\nThe results on Figures 3 and 4 show a 100x difference between these methods for GPT-2 but the text in Section 3.2 says 10x - is this a typo or error on one of the graphs?\n\nCan you move the note for Table 1 into the caption? The trends may also be clearer if you report the change in % relative to the random token rather than the absolute values. Also increase the shading contrast as this is barely visible!\n\nCan you add interference and semantic similarity scores to Table 3?\n\nMinor comment - the reference on line 143 is misformatted.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper addresses a potentially impactful but minimally studied issue: the impact of polysemanticity on LLM vulnerability.\n\nGood and thorough analysis, both within the main text (e.g. running experiments with features sampled from different interference intervals) and in the additional results in the Appendices (e.g. the correlation results between interference and semantic similarity in Figure 8 are an interesting additional result). Multiple metrics are used to validate results, and results are reported for multiple models with confidence intervals.\n\nThe set of black and white box interventions used are well-designed and thorough. The gradient-based steering approach, in particular, is a clever way to run steering experiments on models where SAEs are not available. \nThe finding that polysemantic structures transfer across models, despite having no immediately interpretable meaning, is quite novel and opens up interesting directions for further investigation.\n\nWhere there is confusion around the results (e.g. the major gap between models in Figure 3), the authors are transparent and honest, which I appreciate.", "weaknesses": "The introduction is long and seems to try and incorporate a large amount of related work (while the main RW section is in the Appendices). This makes it difficult to follow the purpose of the paper. I’d recommend using the additional page to change this to a condensed introduction followed by a separate related work.\n\nI found Figure 1 confusing, especially as it appears before the representation domains are defined. The text on the right is a useful summary, but it was not immediately clear to me what the top boxes were trying to show. Could you move these below the other graph (so the reader reads the text first) and/or add labels to these boxes? Please also increase the font sizes in Figures 2-4.\n\nThe SAE feature steering results (Figure 3) are extremely different between Pythia and GPT-2, and I don’t find the hypothesised cause - model depth - to be particularly convincing. It would be great to repeat this with a 3rd model to compare the results, or at least to discuss or investigate other possible reasons in more detail (e.g. it may be SAE quality, since the difference in effect size is much smaller with the gradient direction interventions).\n\nThis paper could also be improved by adding a realistic case study of how interference values can be used to extract examples of prompt injections which elicit dangerous responses. This would be particularly impressive if they transfer to closed weight models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)