# (tentatively) Found 600+ Monosemantic Features in a Small LM Using Sparse Autoencoders

Using a sparse autoencoder, I present evidence that the resulting decoder (aka "dictionary") learned 600+ features for Pythia-70M layer_2's mid-MLP (after the GeLU), although I expect around 8k-16k features to be learnable.

Dictionary Learning: Short Explanation
======================================

Good explanation [here](https://aizi.substack.com/p/51946f63-afdc-4626-b762-1828ccd55f1e) & original [here](https://www.lesswrong.com/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition), but in short: a good dictionary means that you could give me any input & I can reconstruct it using a linear combination of dictionary elements. For example, signals can be reconstructed as a linear combination of frequencies:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a333354e3fea9d087e9a350ecc8cbfab877edaac83cf41c1.png)

In the same way, the neuron activations in a large language model (LLM) can be reconstructed as a linear combination of features. e.g. 

neuron activations = 4*(\[duplicate token\] feature) + 7*(bigram " check please" feature).

Big Picture: If we learn all the atomic features that make up all model behavior, then we can pick & choose the features we want (e.g. honesty)

To look at the autoencoder:

![](https://lh6.googleusercontent.com/1kNFe6gvrMzaWhg-WZr8g3z63UJDupbQQQLb-VzIxgL0oVQjK4UNlWWo4j6csyPiFGf9h5I18TUo0wJUFXmlMTm6j6KJBgoYbfGaQfIIS0epfsYvB2Om4lz6jRWONXRpdRGq8X9b2-BbHIYWklYgG0U)

So for every neuron activation (ie a 2048-sized vector), the autoencoder is trained to encode a sparse set of feature activations/magnitudes (sparse as in only a few features "activate" ie have non-zero magnitudes), which are then multiplied by their respective feature vector (ie a row in the decoder/"dictionary") in order to reconstruct the original neuron activations.

As an example, an input is " Let u be f(8). Let w", and the decomposed linear combination of features are:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b9eaa580b7cbd60499f15872416925458e470be3c534f740.png)

Most of the features are zero, but there are two features ("w" & " Let \[x/o/w/etc\]") that activate highly to reconstruct the original signal. This is a real example.

\[" Let u be f(8). Let **w**"\] = 4*(letter "w" feature) + 7*(bigram "Let \[x/w/n/p/etc\]" feature) 

Note: The activation here is only for the last token " w" given the previous context. In general you get 2048 neuron activations for \*every\* token, but I'm just focusing on the last token in this example.

For the post, it's important to understand that:

1.  Features have both a feature vector (ie the 2048-sized vector that is a row in the decoder) & a magnitude (ie a real number calculated on an input-by-input basis). Please ask questions in the comments if this doesn't make sense, especially after reading the rest of the post.
2.  I calculate Max cosine similarity (MCS) between the feature vectors in two separately trained dictionaries. So if $feature_0$ in dictionary 0 is "duplicate tokens", and $feature_{98}$ in dictionary 1 has high cosine similarity, then I expect both to be representing "duplicate tokens" and for this to be a "real" feature \[Intuition: there are many ways to be wrong & only one way to be right\]

Feature Case Study
==================

Top-activating Examples for feature #52
---------------------------------------

I ran through 500k tokens & found the ones that activated each feature. In this case, I chose $feature_{52}$ which had an MCS of ~0.99. We can then look at the datapoints that maximally activate this feature. (To be clear, I am running datapoints through Pythia-70M, grabbing the activations mid-way through at layer 2's MLP after the GeLU, & running that through the autoencoder, grabbing the feature magnitudes ie latent activations).

![](https://lh3.googleusercontent.com/eP3CYJgvqi3XPULz4xjsVwiXFZJEMp8RmRlMb9QobTTzLwC-Egr2_scwCvnHpSqjkYa4M8u-5LJxm2k9kWT_HXpC8rxLJb138yWLXsZGd2v7xUQ-EavbKvfO8G-f1SATkra60Ffkg82zRO4F0w62OdA)

Blue here means this feature is activating highly for that (token,context)-pair. The top line has activation 3.5 for the first " $" & 5 for the second. Note that it doesn't activate for closing $ and is generally sparse.

Ablate Context
--------------

Ablate the context one token at a time & see the effect on the feature magnitudes on last token. Red means the feature activated less when ablating that token. 

![](https://lh3.googleusercontent.com/VysaRBAI3xwbaz-XdtzyCZ1QnEcMgCRsfTHm_c7mOiWfbDp2w0tpEGMMZ6ZWFMvknKCmoXiVM1Y0inicNDc7TV1gxc3Z-fklFGSmjCtO1vH7LbX6hadGKMfJTUqDOblh6AzqVV9p17T-CWxBzYIyV4Y)

To be clear, I am literally removing the token & running it through the model again to see the change in feature activation at the last position. In the top line, removing the last " $" makes the new last token to be " all" which has an activation of 0, so the difference is 0-5 = -5, which is value assigned to the dark red value on " $" (removing " all" before it had no effect, so it's white which means 0-difference)

Proposed Meaning: These detect $ for latex. Notably removing token )$ makes the final “ $” go from 5 to 3.5 activation. Similarly for “where/let/for”. 

For reference: The darkest blue (ie “ =”) makes it go up by 0.08, so there really isn't much of an effect here.

Ablate Feature Direction
------------------------

We can ablate the feature direction by subtracting the original neuron activation by the feature’s (direction*magnitude), and see the effect on the model's output logits on the actual tokens.  To clarify, the dictionary features are supposed to reconstruct the neuron activation using a sparse linear combination of features; I am removing one of those feature directions.

![](https://lh3.googleusercontent.com/3aXYzu-IKJnaX0xT4sVmsbe8hbx5a5Wb_Lh7QthBRHp75-HhUtKWcBqBzbaUK7tsyFb6ICFAfbCFy2vWY9_kVqNo319m63PehJx3lSeOLHxFk1P8fP6_siS4Gu2xhN0m8pwwc-zEr29hdIV4kPSzxpQ)

As an example, removing this feature direction means the model is worse at predicting the tokens in red & better at predicting the tokens in blue.

Ablating this feature direction appears to only affect latex-related tokens, but the direction can both increase & decrease the log-prob. It will trivially not affect any tokens before the first feature activation (because I subtract by direction*magnitude, and the magnitude is 0 there).

Uniform Examples
----------------

Maybe we’re deluding ourselves about the feature because we’re just looking at top-activating examples. If this is \*truly\* a monosemantic feature, then the entire activation range should have a similar meaning. So I look at datapoints across the bins of activations (ie sample a feature with activation \[1,2\], another from \[2,3\], ...)

![](https://lh4.googleusercontent.com/JgPmdMLoL7OhV-B7JsJNGzT0czp7Keh2YO1jRhhcxOg83-K0e80ul5UnfVljlYaCaX5CSLlyFpeCWbDKCumhhOgVi3Fv5vFj1aaSEcllN1e0SQo1FZJU85nowIEO8TiyUbqplXcVHJ-frZkCC__E64s)

It does seem to be mainly latex. See the second to last line which barely activates (.14), and is a dollar sign ("$ 100 billion tariff"). Maybe the model is detecting both math words & typically money words. Let's check w/...

Created Examples
----------------

![](https://lh3.googleusercontent.com/Qj8ymQvXGbI-4DZWG1_8_U4LaKkwXhQEGzCugUA_1Iflanhl2sUkXta7bAlbsksP50qQul-edI5FSL2M6WWoqsB2Q9iZ8esXVKvcFVmckm8g0u7O8QjuPNE6UwpzHkaR1b0KVp20i9VDUkXNk9OIYZo)

Notably, including the word “sold” immediately shoots down the activation! Also, it seems that combining math words increases it. 

To verify this, I’ll run the sentence “ for all $”, but prepend it w/ a token & see the effect on the feature activation. Instead of choosing a specific token, I can simply run ALL tokens in the vocab, printing the top-increasing & top-decreasing tokens.   
 

![](https://lh6.googleusercontent.com/Fyem4NKwqpPkVOTqJBEpS3LRyJdEJv4G3nQ3DF6A991zjBD70vmxOB-zxQp9L8ZdJu7tVziD-S5n9Llz265T2JhSZ-y0FdWf2iZBh1AzauM_Fj2Tkgm-wWqTK7ZwwXRoqn8NBU1RqA_MqBroeEGl7qg)

The most-increasing are definitely ending $ latex, but I’m not quite sure what all of them are. Like detecting an ending $ is most indicative?

The most-decreasing are indeed more related to the money-version of $, and there’s only 97 of them!

Checking with appending or prepending to the word “ tree”:

![](https://lh5.googleusercontent.com/Le1GbHW47v-Q5z7rwBwPbPQohYFO-t3bBB_3fdXlP1jGkTpfbVKFDuPaHwA5cQl8KSoiWhHl8aRTTAtQAXRQpBI7Xvl7kreYBApTO8nzMSklwB4vRUNKLIomyE5cYe87WKPnHpV806MidoY8sCUobag)

Yep, this fits within the hypothesis.

We can also just check the most activating tokens on its own

![](https://lh4.googleusercontent.com/A_xBD3opEbUM-xVUKPS4ftCNoR3dDEzj9PacB5YrSFg0SYPZW-Q7BEe4sUnilo1lbJ-kl8dZYqpbH64I5_uit8-JyAtYGx1xPJ7CXrFVAJe-ehQ6xhxYFFEwxneDFPQgejgZNPiCX-bXDnbkHdGSPuw)Beyond the first token, this doesn't fit within the hypothesis, though may be OOD. A check (for the future) would be to constrain by the model's next word prediction (and maybe do direct soft prompts).

Comparing to the Neuron Basis
-----------------------------

Does this feature net us anything over using the normal neuron basis? If this is only learning a monosemantic neuron, then that's pretty lame! 

We can first check how many neurons activate above a threshold for the top-10 feature activating examples (ie a neuron must activate above threshold for all 10 examples)gh-MCS Features

![](https://lh6.googleusercontent.com/YXoI3rLZVlwyAb1XdBr2QJOh1JPPJ5w58wwqhCVIk86kmvj27jghXkJ1iuMs1rOtr-Eb5nf5M_B6gC8g0nlmnx3bdWCdt5cp3-0MZM3k9DWWLA3CThtJeqx-ydTyw5fkAN1ISAZccWq7RMJaxos85HY)

Notably it's a lot of neurons for above 0, but then goes to 3 for 0.5. However, we don't really know the statistics of neuron activations. Maybe some neuron's entire range is very tiny? So, we can see how many neurons are above a threshold determined by that neuron's quantiles.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e610e71bd63e5460f1c36f17a75b57066cb4bd1a980dcd68.png)

Here, 60 neurons are activating in above their 80th quantile, but it's unclear where to draw the line still.

Another way is to look at the feature vector associated w/ this feature. It is 2048 numbers representing how much it affects each neuron, so if it's 0, it doesn't affect a neuron at all. Additionally, I'll multiply by the max-activation of that feature to show the scale of the effect on the neurons.

![](https://lh3.googleusercontent.com/uc_fTG7RyRK5U4ND8oQRCUbJesEOQOOcAw5Hjo1fgoMsN0rBmHEdue3P7JqiyolHnGU1hz_EQt2CF6otAgDzRDZqH5V-lopSKYtavwY-68quE9yE_9Yth8d5x_0J0j-0bF-E4z_o4nU3rLQgZgHe_Y8)

So a few datapoints to indicate somewhere between 3 & 80 neurons, maybe several hundred depending on how you interpret it.

We can also check across the top-50 high MCS features for activations

![](https://lh6.googleusercontent.com/CyEuK2aH6yABV8BzT25slUqXf6ERb6x_jSSRpZif8L1lHyCsf5tvAi9uT6leRfZbNuQPmloy3ekKg3HtcceXhLRNA_6idVvxCe5ijoSJE0ADTJVyN2vxndS-kM6mu_jDrQaRuPROYS2pFlIfqazjoZY)

As a contrast, I will also being showing an equivalent graph, but from a different dictionary that seems to have learned the identity (ie we're just looking at the neuron basis)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/84ba222e9f68bc6870e25fdf6fb036592f3a6b5af8e5a343.png)

Here, the vast majority of features correlate w/ only 1 neuron activation (at least above 0.3).

Going through the images quickly for a "feature" here:

Max activation:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f3a63dfd4a93bd8c908f709a97489d4e7ab92760b4cc1995.png)

Ablating context:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/006acfa5740e325be2683464665187831b0ed6f94eba78d2.png)

Logit Diff:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/13b3e3d52c4a11d362278b33cc27c63c561f88aea5760365.png)

Uniform Examples:  
![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4256f4ecfa6c37f1538a8bd1ace9fcb905f76b130028bb3d.png)

I'm really unsure on what this feature could even represent to even come up w/ a testable hypothesis

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/ad309cfc9cc5fafb3f117e8a9e604826294ff8eff41d23e3.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8ee990002c7274844aa8d7d584157e73ac149318d9e3d503.png)

Notably, it drops to 1 neuron after 0.8, so w/ a threshold of 0.8, so some evidence for the original dictionary feature representing ~60 neurons.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b7555a1ba3b264b3d43bbc1c5e1e0b9a63601786a2b52f14.png)

Here, there's only 1 feature weight that's > 0.04 (it's 2.9)

So pretty strong evidence that the original dictionary is giving us a meaningful features that aren't just monosemantic neurons.

\[Note: One idea is to label the dataset w/ the feature vector e.g. saying this text is a latex $ and this one isn't. Then learn several k-sparse probes & show the range of k values that get you whatever percentage of separation\]

\[Note2: There are also meaningful monosemantic neurons in the model, but I specifically chose a feature learned that represents a polysemantic neuron. The point here is: can our dictionary learn meaningful features that are linear combinations of neurons?\]

Range of MCS Features
=====================

That was a case study: it could be cherry picked. So I went through the top 70 MCS features w/ a quick check (~2 min each).

[Excel link](https://docs.google.com/spreadsheets/d/1p1Wu4vJ1fKYsMtjrXFboQpIOl_sKue-dgabFt2EYw0s/edit?usp=sharing)

![](https://lh5.googleusercontent.com/Q1VdLp-9BDSHvxN7Iti15Pi2jlszE_Tc8z1meMPrtMPEu-Cb8q7vkO_O3_QQvNU9H_zeCc3XQr_TodvmUWDoX2nXFvKWpSanR98WUdtGjzCjMIw0D9IHE76IYWla1492RiYSUOIKeQJH4ezNcNeEFgw)

![](https://lh4.googleusercontent.com/MPmZlA0DJat_HNDiRNsIRlgv8pYg72m13Dvh_jm9iEGAeMOnxDhaKlnUZbApct0ITIMZfW604Cth5SQyYglMkPQ9BB6uWPUl8DOHmp4gxkMVRIonKa50sX1iVijwfhhusUAozKp3xpDkadULbI-2v8k)

But we’re using high MCS to say it’s good. What if low MCS is also good?

![](https://lh5.googleusercontent.com/9sgbrj9tW-kHQs2BtEywXeBZ2CilmVxwl1CA20ud5JODurnY6MClzXHMKaovsWZ-jdvmvmRnG6_3O_ucLUJ9j2LXr5mB1BZX7I2H4MTDoT_9mjbzIgU5UeXoGxSHl_D27hlUvZsoKLejOsiUBhgc9h8)

Some do look clearly like features! 1580 specifically activates more if you prepend a meaningful word & only affects the word right after "rom" (e.g. "berg"/"eters"), however, there’s a clear trend that lower MCS features appear more polysemantic. Most notable is that I couldn’t even check the majority of low-MCS features here because they were “dead”: there were < 10 activations, usually 0 for those features! To illustrate: 

![](https://lh5.googleusercontent.com/CIiZdHmmCkJqGw9I4kjjFjI_WtUZc1Zc5Klqxbq-Kt5o3j8qaS5llGdpD6eOr3Y592tNafYGG89O7EXwLgwi-HK1V0F7KPrj_h4e79jywfk9kN3i4IqswhiS9izCKfCO3TjnjkzuzksuKyXRSxTnhZw)

\[Update: After training other dictionaries on more data, I've found more low-MCS features that seem meaningful, meaning one dictionary found features that a separate dictionary didn't find. Later, I will provide a better statistic of the result e.g. N% of low-MCS features seem meaningful to Logan\]

Failures
========

1.  Logit Lens: logit lens of the $ feature didn’t show any meaningful features. Neither did the first 10 highest MCS features  
    ![](https://lh4.googleusercontent.com/ynr78xeD_-GFJ4zKo2rHa2-3IYv7hTezoVPNJRuSuK-JTZW-Ht8Yq6V-japEMTSZPl--EhC6iBotltweoauTXB2NAkS-5iuiekrrsKwc8elyWwnSTaIEYojRE80KZ7rSlE-st-06wMc-hOJSfUzM_U8)
2.  Max-ablation-diff: related, showing the max logit-diff when ablating the feature direction also showed nonsense tokens.

Conclusion
==========

It'd be huge if we could find decompositions of mid-MLP activations that faithfully reconstruct the original neuron activations & allows us to easily specify circuits we want (e.g. honesty, non-deception, etc). There's work to be done to clarify what metrics we care about (e.g. maybe simplicity, explainability, monosemanticity, can predict OOD, model-editing benchmarks) & comparing w/ existing methods (e.g. PCA, other loss functions, etc).

What's currently most interesting imo:

1.  Learning all the features for a layer w/ low-reconstruction loss (concretely: low perplexity diff when replacing w/ reconstruction)
2.  Show/Falsify feature connections across layers (e.g. layer 1's duplicate word feature is commonly used in layer 2's features X, Y, & Z)
3.  More rigorously show what percentage of low-MCS features are\[n't\] meaningful in the other dictionary I trained.
4.  Training dictionaries for much, much larger models to see if more interesting features pop up (e.g. personality traits)

If you'd like to continue or replicate results:

*   [main code-base](https://github.com/HoagyC/sparse_coding)
*   [Notebook for this post](https://github.com/loganriggs/sparse_coding/blob/main/interpreting_sparse_dictionaries.ipynb)
*   [Autoencoder](https://huggingface.co/Elriggs/autoencoder_layer_2_pythia70M_5_epochs) 

Feel free to reach out on our discord project channel:

*   [Discord Link (EleutherAI)](https://discord.gg/eleutherai)
*   [Channel Link](https://discord.com/channels/729741769192767510/1115338262626709567) (Or go to the #community-channels channel & we're the sparse coding project)

*Special thanks to Wes Gurnee for advice (& picking out this surprisingly interesting feature!), Neel Nanda for pushing for being more exact & rigorous w/ understanding this feature, Hoagy for co-developing the original dictionary learning codebase, and Nora Belrose, Aiden, & Robert for useful discussions on the results, & EleutherAI for hosting our discussions.*