# Really Strong Features Found in Residual Stream

\[I'm writing this quickly because the results are really strong. I still need to do due diligence & compare to baselines, but it's really exciting!\]

[Last post](https://www.lesswrong.com/posts/wqRqb7h6ZC48iDgfK/tentatively-found-600-monosemantic-features-in-a-small-lm) I found 600+ features in MLP layer-2 of Pythia-70M, but ablating the direction didn't always make sense nor the logit lens. Now I've found 3k+ in the residual stream, and the ablation effect is intuitive & strong.

The majority of features found were single word or combined words (e.g. not & 't ) which, when ablated, effected bigram statistics, but I also wanted to showcase the more interesting ones:

German Feature?
---------------

Uniform examples:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4bc6a2279ebd9d3682caea2b2c9f6eb2bc9864c1ac8c83cc.png)

Logit lens:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1b3fe81564a578260bbed61d4c98fb09be2d1ccd200f52cf.png)

Stack Exchange
--------------

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/de8adada170c0fd900dde88fed5a26e7e2b0d88445fd03bf.png)

Ablated Text (w/ red showing removing that token drops activation of that feature)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/ecd924675c15c164eebd894252a380a41267167fd9ddf3eb.png)

In the Pile dataset, stackexchange text is formatted as "Q:/n/n....", and removing the "Q" greatly decreasing the feature activation is evidence for this indeed detecting stackexchange format.

### Title Case

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/76b2392742bd8ec5bb1f2b2a333e72f30ba24dbf03c2e101.png)

Ablating Direction (in text):

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/3064cb2302bc3a915862768c364d7851bcc45d8a9855a348.png)

Here, the log-prob of predicting " Of" went down by 4 when I ablate this feature direction, along w/ other Title Case texts.

Logit lens

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/87fa72e83bb91e058516af6ebade83b3ed9653685068d93f.png)

### One for the Kids

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/348d4b492c2b762660e129a75aace754de13b4878d1d4149.png)

Last Name:  
![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/2724b290be1c24d0af01b7c4182f38f85bef052c4c90853f.png)

Ablated Text:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/3de52a04c37f1a8895c6e2351bfcd01916895728f8034bda.png)

As expected, removing the first name affects the feature activation

Logit Lens:  
![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4bb0d06896236cb68c3471e48d2291f03be2731ea7eb7aeb.png)

More specifically, this looks like the first token of the last name, which makes since to follow w/ Jr or Sr or completing last names w/ "hoff" & "worth". Additionally, the "(\\@" makes sense for someone's social media reference (e.g. instagram handle).

### \[word\] and \[word\]

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5ba68829c1c05f6350b6920d6dd6f3e837a4a254462b0ace.png)

Ablated Text:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/9e55c54f95e18985a133b4baa442f839d8fed6ddd182894e.png)

Logit Lens:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/ab437e1d2883fb2d8076978389491bf96e166f3b82a966a7.png)

alike & respectively make a lot of since in context, but not positive about the rest.

### Beginning & End of First Sentence?

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6f3a8c08d11c3bfc71f63262cb8d7a0e3e213bdab1e5592a.png)

On

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6aab7ff249ca4638260a1ff54d3e672161f775782df95bb1.png)

logit lens:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/9bb3a6fbac04e1baa6b2c7f831014aa5fe6609f44c11b427.png)

"on behalf", "oncology", "onyx" (lol), "onshore", "onlook", "onstage", "onblur"(js?), "oneday" ("oneday" is separated as on eday?!?), "onloaded" (js again?), "oncomes"(?)

High Level Picture
==================

I could spend 30 minutes (or hours) developing very rigorous hypotheses on these features to really nail down what distribution of inputs activate it & what effect it really has on the output, but that's not important. 

What's important is that we have a decomposition of the data that:  
1\. Faithfully represents the original data (in this case, low reconstruction loss w/ minimal effect on perplexity)

2\. Can simply specify desired traits in the model (e.g. honesty, non-deception)

There is still plenty of work to do to ensure the above two work (including normal hyperparameter search), but it'd be great to have more people working on this (I don't have time to mentor, but I can answer questions & help get you set up) if that's helping our work or integrating it in your work.

This work is legit though, and there's [a recent paper](https://arxiv.org/pdf/2103.15949.pdf) that finds similar features in a BERT model.