# Knowledge Neurons in Pretrained Transformers

This is a link post for the Dai et al. paper “[Knowledge Neurons in Pretrained Transformers](https://arxiv.org/abs/2104.08696)” that was published on the arXiv last month. I think this paper is probably the most exciting machine learning paper I've read so far this year and I'd highly recommend others check it out as well. _Edit: Maybe not; I think [Paul's skeptical take here is quite reasonable](https://www.lesswrong.com/posts/LdoKzGom7gPLqEZyQ/knowledge-neurons-in-pretrained-transformers?commentId=tqgQ9eyqWvk28nBG5#comments)._

To start with, here are some of the basic things that the paper demonstrates:

- BERT has specific neurons, which the authors call “knowledge neurons,” in its feed-forward layers that store relational facts (e.g. “the capital of Azerbaijan is Baku”) such that controlling knowledge neuron activations up-weights/down-weights the correct answer in relational knowledge prompts (e.g. “Baku” in “the capital of Azerbaijan is <mask>”) even when the syntax of the prompt is changed—and the prompts that most activate the knowledge neuron all contain the relevant relational fact.
- Knowledge neurons can reliably be identified via a well-justified integrated gradients attribution method (see  also “[Self-Attention Attribution](https://arxiv.org/abs/2004.11207)”).
- In general, the feed-forward layers of transformer models can be thought of as key-value stores that memorize relevant information, sometimes semantic and sometimes syntactic (see also “[Transformer Feed-Forward Layers Are Key-Value Memories](https://arxiv.org/abs/2012.14913)”) such that knowledge neurons are composed of a “key” (the first layer, prior to the activation function) and the “value” (the second layer, after the activation function).

The paper's key results—at least as I see it, however—are the following:

- Taking knowledge neurons that encode “the $r$ of $h$ is $t$” and literally just adding $\mathbf t^\prime - \mathbf t$ to the value neurons (where $\mathbf t,~ \mathbf t^\prime$ are just the embeddings of $t,~ t^\prime$) actually changes the knowledge encoded in the network such that it now responds to “the $r$ of $h$ is <mask>” (and other semantically equivalent prompts) with $t^\prime$ instead of $t$.
- For a given relation (e.g. “place of birth”), if all knowledge neurons encoding that relation (which ends up being a relatively small number, e.g. 5 - 30) have their value neurons effectively erased, the model loses the ability to predict the majority of relational knowledge involving that relation (e.g. 40 - 60%).

I think that particularly the first of these two results is pretty mind-blowing, in that it demonstrates an extremely simple and straightforward procedure for directly modifying the learned knowledge of transformer-based language models. That being said, it's the second result that probably has the most concrete safety applications—if it can actually be scaled up to remove _all_ the relevant knowledge—since something like that could eventually be used to ensure that a [microscope AI](https://www.alignmentforum.org/posts/X2i9dQQK3gETCyqh2/chris-olah-s-views-on-agi-safety#Building_microscopes_not_agents) isn't [modeling humans](https://www.alignmentforum.org/posts/BKjJJH2cRpJcAnP7T/thoughts-on-human-models) or ensure that an agent is [myopic](https://www.alignmentforum.org/posts/BKM8uQS6QdJPZLqCr/towards-a-mechanistic-understanding-of-corrigibility) in the sense that it isn't modeling the future.

Furthermore, the specific procedure used suggests that transformer-based language models might be a lot less inscrutable than previously thought: if we can really just think about the feed-forward layers as encoding simple key-value knowledge pairs _literally in the language of the original embedding layer_ (as I think is also independently suggested by “[interpreting GPT: the logit lens](https://www.alignmentforum.org/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens)”), that provides an extremely useful and _structured_ picture of how transformer-based language models work internally.