# (OLD) An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers

# This post is out of date, see [v2 here](https://www.alignmentforum.org/posts/NfFST5Mio7BCAQHPA/an-extremely-opinionated-annotated-list-of-my-favourite-1)

## Introduction

This is an extremely opinionated list of my favourite mechanistic
interpretability papers, annotated with my key takeaways and what I like
about each paper, which bits to deeply engage with vs skim (and what to
focus on when skimming) vs which bits I don’t care about and recommend
skipping, along with fun digressions and various hot takes.

This is aimed at people trying to get into the field of mechanistic
interpretability (especially Large Language Model (LLM)
interpretability). I’m writing it because I’ve benefited a lot by
hearing the unfiltered and honest opinions from other researchers,
especially when first learning about something, and I think it’s
valuable to make this kind of thing public! On the flipside though, this
post is explicitly about *my* personal opinions - I think some of these
takes are controversial and other people in the field would disagree.

The four top level sections are priority ordered, but papers within each
section are ordered arbitrarily - follow your curiosity

## Priority 1: What is Mechanistic Interpretability?

-   [Circuits: Zoom
    In](https://distill.pub/2020/circuits/zoom-in/)

    -   Sets out the circuits research agenda, and is a whirlwind
        overview of progress in image circuits

    -   This is reasonably short and conceptual (rather than technical)
        and in my opinion very important, so I recommend deeply
        engaging with all of it, rather than skimming.

    -   The core thing to take away from it is the perspective of
        networks having legible(-ish) internal representations of
        features, and that these may be connected up into
        interpretable circuits. The key is that this is a mindset for
        thinking about networks *in general*, and all the discussion
        of image circuits is just grounding in concrete examples.

        -   On a deeper level, understanding why these are important and
            non-trivial claims about neural networks, and their
            implications.

    -   In my opinion, the circuits agenda is pretty deeply at the core
        of what mechanistic interpretability *is*. It’s built on the
        assumption that there is some legible, interpretable structure
        inside neural networks, if we can just figure out how to
        reverse engineer it. And the core goal of the field is to find
        what circuits we can, build better tools for doing so, and do
        the fundamental science of figuring out which of the claims
        about circuits are actually true, which ones break, and
        whether we can fix them.

        -   An important note is that mechanistic interpretability is an
            extremely young field and this was written 2.5 years ago -
            I take the specific claims in this article as a starting
            point, not as the definitive grounding of what the field
            must believe.

    -   **Meta:** The goal of reading this is to understand what the
        fundamental mindset and worldview being defended here is. The
        goal is *not* necessarily to leave feeling convinced that
        these claims are true, or that the article adequately
        justifies them. That’s what the rest of the papers in here are
        for!

    -   A useful thing to reflect on is what the world would look like
        if the claims were and were not true - what evidence could you
        see that might convince you either way? These are definitely
        not obviously true claims!

-   [A Mathematical Framework for Transformer
    Circuits](https://transformer-circuits.pub/2021/framework/index.html)

    -   The point of this is to explain how to conceptually break down a
        transformer into individually understandable pieces.

    -   Deeply engage with:

        -   All the ideas in the overview section, especially:

            -   Understanding the residual stream and why
                it’s fundamental.
            -   The notion of interpreting *paths* between interpretable bits (eg input tokens and output logits) where the path is a composition of matrices and how this is different from interpreting every intermediate activations

            -   And understanding attention heads: what a QK and OV
                matrix is, how attention heads are independent and
                additive and how attention and OV are
                semi-independent.

        -   Skip Trigrams & Skip Trigram bugs, esp understanding *why*
            these are a really easy thing to do with attention, and
            how the bugs are inherent to attention heads separating
            *where* to attend to (QK) and what to do once you attend
            somewhere (OV)

        -   Induction heads, esp why this is K-Composition (and how
            that’s different from Q & V composition), how the circuit
            works mechanistically, and why this is too hard to do in a
            1L model

    -   Skim or skip:

        -   Eigenvalues or tensor products. They have the worst effort
            per unit insight of the paper and aren’t very important.

    -   Maybe check out [my (long-ass) walkthrough of the
        paper](https://www.youtube.com/watch?v=KV5gbOmHbjU), and
        comments on how I think about things

        -   If you prefer video over reading I expect it to be high
            value

        -   Either way it’s probably useful to check the relevant
            section it if there’s part of the paper that confuses you.

## Priority 2: Understanding Key Concepts in the field

-   [Induction
    Heads](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html)

    -   This is a study of how induction heads are ubiquitous in real
        transformers, and form as a sudden phase change during
        training.

    -   Deeply engage with:

        -   Key concepts + argument 1.

        -   Argument 4: induction heads also do translation + few shot
            learning.

        -   Getting a rough intuition for all the methods used in the
            Model Analysis Table, as a good overview of interesting
            interpretability techniques.

    -   Skim or skip:

        -   All the rigour - basically everything I didn’t mention. The
            paper goes way overboard on rigour and it’s not worth
            understanding every last detail

            -   The main value to get when skimming is an overview of
                different techniques, esp general techniques for
                interpreting during training.

    -   A particularly striking result is that induction heads form at
        ~the same time in all models - I think this is very cool, but
        somewhat overblown - from some preliminary experiments, I
        think it’s pretty sensitive to learning rate and positional
        encoding (though the fact that it *doesn’t* depend on scale is
        fascinating!)

-   [Mechanistic Interpretability, Variables, and the Importance of
    Interpretable
    Bases](https://transformer-circuits.pub/2022/mech-interp-essay/index.html)

    -   Short-ish conceptual essay on what the point of mechanistic
        interpretability is and how to think about it.

    -   This is similar in flavour to Circuits: Zoom In, but is more
        conceptual and less grounded in very concrete examples +
        progress - your mileage may vary in how much this works for
        you.

-   [A Toy Model of
    Superposition](https://transformer-circuits.pub/2022/toy_model/index.html)

    -   Building a simple toy model that contains superposition, and
        analysing it in detail.

    -   Deeply engage with:

        -   The core intuitions: what is superposition, how does it
            respond to feature importance and sparsity, and how does
            it respond to correlated and uncorrelated features.

        -   Read the strategic picture, and sections 1 and 2 closely.

    -   Skim or skip:

        -   No need to deeply understand the rest, it can mostly be
            skimmed. It’s very cool, especially the geometry and phase
            transition and learning dynamics part, but a bit of a nerd
            snipe and doesn’t obviously generalise to real models.

    -   A good intro paper for concrete projects. The models are tiny,
        the core results should be easy to replicate (and have short
        training times), there’s an [accompanying
        Colab](https://colab.research.google.com/github/anthropics/toy-models-of-superposition/blob/main/toy_models.ipynb)
        and [a list of follow-up
        ideas](https://transformer-circuits.pub/2022/toy_model/index.html#open-questions),
        so this is a great paper to play around with!

-   [Curve
    Detectors](https://distill.pub/2020/circuits/curve-detectors/)
    & [Curve
    Circuits](https://distill.pub/2020/circuits/curve-circuits/)
    (Image interpretability)

    -   An extremely detailed and rigorous study of a family of neurons
        in Inception; a gold standard of what good interpretability
        can look like. Culminates in them hand-coding the weights of
        artificial neurons and substituting those into the circuit,
        and comparing performance. Note that a bunch of the techniques
        won’t generalise.

    -   Deeply engage with:

        -   Understanding what they did as a gold standard, and thinking
            about *why* what they did is deep and meaningful evidence.

        -   Think about which techniques will and will not generalise to
            LLMs

## Priority 3: Expanding Understanding

### Language Models

-   [Indirect Object
    Identification](https://openreview.net/forum?id=NpsVSN6o4ul)

    -   A paper about reverse engineering a complex (28 head!) circuit
        in GPT-2 Small

        -   The most detailed “we actually have a circuit, and can drill
            into it in detail and really get how it works” paper that
            I know of.

            -   The circuit in question is for the task of completing
                “When John and Mary went to the shops, John bought a
                bottle of milk for” -&gt; “ Mary” but “Mary bought a
                bottle of milk for” -&gt; “ John”

    -   Particularly good for a vibe of “ways interpretability is hard
        and you can trick yourself” + “but it is actually possible and
        we can fix these”

-   [SoLU](https://transformer-circuits.pub/2022/solu/index.html)

    -   A paper on a neuron activation function that makes transformer
        neurons somewhat more interpretable.

    -   Deeply engage with:

        -   Section 3 (Background). For the core ideas, esp
            superposition, privileged bases and why they matter.

            -   See “A Toy Model of Superposition” for much more on
                superposition.

        -   Section 6 (on the neurons found). For getting the vibe of
            what kind of features LLMs learn - I think this is the
            best resource I know of for getting a vibe of what kinds
            of things MLP layers are doing at different layers of a
            transformer.

    -   Skim:

        -   Section 4 (on the exact function and how it works) - the
            main intuition to get is *why* you might expect this to
            work (in particular, why lateral inhibition seems
            important)

    -   Skip:

        -   Section 5 (showing that the model works as well as normal
            activation functions).

-   [ROME](https://rome.baulab.info/)

    -   A paper on locating and editing factual knowledge in GPT-2 - a
        strong contender for my favourite non Chris Olah
        interpretability paper

    -   Deeply engage with:

        -   Causal tracing + activation patching stuff (including the
            appendix on it). It’s a really cool, elegant and general
            technique, and demonstrates that certain computation is
            extremely localised in the model, and uses careful
            counterfactuals to isolate this computation.

    -   Skim or skip:

        -   The model editing stuff. It’s way less interesting from an
            interpretability point of view than the above.

-   [Logit
    Lens](https://www.alignmentforum.org/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens)

    -   A solid early bit of work on LLM interpretability. The key
        insight is that we interpret the residual stream of the
        transformer by multiplying by the unembedding and mapping to
        logits, *and* that we can do this to the residual stream
        before the final layer and see the model converging on the
        right answer

        -   Key takeaway: Model layers iteratively update the residual
            stream, and the residual stream is the central object of a
            transformer

    -   Deeply Engage with:

        -   The key insight of applying the unembedding early, and
            grokking *why* this is a reasonable thing to do.

    -   Skim or skip:

        -   Skim the figures about progress towards the answer through
            the model, focus on just getting a vibe for what this
            progress looks like.

        -   Skip everything else.

    -   The deeper insight of this technique (not really covered in the
        work) is that we can do this on *any* vector in the residual
        stream to interpret it in terms of the direct effect on the
        logits - including the output of an attn or MLP layer and even
        a head or neuron. And we can also do this on weights writing
        to the residual stream.

        -   [Analyzing Transformers in Embedding
            Space](https://arxiv.org/pdf/2209.02535.pdf) is a more
            recent paper that drills down into this insight, focusing
            on weights.

            -   I’m somewhat meh on the paper as a whole, but sections
                3, 4.1 and Appendix C are cool for seeing what head
                and neuron circuits can look like

            -   Note that they make the (IMO) mistake of treating
                embedding and unembedding space as the same space -
                the input and output are different spaces! Even if
                most people make the mistake of setting the embed and
                unembed maps to be the same matrix :(

        -   Note that this tends only to work for things close to the
            final layer, and will totally miss any indirect effect on
            the outputs (eg via composing with future layers, or
            suppressing incorrect answers)

-   [An Interpretability Illusion for
    BERT](https://arxiv.org/pdf/2104.07143.pdf)

    -   Good early paper on the limitations of max activating dataset
        examples - they took a seemingly interpretable neuron in BERT
        and took the max activating dataset examples on different
        datasets, and observed consistent patterns *within* a dataset,
        but very different examples *between* datasets

        -   Within the lens of the Toy Model paper, this makes sense!
            Features correspond to directions in the residual stream
            that probably aren’t neuron aligned. Max activating
            dataset examples will pick up on the features *most*
            aligned with that neuron. Different datasets have
            different feature distributions and will give different
            “most aligned feature”

            -   Further, models want to minimise interference and thus
                will superpose anti-correlated features, so they
                *should*

    -   Deeply engage with:

        -   The concrete result that the same neuron can have very
            different max activating dataset examples

        -   The meta-level result that a naively compelling
            interpretability technique can be super misleading on
            closer inspection

    -   Skim or skip:

        -   Everything else - I don’t care much about the details beyond
            the headline result, which is presented well in the intro.

### Algorithmic Tasks

-   [A Mechanistic Interpretability Analysis of
    Grokking](https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking)

    -   *Conflict of interest note - I was the main person working on
        this project!*

    -   A very detailed reverse engineering of a tiny model trained to
        do modular addition and interpreting it during training, plus
        a bunch of discussion on phase changes, an (attempted)
        explanation of
        [grokking](https://arxiv.org/abs/2201.02177) and
        showing grokking on other tasks.

        -   Grokking probably isn’t that relevant to real models and the
            techniques don’t really generalise, but a good example of
            detailed reverse engineering + fully understanding a model
            on an algorithmic task, and of applying interpretability
            during training.

            -   Also a good example of how actually understanding a
                model can be really useful, and push forwards science
                of deep learning by explaining confusing phenomena.

        -   I also just personally think this project was super fucking
            cool, even if not that useful.

    -   Deeply engage with:

        -   The key claims and takeaways sections

        -   [Overview of the modular addition
            algorithm](https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking#Overview_of_the_Inferred_Algorithm)

            -   The key vibe here is “holy shit, that’s a
                weird/unexpected algorithm”, but also, on reflection,
                a pretty natural thing to learn if you’re built on
                linear algebra - this is a core mindset for
                interpreting networks!

    -   Skim:

        -   [Reverse engineering modular
            addition](https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking#Reverse_Engineering_the_Algorithm) -
            understanding the different types of evidence and how they
            fit together

        -   [Evolution of modular addition circuits during
            training](https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking#Evolution_of_Circuits_During_Training) -
            the flavour of what the circuits developing looks like
            during training, and the fact that once we understand
            things, we can just literally watch them develop!

            -   [The interactive graphics in the
                colab](https://colab.research.google.com/drive/1F6_1_cWXE5M7WocUcpQWp3v8z4b1jL20#scrollTo=Analysis_During_Training)
                are *way* better than static images

        -   [The Phase Changes
            section](https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking#Phase_Changes) -
            probably the most interesting bits are the explanation of
            grokking, and the two speculative hypotheses.

    -   Maybe a good intro paper to replicate! It has an
        [accompanying colab](http://bit.ly/neelgrokking) and a
        list of future directions at the end

### Image Circuits

-   [Feature
    Vis](https://distill.pub/2017/feature-visualization/) (fairly
    short)

    -   An early paper with a really core technique for image
        interpretability. Doesn’t really transfer to LLMs, but worth
        getting the vibe, and seeing how this made image
        interpretability much easier and more rigorous in certain
        ways - the vibe that this basically automatically gives
        variable names to neurons.

-   [Multimodal Neurons in Artificial Neural
    Networks](https://distill.pub/2021/multimodal-neurons/)

    -   An analysis of neurons in a text + image model (CLIP), finding a
        bunch of abstract + cool neurons. Not a high priority to
        deeply engage with, but very cool and worth skimming.

    -   My key takeaways

        -   [There are so many fascinating
            neurons!](https://distill.pub/2021/multimodal-neurons/#guided-tour-of-neuron-families)
            Like, what?

            -   There’s a
                [teenage](https://microscope.openai.com/models/contrastive_4x/image_block_4_5_Add_6_0/2231)
                neuron, a
                [Minecraft](https://microscope.openai.com/models/contrastive_4x/image_block_4_5_Add_6_0/3)
                neuron, a
                [Hitler](https://microscope.openai.com/models/contrastive_4x/image_block_4_5_Add_6_0/309)
                neuron and an
                [incarcerated](http://microscope.openai.com/models/contrastive_4x/image_block_4_5_Add_6_0/2297)
                neuron?!

        -   The intuition that multi-modal models (or at least, models
            that use language) are incentivised to represent things in
            a conceptual way, rather than specifically tied to the
            input format

        -   The detailed analysis of the [Donald Trump
            neuron](https://distill.pub/2021/multimodal-neurons/#person-neurons),
            esp that it is more than just a “activates on Donald
            Trump” neuron, and instead activates for many different
            clusters of things, roughly tracking their association
            with Donald Trump.

            -   This seems like weak evidence that neuron activations
                may split more into interpretable segments, rather
                than an interpretable directions

        -   The “[adversarial attacks by writing Ipod on an
            apple](https://distill.pub/2021/multimodal-neurons/#typographic-attacks)”
            part isn’t very deep, but *is* hilarious

-   [The rest of the circuits
    thread](https://distill.pub/2020/circuits/)

    -   A lot of really cool ideas and scattered threads! Worth skimming
        and digging into anything that catches your interest. Each
        individual article is short-ish

    -   This thread represents, in my opinion, the first serious attempt
        at reverse engineering a real model (inception)

    -   My personal favourites:

        -   [An Overview of Early Vision
            Neurons](https://distill.pub/2020/circuits/early-vision/) -
            it’s just fascinating to see the weird shit that happens,
            super cool to the hierarchy where see simple shapes are in
            early layers and are built into more abstract shapes in
            layer layers, and to see neurons being sorted into
            families

            -   If you click on a neuron, you’ll see the [weight
                explorer](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_47.html) -
                this is a really fun tool to play around with, and
                practice just reading off the weights what they do!

        -   [Visualising
            weights](https://distill.pub/2020/circuits/visualizing-weights/) -
            somewhat image specific, but a fascinating exploration of
            the data visualisation questions underlying mechanistic
            interpretability - visualisations are super useful, but
            *how* can we do them in a properly principled way, and how
            can they mislead?

            -   I really want to see more papers like this! These meta
                questions are really important, but it’s rarely
                incentivised to publish on them

        -   [Branch
            Specialisation](https://distill.pub/2020/circuits/branch-specialization/) -
            networks spontaneously learn to be modular *and* the
            modules seem to be consistent and semantically
            meaningful?! WTF?

## Priority 4: Bonus

-   Not a paper: The [codebase of
    EasyTransformer](https://github.com/neelnanda-io/Easy-Transformer),
    a transformer mechanistic interpretability I’m writing - I think
    it’s worth reading for a fairly clean and conceptual-focused
    implementation of a transformer, specifically reading
    [EasyTransformer.forward](https://github.com/neelnanda-io/Easy-Transformer/blob/ff0a4ef52d606277894acd306a97018999fee958/easy_transformer/EasyTransformer.py#L136)
    and
    [components.py](https://github.com/neelnanda-io/Easy-Transformer/blob/54f4b4bfd15448b98288a51eef7ddddeac23b68d/easy_transformer/components.py)
    (a file for the various layers) (the actual codebase is pretty
    long!)

-   [Everything else Chris Olah](https://colah.github.io/) has
    ever written

    -   I’m somewhat biased on this, but I think Chris is just clearly
        far and away the best interpretability researcher in the
        world.

    -   He’s also a massive nerd for good technical communication,
        interactivity and good graphic design, and I find his work a
        joy to read.

-   [Interpreting RL
    Vision](https://distill.pub/2020/understanding-rl-vision/)

    -   Interesting application of image circuits techniques to get some
        insight into an RL model - [unclear how much it
        generalises/works](https://www.alignmentforum.org/posts/iJDmL7HJtN5CYKReM/empirical-observations-of-objective-robustness-failures)

    -   The parts about the impact of the amount of and diversity of
        data on interpretability feel most interesting and general to
        me.

    -   Probably the best RL mechanistic interpretability paper I know
        of (but it’s a pretty low bar :( )

-   Not a paper: Playing around with [OpenAI
    Microscope](https://microscope.openai.com/) - visualizations
    and top dataset examples of every neuron in a ton of image models!
    Challenge: What’s the weirdest neuron you can find?

-   [Visualizing and Interpreting the Geometry of
    BERT](https://arxiv.org/abs/1906.02715) (+ [blog
    post](https://pair-code.github.io/interpretability/bert-tree/))

    -   An early LLM interpretability paper about understanding how BERT
        represents language in the residual stream.

    -   Deeply engage with:

        -   Applying t-SNE to the residual stream + getting resulting
            visualizations. This was really clever and cool, and
            understanding it is valuable.

    -   Skim or skip:

        -   The detailed syntax tree stuff.

-   [Acquisition of Chess Knowledge in
    AlphaZero](https://arxiv.org/pdf/2111.09259.pdf) - analysing
    AlphaZero’s chess knowledge, including during training

    -   Notable for the hilarious stunt of getting a chess grandmaster
        commenting, and for co-authoring (even if this isn’t that
        interpretability related)

    -   Focuses on feature analysis rather than really mechanistic
        engagement, but still very cool! The main things I think are
        cool were successfully applying interpretability during
        training, and on the weird and fucky task of playing chess
        (and that models trained on non-image/language tasks are
        somewhat interpretable!).

-   [Toward Transparent AI: A Survey on Interpreting the Inner
    Structures of Deep Neural
    Networks](https://arxiv.org/abs/2207.13243) - a decent survey
    paper on what’s up in the rest of interpretability.

    -   I’m personally pretty meh about the majority of the academic
        field of interpretability (I rarely find insights from there
        useful in my work) and would prioritise reading the papers in
        the previous sections, but it’s worth skimming to get a sense
        for what’s out there, and digging into anything relevant to a
        specific project you’re pursuing!

        -   Also, for sanity checking whether I’m just being
            overconfident/arrogant, and there’s actually a ton of
            useful insights in standard interpretability for
            mechanistic stuff! Again, this post is just a list of my
            personal hot takes.

    -   [A Primer in
        BERTOLOGY](https://arxiv.org/pdf/2002.12327.pdf) - a
        survey paper specifically on BERTology, a subfield about
        specifically interpreting BERT. I feel pretty meh about this,
        but am not very familiar with the field.

-   [The Building Blocks of
    Interpretability](https://distill.pub/2018/building-blocks/)

    -   A cool and fun whirlwind tour of a bunch of different tools and
        approaches for image interpretability. Worth skimming.

-   Not a paper, but I find [Chris Olah’s interview on the 80,000
    Hours
    podcast](https://80000hours.org/podcast/episodes/chris-olah-interpretability-research/)
    super inspiring