Successor Heads: Recurring, Interpretable Attention Heads In The Wild
Authors: Rhys Gould
Euan Ong
George Ogden
Arthur Conmy
Venue: ICLR 2024 poster (2024)
Keywords: Interpretability
Large Language Models
Natural Language Processing
Science of Deep Learning
Representation Learning
Primary area: visualization or interpretation of learned representations
TL;DR: We find "Successor Heads": a type of attention head that transforms numeric representations in language models. We find successor heads across language models of different sizes and find the abstract number representations that make them work.

Abstract: In this work we describe successor heads: attention heads that increment tokens with a natural ordering, such as numbers, months, and days.
For example, successor heads increment 'Monday' into 'Tuesday'.
We explain the successor head behavior with an approach rooted in mechanistic interpretability, the field that aims to explain how models complete tasks in human-understandable terms.
Existing research in this area has struggled to find recurring, mechanistically interpretable large language model (LLM) components beyond small toy models. Further, existing results have led to very little insight to explain the internals of the larger models that are used in practice.
In this paper, we analyze the behavior of successor heads in LLMs and find that they implement abstract representations that are common to different architectures. 
Successor heads form in LLMs with as few as 31 million parameters, and at least as many as 12 billion parameters, such as GPT-2, Pythia, and Llama-2.
We find a set of 'mod 10' features that underlie how successor heads increment in LLMs across different architectures and sizes.
We perform vector arithmetic with these features to edit head behavior and provide insights into numeric representations within LLMs. Additionally, we study the behavior of successor heads on natural language data, where we find that successor heads are important for achieving a low loss on examples involving succession, and also identify interpretable polysemanticity in a Pythia successor head.

Published as a conference paper at ICLR 2024
SUCCESSOR HEADS: RECURRING, INTERPRETABLE ATTENTION HEADS IN THE WILD
Rhys Gould', Euan Ong's George Ogden', Arthur Conmy?
University of Cambridge Independent
Correspondence to rg66 cam ac. uk
ABSTRACT
In this work describe successor heads: attention heads that increment tokens with a natural ordering, such as numbers, months, and days. For example, succes- sor heads increment "Monday" into "Tuesday'. We explain the successor head be- havior with an approach rooted in mechanistic interpretability, the field that aims explain how models complete tasks in human-understandable terms. Existing research in this area has struggled to find recurring, mechanistically interpretable large language model (LLM) components beyond small toy models. Further, ex- isting results have led to very little insight to explain the internals of the larger models that are used practice. In this paper, we analyze the behavior of succes- sor heads in LLMs and find that they implement abstract representations that are common to different architectures. Successor heads form in LLMs with as few 31 million parameters, and at least many as 12 billion parameters, such GPT-2, Pythia, and Llama-2. We find a set of 'mod 10' features I that underlie how successor heads increment in LLMs across different architectures and sizes. We perform vector arithmetic with these features to edit head behavior and provide insights into numeric representations within LLMs. Additionally, we study the behavior of successor heads on natural language data, where we find that succes- sor heads are important for achieving low loss on examples involving succession, and also identify interpretable polysemanticity in Pythia successor head.
INTRODUCTION
'Monday' ! 'Januay' 'First'
'Tuesday' February'
WE MLP,
Wov - WU
'Second'
'A'
'B'
'1'
'2'
Figure I: A successor head with OV matrix Wov takes a numbered token such as 'Monday" in embedding space and maps it to its successor value in unembedding space, e.g. 'Tuesday'. The circuit is the simple composition of the embedding matrix, the first MLP block, a single attention head, and the unembedding matrix.
Mechanistic interpretability (Olah 2022) is the process of reverse-engineering the algorithms that trained neural networks have learned. Recently, much attention has been paid to interpreting transformer-baeee large language models (LLMs), these models have demonstrated impressive capabilities (OpenAl, (202)) but there is little understanding of how these models produce their out- puts. Existing interpretability research includes comprehensive reverse-engineeriig efforts into toy models (Nanda et al.) 2023) and small language models (Olsson et al. 2022 Wang et al. 20233 been gained about how frontier L.MMs function.
though few insights
In this work, use 'feature' to mean interpretable (linear) direction in activation space, inspired by the second ppotential working definition' from Elhage al. 2022

---

Published as a conference paper at ICLR 2024
In mechanistic interpretability, universality (Olah et al. 2020 Li al. 2016) is a hypothesis that there are common representations in neural networks. The universality hypothesis asserts that neu- ral networks with different architectures and scales form common internal representations. Strong evidence for (or against) the universality hypothesis could significantly affect research priorities in interpretability. If common representations form across different language models and tasks, then research on small or toy language models (Elhage et al. 2022 2021) and narrow tasks (Wang et al.)
Heimersheim & Janiak 2023; Haana et al. 2023) may be the best way gain insights into LLM capabilities. Conversely, if the representations used by language models do not generalize to different model scales and/or tasks, then developing methods that can applied larger language models and don't rely on lessons from small models generalizing (such Wu et al. (2023). Bills al. 2023), Conmy et al. (2023) may be the most fruitful direction for interpretability.
2023
In this work, we find an interpretable set of attention heads we call successor heads in models of many different scales and architectures. Successor heads are attention heads that perform incremen- tation in language models. The imput to successor head is the representation of token in an ordinal sequence such as 'Monday', 'first', 'January', or 'one'. The output successor head assigns higher likelihood the incremented token, such as 'Tuesday', 'second, 'February', 'two', In our work, we find evidence for a weak form of universality (Chughtai et al. (2023); points and 2.) in finding Successor Heads across different models, well finding that numeric representations in language models are compositional (point 3.), as
1. Successor heads form across language models from the scale of 31 million parameters and to at least as many as 12 billion parameters.
2. Successor heads form across models with different architectures, including Pythia, GPT-2
and Llama-2 (Tourron et al. 2023).
3. Language models use compositional numeric representations encode the index of these tokens within their ordinal sequence; these representations exhibit abstract structure, such as mod-10 features.
Our contributions can be summarised as follows:
Introducing and interpreting successor heads (Section 23
(a) We introduce and explain successor heads, and show that they occur in language models across almost three orders of magnitude of model parameter count.
2. Finding abstract numeric representations in language models (Section
(b) We isolate common numeric subspace within embedding space, that for any given token (e.g. 'February') encodes the index of that token within its ordinal sequence (e.g. months).
(c) We find evidence for interpretable, abstract features within successor heads' numeric inputs: an unsupervised decomposition of token representations yields crucial set of features we call the mod-10 features (o.,J) Jn present all tokens whose numerical index = n (mod 10), e.g. fo is present in the model's representations of '2', '32', 2 '172'. 'February'. 'second' and 'twelve'.
(d) We steer the semantics of successor heads' numeric inputs with vector arithmetic.
3. Showing that the succession mechanism is important in the wild (Section 4
(d) Finally, show that successor heads play important role in incrementation-Dasee based tasks in natural language datasets for instance, predicting the next number a numbered list items.
SUCCESSOR HEADS
LLMSs are able to increment elements in an ordinal sequence, For instance, Pythia- 1.44 will com- plete the prompt "If this is 1, the next is" with "2", and the prompt "If this is January, the next is" with " "ebruary"" Given this observation, we find evidence for attention heads within LLMs (which we refer to as successor heads) responsible for performing this type of incrementation. To get evi- dence for successor heads require three definitions: i) the succession dataset of tasks involving abstract numeric representations. ii) an effective OV circuit to measure how attention heads affect model outputs, and finally iii) successor score.
2

---

Published as a conference paper at ICLR 2024
The succession dataset is the set of tokens across eight different tasks that can be incremented
Task
Tokens
Numbers
'T', '2' ..... '199', '200' 'one", 'two', 'nineteen', "twenty" 'first', 'second', ... 'tenth' 'Monday', "Tuesday'.. ... 'Sunday' 'Mon', 'Tue', ... 'Sun" 'January', 'February, 'December' 'Jan'. 'Feb" 'Dec' 'A', B 'B....,. 'Z'
Number words
Cardinal words
Days
Day prefixes
Months
Month prefixes
Letters
Table 1: Tokens the succession dataset
We also include different forms of these tokens, as language model tokenizers often have several tokens corresponding the same word (e.g. words with/without space at the start being different tokens). Full details of our dataset can be found in our open-sourced experiments
Notation. For consistency with prior work follow all Elhage et al. (2021's 's notation choices, though the following definitions are sufficient and self-contained for this work regardless. Trans- former language models use embedding matrix WE E IR (Rommte X.wcch to map tokens vectors in the residual stream (the cumulative sum of embeddings, attention heads and MLPs). After additive application of attention and MLP layers, the unembedding matrix We E R Xdamnal maps the final state of the residual stream to logits for all next token predictions. An OV matrix Wov E Redasss Xd maps the residual stream to the output of an attention head, assuming the tention head solely attended to that residual stream. Altogether, our diagram in Figure 1 shows one shallow path through a transformer model.
Effective OV Circuit. We determine whether attention heads perform succession by studying their effective OV circuit, which measures the direct effect of input tokens when multiplied by an OV matrix Wov, as in concurrent work (McDougall MLPO. The (non-effective) OV circuit WoWooll We (L) from Elhage et al. (202I) is the inspiration for effective OV circuit Wu WoWon Wor MLLP (W E))
al. 2023) which surveys the importance of
(2) columns represent input tokens to the head and the rows represent the logits on each possible output token.
Intuitively,
Successor Heads then operationalized by considering input token t from succession dataset (e.g. = Monday'). the effective OV circuit column for input has a larger output on the successor to ('Tuesday') than any other of the tokens in that task ("Monday' or 'Wednesday' or 'Thursday' or ...) then we consider the head to have performed succession in this case. Successor Heads are then defined as the attention heads that pass this for more than half of the tokens in the succession dataset. We call the proportion of succession dataset tokens on which an attention head performs succession the succession score. The succession scores across range of models are displayed in Figure
DECOMPOSING NUMERIC REPRESENTATIONS
Having found behavioral evidence that successor heads exist across range of models, we now perform à case study on the attention head (L12H0) with the maximal successor score in Pythia- 1.4B. Indeed, not only do we find that the representations outputs of the MLPo layer) share a common numeric subspace (Section Aj, that for any given token (c.g. 'February") encodes the index of that token within its ordinal sequence (c.g. months), but we also find mechanistic evidence for transferable arithmetic features within these representations
which the successor head acts (i.e. the
"The days and months tasks are special as the final tokens in these classes ( ('Sunday' and 'December') have cyclical successors ('Monday' and 'January'). ) We don't 't consider end tokens of the other tasks to have cyclical successors.
'Available soon https:::/gtttubb.coo///eeaannnno
ob/
ma iinn mein/exxppp htt/xppnumerrii www.nnuneen.cmm cearrricreeeeeetattteee ins www.oonel//oooooeeooooeemm.eee.. pyf
NNtte that also observe similar abstract numeric representations across other models too see Ap-
pendix B.3
3

---

Published as a conference paper at ICLR 2024
Successor Heads Language Models
Task performance of best Successor Heads
Model family:
Model family:
Propertion of tokens Incremented
# pythia
Pythia
0.8
lama
0.8
GPT-2
Score
Llama-2
0.5
0.4
Successor Head Cutoff
0.4
1. Numbers
2. Number words
3. Cardinal Cardinalwwrdd words
A. Days
Day prefizes
Go Months
7. Month prefiees
8. Letters
1M
10M
100M
1B
108
Number model parameters
Task
Figure 2: Plots of successor scores (proportion of tokens where succession occurs) for each model tested. A plot of the highest successor score observed across all attention heads for each model tested (left) and successor scores of the best successor heads in models (Pythia-1.4B, GPT-2 XL, Llama-2 7B) across different tasks (right).
Specifically, we use sparse autoencoders to isolate abstract 'mod-10" features within the outputs of the MLP, layer (Section 3.I), provide further evidence for these features through linear probing and ablative experiments on individual neurons (Section 3.2 and use these features to steer model
behavior across different successor tasks (Section 3.3).
3.1 FINDING MOD- 10 FEATURES
To uncover more structure within these MLP ML.p-rrepresentation, we train a sparse autoencoder (SAE) (Ngl (2011); CCunningham et al. (2023); Bricken et al.] (2023), see Appendix [B) on tokens & from range of ordinal sequences. In short, SABS find sparse set of linear features that can reconstruct activations in neural networks. We apply SAEs to the rerperrrprssnttaiion of all tokens I and call this their SAE-decomposition The ith SAE feature has activation (on input token t) (t( 2 0.
Given ordinal sequence token t and trained SAE, define t's most important feature t* as the SAE feature that, when ablated from the reconstruction of MLP MLPorrepresentation causes the biggest decrease in the probability of the successor of t (by calculating probabilities from the logits obtained by multiplying by We Wov).
For numeric tokens t, (e.g. f13 '13'), we find that their most important feature is usually shared by other numeric tokens (m for which m = n mod 10 (e.g. (3, (23...,, loa). Indeed, for each of the 'mod-10 10 classes' (lo, ..., ) (l1,...,99) ...., (19.-.., (9 the modal most important feature for tokens in that class is shared on average by 58.5% of tokens in that class. Moreover, we find that the most important feature of t; typically only has a high activation a; (t²) in the SAE-decomposition of t, i j mod 10, which we visualise as 'mod-10 bands' in Figure
We also observe that these features are causally important for the successor head perform incre- mentation: if we apply the successor head to the most important features t of tokens t, (i.e. com- pute logits as We Wovt?), the resulting distribution places high weight on tokens ty for j i mod 10, which we visualise 'mod-10 10 bands' in Figure 5 Furthermore, the weight placed on tokens ly when 4; a single-digit number is much larger than that when I, a double-digit number.
Given these observations, we hypothesise that the most important features of numeric tokens t; might in some way encode the value of mod 10. As such, we define the mod-10 features fo. .... fo as the modal most important features from mod- 10 classes {fo..., (..,,,,)) ), {f,,.., ) (19, ..., (90) averaged over 100 SAE training runs. We verify that these mod-10 features are causally important for incrementation, by observing that the logit distribution obtained by applying the successor head to features f. (i.e. Wi Wovft fa) places high weight on tokens t, for j = i mod 10 (which we visualise as 'mod-10 10 bands' in Figure 4).
4

---

Published as a conference paper at ICLR 2024
3.2 TRANSFERABILITY OF MOD- 10 FEATURES
Are the mod-10 10 features we found in Section 3.1 simply an artifact of the SAE technique? We pro- vide evidence that these are natural, causally important features by using two independent methods to recover them: (1) linear probing, and (2) identifying MLPo neurons. We also demonstrate that these features transfer to other tasks in the succession dataset (Section
(1) Linear probing. We train a linear probe P : IR 10xdaaaa to predict the value of i mod 10 from the MeLee-eeprssnntation of numeric tokens t,. We find that P., the ith row of our linear probe, has a high cosine similarity (on average 0.70764) to the corresponding mod-10 feature fi obtained from the SAF. Surprisingly, the probe even generalizes non-numeric tokens, correctly predicting the index value mod 10 for 94/102 examples from succession dataset tasks 2-8 (Section (2) Our full experimental setup is described in Appendix C.I Additionally, we provide an analysis of linear probes for moduli other than 10 in Appendix R
(2) MLP, neurons. We perform ablative experiments on individual MLP, neurons (activations im- mediately after the final ReLU/GELU) to find the most important neurons for incrementing numeric tokens t (measured by decrease probability of the successor token 1,+1, as per the definition of most important feature). Observing the behavior of these neurons on tokens i, reveals periodic spik- ing patterns in firing intensity (the neuron's s activation value), with the most common period across the top 16 most important neurons being 10. Figure 22 presents the firing patterns of some of these neurons. Indeed, we also find that the neurons increase probability on successor tokens by multi- plying the neuron's 's corresponding direction with Wy Woy in the same figures. Further technical details can be found in Appendix D Our results on the interpretability of individual neurons may seem surprising in light of recent work suggesting that the individual neurons of language models may be inappropriate as the units of understanding (Elhage et al. 2022). However, our results do not contradict previous findings that understanding MLPs requires understanding distributed behav- iors, since, for example, feature fc appears to be in superposition across at least two neurons (see Figure 22).
3.3 VECTOR ARITHMETIC WITH MOD- 10 FEATURES
The generalization of our mod-10 10 linear probe to unseen numeric tasks suggests that token repre- sentations across different tasks might be compositional and share some common mod-10 struc- ture. In this section, we test our understanding of this structure by performing vector arithmetic with our mod- 10 features to manipulate the index of ordinal sequence tokens. For instance, just as Mikolov et al.) (2013) found that work vectors satisfied vec(" (xing")-rees (ree("Man")-aee vec(" Woman') = vee(" Que'en' ), we expect MLPo(WE g( ('fifth')) - kfs + kfr (3)to to be causally used by the model in a similar way to (W ("seventh") ),, where & is scaling factor (Appendix B
We use our successor head to test this hypothesis. (LeooWWe("svenn
Observe that, if (3) behaves like
applying the successor head to (3) (i.e. multiplying by Wu Woy) should yield a distribution with more weight on "eighth" than on any other token from the cardinal-word
task in the succession dataset (Section 2 Indeed, this is correct as indicated by the circled check- We can perform a similar experiment with all tokens in the succession dataset and with features other than fr added. The cases where the max logits are on the successor token are check-marker also display how logits are distributed across tokens for individual cells in Appendix Q We find that when the mod 10 addition feature is larger than the source value (modulo 10), vector arithmetic works on 53% (for months) and 89% (for digits 20-29) of cases.
mark @ in Figure 6
in Figure (6] We describe the experiment in more detail in Appendix E and we
than bias. The vector arithmetic experiments (Figure 6 work much worse when the mod- 10 addition is smaller than the source tokens's ordinal sequence position mod 10 (e.g. experiments involving 'March' and adding fi or f2 do not go well). This is because Successor Heads are bi- ased towards values greater than the successor, compared to values less than the successor. This effect can be seen in Figure 7a on the tokens 'first', 'second', .... 'tenth'. However, our mod-10 10 features do not exhibit a greater- bias, as seen in Figure 7b] We survey these effects across all tasks in Appendix ! As result, using the mod-10 features to shift logits towards tokens of
lower order than the input token fails, as the changes in logits are not significant compared to the large logits higher-order tokens. In the case of numbers, this leads to the effect that, for exam-
5

---

Published as a conference paper at ICLR 2024
ple, Wo Wov (MLPo(W2 (We((3))) - kfs kifs) has high logits '43', rather than '33' (this '+10' effect occurs for 2/3 entries below the diagonal the 20-29 numbers table of Figure 6
Feature activations the most important SAE features across numbers
Logit distributions the most important SAE features across numbers
20
80
Feature source
Feature
40
40
-10
0
80
80
20
60
Output number
Output number
Figure 3: The activations of 4ss most important feature (g-axis) in the SAE-dccomposition of t, (r-axis), for t4, ty numeric tokens. Values aver- aged over 100 SAE training runs.
Figure 4: The logit value for lj (z-axis) when unembedding the most important feature t, (y-axis), for t, tú numeric tokens. Values aver- aged over 100 SAE training runs.
mod-10 feature
Logit distribution of each mod-10 feature across numbers
-10
20
40
60
80
Output number
Figure 5: Logit distribution WoWovv. for each mod-10 feature St.
The absence of strong greater-thal than bias in mod 10 features suggests this feature- level description missing some details specifically, that successor heads must use other numeric information to produce the greater-than bias observe. Additionally, while alization of the mod 10 features across various tasks in the table in Figure the mod 10 features are not able steer the Days and Letters tasks from Section 2 We describe this in Appendix
Limitations:
good gener-
SUCCESSOR HEADS IN THE WILD
In this section, analyze the behaviour of successor heads within natural-language datasets, and observe that they aren't i simply responsible for incrementation: indeed, we identify four distinct, interpretable categories of successor head behavior, highlighting successor heads as an example of an interpretably polysemantic attention head 'in the wild'.
In order to characterize the behavior of Pythia- 1.4B's successor head on natural-language data, randomly sample 128 length-512 contexts from The Pile, and for each prefix of each context, we assess whether the successor head important for the model's ability to predict the correct next token. To measure importance, output of a head with its mean output over a chosen distribution (in this case, the same batch), and, the very end of the model, subtracting this mean output and adding the original head output to the residual stream (other effects and ablation methods are explored in Appendix M). We evaluate prefixes using two different metrics for per-pon per-prompt successor head importance:
use direct effect mean ablation, which involves patching the
Winning cases. We identify prefixes where the head that most decreases the logit for the correct next token under direct effect mean ablation is the successor head, denoting them winning cases.

---

Published as a conference paper at ICLR 2024
2
3
6
7
0
4
5
6
7
8
9
Janutyy
Fehnury
Apáil
x
v
May
25
July
Augue
Septeibeer
second
third
x
fourth
d
siuth
x
6lam
seventh
x
estan
righth
seventenn
i
Example: The circled case (3) has a source token "fifth' and target residue of 7, and it is ticked because
tweyy
vector arithmetic leads to this token heing successfully treated like 'seventh')
Figure 6: Table displaying cases in which vector arithmetic such as (3) is successful for various ranges of tokens. Other ranges of tokens give similar results, displayed in Appendix Rows: source tokens. Columns: target residues modulo 10.
Logits
Logits
rinth
Source token
mod 10 feature
soth
founh
133
puDos
Spit
mmn
TEAL
Output token
(a) Evaluating the effective OV circuit on the input and output tokens 'first', 'second', "tenth".
Output token
(b) Multiplying all mod 10 features fs by Wow.v
Figure 7: The Successor Head OV circuit displays clear bias against decrementation (Figure 7a i.e. the logits on or above the main diagonal are less than the logits below the main diagonal. This bias isn't captured in the mod 10 feature (Figure 7b).
Loss-reducing cases. We identify prefixes where direct effect mean ablation of the successor head increases next-token prediction loss (by AC(P)), denoting them as loss-reducing cases.
4.1 INTERPRETABLE POLYSEMANTICITY IN SUCCESSOR HEADS
On analyzing prefixes where the successor head is particularly important for next-token prediction observe four main categories of behavior, which
i.c. loss-reducing and winning
operationalize as follows (denoting a top-n-attended token as a token at one of the top n positions to which the successor head attends most strongly):

---

Published as a conference paper at ICLR 2024
Successorship behavior: the successor head pushes for the successor of token in the context. We say this behavior occurs when one of the top-5-attended tokens is the successorship dataset, and the conrect next token is the successor of t.
Acronym behavior: the successor head pushes for an acronym of words in the context. We say this behavior occurs when the correct next token is acronym whose last letter corresponds the first letter of the top-1-attender token. (For example, the successor head attends most strongly to the token 'Defense.. and the correct next token is 'OSD'.)
Copying behavior: the successor head pushes for a previous token in the context. We say this behavior occurs when the correct token has already occurred in the prompt, and token is one of the -op-5-attended tokens.
the successor head pushes for a token greater than a previous token in the context. We say this behavior occurs when we do not observe successorship behavior, but when the correct next token is still part of an ordinal sequence and has greater order than some top-5-ate oo---dete token (e.g. g if the successor head attends to the token 'first' and the model predicts the token 'third").
Greater-than behavior:
Proportions of behaviours in winning cases
Prompt
Completion
(...) B2] Hence, bonding to ceramic requires strict attention to detail for op- timal clinical outcomes. [@ B
Acronym
(...)
designated boxazomycin. A and
Successorship
Greater 4bbrn 18,9e
(...) called Generalized Single Step
GS
Single Solve (
(...) More than
thirds
where one or more access points (
AP
14%
Figure 8: Proportions of three dominant behaviors across winning cases.
Figure 9: Random sample of 5 winning cases. Negli- gibly many winning cases were copying.
Proportions behaviours reducing
Prompt
Completion
low-dose administration of 14- and in the second round, let's get it to the
15
(...)
third
20. 7%
Successorship
29h
(...) for 1) investigators to ask the mis- sionaries.
Copying 11.79
(...) You using Microsoft Test Man-
MT
Acronym
10.00 8i
ager
(...) known as the Medical Research
MR
Proportions of reduced loss (AC) attributable to prompts of each be- haviour.
Figure 10:
Council Technology (
Figure 11: The top 5 most loss reducing examples.
We plot the proportions of each behavior observed across winning cases in Figure and the fraction of total reduced loss over all contexts (AC) attributable to contexts of each behavior in Figure I0 We also illustrate a random sample of winning cases in Figure 9 and of 5 loss-reducing cases in Figure 11 We observe that, while successorship is the predominant behavior across both winning and loss-reducing cases, acronym and greater-than behaviors also form a non-negligible fraction of successor head behavior. In other words, the successor head is example of attention head with interpretable polysemanticity5 While polysemanticity has been observed in both vision models (Olah et al. 2020) and toy models trained to perform simple tasks (Elhage et al. 2022), the best of our knowledge the presence of both successorship and acronym behavior in head L12HO is the cleanest example of polysemantic behavior identified so far in an LLM, where we show two clear distinct behaviors one model component has. Finally, this finding is surprising given research into polysemanticity and superposition (Elhage et al.) 2202, [Bricken et al.) (023). The succession and acronym behaviors are different tasks that L12HO completes, yet they are not independent tasks
A component of network is said to he interpretably polysemantic if performs multiple distinct, inter-
pretable functions.
8

---

Published as a conference paper at ICLR 2024
occur in completely different contexts (this is because token completion could involve both suc- cession and an acronym, e.g 'the First Limited Corporation, could be completed with Secoond' or FFCC).
Note that in this section, while we identified that successor heads are often used in tasks involving incrementation, we did not explicitly demonstrate that successor heads necessary for incremen- tation. In Appendix we describe an experiment that reveals that successor heads are necessary for specific incrementation task (numbered listing).
RELATED WORK
Mechanistic Interpretability research aims to reverse engineer trained neural networks analo- gously to how software binaries can be reverse-engineered (Olah] 2022). This research was largely developed in vision models (Bau et al.
2017 Olah et al., 2017) though most recent research has
studied language models (Elhage et al.
2021 Olsson et al. [202 Gurnee et al.) (2023) and trans-
formers (Nanda et al. 2023). Olah et al. (2020) introduces the universality hypothesis and we use
Chughtai et al. (2023) 'weak universality notion in this work (Section
Transformer circuits. More specifically, our work builds from the insights of Elhage et al. (021) framework for understanding circuits in transformers, including how autoregressive transtormers have a residual stream. Due to the residual stream, different paths from input to output can bypass as many attention heads and MLPs as necessary. This has further been explored in specific case
studies (Wang et al. 2023 Goldowsky- Dill et al. (202)) and generalizes to backwards passes (Du jet al. (202)) One related case study to work is Hanna et al. (2023) which studies Greater-Thie circuit in GPT-2 Small, similar to how we indirectly found the Greater-Thha operation in Section 3
Hanna al. (023) focus mainly
numbers, not other tasks. Our work is inspired by Olsson et al. (2022) who study induction heads and find that heads with similar attention patterns exist in larger models. In our work we provide an end-to-end explanation of generalizing language model
components (Figure D. though induction heads are related to in-context learning and a consistent phase change in training, which we didn't observe for successor heads (Appendix H).
LLMs and vector arithmetic. MMkolovv et al. (2013)'s 's seminal work on word embedding arith- metic showed that latent language model representations had compositionality. e.g. vec( King") vec("Man) + vec( (eee"a"""") x vee("Queen)) ). Recently Merullo et al. (2023) showed some exten- sion of these arithmetic results to L.Mss. (203)) found that 'one GPT-2 Small boost successors numbers, months and days, which we generalize to more architec- tures and to an end-to-end circuit (Figure T}. Lan & Barez (2023) also use an automated approach to study the overlap of these tasks. Finally, Subramani et al. 2022); Li et al. (2023) and Turner et al. (2023) use residual stream additions to steer models. Our work differs in that considers shallow targeted paths through networks, rather than deep hidden states in networks.
and that similar heads in
6 CONCLUSION
In this work, discovered and interpreted class of attention heads we call successor heads. We showed that these heads increment tokens in ordinal sequences like numbers, months, and days, and that the representations of the tokens are compositional and contain interpretable, abstract 'mod- 10' features. We provided evidence that successor heads exhibit a weak form of universality, arising in models across different architectures and scales (from 31M to 12B parameters), and using similar underlying mod-10 10 features in all cases. Finally, we validated our understanding by demonstrating that a successor head reduced the loss on training data by predicting successor tokens.
Additional numeric representation findings relevant to future work include:
1. Finding a "greater-than bias', where language model was much more likely to predict numeric answers larger than the values in the prompt, compared to smaller values than tokens present the prompt, that was observable by a leights-level analysis.
2. Surprisingly interpretable individual MLPO neurons on this narrow task.
3. A novel example of attention head polysemanticity (successor heads predicting acronyms).

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENTS
We would like to thank Bilal Chughtai, Théo Wang and reviewers for comments on a draft of this work and Neel Nanda for helpful discussion, as well as Lawrence Chan and Sebastian Farquhar for pieces of advice. Elizabeth Ho, Will Harpur-Davies and Andy Zhou worked on an early version of this work in GPT-2 Small with help from Théo Wang.
Contributions from each author: Rhys Gould first found successor heads GPT-2 Small and Pythia- 1.4B, and identified the mechanism from Figure as well as the mod-10 features. Euan Ong de- veloped Appendix [ improved our understanding. and worked on writing across the paper. George Ogden found successor heads in larger models (e.g Figure Arthur Conmy led the project, framed the paper's s contributions and suggested and implemented several experiments throughout the work.
REFERENCES
David Bau, Bolci Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations, 2017.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardr. Eliciting latent predictions from transformers with the tuned
lens, 2023. 18
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023. 18
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. attps://opeeaipublicc wlw.licccblbb.o......... www.wws.nee......
neuronnexplainer/ wwwpeeewepaaeeeeeeeeeeeeeeeeeeee htwpee/eeeeeee..appppaepppaaaeee index html 2023. 2
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https:/ttrassfommer circuits. htttsssuu/b/22//o//mmmmee hetpreemmantttffttttteenneeeeen
Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineer- ing how networks learn group operations, 2023. URL httpps:/aaxx....rr///bb//////
03025
2 9
Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià
Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. 2023.
Stanford CRFM. Mistral: A framework for transparent and accessible large-scale language model
training. https: //tp::///iii http://gtttttttt wtt.c/oo//sss/eeeeeeeeee /tt.fforadorroooooooo condddrrfmmmaei ttffm/mmitraa stall] 2021.
[8]
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models, 2023. 13
14
Kevin Du, Lucas Torroba Hennigen, Niklas Stoehr, Alexander Warstadt, and Ryan Cotterell. Gen- eralizing backpropagation for gradient-bused interpretabilityy 2023.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Gan- guli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. URL
cirumttrrrrrrrttteeeee www.eeittttttttteuue.eeeouetettt httpub///2112 Erweeworkkooorroo winein..xx.. html
10

---

Published as a conference paper at ICLR 2024
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dowd Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposi- tion. arXiv preprint arXh::2209.10633 2022.
Nicholas Goldowsky-Dill Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching, 2023.
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsi- mas. Finding neurons in a haystack: Case studies with sparse probing, 2023. g
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: In- terpreting mathematical abilities in pre-trained language model, 2023.
Stefan strings in a 4-layer attention-only transformer,
Heimersheim and Jett Janiak.
A eircuit for
Python doc-
2023.
URL https:
/ www.
a
Mikhail I.. One is 1: Analyzing activations of numerical words vs digits 1, Jul 2023.
Michael Lan and Fazl Barez. Locating cross-task sequence continuation circuits in transformers, 2023. 0
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time
intervention: Eliciting truthful answers from language model, 2023. |
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hoperoft. Convergent learning: Do
different neural networks learn the same representations?, 2016. 2
Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy sup- pression: Comprehensively understanding an attention head. arXiv preprint, 2023. B
Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra
effect: Emergent self-repair in language model computations, 2023. 21
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple word2vec-
style vector arithmetic, 2023. 9
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representa-
tions of words and phrases and their compositionality, 2013.
Neel Nanda, Lawrence Chan, Tom Lieberum. Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learn- ing Representations, 2023. URL /tllittts////ppeevviieee het.eewwwwwwwweeeeeeeeeemeeeeeee
wttp:e/wwwweeeeeeeoounneeeeee. forum 9X PmDP
Andrew Ng. Sparse autoencoder. Online Course Notes, Stanford University, 2011. URL https:
pdf
Chris Olah. Mechanistic interpretability, variables, and the importance of interpretable bases.
doi: http://wwww..aaaoooooeeeeaaaaaaa
2022.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Featture visualization. Distill, 2017.
visuuuuiirriiiuuaaitiooooo 9
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: introduction to circuits. Distill, 2020. doi: 10.23915/disstil..000
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma. Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, al. In-context leam- ing and induction heads, 2022. URL /ttps::/rranssoomeeee ott.aaaaoo...arroorrooooooo////2 in-context In-coontext-learniie
wearnlannnnn i induct hewattiiineeeeeeneenneee html

---

Published as a conference paper at ICLR 2024
OpenAl. Gpt-4 technical report, 2023.
Judea Pearl. Causality, Cambridge University Press, 2 edition, 2009. ISBN 978-0-521-89560-6
doi: 10.1017/C80978051180011 21
Lee
Sharkey, taking
Dan Braun, and Beren Millidge. finterim research re- features out of superposition with sparse autoencoders. 2022.
port]
URL
https: /ww.
interime
Nishant Subramani, Nivedita Suresh, and Matthew E. Peters. Extracting latent steering vectors from pretrained language models, 2022. 9
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. 2
Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDi- armid. Activation addition: Steering language models without optimization, 2023.
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt.
Interpretability in the wild: circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https:
//openrevi iew.wwwww.. net. form/f www.:mmm..ddNNsssssSsess.
Zhengxuan Wu, Atticus Geiger. Christopher Potts, and Noah D. Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca, 2023.
A ORDINAL SEQUENCES ARE REPRESENTED COMPOSITIONALLY
Let i, denote the ith token in ordinal sequence s (such that e.g. MMnnth corresponds to the token 'February'), and let [i. token i, (the output of MLP, in Figure
(We(t)) denote the model's internal MrPPrrrepresentationn of
Given successor heads S Woy can increment tokens s; from a range of ordinal sequences s (c.g. numbers, months, days of the week), one might hypothesise that the repperreprssnttttionn of such tokens have compositional structure i.e. that information about token's position in its ordinal sequence is encoded independently from information about which ordinal sequence it comes from. More precisely, we claim that we can decompose representations [i..] into features v, living in some 'index space" and v, living in some 'domain space', such that [i.] v, + V..
Method. To test this compositionality hypothesis, we wish to learn two linear maps an index-space projection TN dudke + Rdust and domain-space projection TD : Bade + Russ such that, for all pairs of tokens i, and je (with i, a valid token), [ := ([a.l) + ([.l)) [[.]]] To do so, we enforce that лH + TD = T, and ensure predicted representations [[]]]] 'hehave like' ground truth representations [i.] for randomly sampled pairs of tokens i, and j. other words, that there is low disttane between [[ and [[.]]. that [&,]] decodes to i, (output-space decoding), and that ((E.]) decodes to (i 1), (successor decoding). For full experimental details see Appendix N
12

---

Published as a conference paper at ICLR 2024
Source index token (dkw)
III
IV
VI
VII
ViIl
Ix
X
XI
XiI
1
9
22
9
24
11
12
one
nine
two
Uhree
Tour
twenty
six
seven
eight
nine
twenty
eleven
tweive
first
ninth
fourth
second
seventh
eighth
ninth
Monday
Sunday
Tuesday
Wedne...
Thursday
Tuesday
Saturday
Sunday
Sequence
Mon
Sop
Tue
Wed
Thu
Tue
Sat
Sun
January
Septem...
Febnuary
March
April
February
June
July
August
Septom
Decam...
Novem
Decem
Jan
Sep
Feb
Mar
Apr
Feb
Jun
Aug
Sep
Dec
Nov
Dec
Table 2: A table presenting top-1 predicted tokens under output-space decoding from (1,) a) + TH TH (dRom). Green cells denote predictions which match their target exactly; red cells denote incorrect predictions, Dashed cells denote pairs of 1, and Rom for which is is not valid (single) token.
Results. On our held-out dataset of token pairs, we obtained top-1 output-space decoding accuracy of 1.00. To explore -utttf-distributionn performance, we also test whether ay can project out the numeric component of Roman numerals (which weren't in the successor dataset), by taking Roman numerals fRom E ,.., 'XII' whether (1,))) + Ty. (fRom) decodes to ig. We present the top-1 predicted tokens under output- space decoding in T observe that we obtain perfect top-1 accuracy (apart from E {1, 10), which attribute to the Roman numerals I, and X being single-letter and impossible to disambiguate from 9etter 22 and 241ene))
} and tokens 1, from sequences in the successor dataset, and testing
These results in particular, ability project the numeric component out of tokens from unseen sequences and transfer indices across domains suggest that there is a shared numeric subspace storing the index of token within its ordinal sequence. Indeed, informal testing suggests that this numeric subspace may be interpretable even for tokens not part of an ordinal sequence: for instance, ((I triangle']] + xD(Num)) yields 3yum- and d( ([" week + π (1Num))) yields N
We note in Appendix O however, that applying the successor head to these learned representations did not always preserve performance (i.e. for constructed representation [. S([&)) did not always decode to (i + 1):). This suggests our numeric projection ты might be capturing slightly more than just the numeric subspace: specifically, there may be some components of domain-space which are ignored by output-space decoding, but which our successor head lifts into output-space.
B SPARSE AUTO-ENCODERS
B. DEFINITION
We refer single-layer autoencoder with a sparsity regularization term in its loss sparse auto-encoder.
For dataset generated from set of underlying vectors (each dataset example is sparse linear combination of such vectors), it has been empirically observed (Sharkey et al. 2022 Cunningham al. 2023) that sparse auto-encoders are capable retrieving the underlying set of vectors. We hope to obtain a set of sparse, interpretable features from the SAEs that decompose some of the structure of MLP, space that we can use analyze the way numeric operations are performed.
B.2 TRAINING PROCESS FOR MOD 10 FEATURES
Training a sparse auto-encoder with D features and regularization coefficient À on a dataset of tokens
in MLP, space results in map F: Tokens - (R& xR_ with F(x) {(Vi, ((vi,)))) ..),...,. (VD,ap)), mapping a token to set of feature and activvatativ pairs, with reconstruction R(z) Eb a,v, Each v, the ReLU of linear transformation with the input to the SAE, represented by W. in Figure 12 Note that we use SAEs in MLP, space, i.e. the reconstruction loss is at the middle stage of Figure IaareeTThhh which we have illustrated in Figure [2]
We train the SAE using number tokens from to 500, both with and without a space ('123' and 123'), alongside other tasks, such as number words, cardinal words, days, months, etc. 90% of
13

---

Published as a conference paper at ICLR 2024
W
ReLU
{a1v1, ..., Vn
WE E
MLP 0
Wov
WU U
Figure 12: SAEs are trained on the activation after MLP, and before Wov.
these tokens go into the train set, and the remaining 10% the validation set. Even with the other tasks, the dataset is dominated by numbers, but creating a more balanced dataset would give us less data work with, and without enough data, the SAE fails generalize the validation set. Hence, we only concern ourselves with the features that the SAE learns for number tokens, and we then separately check whether these features generalize to the other tasks on the basis of logits, rather than SAE activations.
We used the hyperparameters D 512 and X 0.3, with a batch size of 64, and trained for 100 epochs. To find these hyperparameters, we used the metric of mean max cosine similarity between two trained SAEs, as described (Sharkyy et al. (2022) and Cunningham al. (2023).
B.3 UNIVERSALITY MOD-10 10 RESULTS
We also observe the mod 10 structure via SAEs across models other than Pythia-1.4B, without any finetuning SAE parameters to these models. We reproduce the SAE figures seen Section 3.I for other models, with Appendix B.3 for Pythia-2.8B, and Appendix B.3 for (edebrrmboorrggt--meddu x81.
C TEST SET EVALUATION
C.I LINEAR PROBING
We train linear probe predict the mod 10 value of tokens. Specifically, we train on number tokens from '0" to '500', both with and without space, assigning 90% of tokens to train set, and the remaining 10% to validation set. We use learning rate of 0.001, and batch size 32, for 100 epochs.
We then evaluate on dataset of unseen tasks, including number words (from 'one' to 'nineteen' ) placements, Roman numerals, months, days, and any valid spaced and capitalized variants. Out of the total 102 such examples, 94/102 are correct, and the 8 failures are: ['January', 'December', 'Friday', 'Saturday', 'Sunday', V', X", XV'].
The failures of out of 7 days are consistent with our inability to interpret the day task well with mod 10 features. Additionally, we 'January' and 'December' failure cases, which is also consistent with our finding that there does not seem to be a mod 10 feature that corresponds to any of them: fi behaves 'November' rather than 'January', and as 'February'.
14

---

Published as a conference paper at ICLR 2024
Feature activations for the most important SAE features across numbers (EleutherAliyythia-222c
Logit distributions for the most important SAE features across numbers (EleutherAlipythia 8b)
12
20
80
10
15
Feature source
Feature source
60
60
5
20
20
-10
40
60
80
0
20
60
Input number
Input number
(a) SAE feature activations
(b) Feature logit distribution
Logit distribution each mod-10 feature across numbers (EleutherA//pythaa22.88)
mod-10 feature
-10
20
60
80
Input number
Logit distribution for each mod- 10 feature
Figure 13: SAE plots for Pythia- 2.88 analogous Figure
Figure and Figure
15

---

Published as a conference paper at ICLR 2024
Feature activations for the most important SAE features numbers
Logit distributions for the most important SAE features across numbers (celebrimbor (celebrinbor-gp1--eeliimm (pd--gpmddmmmxxx11
gellebrimbor-gp12-mmeduu
1.5
Feature source
60
60
6
Feature
1
40
2
0.5
20
0
Input number
Input number
(a) SAE feature activations
(b) Feature logit distribution
Logit distribution of each mod-10 feature across numbers seabbrimborggtt--meddm-
mod-10 feature
10
20
40
60
80
Input number
(c) Logit distribution each mod- 10 feature
Figure 14: SAE plots for celebrimbor-gple -p22-medium--x8 analogous to Figure
Figure
and Fig-
ure
16

---

Published as a conference paper at ICLR 2024
Roman Numeral Successor Score
Proportion of tokens incremented
Model family:
In this work, used all tokens in numeric sequences in the models' vocabularies, except Roman numerals, so can these
* gpt2
EleutherAl/pythia
0.8
Ilama
test set, as in Appendix [C.]] and Ap- pendix N We tested all the OpenAl GPT-2 models as well as the Pythia models with at least 1B parameters. We find that Successor Heads have variable performance on this held out task, with many (including all Pythia models) achieving a high succession score. However, the original Llama-7B does not generalize well this task.
0.6
0.4
0.2
0
Figure 15: (Left) Succession scores for the Roman numer- als task only.
1M
10M
100M
1B 10B
Number of model parameters
C.2 HELD-OUT TASK
D MLP, NEURONS
In our MLP, neuron experiments, we do the following: for each T E '''','',..... '99'), we ablate each neuron from the final activation in MLP, (the final activation is just before the final linear layer of MLP,), and store the probability attributed to the successor of T after passing the modified (due ablation) MLP, output through the successor.
Averaging the correct probabilities across all 100 prompts then gives an averaged correct probabil- ity for each neuron after ablation. We then look at the intensities (neuron activation values) and logits across inputs of number tokens for neurons with the lowest correct probability after ablation, meaning they have the most impact on successorship when ablated. This gives us the plots seen in Figure 22
E ARITHMETIC EXPERIMENTS
For token é (row of arithmetic table) with order ord(t), and mod-10 feature fi (column of arithmetic table), we consider how x = MLPo(We(() + k(-fa + fi) attributes logits to tokens within the task I, with & E R, scaling. We denote whether correctly attributes maximal logits to the successor token 1+ of (defined by the property that ord(t+) = n + 1) by a checkmark, as seen in Figure 6 Since the mod-10 features {fa} obtained from the SAE are normalized unit norm, the scaling & is necessary in order to modify the order of numeric token. For example, though fi may be present in the MLP, embedding of eleven', we not know the intensity of the feature (analogous to SAE feature strengths {o,(t)):) in the embedding. We use the heuristic of a scaling & : A(MLPo (W) fn E R for some À R.. The appropriate should be such that performing the arithmetic described by æ has effect the embedding's numeric order (i.e. À large enough) while not corrupting the task identity of the original embedding (i.e. not too large), which is checked by multiplying by Wov and observing whether the top token has these two properties (that is, whether the numeric order of the token has been altered while maintaining the same task identity of, say, month). By checking this criterion for range À
17

---

Published as a conference paper at ICLR 2024
Loyat
Loges
Logis
Logis
mod 10 frature
Sourcee pronet
mod 30 feature
prompn
Prompts
Prompis
Prompes
(a) Output logits for suc- cessor heads on day tokens (cf. Fig. 7a
(b) Presence of mod-10 10 (c) Output logits for suc- (d) Presence of mod-10
Features Day Token cessor heads on letter to-
features on letter tokens
Predictions (cf. Fig. 7b) kens (cf. Fig. 7a
(cf. Fig. 7b)
Figure 16: Comparative analysis of output logits for day and letter tasks and presence of mod-10 10
features. This complements the main text analysis in Figures 7a and 7b limitations of mod- features across certain tasks successor heads perform.
{0.0, .... 0.75, 1.0, 1.25, .... 3.0). find that À achieves this criterion for all tasks other than months, where months instead have À =
F FAILURE CASES OF MOD 10 FEATURES
For the day and alphahet task, analogously Figure 7 we look the logits across the task and the mod 10 features. These are displayed in Figure 16, and demonstrate that our mod 10 features are not very interpretable the context of days and the alphabet in terms of logits, with no clear diagonal of high logits.
G RESIDUAL CONNECTIONS ARE NOT IMPORTANT FOR SUCCESSION
To show that there is no relevant information in the residual stream, i.c. the path WoMLP (M))) is sufficient to predict successors, we perform an experiment using the Tuned Lens (Belrose al. 2023). which approximates the optimal predictions after given layer inside a transformer.
For all tasks in the succession dataset (Section between tokens)
we used prompt formats (where denotes a gap
1. Herel isl l l alphal betal gammal and here! is!
another <tokenl>|
2. | The | Monday /Tuesday wednesdayl andl The <tokenl>|
in order to measure how well models were able predict the successor <taken2> (e.g 'February') given the final token of the prompt was <tcken1> (e.g 'January'), as LLMs, predict successors given these prompts.
We then took GPT-2 Small and Pythia-1.4B3 output after MLP, and used Tuned Lens to get logits on output tokens | The resulting successor score <1% and commonly predicted bigrams, such as <token1>>>> first" giving time" as a completion and <tokeni>=> Sunday' giving morning' prediction. This suggests that MLP, information insufficient for incrementation and the successor head is critical for succession.
H TESTING SUCCESSOR SCORE OVER TRAINING STEPS
Another line of evidence that Successor Heads are an important model component for low training loss be found by studying successor scores across training points. We study a Pythia model (Bi-)
derman et al. 2023) as well as a Stanford GPT model (CRFM 2021). as these models have training checkpoints. The emergence of Successor Heads throughout training is displayed in Figure 17
'Note we did run with GPT-2 Small's attention layer 0 maximise the model's chances being able to perform succession. Pythia- 1.4B has parallel attention we just take the MI.P) output this case.
18

---

Published as a conference paper at ICLR 2024
Successor scores across checkpaints in Pythia models
Best Best sucoesssr Store
pythia-70m
Best succeeson store
stanlard gp22
Pythie-1.44 1.4b
0.7
Pythia-2.88
0.6
0.6
0.5
0.4
0.2
0.1
100K
200k
300k
400k
Checkpoints
Checkpoints
(a) Pythia models
(b) Stanford GPT-2 Medium-C
Figure 17: Best successor scores across successor heads throughout training checkpoints for Pythia and Wanford-gpt2 models.
Looks
Logis
Logits
Sourte poreet
Source toles
Souree
Prompts
(a) Numbers
(b) Number words
Placements
(d) Months
Figure 18: Plots of logits across various numeric classes, analogous to Figure 7a Source tokens on the y-axis and output tokens are on the xaaxis.
I DECREMENTATION BIAS ACROSS DIFFERENT TASKS
We show the strength of the decrementation bias in figures Figure and 19
J ALL SUCCESSOR SCORES IN A MODEL
In Appendix ! we find that for both Pythia- 1.4B (the mainline model in the paper) and GPT-2 Large randomly selected model without successor head from (Figure 2 left)), the heads with highest successor score are sparse: in Pythia-1.44 1.4B 1.12H has eight times as great successor score to the next higher successor score.
Logits
Logits
Logits
med ao feature
mod s0 feeture
moc 10 feature
Prompts
Prompts
Prompos
(a) Numbers
(b) Number words
(c) Placements
(d) Months
Figure 19: Plots of mod 10 feature logits across various numeric classes, analogous to Figure 7b
19

---

Published as a conference paper at ICLR 2024
Fraction of takens which are weakly successed
Fraction of takens which are weakly successed
(a) Pythia-1.49
(b) GPT-2 Large
Figure 20: Successor scores for Pythia- 1.4i and GPT-2 Large.
Prompt
Answer
(...) (A) Colony formation and
B
(...) (i) (2()) equals the factual density ('y) for all g E C:
(...) [22]: Conceived and designed the experiments (...)
(...) Kirovsky Zavod Station St. Petersburg, Russia (...) you can see a
statue of Lenin here.
(...) [9] Minutes, Criminal Law Revision Commission, January 28, 1972,
16.
Figure 21: Some examples of numbered listing prompts from the Pile dataset.
K
CASE STUDY: NUMBERED LISTING
In Section 4 we demonstrate that when the successor head is contributing usefully, the prompts often required some kind of incrementation. However, we want to investigate whether the converse holds: are prompts requiring incrementation mostly solved by successor heads?
Numbered listing is widespread across real datasets and requires incrementation. Additionally, blog post discussior suggests that even small LLMs are capable of this task in the case of incrementing citations. Examples of prompts involving numbered listing can be seen in Figure 21
We collect 64 such prompts and check for whether the successor head in Pythia- 1.4B winning case (as in Section f. and find that the successor head is indeed the winning head across all 64 prompts. Hence this provides some evidence prompts requiring incrementation real datasets are indeed mostly solved by successor heads.
L FIRING PATTERNS OF MLP, NEURONS
Moved to appendix based suggestions emphasise technical background more in the main text.
Firing and logit patterns for the top 16 most important MLP, neurons are displayed Figure 22
We see superposition effect, with Figure 22c and Figure 22i both representing 6 mod 10.
hel pu-cut- hagp
20

---

Published as a conference paper at ICLR 2024
1st most important neuran
Fining intenslty
7th most important neuron
Logits
Firing interstyy
Logit
Across prompts
Across prompts
(a) 1st place neuron, firing for odd numbers, and slightly more for mod
Across prampts
Across prompts
(b) 7th place neuron, 0 mod
10 neuron
8th most important neuron
most important neuron
intensty.
Firing intensay
Logits
30
Logits
5.5
-10
20
Across prompts
Across prompts
Across prompts
Across prompts
(c) 8th place neuron, 6 mod 10 neuron
(d) 10th place neuron, 4 mod 10 neuron
15th most important neturnn
16th mosz important neuron
intenslyy
Fining intensity
Logits
Logits
0.5
3.
20
60
Across prompts
Across prompts
Across prompts
Across prompts
(c) 15th place neuron, 5 mod 10 neuron
16th place neuron, another mod 10 neuron
Figure 22: Some examples of neurons firing strongly in modulo 10 patterns out of the top 16 most important MLPO neurons for successorship.
M DIFFERENT ABLATION METHODS
To analyze the effect of language model components when running ablation experiments it is im- portant to distinguish the direct, indirect and total effect language model components (McGrath) et al. 2023 Pearl 2009) model outputs, which are illustrated in Figure 23 To measure the di- rect effect of a component (for a given ablation method) the direct effect involves, at the end of the model, subtracting the head's output and adding the ablated output of the head to the residual stream. In Section 4 we analyzed direct effect under the ablation method of mean ablation. This appendix argues that our the direct effect is the largest effect and results hold under different ablation methods.
Dasic Cruph
Dirsci Effect
Indrect Effect
A
Figure 23: The types of effect an Attention Head (A) could have on model output (O), possibly through mediating downstream model components (D).
The indirect effect instead involves replacing head's output with ablated output and, at the very end of the model, subtracting this ablated output and adding the head's original output. This effectively ablates the downstream effects of head.
21

---

Published as a conference paper at ICLR 2024
The total effect replaces the head's output with an ablated output, which effectively ablates both the direct and indirect effect of the head.
We ablate head outputs using one of the two techniques:
1. Mean ablation: replacing the output of head with the average head output over a chosen distribution. We choose this distribution to be the current batch we are using.
2. Resampling ablation: replacing the output of a head with the head's activation on ran- domly sampled example from a dataset (e.g. the Pile). When using this method, we repeat this process a number of times and average the results.
Rerunning the loss reducing experiments described in Section for these different methods gives:
1. Direct effect resampling ablation: 33% for successorship, 11% for acronym, 10% for greater-than, and % for copying behaviour.
2. Indirect effect mean ablation: 6.5% for successorship, 2.8% for acronym, 12% for greater- than, and 9.0% for copying behaviour.
3. Indirect effect resampling ablation: 5.8% for successorship, 2.7% for acronym, 11% 1% for ghea and 8% for copying behaviour.
4. Total effect mean ablation: 27% for successorship, 8.2% for acronym, 10% for greater- than, and 8.4% for copying behaviour.
5. Total effect resampling ablation: 30% for successorship, 6.5% for acronym, 1 11% for greater-than, and 6.2% for copying behaviour.
Result demonstrates that using resampling ablation instead of mean ablation for analyzing the loss reducing effect has little effect on the results in Figure 10 Additionally, the indirect effect results (2 and 3) provide some evidence that downstream effects of the successor head are not highly significant to successorship
N TRAINING DETAILS FOR COMPOSITIONALITY EXPERIMENTS
Remark: obtaining decoding function. Recall that we wish to learn two linear maps - an index-
space projection TN: Rdmaid + Rdaat and a domain-space projection TD Rddmne - Redans such
that, for all pairs of tokens and ju (with i, valid token), [ (Te(i..) + (Dil ~ [+l
To evaluate the above identity, we must first learn a decoding function d IR drnake - IR dhaah such
that arg max, d(([i,])
dg. Given the informal observation that directly unembedding [1,]] vields next-token predictions for i, whereas unembedding S(i.) yields (i + 1), (see Appendix hypothesise that the unembedding matrix Wy reads from some 'output space' O and the embedding transform [.] writes to some 'input space' I and that the successor head reads from I and writes to O. Indeed, when training an output-space projection To + O over tokens in the vocabulary such that
we
i. obtain 97.4% top-1 accuracy both confirms the output-space hypothesis, and gives us a decoding function d(x) = (TO
set of 1000 held-out tokens which
Method. With our decoding function in hand, we can train TN and aD to satisfy our identity. Specifically, we define Ty and TD to be matrices such that my + TD = For valid token pairs i, and je. we obtain predicted representations [+a]] ([(.) +Tpl) and minimise a combination of "closeness metrics':
| - [1111 (()) + L(Wo
(c(((())
for L the cross-entropy loss. Specifically, ensure that predicted and ground truth representations 'behave in the same way" - in other words, that they are close together, that predicted representations [ decode tokens is (output-space decoding), and that the logit distribution when decoding in- cremented predicted representations (([,,] matches that when decoding incremented ground truth representations S ([&,,) (successor decoding).
More succinctly, we can frame the training procedure as learning TN₂ Tp such that the following diagram commutes:
22

---

We see that the Published as a conference paper at ICLR 2024
RN Rn
I
Wy
idr
By X
WooTo
I I
Logits N DJ
NxD D
I
Idwx
WyoSel-1
F
(N D) x (N D)
N
We trained for 10 epochs over valid token pairs sampled from the succession dataset, and evaluated on a held-out dataset of 500 randomly-samper token pairs.
O EFFECTS OF APPLYING THE SUCCESSOR HEAD TO COMPOSITIONAL. REPRESENTATIONS
assessed the performance of our factoring mechanism through outpur-space de- coding (i.e. evaluating a representation [[+]] by testing whether [ decodes to it). Below, we also present results for successor-space decoding (i.e. evaluating representation [ by testing whether ((E]) decodes to 1);)-
In Section A
In contrast to the high performance of output-space decoding, we found successor-space decoding yielded top-1 accuracy of 0.90 the held-out dataset of token pairs, and top-1 accuracy of 0.5 on the out-of-distribution Roman numeral dataset (see Table 3
Source index token (ism)
VI
VIII
XI
Third
fourth
fith
seventh
eighth
ninth
12
13
first
lenth
Third
fourth
fifh
VI
seventh
ninth
ninth
tenth
Monday
RS
Third
MC
Friday
MV
VII
Sequence
January
CS
Sept
MC
Fift
WV
VII
VIII
Y
XII
Table 3: A table presenting top-1 predicted tokens under successor decoding from (p()) + TN TH (dRom). Green cells denote predictions which match their target ((i + 1)a) exactly: yellow cells denote predictions which match the target index but the target domain; red cells denote incorrect predictions. Dashed cells denote pairs of 1, and itm for which i, is not valid (single) token.
This drop in performance when switching from output-space to successor decoding (and in partic- ular, the leakage of Roman-numert information into (1,) + ma((Rom) - notice the VII and VIII in Table 33 suggests numeric projection ли might be capturing slightly more than just the nu- meric subspace. Specifically, there may be some components domain-space which are ignored by space decoding, but which our successor head lifts into output-space.
ADDITIONAL ARITHMETIC TABLES
We display additional arithmetic tables analogously to those in Figure with number tables (randomly sampled ranges) in Figure 24 and number word table in Figure 25 results are similar those in Figure
Note, though, that, as successor dataset contains 1041 tokens, a random classifier (even when restricted tokens in the successor dataset) would achieve accuracy 0.001.
23

---

Published as a conference paper at ICLR 2024
50
52
54
55
56
57
59
(a) Arithmetic table for range 50 to 59
30
31
33
34
35
36
38
(b) Arithmetic table range to 39
120
121
122
123
125
126
128
129
(c) Arithmetic table for range 120 129
Figure 24: Additional randomly selected number arithmetic tables, analogous to those in Figure
24

---

Published as a conference paper at ICLR 2024
zero
three
four
six
seven
eight
(a) Arithmetic table for range zero to nine.
Figure 25: An additional arithmetic table for number words, analogous to those in Figure 6
Q LOGIT DISTRIBUTION FOR CELLS IN THE ARITHMETIC TABLE
The logit distributions across tokens for randomly sampled correct arithmetic examples displayed in Figure 26
succ('87")
'50' succ("S6')
18
16
15
Lagits
10
8
6
Tokens
Tokens
(a) Source token '83', target residue of modulo
(b) Source token '50', larget residue 6 modulo
10.
10.
eleven' suco(' fifteen')
'September' (ucoo'October')
15
10
Logits
eleves
twelve
thirkoan
Tourteen
fifteen
sixtsen
seventaen
eighteen
nineteen
twenty
Avenues
May
June
July
August
September
October
Novemben
Dacamber
Tokens
Tokens
(c) Source token eleven', target residue of modulo 10.
(d) Source token 'September', target residue of modulo 10.
Figure 26: Logit distributions for randomly sampled checkmarked cells, sampling two cells for numbers, one cell for number words, and one cell for months.
25

---

Published as a conference paper at ICLR 2024
R
LINEAR PROBES FOR MODULI OTHER THAN 10
1.0
0.8
Accuracy
0.6
0.4
Validation set
sest
2
3
4
5
7
9
10
11
12
14 Modulus
13
15
16
17
18
19
20
21
22
23
24
25
Figure 27: Validation (in-distribution task) and test
task) performance for linear
probes p(m), The vertical purple lines denote moduli divisible by 5.
While our experiments in Section focus on the study of mod- 10 features, one might hypothesise that there exist similar natural mod-k features for other explore this hypothesis, we repeat the linear probing experiment detailed in Section 3.2 for range of moduli m E {2, .... 25). Specifically, we test whether can learn linear probes p(m) Rm to predict the value of i mod m from Ni.-Prepresentations numeric tokens 44 and whether these probes generalise to non-numeric tokens. As per Appendix C.I train probes numeric tokens from "0" to '500' (both with and without leading space), holding out 10% of these tokens as validation set, and we test our probes dataset of unseen tasks including cardinal numbers, Roman numerals, months and days.
We present the results of these experiments in Figure 27 Observe that, while for almost all moduli, validation and test performance above random chance, cannot easily extract 'mod-m data' from token representations for all m. Indeed, the only probes with out-of-distribution performance above 0.5 those for moduli 2-5, 10, 12, 15 and 20 (and the six probes with the lowest validation performance are those corresponding to the six primes between and 25).
We see some evidence, however, that 10 a particularly significant modulus for token MLPo- representations: indeed, p(10) has both the joint highest validation performance (together with p(2) p(4), p(5) and p(20), and the highest iitoff-distribution performance. Moreover, of all the probes whose validation accuracy is above 0.5, p(10) has the smallest drop in performance from in-distribution to dittrf-distributiin tasks.
In particular, for weeks (which we might expect to have 'mod-7 features") and months (which we might expect have 'mod-12 features'), not only are the performances of ((7) and p(12) substan- tially lower than that of p(10), but p(7) fails correctly identify the index mod 7 of any day of the week, while P(12) only correctly identifies the index mod 12 of 8/12 months (failing February, May, July and August). By contrast, p(10) correctly identifies the index mod 10 of 4/7 days of the week, and 10/12 months.
26


Reviews:
[{"review_id": "XpaDH0136J", "reviewer": "Reviewer_uo7e", "summary": "This paper digs inside the mechanism of attention heads in LLMs and discovers some particular attention heads are able to fire for predicting naturally-ordered tokens, which is termed as successor heads. This paper belongs to a recent line of work, the mechanical interpretability of transformer models. The findings of successor heads appear to be common for different prompts and also across models, showing some level of polysemanticity in the successor attention heads’ activation space.", "questions": "How does the choice of n in top-n-attended tokens affect the findings? Authors only pick a particular k (i.e. k=5 or k=1) and present the result. Can you demonstrate the full story about polysemantics when you relax the constraint of importance by choosing large n?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The findings presented in this paper are significantly novel. Authors have clearly described the functions of successor heads and designed multiple experiments to validate their hypothesis. I especially appreciate Section 3.3 where the evidence in arithmetic is a strong proof that the activation of success attention indeed captures the natural ordering of words and is responsible for the LLM’s reasoning.", "weaknesses": "There are a few issues mostly in the presentation of the work. \n\nI am getting really annoyed when the authors place all definitions, i.e. the Glossary section, at the end of the appendix. It is really inconvenient for the reader to go back and forth during the reading. It has to have better ways to present the definitions in the context. Please do not do this. \n\nIt lacks sufficient descriptions for the reader to understand the process that parses the original output of the attending heads to the sparse encoder’s output. I understand that this is to make more room to present the findings; however, it makes the methodology part pretty unclear from reading the current version. I have to go back and forth and spend a lot more time on Section 2 and 3 to make sure I understand the way each figure is plotted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6SwWiTBpXv", "reviewer": "Reviewer_Vpua", "summary": "This paper discovers and analyzes Successor Heads, a type of attention heads in transformer language models that increases the probability of the next token in a sequence such as 1 -> 2, or Monday -> Tuesday. This is discovered mainly through ablations on a specific dataset crafted for this task, and analyzed through various means.", "questions": "How was the set of succession datasets chosen? Did you experiment with other tasks that were eventually not included? It would be interesting to measure successor behavior on some held out succession task, as currently behavior on all the tasks was used to find successor heads.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Good informative figures such as Fig 1 and Fig 7, clear writing. The use of OV circuits in the discovery and analysis seems smart and somewhat novel to standard methodology for these kinds of findings. Interesting behavior and good multi-pronged analysis of it.", "weaknesses": "Somewhat overclaiming the contribution:\nFor example abstract says:  \"Existing research in this area has found interpretable language model components in small toy models. However, results in toy models have not yet led to insights that explain the internals of frontier models and little is currently understood about the internal operations of large language models.\" This makes it sound like existing work has only studied toy models which is not true, while also making it sound like this work would study frontier models which is not the case. While they look at larger models than most related work, the wording makes it sound like difference is larger than it is.\n\nAlso the findings about mod 10 features are almost entirely based on the setting of incremental numbers which makes sense, while the writing makes it sound like they are behind successor head behavior on all tasks. The only evidence of these being used on other task is a low success percentage on changing output month with vector arithmetic. I would expect for tasks like months and days there would be other mod-12 or mod-7 features for example that could explain this behavior, was this studied?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0rihrnLYVc", "reviewer": "Reviewer_ExhM", "summary": "This paper demonstrates findings of a set of attention heads called successor heads that perform incrementation on tokens from ordinal sequences. It also shows evidence, in MLP0 layers, for \"mod-10\" features which are present in tokens belonging to the same numerical index. Experiments were performed to modify numeric inputs using vector arithmetic with the mod-10 features. Finally, the paper analyzes the polysemanticity of successor heads on natural language data samples. The paper's authors claim that these findings across several models of various scales demonstrate a weak form of universality.", "questions": "Questions:\n1. While using MLP0 with successor heads, in isolation, was shown to be sufficient to perform incrementation, it is not shown how they interact with the rest of the models they belong to that the paper studied. How is this information about numerical features processed in later layers, from MLP0 to the successor head L12H0, then to the rest of the model? How does this end-to-end path interact with alternative paths, and inhibitory ones?\n\nThe mod-10 features are obtained from MLP0's outputs, to study how the end-to-end path of MLP0 to the successor head processes these features. However, the paper does not show how this information is processed through other layers and MLPs in the models. Can such mod-10 features be found in other layers?\n\n2. To continue on weak point #2:\nThere are also the cases of adding non-mod-10 features to numeric tokens, and adding the features to non-numeric tokens. There may be cases of \"confounding\" factors where U(OV(MLP0(E('twelve')) - f_2 + f_4)) has high logits for 'fourteen', but is it possible that adding non-numeric features will also shift twelve to thirteen, fourteen, etc.? If so, then perhaps these mod-10 features are just obtaining the correct change because of how the numeric features are scaled with one another? In other words, would subtracting non-numeric features R then adding RR obtain the same result? This is not a weak point of the paper as it may be beyond its scope, and this situation seems unlikely to be the case, but it can be further investigated for thoroughness. \n\n3. To continue on weak point #3:\nIn terms of interpretable polysemanticity, how do other heads compare to successor heads? To ensure acronym handling is not commonly done for many attention heads, how well do other heads handle acronyms?\n\n\n- Other comments:\n\nThis paper tackles a similar topic as a previous project from earlier this year, which also found a successor attention head by performing OV circuit analysis on inputs of numbers, number words, days, months, and letters. It also was stated to have inspected the effects of vector addition on number tokens [3]. Could the authors elaborate on the similarities and differences? \n\n[3] https://alignmentjam.com/project/one-is-1-analyzing-activations-of-numerical-words-vs-digits", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "Strengths:\n1. The paper is clear and easy to read.\n\n2. To show claims of a weak form of universality, the paper thoroughly tests for successor scores across several models for various numbers of parameters.\n\n3. The experiments to find and verify the mod-10 features are also thoroughly performed. These features were confirmed by several different methods: first by training a sparse autoencoder using reconstruction loss on the MLP0 outputs, and then by further comparisons to linear probing and ablation to reinforce these observations. This is an interesting result uniting various tokens under mod-10 classes.\n\n4. There are also interesting results with the natural language experiments that demonstrate interpretable polysemanticity.", "weaknesses": "Weaknesses:\n1. In Section 4, direct effect mean ablation is used to show \"that when the successor head is contributing usefully, the prompts often required some kind of incrementation\". Then in Appendix J, direct effect mean ablation is used again to show that the successor head is the most important head across 64 prompts. Though this is stated with some evidence, not enough quantifiable evidence is shown here to justify the reach of these claims, such as the statement of \"mostly solved\". More analysis can also be shown about the \"direct effect\" to separate it from \"indirect effects\".\n\nThe paper also did not clarify the details of the ablation, such as if it used resampling ablation (there may be issues if it used mean ablation from the same dataset, as there are known issues with mean ablation [1]), and/or path patching (to obtain direct effects).\n\n[1] https://www.lesswrong.com/posts/kcZZAsEjwrbczxN2i/causal-scrubbing-appendix#3_Further_discussion_of_zero_and_mean_ablation\n\n2. The paper mentions that vector addition was performed successfully for 89% of the cases for digits 20-29, and mentions how it was performed on token '35'. It does not mention how this performed for other digits. This is likewise the case for number words only showing ten to twenty. Presumably, the performance is similar, but the paper should explicitly mention this to avoid criticism of cherry-picking.\n\nIn Figure 7, it's also unclear what \"target residues modulo 10\" means when referring to the column headers. Presumably, this is stating something similar to how the vector arithmetic on MLP0(E('fifth') makes it \"behave more like MLP0(E('seventh')\". The wording can be made clearer to avoid confusion that it means the number \"7\" rather than the word \"seventh\". Additionally, the checkmarks are given when \"the max logits are on the successor token\". This is an interesting result, but how big is the logit difference between logits for the successor token and other tokens? \n\nAppendix D states that scaling was used on the additive feature terms. A quick explanation of why a particular scaling factor was used would be helpful.\n\n3. The paper states: \"to the best of our knowledge the presence of both successorship and acronym behavior in head L12H0 is the cleanest example of polysemantic behavior identified so far in an LLM.\" Why is this the cleanest example of polysemantic behavior, compared to other studies on the topic such as in [2]? Similarly for this statement, \"which to the best of our knowledge are the most closely studied components in LLMs that occur in both small and large models\", what other components are you comparing to that are not as closely studied?\n\n[2] Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing, 2023\n\n4. This paper discovers novel and interesting observations, but it does not elaborate much on why this observation is impactful enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3UmSZ4IwHn", "reviewer": "Reviewer_uBkt", "summary": "The paper leverages the transformers circuits work (Elhage '21) to uncover successor heads, which are attention heads in a transformer responsible for incrementing words / numbers which have a natural ordering. To do so, the authors build an Output-Value circuit for these successor heads, which unlike (Elhage '21), leverages a non-linear transformation of the word embeddings. The authors show that \n\n1. successor heads occur in many decoder-only transformer architectures, and across different model scales, and different types of incrementations\n2. Through several analyses and ablations (the use of linear probing on the MLP0 features, as well as the use of sparse autoencoder to extract important features for successor heads), the authors argue that the model's internal representaiton for numerical / incremental tasks is in modulo 10, i.e. that successor heads, when given input $f_{i}$, will increase the logits of $f_{i+1 \\  \\text{mod} \\  10}$. \n3. The authors show that these successor representations are amenable to vector arithmetic operations, and that while the MLP$_0$ representations are not biased towards incrementation, the OV circuit for successor head is. \n4. The authors argue that successor heads enable the identification of polysemantic behavior, finding evidence that they are responsible for both acronym prediction and incrementation.", "questions": "1. Effective OV circuit : what is meant by effective here ?\n2. What was the motivation for using MLP$_0$ rather than the original word embeddings ? Given that MLP$_0$ occurs after a first self-attention layer, I would have not expected it to map embeddings to a representation amenable for the analysis presented in the paper. \n\nOn section 4 : \n3. successor heads in the wild. I am not sure I understand why the authors distinguish between features that, when ablated, make the correct prediction less likely, and features that, when ablated, increase loss the most. I understand that these two are not exactly the same thing, but I don't understand what we gain by treating them differently. \n4. It is mentioned that 128 samples are sampled from the Pile dataset. Did you happen to find a token where the successor head was important for each of the 128 samples ? Or did you bias your sampling towards extracts where the successor head is important ?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. The empirical rigor of this work is high. The authors provide several ablations to argue the existence of mod_$10$ features in transformers. They moreover provide detailed additional information for experiments in the appendix.\n2. The finding of successor heads is interesting and provides a good framework for understanding how transformers reason about incrementation. \n3. The connection between incrementation and acronym prediction observed in successor heads is interesting.", "weaknesses": "1. The paper lacks a proper background section. Terms like OV matrix are not introduced, and more generally, the notion of circuits, or what un enembedding is, are never properly defined. This makes the paper very hard to digest without being familiar with the concepts of transformer circuits (Elhage '21). Given that the paper is already quite dense, you can e.g. move figure 7 to the appendix, and properly lay the appropriate terminology to understand this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)