The Dual-Route Model of Induction
Authors: Sheridan Feucht
Eric Todd
Byron C Wallace
David Bau
Venue: COLM 2025 (2025)
Keywords: interpretability
induction heads
in-context learning
ICL
detokenization
TL;DR: We find that LLMs can do in-context copying in two different ways: either by copying individual tokens verbatim, or by copying entire word meanings (which may span multiple tokens).

Abstract: Prior work on in-context copying has shown the existence of *induction heads*, which attend to and promote individual tokens during copying. In this work we discover a new type of induction head: *concept-level* induction heads, which copy entire lexical units instead of individual tokens. Concept induction heads learn to attend to the ends of multi-token words throughout training, working in parallel with token-level induction heads to copy meaningful text. We show that these heads are responsible for semantic tasks like word-level translation, whereas token induction heads are vital for tasks that can only be done verbatim (like copying nonsense tokens). These two "routes" operate independently: we show that ablation of token induction heads causes models to paraphrase where they would otherwise copy verbatim. By patching concept induction head outputs, we find that they contain language-independent word representations that mediate natural language translation, suggesting that LLMs represent abstract word meanings independent of language or form.

Published as a conference paper at COLM 2025
The Dual-Route Model of Induction
Sheridan Feucht, Eric Todd, Byron Wallace, & David Bau" Northeastern University {feucht . S, toda. ez, wallace, d. d.bau @eootteasteeeeeeeeee
Abstract
Prior work in-context copying has shown the existence of induction heads, which attend to and promote individual tokens during copying. In this work we discover a new type of induction head: concept-level induction heads, which copy entire lexical units instead of individual tokens. Concept induction heads learn to attend to the ends of multi-token words through- out training, working in parallel with token-level induction heads to copy meaningful text. We show that these heads are responsible for semantic tasks like word-level translation, whereas token induction heads are vital for tasks that can only be done verbatim (like copying nonsense tokens). These two "routes" operate independently: we show that ablation of token induction heads causes models to paraphrase where they would otherwise copy verbatim. By patching concept induction head outputs, find that they contain language-inde indeeendenn word representations that mediate nat- ural language translation, suggesting that LLMs represent abstract word meanings independent of language or form.
1 Introduction
How do language models repeat sequences of tokens in-context? Imagine that you are asked to copy random string of characters from a leaflet of paper into a notebook: 'oane dnn t ephzawfeew eausr 1thii', This would be achievable but laborious task, requiring careful attention to each subsequent character. Now, imagine instead that these characters rearranged into the phrase 'the false azure in the windowpane' There are now two ways of copying the sequence: character-by-charactee, or by leveraging your understanding of English to copy large swaths of characters at a time.
LLMs can copy text using induction heads, a type of circuit found in decoder-only transformer models that enables them to copy sequences in-context (Elhage al., 2021; Olsson et al., 2022). Given the sequence w ax wwwgg.g......................... .iiig........................... w, an induction head at the second occurrence of would attend to and promote the earlier token ax by scanning for previous token information. Prior work has argued that induction heads are responsible for broader in- context learning capabilities (Olsson et al., 2022). However, it is unclear how attention heads operating on bakken-byttoker basis might handle "fuzzy" copying tasks like translation, which often requires conversion between words of differing token lengths (e.g., P ommes de I terre - pot latee). In this paper, we show that LLMs use concept induction heads in parallel with token induction heads to copy meaningful text.
The Dual-Route Model of Induction. Psychologists who study reading in the brain describe two parallel routes through which people read: a sublexical route that converts letter strings into speech sounds, and lexical route through which word meanings can be directly accessed as entire units (Marshall & Newcombe, 1966; 1973; Dehaene, 2009). If the sublexical route damaged, patients exhibit condition known as deep dyslexia, where they can understand the meanings of words without being able to access their sound or spelling: Marshall & Newcombe (1966) describe patient who, after brain injury, would read the word CANARY as "parrot" and COLLEGE as "school," indicating that he could still access word meanings, but was unable to read individual graphemes. "Code and data available https: / /dual route. info.

---

Published as a conference paper at COLM 2025
ase
uc
of
the
3X
srang
of
the
of
the
BX
wing
du
Token Induction: verbatim copying
Concept Induction semantic copying
Figure 1: The dual-route model of induction. LLMs develop token induction heads, which used for verbatim copying, alongside concept induction heads, important for translation and "fuzzy" copying tasks. These two routes work in parallel to copy meaningful text.
In this work, we consider the possibility of an analogous dual-route model of induction in LLMs, where tokens are equivalent to graphemes. Given piece of meaningful text, models can either copy by shifting individual subword tokens, or by accessing detokenized lexical information over an entire span of tokens. We characterize two types of induction:
Token Induction. Figure illustrates token-level induction circuits discovered in previous work. Elhage al. (2021) show that in two-layer transformers, models load previous token information into each hidden state, which allows induction heads in future layers to attend to these states and copy the corresponding token. We design causal intervention token induction heads in Section 2. When these heads are ablated, models lose the ability to copy sequences verbatim (Section 4), exhibiting "symptoms" analogous to patients with deep dyslexia (Hinton & Shallice, 1991; Zorzi et al., 1998).
identify
Concept Induction. Figure also depicts our current understanding concept induction heads, which are responsible for copying lexical information. To isolate these heads, we define a concept copying score based on causal mediation for a simple multi-token copying task (Section 2). Instead of attending to the next token, we find that these heads attend to the ends of multi-token words (Section 3), where concept information is more likely to be stored (Meng et al., 2022; Geva et al., 2023; Nanda et al., 2023; Feucht et al., 2024; Kaplan et al., 2025). Concept induction heads are vital for semantic copying tasks (Section 4), and output word representations that are language-agnostic (Section 6).
2 Token & Concept Copying Scores
2.1 Approach
To identify concept induction heads, we search for attention heads responsible for copying multi-token words by measuring the effects of causal interventions (Vig et al., 2020; Geiger et al., 2023). We hypothesize that if head increases the probability of future tokens for multi-token concepts, it is actually copying the entire concept.
Figure 2 illustrates our approach. We first sample random tokens to create an induction prompt X1 | xx.,g... using the newline token as a separator between the first and second repeating occurrences of X1...xn. Then, we append single concept made of m tokens c1..ccm to each half of the repeated sequence s. These concepts are sampled from a set of multi-token concepts C, with lengths uniformly distributed over 2 < < 5. Prompts are truncated so that the final token is always of regardless of m. We set n = 30 m.
This yields clean prompt Pelewn = X1 XI.....InII....mm x/.22"..//111, which we corrupt by replac- ing the first half with different random tokens: Peorrupt = //1/2...uu-m / We patch the activations of each attention head (!.)) (layer 1, head h) from the penultimate token
'oo avoid confounds from undertrained tokens (Land & Bartolo, 2024), we sample tokens by randomly selecting document from the Pile (Gao et al., 2020), tokenizing it, and shuffling the token order. We adopt this approach for all random token sampling henceforth.
2

---

Published as a conference paper at COLM 2025
P(cz Paae
Peoorua
P(ax)
Patch x
(GA)
head (l, ():
Prinee
Petocn
wing
foo
bar
opr
Eao
Yese
Polean
Poorrupt
Figure 2: We patch the outputs of each attention head from bar in Pelean to bar in Pearrupt to see whether that head has an impact on the "next-next" token P(c2), which is P(ax) in this example. Our hypothesis is that the heads that increase P(ax) in this setting actually carry the entire concept of "waxwing." See Section 2.1 for notation details.
Model
Hugging Face ID
[
Citation
Tlama-2-7b
meta- reta-11ama/l.i ama 2-7hh
32k
2T
Touvron al. (2023)
Llama-3-8b
mata-11aamaMMttall12
a-8-3-88
128k
15T+
Grattafiori
(2024)
OLMo-2-7b
alLenaa/0OMo-221144-7
100k
4I
OLMo et al. (2025)
(OLMo-2-1b)
aLLena3/00MM-22042333
100k
4T
OLMo et al. (2025)
Pythia-6.9b
EleutherAI/pythiie
50k
0.3T
Biderman al. (2023)
Table 1: Models used in this paper. | is the model's token vocabulary size, and f is the number of tokens the model was trained on (in trillions). We evaluate OLMo-2-1b on only subset of experiments.
position of Pelean (i.e., from x²y) into the same position in Peorrupt. Then, the concept copying score for head h1 over concept set is defined as
CONCEPTCOPYING(L.)
Σ CEC
(Lbb (
+ Peorrupt) -P(-- Pcorrpp))
(1)
CC|
where 2 indicates that activations (l.b) are being patched into the Peorrupt context at the
position corresponding to x/y- Because C2 predicted at the token position afler x we are measuring increase in probability for the "next-next token" (Pal et al., 2023; Wu et al., 2024). Our hypothesis is that an attention head increases probability for the future token c2, this may be because it is carrying information about the entire concept Cj...cm
We also want to find attention heads that are responsible for copying one token at time (i.e., token induction heads). We run the same procedure to find these heads, with two differences: (1) We use random tokens Fm...'m E instead of concepts, and (2) we measure the impact of each head on ((r)), instead of P(rz). This gives us our token copying score:
TOKENCOPYING(I,1)
Σ PCR
(1h) P(rilapahi
7 Peoorupt) P(ra Peorrupp)
(2)
R
For this experiment and for Section 3, we take concepts from the COUNTERFACT dataset (Meng et al., 2022), which consists of subject-object relations (e.g., "Paul Chambers plays bass"). We sample subset of subjects from this dataset to use as our set of concepts C, where [C| 1024. We could also use generic multi-token words, but results from Feucht et al. (2024) suggest that models treat both types of sequences similarly. We also measure scores for P(c1) and P(r2), but do not focus them in this work (see Appendix A).
3

---

Published as a conference paper at COLM 2025
(4)
Top-16 Copier Head Positions
(b)
Llama-2-7b: Token Copier Heads
Llama-2-7b: Concept Copier Heads
10
1.42.
15.3
11:15
16.9
50
11.22
Lla
6ee
6ee
12.25
13.23
58
B25
012
00
00000
44
Pyth
0000
tie
false
Dan
window
ano
unurs
lo
witdoo
ane
20
23
30
Layer
Takiro Canceg
Figure 3: We use causal mediation to identify token copier heads that copy the next random token, well as concept copier heads that copy future concept tokens. (a) Distribution of the top-16 token and concept copier heads across model layers. (b) Value-weighted attention scores over repeating phrase for the top causally-ranked heads (I.h) in Llama-2-7b. At the final token position "the", token copier heads attend to the next token wwinddw.p.ane"" whereas concept copier heads attend to the end of the next word (" ("window.p.anee
2.2 Results
We calculate causal scores for all heads in each model Table 1. This gives us two ways of ranking heads in model: by concept copying score and by token copying score. Figure 3a shows that concept copier heads seem to be concentrated in mid-early layers, whereas token copier heads are more sporadic, and are more likely appear at late layers. We find that there is little overlap between the top-k heads of each ranking (Figure 32) and that token & concept scores are not correlated (Appendix C.2), implying that these are two separate roles.
As a baseline, we also calculate increases in probability for future random tokens P(r2) when patching individual heads (Appendix A). This gives a sense of whether heads are copying several arbitrary tokens at a time. While some heads do increase the probability of future random tokens, their maximum intervention effects are at least three times smaller than heads that promote future entity tokens (for all models). We take this as preliminary evidence that concept copying heads copy meaningful units, not just arbitrary lists of tokens.
We also calculate copying scores throughout training checkpoints for OLMo-2-7b and Pythia- 6.9b. Some of the top concept induction heads OLMo-2-7b have high token copying scores before they develop into concept induction heads, suggesting that concept induction heads might develop from token induction heads. However, this pattern not as clear for Pythia-6.9b. We show examples of head trajectories throughout training in Appendix B.
Next-Token & Last-Token Attention Scores
Where do concept copying heads attend? In previous work, Olsson et al. (2022) found that token induction heads always attend to the next token-the one that they are about to copy. concept copier heads transfer entire concepts at once, we would expect them attend to the ends of multi-token words, where concept information is usually stored (Meng et al., 2022; Geva et al., 2023; Nanda et al., 2023). Figure 3b shows an example of attention patterns for both types of heads in Llama-2-7b. When the model is about to copy multi-token word, its token copier heads attend to the next token "window.ppane" while its concept copier heads attend to the end of the next word ("window.p.ane)
3.1 Approach
To capture this attention behavior, we design a last-token matching score which measures how much attention is paid to the last token of a multi-token concept. First, we select concept c Cj...Cm from COUNTERFACT subjects C, evenly sampling across concept lengths 2 < m < 5. We then construct random repeated sequences of tokens as before (see $2) and insert the concept at the end of the first half to create the prompt x1x22_..xx(1-..c.. x x/.2 The last-token matching score is then calculated as an average over the concept set C:
We the Hugging Face (Wolf al., 2020) implementation of each model.
4

---

Published as a conference paper at COLM 2025
LASTTOKENMATTCHNN(.M))
Σ CEC
(bb) [x cml)
(3)
[c|
where A represents the value-weighted attention scores for head index it at layer I, and the square brackets indicate that we collect attention paid from x, to
For comparison, we also calculate attention paid to the next token over random sequences, known in previous work as "prefix-matchnng scores (Olsson et al., 2022; Bansal et al., 2023). In this work, we refer to this as a head's ' next-foken matching score. Just like in $2, we use the same procedure that we do for last-token matching scores, with two differences: (1) we replace concepts with random spans of tokens fi.../m, and (2) we calculate attention paid to the next random token F1, instead of the last token.
NEXTTOKENMATCHING(..)) =
Σ FER
(A (..l)
R
Following Kobayashi et al. (2020), all attention scores in this work are calculated using value-weighting: i.e., we multiply attention weights for each token by the I,2 norm of their value vectors, then renormalize so the scores sum to one. This tends account for cases where attention "rests" on unimportant tokens like <>> at the beginning of prompt.
3.2 Results
Figure 4 shows next-token and last-token matching scores for heads of each type in Llama- 2-7b, averaged over 2048 COUNTERFACT entities. Token copier heads tend to have high next-token matching scores, whereas concept copier heads tend to have high last-token matching 3cores.3 3 We show only the top 16 heads (ranked using the scores defined in Equations and 2) for readability, but include top-64 scores for all models in Appendix C.1.
We also calculate correlations between causal and bbased scores. As expected, token copying scores positively correlate with next-token matching scores (r=0.63, p<0.001) whereas concept copying scores correlate with last-token matching scores (r=0.44, p<00.001) for Llama-2-7b. We find significant correlations for all models; see Appendix C.2.
Some heads seem to attend to both next-tokens and last-tokens. In Llama-2-7b, head 11.2 is the third-highest token copier head, but also has a relatively high last-token copying score (and attends to ane in Figure 3b). We also concept copier heads with next-token matching scores up to 0.20, such as head 13.23. This suggests that some heads may just copy the next "thing," " whether that be the next token the next concept.
Lesioning Concept and Token Copier Heads
We have found set of attention heads that attend to the ends of multi-token entities (Sec- tion 3) and help to copy the second tokens of those entities (Section 2). We hypothesize that these are concept induction heads, which copy by transferring entire concept representations at once. In this section, we show through targeted ablations that concept heads are vital for "fuzzy copying" tasks that deal with lexical semantics, whereas token induction heads are important for verbatim copying tasks.
4.1 Approach
First, we define new "vocabulary list" task that requires models to copy list of words in-context. Using word-pair data from Conneau et al. (2017) for five languages, we enu- merate n = 10 words, followed by parallel list of English translations for each word (see Appendix D.2 for full examples). Using data from Nguyen al. (2017), we also build similar
Last-token attention scores are lower overall because attention can spread over the length of the concept; for example, "window,p" could be treated as concept if the model guesses that the word is "windowpane" (we already this for some heads in Figure 3b).

---

Published as a conference paper at COLM 2025
Llama-2-7b: Token Copier Heads
Llama-2-7b: Concept Copier Heads
Atteation Matching Score
Attention Matching Scom:
Next-tok
Next-tok
0.8
Lasi-tok
Last tok
0.6
maa last-token matching soore
0.4
0.4
0.2
0.2
0.0
16.19
11.15
11.2
12.26
8,26
24.29
21.28
633
11.18
17.16
17.22
16,1
29.2
6.30
28.13
29.21
14.1
14.9
11.22
13.23
9.25
14.7
10.14
16:29
1322
1410
8.5
12.20
22.17
15.19
18.31
Arcontion Head
Allention Head
Figure 4: Attention-based matching scores for the highest-scoring causal heads in Llama-
2-7b. Consistent with prior work, heads that are responsible for copying the next random token have high next-token matching scores. On the other hand, heads that are responsible for promoting future entity tokens have the highest last-token matching scores.
prompts for uppercased and title-cased English words, synonyms, and antonyms. We calculate first-token accuracy and exclude word pairs that have the same first token. Finally, we add two verbatim tasks, where the first and second list are identical: English copying, using words from Conneau et al. (2017), and "nonsense copying," using randomly-sampled tokens. Models are evaluated on 1024 prompts per task.
For these tasks, we mean-ablate (Wang et al., 2023a) sets of concept and token induction heads to observe impact on model performance. We use causal scores from Section 2 to rank according to concept copying score, and once according to
heads in each model twice: token copying score. We ablate the top-k heads in each of these rankings by replacing their activations across all token positions with their mean activations random Pile documents (n = 1024).
4.2 Results
Figure 5 shows ablation results for Llama-2-7b. We see two separate "routes" through which models can copy: a semantic route, via concept induction heads (heads with high concept copying scores), and a verbatim route, using token induction heads. When concept induction heads are ablated, we see a drop in translation, synonym, and antonym accuracy, with no effect on surface-level tasks. When token induction heads are ablated, nonsense copying fails, but the model can still use concept heads to complete the rest of the tasks. English copying remains unaffected for both types of ablation (e.g. peal coat - pea I coat), as it can be solved using either type of induction. Like English copying, uppercasing tasks can also done semantically or without regard meaning. We report similar results for other models in Appendix D. For all models, ablation of concept induction heads damages semantic tasks much more than it does surface-level tasks. Llama-3-8b and OLMo-2-7b behave similarly to Llama-2-7b, but token ablation results for Pythia-6.9b are less clear-cut.
In Appendix D.3, we repeat this experiment while controlling for word length to test whether are simply distinguishing between multi- and single-token tasks. For single-token words, we see the same pattern, with a weaker (but still distinct) separation between translation and verbatim copying when ablating concept heads. Thus, in cases where words are constrained to a single token, token heads may also be able to assist in semantic tasks.
Qualitative Examples. To get sense of how model outputs look when token induction heads are ablated, we prompt token-ablated models to copy entire sentences verbatim. Ablating a small number k of token attention heads results in "paraphrasing" behavior instead of copying behavior; in other words, the model is able to copy the meaning of the sentence, but doesn't get every token exactly right. Box 4.1 shows that Llama-2-7B starts to paraphrase when k 32 token induction heads are ablated.' We find that k 32 is
"To encourage copying, we feed the sequence twice; we omit the first occurrence here for brevity.

---

Published as a conference paper at COLM 2025
(a) Example Task
(b) Ablating Token Induction Heads
(c) Ablating Conoept Induction Heads
French vocabulary:
Nonsense Oopstig
draite
Explic Croetic
fin
0.8
Thtle Ceeing
Unpeecoddr
roi
Antreyyn
Symenyyrz
English translation:
9.4
Translstion
right
end
10
2000
2:0
300
Tog Indection Heada Mant-ciiteed
Top- Ocoeept Todaction Roads Moar-Alloo
Figure 5: Mean-ablating the top-k copier heads for "vocabulary list" tasks in Llama-2-7b. Ablating token copier heads damages copying for nonsense tokens, without affecting tasks that can be performed semantically. Ablating concept copier heads damages performance for translation, synonyms, and antonyms, without affecting surface-level copying. English copying, which can be done either semantically or token-by-toker remains high for both types of ablation, as do uppercasing tasks (which can be done without regard to semantics). We plot results relative to models' original task accuracies; see Appendix D for details.
sweet spot that allows us to observe this effect without causing the model to stop copying altogether. Box 4.2 shows ablated Llama-3-8b generations when prompted copy a piece of Python code. When token induction heads are ablated, the model still copies the meaning of the original snippet, but writes using list comprehension. We provide examples of this paraphrasing behavior for all models in Appendix D.4.
Box 4.1: (Llama-2-7b) Original Model vs. Top-32 Token Induction Heads Ablated
I have reread, not without pleasure, my comments to his lines, and in many cases have caught myself borrowing a kind of opalescent light from my poet's 's fiery orb. I have reread, not without pleasure, my comments to his lines, and in many have caught myself borrowing kind of opalescent light from my poet's fiery orb.
I have reread my comments his lines, and have caught myself many times borrowing from his fiery orb kind of opalescent light.
Box 4.2: (Llama-3-8b) Original Model vs. Top-32 Token Induction Heads Ablated
foo = []
foo []
for i in range (len (bar)):
for i in range (len (bar)) )
if $ == 0:
if 2
foo. append (bar [i])
Eoo app.nd (bar [i])
foo []
for i in range (len (bar))
if & 2 == 0:
foc [bar for i in range len (bar)) if i 8 2 == 0]
asdf append (bar [i])
5 Concept Heads Reveal Semantics of Hidden States
If concept induction heads work with meaningful representations, the subspaces that they read from and write must contain semantic information. What do these subspaces look like? We use the attention weights of concept heads to define concept lens Lc, E R that reveals the semantic information contained within any particular hidden state. Specifically, sum the OV matrices (Elhage et al., 2021) of the top-k concept induction heads C₁ to obtain LC,- We then apply this transformation to a hidden state, followed by the model's final normalization and decoding head (see Appendix E for details).
We can use this linear transformation to visualize how a model represents a word in particular context. For example, consider the word cardinals, which could refer to sports
7

---

Published as a conference paper at COLM 2025
Prompt
Concept Lens Outputs 20)
cardinals
Card', cardinal', 'Card' Card', " football', baseball', Rome', Catholic', Card' birds', bird', Bird',
he was lifelong fan of the cardinals the secret meeting of the cardinals in the morning air, she heard northern cardinals
Table 2: Applying concept lens to hidden states for the token inals in the word cardinals reveals context-sensitive semantic representations. We show layer 20 with weights from the top k = 80 concept induction heads for Llama-2-7b. See Appendix E.
(a) Translation Patching
(b) Llama-2-7b: Patching Concept Heads
Ispañol:
"fuego"
Italiano:
"fuoco"
Accuracy
0.4
Español:
"niño"
Italiano:
W3 1
Patch top-k heads
0.2
B:4:5
No patching: M
0.0
B#1h:
1881""
#*
Concept
0
25
50
75
100
125
Top-k Concept Induction Heads Patched
Figure 6: (a) Patching the top-k concept induction head outputs from Spanish-Italian prompt into a Oppanese-Chineee prompt. (b) Patching concept induction heads changes the concept output by the model without affecting the language (i.e., niño, the Spanish word for "child," is translated into Chinese instead Italian). This effect is strongest for k = 80, which is also where the largest separation between semantic and literal copying performance is found in Figure 5c. Across π 128 examples, this approach causes the model to output the source Spanish word in the base output language with an accuracy of about 0.40 (solid red line). This is comparable to the model's original Japanese-Chie Chinese translation accuracy of 0.48 (dotted gray line).
team, senior members of the Catholic church, or group of red birds. By transforming the hidden state for the token inals with Lc, and projecting to vocabulary space, we can see that Llama-2-7b has different semantic representation for cardinals depending on the context that the word is in, further supporting the idea that concept heads represent word meanings rather than surface-level token information.
Concept Induction is Language-Agnostic
concept induction heads are important for translation tasks, what do they output? We hypothesize that the representations being copied by concept induction heads are abstract representations of word meanings. In other words, we posit that concept induction heads have the same activations when copying "waxwing" as they do when copying "снириетеив," " as these two words refer to the same concept, and are only expressed differently on a surface level. This in line with previous work suggesting that concept information in LLMs may be language-agnostie (Wendler et al., 2024; Dumas et al., 2024; Brinkmann et al., 2025), though Schut et al. (2025) argues concepts for some tasks may be biased towards English. For model representations to be truly language-agnostic they must be trained on those languages- -thgg even if high-resource languages are represented in unified semantic space, this effect may not hold for low-resource languages. Unfortunately, as we do not evaluate on low-resource languages in this work, we cannot make claims as to how models represent low-resource languages.
6.1 Approach
We adopt similar experimental approach to Dumas et al. (2024). They provide dataset ( of word-level translation prompts, where the model is shown five (elinf, g(out) pairs and prompted to translate word w from ((m)) {(out), Following their approach, we define
8

---

Published as a conference paper at COLM 2025
(a) Translation Patching
(b) Llama-2-7b: Patching FV Heads
Françaist "jambe"
English:
'leg"
0.0-
Frençais: "conité"
Englisht
Fatch top-kh
Accurar
6.4
My 4
0.2
my 22
"Tahrrad"
Pycciani:
"senocunea"
Deatech:
"Klocett"
Русскмяи
Concept heads: KoMiTer
FV heads: toilet
20
50
70
100
126
Top la Fazetion Vector Hoads Patched
Figure 7: (a) Patching concept heads changes semantics without affecting language, while patching FV heads changes output language without affecting meaning. In red, we show the same experiment from Figure 6: here, patching concept heads causes the model to output the source concept "committee" in Russian. In green, we show that patching FV heads at the same position for the same prompt causes the model to output the base concept "toilet" in English. (b) Across 128 examples, patching FV heads from a French-English prompt into a Bermann-Russian prompt flips the output language to English with an accuracy of about 0.80 (green line). This is higher than the model's s original German- Russian translation accuracy (gray dotted line, approximately 0.41), perhaps because translating into English is easier for Llama-2-7b than translating into Russian.
source prompts : ef(n) (out) and base prompt b efin) - e(out), where input and output
languages differ between source and base prompts. For example, a source prompt might translate from Spanish Italian, and a base prompt might translate from Japanese to Chinese. These prompts translate two different words W, and wp. Their target outputs are those words expressed in their respective output languages: 20g : !!ou)) and wh : Lg el
We patch the set of activations at | e C*} of the top-k concept copying heads Ck from the last token position of s into the last token position of b. We generate = 10 tokens in this manner with NNsight multi-token generation (Fiotto-Kaufman al., 2025), intervening for each newly-generated token. Then, we measure performance by evaluating accuracy of model generations. Specifically, given a generated string w, we consider w equal to a ground truth label if it is contained within or contains the ground truth string. Dumas et al. (2024) provide multiple possible translations for output words, and marked correct if it is equal to any of these synonyms.
6.2 Results
Figure 6 shows results for Llama-2-7b when patching outputs from Spanish-Italian prompt into a lapanese-Chinese prompt. Patching concept induction heads causes the model to output the source word wt in the base language e p(chif) For example, patching the outputs of concept heads from Spanish-Italian prompt translating "child" into Japanese- -hhnneee prompt causes the model to output the word for " "child" in Chinese. This intervention is the most effective at about k = 80, which also corresponds to the point where translation and nonsense copying are the most separate in Figure 5c. We show similar results for Llama-3-8b and more language pairs in Appendix F. This suggests that concept induction heads transport semantic representations that are expressible across multiple languages.
Function Vector Heads Complement Concept Induction Heads
Function vector (FV) heads are attention heads whose outputs help to promote in-context tasks. The outputs of FV heads for few-shot antonym task, for example, will cause the model to output antonyms when added to new contexts (Todd et al., 2024). We argue that FV heads can be thought of as "sisters" to concept induction heads: in the case of translation, concept heads copy semantic information, whereas FV heads copy language information.
We calculate correlations between FV scores and concept copying scores and find significant, albeit weak, positive correlations for all models (Figure 48), suggesting there may be some relationship between these two types of heads. This is perhaps because they can both
9

---

Published as a conference paper at COLM 2025
be thought of as "soft" induction heads, responsible for copying high-level conceptual information in-context.
However, FV and concept heads seem to play two distinct roles in the tasks we examine. We focus on translation, and patch outputs of FV heads between translation prompts using the same approach that we do for concept induction heads in Section 6. Our experimental
setup and data is identical, except we patch activations a = fas ( (1, E Ft where pk represents the top-k FV heads for given model, We then measure the output of the base word in the source output language, Wp ls e In other words, we measure whether
patching FV heads changes the output language while retaining the original base concept.
Figure shows that for the exact same prompts, patching FV heads causes the model to output the same concept in a different language, whereas patching concept induction heads causes the model to output a different concept in the same language. The outcome of patching FV heads more sensitive to output language (i.e,, the effect is strongest when the output language is English), but nonetheless suggests that FV heads play distinct role from concept induction heads (see Appendix G).
We also replicate ablations from Section 4 using FV rankings instead of concept and token copying rankings (Appendix G). Ablation of FV heads damages performance for non- verbatim tasks significantly, which makes sense: models cannot perform task without knowing what the task is.
8 Related Work
Induction Heads and ICL. The in-context learning capabilities of LLMs as demonstrated
by Brown et al. (2020) motivate a body of research on ICL (Dong et al., 2024; Lampinen et al., 2024). We build on the discovery of Elhage et al. (2021) and Olsson et al. (2022), which characterizes token induction heads. Concurrent with our work, Yin & Steinhardt (2025) argue that function vector (FV) heads (Todd et al., 2024), not token induction heads, are primarily responsible for few-shot TCL performance. Similarly, Yang et al. (2025) find that FV heads, which they call symbolic induction heads, perform induction over abstract variables. Akyürek et al. (2024) document n-gram heads, an n-gram generalization of induction head behavior, and Ren et al. (2024) study semantic induction heads that encode syntactic and semantic relations. Our work differs from these previous studies that we identify concept induction heads via their ability to copy multi-token concepts.
Concept Representations in LLMs. We build upon previous work showing that LLMs contain internal representations of "concepts" beyond individual tokens (Kaplan et al., 2025; Hewitt et al., 2025). Some work has studied how individual tokens are converted into abstract concept representations via neurons (Elhage et al., 2022; Gurnee et al., 2023; Nanda et al., 2023) or attention heads (Correia et al., 2019; Ferrando & Voita, 2024). Other work has shown that this concept-level information is stored at the ends of multi-token entities/ /prrases across various settings for factual recall (Meng et al., 2022; Hernandez et al., 2024; Feucht et al., 2024; Ghandeharioun et al., 2024; Ferrando al., 2025), and classification tasks (Wang et al., 2023b; Tigges et al., 2024; Marks & Tegmark, 2024). Our work identifies particular heads that transport this conceptual information between token positions.
9 Conclusion
In this work we investigate how LLMs copy lexical information. We find concept induction heads that are responsible for copying multi-token words, and whose representations are expressible across languages. These heads act as second "route" through which models can copy meaningful text, alongside previously discovered token induction heads (Olsson et al., 2022). Concept induction heads are one example of how LLMs might use induction in a general way to transport abstract contextual information between hidden representations.
10

---

Published as a conference paper at COLM 2025
References
Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-context language learning: Architectures and algorithms. In Forty-first International Conference on Machine Learning, 2024. URL https: http///ppwreeeeeeeeeeeeeeeeeeeep net/
Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. Rethinking the role of scale for in-context learning: An interpretability- based case study at 66 billion scale. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceddings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 11833-11856, Toronto, Canada, July 2023. As- sociation for Computational Linguistics. doi: 10.18653 /v1//v////// h 102222 acl-long.660. URL ht / htt///aclaan https///aaaaaatlogg....o. org/ 2023. acc acl -nng 660/.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, 23. JMLR.org, 2023.
Jannik Brinkmann, Chris Wendler, Christian Bartelt, and Aaron Mueller. Large language models share representations of latent grammatical concepts across typologically diverse languages. Xive preprint XXiv:2501.06346, 2025. URL https: //tp///aavvva/aa///a//ee.ea.ae.a org /bs/ 25C1. 06346.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https: https: htpp:ceess///ooeeigggs//. paper/2020 1ailee144 0tte:///7700000000000411100aaaa. Pape....
neurips. ctwp://rooeeereoooooooooo....... htwp://re...oooi.....///eeeeeeee
Alexis Conneau, Guillaume Lample, Marc' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. Xiv preprint ar XfXi::1700.04087, 2017.
Gonçalo M. Correia, Vlad Niculae, and André F. T. Martins. Adaptively sparse transformers. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNN.P-IJCNIP)) pp. 2174-2184, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/vll D19-1223. URL https: / acl anthology org/D1 D1 - 1223/.
Stanislas Dehaene. Reading in the Brain: The New Science of How We Read. New York: Penguin, 2009.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in- context learning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1107-1128, Miami, Florida, USA, November 2024. Association for Computational Lin- guistics. doi: 10.18653/v1.11 01/553//1/2244emmlpmmmm mainem URL https: ///aa/ccaaantloogoe// org/ 2024.emnlp-main 64/.
Clément Dumas, Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Separating tongue from thought: Activation patching reveals language-agnostic concept representations in transformers. ar preprint arXiv:2411.087.3 2024. URL https:
// farxiv.org/bbs. 2241 08745.
11

---

Published as a conference paper at COLM 2025
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Heranddez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical frame- work for transformer circuits. Transformer Circuits Thread, 2021. https:/ / transformer- circuits.pub 202/ 21/1/22211/////
Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott John- ston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernan- dez, Amanda Askell, Kamal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Doder Jackson Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Kadavath, Josh Jacobson, Eli Tran-Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCandlish, Dario Amodei, and Christopher Olah. Softmax linear units. Transformer Circuits Thread, 2022. https:/ /tsss///traaae /transformmr-- /imlex.ttll.
circuits.pub/ /ttp://///ppo///eeeee/e
Javier Ferrando and Elena Voita. Information flow routes: Automatically interpreting language models at scale. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Pp. 17432-17445, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v /v11 22222224.2mm/ 2024.emmlp-main.966 URL https: a ogy. org/2024.ems 1omnlp-mun p-mainn m....aainn..........i..m 965/-
antho
Javier Ferrando, Oscar Balcells Obeso, Senthooran Rajamanoharan, and Neel Nanda. Do i know this entity? knowledge awareness and hallucinations in language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.o net/ http:/ommmmm-oo000111000---
Sheridan Feucht, David Atkinson, Byron Wallace, and David Bau. Token erasure as footprint of implicit vocabulary items in Ilms. In The 2024 Conference on Empirical Methods in Natural Language Processing, 2024. URL https: / /awxiv.iviv.////aa///..aa/aaaava org/ /bbs/406.. 20086.
Jaden Fried Fiotto-Kaufman, Alexander Russell Loftus, Eric Todd, Jannik Brinkmann, Koyena Pal, Dmitrïi Troitskii, Michael Ripa, Adam Belfki, Can Rager, Caden Juang, Aaron Mueller, Samuel Marks, Arnab Sen Sharma, Francesca Lucchetti, Nikhil Prakash, Carla E. Brodley, Arjun Guha, Jonathan Bell, Byron C Wallace, and David Bau. NNsight and NDIF: Democratizing access to foundation model internals. In The Thirteenth Inter- national Conference on Learning Representations, 2025. URL https: /ttp////pp/eeeeeee/////e/......m net// forum? M.MMMMXEFFFFF..
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arviv preprint arXiv:2101.00027, 2020. URL https: : arxia.iiivv..... org/ wwwp/r/g//////oeeeee..eeeeeeeeee 2bs 00027.
Atticus Geiger, Duligur lbeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, et al. Causal abstraction: A theoretical foundation for mechanistic interpretability. arXiv preprint (rXiv:2301.04709, 2023. URL https: /ttp://aaxvvv....i org/ abs//bb////// 2301 04709v3
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12216-12235 Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653 /v1//vv1////// /ttp//////2/eeeeee/ee www.0//.e.nl--mmmmm htt./2223.emmmpp mai.:.pmmmi.... 751/.
emnlp-main.751. 75.. URL https: /awaaanannnnnoo.....
Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patch- scopes: A unifying framework for inspecting hidden representations of language mod- els. In forty-first International Conference on Machine Learning, 2024. URL https:
//openreview n forum? httt://oorrmm..-55ssc...... d=5uw82228855
12

---

Published as a conference paper at COLM 2025
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Ro- driguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bob- bie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta lbarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Isimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Praijwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Gird- har, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Ra- parthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe I'apakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Chang- han Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide,
13

---

Published as a conference paper at COLM 2025
Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpem, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Trina-Flena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Gro- shev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bon- trager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Paatthasaraty, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL
org/ abs///////// 2407.2177.22 .21783.
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: /wppeeevee....ee////e net/
Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. In The Tweeffh International Conference on Learning Representations, 2024. URL https: http::::///ppeeeiieeee net/ ftrmmr/rrm////////////
14KE.
John Hewitt, Robert Geirhos, and Been Kim. We can't understand ai using our existing vocabulary. arXiv preprint 2rXio:250..0756, 2025. URL https: http:s////xxvooooa org/ abs/ 2502. 07586.
Geoffrey E Hinton and Tim Shallice. Lesioning an attractor network: investigations of acquired dyslexia Psychological review, 98(1):74, 1991.
14

---

Published as a conference paper at COLM 2025
Guy Kaplan, Matanel Oren, Yuval Reif, and Roy Schwartz. From tokens to words: On the inner lexicon of LLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL https: httpt/:p:///peeevviieeee
net/
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only a weight: Analyzing transformers with vector norms. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7057-7075, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ve 2tt////1/22200mmmmmmmmm... https: / /aacaalcahtooo logy. org/ 2020. emn comnl-mma 574/.
URL
Vedang Lad, Jin Hwa Lee, Wes Gurnee, and Max Tegmark. The remarkable robustness of llms: Stages of inference?, 2025. URL htttt:://aaxi...oooto /ttp:///.rrr////////////// 2406. 19384.
Andrew Kyle Lampinen, Stephanie CY Chan, Aaditya K Singh, and Murray Shanahan. The broader spectrum of in-context learning. ar preprint ar arXin:2412.03782, 2024. URL
https:: /taps://arxee //trx/aavvvvv//////// org/abs/ 241 03782.
Sander Land and Max Bartolo. Fishing for magikarp: Automatically detecting under- trained tokens in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11631-11646, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ h22024.emn/pmmmaiii maaill-pmaii..499 URL https: //aclanchology 2r2/2024..... emnlp-main. 649/.
Samuel Marks and Max Tegmark.
The geometry of truth: Emergent linear structure in large language model representations of true/ datasets. In First Conference on Language Modeling, 2024. URL h www.:eewwwwwwweeeeeeeeeeeeeeeeee www..neeoooommdddaaaaaaaa.aaaa
John C. Marshall and Freda Newcombe, Syntactic and semantic errors in paralexia. Neuropsy- chologia, 4:169-176, 1966. URL https: http:/pppp.aaiiiiiiiooocaam wwwaaaarorroooooooboa.aa 144782251.
John C Marshall and Freda Newcombe. Patterns of paralexia: A psycholinguistic approach. Journal of psycholinguistic research, 2:175-199, 1973.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing
factual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022. arXiv:2202.05262.
Neel Nanda, Senthooran Rajamanoharan, Janos Kramar, and Rohin Shah. Fact finding: Attempting to reverse-engineer factual recall on the neuron level, Dec 2023. URL https: /www:/wwwwwooooo al wwwgnmentttiinnneee.ooomm.nn.. forum. org http://pppsttttttttwii////./.. fact www.iiinnnnntteepaaaaaaeaeaee
to eeggee-ee--eenggnnerr--------e
Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. Distinguishing antonyms and synonyms in pastern-based neural network. In Mirella Lapata, Phil Blunsom, and Alexander Koller (eds.), Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pp. 76-85, Valencia, Spain, April 2017. Association for Computational Linguistics. URL ht.ps:::p.... / /acl antho erg/E17-1005
nostalgebraist. interpreting gpt: the logit lens, 2020. URL wwwprreeeiing--t---ttete-- /ttpotts//ccBBBBtttttoorrrrreeee
the
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane A Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Ji- acheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo 2 furious, 2025. URL https:/ /tops///rxxiv org/ wtg/abs/////ooooooo//////////a /os/2501 00656.
15

---

Published as a conference paper at COLM 2025
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformeer Circuits Thread, 2022. h /tt.ansfooomrrrrirtttttt.o pubp htpp//22////a in- mameexxlemiingaan--dddciioo-oo
index.html.
Koyena Pal, Jiuding Sun, Andrew Yuan, Byron Wallace, and David Bau. Future lens: Anticipating subsequent tokens from a single hidden state. In Jing Jiang, David Reitter, and Shumin Deng (eds.), Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pp. 548-560, Singapore, December 2023, Associa- tion for Computational Linguistics. doi: 00.16658/112223.comll //aca //aclanthology org/ 2023..2...22 conll- 11111111377 37/.
URL https:
Jie Ren, Qipeng Guo, Hang Yan, Dongrui Liu, Quanshi Zhang, Xipeng Qiu, and Dahua Lin. Identifying semantic induction heads to understand in-context learning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Compulational Linguistics: ACL 2024, pp. 6916-6932, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653 /t1//224..nniingg/... //aclanthology.. org/ 2024. findings acl. 412/.
URL https:
Lisa Schut, Yarin Gal, and Sebastian Farquhar. Do multilingual Iims think in english? arv preprint ar Xio:2502.15603, 2025.
Curt Tigges, Oskar J. Hollinsworth, Atticus Geiger, and Neel Nanda. Language models linearly represent sentiment. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th BlackboxNI.P Workshop: Analyzing and Interpreting Neural Networks for NLP, Pp. 58 87, Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024 blackboxnlp-1..5 http:::: / aclanthology org/ 2024 b blackboxnlpp1..5/
Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. In Proceedings of the 2024 International Conference on Learning Representations, 2024. URL https::/p////. http::::///ppeeevieeee wwwnteettoooue
id=RwyxttyWwaG, arXiv:2310.15213.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brían Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: / /ttxiv..vvvv./////////a//////vvl org/ abss//////// 2207 09288.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal media- tion analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12388-12401. Curran Associates, Inc., 2020. URL
neurips cctpaper_fileeeeeeeeeeeeeeeeeppe files/ hwtp:/eeewe..cccceepeee.......e
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in GPT-2 small.
16

---

Published as a conference paper at COLM 2025
In The Eleventh International Conference Learning Representations, 2023a. URL https:
//tp///a/a/// //ppneeview.... www.ee.weeee......... wwt.:/etttuoneeeeeee forum? Vonniii
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, PP. 9840-9855 Singapore, December 2023b. Association for Computational Linguistics. doi: 10. 10.18553/v1/2000 emnlp-main.600 609. URL https : / a c ho ogy . wrgg20233 emn palpppmammm ma maain... 609/.
Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Do llamas work in English? on the latent language of multilingual transformers. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meefing of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 13366-1539) Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18633/v1. 0663//v//2024.cc-lonng 820. URL https: hatper//allaattt attpaaaaaannooooooooooooooooooat org/ 2024 acl aal.accllon... 820/.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. URL https: www. ac aclweb. orq/ arg/anthology/// /202 2020 emnlp-demos. . 6.
Wilson Wu, John Xavier Morris, and Lionel Levine. Do language models plan ahead for future tokens? In First Conference on Language Modeling, 2024. URL https:/ openreview wtw.://wwwn//oorummm/mm /ttp://rrrrrdaaaAAAPPPPP
Yukang Yang, Declan Campbell, Kaixuan Huang, Mengdi Wang, Jonathan Cohen, and Taylor Webb. Emergent symbolic mechanisms support abstract reasoning in large language models. Xrv preprint arXiv:2502.20332. 2025. URL https: /ttp///axvvv//////////a aabs//////////// 2502. 20332.
Kayo Yin and Jacob Steinhardt. Which attention heads matter for in-context learning? leanmiin arXiv preprint ar Xie:2500.144010 2025. URL
14010.
Marco Zorzi, George Houghton, and Brian Butterworth. Two routes or one in reading aloud? connectionist dual-process model. Journal of Experimental Psychology: Human Perception and Performance, 24(4):1131, 1998.
17

---

Published as a conference paper at COLM 2025
A Token & Concept Copying Scores
We provide copying scores for Llama-2-7b in Figure 9 and Figure 10, for Llama-3-8b in Figure 11 and Figure 12, for OLMo-2-7b Figure 13 and Figure 14, and for Pythia-6.99 in Figure 17 and Figure 18.
Although we calculate intervention scores for the next token over entities (c) and the next-next token for random tokens (r2), we do not focus on these scores in our work. In the former case, heads that are responsible for promoting the next concept token (i.e., c1) could be either concept induction heads or token induction heads; we assume that this score would not help us to differentiate between the two. The latter score gives us information on heads that copy multiple arbitrary tokens (e.g., 12) at a time (heads that copy "lists" of random tokens). Careful comparison of the right-hand plots for entity tokens versus random tokens (e.g., Figure 9 vs. Figure 10) shows that these heads have weaker effects, which helps to support our hypothesis that concept heads are not just "copying lists of tokens." " However, we do not utilize these scores further in this work. In Figure we plot the concept copying score against token copying score for each model. We find that concept and token copying scores are either negatively correlated (Llama-2-7b, OLMo-2-7b, and Pythia-6.96 or uncorrelated (Llama-3-8b). We take this to mean that concept copying and token copying are two distinct roles.
9.22, pw1.15c
t- p-0.817
Pythis p=0.00160
0.623
60000
0/011
00006
0:301
0
00000
0.00
-0.25 0.91
0.73 Coocept Capiigg Soone
6:55
-0.3
80
05
100
1e-3
Conneyt Drppiing
Cocagg CaporptCopptigg Scars
Sen4
Consup Coppang Scars
Figure 8: Concept copying scores plotted against token copying scores for every model. These scores are either not correlated negatively correlated. In other words, there few heads, if any, that are causally important for both random next-token copying and copying of future concept tokens.
Copylng Soctes for Llumo-2.77 Over Enthy
PlojoPM -2 -ungg) Ple Roomel
ExnryCorvsG Ploy -1 Parrge) PiolBanngl
0,0011
10
-00000
-0 0000
--00000
01002
-L0003
Figure 9: Probability differences when patching head activations for Llama-2-7b over entity tokens." The right-hand side shows concept copying scores; we do utilize the left-hand scores in this work. We patch at token position x, and measure increase in probability for e at that token position (left), and increase probability for C2 the following token position.
See Figure 2 and Section for details.
"Head 14.1 scores an order of magnitude higher than the second- highest scoring head on the right plof. Its concept copying score 0.0011. For visibility, we scale instead by the second highest concept copying score.
18

---

Published as a conference paper at COLM 2025
Copging
TorvinG Pinjoll Borngt PinlBannel
Piryjo Boragi PiryPoornel
+0001
CJEND
- 0.0005
-FLD010
-010000
-1L0013
0.0000
-4.0003
10
14 Hoad Badox
16
18
20
301
201
14
00
Beal Jackss
Figure 10: Probability differences when patching head activations for Llama-2-7b over random tokens. The left-hand side shows token copying scores; we do not utilize the right- hand scores in this work. We patch token position x, and measure increase in probability for i at that token position (left), and increase in probability for r2 at the following token position. See Figure 2 and Section 2 for details.
Copying
Tokees
Piojiel Porrgt)) Pie (Noes))
ENTIYYCOPVING Play Purrge) Plol
-8000
LJ3222
22 11
-00001
100
-C/0031
DAVE
10
0.0000
--0000
-L0002
-00000
Howd
Iuako
Boal Taakes
Figure 11: Probability differences when patching head activations for Llama-3-8b over entity tokens. The right-hand side shows concept copying scores; we do not utilize the left-hand scores in this work. We patch at token position 2,, and measure increase in probability for e at that token position (left), and increase in probability for C2 at the following token position. See Figure 2 and Section for details.
19

---

Published as a conference paper at COLM 2025
Copging
TORUNCOPPVN Ptr (off Hommsal Plomed
60033
10
100CS
--100003
Boad Irdoe
Bead Trakes
Figure 12: Probability differences when patching head activations for Llama-3-8b random tokens. The left-hand side shows token copying scores; we do not utilize the right- hand scores in this work. We patch token position x, and measure increase in probability for I at that token position (left), and increase in probability for r2 at the following token position. See Figure 2 and Section 2 for details.
Copying O1Mo-2-7b Fatiity
Plojull Porrgt) Pie (Noes))
ENTIYYCOPVING Play Perrye) Plol
0000
0/000
0,0022
CJ393)
0,0000
-00001
--0DE04
-U0000
-ILIEE
Howd
Iuako
Beal
Taakes
Figure 13: Probability differences when patching head activations for OLMo-2-7b over entity tokens. The right-hand side shows concept copying scores; we do not utilize the left- hand scores in this work. We patch at token position x,, and measure increase in probability for at that token position (left), and increase in probability for at the following token position. See Figure 2 and Section 2 for details.
20

---

Published as a conference paper at COLM 2025
Copying Secers OLM--2-7u
TORESCOPYING Pirija Poorge Piril
Pirzio Porrged PirilRermet)
0.000
11M5
0.000
23
16
-U0022
75
Figure 14: Probability differences when patching head activations for OLMo-2-7b over random tokens. The left-hand side shows token copying scores; we do not utilize the right- hand scores in this work. We patch token position X, and measure increase in probability for at that token position (left), and increase in probability for at the following token position. See Figure 2 and Section 2 for details.
Copying Stores for OLMo 1.1h Ove Darity Tokens
Piojal Bowng)) Pie. (Borgel
ENIT/YCOPYIN Play oM Parrge) PiclRewng!
-0100
10000
Last
-11000
--LDOO
Figure 15: Probability differences when patching head activations for OLMo-2-1b over entity tokens. The right-hand side shows concept copying scores; we do not utilize the left- hand scores in this work. We patch at token position X and measure increase in probability for at that token position (left), and increase in probability for C2 at the following token position. See Figure 2 and Section for details. Unlike other models in this paper, OLMo-2- lb has only 16 layers and 16 heads per layer.
21

---

Published as a conference paper at COLM 2025
Copying Secers OLMM-2-11
TORESCOPYING PirjjalAA Porrge) - Piril
Pirzio Porrgel PirslRornge)
Loos
--003
-0000
Figure 16: Probability differences when patching head activations for OLMo-2-1b over random tokens. The left-hand side shows token copying scores; we do not utilize the right- hand scores in this work. We patch token position X and measure increase in probability for at that token position (left), and increase in probability for at the following token position. See Figure 2 and Section for details. Unlike other models in this paper, OLMo-2- lb has only 16 layers and 16 heads per layer.
Copying Sccces for Pythin 6.th Ove Earlty Tokens
PiojollN Poong)) Poomee
Extr/CCpMnso Parrge Picy
0,000
RANG
-0,000
0,000
0L092
10
Maal
Tocks
Blad Tralos
Figure 17: Probability differences when patching head activations for Pythia-6.9b over entity tokens. The right-hand side shows concept copying scores; we do not utilize the left-hand scores in this work. We patch at token position 2 and measure increase in probability for e at that token position (left), and increase in probability for c2 at the following token position. See Figure 2 and Section for details.
22

---

Published as a conference paper at COLM 2025
Copying Score for Pythis-6990 Bls Random Tikems
ToRESCorYING - Pin Pinjee -t Poorel Pin (erre))
Piryje - Borngl) Pirs/oorael
0.033
-000015
10011
11
--0.00]
-0.00.
8
1
- 00003
-0.00
14 Real Inake
Hoad Iaake
Figure 18: Probability differences when patching head activations for Pythia-6.9b over random tokens. The left-hand side shows token copying scores; we do not utilize the right- hand scores in this work. We patch token position x,, and measure increase in probability for at that token position (left), and increase in probability for at the following token position. See Figure 2 and Section 2 for details.
B Token and Concept Induction Heads Throughout Training
We include analysis of causal and attention-based induction scores over time for OLMo-2-7b and Pythia-6.9b (models that provide access to training checkpoints). In Figures 19 and 21, show token and concept copying scores from Section 2 over checkpoints. Figures 20 and 22 show next-token and last-token attention scores from Section over checkpoints.
Token and Entity Copying Scores Over Training
Token Copter Heads
Oogeang
Copptng
0/0025
Kalls Draudngg
2.0055
0,003
80005
33865
Gaae
13001
-C.8GG
00000
10
137
12
Skipe
Concept Copter Heeds
Hoad
Chaggay Soures
01X0 Gagsany Sourse
C1Me22 Head 10.21: Copeing Seares
01.b-22 Ougyuny
1.903
Cigeul
13000
2,003
3/0/C
19827
00000
13001
-20804
0/632
Tramiing Kinge
Figure 19: Causal copying scores from Section 2 for OLMo-2-7b throughout training check- points. Top row: top-4 token induction heads. Bottom row: top-4 concept induction heads.
23

---

Published as a conference paper at COLM 2025
Next- and Last-Token Matching Scores Over Training
OLMoa Bead B: Altention Scores
0L Head 18.15 Arteuriun Souns
OLNo Bead 9.2 Attemiion Soores
Token Copier Heads
66-
3.000
60
2.00
10"
10*
Stace
10
33'
100
10
Concopt Copler Heads
01Mb-2 Head 10.12, Absntoo Scames
(1MMox2 Head 1511 Abvetion
01.0022 iteed Alberieen Strens
0.8
Luk 14
94
00
3.2
157
100
100
Troakin Stips
Figure 20: Attention-based matching scores from Section 3 for OLMo-2-7b throughout training checkpoints. Top row: top-4 token induction heads. Bottom row: top-4 concept induction heads.
Token and Entity Copying Scores Over Training
Token Copter Heods
Petlard Heal 1511 Oxpving
Head
Pythaa Hoad B.B: Coppzng Scerca
1.800
6.005
E3000
29091
Concept Copler Heads
Copyite
Ty-biad Blod 9. Copping
Pethia Shaad 13.28 Copptng Soves
08972
Eatn Contre
300000
1.000
0803
00030
00831
0,000
00031
30030
-c.00
1.00
237
200
10"
100
10'
25'
Trackng
Figure 21: Causal copying scores from Section 2 for Pythia-6.9b throughout training check- points. Top row: top-4 token induction heads. Bottom row: top-4 concept induction heads.
Next- and Last-Token Matching Scores Over Training
Taken Copier Heads
Pythoo-6.00 Sh Thead 15, 1h, Alteetinn Sconns
Pythio-C 6.9b Food B Amerrtia Scoces
Pythia 6:9b Boad 8.4: Altertion Scoeps
Pycais do Head 9.13 Amostior Sccees
00
107
107
107
15'
36
Trannng Sage
Concept Copier Heads
Pythoo
Pyttia
Pyslis 6.3b 5:3:
Pythis
033
Sosetan Soam
0.39
6000
6:35
0.5
6:65
6.55
Tra.ung Stoge
Tralsug Stgg
Figure 22: Attention-based matching scores from Section for Pythia-6.9b throughout training checkpoints. Top row: top-4 token induction heads. Bottom row: top-4 concept induction heads.
24

---

Published as a conference paper at COLM 2025
Llama-2-7b bf: 64 Token Copier lleads
Next bok Last-iok
Attostkka Marching
0.8
04-
00
(1),
(32, 201
(6 281
(11 101
(37 10)
(17, 22)
(26.11
(8 16.30
21
68 71
(11 20
(1%
171
220
(39 20
16
16
10 4
(17, 211
(08, 51
(12 21
(1% 300
(22. 10
122
(6 200200
200
on 37)
11
(2), 30
(21.21
(13 1m
(12, (12:20
18,
26.26
10,15)
(2n am
(27 (27.1214 1211
ita 14]
G.
118 10
201
G28 51
(24,
100. 21
Hledd Index (Dveriap with Concegt leods Uolded)
Figure 23: Next-token and last-token matching scores from Section 3 for the top-64 Llama-2-
7b token copier heads. Heads that are also in the top-64 concept heads are bolded.
Llama-2-7b-hd: Top-64 Concept Copier Heads
0.5
Next-tok
Attonties Matzing Scones
04-
0.-
0.1
(1
(34,
(11 (1
(13 231
(9. 291
(14,
110 201
(13 71
n1 14100
(8
(12 201
(22 171
(U8.)
[10 311
(12 1S)
(1) 20)
(18, [16 17)
(0.7)
(7, di
(19 12)
(39 di
116 16)
(1 12]
(3,38
(16,
(6,714
(13 200
()6.0)
(12, 1222
31
(13,
no 16)
(20
19 16:
(18.20 20
110
[1 10:
(24, 150
115 26
(16.7) 7:
07,220
(1
(1.214
00 22
(9 131
(35
017300 3m
(17.15) 290
(20. 221
(12 201
117 10
(9.28)
(13, 291
Head Index (Overiig with
Heade holded))
Figure 24: Next-token and last-token matching scores from Section 3 for the top-64 Llama-2-
7b concept copier heads. Heads that are also in the top-64 token heads are bolded.
C Next-Token and Last-Token Matching Scores
C.1 Attention Scores for Top 64 Heads
We include expanded versions of Figure 4 for the top-64 token and concept copier heads for each model. See Figures 23 through 30. We see that concept copier heads tend to have high last-token matching scores ($3), but also that each model has few heads that seem to do both token and concept induction. Heads that appear in the top-64 of both token-copying scores and concept-copying scores are bolded.
C.2 Correlations With Causal Scores
Figure 31 shows correlations between causal scores (Section 2) and attention-based scores (Section 3). For token induction, we compare token copying scores with next-token match- ing scores. For concept induction, we compare concept copying scores with last-token matching scores. Although these scatterplots appear quite noisy, there does seem to be significant correlation between attention paid to the next/last token and propensity to copy tokens/ tokens/entities entities respectively. Correlations for concept induction are comparable to correlations for token induction heads (which have been established in prior work).
25

---

Published as a conference paper at COLM 2025
Meta- L-at---lam-----B Top-od Token Copier Heads
Nect-tok Last-tok
Attontes Scores
0.8-
0.-
(1% 300
(33 241
(1K 231
(15, 201
(27 200
(15, (15,8)
(29 201
(5) M1
120 18)
(23 01
(16 31
037 11
(3. 221
(10) 121
127 10)
200
(3). 11
(21 21
(10 141
(24, 13)
(21) 221
(35,3)
(17 24
124 19)
(20 311
(20 201
(10, 13)
(16. (16
(M 201
(24 201
181.30 111311
d17 251
(18 41
n1 191
(20
(31
(29
(1S
(30
(35
120
128,
(20
022
(2%,
(25,
(03,
(22,
(2)
(17,
(22
Head Index (Overiap with Concept Heeds Bolded
Figure 25: Next-token and last-token matching scores from Section 3 for the top-64 Llama-3- 8b token copier heads. Heads that are also in the top-64 concept heads are bolded.
Meta-I.lmma33 8B: Top. 64 Concept Copier Heads
0.7
Noxt-tok
Last-lok
Adteation Katshing Soores
0.5
0.3
0.2
0.0
(22, 20)
321 1)
(16 25)
(1D. 1)
(15.100
(20. 01
(15-211-
19 277
(23.0)
113 12
(10.1) 1)
(1S, 17)
[15:11] 21)
(14-31)
(15 3)
[10. 200
(22, 19)
1M 113
10)
126. 1)
114 20)
(21, 10)
113 ()
11)
14)
16)
(19) 113
13)
(6. 2)
(b. 11
(23. 25)
(20 20)
(14 22)
pa 11)
(15 2)
10 00
(24 12)
(L 0
(17, 10)
(to. 10)
63,
(0)
19
10.
cio
Heed Inces (Ovedlag with Tekon Heads Bolded)
Figure 26: Next-token and last-token matching scores from Section 3 for the top-64 Llama-3-
8b concept copier heads. Heads that are also in the top-64 token heads are bolded.
OLMo 2-1124-70: Top 64 Taken Capier Lleads
Next
too
0.8
Attestion Maitcing Scoees
0.6
0.2
(23. 20
(16 10
(0)
01. 11
(25.01
(30 25)
(90
(31 25)
(3, 20
(22
(30, 41
(31.01
(27.
117 313
(21. 24)
19 2h1-
(30) 17)
(2) 14)
(32,21) 201-
271
(8. 100
128 111
(22 200
311
(26,30
171
06 ti
(10 18
(31, 11
(30 0-
(2.24
(20,200
(21, m
03.26) 20
(20) 221
1.3311 10
123,
03 291
(3) 201
(2) 271
(30, 71
(27112
84,10
19
(25 10)
no 10
2h.15 13
201
as. 1m
(29 211
(24,
G3. 20
Head Indox (Overlap with
Bodoa
Figure 27: Next-token and last-token matching scores from Section 3 for the top-64 OLMo-2-
7b token copier heads. Heads that are also in the top-64 concept heads are bolded.
26

---

Published as a conference paper at COLM 2025
OLMe-2-1124-7B Top-64 Concept Capier Heads
Nect-tok Last-tok
0.7-
0.6
04-
Attontes
02-
(1
(15
de 211
(17, 31)
(21,
117,18
(11
118
11 70
(18 221
(6 28
(19,
(18,
(15 19)
(112 01
016, 11
(17
(11
(0
115
122 101
(16, 18)
12% 21
(19.
(10 211
(18 01
(11 31
(28, (28.17) 171
02, 211211--
118
(10 221
15
(30 8
11 171
(7.
(1)
(15 700
G. 19)
115.16 16)
(2% 131
17, 28.20
(14
(2) 22
17.00
200
(19
(34
(12-11
(6 111
(17 21
018 71
Head Index (Overlag with Token Heads Bolded)
Figure 28: Next-token and last-token matching scores from Section 3 for the top-64 OLMo-2-
7b concept copier heads. Heads that are also in the top-64 token heads are bolded.
pythia-6.9b: Top-64 Tooken Copier Heads
Noxt-tok
0.3-
Last-lok
0.5
Adtontion Katching
0.0
(8. 4.
18. 10
(0. 15)
17 76)
(10 22
(20. 177
(N 15)
(8 289
(10.22)
100. 100
(31 4)
(% 200
19) 19)
18 17)
100
1th 24)
10. 300
(0. 1)
99
(7, 77.10-
100
127 2)
(15, 109
110 1)
(11.22)
(20. 135
(27) 200
(10. 14)
25)
15 20)
130
[17, 22)
(11 3
114, 30
11)
20
(1,110)
(1. 50
(20. 31
(12. 51
(10. 10)
15)
10)
127 8)
24
(9. 31
(29, 10)
112 m)
(15,
(55
113
(6
Haod Index (Ovrriap with Concept Hendx Roldwa)
Figure 29: Next-token and last-token matching scores from Section 3 for the top-64 Pythia- 6.9b token copier heads. Heads that are also in the top-64 concept heads are bolded.
pythin-6.9b. Top-64 Concept Copler Hoads
Next
0.6-
0.1
0.3
0.1
03 241
12)
(N, 9
03:20 201
IN 13)
(13 201
20)
02 11
(11
(9 23:
177
(11,24)
099 200
6, 19)
(12 9-
114,8)
(0.50) 311
n.. 00
(19 20
(122
13. 00
(26,
(3. 10
011 10
13)
(21
01. 25
(12.3) 31
60, 13
(35.1 15)
011211 21
13320)
03. 23
(20 fM)
017 15
(18 10
16 0
16,71
07.300 201
(1) 30
63 15)
(4 14
07:20 2m
10. 22)
18.23
(20,
(15 m
(11.3
Head Index (Overtap with lleads Bolded)
Figure 30: Next-token and last-token matching scores from Section 3 for the top-64 Pythia-
6.9b concept copier heads. Heads that are also in the top-64 token heads are bolded.
27

---

Published as a conference paper at COLM 2025
2-15-3
4.02.0
0.03
0.004
0,000
0002
8.010
0.005
0102
0.0000
0.000
0.
04
26
6.
28
04
24
02
04
Cleme 2/ tr-0 (e1.0e661)
Mota Laree 88 io 5o, pw2 20
0L 1124 (uw 33, pwdt
gythin (owd be
3.310
28906
EXAM Chaling
33005
00033
33000
-64032
38005
Camb-hore Massiigg Store
Lan Manciing Store
Figure 31: Relationship between attention-based matching scores (Section 3) and causal copying scores (Section 2). Top: correlations between next-token matching scores and token copying scores for each head, indicators of token-level induction. Bottom: correlations between last-token matching scores and concept copying scores for each head, indicating concept induction.
Number of Everlapping Heads
40
Llama-2-7b
1Jama-3-8b
30
OLMo-2-7h
6yth
20
10:
2
322
64
Top-K Hoads Chosen
Figure 32: Number of overlapping token and concept copying heads for increasing values of k. For smaller values of k, OLMo-2-7b and Pythia-6.99 have more overlapping heads than Llama models.
28

---

Published as a conference paper at COLM 2025
D Lesioning Concept and Token Copier Heads
D.1 Full Results
We show the results of ablating token copier heads and concept copier heads on our "vocabulary list" tasks for all four models in Figures 33-37. We also show the number of heads that overlap for increasing choices of k in Figure 32.
Llama-2-7b: Token Heads
Llama-2-7b: Concept Heads
1.0
nonsense aco=0.06
Perceet Orrggnal Artureey
Percott of Orogiial Accurscy
CAPS a.c
0.8
Utll a.c-0.99
0.5-
syscoym acc= 0.000
Bhen 300-0.56
0.4
0.4-
pt 1t.on_3ch ann_scom..0 0.50
[ten ace=0.55
0.2
0.0
0.3-
50
100
150
200
230
300
150
200
Top Tolows Induction Haads Meun-Ableied
Contapt
Figure 33: Mean-ablating the top-k copier heads for "vocabulary list" tasks in Llama-2-7b. As results shown as percentage of original task accuracy, we display these full model accuracies in the legend. Even though synonym accuracy is initially low, it is still unaffected by ablation of token induction heads.
Llama-3-8b: Token Heads
Llama-3-8b: Concept Heads
1.0
copy
Percent of Original Accunnty
CAPS, aoc-0.3 0.97
Peroent of Original Armmney
Utin, scow1.1 1.0
antonym
synosym aec=-.11
0.0
friaa aco-0.51
ee-en =--m_acc:0055
pe-en, arem0.50
Jt. es, a05-0.53
0.2
0.0
0.0
50
150
200
250
300
50
300
Tokeen Induction Elends Mean-Ablated
Induction
Figure 34: Mean-ablating the top-k copier heads for "vocabulary list" tasks in Llama-3-8b As results shown percentage of original task accuracy, display these full model accuracies in the legend. In contrast to Llama-2-7b, capitalization tasks seem to use a blend of concept induction and token induction heads.
29

---

Published as a conference paper at COLM 2025
OLMo-2-7b: Token Heads
OLMo-2-7b: Concept Heads
1.0
tonsense_acoe 0.000.50
espy_auc=11
Porcent of Orgginal Accuzary
Acturacy
GAPS acc-0.85
0.5
0.8
title
antorym. a0c-0.11
0.65
synouyaa oco-6.13
acc-0.48
wwvtr mnos=0u4
p--n noc.m.ck 44
0.2
accr0.36
0.0
0.0
150
1000
150
200
250
300
Top Induction
Concept Industion Uleads Meen Ablatte
Figure 35: Mean-ablating the top-k copier heads for "vocabulary list" tasks in OLMo-2-7b. As results are shown as percentage of original task accuracy, we display these full model accuracies in the legend. Although English copying accuracy remains high in the right figure, ablation of concept heads in OLMo-2-7b does damage nonsense token accuracy; this indicates that there may be more overlap between these heads for OLMo-2-7b than for Llama models.
OLMo-2-1b: Token Heads
OLMo-2-1b: Concept Heads
nonsemse aco-0.92
1.0
acc-0.99
Percent of Origital Accucary
oogy m.001.0
Purcent of Original Accuracy
copy, s
0.8
CAPS a.cm0.9
0.0
CAPS acem0.9
fr on, acp=0.37
fr on. acc=0.37
0.6
0.6
0.4
0.4
0.2
0.2
0.0
1.0
20
30
40
50
70
10
30
Top Token Induction Heads Mean Ablated
Top Concept Induction Heads Mean Ablated
Figure 36: Mean-ablating the top-k copier heads for "vocabulary list" tasks in OLMo-2-1b. As results are shown as percentage of original task accuracy, we display these full model accuracies in the legend. To be consistent with larger models, we show approximately the top 30% of heads.
Pythia-6.9b. Token Heads
Pythia-6.9b. Concept Heads
1.0
honsenso 30c=1,0
coce 1.0
Percent of Original Accurscy
title aoc-0
antonym acp-0.11
Original
0.8
0.5-
synorym s0cm0.014
free acce0.42
0.4-
0.4
pt
2. as, aoc-B.4
0.2
0.2
woo-0.3
0.0
0.0
50
100
150
200
250
300
100 150
200
250
300
K'lbben Indictton Heaas Moan Ablated
Ucncept Induction Uteds Mran-Abiat Ablated
Figure 37: Mean-ablating the top-k copier heads for "vocabulary list" tasks in Pythia-6.9b As results are shown as percentage of original task accuracy, we display these full model accuracies the legend. Unlike previous models, semantic tasks (translation, synonyms, and antonyms) are also damaged when we ablate token induction heads (left). However, capitalization and English copying accuracy remains relatively high. Because Pythia is trained on order of magnitude fewer tokens than other models in this work (0.3 trillion, versus 2 2 trillion), may be that concept induction and token induction are somewhat fused.
30

---

Published as a conference paper at COLM 2025
D.2 Examples of Prompts
We show examples of two semantic task prompts from Section 4 in red boxes below. We also show examples of verbatim tasks in blue.
Vocabulary List Prompt Example (French-English)
Vocabulary List Prompt Example (Antonym)

 French vocab:

 Words:
1. completement
1. hydrous
2. incosttitutionsaliee
enlarge
3. flocon
3. alternative
4. chiens
4. messy
5. vaccinés
5. traditionalism
racontée
simplicity
7. spécialités
7. underwater
8. parfumée
8. transitive
9. condensateur
9. omit
10. saumons
10. contract
English translation:
Antonyms:
1. completely
1. anhydrous
2. unconstitutioonaltyy
2. cut
3. flake
3. primary
4. dogs
neat
5. inoculated
rationalism
6. told
6. difficulty
7. specialtics
7. amphibious
8. fragrant
8. intransitive
9. capacitor
remember
10.
10.
Vocabulary List Prompt Example (English)
Vocabulary List Prompt Example (Nonsense Copying)
English vocab:
Vocab;
1. live
any insp
2. begin
2. comes look
3. stumble
3. the like
4. good
4 peints__
5. ostracize
5. fix
6. important
$$_
7. coin
enec Camer
8. col cred
8. there
9. manner
9. object currently
10. recover
ercase
English vocab:
Vocab:
1. live
1. any insp
2. begin
2. comes look
3. stumble
3. the like
4. good
4 4.points_
5. ost
5. fix
6. important
6. $$_
7. coin
e.ence Camer there
colored
manner
object currently
10.
10.
31

---

Published as a conference paper at COLM 2025
Word Length =1
Word Length 2
Word Length =
Token Copier Heads
20-14
100
Tap
Concept Copier Heads
2/1
34
Figure 38: Ablations for Llama-2-7b, controlling the number of tokens in each word. When translating single-token French words to English, ablation of concept heads is less effective, suggesting that token heads may also do semantic copying when words are restricted to single tokens. However, the separation is still apparent, and is even clearer for two- and three-token words.
D.3 Word Length Breakdown
We run the same ablation experiment as shown in Figure 5 for Llama-2-7b, except we control the number of tokens in each word. We only on subset of representative tasks and for и = 256 prompts per task. Figure 38 shows these results. While single-token words still have a similar pattern multi-token words, the effect is less drastic, likely because the difference between concepts and tokens is less clear for single-token words.
D.4 Qualitative Examples
We show examples of model generations with token induction heads ablated, expanding on examples shown in Section 4.2. We generate text with token induction heads mean-ablated at all token positions using the nns igt. LanguageModel generate function (Fiotto- Kaufman et al., 2025), with default sampling. For the first three models, ablating token induction heads causes the model to start paraphrasing instead of directly copying, for natural language (Boxes D.1, D.2, D.3, D.4) and code snippets (Boxes D.5, D.6, D.7).
32
We notice that for k = 32, Pythia-6.9b's "paraphrases" are much poorer than other models (Box D.3, D.7, D.11). We scale back to = 16 and provide more than the first token of the copied sequence in Box D.4, finding that this causes Pythia-6.9b to copy more faithfully.
Interestingly, when paraphrasing a Python snippet, Llama-3-8b (Box 4.2) is the only model to write snippet that is "correct," " i.e. semantically equivalent to the original input.
We also show what model outputs look like for vocabulary list prompts: Llama-2-7b generates sentences that use previously-seen words (Box D.8), Llama-3-8b copies the list but without numbering (Box D.9), and OLMo-2-7b (Box D.10) and Pythia-6.9b (Box D.11) fail to copy any of the previously-seen words.
32

---

Published as a conference paper at COLM 2025
Box D.1: (Llama-3-8b) Original Model vs. Top-32 Token Induction Heads Ablated
I have reread, not without pleasure, my comments to his lines, and in many cases have caught myself borrowing a kind of opalescent light from my poet's s fiery orb I have reread, not without pleasure, my comments to his lines, and in many have caught myself borrowing kind of opalescent light from my poet's fiery orb
I've reread my comments on his lines, and found that I'd borrowed from his fiery orb kind of opalescent light
Box D.2: (OLMo-2-7b) Original Model vs. Top-32 Token Induction Heads Ablated
have reread, not without pleasure, my comments to lines, and in many cases have caught myself borrowing a kind of opalescent light from my poet's fiery orb I have reread, not without pleasure, my comments to his lines, and in many cases have caught myself borrowing kind of opalescent light from my poet's fiery orb
I have read his lines again, and I have not found them without pleasure. In many cases, I have caught myself borrowing kind of opalescent light from his fiery orb.
Box D.3: (Pythha-6.9b) Original Model vs. Top-32 Token Induction Heads Ablated
I have reread, not without pleasure, my comments to his lines, and in many cases have caught myself borrowing a kind of opalescent light from my poet's fiery orb I have reread, not without pleasure, my comments to his lines, and in many cases have caught myself borrowing kind of opalescent light from my poet's fiery orb
T have reread my comments on his lines, and I have found them to be, in general, and
have found myself to be agreement with them.
Box D.4: (Pythia-6.9b*) Original vs. Top-16* Token Induction Heads Ablated
I have reread, not without pleasure, my comments to his lines, and in many cases have caught myself borrowing kind of opalescent light from my poet's fiery orb have reread, not without pleasure, my comments to his lines, and many cases have caught myself borrowing kind of opalescent light from my poet's fiery orb
I have reread, not without pleasure, my comments to his lines, and have caught myself in many cases, borrowing kind of opalescent light from my poet's fiery orb
Box D.5: (Llama-2-7b) Original Model Top-32 Token Induction Heads Ablated
foo []
foo = []
for in range (Len (bar))
for i in range (len (bar) if i 2 0: f (bar [i]]
if 8 0:
coo.append (bar [i])
foo []
foo = [item for sublist bar for item in sublist if i 2 == 0]
for i in range (len (bar))
if & asdf append (bar [i])
33

---

Published as a conference paper at COLM 2025
Box D.6: (OLMo-2-7b) Original Model vs. Top-32 Token Induction Heads Ablated
200 II
for in range (1en (bar)):
f00 II
if $ 2 == 0:
for i in range (Len (bar)
foo. append (bar [i])
if i 2 0:
foo
append (bar [i])
for range (len (bar)) )
foo = [x for in bar if x 2 == 0]
if 8 == 0:
asdf append (bar [i])
Box D.7: (Pythia-6.9b) Original Model vs. Top-32 Token Induction Heads Ablated
foo for range (len (bar))
foo []
for i in range (len (bar) )
if & = 0:
if 2
append (bar [il)
foo append (bar [i])
foo for range (len (bar))
foo foo. aboend (foo)
if & = 0: asdf append (bar [i])
print (foo)
Box D.8: (Llama-2-7b) Original Model vs. Top-32 Token Induction Heads Ablated
English vocab:
1. hospital
4. lesbian
2. visage
5. know
unlucky
6. wizard
lesbian
7. unfair
know
8. sound
wizard
9. cortical
7. unfair
10. obsolete
sound
English vocabulary:
cortical
1. She was hospital for a long time after
10. obsolete
the accident.
English vocab:
2. Her face was covered in bandages.
1. hospital visage
3. The accident very unlucky.
4. She was lesbian.
unlucky
5. He knows everything about the wizard.
lesbian
6. It is unfair to treat people like that.
5. know
7. The sound of the engine was very loud.
wizard
8. don't know why the doctor told that should more fish.
unfair
sound
9. The new cortical implants can help the blind to see.
9. cortical
10. obsolete
10. The
34

---

Published as a conference paper at COLM 2025
Box D.9: (Llama-3-8b) Original Model vs. Top-32 Token Induction Heads Ablated
English vocab:
English vocab:
1. hospital
1. hospital
visage
2. visage
unlucky
3. unlucky
lesbian
4. lesbian
5. know
5. know
6. wizard
6. wizard
unfair
7. unfair
sound
8. sound
cortical
9. cortical
10. obsolete
10. obsolete
English vocab:
English vocab: hospital visage
hospital
visage
unlucky
unlucky
4. lesbian
lesbian
know
know
wizard
wizard
unfair
unfair
8. sound
sound
cortical
cortical
10. obsolete
obsolete
Box D.10: (0LMo-2-7b) Original Model vs. Top-32 Token Induction Heads Ablated
English vocab:
1. hospital
English vocab:
2. visage
1. hospital
3. unlucky
2. visage
lesbian
3. unlucky
know
4. lesbian
6. wizard
5. know
7. unfair
6. wizard
sound
7. unfair
cortical
8. sound
10. obsolete
9. cortical
English vocab:
10. obsolete
1. hospital visage
English French French to English
unlucky
English to French
lesbian
French to English
5. know
German to English
wizard
Russian English
unfair
German to Russian English to German< endoftext
sound
9. cortical
10. obsolete
35

---

Published as a conference paper at COLM 2025
Box D.11: (Pythia-6.9b) Original Model vs. Top-32 Token Induction Heads Ablated
English vocab:
English vocab:
1. hospital
1. hospital
visage
2. visage
unlucky
3. unlucky
lesbian
4. lesbian
know
5. know
6. wizard
6. wizard
unfair
7. unfair
sound
8. sound
cortical
9. cortical
10. obsolete
10. obsolete
English vocab:
English vocabularies
hospital
English-French
visage
An English-French dictionary
unlucky
An English-French dictionary
4. lesbian
An English-French dictionary
know
English-French dictionary
wizard
English-French dictionary
unfair
An English-French dictionary
8. sound
An English-French dictionary
cortical
English-French dictionary
10. obsolete
36

---

Published as a conference paper at COLM 2025
E
Concept and Token Lens
E.1
Approach
Every attention head in an autoregressive transformer can thought of as "reading" some small amount of information from all previous token positions and subsequently "writing" that information back into the current residual stream. Can we visualize the semantic information that concept induction heads are contributing to the residual stream?
Let d be a model's hidden dimension and m < be the dimension of a single head. We rely on key insight from Elhage et al. (2021): that the value and output projections for a particular head It at layer I, V(aaa) E R (mma) and ((t)) E R (d.m) respectively, are solely responsible for whatever information head writes into the residual stream. Specifically, they point out that the product of these two matrices (1,)) ((()) is low-rank d x matrix (at most rank m) that determines the effect of head kl) on the residual stream. In other words, multiplying a hidden state xj by this matrix extracts whatever information within x that this head typically contributes to the residual stream.
To build a concept lens Lc, E (ddd) that reads from all of the concept induction head sub- spaces simultaneously,
combine the weights from the top-k concept induction heads Cj- We choose k = 80 based on results from Section 6, and calculate the sum of all concept OV matrices:
La= E 1 V(In) (LA)EC
(5)
If all attention heads in C are in the same layer, L is mathematically equivalent to taking the sum of the outputs of those attention heads. However, we also allow for summation of heads across layers, which was empirically effective in prior work (Todd et al., 2024), possibly because transformer representations are interchangeable in intermediate layers (Lad et al., 2025).
Finally, we can project Lç, Xi to token space by applying the model's final normalization module and decoder head (nostalgebraist, 2020). This approach works for any set of heads: can create token lens Lr, R (d,d) out of the top-k token induction heads, or baseline lens L E (d.d) as the sum of all OV matrices in the model.
E.2 Lens Output Examples
Figures 39 through 42 show concept lens outputs for three different models. Compared to a baseline lens that sums all OV matrices (Figure 44), these plots reveal differing semantics of the same token in three different contexts. We also show examples of token lens in Figure 43, which appears ineffective for inals (the last token of multi-token word) but clearly reveals token-level information at other positions. We posit that this discrepancy is consequence of the "token erasure" effect found by Feucht al. (2024).
37

---

Published as a conference paper at COLM 2025
Teant meyt
NC leng.
for ... The
35540
500
04
062
Locdl
14
Figure 39: Concept lens outputs for Llama-2-7b. We multiply the hidden state for inals at every layer by Lc, (Equation 5) before projecting to vocabulary space. Applying this lens reveals the semantics of the token inals, which depends on the context.
Cantua
Miroed
04
Mhane
Cacoal
Ney
Hight
Figure 40: Concept lens outputs for Llama-3-8b. We multiply the hidden state for inals at every layer by Lc, (Equation 5) before projecting to vocabulary space. Applying this lens reveals the semantics of the token nn ls,s, which depends on the context. Unlike other models, early-middle layers are less decodable than middle-late layers. STL is likely reference to the St. Louis Cardinals.
38

---

Published as a conference paper at COLM 2025
Phet
STL
Lew care
1oog
Lm000
Jence
kmp
Figure 41: Concept lens
lens outputs for OLMo-2-7b. We multiply the hidden state for inals at every layer by LG, (Equation 5) before projecting to vocabulary space. Applying this lens reveals the semantics of the token inals, which depends on the context. Unlike Llama models, we must add "STL" to the leftmost prompt to decode semantics related to sports.
The Canvent Levs Oary
The
seang
Livee
Berdys
Figure 42: Concept lens outputs for OLMo-2-1b. We choose 20, ie. the top 8% of heads, to be consistent with larger models. Unlike larger models, the signal provided by concept lens is quite noisy. However, for a simple multi-token concept like "New York", concept lens still reveals more semantic information than a baseline sum of all OV matrices.
39

---

Published as a conference paper at COLM 2025
<B> in the morning ais, she heard nerthern card, inals
tokke lens/northern)
token_Jnnsfcard
token lans( nals)
Figure 43: Token lens outputs for Llama-2-7b at three token positions. Applying token lens reveals the token that corresponds to given hidden state. We multiply the hidden state for inal ma at every layer by Lr₃ before projecting to vocabulary space, with k = 80. Token lens is not effective for (inals) 1s), likely due to the "token erasure" effect for multi-token words described by Feucht et al. (2024).
Foor the Mar
ThA CATT (HA)) Indiaa
The
4-
2-
54
198
1008
390
3000
Figure 44: Baseline "all heads" lens outputs for Llama-2-7b across three prompts. We multiply the hidden state for inals at every layer by L, the sum of all OV matrices in the model, before projecting to vocabulary space. Although we can still observe some semantic information when using all heads, concept lens (Figure 39) provides cleaner signal.
40

---

Published as a conference paper at COLM 2025
F Concept Induction is Language-Agnostic
We show results from Section 6 for more languages, with both Llama models, Figure 45 shows results for the same experiment in Figure 6, expanding evaluation to Llama-3-8b and two more language sets: patching from French-English into German-Russian, and patching from Russian-Spanish into English-Japanese. Figure 46 shows results for smaller model from different family, OLMo-2-1b.
Liama-2-7b-hf: je-2b
Llama 2-76-u- fi-en de-tu
Llema-2-7b-bff rures en-je
Naturscy
0.50
0.50
Wy
0.25
0.25
Wa
0.00
0.00
0
25
75
125
Top k Concept Induction Heads Patched
Top-k Concept Induction Heads Patched
Top Concept Induction Heeds Palched
Meta-Llame-3-44 es-it jezzh
Meta-Lloma---888 fr-en_de-ru de-ru
Meta-Llama-3ee ruees, ea-ja
Acturary
0.5
0.5
0.5
0.0
0.0
0.0
25
50
75
100
125
25
50
75
100
125
25
50
100
125
Induction Heads Patched
Top-k Concept Induction Heads Patched
Induction Heads Patched
Figure 45: Results for patching the top-k concept induction heads from one translation prompt to another. Top-left is the same plot as Figure 6. We evaluate Llama-2-7b and Llama-3-8b on Spanish-Italian Japanese-Chinese, French-English -> German-Russian and Russian-Spanish - English-Japanese, Interestingly, the language set for which concept patching is the least effective is the same language set for which FV head patching is most effective.
OLMo-2-0425-1B. es-it, es-it_ja-zh
OLMo-2-0425-1BB fr-en de-ru
0.2
W, : lout) g
Accuracy
Accuracy
0.05
0.1
W; (out) loutt)
0.0
0.00
0
25
50
75
100
125
25
50
100
125
Top-k Concept Induction Heads Patched
Top-k Concept Induction Heads Patched
We also include results for OLMo-2-1b 1b, patching from Spanish-Italian - Japanese- Chinese and French-English - German -errman-Russian. The effect of patching concept heads is less clean than for larger models. This may be due decreased separation between token and concept induction, or because of OLMo-2-1b has lower overall translation performance. Like larger Llama models, the latter language pair -i--nn) shows weak results. Gray dotted lines indicate base model accuracy for Japanese-Chinese and German-Russian translation respectively.
Figure 46:
41

---

Published as a conference paper at COLM 2025
G Function Vector Versus Concept Induction
As mentioned in Section 7, we find weak correlations between FV and concept copying scores (Figure 48). We also find that ablation of FV heads also causes a large drop in performance for vocabulary list tasks that cannot otherwise rely on token induction heads (Figure 49). However, this does not necessarily mean that FV heads play the same role as concept induction heads. As Section 7 explains, patching FV heads and patching concept induction heads for the same prompt yields very different results. While patching FV heads changes the language that the model outputs (depending concept induction heads changes the meaning the output word. In order for the model to do semantic copying tasks, it needs both FV heads and concept heads-thereforr, if either are ablated, we see the same drop in translation accuracy. We show results for patching FV heads for Llama models for two language sets in Figure 47.
the language pair), patching
Llama---77bhh fi-en_de fr-en_de-ru de
Llama 2-7hh----hhh es es-it_ja-zh ju-zh
0.8
0.8
Wy Wa::fp
Acnurany
0.8
Accuracy
0.6
Wy : 43
0.4
Wa g
12
25
50
100
125
25
75
100
125
Top-k
Top-k Heads
1aaaa-Llama---880 fiven de
Mela: Maata---ama-33331 3-8B: es-il_jaezih
0.8
0.8
Accuraty
0.5
Hy gost)
0.6
M, d
My 1/
My 1/
0.2
0.2
0.0
100
25
75
Top-k Function Voctor
Patched
Top k
Vector Hoads Ratched
Figure 47: Results for patching the top-k FV heads from one translation prompt to another. Top-left is the same plot as Figure 7. Patching FV heads from Spanish-Italian to Japanese- Chinese does not flip the output language to Italian, but patching from French-English to Cerman-Russiant flips the output language to English.
42

---

Published as a conference paper at COLM 2025
Llama-2-7b: r=0.57, p=2.33e-89
Llama-3-8b: r=0.39, 26-386
0.13
Function Ventor Score
0.04
Fanction Vector Score
0.03
0.10
0.02
0.05
0.01
0.00
0.00
-0.01
-0.25
0.25
0.75
Fintlty Copying Score
le-3
Entity Copying Score
1e-4
OLMo-2-7h: r-0.27, 5.01e-19
Pythia-6.9h r=0.23, p=1.333-111
0.06
Fanction Score
0.10
Function Vector Score
0.08
0.02
0.04
0.02
0.00
-0.02
0.5
Entity Copying Score
1e-1
Entity Copying Score
1e-3
Figure 48: Correlations between FV scores and entity copying scores for all models. While we do find significant correlations, they are strongest for Llama models with strong outliers, and do not seem particularly strong for other heads and models.
Llama-2 7h-hf: Ablated Performance
Mata: Llama 3 Ablatnd Performance
1.0
Nassense Copylig
Dopying
English Cupsing
1.0
linglish Copying
0.88
Title Cosing
Uppercasing
Percent ef Oinginal Accurscy
0.8
Uppercasing
Casing
Antonyms
Antonyms
Orginnal
Synonyms
Synoryms
Translation
0.5
Translation
0.4
0.2-
150
Top Panction Vector Heads Mean Ablated
Top K. Punction Vector Heads Mean Ablated
OLMo-2-1124-7B: Ablated Performance
pythis-6. 6.9b: Ablated Performance
1.0
Nonsense Copying
1.0
Copying
English Copsing
English Copylag
of Original acameyy
0.8
Title Castng
Uppercasing
Percest of Original acturscy
0.8
Uppercasingg
Castng
Antongms
Antonyas
0.6
Synotyms
0.5
Synntimms
Translation
Translation
50
100 150
200
250
300
100
150
200
250
300
Top K Function Vector Heads Mean-Ab
Thp Fonction Victorr Heeds Meenn-bllieed
Figure 49: Ablation experiment described in Section 4 where ablate the top-k function vector (FV) heads. We see that FV heads are also vital for semantic copying tasks. As Section 7 describes, this result likely not due to an overlap with concept induction heads, but rather because FV heads help the model output the correct language (whereas concept heads copy the word meaning).
43


Reviews:
[{"review_id": "FmYg0oxNPM", "reviewer": "Reviewer_3CWa", "summary": "This paper explores LLM copying behavior, expanding on prior work on induction heads. This paper fits into the “careful and deep analysis” dimension for COLM contributions. While there may be some near-term technological impact from the results, the main contribution is a deeper understanding of components of LLMs and how they impact LLM output. The insights may also contribute to understanding of translation processes in LLMs and will likely also inspire additional research and evaluation across a wider range of languages in the future. The paper uses established approaches to measure effects of causal interventions and ablations to examine the behavior of LLM heads.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a thoughtful piece of work that begins with a clear presentation of the topic that the paper will explore. It makes analogy to concepts from psychology (while walking a reasonable line not to make assumptions that the model behavior is identical, just analogous in some ways) and uses clear examples to set the reader up to understand the main concepts before following up with clear and careful definitions.\n\nThe experimental setup is clear and generally seems appropriate. I found that the writing did a good job of anticipating and addressing questions I was likely to have as a reader, such as controlling for word length and exploring casing-related tasks (I was thinking of some issues that arose in https://aclanthology.org/D18-1339.pdf, though that examined other copying-related topics, with pre-Transformer architectures).\n\nThe work does a good job of balancing the broader picture through results across full datasets and giving the reader specific qualitative examples.", "reasons_to_reject": "There are some areas (see questions below) that could use additional clarification. There are also some areas that are quite dense; it may be appropriate to provide the readers slightly more detail on approaches like patching as they are introduced.\n\nThe claim that “Concept Induction is Language-Agnostic” may be somewhat too strong; the languages evaluated in this are all fairly high-resource (though this is somewhat acknowledged in the cited work). It would be helpful to do additional evaluation with very low-resource languages. That said, this is beyond the scope of this paper to fully explore, and I think publishing this paper is likely to inspire future work that takes a similarly careful look at this question.\n\nThe section on Function Vector Heads feels like it arrives quite suddenly and it would be nice to have an expanded analysis of the relations between the different types of heads explored, given additional space.", "questions_to_authors": "Questions and Comments:\n\n- Line 31: This should probably be “pommes de terre” (not “des”)\n- Line 73-74: “resampling” – to clarify, this is a sampling as described in the footnote, not simply a shuffling?\n- Line 74: Should $y_{n+l}$ be $y_{n+m}$ ?\n- Perhaps move lines 87-92 earlier in this section, e.g., between 72 and 73? (Not completely necessary and may not flow well, but I found myself wanting to be sure I understood what kinds of concepts were going to be used around that earlier point.)\n- Figure 3: Consider using different shapes for the Token/Concept (will be clearer to readers who print in grayscale).\n- Line 125: I was slightly confused by the description “insert the concept on the left”\n- Figure 4: The two colors are very similar when printed in grayscale.\n- Figure 5: Nearly impossible to read in grayscale. Would be nice to use different line patterns to distinguish.\n- Figure 6: I found the parenthetical in the caption somewhat confusing and had to reread the section of the text to be sure I understood it. Perhaps there is a way to rephrase. If there’s not an easy way to fit a gloss into the figure, it may help to give English glosses in the caption.\n- Figure 7 appears quite a bit before Function Vectors are first discussed, so it may be helpful to either move that or expand the acronym the first time that it appears in the caption.\n- Lines 258-260: “FV heads cause the model to output the same concept in a different language, whereas concept induction heads cause the model to output a different concept in the same language” – should “patching” appear somewhere in this sentence?\n- Line 683, box D.1: Is there a reason (other than space-saving) that you didn’t split these into two separate figures? As a reader, my eyes went (very confusingly) from “1. completement” to “1. hydrous”. (I think especially because the coding examples do expect you to read more in that manner.)", "extra_scores": {}}, {"review_id": "T7h9Obo7dC", "reviewer": "Reviewer_ZnwZ", "summary": "This paper introduces the concept-level induction at a multi-subword token level. The paper first measures the causal interventions effects by measuring the differences between concept vs token-level copying behavior. Then the next part of paper compares the difference between the concept-level vs token-level attention activation with respect to previous or next token and they showed that the concept level activation heads attends to the previous token. Next using various word pairs and perturbations/lesioning the word pairs to show that different pertubations affects tokens vs concept-level induction differently. And after, the paper did an experiment to patch/transplant the concept induction head from a translation prompt to another translation prompt for a different language pair and showed that the translation in the transplanted induction head can correct predict next token in the different language pair, proving the concept-level induction head as language agnostic (but this is only effective up to the top-80 concepts). Last hypothesis that the paper tested was the idea of function vector vs concept-level heads, the function level heads were supposed to be task-specific, some inconclusive results here and there for this part of the paper but the authors have shown how FV heads and concept-level heads perform differently.", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Really good job done at breaking down the proposal into provable hypotheses and gradually showing the effectiveness of the concept-level head empirically in each hypothesis test\n- A lot of effort was put into the ablation studies for each hypotheses (though not in the main paper, but it's a comprehensive appendix of results) \n- The proposed concept-level head was shown to behave consistently across various LMs (Olmo, pythia, llama)", "reasons_to_reject": "- Nothing, good paper, clearly written, will fight to have it accepted.", "questions_to_authors": "Food for thought: When the feature vector head is actually just 1 token, e.g. in older BERT-style language modeling we'll use a specific special token to simplify the preprocessing and training, wouldn't the FV head be the same as concept head?\n\nPlease do also take a look at pointer-network style attention, although it's not directly relevant to decoder-only architecture, the idea is similar when the head is pointing to prior tokens. The pointer-style would be similar to token-level heads https://arxiv.org/pdf/1506.03134 and even older morpheme vs concept level representation https://nlp.stanford.edu/pubs/conll13_morpho.pdf (which is something between concept vs token level)", "extra_scores": {}}, {"review_id": "eyCZQv5B0T", "reviewer": "Reviewer_sPEp", "summary": "In this paper, concept-level induction heads which copy entire lexical units in LLMs is introduced as a new type of induction head, not individual tokens. It works in parallel with token-level induction heads to copy meaningful text. It was indicated that these heads are important for semantic copying tasks, and they are language-agnostic.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper indicated that patching concept induction heads is effective in various LLMs based on detail experimental results. \nMoreover, it showed concept heads attend to the end of the next word. That is, concept heads can transfer entire concepts at once.", "reasons_to_reject": "The effectiveness of the proposed method is not clear in semantic task for the following reasons.\n\n1. In language-agnostic of section 5, the accuracy by patching concept heads is about 0.4. Moreover, in also Figure 36, the accuracies by patching concept heads are under 0.5 except Spanish-Italian -> Japanese-Chinese and Russian-Spanish -> English-Japanese. Therefore, it is not clear what these experimental results mean in word-level translation task. This paper needs to indicate the effectiveness of the proposed method through comparison experiments with other previous works.\n\n2. The experimental results by patching function vector (FV) heads of section 6 indicated that patching FV heads obtains output the same concept in a different language. However, patching FV heads may be effective comparison with patching concept heads from the viewpoint of accuracy because of accuracies French-English -> German-Russian are over 0.8. That is, it is not clear what was obtained through the experimental results in section 6.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aS6h6OrP0S", "reviewer": "Reviewer_NwaB", "summary": "This paper proposes a dual-route model of induction in LLMs, which consists of two distinct types of attention heads: (1) token induction heads that copy tokens verbatim, and (2) concept induction heads that copy entire lexical units. Using causal interventions and ablation studies across several models, the authors argue these two mechanisms support different in-context tasks, i.e., verbatim copying vs. semantic copying (e.g., translation). The work draws an analogy to human reading systems and explores language-agnostic representations via activation patching.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper introduces \"concept induction heads\" that enriches the taxonomy of known transformer circuits. The paper also presents analysis combining causal tracing, attention analysis, and ablations across multiple models and tasks. The paper can potentially offer a novel perspective of understanding in-context mechanisms.", "reasons_to_reject": "- The paper is dense and difficult to follow, especially in key methodological sections (e.g., causal interventions, score definitions).\n\n- The evaluations rely heavily on synthetic or narrow tasks. There's little evidence the phenomenon generalizes to real-world in-context learning.\n\n- It's also not evident how identifying concept heads helps with model design, prompting, or downstream tasks. The contribution feels more observational than actionable to me.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received mixed reviews: two very positive ones (including a reviewer fighting for acceptance) and two somewhat negative ones.

Synthesizing the reviews and my reading, I believe the paper has the following relevant strengths and weaknesses.

Reasons To Accept:
* The paper was described as well-written at various levels by the reviewers [3CWa,ZnwZ]
* Extensive ablation studies [ZnwZ]
* Consistent results across different LMs [ZnwZ]
* The paper provides foundational insights into in-context learning [sPEp,NwaB]

Reasons To Reject:
* The paper was perceived as dense and not sufficiently clear in some technical regards [3CWa]. I urge the authors to do what they can to improve this aspect.

There are some other listed “reasons to reject” that I believe do not pose major concerns:
* The patching effect might be not very large [sPEp], and it’s not evident how the results help applications [NwaB]. I consider this fine for foundational research about language models.
* Evaluation relies on synthetic or narrow tasks [NwaB]. I concur with the authors that such tasks are commonly used to study ICL, and their constrained nature can help get a clean picture.

Taken together -- and despite the mixed scores provided by the reviewers -- I believe, taking together the overall set of strengths and weaknesses, the ensuing discussion, and my reading of the paper, that the paper is of sufficiently high quality to merit publication in COLM.
Decision: Accept