InversionView: A General-Purpose Method for Reading Information from Neural Activations
Authors: Xinting Huang
Madhur Panwar
Navin Goyal
Michael Hahn
Venue: NeurIPS 2024 poster (2024)
Keywords: interpretability
explainability
mechanistic interpretability
Primary area: interpretability_and_explainability
TL;DR: We develop a method that reads out information from neural activations.

Abstract: The inner workings of neural networks can be better understood if we can fully decipher the information encoded in neural activations. In this paper, we argue that this information is embodied by the subset of inputs that give rise to similar activations.  We propose InversionView, which allows us to practically inspect this subset by sampling from a trained decoder model conditioned on activations. This helps uncover the information content of activation vectors, and facilitates understanding of the algorithms implemented by transformer models. We present four case studies where we investigate models ranging from small transformers to GPT-2. In these studies, we show that InversionView can reveal clear information contained in activations, including basic information about tokens appearing in the context, as well as more complex information, such as the count of certain tokens, their relative positions, and abstract knowledge about the subject. We also provide causally verified circuits to confirm the decoded information.

Inversion View: A General-Purpose Method for Reading Information from Neural Activations
Xinting Huang
Madhur Panwar EPFL
Saarland University xhuang@lst. uni-saarland.cr
madhur pwwpwraeppl.cpcmmm ch
Navin Goyal
Michael Hahn
Microsoft Research India navingo@microsofft com
Saarland University mhahn@lst. uni www.rlaaaddd....
Abstract
The inner workings of neural networks can be better understood if we can fully decipher the information encoded in neural activations. In this paper, we argue that this information embodied by the subset of inputs that give rise to similar activations. We propose Inversion View, which allows this subset by sampling from a trained decoder model conditioned on activations. This helps uncover the information content of activation vectors, and facilitates understanding the algorithms implemented by transformer models. We present four case studies where investigate models ranging from small transformers GPT-2. these studies, we show that Inversion View can reveal clear information contained in activations, including basic information about tokens appearing in the context, as well as more complex information, such as the count of certain tokens, their relative positions, and abstract knowledge about the subject. We also provide causally verified circuits to confirm the decoded information
practically inspect
Introduction
Despite their huge success, neural networks are still widely considered black boxes. One of the most important reasons is that the continuous vector representations in these models pose a significant challenge for interpretation. If we could understand what information is encoded in the activations of neural model, significant progress might achieved in fully deciphering the inner workings of neural networks, which would make modern AI systems safer and more controllable. Toward this goal, various methods have been proposed for understanding the inner activations of neural language models. They range from supervised probes 2
projecting to model's vocabulary space B on model's 's inner states. However, to this date, decoding form remains major challenge. Supervised probing classifiers require the researcher to decide which specific information probe for, and does not scale when the space of possible outputs is very large. Projecting the vocabulary space is restricted in scope, as only produces individual tokens. Causal interventions uncover information flow, but do not provide direct insight into the information present in activations.
5
4 55
42 to causal intervention 21 54
26
the information present in neural network activations in Humaa--undessaaddde
Here, we introduce Inversion View as a principled general-purpose purpose method for generating hypotheses about the information present in activations in neural models on language and discrete sequences, which in turn helps us identify how the information flows through the model-crucial for obtaining the algorithm implemented by the model. Inversion View aims at providing direct way of reading out
Code available at mtthtpss///githhe hlttte///ggtaaal..cc/naagtttttts
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the information encoded in an activation. The technique starts from the intuition that the information encoded in an activation can be formalized as its preimage, the set of inputs giving rise to this particular activation under the given model. In order to explore this preimage, given an activation, we train a decoder to sample from this preimage. Inspection of the preimage, across different inputs, makes it easy to identify which information is passed along, and which information is forgotten. It accounts for the geometry of the representation, and can identify which information is reinforced or downweighted at different model components. Inversion View facilitates the interpretation workflow, and provides output that is in principle amenable to automated interpretation via LLMs (we present a proof of concept in Section
We showcase the usefulness of the method in three case studies: a character counting task, Indirect Object Identification, and 3-digit addition. We also present preliminary results on the factual recall task, demonstrating the applicability of our method to larger models. The character counting task illustrates how the method uncovers how information is processed and forgotten in a small transformer. In Indirect Object Identification in GPT2-Small [44 we use Inversion View to easily interpret the information encoded in the components identified by Wang et al. [54), substantially simplifying the interpretabilityy workflow. For 3-digit addition, we use Inversion View to provide for the first time a fully verified circuit. Across the case studies, Inversion allows us to rapidly generate hypotheses about the information encoded in each activation site. Coupled with attention patterns or patching methods, we reverse-engineer the flow of information, which we verify using causal interventions.
2 Methodology
Interpretation Framework What informa- tion does an activation in a neural network en- code? Inversion View answers this in terms of the inputs that give rise to this activation (Fig- ure [.. For instance, if certain activation en- codes solely that "the subject is John" and noth- ing else (Figure [I] right), then will remain un- changed when other parts in the sentence change while preserving this aspect (e.g., "John is on leave today." = "John has cute dog.") "). From another perspective, if all sentences where the subject is John are represented so similarly that the model cannot distinguish them, given one of these representations, the only information is the commonality "the subject is John" (assum- ing sentences are represented differently when it does not hold). Building on this intuition, given an activation, Inversion View aims find those inputs that give rise to the same activation, and examine what's common among them to infer what information it encodes. In realistic networks, different inputs will rarely give rise to exactly the same activation. Rather, different changes to an input will change the activation to different degrees. The sensitivity of an activation to different changes reflects the representational geometry: larger changes make it easier for downstream components to read out information than very small changes. This motivates a threshold baashedd--baedd definition of preimages, where we consider information as present in an activation when the activation is sufficiently sensitive to it. Formally speaking, given space A' of valid inputs, query input x" X' function ff that represents the activation of interest function of the input, and a query activation z f(x"), define the e-preimage
John is on leave today
Mary has the day off
Bob is not working today
Activation Sia
John leave today John plays the guiter his free time- John a dog
Activaatio Site)
Figure 1: Illustration of the geometry at two differ- ent activation sites, encoding different information about the input. Top: the semantics of being on leave are encoded. Bottom: the information that the subject of the input sentence is John is encoded.
B f.c {x E X: D(F(), x²) Se).
(I)
where e > is a threshold and (:--)) is a distance metric. Both and D(--)) are chosen by the researcher based on representation geometry; we will define these later in case studies. In practice,
"Strictly speaking, when x" is a sequence, we study vector z" corresponding to specific position t in this sequence, i.e. z² (x"), where f(x",, represents taking the activation from the site of interest (abstracted by f) at position in input sequence x". In this case, the preimage can more rigorously be defined as Bga.fe (x X, 3t [1, [x]] D(f(x),, 29) S Zc).

---

detanet
Output X (Shilted)
datiset
Output
(anited)
train
N2
Model m
train
Decoder
I
Input
Ieput z
(5) Train Probed Model
(b) Train Decoder
x didance
d
d
Model
Model +
preimage
Decoder
Inpurt
x²
Input
[c] Interpret 2t!
Figure 2: (a) The probed model trained on language modeling objective. (b) Given trained probed model, we first cache the internal activations together with their corresponding inputs and activation site indices (omitted in the figure for brevity), then use them to train the decoder. The decoder is trained with language modeling objective, while being able attend z. (c) When interpreting a specific query activation z we give it to the decoder, which generates possible inputs auto-regressively. We then evaluate the distances the original probed model.
in all our three case studies, we vary and set it so we can read out coherent concepts from the e-preimage (Appendix A.4). With threshold-based definition, we consider only those pieces of information that have substantial impact on the activation. See more discussion in Appendix A.I
Conditional Decoder Model In this paper, study the setting where x is sequence. Directly enumerating B2 ffe is in general not scalable, as the input space grows exponentially with the sequence length. To efficiently inspect f, ee we train a conditional decoder model that takes as input the activation z and generates inputs giving rise to similar activations in the model under investigation. In the following, we refer to the original model that we are interpreting as the probed model, the conditional decoder the decoder, the place in the probed model from which we take the activation as the activation site (e.g., the output of ith layer), the inputs generated by the decoder as samples, and the index of a token in the sequence as position.
We implement the decoder an autoregressive language model conditioned on zº, decoding input samples x (see Figure [ and details in Appendix C As the decoder's training objective corresponds to recovering x exactly, sampling at temperature 1 will typically not cover the full e-preimage. Thus, for generating elements of the e-preimage, we increase diversity by drawing samples at higher temperatures and with noise added z (details in Appendix A.2). We then evaluate (f z²) at each position in each sample x, select the position minimizing D determine membership in Bz and subsample in-c-preimage and out-of-c-pre samples for inspection.
An important question is whether this method, relying on a black- box decoder, produces valid e-premmages Correctness (are all generated samples the e-preimage?) is ensured by design, as we evaluate (x 2²) for each generated sample. The other angle is completeness (are the samples representative of the e-preimage??. If some groups of inputs in e-preimage are systematically missing from the generations, one may overestimate the information contained in activations. But this behavior would be punished by the training objective, since the loss on these examples would be high. We explicitly verify completeness by enumerating inputs in one of our case studies (Appendix B). Another approach is design counter-examples not satisfying a hypothesis about the content of Ba e- In our experiments, we found that these examples were always outside of Bzg.f.e-
3 Discovering the Underlying Algorithm by Inversion View
Notation. In the transformer architecture, outputs from each layer are added to their inputs due to residual connection. The representations of each token are only updated by additive updates,
In some cases, including Figure
4b fix particular position for interpretation. See Appendix A.3

---

Activation Site: gfrred
Acthvation poot
Actioot Site: a stwoiot
Query
Query
Mord
8,000;
Overy Teput
Gererated Sumples
Cenerated Seep Les
0.002; B Baiaiaiiiii
T1o Count
0:007;8:3; g a 1 13
[001) 3; B aailaga ailiaaaaae iilaaaaaa a (a) :
Generated Sauples
0.002 ;9:3;
0.1 2 (::3E
0:035;::3 0 a
33
0.023 Brcpew
0:004:8:3::
001T;A:3;
8 a
1::03:0:3; m
0:004:4:3:
33
6d18;
1::38::33
00005:8::;;
3
8:20;8:3;
Bwg.aggaiiii
10007;::3 mm
0:079;1:3;
ggggnaiaagaaa 3.
3:66;;:;; wtw.://wwwwwvggg......
10009:::3 ww..vavvook.....................
0.001:4:4;
0058; 8 wwwowwossww...... (0)
00006;8:3; 5 W www.owwwn www.mwwwonmwlll
0:062;8:4;
Ball::gglllgllg.o. ggglllgggaaa 4.
8:26;;:3; fgg. Bgw.skkkkg.allp ::
0003;::3;; pt
0:31::4:::
kg
g
4412; B kkjggkknggkk 4
0.12:::4; b 001e:400000@@(000000000000
4: k
0:16: B ppppppp htt.s/iiijjjjiiiiihhhihh.aa
0:220:1:2;
3:
2:
0:
(a)
Figure 3: Inversion View Character Counting Task. The model counts how often the target character (after 'I') occurs in the prefix (before 'I'). B and E denote beginning and end of sequence tokens. The query activation conditions the decoder to generate samples capturing its information content. We show non-cherrypicker activation sites on the same query input. Distance for each sample is calculated between activations corresponding to the parenthesized characters in the query input and the sample. "True count" indicates the correct count of the target character in the samples (decoder may generate incorrect counts). (a) MLP layer amplifies count information. Comparing the distances before (left) and after (right) the MLP, we see that samples with diverging counts become much more distant from the query activation. (b) In the next layer (":" exclusively attends to target character copying information from residual stream of target character to the residual stream of ":"), the count is retained but the identity of the largel character is longer encoded ("c", "m", etc. instead of "g"), as it is no longer relevant for the predicting the count. Therefore, observing the generations informs us of the activations' content and how it changes across activation sites.
samples inside and outside the e-preimage (e = 0.1) at three
forming a residual stream 117. Using notation based on 1r and 40, we denote the residual stream as pr.(pre.mid.poat) R-xxd where is the layer (an attention (sub)layer + MLP (sub)layer) index, N is the number of input tokens, d is the model dimension, pre, mid, post stand for the residual stream before the attention layer, between attention and MLP layer, and after the MLP layer. For example, Ore is the sum of token and position embedding, 20.mid is the sum of the output of the first attention layer and 2rrr and 2.psttt is the sum of the output of the first MLP layer and z U, Note that ppttt = pirpppr We use subscript / to refer to the activation at token position r, e.g., xt h(d i.e., x p..mid = x.ppre+
i.mid Rdd. The attention layer output decomposes into outputs of individual heads (LN(((prr)))) where LN(-) represents layer normalization (GPT style/pre-laer--norm)) We denote the attention head's output as add i.e., a.d hi (Lr(x))))))
+E, /2² x
Decoder Architecture. We train single two-layer transformer decoder across all activation sites of interest. The query activation z is concatenated with an activation site embedding e, a learned embedding layer indicating where the activation comes from, passed through multiple MIP layers with residual connections, and then made available to the attention heads in each layer of the decoder, alongside the already present tokens from the input, that each attention head can also attend the post-processed query activation in addition to the context tokens. Each training example is a triple consisting of an activation vector z Rd, the activation site index, and the input, on which the decoder is trained with a language modeling objective. Appendix C has technical details.
3.1 Character Counting
We train a transformer (2 layers, head) on inputs such as "wwzccvezwwvveeeely.a" to predict the last token "8", the frequency of the target character (here, "v") before the separator "I". For each input, three distinct characters are sampled from the set of lowercase characters, and each character's frequency sampled uniformly from 1-9. The input length varies between 7 and 31. We created 1.56M instances and applied 75% 25% train-test split; test set accuracy is 99.53% (Details in

---

Appendix D).
We use D(z, 1z==="la (i.e., normalized euclidean distance, as the magnitude of ||z0||2
activations varies between layers), where z denotes the aforementioned f(x), and set e 0.1.
Interpreting via Inversion View and attention. In layer O, the target character consistently attends to the same character in the previous context, suggesting that counting happens here. In Figure Ba we show the c-preimage of Tte 0.mid and #te 2..post where the subseript te denotes the target character. We show ~ 10 random samples at single query input, but our hypotheses are based on- and easily confirmed by-rapid visual inspection of dozens of inputs across different query inputs On the left (before the MLP), the activation encodes the target character, as all samples have "g" as the target character. Count information is not sharply encoded: while the closest activation corresponds to "g" occurring times, two activations corresponding to count-4 input ("g" occurring 4 times) are also close, even closer than a count-3 input. On the other hand, on the right (after the MLP), only count-3 inputs are inside the e-preimage, and count-4 inputs become much more distant than before. Comparing the e-preimage before and after the ML. in layer O, we find that the MIP makes the count information more prominent in the representational geometry of the activation. The examples are not hherry-picked, count information is generally reinforced by the MI. across query inputs.
In the next layer, the colon consistently attends to the target character, and Inversion confirms that count information is moved to the colon's 's residual stream (Figure 3b), More importantly this
illustrates how information is abstracted: We previously found that Zic ".post encodes identity and frequency of the target character. However, the colon obtains only an abstracted version of the information, in which count information remains while the target character is largely (though not completely) removed. Inversion View makes this process visible, by showing that the target character becomes interchangeable with little change to the activation. See more examples in Appendix [D.2] Overall, with Inversion View, we have found simple algorithm by which the model makes the right prediction: In layer O, the target character attends to all its occurrences and obtains the counts. In layer I, the colon moves the results from the larget character to its residual stream and then produces the correct prediction. Accounting for other activation sites, we find that the model implements a somewhat more nuanced algorithm, investigated in Appendix D.4 Overall, Inversion View shows how certain information is amplified, but also how information is abstracted or forgotten.
Quantitative verification. We causally verified our hypothesis using activation patching 533 on (position, head output) pairs. As the attention head in layer I attends almost entirely to the target character, only head outputs dze a..0, and a! can possibly play a role in routing count information. We patch their outputs with activations from contrast example flipping a single character before "I"" We patch activations cumulatively, starting either at the lowest or highest layer, with some fixed ordering within each layer. For example, we patch a.00 and observe how final logits change compared
0,0
to the clean run, then we patch both a..0 and aze 0,0 and do the same, and so forth. By the end of patching. the model prediction will be flipped. When adding activation to the patched set, attribute to it the increment in the difference of LD before and after patching, where LD denotes the logit difference between original count and the count in the contrast example. Cumulative patching allows us to observe dependencies: For instance, as we hypothesize that - is completely dependent 0.0
0.0 on atc we expect that, when ate is already patched, patching a.'0 will have no further effect, 0.0, is not patched, patching a.'0 will have a significant effect. Results (Figure 4a) 0.0 match our prediction: Patching either of the activation in the hypothesized path (ate and a! is sufficient to absorb the entire effect on logit differences, confirming the hypothesis. See Appendix D.3 for further details and D.4 for further experiments.
whereas when azc.
3.2
I0I circuit in GPT-2 small
To test the applicability of Inversion to transformers pretrained on real-world data, we apply our method to the activations in the indirect object identification (IOI) circuit GPT-2 small [48] discovered by Wang et al. (54) We apply Inversion View to the components of the circuit, read out the information, and compare it with the information or function that Wang et al. (54) had ingeniously
inferred using a variety of tailored methods, such as patching and investigating effects on logits and
'The reader can check generations conveniently httttts: http://wwwerrrooooooooooooeeeaee streamlit..ap
5

---

-jendoftwet--Afee Juadie
gavw Justin
Patching tffeci
Overy Trpat
wwweerbbaaaaaaaeeeeeeee and weat he srin fel
Conerated Senolee
to) kadribbo))
10004 Heer the Phur asti yoet as the school, Paus dedided to give kt Juste (loO)
1000 -to- deeeerloeed teeeeeeerreeee reee dailk and lecile hut long. rgg alkorwards dlick soit xsteéeos)
hascring Effict
thashertall
coee ope anrteaeee the funn the vforeoon, str terter serer to station wiicee glve sieitine luck
0.000 ben long breas foin barty and zustin went toere Kimber ahern Juarinttio
austigios) 0.00 -er bereelyteee Lustle ane wear want the school, ntvie gove boow/ ieste gog
worh nz poodin, piviagar
0L000 ype barbet santted vodite and Justin anda srack at parcies Just pase bs su (nseejoos)
0006 Aoo hwgh spers tagg olx corennnng ottieracional Nourddo ring ldt Missckleps)
(a)
(h)
0.0 Figure 4: (a) Character Counting, Activation patching results show that d'e and a. play crucial roles in prediction, as hypothesized based on Figure 3 and Sec. [.3] In contrast examples, only one character differs. Top: We patch activations cumulatively from left to right. We can see patching ate 0.0 accounts for the whole effect, and when atc is already patched, patching a!'0 has almost effect. Bottom: On the other hand, if we patch cumulatively from right to left, a. accounts for the whole 0.0 effect while patching Ote has no effect if a.'" has been patched. So we verified that a. solely relies 0.0
0.0
on aze and this path is the one by which the model performs precise counting. The patching effect averaged across the whole test set. (b) IOI. Inversion View applied to Name Mover Head 9,9 at "to";; we fix the compared position to "to". Throughuut the e-preimage, "Justin" appears as the IO, revealing that the head encodes this name. This interpretation confirmed across query inputs.
attention. We show that Inversion View unveils the information contained in the attention heads' outputs, with results agreeing with those of Wang et al. (54).
The 1OI task consists of examples such as "When Mary and John went to the store, John gave drink to", which should be completed with "Mary". We use S for the subject "John" in the main clause, 10 for the indirect object "Mary" introduced in the initial subclause, SI and S2 for the first and second occurrences of the subject, and END for the "to" after which IO should be predicted. To facilitate comparison, we denote attention heads in Wang et al. [54] with i.j denoting h Wang et al. [54] discover a circuit of 26 attention heads in GPT-2 small and categorize them by their function. In short, GPT-2 small makes correct predictions by copying the name that occurs only once in the previous context. For Inversion View, we train the decoder on the IOI examples (See details in E. Despite the size of the probed model, we find the same 2-layer decoder architecture as in Section 3.1 to be sufficient. We use D(z, 22) = | |
(i.e., cosine distance), and 0.1. Euclidean distance leads to similar results, but cosine distance is a better choice for this case (Appendix [E.4).
We start with the Name Mover Head 9.9, which Wang et al. 54 found moves the IO name to the residual stream of END. shows the e-preimage at "to". The samples in the e-preimage share the name "Justin" as the 10. The head also shows similar activity at some other positions ( (Appendix [A.3). Results are consistent across query inputs. Therefore, Inversion View agrees with the conclusions of Wang et al. (54) on head 9.9. Applying the same analysis to other heads (Table we recovered information in high agreement with the information that Wang et al. (54) had inferred using multiple tailored methods. For example, Wang et al. (54] found S-Inhibition heads were outputting both token signals (value of S) and position signals (position of S1) by patching these heads' outputs from a series of counterfactual datasets. These datasets are designed to disentangle the two effects, in which token and/or position information are ablated or inverted. These two kinds of information can be directly read out by Inversion View (Figure 19 shows an example for an S-Inbibition head that contains position information), and there is no need to guess the possible information to design patching experiments. Overall, among the 26 attention heads that Wang et al. (54) identified, Inversion View indicates a different interpretation in only 3 cases; these (0.1, 0.10. 5.9) were challenging for the methods used before (Appendix E.3). In summary, Inversion View scales to larger models.
3.3 3-Digit Addition
We next applied Inversion View to the problem of adding 3-digit numbers, between 100 and 999. Input strings have the form "B362+405-767E" or "B824+692=1516E" and are tokenized at the

---

Activetion
Acthaation
sum Σ
d22
Drre
6476
6 8
Activation Site A
Query Irput
Query Input
Query Input
0,000;
0.000; B 8615+861
5,000; B 8615+861()11
Generated Somples
Seserated Samples
Generated Samples
Decoder Liwihood
0.00;
6 (5 15
0.001:
3 87
9:003; 6 6 0
177
0:02: 0 3 4
2
0.001 8
1 7
0,366 0 8
0:03;; 4
0
14
0:0; 8
2
LITUE
0.03; B 85
(4
0.034; 8 0 - 1437E
0,05; 8 1 6 12800
0041; 8 a (88
0,008; 9 0 - 0
1437E
0.006; 8 a 1200E
0042; B
0,04; B 8 1453E 14
0009; 8
+ 9
1476E
0,000 B 4
4 14566
0.03; 2 (M
0.000; 6
9
1503E
0:09; D 2
14446
0.15; 5 9 66
n.00;; 8 0 966E
0.00;; D 2
1 14TE
0.14; 54
D:44T;
0.20;
1520E
(c)
Figure 5: Inversion View applied to 3-digit addition: Visually inspecting sample inputs inside and outside the e-preimage of the query allows us to understand what information is contained in an activation. The color on each token generated samples denotes the difference in the token's likelihood between a conditional or unconditional decoder (Appendix G). The shade thus denotes how much the generation of the token caused by the query activation (darker shade means a stronger dependence). In (a-c), the colored tokens are most relevant to the interpretation. We interpret two attention heads (a.b) and the output of the corresponding residual stream after attention (c). In (a), what's 's common throughout the e-preimage is that the digits in the hundreds places are 6 and 8. Inputs outside the e-preimage don't have this property.
(b), what's common is that the digits in tens places are 1, 6. or numerically close. Hence, we can infer that the activation sites a and a.3 encode hundreds and tens place in the input operands respectively; the latter is needed to provide carry to Al. Also, the samples show that the activations encode commutativity since the digits at hundreds and tens place are swapped between the two operands. In (c), the output of the attention layer after residual connection combining information from the sites in (a) and (b) encodes "6" and "8" in hundreds place, and the carry from tens place. Note that o. and a.2 contains similar information as a 0.00 These observations confirmed across inputs. Taken together, Inversion View reveals how information is aggregated and passed on by different model components.
character level. We use FI, F2, F3 denote the three digits of the first operand and S1, S2, S3 for the digits of the second operand, and AI, A2, A3, A4 (if it exists) for the three or four digits of the answer, and C2 C3 for the carry from tens place and ones place (i.e., C2: whether F2+S2>10 10, C3: whether F3+S32 10). Unlike (47). we do not left-pad answers to have all the same length; hence, positional information is insufficient to determine the place value of each digit.
The probed model is a decoder-only transformer (2 layers, 4 attention heads, dimension 32). We set attention dropout O. Other aspects are identical to GPT-2. The model is trained for autoregressive next-token prediction on the full input, in analogy to real-world language models. In testing, the model receives the tokens up and including "=", and greedily generates up to "E". The prediction counts as correct if all generated tokens match the ground truth. The same train-test ratio as in Section 3.1 is used. The test accuracy is 98.01% %. For other training details see Appendix F.1
Interpreting via Inversion View and attention. As Section 3.1 we use normalized Fuclidean distance for D) and the threshold 0.1. We first trace how the model generates the first answer digit, A1, by understanding the activations at the preceding token, "=". We first examine the attention heads at "_" in the 0-th layer (Figure 5). As for the first head (a..)), only Fl and SI matter in the samples- indeed, changing other digits, or swapping their order, has a negligible effect on the activation (Figure 5). Across different inputs, each of the three heads 0.00 a..1 0.2 encode either one or both of F1 and S1 (Figure 26); taken together, they always encode both. This is in agreement with attention focusing on these tokens. The fourth and remaining head in layer 0 (a) encodes F2 and S2, which provide the carry from the tens place to the hundreds place. Combining the information from these four heads, 2. mi consistently encodes and S1; and approximately represents F2, on-only the carry to Al (whether F2+S2>100 10) matters here (Figure 5p). Other examples are in
7

---

Ver fication informacion flow from F1,
vertfcction infermation "ine from F2,
10
8-
Incemment
6-
4-
2-
B
422 a5 ag a aL ali al? Petched Activation
a2 #2 ag 2 al at
Patched Activetion
(a)
Figure 6: 3-Digit Addition Task: (a) Information flow diagram for predicting Al inferred via Inversion View. The colors denote which places are routed; alternating colors indicate two places routed. This is a subfigure of Figure 31 (b) Validation of (a) via activation patching for the prediction of A1. Like Figure 4a + (+) means cumulatively patching activation from left to right (right left) on the horizontal axis. Left: Patching with activation containing modified FI and S1 information. Right: Patching with activation containing modified F2 and S2 information. As we can see, components from (a) show substantial increment if and only if they have a not-yet-patched connection to output (when patching right to left) or input (patching left to right), verifying that (a) causally describes the flow of information. Therefore, Inversion View helps us uncover both information flow and content of activations.
We summarize the function of layer 0 at "=": Three heads route FI and SI to the residual stream of "=" x=. The fourth head routes the carry resulting from F2 and S2. Layer 1 mainly forwards information already obtained in layer 0, and does not consistently add further information for A1. See more examples in Appendix F2
Figure 27
Figure 6a shows the circuit predicting A1. Inversion View allows us to diagnose an important deficiency of this circuit: Even though the ones place sometimes receives attention in layer 1, the circuit does not consistently provide the carry from the ones place to the hundreds place, which matters on certain inputs-we we find that this deficiency in the circuit accounts for all mistakes made by have provided a circuit allowing the model predict Al while also understanding its occasional failure in doing so correctly. Corresponding findings for A2, A3, and A4 are in Table B and Figure 31 From A2 onwards, Inversion allows uncover how the model exhibits two different a algorithms depending on whether the resulting output will have 3 or 4 digits. In particular, when predicting A3, the layer 0 circuit the same across both cases, while the layer circuit varies, since this determines whether A3 will be tens place or ones place. Beyond figures in the Appendix, encourage readers verify our claims in our interactive web application.
the model (Appendix F33 Taken together,
Quantitative verification. We used causal interventions to verify that information about the digits in hundreds and tens place is routed to the prediction of Al only through the paths determined in Figure [a]] and none else. Like before, we cumulatively patch the head output on "=" preceding the target token A1, with activation produced at the same activation site by contrast example changing both digits in a certain place, Results shown in Figure 666 strongly support our previous conclusions. For example, n.3 and a are not relevant to Fl and S1. Important heads detected by activation patching, a 0.1 0.2 ".2,a""'' at.1, all contain F1 and S1 according to Figure 6al Furthermore, also confirm that a relies on the output of layer 0 depicted in sub-figure (a): When heads in layer are already patched, patching a 1.1 has no further effect (value corresponding to -+ is zero), but has an effect when patching in the opposite direction. On the contrary, a..0 shows little dependence on layer O, consistent with Figure 6al On the right of Figure 6bb
we can confirm that a 9.3 important for routing F2 and S2, and the downstream heads in layer rely on it. Findings for other answer digits are similar (See Appendix F.5). Overall, the full algorithm obtained by Inversion View well- cellosuppo by causal interventions.

---

Inversion View reveals granularity of information. Heads often read from both digits of place, but only the sum matters for addition. Are the digits represented separately, only as their sum? Unlike traditional prob- ing, Inversion answers this ques- tion without designing tailored prob- ing tasks. In Figure 7 (left), a.22 ex- actly represents F2 and S2 (here, and 5). Other inputs where F1+S1=5+2 have high D. contrast, on the right,
Activation Site: a
Activation Site: a
8
6
B 9 0 8
(1)
Query Input
Query Input
0.000; 5
0.000; 0
Generated Samples
Generated les
0.000; 5
2 ( 9
1 0.015; 8 2
7 (1) 50
0.000; 8 2
99 0177
0.041; 8 6
B3 2 7
0.004;
99
8 9
0.004; 2
863
0.063; 9
8
0.675; 6 55
0.064; 0 8 8 9
0.06;; 8 8 6 9
6E
F2 and S2 are represented only by their sum: throughout the c-preimage, F2+S2-9. In fact, we find such sum- only encoding only when F2+S2=9- special case where the ones place of operands affects the hundreds place of the answer via cascading carry. We hypothesize that the model encodes them similarly because these inputs require special treatment. Therefore, even though encoding number pairs by their sum is good strategy for the addition task from human perspec- tive, the model only does it needed. We also observe intermediate cases (Figure 29
Nanual Samples
1013; 8156:86666660000
0.069;
(1)
0.729;
0.120;
6
801E
0.723; 4355-450
0.242: 0
87411
691E
691E
1,012; 6.6.5 1 180
0.405;
977
(1)
11EE
Figure 7: 3-Digit Addition Task: Inversion View uncovers different ways which digit representation encoded in ac- tivations. Left: The digits in the hundreds place are encoded separately and hence generations denote them as separate entities. Right: The digits in the tens place are encoded as sum (9 in this case) and the generations represent different 2-partitions (7+2, 6+3, 1+8, S+4, etc.) of that sum.
3.4 Factual Recall
To test whether Inversion View can be applied larger language models, we explore how GPT-2 XL (1,5B parameters) performs the task of recalling factual associations. In this case study, our intention not to provide a full interpretation of the computations performed to solve this task, which we deem out of scope for this paper. Instead, we show that Inversion View produces interpretable results on larger models by focusing on a relatively small set of important attention heads in upper layers. The decoder model in this case study is based on GPT-2 Medium, because we expect a more complex inverse mapping from activation to inputs to be learned. We observe the resulting c-preimage can express high-level knowledge (Figure (4444), and sometimes can predict the failure of the model (Appendix (..6). Using Inversion View, we again shed light on the underlying mechanism of the model. We present detailed findings in Appendix H
Discussion and Related Work
Comparison with other Interpretation Methods Supervised probing classifiers, assessing how much information about variable of interest is encoded in activation site, arguably the most
common method for uncovering information from activations [e.g. 2 K 5 55
52
33
34].
requires a hypothesis in advance and is thus inherently limited hypotheses conceived priori by the researcher. Inversion View, on the other hand, helps researchers form hypotheses without any need for prior guesses, and allows fine-grained per-activation interpretation. Inspecting attention patterns [e.g. 12] is a traditional approach to inferring information flow, and we have drawn on it in our analyses.
28 35 causally identifies paths along which information flows. While the information flow provides an upper bound on the information passed along by tracing back the input token, it insufficient for determining how information is processed and abstracted. For instance, in Section 3.1 occurrences of the target character are causally connected to 0.0 aze which then connects to a! (direct or mediated by MLP layer O). Without looking at encoded information, we only know that the information these paths is related to the occurrences of the target character, but not whether it is their identity, positions, count, etc. More generally, when a component reads a component that itself has read from multiple components, connectivity does not tell us which pieces of information are passed on. Similar considerations apply to other intervention methods.
More recently, path patching [54] 26 13

---

causal intervention method, can be used to study the causal effect of an activation on the output. and can help localize where information is stored [37 49] or find alignment between a high-level causal model and inner states of a neural model 21 22 57 Many recent works obtain insights about information content by projecting representations or parameters into the vocabulary space [42] 44, 31 54, 23 24 [6]. This technique is sometimes referred to as Direct Logit Attribution (DLA). We argue that DLA is only suitable for studying model components that directly affect model's final output. For those components whose effect is mediated by other components, their output information is meant to be read by a downstream component, thus not necessarily visible when projecting to the vocabulary space. We provide further discussion in Appendix [ Generalizing this Geva et al. 24 intervene on attention weights to study information flow. Activation patching,
approach, Ghandeharioun et al. (5 patch activations into an LLM. Another line of recent research 110 51 14
decomposes activations into interpretable features using sparse autoencoders.
Some other interpretation methods also generate in input space, but differ from Inversion View in goals and methods. This includes feature visualization 45 41 adversarial or counterfactual example
generation 227 59 46 45 and GAN inversion methods (58) We discuss the similarities and
differences of these works compared to Inversion View in Appendix I.4
Inversion View offers distinctive advantages and makes analyses feasible that are otherwise very hard to do with other methods. It can also improve the interpretability workflow in coordination with other methods. For example, one may first use methods such as path patching or attribution [50] [8] to localize activity to specific components, and then understand the function of these components using Inversion View. In sum, Inversion View is worth adding to the toolbox of interpretability research.
Transformer Circuits for Arithmetic Related to Section 3.3 [47] interpret the algorithm imple- mented by 1-layer 3-head transformer for n-digit addition (n E (5, 10, 15)). finding that the model implements the usual addition algorithm with restrictions on carry propagation. In their one-layer setup, attention patterns are sufficient for generating hypotheses. Lengths of operands and results are fixed by prepending 0. Our results, in contrast, elucidate a more complex algorithm computed by two-layer transformer on a more realistic version without padding, which requires the model to determine which place it is predicting. We also contribute by providing a detailed interpretation, including how digits are represented in activations.
Automated Interpretation for Inversion View Recent work has started using LLMs generate interpretations [8. [0] The samples produced by Inversion View can be easily fed into LLMs for automated interpretation. We show proof of concept by using Claude 3 to interpret the model trained for 3-digit addition. See results in Table The --r-viiedeedd interpretation reflects the main information in almost all cases of the addition task. Despite some flaws, the outcome is informative in general, suggesting this as a promising direction for further speeding up hypothesis generation.
Limitations Inversion View relies on a black-box decoder, which needs to be trained using relevant inputs and whose completeness needs to be validated by counter-examples. Also, Inversion View, while easing the human's 's task, is still not automated, and interpretation can be laborious when there many activation sites. We focus models up to 1.5B parameters; scaling the technique to large models is an interesting problem for future work, which will likely require advances in localizing behavior tractable number of components of interest. Fourth, interpretation uses metric D(:,) +). The geometry, however, in general could be nonisotropic and treating each dimension equally could be sub-optimal. We leave the exploration of this to future work.
5 Conclusion
We present InversionView, an effective method for decoding information from neural activations. In four case studies character counting, 1OI, 3-digit addition, and factual recall- -we showcase how it can reveal various types of information, thus facilitating reverse- wveere-enggieering of algorithm implemented by neural networks. Moreover, we compare it with other interpretability methods and show its unique advantages. We also show that the results given by Inversion View can in principle be interpreted automatically by LLMs, which opens up possibilities for a more automated workflow. This paper only explores a fraction of the opportunities this method offers. Future work could apply it to subspaces of residual stream, to larger models, to different modalities such as vision.
10

---

Acknowledgements
Funded by the Deutsche Forschungsgemeinnchaatt (DFG, German Research Foundation) Project-II) 232722074 - SFB 1102. We thank anonymous reviewers for their encouraging and constructive feedback.
References
[1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint 16xvvv:610.016444 2016.
[3] Authropic. Introducing the next generation of claude, 2024.
com/
news/ claude- family
[4] Y. Belinkov. Probing classifiers: Promises, shortcomings, and advances. Comput. Linguistics. 2871)2207- 2022. doi: 10.1162/COLI A A1_00422 URL https://doii org/10. 1162/
coli_a_ 00422
[5] Y. Belinkov and J. Glass, Analysis methods in neural language processing: A survey. Transac- tions of the Association for Computational Linguistics, 7:49-72, 2019.
[6] Y. Belinkov, N. Durrani, F. Dalvi, H. Sajjad, and J. Glass. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 861-872, 2017.
[7] N. Belrose, Z. Furman, L.. Smith, D. Halawi, I. Ostrovsky, L. McKinney, S. Biderman, and J. Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2023.
[8] S. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, Wu, and W. Saunders. Language models can explain neurons in language models. URI. https://openaipublic.. blob. core. windows. httphheeurooeepplieeeeeeeeeeeeee cessed: 14.05. 2023), 2023.
himl. ac-
[9] S. Bird and F. Loper. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214-217, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL htttps::/acaaattttttttt wtwpe//latttoolloooooo--/-------
[10] T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, page 2, 2023.
[11] B. Chughtai, A. Cooney, and N. Nanda. Summing up the facts: Additive mechanisms behind factual recall in llms. arXiv preprint arXiv:2402.07321, 2024.
[12] K. Clark, U. Khandelwal. O. Levy, and C. D. Manning. What does bert look at? an analysis of bert's 's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP., pages 276-286, 2019.
[13] A. Conmy, A. Mavor-Parker, A. Lynch, S. Heimersheim. and A. Garriga-Aonso. Towards automated circuit discovery for mechanistic interpretabiility. Advances in Neural Information Processing Systems, 36: 1111661111115221 2023.
[14] H. Cunningham, Ewart, L. Riggs, Huben, and Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint SXXi::3309006000 2023.
[15] Dao, Y.-T. Lao, C. Rager, and J. Janiak. An adversarial example for direct logit attribution: Memory management in gelu-41. arXiv preprint arXiv:2310.07333 2023.
[16] G. Dar, M. Geva, A. Gupta, and J. Berant. Analyzing transformers in embedding space. In The 61 st Annual Meeting Of The Association For Computational Linguistics, 2023.
[17] N. Elhage, N. Nanda, C. O. T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, Drain, D. Ganguli, J. Kernion, L.. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCan-
Conerly, N. DasSarma,
Hatfield-Dodds D. Hernandez, A. Jones,
dlish, and C. Olah. A mathematical framework for transformer circuits, 2021. https //transf owwerr circuits, swweiiiss.pub//00 http2////////aeee///eee wwweeewwwwwoi/i..oeeeee httml

---

[18] J. Ferrando and E. Voita. Information flow routes: Automatically interpreting language models at scale. 2024.
[19] J. Ferrando, G. I. Gállego, I. Tsiamas, and M. R. Costa-jussà. Explaining how transform- ers use context to build predictions. In A. Rogers, J. Boyd-Graber, and N. Okazaki, ed- itors, Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 5486-5511 Toronto, Canada, July 2023, Asso- URL https
ciation for Computational Linguistics. doi: 10. 10.18663///2023.ac--oogg
7 //aclanthology wrg./0g............ a acl-long......... 301
[20] Ferrando, G. Sarti, A. Bisazza, and M. R. Costa-jussă. A primer on the inner workings of Transformrr-baeee language models. arXiv preprint arXiv:2405.0022 00208, 2024.
[21] A. Geiger, H. Lu, T. Icard, and C. Potts. Causal abstractions of neural networks. In M. Ranzato, A. Beygelzimer. Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Anmual Conference on Neu- ral Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 9574-9586 2021. URL httpps://ppoceeednngg htwpocceddingn..eriipssss.. neurips newrips.cc.paper// /201//as/////
$f5c422f4d49a5a80 0at1aaa5550077020244333233300000 Abbtaaatt. httmracctttttll
[22] A. Geiger, C. Potts, and T. Icard. Causal abstraction for faithful model interpretation. arXiv preprint arXiv:2301.04 04700, 2023.
[23] M. Geva, A. Caciularu, K. Wang, and Y. Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 30-45, 2022.
[24] M. Geva, J. Bastings, K. Filippova, and A. Globerson. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216-12235, 2023.
[25] A. Ghandeharioun, fying framework for inspecting hidden representations of language models. arXiv preprint arXiv:2401.06100 2024.
Caciularu, A. Pearce, Dixon, and M. Geva. Patchscope: A uni-
[26] N. Goldowsky-Dill, C. Macleed, L. Sato, and A. Arora. Localizing model behavior with path patching. arXiv preprint (xXiv:2304.05900 2023.
[27] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. 2014.
arXiv preprint
[28] M. Hanna, O. Liu, and A. Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in pre-trained language model. Advances in Neural Information Processing Systems, 36, 2024.
[29] M. Hanna, S. Pezzelle, and Y. Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. arXiv preprint arXiv:2403. /7806, 2024.
[30] Kaddour. The minipile challenge for data-efficient language models. arXiv preprint arXiv:2304. 08442, 2023.
[31] S. Katz and Y. Belinkov. Visit: Visualizing and interpreting the semantic information flow of transformers. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14094-14113, 2023.
[32] G. Kobayashi, T. Kuribayashi, S. Yokoi, and K. Inui. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057-7075. 2020.
[33] B. Li, M. Nye, and J. Andreas. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1813-1827, 2021.
[34] K. Li, A. K. Hopkins, D. Bau, F. Viégas, H. Pfister, and M. Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations, 2022.
[35] T. Lieberum, M. Rahtz, J. Kramár, G. Irving, R. Shah, and V. Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla, arXiv preprint 2307.0337 2023.
12

---

[36] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
[37] K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372, 2022.
[38] Michel, O. Levy, and G. Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
[39]
N. Nanda. Attributionn patching: Activation patching at industrial scale, 2023. URL https
/www www.neneenndd.d.mmm wwt.://aaiiiiiiiiiaeaeiaaaaa
[40]
N. Nanda and J. Bloom. Transformerlens, attp:://gitaab..oo//raaattttoooe
oransformerLens] 2022.
[41] A. Nguyen, J. Yosinski, and J. Clune. Understanding neural networks via feature visualization:
A survey. Explainable Al: interpreting, explaining and visualizing deep learning, pages 55-76, 2019.
[42] nostalgebraist. interpreting gpt: the logit lens. LESSWRONG, 2020. https: www.weesswoogg
com/posts Socff/BwppdaaNGvrn//nnttrrrrrrre gptnnngg--pttttt Logit- lens
[43] C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. Distill, 2(11) 2017.
[44] K. Pal, J. Sun, A. Yuan, B. C. Wallace, and D. Bau. Future lens: Anticipating subsequent tokens from single hidden state. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 548-560, 2023.
[45] M. Pawelczyk, C. Agarwal, S. Joshi, S. Upadhyay, and H. Lakkaraju. Exploring counterfactual explanations through the lens of adversarial examples: A theoretical and empirical analysis. In International Conference on Artificial Intelligence and Statistics, pages 4574-4994, PMLR, 2022.
[46] X. Qi. K. Huang, A. Panda, P. Henderson, M. Wang, and P. Mittal. Visual adversarial examples jailbreak aligned large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21527-21536, 2024.
[47] P. Quirke and F. Barez. Understanding addition in transformers, In The Twelfth International Conference on Learning Representations, 2024. URL h www.wewww.etforontt/
id=r Xd=FI2
[48] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[49] A. Stolfo, Y. Belinkov, and M. Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7055-7052. Singapore. Dec. 2023. Association for Computational Linguistics. doi: 10. 18653/v1/2023.emmg 1op-mmii.4333 4335. URL https: https://aclanthologyy hwwp:/hwoooooooooo//...o 2023. emnlp- main
435
[50] A. Syed, C. Rager, and A. Commy. Attribution patching outperforms automated circuit discovery. arXiv preprini arXiv:2310. 10348, 2023.
[51] A. Tamkin, M. Taufeeque, and N. D. Goodman. Codebook features: Sparse and discrete interpretability for neural networks. arXiv preprint 0rXiv:2310.172333 2023.
[52] I. Tenney, D. Das, and E. Pavlick. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4601, 2019.
[53] Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, Y. Singer, and Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 01112288-124001 2020.
[54] K. R. Wang. A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. 2023. URI. https /openreview wtt.:///fforrrri/eessss/ssooo..
[55] Z. Wang, A. Ku, J. M. Baldridge, T. L. Griffiths, and B. Kim. Gaussian process probes (gpp) for uncertainty-aware probing. In Thirth-seventh Conference on Neural Information Processing Systems, 2023.
13

---

[56] J. Wiland, M. Ploner, and A. Akbik. BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models, 2024. URL URL http:://aaa/...oooo abs/ 2404.04113
[57] Z. Wu, A. Geiger, T. Icard, C. Potts, and N. Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. Advances in Neural Information Processing Systems, 36, 2024.
[58] W. Xia, Y. Zhang, Y. Yang, J.-H. Xue, B. Zhou, and M.-H. Yang. Gan inversion: A survey.
IEEE transactions on pattern analysis and machine intelligence, 45(3):3 2022.
[59] C. Xiao, Li, J.-Y. Zhu, W. He, Liu, and Song. Generating adversarial examples with adversarial networks. arXiv preprint ar0iv:1801.0 (a01.0201000 2018.
[60] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le. Xlnet: General- ized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
[61] Y. Yao, N. Zhang, Z. Xi, M. Wang, Z. Xu, Deng, and H. Chen. Knowledge circuits in pretrained transformers. arXiv preprint arXiv:2405.17965 2024.
[62] Q Yu, J. Merullo, and E. Pavlick. Characterizing mechanisms for factual recall in language models. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pages 9924-9955 2023.
14

---

Contents
Introduction
2
Methodology
3
Discovering the Underlying Algorithm by Inversion View
3.1
Character Counting
IOI circuit in GPT-2 small
3.3
3-Digit Addition
3.4
Factual Recall
Discussion and Related Work
5
Conclusion
10
Practical Guidelines
16
A.I
Observing Larger Neighborhoods is Important
16
A.2
Sampling with Decoder Model
16
A.3
Selecting Position in Samples
18
A.4
44
Thressolld-Deppendence of Claims about Activations
18
Experimental Verification of Completeness
20
Decoder Model
20
Character Counting: More Details and Examples
23
D.I
Implementation Details
23
D.2 More Examples of Inversion View
23
D.3 Causal Intervention Details
25
D.4
Extended Algorithm with Positional Cues
26
101
Task; Details and Qualitative Results
27
E.
Implementation Details
27
E.2 More Examples of Inversion View
28
Qualitative Examination Results
28
Choice of Distance Metric in [0]
29
3-Digit Addition: More Details and Examples
32
Implementation Details
32
R2
More Examples of Inversion View
32
Model Deficiency
32
Qualitative Examination Results
33
F.S
Causal Intervention: Details and Full Results
33
15

---

Decoder Likelihood Difference
38
Factual Recall: Detailed Findings
39
H.1
Background
39
H.2
Selecting Activation Sites
40
H.3
Decoder Training
................................
H.4
Sampling from Decoder
H.5
Observation
42
H.6
More Examples Inversion View
45
Notes on Attention, Path Patching, DLA and others
45
Inversion Reveals Information from Un-attended Tokens
45
Additional Discussion about Path Patching
47
Additional Discussion about DLA
47
1.4
Methods Generating in Input Space
47
J
Automated Interpretability
48
Compute Resources
54
Practical Guidelines
A.1 Observing Larger Neighborhoods is Important
Here, we illustrate the importance of inspecting <-preimages up to the threshold rather than just top-k nearest neighbors of the query activation. In Figure 8 an initial glance at the samples on the left may suggest that the residual stream of "+" encodes Fl and F2. However, observing broader neighborhood (as depicted on the right) reveals that this conclusion not even robust to tiny perturbations of the activation. Indeed, after more comprehensive calculation over all possible x4 d.ppost we find that the maximum possible metric value between any pair of 0.post is 0.0184. So for any e 2 0.0184 the e-preimage covers the entire input space. Hence, the activation is unlikely to contain usable information.
O.post has no effect on the model's We further prove this by causal intervention. We found that x
output. Concretely we patch x O.ps with its mean on the test set (mean ablation (54)) and for each prediction target (A1, A2 etc.), we compare 1) the KL divergence between the distribution before and after patching. 2) logit decrement rate, which is the difference between the maximum logit value before patching and the logit value of the same target token after patching, divided by the former. E.g., 1.0 means the logit is reduced to zero (assuming it is originally positive). The results are shown in Table 1 We can see the effect of U.post is negligible.
4
A1
A2
A3
A4/E
KL divergence
8.1 x 10-8
5.4 x 10-7
10
3.6
10-9
Logit decrement rate
-2.3 x 10-5
8.7
1
1.4 x 10
2.8 x 10-7
Table 1: Activation patching results for x S.psst
A.2 Sampling with Decoder Model
In Section 2 we mentioned that the distribution p(x[29) is modeled by the decoder. Strietly speaking, p(x[2?) represents the data distribution in the e-preimage defined by 0. For example, when the
16

---

Increasing Coverage by Temperature and Noise. In our experiments, we use both ways to control the generation, i.e., by adjusting the temperature and adding random noise to the query activation. We denote temperature as and noise coefficient as The noise vector consists of independent random variables sampled from the standard normal distribution and then multiplied by std(z") -7 where std(.) stands for standard deviation. In our web application, we provide multiple sampling configurations: four configurations which 7 (0.5, 1.0, 2.0, 4.0) and 1) 0.0 (only for addition task); one figuration named "Auto" which is sampled by following procedure: we iterate over few predefined r (ranging from 0.5 to 2.0) and 7 (0.0 or 0.1) and sample a certain amount of inputs (c.g.. 250) for each parameter combination. We then calculate the metric value for all inputs collected from different sampling configurations. We then randomly choose small part of them (100) with different probability for in-e-preimage inputs and out-of-e-preimage inputs. We dynamically adjust the probability such that the in-c- in-c-preimen preiiage inputs account for 60%-80% of the chosen set of inputs different noise in factual recall task, which will be described Activation Site: x post
Query Input
0.000; B 8 2 (+) 344 9
Generated Samp les
0.000; 8 (+)-..
0.005; B 0 (+)
0.000; B 8 (+)
0.005; B 8.188 (+)
0.000;
(+)...
0.005; 8.8 (+)...
0.000;
(+)-.
0.006; B 9 (+)-.
0.000;
8 9 (+)
0.006; 8 0 (+)
0.000:
(+)..
0.006; B 8 (+).. ...
0.000:
(+)-..
0.007; 4 (+)-..
0.000;
8 (+)..-
0.007; B 4 0 (+)...
0.003; 8 8 (+)..-
0.007; B 1 2 (+)
0.003: B (+)...
0.008; B 9 (+)-..
Figure 8: Addition Task: Inspecting e-preimage avoids pitfall of inspecting simple top-k similar activations. Generation based on query activation 24 0.post of random example. Contents after "+" is omitted since they do not affect the activation due to causal masking.
probed model is using causal masking, and certain activation is relevant to all previous context (by non-zero attention weights), then p(x(29) is the distribution over those inputs that share the same previous context (i.e., they have same prefix). This requires that the decoder can distinguish any tiny difference in activation and decode the full information (imagine a token attended with 0.0001 attention weight). Such a decoder must be very powerful and perhaps trained without any regularization. But in practice, the decoder is a continuous function of activation and tiny changes in activation sample that lies at the same point (producing the same activation) as the query input in vector space, instead usually generates samples that are in the neighborhood of the query input. Because we need to observe the whole neighborhood of the query input and prevent samples from being too concentrated, we adjust the sampling temperature to control how concentrated they are. Importantly, even if the decoder is too powerful and can always recover the same activation, we can still obtain the neighborhood by adding random noise to the query activation before giving to the decoder. This motivates decoding with temperature and noise, as described in the next paragraph.
perceivable by the decoder. We observe that the decoder rarely generates the
(when this is possible). Note that later in Appendix [H.4
When inspecting the samples, we choose configuration for which the distances D(:,.) -) to the query activation best cover the interval [0, <). The choice is usually specific the activation site that we are inspecting and can be performed manually in the web application.
17

---

<lendoftext><After Erin and Justin the house, Erin gave ringl Justin
Query Irput
0.000; : Hendoheec]-eeer Erin and Justin the house, gave ringf Justin
0.024; -jnndffaxc--i The station Sara and Justin went to had kiss. Sara gave to) Justin(EOS)
0.024; -innd/ahosss--heee Paul and Justin got kiss at the school, Paul deciced give to) Jusinn EOS)
<inndfteec--The Alicia and Justin had long argument. Afterwards
Justin(EOS)
0 -eenol:-esolllt---T Justin and Erin the garden. Erin gavel a basketball to Justin(EOS)
0.007; -enddfleecc---lee Alter the lunch in the afternoon, Justin and Kristen went to the station. Kristen gave a kissl tot Justin(EOS)
0. o,609: jendeefoooos------- taking long break Kimberly and Justin went house, Kimberly give bone tol Justi[[EOS]
0.042; (endoftec--Whileeeee spending time together Justin Alicis working the garder, Alicis) gawe)
JuS)in((OS)
0.0 0,048; lendoffeeeeeeee---- Ther, Justin and Kristen went to the school. Kr Kristeni gave) bone to Justin[EOS]
0.196; sjendhhest-Frieee separated at birth Kristen and Justin found a snack the garden(] Justin gare Kristen[EOS]
0.575 sendoffess While spendiig time together Michelle and Joshus were commuting to the restaurant(] Alexander gave a ring MichelalEOS)
Figure 9: IOI: InversionView applied to Name Mover Head 9.9 at "to"; Unlike Figure 4b here the position minimizing ((::) :) is in parentheses. The head also copies the name "Justin" in other circumstances, e.g., at "gave". The name "Justin" is always contained
A.3 Selecting Position in Samples
As the decoder outputs an input but not the position of the activation, we then assign the position minimizing :): to the query activation. Usually, there is only one position with a small D(.)), matching the structural position (not necessarily the absolute position) of the position the query activation was taken from (e.g., the target character in Figure [a). In certain cases, we visualize -(-,)) for an activation from a position not minimizing --,-)) for expository purposes. For example, in Figure 3b because the target character exclusively attends to itself in layer 1, resulting azc ~ a!:0, so sometimes the metric value of aze is smaller than a.:0. Throughout the appendix and our web application, we use italic font and rounded bars to visualize ((-,)) in such cases.
1.0
We also find that selecting the position minimizing D(--)) can reveal that components are active in similar ways other positions than the one originally investigated. For example, in Figure g we can see sometimes "gave" is selected. This is reasonable, because the IO is also likely to appear right after "gaae" and the head needs to move the IO name for this prediction. We can see that activation at the period "." can also be somewhat similar to the query activation, this is not surprising. Because the model needs to predict the subject for the next sentence and copying a name from the previous context is helpful. In summary, the copying mechanism can be triggered in circumstances different from 1OI, selecting position minimizing ((...) reveals more information about this.
A.4 Thhreshodd-epeendeee of Claims about Activations
One question people may have is whether our conclusion about the information in activation depends significantly on the threshold we choose. To address this potential concern, we show more details about the geometry of the vector space in Figure [0 On the one hand, we can see that with different thresholds we can make different conclusions about the query activation. On the other hand, the conclusions made with different thresholds are "in alignment". In other words, the conclusions do not differ fundamentally, instead, the difference between them is about granularity or the amount of details being ignored.
results in the conclusion that the count is 5, the target character is either 't' or "m', and also approximate sequence length is retained. E2 results in a conclusion only about the count and the sequence length. In Figure [0b] if we set the threshold to 6j (i.e., value between 0.000 and 0.009), the obtained information will be F1=5, S1=7. If we set the threshold to c2, the information will be 5 and 7 are in the hundreds place. If we set the threshold to cy the information will be is in the hundreds place". Figure [0c| é results conclusion that 9, 8 are hundreds place and 2, 7 are in tens place; results in conclusion that 9, 8 are in hundreds place and F2+S2=9; <3 results in conclusion that F1+S1 and F2+S2=99. Therefore, changing the threshold value will not lead us in a different direction, because the e-preimage is based on the same underlying geometry.
Specifically, in Figure 10al
18

---

B m
m m m
m
(
B 5 5 1287
8920 +
Query Inpur
Query' Input
Query Irpur
0,000; m m
0.000: B 7.37 1287
0.000; 3 9 2 87.8
Generated Semples
Conerated Saeples
Gererated Sumplex
0:012;
0.000;
0.003; 9
0,015; 5; www.pooppooooooooooooooooooooooo (
0.000;
0.004;
E
0,015;
0.000; 8
8
8
0.005; 8.2 220 9
4
6:
0.000; 3 2811-
0
1223E
0,012; 8 211 B
T95E
0.07; 5:
m H.
0,000;
0,015; T44
801E
0: 5: q Qttc aeqati99ll0
0.000;
0.01;;
0.000; 0 8
1 1260E
0.015; 2
;1:
a
0.000; 8 46
0,016; 4
DE
0,023; 5: 8 m t qtmpqtq [t:0
0.000; 8 17 1
8:000; 6 6 B 1
TE
0,024
h S.E.
2
0,010;
0:02;
0.000; 50
4 1291E
0.01;;
6,025 5, 8
thezch]t
0.000; 8
8
1316E
0.047;
801E
0,02;;1:5
1000ott]:888
0.000: 0 65 9
1364E
0.061; 8 35
B
9E
0,000; 9
1352E
0.058; 3
8 1
0:03;;1:5;
(0000000
0,000;
0:0
Gho0s 30 7
1281E
0.060; 6
DE
0::7::::5;
0.009 8 4 T 12S1E
0.062; 8 111111
B
5E
0.009: 8 87 0
0.063; 9
B
#::
0000|m]]5E
0,009; 2 2
0::::;t:5;
hddmdm/m
0.009; T 1280E
0.004;
T
0:042;1:5;
8
0.009; 9 9 6 1392E
0.065; 8
85
T
0:04):8:5;
0.009; 87 -
1251E
0.066; 8
0
7966
;:::;
B
p m 6
0.009; 17 2
1238E
0.067; 14-
TE
0:043;1:5;
8
h (
1
()
0.009;
1284E
0.069; 9914-994441
BE
8::4;;::;;
0.009 8
22822
2
0.082; 89
B
7999
0:01;;1:5;
8
Rur3: 6
161111261
0.099; 8 4 4
9
BO2E
(c
0.073; 9
5 1172E
0,120;
801E
8:
0,073;
8:068;
B
a 5
0.073:
0.124; 3995
D
0.302; 8 5
B
1312E
2211; 0901:8901-100
78
69
0:
0.102: 8 6 4
1402E
0.226; 3 5
B
TOTE
0.129 0:129;
8:
B
m
m m m 5
69
0.13S
B
h | 5
0.106;
13STE
0,249;
8:14
0.306;
8 19196
1396E
0,251;
711E
0:
0.106; 0 6 9 14660
0,255: 8
B
7OBE
0:148;0:6;
B
0.106;
1410E B
0 98 B
TO2E
begg8gl
9:
a ggarra/ri
0.129; 99 -
0,289;
694E
0:31:::3:
B
(m
0.130; 8 750E
0,89; 60
8:
2146; 9
10
0294; 8894
60
85
69TE
0:
8:33; 817-1472
0,29;; -
695EE
0:
B SPPpspppbipqi10/
0:436;
606E
0
Bty t
0.47: 8
0.406;
0::0;#:0;
0,509 185
0.417:
26
8115
(b)
(c)
Figure 10: (a) Activation site a 1.0 (b) Activation site 0.2 (c) Activation site a In all three cases, we use normalized Fuclidean distance as the distance metric. We use 61, 62, ... mark varying threshold values by which different interpretations will be made.
19

---

In practice, rather than selecting threshold first and treating inputs in saack-and-whie manner, we first observe the geometry of the vector space and obtain a broad understanding of the encoded information, then choose reasonable threshold that best summarizes our findings. In other words, the threshold value is used to simplify our findings so that we can focus more on the big picture of the model's overall algorithm, and it should also be set according to the difference that is likely to be readable for the model. As the interpretation progresses, one can see if the chosen threshold leads to a plausible algorithm and can adjust it if necessary. Finally, verification experiments are conducted to verify the hypothesis.
B Experimental Verification of Completeness
we described that an ideal strategy for obtaining samples in the e-preimage satisfies two desiderata: it only provides samples that are indeed within the 6-preimage (Correctness), and it provides all such samples reasonable probability (Completeness). As further described there, can directly ensure Correctness by evaluating D-) for every sample. Ensuring completeness is more challenging, due to the exponential size of the input space; the most general approach is to design counterexamples not satisfying a hypothesis about the content of the e-preimage, and verifying that D--) indeed large.
In Section 2
Here, we provide a direct test of completeness in one domain (3-digit addition). The primary concern with completeness is that, if some groups of inputs in Bze J.e systematically missing from the generated samples, one may overestimate the information contained in activations. To see if may happen in reality, plotted log-probability against distance, each of which includes all inputs in digit addition task, as shown in Figure TT] We next evaluated the sampling probability for different sampling configurations, as described in Appendix A.2 When adding noise, we calculate probability of an input using a Monte Carlo estimate: Concretely, because the probability of inputs is conditioned on the noise vector added to the query activation, randomly sample 500 noise vectors from the normal distribution (with the standard deviation described in Appendix (A.2) and calculate input probability given these noise vectors, then average obtain the estimated probability, and then compute the logarithm.
Across setups, we can see that there is triangular blank area in the bottom left corner, i.c., the bottom left frontier stretches from the upper left towards the lower right. In all sub-figures, not a single input close to the query input is assigned disproportionately low probability. All inputs in the e-preimage (the dots on the left of the red vertical line) are reasonably likely to be sampled from the decoder, with probability decreasing the input becomes more distant, alleviating concerns about completeness for these query activations. On the other hand, some inputs distant from the query input are also likely appear in samples, but this is not a problem for our approach, as can easily tell that they are not in the e-preimage by calculating the distance (correctness is ensured).
We can see that sometimes the distribution of the decoder itself (at temperature I and no noise) is quite sharp, and in-e-preimage inputs can have low probability as they are near the boundary of preimage. By comparing the sub-figures, we can see both increasing the temperature and adding noise substantially smooth the distribution within the c-preimage, lowering the difference of the probability of inputs that are at similar distance.
C Decoder Model
The overall training and sampling pipelines are shown in Figure 2 In this section, we describe the architecture of the decoder model in detail.
The decoder model is basically a decoder-only transformer combined with some additional MLP layers. In order to condition the decoder on the query activation, the query activation is first passed through a stack of MLP layers to decode information depending on the activation sites and then made available to each attention layer of the transformer part of the decoder, as depieted in Figure T2
Processing Query Activation. The query activation 2 Rd is first concatenated with trainable
activation site embedding eact E Bddase, producing the intermediate representation z [2*; emet] We chose daite to be the number of possible activation sites in the training set. The result z (0) is
20

---

gf 9282-717-9911EEP
gh B876+225-11101E
2 8188-558-74411
A 96:4-940- (21554
-23
lag Probability
0.00
0.33
1:00
1:5
5.00
5.75
100
(1111730
-25
-0
-50
-15
110
0.35
8.91
02
0.5
Distance
(a)
*i 9282-717--90115P
2Az 087744225-1110011
x² pMst B6:44 ()
10
-10
log Probability
300
*5 075242000100178
09074230-1111077
-00
40
-40
0.00
0.22
0.30
0.75
1.0
0.
03
1.0
15
0.0
0.
32
03
0.
000
0.
1.0
15
Distance
(b)
6-f 9282-737--99D1EP
g 86751225-111011
2 8189-559-74491E
2 2.Mrt 36.4- 96:4-940-(15544
-20
-0
-0
log Probability
-.0
-43
28
1:00
a/ 0752+286- 1013GE
311 9406+334--9X400E
2307-230-1111871
B0GB 3711-1844F
-25
-0
-10
-30
3.00
0.2
0.50
0.75
1.00
0.
1.0
15
3.0
0.2
32
63
04
0.0
0.5 1.0
15
Distance
(c)
Figure 11: Addition Task: Exhaustive verification of the decoder's completeness for 8 random query activation. Failure of completeness would mean that some inputs result in activation very close to the query activation but nonetheless are assigned very small probability. Here, we show that this does not happen, by verifying that all inputs within the e-preimage are assigned higher probability by the decoder than most other inputs. We also show that by increasing the temperature and adding random noise, can increase the probability of inputs near the boundary of e-preimage. Each sub-figure (a), (b), (c) contains 8 scatter plots, each which contains 810000 dots representing all input sequences in the 3-digit addition task. The y-axis of scatter plots is the log- pesbaabbbblity of the input sequence given by the decoder (which reads the query activation), the x-axis is the distance between the query input and the input sequence. As before, distance is measured by the normalized Euclidean distance between the query activation (the activation site, query input, and selected position are shown in the scatter plot title) and the most similar activation along the sequence axis. In addition, the red vertical line represents the threshold e, which is 0.1 in the case study. (a) Temperature T = 1.0, 2.0, no noise is added. (c) Temperature 1.0, noise coefficient 0.1 (See Appendix A.2 for explanation of 1)).
no noise is added. (b) Temperature
21

---

zUn)
N8OS
Mika
MLP
LN
UN
Linser
Xyhe
MLP
LN
UN
00)
Unnear
N8OS
Cart
[BOS]
In
the
Figure 12: The decoder model architecture used in this paper. The query activation is processed by stack of MLP layers before being available as part of the context in attention layers. We use transparent blocks to represent model components inherited from original decoder-only transformer model.
then fed through multiple MLP layers (each layer indexed by p (0, P 1)) with residual connections:
(x+1) MLP(LN(Z())) 22))
where LN represents layer normalization. For each MLP layer, the input, hidden and output dimen- sions are d + deite. d, and d + daite. respectively. The activation function is ReLU. There is also final layer normalization, ((f)) = (P))
Integrating Query Activation. As we want make the query activation available to each attention layer of the decoder, we separately customize it to the needs of each layer using a linear layer. That is, for each layer of the transformer part of the decoder (indexed by L, so {0, 1,... 1}).
we define a linear layer (f iddtee - Rdaccder and a layer normalization LN(3) where ddecoder is the model dimension of the decoder model:
( (Linear (0 (z()))
We add superseript (f) to the model components to emphasize they are layer-specific. In the E-th layer of the transformer part, (() is concatenated with the input of the attention layer along the length axis before computing keys and values, so that each attention head can also attend to (() in addition the
context tokens. This means that each head in the ê-th layer, instead of attending si .22 now
(e) (e)
(f) computes its attention weights over ((), .i I42 .... Here xt is the residual stream corresponding to the /-th token input into the E-th layer.
(6) (e)
Motivation for Architecture Design. At the beginning, we train separate decoders for each activation site, but this not very scalable when there are many activation sites. In the architecture activation site embedding Cact signal trigger different processing, and functions or model components that are needed for all activation sites are shared. Similar reason applies to the linear layer (f) we expect the z should be transformed differently for each transformer layer, but having separate MLP stacks to process z for each layer would largely increase the number of parameters. In preliminary experiments, we also try "encoder-decodes" attention layer. That instead of providing query activation in self attention layer, we add new attention layers that analogous to the encoder-decoder attention layer in original transformer architecture, where each token can attend the processed query activation as well as blank representation (similar to the function of "BOS", so that "no-op" is possible). However, we do not find significant difference between this design and the aforementioned one. Therefore, other than adding components for processing z we do not modify the decoder-only transformer architecture, so that we can also choose use pretrained models. We note that there are other possible choices for conditioning the generation the activation, and we didn't 't optimize this choice thoroughly.
above mentioned,
22

---

Decoder Hlyerpaaaameters. Regarding processing query activation, the decoder has 6 MLP layers, ie.. = 6. The decoder model has 2 transformer layers (L 2). 4 heads per layer, and model dimension of 256. The attention dropout rate is O. Other settings are the same as the GPT-2. We use the this architecture for the characcrr counting, IOI, and 3-digit addition- in the paper. Regarding the factual recall task, we use the same architecture for processing query activation, i.e. P = 6, and use pretrained GPT-2 Medium (24 layers) as the transformer part of the decoder.
Training Details. We construct the training dataset by feeding in-domain inputs to the probed model, and collect activations from random activation sites and random position as query activations (the choice of activation sites and position is specific to each task and described later), we also record the activation site they come from. For each input, we could obtain many possible training examples because of many choices of activation sites and position. So we do not iterate over all possible training examples. We sample certain amount of examples to train the decoder for I epoch, using constant learning rate of 0.0001 and Adam optimizer with weight decay of 0.01. Other details (e.g., amount of examples, training steps) are task-specific, and can be found later in their own section.
During training, we regularly calculate the in preimage rate, which serves proxy for generation quality. Concretely, for a fixed set of query activations used for testing, the decoder generates samples with temperature=1, we then compute the fraction of samples inside of e-preimage (with e = 0.1, D as normalized Euclidean distance). The rate is calculated for each activation site. We usually observe difference between the ratios for each activation site, indicating some inverse mappings are easier to learn (we also observe these activation sites tend to have clearer information). Overall, we usually see a continuous improvement on the average rate during training.
D Character Counting: More Details and Examples
D.1 Implementation Details
To construct the dataset, we enumerate all the 3-combinations from the set of lowercase characters defined in ASCII. For each combination, we generate 600 distinct data points by varying the occur- rence of each character and the order of the string. The occurrences are sampled uniformly from 1-9 (both inclusive). So the length of the part before the pipe symbol ("I") lies in [3, 27] (not considering "B"). Like 3-digit addition, we split the dataset into train and test sets, which account for 75% and 25% of all data respectively. The input is tokenized on the character level.
The model is a two-layer transformer with one head in each layer, the model dimension is 64. All dropout rates are set O. The model is trained with cross-entropy loss the last token, the answer of the counting task. The model is trained with batch size of 128 for 100 epochs, using a constant learning rate 0.0005, weight decay of 0.01, and Adam W (36) optimizer. The training loss shown in Figure [3] we can see the siair-like pattern. An interesting future direction is to investigate what happens when the loss rapidly decreases using inversionView.
With regard to the decoder model, the architecture is described in C We select ,.0prre ..f.imdd p.f.post a²-J mm as the set of activation sites we interested in, where i (0), and m denotes MI.P layer output. The query activation is sampled from those activations corresponding to only the target character and colon. We sample 100 million training examples (all activation sites are included) and train the decoder with batch size of 512, resulting in roughly 200K steps. During training, as we mentioned before in[ we test the generation quality by measuring in-preimage rate. For those activation sites for which the decoder has a low generation quality, we increase their probability of being sampled in the training data. The final in- prerrimaee rate averaged across activation sites is 67.7%.
{0, 1).j
D.2
More Examples of Inversion View
See Figures
14 and 15
Here, we show results similar to Figure for other query inputs.
23

---

1.00
0.50
spoch
Figure 13: Training loss of the Character Counting task. Each data point is the averaged loss epoch.
paid
Paost
Query
0.000;
0.000; q a
Conerated Samples
Generated Samples
0.009;
8
0.006; 7; q
7
0.013;
8
0.030;
0.049;#:7;
q aeeeaqaea..
88 9
0.011:
q
0.054;
8
0.016; 8
E
0:
9
9
0.016; B
(7)
0.065;
a
8
8.158;
(.) 8
(x)
8: 0
0.180;
6:
0.153;#:;;
B
7
0.117;
a
0.207;
B
0:122;
B
w E
Figure 14: e-preimage showing function of MLP layer 0
Activation Site:
Activation Site:
www.
B Buooojojjo0ol
Query Input
Query Input
0.000;
0.000: B 000000o00000
Genorated Samplas
Ganerated Samples
0:02;;::;;;
0.081;
0:04;;::4
C
0.05;;
0:048;
8
0.069:
8:
0.075;
8:057;8:4;
8
0.072:
8
0.321;
a:;7;
8
0:323:::;
8 k mmkk0kkkkkkkkkkkk..k.kkkkkkkkkkk
5:
0:124;
8:
8: daddaaadd
0:263;
()
8:430; B mlmmmm
m
Figure 15: e-preimage of a. 10, As mentioned, we hypothesize that the attention head is reading the subspace where the count information is stored. One can presumably find this "count subspace" by optimizing projection matrix such that after projecting the activation there is only pure count information in the e-preimage, and compare it with the subspace read by the value matrix of the attention head. Therefore, Inversion View be potentially useful for subspace study.
24

---

Increment of (LDong LDpon)
LDong
10pm
Patching Effect ->>
Increment
30
20
20
10-
none
a
07.0
Patching Effect
40
Increment
30-
20
10-
-20
0.
Patched Activation
Figure 16: Results of activation patching for model trained on character counting task. Same figure as 4a with intermediate steps of calculation shown using line plot. Note that the gray lines correspond to the y-axis on the right. In contrast examples, only one character differs. L.D stands for logit difference between the original count and the count in the contrast example. L.peh and LDorrg correspond to the LD with and without patching, respectively. Top: We patch activations cumulatively from left to right, flipping the sign of LD. The "none" on the left end of x-axis denotes the starting point, i.e., nothing is patched. Bottom: We patch from right to left. Similarly, "none" on the right end of x-axis denotes the starting point.
D.3 Causal Intervention Details
For an input example Xorig with forig as the count (final token), we construct a contrast example Xcon with different count foom by changing random character before "I"". The contrast example is valid input (the count token the count the target character). We also ensure that the contrast example is within the dataset distribution (the count is in the range [1-9] and there are 3 distinct characters in the input).
We run three forward passes. 1) The model takes as input Xorig and produces logit values for count prediction, we record the logit difference LDorig between torig and foon (former minus latter). 2) We feed the model with Xcon and store all activations. 3) We run a forward pass using Xorig: replacing the interested activations (e.g, {a..0 4.00)) with the stored activation in the sites, and record the new logit difference LDpeh. Because the model can make the right prediction patch
position and activation
in most cases, can that average 1.Dpchh changes from positive negative values more and more activations. We do the same for all inputs in the test set and report the average results.
study, For example, the top of the figure, patch {a..", (a..0 are {a. age a..²3 respectively. Patching more activation results in increases of LD - L.D we attribute the increment to the newly patched activation. Hence, the causal effect of each activation is measured conditioned on some activations already being patched.
Figure 16 shows the LDorig and LD We cumulatively patch the activations 0,0
We sort the activations according their layer indices and show the results of patching from bottom to top (->) and from top to bottom (+-). In this way, we can verify the dependence between activations in the top and bottom layer. For example, at the top Figure 16 when 0,0 is already patched, patching a. has almost no effect. On the bottom, we also see patching azc has no effect if a. has been patched. So we verified that atc is the only upstream activation that a. relies on, and a! is 0.0 the only downstream activation that reads agc
0.0
25

---

Patching Effect
Patching Effect
Increment
30
Increment
30
20
20
10
10
0
0
e pns
a0.0
a1.0
a0.0
a
ac
tc eos
atc
e0os
e
Patched Activation
Patched Activation
Figure 17: Results of activation patching for model trained on character counting task. + and <- means the same as previously. Left: Patching with activation from examples with different counts. Right: Patching with activation from examples in which only one character differs.
D.4 Extended Algorithm with Positional Cues
In Section 3.1 we verified the information flow by an activation patching experiment which the contrast example only differs by character. These experiments verified that the algorithm described complete in distinguishing between such minimally different contrast examples. We now show that the model implements somewhat more complex algorithm that combines this algorithm with position- based cues, which become visible once we consider contrast example that differ in more than character, in particular, those that differ in length.
To show this, we conduct another activation patching experiment in which the contrast example is a random example in the dataset with different count. In other words, everything can be different in contrast examples, including the sequence length and the target character. Thus, we cumulatively patch four places:
1. etc and eos where gos stands for position embedding, because the final count correlates with positional signal, so the model may utilize it. They are patched together because the attention pattern of the colon in layer relies on their adjacency.
2. 0.0
atc (as before) and don where chnn stands for token embedding. They are patched together because patching only one of them would result in conflict between character information in the patched and the un matpathhed activation;
3. 3) a.:D; (as before)
4. a. (as before).
The result is shown on the left of Figure We can see that, besides the components we had detected previously based on minimal contrast examples (ate a,.'),, some other signal also contributes notably to the final logits. We compare with patching experiments for the same set of activations on contrast examples that differ in one character, shown on the right of Figure I7
Overall, besides the algorithm identified in Section 3.1 we find other 3 sources of information influencing the model's output. The position embedding, epo and e.oo, This is observable on the left of Figure 17 from which we can also know a. contains the position information (because the bars of ete and e are not symmetric). This is confirmed by Inversion View. As shown in Figure we see the inputs in c-preimage roughly follow the query input length, being independent of the count. Therefore, the model also utilizing the correlation between the count and the sequence length. 2) Attention output of colon, a. 0.0, which attends to all previous token equally. From Inversion View, we observe contains fuzzy information about the length (same as position signal), and the characters that occur in the context, well as their approximate count. Our causal experiment also shows that
26

---

Activation Site:
Activation Site:
B
(1)
Bettetnnntt
Input
Query Input
0,000;
Go Generated Samples
Generated Sarples
0.038; 2::2;
0.022; ttt
0.058 2:
0.023; 9; wtw.mtiiimmmmmmmmmmmmmmm.mmm.mmt
2:
0:028; B
0.083 F4:2; 22
8
0,034; BELtttthhrrrr//)))
0.072;
0.053; c/c.c//ccccccccccccccccp.
0.086;1:22 hww.
0.100; v: 8; wtw.:tttttttccccccccc1111000....
0:092; m]
0,122; 8; B
2::2;
0:
0.170;
0.67;;1:7;
0,458;
Figure 18: e-preimage of a! a. to show the position information is also encoded and independent of count information.
it does not contain precise count. Therefore, it contributes to the model's prediction in manner 0.0 From the attention pattern
similar to the position signal. 3) Attention output of the pipe sign, a. observe sometimes in layer 0, pipe sign attends selectively to type of character, e.g. "x", "k", or "J". Inversion View shows that it indeed contains the approximate count in that case (though the decoder has not been trained activation corresponding to pipe sign). In next layer, the colon also attends to the pipe sign if target character is the same as the character attended by pipe sign in layer 0. This explains why we can observe nonzero effect of patching a. when other activation is already patched (the red bar corresponding to a." on both sub-figures of Figure 17
Whereas patching with minimally different contrast examples allowed us extract an algorithm sufficient for solving the task in Section 3.1 patching with arbitrarily different contrast examples allowed us uncover that the model combines this algorithm with position-basee cues. The model performs precise counting using the algorithm we found earlier in Section 3.1] while also makes of simple mechanisms such correlation to obtain a coarse-grained distribution over counts. Overall, we have found the full algorithm by alternating between different methods - Inversion View, traditional inspection of attention patterns, and causal interventions, and confirming results from with others.
E
10I Task: Details and Qualitative Results
E.1
Implementation Details
In order to train decoder model, we construct dataset that consists of IOI examples. We used the templates of IOI examples from ACDC [13] implementation. For example, "Then, [B] and [A] went to the [PLACE]. [B] gave [OBJECT] [A]", in which "[B]" and "[A]" will be replaced by two random names (one token name), "[PLACEJ" and "[OBJECT]" will also replaced by random item from the predefined set. Besides "BABA" template (i.e., S is before 1O) we also use "ABBA" templates (S is after IO) by swapping the first "[B]" and "[A]". We generate 250k data points.
The architecture of the decoder model is the same as before, as described in Appendix C The set of activation sites the decoder is trained consists of the output of all attention heads and MLP layers (no residual stream). Note that when producing query activation using GPT-2, we always add the "< |endoftext] >" token as the BOS token. We do so as during the training of GPT-2 this or multiple such tokens that usually appear in the previous context can be used as a BOS token, which is possibly important to model's functioning. We use new token "[EOS]" as the EOS token when training the decoder. The query activation is sampled uniformly from all positions excluding FOS and padding tokens, and uniformly from activation sites the decoder trained for. We sample 20 million training examples (all activation sites are included) and train the decoder with batch size of
27

---

Tjendoftext/-Then in the morning, Stephanie and Nicole thinking about going to the hospital. Stephanie wanted give ringi Nicole Query
0.000; <ondehaxxi-Tees in the momning Stephanic Nicolo wore chinking goling hospizal, Stephanic wonted give ringt to)
Generated Sempless
-yndeffrett--hher then in the morning, Eris and Samah were thinking about going to the house wanted give ringl ta) Sarafiioo)
0,27; -jendcffeell-Teeee Then in the morning, Sanah and Sanah mere thinking about going the station. Surah-ownnted counted give ring! tu) Serah(BOS)
0,028, <jondettwwll-Theee Then in the momning, Lindsey and Emiiy were thinking about going thasiiaion. Lindsey wonted to give
Eriily [E0S]
0,055; <jndeehowii-Ther Then, Undsey and anthony were thinking shour going to the hou se. Lindsey wanted to give necklace to) Anthory(EO5)
0,002; =jendotreeit-Tee Then in the morning Relly and Richard were thinking about going the office. Relly w enteed gave a aingttg not Richerdlios)
0,075; -lendofeenl--Tee Ther, Kelly and Patrick were thinking about going the office. Kelly warned to give ring hol Pptrickfpoos"
0,085; <jnndohaett-Thie Brian and Katie wore thinking about going to the school. Brian wa rted to give ringl to) kat
0.129; sjerdefheett--hee
thinking going to
Devid
ringf Michwlle]ross)
Sendeehtttt--- the morning.Jo Amber thinking about going office,
John(EOS) Sendehaettteeeeeeee the Elizabtt Tyler thi thinking going the station. wanted nodklace) eliaabchhEEES]
e- e-preimage of S- Inhibitiio Head 7.3, The relative position of S1-but not its identity-is contained in the head output (together with some template information). That means, in the samples within the e-preimage, S1 always appears before the IO, While the relative position is encoded, the absolute position vary, can the identities of the names.
Figure 19:
<jendoftext/~Friendds separated at birth Bradley and Cody found a computer at the store. (Cody) gave to Bradley
Query Inpu:
0,000 Fenddctttttteeeieeddd separated Bradley and Cody computer the store. Cody) gave to Bradley
Generated Sumples
0,000 =lendoftet)) the morning afterwards, and Jacob had argument. Afterwards Jacob said CodylEOS) morning, Cody
0.000; [end:::-ooooteeeTThh
fun the school. Cody) gave bone Scott[FOS]
0,000 0,000; (eno::/jjoootott-Theeeee Then in the morning efterwards, Cody and Ryen hed long argument. Afteraadss Cody) said to Ryen[EOS)
0,000; #knocttssoot taking a long break Cody and
the station, Cody) gave camputer Lauren[EOS]
0.001, 0,001; [eoos===ennoeeeet-eeeeere Cody and Danielle went to the restaurant. (ody) gave besket to Danielle[FOS]
0,001 Then Cody and Jonathan had a lot of the house. Jonathan gave a bone toj Cody) EOS)
0.001
Cody had long argument. Afterwerds| Cody) said to Paul[EOS;
0,382; [ Then morning afterwards, Courtney Dustin long argument. Mitewwrrss Dustin) to Courtney(5O5)
0,392; FFlnddddtttttt--iddddssss separated at birth Alicia and Dustin found a snack at the school., Dustin) gave it Alicia[EOS)
0.385; =lendoftet soon afterwards Jason and Nicholas snack hospital, Nicholas decided the snack tol Jason][EOS]
Figure 20: e-preimage of Duplicate Token Head 0.1. S name is contained in head output.
256, resulting in roughly 80K steps. The final average in in-preimage rate is 58.0%, despite that the decoder trained for 157 activation sites.
E.2
More Examples of Inversion View
See Figures
19
20
21
E.3
Qualitative Examination Results
The qualitative examination results shown in Table 2 We summarize the description in 54 of each head category to facilitate comparison. Figure 22 shows the 1OI circuit in GPT-2 small, which is taken from their paper. For more details, please refer to [54].
There are some heads for which Inversion indicates a different interpretation. First, Head 0.1 and 0.10: (54] only shows that they usually attend the previous occurrence of duplicate token and validates the attention pattern on different datasets. However, there is no evidence for the information moved by these heads. Thus they only hypothesize that the position of previous occurrence is copied. Second, Head 5.9: The path patching experiments in [54] show that head 5.9 influences the final logits notably via S-Inhibition Heads' keys. But there are no further experiments to explain the concrete function of this head. While the authors refer to it
Fuzzy Induction Head, the induction score (measured by the attention weight from a token T to the token after T's last occurrence) of this head shows a very weak induction pattern. Even if such pattern occurs, it cannot tell us what
28

---

<jendoftexti<After taking long break Bradley and Rebecca went to the store, Bradley) gave a basketball to Rebecca
Qvery Inout
6. 6,000; =jerdofteri/siiee taking long break Dradley and Rebecca went the store, Bradley) gave a baskerballl to Rebecca
Generated Samples
6,002; -erdoffles)-AAee taking long break Bredley end Gregury went station, Bradley) computer Cregory(EO5)
0.014; 4endohaxtpphhtr taking long break Kelly
went the rasturant, Kally) gave bone Aaron:OOS)
0.015; dfndoooooo--iteerr taking long break Kelly Aaron went the hospitall Kelly) gave necklace to Karon EOS)
0. 0.019; (erd://eerdofee)ekeeee taking long break Kyle and weul/went went to the hospital Kylel gave snack Psul
o 0.020; <erdoouaeeeeetete taking long break Travis and Andrea went to the restaurant, Andrea gave a kiss toi Travintt:OS)
0.022: <endottocc-tttter taking long break Megan and Benjamin went the restaurant, Bonjamin gave ring tot Megani]EOS)
0. 0,024: =eddoffoeti--eeerrrr taking long break
and Jeshua
garden, Joshua gave necklace Lauren[[EO5]
=/erdoffooeee//frrr taking long Nichelle
went restaurant) Michelle) gave kiss Amands(:Os))
6.032; www.deodeee/hiiileoeeooo spending time together Dust n Sarah were working at the station, Sarah gave a toa DustiniJBOS)
6.370; -lendofooet) taking long break tradley Andres
restaurant, gave computer (ndree)((
Figure 21: e-preimage of Induction Head 5.5. Position but not identity of the current token (token
in parenthesis)'s 's last occurrence is contained in head output
Legend
When
Mary
Key/Volue
John
Query
Class Heads LayerHead
Output
Frevious Takan Heads
51+1 went
to
store
Negatiie Narne Mover Heads
Duplizate Toxan Heads
Induction Heads
S2 Dohn
as 69 [58
gave
Narne Mover leach
drink END 16
5-Inhibition Heads
Backup Name Mover -eeds
Figure 22: IOI circuit in GPT-2 small. Figure from
information is captured by this head. Interpretation with Inversion suggests that the head barely contains any information, within the input space of IOI-like patterns. One possibility is that head 5.9 recognizes the IOI pattern (i.e., there are two names and one is duplicated in the previous context), so that if an 1OI-like pattern exists, S2 should be attended to by S-Inhibition heads. As the decoder model is trained on IOI examples and generates mostly IOI examples that is, the input space X in [I consists of IOI-like inputs, this information is by definition not visible. Expanding the input space to arbitrary language modeling would allow capturing such information; we leave this to future work.
E.4 Choice of Distance Metric in IOI
As described in Section 3.2 we used cosine distance for the IOI task, while for the other two tasks, we use normalized Euclidean distance. In this section, show that both distance metrics produce similar interpretations while cosine distance makes the meaningful patterns easier to identify.
In Figure 23 and 24 we show two examples of using normalized Fuclidean distance as the distance metric. Readers can see more examples on our web application. We see that the samples generated by the decoder tend to have larger distances under normalized Euclidean distance. Nonetheless, we can still see the top-ranked samples show meaningful commonalities - indeed if we set e = 0.4, we obtain the same interpretation as the one we obtain based Figure 4b and 19
While normalized Euclidean distance can produce a similar interpretation, we have to set much larger which we find less intuitive. We believe this to be because of the differences in the activation dimensionality, which is 768 in the 1OI task, much larger than character counting (64) and addition (32) tasks: Under Euclidean distance, the ratio of all close samples to all possible samples becomes lower and lower when the dimension becomes higher. In other words, the volume of the e-preimage accounts for very tiny proportion of the whole space in the nigh-dimensional case. By using cosine
29

---

Whether Consis-
Head Category
Function According 54
Observation from Inversion View
Position
tent
Copy whatever it attends to (in- crease its logit). It will always attend to 10 because of previous components.
Name Heads
Mover
END
9.6 10.0: 10 name
Yes
Negative Name
Copy whatever attends (de- crease its logit), attends IO Move info about and cause Name Mover Heads to attend less to They attend to $2. They adds two signals to resid- ual stream, one is token value
END
10.7 11,10: 10 name
Yes
Mover Heads
7.3 8.6: Position of SI (relative position IO, same for the fol- lowing when say posi- tion)
S-Inhibition Heads
END
Yes
7.9: name; Position of
of S, the other is position of S1, and position signal has greater effect.
8.10: S name; Position of S1 (most of the time)
Attend from S2 SI, gen- erally, attend previous dupli- loken, and copy the position this previous occurrence.
0.1 0.10: S name (instead of position)
Duplicate To- ken Heads
S2
Partly
3.0: Pusition of the duplicated name
2.2: S name (most of the time, sometimes attend to "and" when S1 is after 10 and make S name less important)
Attend the previous token, the token (name) S1+1, then this information used key S1+1 in Induction Heads
Previous Token Heads
S1+1
Yes
4.11: S name; Position of the previous token (relative to the other name)
5.5 6.9; Position of the current token's last occurrence
Attend S1+1 and posi- signal (similar the func- of Duplicate Token Heads, while different from normal in- duction heads)
5.8: Position of the current ken's last accurrence (most time)
Induction Heads
Partly
5.9: Almost no info, possibly some info about the template.
Do not move 10 normally, but act as Name Mover Heads when they knocked out.
9.0 10.1 10.10: IO name 10.6 S and/or IO name (Most of the time: IO, sometimes: IO and S, occasionally: S) 10.2: S and IO name (most of the time), S or IO name (some- times)
There are 4 categories:
9.0, 10 10, 10.6: similar
Backup Name
Name Mover
END
Yes
Mover Heads
10.2, 11.9: attend to both S1 and
10, and move both
11.9: S and/or 10 name (no ob- vious difference frequency) 11.2: and name (most time). name (sometimes) 9.7: name
11.2: attends and
attends to and writes negatively
Table 2: Column "Position" means the query activation is taken from that position. "S1+1" means the token right after Rows are ordered according to narration we say "S name", it means the the name of S in the query input, but the name is not necessarily $ in the samples. This also applies to "IO name". The information Iearned by Inversion View which is different from the information suggested by Wang et al. [54] is in bold.
original paper. When
30

---

:|endoftaxt/<After Erin and went to the house, Erin gave ring( to) Justin
every
wjendhhud/sfftee Erin Jusin went to house, Erin gove a to) Juain
Samp
0:208: -jendsfedd-WWhlee spending time together frin and Justin working the house, Erin give computert to) Justin[[05)
0:243; #edds/edndseeee
working the hospital,
JustingeOS]
Teendottes--henn the mominng Timethy
the garden. Timethy gave snaca(ta) Justin
-fenddfladd murning, Benjenin
king engument, afla munds Bunjanis sardf (u) (EOS)
0:200: +jendsftedd spending together
commuting to
gave Justin)tOs)
0.73; (endo//enootted/ rinds reparated Petrick and Justin computer at hospital. Patrick gave it; Justn[[COS]
0.275 275d-jenndsttod--iii
Justin were commucing to the office, Daniel ge a Justin(EOS)
0.302 wendn/oodddddwwllcco... apending time together Thomas Justin were wor king school, Thomas gave ringt toj JustinEE5))
0.311 HWhile spending time together Sean and Justin were working at hospital, Sean gave computer nol Justi ([os)
33Td:-eensttee--1hhee Justi and kristen went to the hospital, Knttten gave) a bone to Justi[
0.464 #endst/doo spending ng time together justin and Krtsten were commuting to the garden, Krnstenn govel a bone Justin(EOS)
0.481 fand::/tndtdtd-e---eo Thon the momiing Justin and kr sten had oo fun at the hospital,) 1, Justin gave baskethall to Wiste(Ees))
0.54: +endshhdd Then in the moriin, Juse in and kr stenn wern wor winking at the officet)) Justin derided to g bone krisen)
0,783 (jenss//ddddddiiie spending time together Joshus and Justin were commuting munttnggg to the stationt) Justin gave ring to (oshux/Fo5)
0.940 wiil Jomes and Kristen were commuting to the school, Jamest gave) a snack to krieten(ED)
Figure 23: c-preimage of the same activation as Figure 4b using normalized Euclidean distance instead of cosine similarity (Appendix E.4). The line shown in the figure still represents threshold of 0.1. While at = 0.1, some query inputs do result a substantial number of in-e-preimage samples, many cases result in very small or even empty (as here) sample sets, suggesting that the e-preimage at 0.1 at least as accessible to the decoder - is extremely small in this model, which we speculate is related to the models higher dimensionality compared to the other tasks (768 vs 32/64), which tends to make Euclidean distances large except for extremely similar vectors (see Appendix (E.4 for more discussion). We find it more convenient and natural to use similar thresholds (e 0.1) across tasks and account for the different geometries using different distance metrics. What is key, however, is that for appropriately higher e (e.g.. 0.4) again always obtain substantial number of samples, and - most importantly these samples lead to the same interpretation as we obtained in our main experiments with the cosine similarity. Here, for example, we can obtain the same interpretation by setting 0.4, i.e., the 10 name is encoded in the activation.
<jendoftex(]- Then in the morning, Stephanie and Nicole were thinking about going to the hospital. Stephanie wanted to give a ring( to) Nicole
Query Input
0.000 - Tonnotteei-Then in the moming, Stephanic an Nicole were think ng aboutgoing the hospttl Stephanic wa nted to give ringt Nice
Demer sted Sumplu les
0.106; + jerdofess--then, Michelie and Patrick wew th minng about going the hosptial. Michelle wanted to ring! Periic(tos)
0.155 sjendoftxiptTeen in the moming, Sa aa and Jason were th thinking about gaing to the station. wamtdd to alve aringg to) Jason[EO5]
0. 0,230 =eendottwwl--Teen in the moming, Ui Lnddo and Joseph were th miing about going the ganden. Indoey wonteed to give ringt Jeseph(EOS)
0.77; Jendohestttht Shannon and Erin were thinking about going to the office sharnon wanted to give a ring! to) trinjcos)
0.277,
wwt..//oododudddoeoooo Then the moming, Kathorine and William were miing about going to the store. Katherine wanted to give ringt to) [illia]OOO]]
0,285 <lendoleeettttteee Then, Kelly and Rachel were thinking about going to the house, Helly wanted to give neckloce) toj Rach elBoo
0,202 jendo:jessssttteeee Then the morning Kelly and Courtney were think ng going to the house, Kelly wanted to give aing hoi Courtnvey [tos)
0,317, Sendddntttt---no Thenn the moming, Jumie and Robart were thinking about going to the garden. Jami
give necklaco to) Rober]]OS]
1/e3: dondotett--Teen moming, Undsey and Bryan thinking toingto station. Lindoey to give ace Bryan(EOS) thinking about gaing
jenstorrstt--hee Lindary
Linckey
give nedicanen (eos
dufuudt Then moming,
William think about going the Allizon wanted bus kuskettalli Willin(E0s)
-len dww.eoototttooooooooooooo Then morning and Any thinking
goi og the Army
ringt Johnteos)
6444; jerdofeest--hhee then the morning Richard and Scott werw thinking about gaing the gardeen Scott weenteed giow ring! (echwrl(
0.512; sjondd dohe Then, Jamio and velissa were thinking about going to the store. Malissa wanted to glve ringl tol Jamie]E05]
0 5388 dendootettt-Teenn in the morning Ur doyy and Chrtatina na were thinking about goingto the gardon. Christina wonted to gave bone) tol Lintrey(BOS
Figure 24: e-preimage of the same activation as Figure 19 using normalized Euclidean distance. The
line shown in the figure still represents the threshold of 0. As explained in Figure 23 due to the representation geometry, normalized Euclidean distance tends to require a much higher threshold to obtain a sufficient sample size for interpretation. Importantly, as also explained there, we still obtain the same interpretation as in our main experiments if we use normalized Euclidean distance but take higher threshold (c..g, 0.4): here, the relative position of S1 is encoded in the activation.
31

---

2.0
16-
14
12
epech
Figure 25: Training loss of the 3-digit addition task. Each data point is the averaged loss epoch. The final loss is still big since the two operands of the addition is unpredictable.
similarity, we allow more input samples to lie in close distance with the query input, as cosine similarity ignores the magnitude difference. This suggests cosine similarity may overall be more suitable when applying Inversion View in high-dimensional activations.
3-Digit Addition: More Details and Examples
F1 Implementation Details
We constructed 810,000 instances and applied random 75%-25% train-test split. The probed model trained with a constant learning rate of 0.0001, a batch size of 512, for 50 epochs (59350 steps). We save the last checkpoint the trained model, and test it on the test set. We use the AdamW (36) optimizer with weight decay of 0.01. The training loss is shown in Figure 25
With regard to the decoder model, we select 0.rr zimidd pripost as the set of activation sites we are interested, where (0, 1}.j E 2,,3, When sampling training data, we select an activation site and token position uniformly at random. We sample 100 million training examples (all activation sites are included) and train the decoder with batch size of 512, resulting in roughly 200K steps. The final average in- in-preimage rate is 88.6%
F.2
More Examples Inversion View
See Figures
26 to 29
F33
Model Deficiency
In Section 3.3
mention that there is firm and clear path of obtaining the carry from ones to tens, so the model may make wrong prediction. We examine those instances for which the model makes wrong prediction and find they all satisfy make the right prediction because the ones place matters. This is consistent with our interpretation of the model. Furthermore,
condition: F2+S2-9. In other words, it fails
check the model's accuracy this special subset where F2+S2-9, and find that it is significantly higher than chance level. The accuracy on training subset (training data where F2+S2-9) is 80.45%, and on subset is 80.06%, while chance level 50%. So, can infer that the probed model obtains some information about the ones place by means other than memorization, Indeed, we observe fuzzy information about ones place in a 1.0 and a occasionally (See Figure 30).
'We have also tried value- weighted attention pattern 32 it makes a negligible difference, so we always show the original attention pattern in paper.
32

---

Activation Site: %1
Activation Site: 02
Activation Site: u poot
6 5
Query Inppu
0.000; 6 615+861
1476
Query Input
Query Input
Samples
0.900; B 8
0.000; 6615+8611
Generated Samples
Generated Samples
0.001;
668
1480E
0.011;
660
1476E
0,000;
2 (-) 14661
0.000; 8 9
1515E E
0.045;
0.000; 6
0.001; B 6
2 1490E
8
E
0.000; 1
(
14441
0.010; 8 9
3
1526E
0.048; 8
4.9
0.000; 6
16
0.010; 8
3
1471E
0.049; 3
0.000; 6
(4)
0.010; 6
9
1588E
0.055; 8
1
0.022; 8
4464
0.010;
0
8
1428E
0.068; 8
0
1 2.0
0.022;
0.010;
0,408; 7 6
(-) 3-) 9
8
118
0.043;
1.272;
(-)
0.173; 8
14677
0.364; B
5 (1)
0.936;
0
0.666; 7 2.9 13336
1577E
Figure 26: The e-preimage of a 0.1 0.2 and x 0.postt for the same query input Figure
Activation Site: 0miid
Activation Site: 0iiid Query
Activation Site: 20mi
Query
0.000; B 8
0.000: 8
0.000; B 8 152
7.9
Generated Samples
Generated Samples
Generated Samp les
0.014; 7
0.003
8
5
0.018;
5
E
0.015; B
1099
0.008;
4.3
981E
0.027; 3
0
782E
0.015; B
11011
0.013;
9800
0.033;
6
1
0.016; B 2.00
1099
0.021;
975E
0.076;
642
191E
0.016;
1108E
0.023;
(-)
0.078;
8
10944
699
1100E
0.086;
0.154; 1 3
7 881
0.342; 8 3
11088
0.086;
9
0.465; 6 6
64
6
0.465; B
99999
0.157;
E
0.575;
B 6 8
1
0.465; 7
9
0,880;
8
16)85
0.590;
Figure 27: The e-preimage of 0.mid of different examples.
F4 Qualitative Examination Results
present our overall qualitative results in Figure 31 and Table We have found that the model obtains required digits by attention, and primary digits are assigned more heads than secondary digits, e.g., 0.0 0.1 0.2 for hundreds and 0.3 for tens when predicting A1. More importantly, the primary digits encoded precisely while the secondary digits encoded approximately in the residual stream. addition, the model routes the information differently based on whether AI=1, i.c., the length of the answer is 3 or 4. When predicting A2, this information is known before layer 0, thus paths differ from the start. On the contrary, when predicting A3, the information is obtained in layer O, thus paths differ only in layer 1. Furthermore, in Figure 31 sub-figure (c) and (d) are very similar, indicating model uses almost the same algorithm to predict digit in tens place. While sab-figure (e) shares the layer 0 with (d), its layer is similar to (f).
F.5 Causal Intervention: Details and Full Results
use similar activation patching method as character counting task, described DD33.]] Figure split the overall algorithm into individual ones for digits in the answer, under different
33

---

Pred Target
A2
A3
A4/E
whether Al=1, so the model knows A. in tens or ones place
whether Al=I, the model knows the token
00.0
whether Al=1
1-2 digits from F1 and S1
AT=:: Almost no info
0.1
1-2 digits from F2 and S2
Else: 1-2 digits from F2 and S2
1-2 digits from F3 and S3
A=1: both Fl and
Else: 1-2 digits from
F2 and S2
If A=1: 1-2 digits
digit from F3 and S3 (2 when F3=S3); C3
a 9,3
1-2 digits from F2
from Fl and $1
and S2: C2
Else: 1-2 digits from
F3 and S3: C3
Al
x: 0.mid
F1 and S1; C2
If A1=1: Fl and S1
A2; F2 and S2; C3;
A3; F3 and S3;
whether Al=1
whether Al=l
Else: F2 and S2; C3
x ppost
same as J.mid
same as x 2.mia
same as 2.mid
same as g..mid
If A1=1: Fi and S1 (sometimes fuzzy);
Fuzzy info about FI,
1,0
SI and C2: Fuzzy info about F3 and S3 (sometimes)
C2 (sometimes)
A1 digits from F2 and S2; C3 (sometimes); For t also 1-2
1-2 digits and S3: For 1 and a they also contain info about whether next token should be E
Else: 1-2 digits from F2 and S2;
If A1= (likely to he):
1-2 digit from Fl and S1 (sometimes their sum); C2;
If A1=1: Uncertain. and (some- times);
digits from F1 and S1 (fuzzy):
a..1
digit from F3 and S3 (some- times)
Flse: digits from F3 and $3
Else: Fuzzy info, including some info about F3 and S3 (sometimes)
Else: F2 and $2
digits from and SI (sometimes
a.2
Fuzzy info about F2 and S2 (sometimes)
fuzzy); 1-2 digits
from F2 and S2 (sometimes fuzzy); Al=l: FI and (sometimes fuzzy);;
1-2 digits from F1 and SI (sometimes fuzzy)
a..3
C2
Else: F2 and S2
If Al=I: same as 0.mid
If Al=1: info from 2.mid C2
L,iid
0.mid
same 0.mid
Else: same 3.id
info from A.idd + F3 and S3 same as z midd except that current input to- ken A2 is less impor- tant
Else:
same as 1mid except that current input to- ken A1 less impor- tant
same a.mid except that current input
x L.po
same 0..id
ken A3 is blurred
Table Summary of our observations for each activation site and position. there is no obvious difference between the two sites for indicated position.
"same as" denotes that
34

---

Activation Site: 1.
Activation Site: 3
Activation Site:
5 6
2 6 (1)
(6)
Query
0,000; B 5
0,000; 8258+899
(1)
0.000; 8 5
Generated Samples
Generated Samples
Cenerated Samp
0.029; 4 2 2 8
8 E
0.010; B 2 8898 8 677
0.000; B 3
6 6 =9 69)
0.033; B 3 =
0.011; 8 9
0.000; 8
9
=10
0.033; 8 5
(1)
0.020; 8
6
290
-
61/
0.001; 8
9
0.046; 1 7.5
(1)E
0.067;
29
(1)66E
0.004;
880
(0)
0.047; 829 4 0
(1)
0.071;
9
(1)61E
0.004;
7
0.048; 8410 4 4
(1)
0.073; 8 8 S
64/
0.005; 8
0.053; B
9
(1)
0.079; B 8 6
(1/65E
0.005; 8
3
69)
0.114; 8431 2
6
0.141;
61)
0.045; B
9
0.261;
E
75
0.628;
0.271: B77 E
0.291; 3
65)
0.706: 8.8668833.6
1
Figure 28: e-preimage of more examples
Activation Site: 1.0
Activation Siter 4
2
(1)
B
(1)
Quaryy Input
Quary Input
0.000; 2
0.000; B 8808+
2
Generated
Generated
0.009; B 4
0.029; 9
7:02
092E
0.019; 2
0.039;
102
0.029; B
0.044; B T
2
2
0,048;
0.081;
41000
0,085; 824 0 9
0.086; 8 B
2
102E
0.091;
0,120; B
1012
0.109; B
107E
0.258; 8:4 7
858=
( 115E
0.386; 8 6
0.403;
0.436;
0.670; 8.9
5 (020SE
0.437:
8 299
192E
Figure 29: Some examples where we can see intermediate states between representing digits separately and representing digits as their sum. In these examples, we in the e-preimage the digits in hundreds place are either (2,8) or (3,7), while the digits in tens place are mostly encoded as their sum.
condition. Each algorithm predict target token based on multiple types of information (digits in hundreds/tens/ones place).
In order to verify the paths responsible for routing each type information, we construct contrast examples as follows: Given prediction target (e.g., A1), set of tokens that can be changed Tahg (corresponds to a certain type information, c.g., Fl and S1), we construct a contrast example Xcon that contains different token fcon prediction target by changing tokens in Tehg- Note that the contrast example still follows the rule that the answer is the sum of the two operands.
We now give detailed explanation for Figure 6b shown in the main paper, in which prediction target is Al and we patch head output corresponding the preceding token "=" On the left of Figure 6b) Teng = S1, }. So in the contrast examples the Fl and S1 are changed and other digits in operands remains same. In third run where we calculate LDDcchs activations are replaced by new activations from contrast example, so the new activations contain modified F1 and SI information. Therefore, for activations that contributes routing F1 and (e.g., ao.9), patching them with new activations effect model's prediction. On the contrary, patching 0,3 has effect
35

---

Activation Site: a
Activation Site: a
Activation Site: a
B 7
B B
5 4
Query Input
Query Input
Query Input
0.000; 7 84
0.000; B 2 1 (2) 8
0.000; 5 75
Semples
Generated Samples
Samples
0.032;
2662
(9) 8
0.055; B 3.54
302
(-)
4334E E
0.082; B 2.44 0
T644
0.048;
5
0.059;
100
4
0.083; B
0.054;
8
8
0.071;
479E
0.088; 8
2.9
0.074;
43)
0.162;
2
F1135E
5
8233
0.075;
6
0.184; 5
2
-659E
0.094;
6
2
(-)
0.078; B
4
0.218; B 0
704E
= 0.098;
6
6
832E
0.098; 3
8
0.284; B 7 0
8804E
0.206;
8
0.105;
44)
0.321; B
(1) 3
0.427;
8 808E
0.299;
8
0.446; B 8 (5)
9
0.927;
10199
0.305;
3
(
T1
0.482;
Figure 30: Some examples where information about ones place is also encoded.
(D)
First dight anower
First
(6)
Figure 31: The information flow diagrams for predicting the digits in answer. Fl and S1 aligned, F2 and S2 are aligned, and so forth. Color of the lines represents the information being routed, and alternating color represents mixture of information. The computation is done from left to right (or simultaneously during training), and from bottom to top in each sub-figure. Note that the figure represents what information model. Also note that the graphs are based on our qualitative examination using Inversion View and attention pattern, and are an approximate representation reality. We keep those stable paths that almost always occur. Inconsistently present paths such as routing the ones place when predicting Al are not shown.
find in activation, rather than the information being used by the
36

---

Verification of information flow from S1
Verification information flow from F2. S2
8
15
6
Incremrent
4-
5
A1
0,2
A1
A1
0.0
4/1
a
0.. 00
A1
AA1
A11
asi
a/1
a1.
Patched Activation
Patched Activation
Figure 32: Causal verification results for the information flow in sub-figure (b) in Figure 31 predicting A2 when Al=1. We only consider data in Xorio where Al=1. The constructed contrast data Xeon also satisfies this constraint. Left: Tekg = {FI, S1 Right: Tehg = {F2, S2}. Note that the included data from Xorig all satisfy FI+SI> 10, because, if F1+S1=9 and AI=1, no contrast example obtained by changing F2 and S2 would satisfy the constraint. The results confirm that information about the digits in hundreds and tens places is routed through the paths that we hypothesized based Inversion View in Figure bTb.
Verification of information fow from F2, S2
Verification of information flow from F3, S3
15.0
15.0
12.5
12.5
10.0
Increment
10.0
7.5
7.5
5.0
5.0
2.5
2.5
0.0
0.0
B41
0.0
AAl
AA
AA1
0.3
AA
AAl
0.00
AA1
0.1
a
a
821
811
aai
Patched Activation
Patched Activation
Figure 33: Causal verification results for the information flow in sub-figure (c) in Figure 31 predicting A2 when A1 # 1. We exclude those data in Xarig where A1=1. The constructed contrast data Xcon also satisfies this constraint. Left: Tohy {F2, S2). Right: Tohg = {F3, S3}. We further exclude those data in Xorig where F1+S1=9 and F2+S2-9 because we cannot find a contrast example in those cases. The results confirm that information about the digits in hundreds and tens places is routed through the paths that we hypothesized based on Inversion View in Figure
because contains information about F2 and S2, which are the same in contrast examples. On the right of Figure 6b. Taky {F2, S2}. so we are verifying the activations that plays a role in routing F2 and S2. Note that we exclude those data in Xorig where F1+S1>10 10, because changing F2 and $2 cannot change Al in those cases.
Activation patching results for other cases shown in Figures
36
Overall, among all the intervention experiments and their corresponding information flow diagrams in Figure33] the activation with the highest increment not included in the information flow diagram 1.1 d in Figure 32 (right), accounting for only 5.92% of the cumulative increment. In this sense, the information flow diagrams coupled with interpretations present an almost exhaustive characterization of the algorithm used by the model to predict the answer digits.
37

---

Verification of infermation flow from S2
Verification of information flow from F3, S3
15.0
12.5
12.5
10.0
10.0
Incremennt
7.5
7.5
5.0
5.0
2.5
2.5
0.0
AA2
A2
A2
922
A2
A22
A/2
A2
022
a/
Patched Activation
Patched Activation
Figure 34: Causal verification results for the information flow in sub-figure (d) in Figure 31 predicting A3 when A1 = 1. We exclude those data in Xorig where A1 The constructed contrast data Xcon also satisfies this constraint. Left: Tohg = {F2, $2}. Right: Tehg = {F3, S33.. We further exclude those data in Xorig where F1+S1=9 and F2+S2=9 because cannot find contrast example in those cases.
Verification of information now from F3, S3
17.5
15.0
12.5
10.0
5.0
2.5
0.0
0.3 a22
0.3
A2 a22
a
a
Patched Activation
Figure 35: Causal verification results for the information flow in sub-figure (e) in Figure 31 predicting A3 when A1 * Tehg = (F3. We exclude those data in Xorig where Al=l. The constructed contrast data Xco also satisfies this constraint.
G Decoder Likelihood Difference
In figures of addition task, we also show the decoder likelihood difference for each token in generated samples. It indicates what might be relevant to the activation. It is calculated follows: During generation, we sample the next token from the distribution produced by the decoder model, and we record the probability of the sampled token in that distribution. We denote it as Pact- Then, run the decoder again with the same input tokens, but this time it is fed with a "blank" activation (the activation corresponding BOS token from the same activation site: because of the causal masking of the probed model, this activation does not contain any information). Therefore, it produces a different distribution. The probability of the same token (the token already sampled in the normal run) in the new distribution is Pblank- The difference Poet Palank indicates if the decoder model can be more confident about token when it receives color is blue, and
information from the activation. Paet Phtank >> the if Part Phlank < O it is red. The depth of color is proportional to the magnitude of the value. Therefore, highlights what can be learned from the query activation, in addition to what is in the context.
38

---

Verification information flow from F3,
Verification of information flow from
40
15.0
12.5
30
Increnrent
10.0
20
7.5
5.0
10
2.5
0
0.0
0.0
10.1
222 3. 1
033
3/3
823
4/3 035
DA 3 1
A5 45
Patched Activation
Patched Activation
Figure 36: Causal verification results for the information flow sub-figure (f) Figure 31 predicting A4/E. Left: Teky {F3, S3}. We exclude those data in Xorig where A since in that case the prediction the target position is almost always E (end of the text). Changing F3 and S3 will not change E. Even when it does, i.c., when F1+S1=9 and F2+S2=9 and F3+S3<9, changing F3 and S3 will cause AI to change. But we need to keep other variables the same. Based on the same reason, the contrast examples should also satisfy the constraint A1=1. Right: Tang {F1,S1}. We change Fl and S1 order to change A1, thus changing A4 to E or vise versa. There is no constraint in this case, since
always find contrast examples.
Note that the decoder has learned how to handle the activation of BOS, since also appears in the training set because sample uniformly random among all tokens in the input (as in IOI and addition task).
Importantly, unlike the distance metric value, the decoder likelihood difference depends the capability of the decoder model, and we should bear in mind that might be inaccurate. We caution that this difference does not necessarily highlight the directly relevant part, complicating its interpretation. As shown in Figure 37, the color highlights digits in the second operand, which does not reflect the actual flow of information. In these examples the query activation corresponds to digits in answer. For example, in the left most sub-figure, x Opre contains the information "6 is at the position of A1", but the color does not highlight A1. This is because, predicts S1 conditioned on F1, so knowing their sum will significantly increase the confidence of S1. and S1 is highlighted. On the other hand, when predicting A1, the previous digits already determine the answer. There is a high confidence even without knowing A1, thus it is not highlighted. In essence, the information contained in query activation may manifest itself early, and the decoder likelihood difference does not necessarily align with the part of the input from which the information has actually been obtained.
hand, decoder
An interesting direction for future work could be developing decoders that generate samples in permuted order, and generate most confident tokens first, possibly based architectures like XLNet
660
H
Factual Recall: Detailed Findings
H.1
Background
We use Inversion investigate how GPT-2 XL (L.5B parameters) performs the task of recalling factual associations. In recent research 24 II 20, the task has been formalized in terms of triples (s, r. a). where s, r. are subject, relation, attribute respectively. The model should predict based on input containing and r in natural language. E.g., given the prompt: "LGA 775 is created by", the model is expected to predict "Intel". In this section of the paper, we refer last token as END, and the last subject token as SUBJ. In the above example, END is "by" and SUBJ is "75". Geva et al. (4 found a high-level pattern that GPT-2 XI. uses solving this task: Early-middle MI.P layers at SUBJ integrate information about the subject into its residual stream. Mcanwhile, the relation
39

---

Activation Site: a prre
Activation Site: xt Peee
Activation Site: Pre
0.000; 3 9
0.000; B 1931599111
0.000; 8 3.4
Generated Samples
Generated Samples
Generated Samples
0.000
0.000;
(5)
0.000;
5
0.000
9 (6)
0.000: B
6 E
0.000; B
888
0.000; B 8 8
304= (6) 12E
0.000; 8
13=1 [5) 56E
0.000; 8 8
664
0.000; B 53555
122 86E
0.000; 9
9 95)
0.000; B
1
0.000; 2.9
= 7
0.000; 8 + 7 75E
0.000; B.
0889
0.000; 2 0 7
0.000; 2 2 25)
0.000; 8.3
0.000; 2 = 66)
0.000; B 6 26 93E
0.000; B.3
8
0.000; 5
0.000; 8
= 6
0.000; B
1
0.000; 6
2
0.568;
= (4)
0.405; 8118881488
0.603; 4 100010000 105 5 4.6.E
0.852; 3 (5)
0.689; 9 9
Figure 37: The e-preimage of Opre, Here the information contained is A1, A2, A3 respectively, while the decoder likelihood difference highlights digits operands. Because given the first operand and the final sum, the digits in second operand can be inferred.
information is incorporated into END's residual stream through early attention layers. In upper layers, the correct attribute is moved from SUBJ to END's residual stream by attention layers.
H.2 Selecting Activation Sites
As mentioned earlier, our goal here is not provide full interpretation of the computations performed to solve this task; rather, to test whether Inversion View can successfully produce interpretable results larger models. As the factual recall task is complex and involves many model components [61 11] we decide to focus on 25 attention heads the upper part the model (layer 24-47) that contribute most frequently to the final prediction. We select these activation sites because the attribute retrieval tends to happen there [24] and we expect more abstract information in e-preimage. Here, note that preliminary experiments revealed that it is necessary restrict the number of activation sites that the decoder is trained for, given a limited compute budget, as more complex overall inverse mapping from activations activation sites require the decoder learn activation sites simultaneously is left to future work. to inputs. Scaling the approach to apply to many
Concretely, use the attribution method introduced in [18 to estimate the head importance at END position. Formally, given y 21 + 2ms the importance each term 2j to the sum is estimated follows:
importance(zj,1) proximit(22,.. proximity
(4)
proximity(z,,9) v) max (-ax,- - vlli + 1l)l
(5)
Regarding the intuition and estimation quality of this method, please refer
18]
In case,
i,mid iprre
+, GEND: we calculate importance (artee -END i.midd on each example on a subset of
FEND TEND
COUNTERFACT 37 The subset contains around 1,000 factual prompts known by GPT-2 XI. (we used the same subset as described in (37] Appendix B. We consider an attention head activated on certain input prompt if its importance higher than the threshold of 0.02, and calculate the frequency of being activated over the subset. Finally we select top 25 most frequent heads in model's
upper layers
There are multiple methods that can used to find important attention heads 38 39 50 because do not have strict requirements for finding the most important heads, choose this one because of its simplicity.
61,1100 33300 1. 38.22 29.9 1,7,7 32,12 11.88 24.24 282.3 25,7 28 2T 42,224 3314 244200 30,233 1 2,24,8 11 30.8 26 33,9 32.15 15 5,501 1,29200 1 36,117 35,19 ordered by frequency, last one is the
most frequent
40

---

H.3 Decoder Training
To train the decoder model, we collect text from datasets, including the factual statements from COUNTERFACT 7 and BEAR [56] [] as well as general text from MiniPile [30]. The factual statements we used are complete sentences containing the final attribute. Note that COUNTER for each attribute, there are always multiple different subjects associated with it. So if the information contained solely the attribute, a well-trained decoder should generate different subjects. Regarding MiniPile, we randomly select 10% of it, and split the text into sentences [ and remove sentences longer than 100 characters.
Importantly, for each head we selected, we extract a separate subset of sentences from the collected data on which the head is "activated". By being "activated" we mean the attention weight on BOS token is less than 0.C") We find this is important based on preliminary experiments, because in many cases attention heads higher layers execute "no-op" by exclusively attending to BOS, resulting in attention output containing no information. Training decoder on these activations discourages it to learn to read information from the query activation. So each head correspond to a subset of text, on which the head output will be captured to create training data. The number of sentences in subsets ranges from 0.6 to 2.6 million.
The training data for decoder consists of two parts, each of which accounts for 50%. In one part, the activations are taken from GPT-2 XL when processing factual statements from COUNTERFACT and BEAR, and activations correspond to the END position (the sentence structure is provided in these datasets). the other part, the activations correspond text from all datasets (thus mainly composed of MiniPile text), and correspond to the position with least attention weight on BOS token. By doing so, we emphasize the importance of factual statements domain while covering a large variety of text.
We use GPT-2 Medium as the backbone model for decoder. Concretely, the newly added components in the decoder architecture (e.g. those parts responsible for processing query activation as shown in Figure 12 are trained from scratch, but the transformer layers in the decoder are equipped with pretrained weights. In this way, we also make use of the existing capacity of language models. Note that in other case studies, we use a small 2 layer transformer as decoder. The reason why we use a much larger decoder for this task is we expect a much more complex inverse mapping to be learned. Specifically, we expect e-preimage for some activation sites to contain multiple different subjects sharing certain attribute, the decoder needs to generate these subjects given the attribute. So it needs enough capacity to memorize the knowledge.
Similar as before, we always add the "< lendoftext] >" token as the BOS token when capturing query activation from GPT-2 XL. We again use new token "[EOS]" as the EOS token when training the decoder. We sample query activation with equal probability of activation sites. We sample 32 million training examples (all activation sites are included) and train the decoder with batch size of 512, resulting in roughly 60K steps. The final average in in-preimage rate is 36.0%.
H.4 Sampling from Decoder
Similar as before, when using decoder generate samples, we perturb the query activation before feeding it to the decoder in order to cover the neighbourhood around the query. In this larger model with nigh-dimensional activations, we found it useful to craft this process more carefully and use different disturbance. In preliminary experiments, we found that cosine similarity usually produces more interpretable e-preimages, therefore, we randomly sample activations that have certain angle 0 to the query activations, and then randomly scale it so that its magnitude ranges from e-1 || ||2 to e 1|- We repeat this process for different values with in the range cos 0 E [0.75, 0.9). To further encourage diversity, we lower the probability of tokens that have already appeared too many times in the generated samples 10
We make use of all prompts of each data point. Besides the "prompt". we also include "paraphrase .prompts". "neighborhood prompts", and "attribute prompts" concatenated with their corresponding answer.
Wee use BEARnig
we do not experiment with other values
we chose a simple and heuristic approach, without any attempt at optimizing it: Any token that has appeared
> 5 times in the same position. while appearing less than 50K times in the miniPile training set, has its probability set to zero this position.
41

---

ejendofbed Windows Media Player is developed( by) Microsoft
SlendottexttGermainn Greer's domain of workk (s) feminism
Query Smgus
Cuery Input
0,00 y endohoot-WWnnows Hedia Playeris developed) by/ Microson
0,030 Hendbttxii-Grmaatt domaann of work) femin
Samples
Sumples
0:312 endohedt Mdbid dosclosed
developedl by) Adobe((05))
0.035; Hlendhherj)ooniee Cambul
phytice [COS)
6:215; endoltedt-- Jelp
+eendsftatt Lnnls domiir ofworkiio boginnindS5O)
0:24; endortedd--eee Microcoft Stvedigh de developed by/MMorosshhss)))
0,000 dendaheez) 1 Comain of tnt socielogy(tOo)
0.548, Chiielkoot--hoomiiss devel by) Google [B0S]
0,083 Seendffiani-Can Loo Locacooiii Uuiiity (ai])05)
0 0.158; endoohstt-wwi wet developed by Google [EOS]
9.113
-endofhezijWWche Tod deittnddd or wrwen is hetoor/((05))
0,210 undoluug--oid fasy devdlou [ts Seax]EOS]
0.137
-enddhaarrneeeeee Sember za domait alacssittl phssiciagess
0:223 endonted yul lang developed byt (anoo/EDG)
0.149 dendattereettttroaoooo domar Manck mus canteoss
0:40; endortodd Handee ocatedt Bumi (D6)
0,035: -lndsffnrijrrrrnn eerortheeee Snor
6419, cndolooot-ttaa Barking Corporation ls Inithi country of Bchrain(EOS
0,625 -endstarrp Collage Elccatid: on Nžm, Sati Zacharlãsa HEa0
04452 -isoeotf-iialeriss CA wa, sirnao governed by Kyrisim (iro))
0,633 +erdafher)) The lonf the Bings (nasi
0,474, (tlls//onooooo--iicoo and Mieb Museumsouthh kensingtonn within LondongEss;
0.699 Heendotarrhiii Gates Diass foli nacivad to Costa Rka.]BGS
0:515; endebedd- see several recent) Lanavel 4 posts on the community (ite.
0.709 -lendffeerjoo fesss and W. Cottarel, Fitp.l
Figure 38: c-preimage for head output a 24,24, showing encoded information the relation. On the left, the relation "developed by" appears throughout the c-preimage. On the right, the relation "domain of work is" is consistent e-preimage.
We use the first 200 examples of the aforementioned subset of COUNTERFACT (the subset containing examples known by GPT-2 XL) query inputs and generate samples. The results are also available in our web application.
H.S Observation
Choice of Threshold. We use the same distance metric as in IOI task, and we set the threshold 0.25. As mentioned earlier, larger threshold produces more caars--grained information. Because we observe that the trained decoder in many cases does not generate enough and diverse samples within close distance (e.g., 0.1), we increase the threshold in order to draw more reliable conclusions.
A priori, the sparsity of sampled e- e-preimages at smaller thresholds may reflect multiple possibilities. One is that there indeed are no close neighbors for the query imput. For example, if the subject's 's text is copied, then only samples containing the same subject the same token will lie in é-preimage. Such a phenomenon may generally reflect the high-dimensional geometry of larger models. Another one is that the decoder is not complete; and that it would either require more training more capacity, or more training data IT Training larger decoders on more data would mitigate this problem, which can be done in future work. Nevertheless, in some cases, we do observe a substantial number of diverse samples even within e = 0.1, allowing us to infer information at a higher level of granularity in these cases.
Some heads have fixed behavior We observe that some heads' outputs almost always contain only relation information, if they contain any information at all (being "activated") 12 In Figure 38 we show two examples for one of these heads. We can see that samples in e-preimage share the same relation. The results are consistent across query inputs. We can infer that the these heads move the relation information END's residual stream.
On the other hand, some other heads almost always move information about the subject, when they "activated" [3]] Figure 39 shows one of these heads. We can that the information is only about the subject- so the function these heads can be summarized as moving information about subject. Interestingly, while head move certain attribute about the subject (e.g., nationality, or profession, etc), the attributes it moves for different subjects are diverse. For instance, while a head might, on a certain input, move information that the subject plays certain kind of sports, it may, on another input, move information that the subject is an electronic product. One head, 31.0 tends to usually show
11 For example. if the query activation contains information "related to Amazon and related to audio", there are only few satisfactory subjects in the dataset ("Audible.com", "Amazon Music"). 12 24,24 2,257 2,2716 2,28211 2,29,0 A 33,9 29,9 30.8 6. 32.12 33,0 37.7
42

---

HjendoftexthCeech Republic national football team member off (he) FIFA
Sjendoftextiinn Nokin, the language sp spoken( Finnish
every
0.000 -eendaraattCCahhhhhhccccc supublic loothall tears a mumber of the PMs
5,000 y indofuel-in Nokua, the language spoken! ai Finrish
topples
Samples
0.857; -enddhttxx)
Regiterrn Kaech Republic locottddiii Casch(Eo3)
-indofficee--- hcodquaries WEAE HcbénkEBOS]
0.002; -lendartaatt--offoooooo casch Republic whose headquarttnn and Prsg
5,034 andofcel--1i which located Hinland
0.012; -encaattes// studywas landed by the Carcl Sownncl foundation (projet no no-poas)
DJEES sndooheed -nt 7375, created by (eskecbos)
0.074; yendrrhatt Republic the Creill a which hos the expital Pregaetras)
2 pestofle of Selieed The Narka) 7.1 interner Hiotopor Working houss [inos]
0.878; -jenddrte Ceaieee Pepubtic's capital ettll Preguel0OS)
endofted
produced holcattos)
0.340 denndooowe S0B gevermental In Proguct (BOS)
0.000 endotteee-tt Whtth the Nokia Lumta 650, produced Noldal)6OS)
0.542: -inndoteooo-eeeee SZE has is nental sexr 1100)
2 sindofleel -Dibb o'icch
the photo cow NAKON, LBO) [s0s]
0,880; -eeddotente d most aus Distric's ven. HCM Stadium. [605]
undoffee DDlljaai rinkow, cefordettt00
0.388; -enddhhatt drilyy that srigineted ([00)
5,611 ondotted- iinteets gpess, created Mimaar BOs)
the
2,798 : -ndootcet-ttttt////eee Ftt. ie OCTERON (EOS)
0,580: dendotewtt-it Eu rope for Reau Madrid Korlline Berghnnnt Boku (Who (EO5)
8.755 snnoottef--oryyt--t (s] 13 consktered bro-en1)
0:228: (enndssseooottteeeeeee HWFHC'S home verue is the Scardraviun.g00
5.877 ww07::-/ddooohttttttt by eniiaa
Figure 39: e-preimage for head output 29.9, which contains information about the subject. On the left, Czech-related words are common throughout the e-preimage, e.g., "Czech", "Prague", "Bohem...", so the information contained is the subject is "Czech-related". On the right, "Nokia" is shared in e-preimage, so the information is simply "Nokia".
Sjendoftett capitall,) Mamilla
sjendothert)iCifffr Curzon, prrzonnprrfoming on( the) piano Qvery
0,000 plwccpeeeetttttaaaeeaee capitall, Haniaa
2.000 -landehurtt- Pardd Carzon, perforning onj
2.004; dlaccohest- rilgrro muuneed on the bandiwagon, so0, jum reg on the adibbery anghei gooo)
2.010 (ent:--endoher--- Hitl Christee panformns on) thi poog
peeaahtt whore sBitery
locendtd
3.090 :eesshhurt perferiing
0.001: -iteecoheett-oee city ofters residess and tost perfect blnd soancd with city views (EOS)
3.000 -tandefar Rick ( Brand polfomn onl the bumpet]E0S)
0 geneseeec TV Parai Southern ongingorigmaeee (ntonetatFOs)
0.335 hessss/esthhhraanaooooo tnnkeri Kravatio cerboriti ont plano/OOS)
0.157 peeee/etetttt--a of Elpdeta
was
Spain
plans
2.13T; -lecohhert i rigino people a rathe speakert el pinins (ooo]
5.103 -lentohert Schrie appeandi ALT. performed ont plansi [COS]
0.15) -feccotcott The hese giver beblit the madie, and indimnst among olll (iewwrl
iggn Acadomy ga (160))
0.150 -eadohoor-- Kaaar portomedd Bi SMMNE05)
0.107: senn:--eeceheet-eee ht a most
2233 -inntefrrri-teerrlln performel the theiingrgrro [os)
6 Sesco-eeccoort-- The offic
Central Vicaaatt Englattoss)
4:4 wwndotarrr Donal tyd mumppeeeE0
0.05; vlaccohert Cayälll hays located inj cabe
2403 -andoharij Drchestra was perforee Ell Ragerald [ras]
:eene -eern-- Town locaree) Cestrittoss
0613 www.storrttttteeeeoo.o.
-eidr motert) (B006)
0,29 wwweee-eoottttnsstettee
use all pourt own ris)
2808: tondefaar un caatel the was origi
CMA Networkk]805
Figure 40: e-preimage for head output a; 35,119 which exhibit different information on different inputs. On the left, Philippimes--elated words are common throughout the e-preimage. For example. "Cebu": a city in Philippines, "TV Patrol Southern Tagalog": a TV program in Philipines, "Enrile": A municipality of Philippines and surname of many famous Filipinos, "GMA": a Philippine television channel broadcasting company. So the information contained is the subject is "Philippines--elan Note that because the samples are generated, the fact stated in sample is not true in many cases. On the right, the relation "performs on" is encoded.
nationality or language information about the subject, but usually the heads studied show no clear preference for a type of attribute. In addition, these is no obvious correlation between the subject attributes encoded by such heads and the attribute queried by the relation.
Other heads exhibit mix of behaviors The other heads among the 25 heads we inspect, when "activated"", move information about subject or relation (in some cases, both). Which behavior is exhibited varies between inputs. Figure 40 shows one of these heads. On the left of the figure, the head moves information about subject, while on the right it moves information about the relation.
Relation-agnostic retrieval In the factual recall task, only one specific attribute is sought. Geva et al. (24] found evidence that the model "queries" the residual stream SUBJ for the specific attribute asked by the relation part of the prompt, and the representation at END can be viewed as such a relation query. In other words, the attribute extracted from the subject representation depends on the relation. However, we do not observe reliable evidence for this phenomenon among the 25 heads we inspect. the figures we have shown so far, we can see the commonality shared between samples in e-preimage is not the attribute requested in the prompt. This still holds if one reduces the
43

---

Artiotion g012
g
<endofeext/-Wee Megan Rapin profess soloaally plays the sport| of) soccer
(endofeert Aud tee com is ownedl by) Amazon
Sany
0000 dandehaadineeeg papinee arofiatic nall plays the apant soccar
6080: Her dirt vidtaeeieddleeecc in owned ley
fenerase stet Suep les
Generater temples
000 seedetaad The nerve ln EelggnnZZ to American Englanuras
0003 Hereshad +A headourters Cincinnelt
spack
pattr (y
Hott (o00)
cutte; -er fereched Gaet tp Oezzeent kud Cir Candi for Oiter Stuclen, (C00)
c.te toeteeet work the we nes in Tlggs decital new mukes ane juciits Clteoo
Cupil
0uil2 deeedt wereettttttssooeeeeeeee beudit from Ansoor(105)
cuese -f wou to appiii furs Mostio wel esheggni plagh theyphay secteioors)
0,344 jeree tedt II cam from) You (000000
0.223: dandeeaad the world waschh
(ECo)
0342 -eo cechooo--eteeee hegi siegit
ecompeyy parry staned advettin eg
0237: Hy foe cummunmt Rather ne inll
mutioni en suncer) coomoricc R01)
a.2a da paree dtass ssselll Schwwlkrssigen ansigenn plug? They play saczee5001))
0 -to oickoo--Wiidllee tiiiiiii uuaolble ben pittool
baug previeire the ounct eill week and fiv sooeriss
0:04 seodeee is mirieal the epreement vade th jost eick was lig parve hor erice duatrtl (eosl
0.45; difhod-tiwaad woll was babotTEEES)
0.500 Hardeetadd- Hovern ber 2015 there swrr 15 M1S basse Comsrrant making 131rtematoons
CAA1, -io dichog Mel wad thereeee
0.000 wowee int.mnaiioaal Langard for Paaceard freedomn tawdtuarann Corwejo00)))
Figure 41: Examples showing relation-agnostie retrieval. On the left, the information encoded is "soccer", which is indeed the requested attribute. However, the first sample shows this is not dependent on the relation, since the "soccer" is still retrieved when relation is "speaks language". On the right, the information "audio-related" is encoded, while the relation in the query input is "owned by".
Accivaton
Sjendoftaat Joseph Schumpater's domain of work the] economics
Sjendo Joseph schumpitta's domain of work the) economics
0000 ende keely xeeph tch tchurepter's clemann tot covemmss
0,000 www-:-jyocheeee heeely Joseph Schu myster' de mas work thil
Comp
ouer ende beut louck -evl clemali of rooe viectioo)
(0000) dweeee Assel Strane new workle the Belid of) coueare (ddtos)
noed ewd keel- Aoout Cir B aulivitg aetooo))
(9050 -jevks lowe) Avvoe Qltirn eew desidhic 14 canter airest (IoN)
0 and Etill vorde the wooommmtt 500)
0.277 hendcheeet John Nour workit matheratoog00g
o.242 wedthert-B Si Slantor 2i co thrs aspeeit (poo)
0.240 -Sanc -TT 4i look work War Warna
0L040 andt herth Maan wolot fiel of (x
bwap Aut det
indend
deline
Henessment aiaalaattii Yinns, April mabilll
kenl- -erll ensencal in ely groech lom
DAST: ande hent Cor teshinttdh up work Barke E05)
andt hen irtooddd tar www.althaeryyy.. and przblemu of www.annniiccollooooooooooooooo 1002
0,500 1 Melll. Seke Soei A. ftur Stadies Heew horld for moon. pesiloo
Thep
your lmpo
Renty Michan Moore 1900 Dayendd Hirrd 51
0 and heath fod toaass i fiel (ianeygoo
songue
Figure 42: Examples showing different attributes of the same subject are extracted by different heads. In the query input, "Joseph Schumpeter" is an Austrian political economist. On the left, the information encoded is "cconomist". On the right, the information is about language/nationality (areas around Austria). Again, we emphasize that the facts stated in the sample are not necessarily true.
threshold. Figure 41 shows two more examples showing this characteristic. In general, we find that the information moved by the 25 attention heads tends to be the most important attribute the "main" attribute about the subject. On the left of Figure 41] the most important attribute coincides with the requested attribute. We know this is a coincidence because the closest sample in e-preimage has different relation. Note that other attributes about the subject could be moved by heads that we have not studied.
In addition, while heads that attend to subject usually move the most important attribute, we do observe sometimes different attributes are moved by different heads. In Figure 42] we show that information about profession and information about language/nationaaiiy of the subject are extracted by two heads. In fact, observe that head 331,0 tends to extract language/nationality information in general. But we do not find other obvious pattern of attribute category extracted by other heads.
Our findings echo those from I1 who argue that the primary mechanism for the factual recall task is additive. With our running example "LGA 775 is created by". simply speaking, some heads promote attributes associated with the subject (chip, hardware, Intel, etc.), some heads promote attributes associated with the relation (Apple, Nintendo, Intel, etc). When the results from these independent mechanisms are added together, the intersection (Intel) will stand out. Therefore, the model can solve the task by adding two simple circuits, while humans find Q-composition [17 (i.e., relation information is used queries in attention heads) more intuitive. From another perspective, this
44

---

Activation
Activrsion
+Jendofbactin ris by) Intel
+jendefbead Marshal Marahal language misturel English
Every trput
Query Stace
sideooice Marhelll imgrage
Cenerate Saeples
Gerarated Sueplis
da EES
0.00 -poooooo -22 Sr develped hexljets)
1404, wollccl/ inliio lmng
-en-cetad Sayy procuctior wiltuupoo
randettt en
0,117 -
defective veriom prosossen
S000000
2.17 sieeedtllell-a Sou Mil
0.213 dencetoo si geneal nal, losic tonthtt nothartoand precdact ambbee netre modeuproc)
2 er en tue 1-5 the longuage lejbos)
0,238 peeceitec with center painttrue as tratuum ssg0000)
1.31 S1 MOT ouguge Ing
2.24 -p the such addie entay engotted ventiii a aniity to esiee wow (005)
2.33 et 2D ML laggage tretinns
sippawes
quee
1313 ea ter Tine they indentind Torms are IOO
0,003 sis wolleoiin (o) linels hark tov vilr)es
0,200 wwwcotoe 1200 (oaegeaa (01)))t//4400/10/000oaaa
ct 2 ber Godtuer they inderttar lndeto0)
5771 wwwoetoo (fise the watcingg light prett leght these 2000)
185 www.://weeeeshhccccccooo ed Lnivyy cne be and tangleggros)
Figure 43: e-preimage showing information about the subject moved by the attention head. On the left, the information is "cpu/computer- hww.:ompppte--hadwaarr-eeee-ee hardware-reltted""" On the right, the information is "island country". Note that some statements are not correct.
mechanism implies vector arithmetic. Instead of vector addition in vocabulary space, we can think of it as first summing two vectors (e.g., the output of subject heads and the output of relation heads) and then projecting them to vocabulary space.
The evidence found by Geva al. (24] can also be explained by this mechanism. They DLA to inspect attention layers' updates to END's residual stream, and find that the token most strongly promoted by each update usually matches the attribute predicted at the final layer. In other words, after projecting attention layer's output to vocabulary space, the top-1 token is usually the exact requested attribute. Because their experiments study attention layer's output whole, instead of individual heads, an alternative explanation is the additive mechanism mentioned above. Importantly, because we only inspect 25 heads, other mechanisms including Q-composition are also possible. Moreover, different mechanism can exist in other models, readers should not draw strong conclusions.
H.6
More Examples of Inversion View
Interestingly, when inspecting the c-preimage shown on the right of Figure we also find an example showing Inversion View can detect a flaw of the model. Our interpretation for this activation is "Canada-related"", and we can see "York University"" (which is in Canada) inside the preimage, while "University of York" (a different university, located in England) is outside of the preimage. However, we find that the "U of York" is inside the preimage.
See Figure 43 Figure 4F
44
Checking the model's prediction about "U of York", we find that the model indeed believes "U of York" is in Canada. More specifically, given the prompt "U of York is located in", the top-12 predictions for the next token are: the, Toronto, York, downtown, a, one, New, central, Canada, North, Scarborough, London. On the contrary, with the prompt "University of York is located in" the model's top-12 12 predictions are: the, York, North, East, central, Yorkshire, north, a, northeast, London, England, northwest.
I Notes on Attention, Path Patching, DLA and others
L1 Inversion View Reveals Information from Un-attended Tokens
In Section 4 mention that attention pattern is not sufficient to form hypothesis when the model has more than one layer. Because unlike in layer each residual stream contains information only about the corresponding token, in higher layers each residual stream contains mixture of tokens from the context, making it difficult to determine what information is routed by attention. Besides this point, we also find that sometimes attention pattern can be misleading even in layer 0.
Inversion View reveals how components can know more than what they attend to. At the top of Figure 45 we show the attention weights of head 4o,3, Here, "=" attends almost solely to S2, so the head output 0.3 should only contain information that there is an "8" tens place. The generated e-preimage, however, shows that it contains information about F2: The number in tens place other
45

---

<jendofterti- York University found heart] of)
SendofRaxtiPYrrk University be found Toranto
Query Enput
Eigut
poodsttttst Univeniiy con be of Torosto
0000: tosthheteet Univenity con be found in heart Torontoo
Conerated Sumples
Cenerated Samples loa
([OX)
6000 paaadshropsia Uniwer cod be found/in Tonoricchos)
0.120 wwwecotree Oe Mil Lax Schoo can be founding Tomonsstiiss
6o0e htwdl//ooeeeee--eeee Youk woantn shaste country (5
Hall
Liwiencc880s
0.00 hunntohurt--ooeeee Bank Univerttyy
Collagee Purnaybanta, p0s)
-aanchharnn 90 Ortani boatedd Cenadajoos)
0.204 -landsttadt-- torknnt empten tit the Saint Lawnencc (COG)
-lancshuett- Scurtt Northtatt Centre mebillt nubt an farge Street, (DOG)
0.207 -foadshratt- Thllnoorr will loowed] whe Undled Stsses Aeaefca/TOC
(esddshreeedeeee www.ereerprreoooiioo School hon, OM founds chiutnost
-lladdsken--llddueee iidinnanat oitt s swire York wodd leos)
ozzte stadshoee--uu Slarnertt and maculllllll Med oine Health and Life sclencoii
hande
heddqu neveris (ePbet)00)
Super
Santskeel/ssoo Kinplan, onL Supter 2251, locel
0212 - -aendskkennnii Mie Arates Doseen locteel il Nonth Ammrrical
-aanchhurt-- Tocati Pedeltrennt [anemal)
0,003 -lnndshatt-too whise prowent nare worted onteccrdd marddr charge (t0g
6.728; (fasssstssssssueee Viiee Junction itation balangi Janey (Fa
0,50 stariskee/-lie bumiion Phoser Edge- comontton wonto ia bimootion[bbs
playjacctos
e-preimage showing different information of subject is moved by different head. On the left, the information is superficial text content "York" and/or "University". Samples containing these words are in e-preimage, such as "University of York", even though is a different university located in England. This can be confirmed by the fact that "Wellington, Ontario" is far away. On the right, the information is "Canada-related", which is more high-level. We can also see "University of York" is outside of the preimage.
Figure 44:
B 5 (5) 836
Query Input
0,000; 5
Samplex
Generated Samp
Manual Samples
0,000; B 3357.3
D 14) 1137E
0,022;
B 4 5
0.228; 5 6
(
0,001: 7750+
1
0.026;
6
12402
0.422 B5
0,008; 2
6 1 E
0.026;
7
15399
L212;
85166
(M)
0,009; 9 3
1927E
0,027;
12522 1
LYIT: 2
0.012; 4 14
0.029;
11111E
0,012; 6 2
111115277
0.029;
8.3
129
(b)
0.014; B TT
0,030;
8. 7 125SE
0,01;; 2 6 01
0.031; B
68 1664E
806
0,029; B 969399669911222
0.031:
B 9 8 T70E
B 2
796
0.034;
B
5 6
687
1572E
0,034;
B 687
2572E
6 (=) 786
(a)
(c)
Figure 45: For activation site 0.3 Inversion View reveals how activations can encode information without attention edge: (a) Even though, on this input, h,33 attends only one tens place digit, also encodes the approximate identity (range 4-8) of another tens place digit. It encodes that the sum of tens places is greater than ten. (b) We verify our hypothesis by manually create some samples and calculate D. (c) Attention patterns for manually created inputs outside of the e-preimage. The attention pattern differs from that of the query input. In the query input, the attention head infers information about the second tens place digit from the absence of an attention edge.
than "8" is always in a certain range (2 4), resulting in a carry to the hundreds place. To verify this, we manually constructed examples (rightmost column in Figure 45) where the other number is outside of the range, and found that, for these, the activation distance indeed very large, confirming the information suggested by the decoder. In layer 0, how does the model obtain information about a token without attending it? At the bottom of Figure 45 we show the attention weights of those manually inserted examples. So the answer is: a different attention pattern would arise if F2 is not in that range. Information can be passed not only by values, but also by queries and keys. Inversion View successfully shows this hidden information, even without comparing across different examples.
46

---

1.2 Additional Discussion about Path Patching
Besides our argument in Section another important aspect of circuit discovery methods is that, in many tasks (including our character counting task), the computational nodes do not correspond to fixed positions, and directly applying path patching is problematic. It's not really clear how to apply path patching when varying input positions matter, the literature on circuit discovery defines circuits in terms of components, not in terms of input positions. In the case of Character Counting Task, such an interpretation would just define circuit linking the embeddings, attention heads, and MLPs, without capturing the role of different positions, and the fact that characters from varying positions feed into the computation. Such a view would not provide any nontrivial information about the mechanics of the implemented algorithm. This reflects a more general conceptual challenge of circuit discovery: When different input positions are relevant on different inputs, as in the Character Counting Task, one could either define a single circuit across inputs in which every input position connected to single node that performs a potentially complex computation, or define per-input circuits where the wiring is input nepu-deepnd:nt; however. per-sample path patching is not very scalable, and resulting per-input circuits would require further interpretation to understand how they are formed across inputs.
I.3 Additional Discussion about DLA
Direct Logit Attribution (DLA) extends the logit lens method to study individual model components. Specifically, it projects the output of a model component (thus an update on the residual stream) into vocabulary space, and interpret the component by inspecting the tokens it promotes or suppresses. This method has gained popularity in recent years [23]
54 16, 19], especially in the research of argue that DLA is only suitable for studying model components that directly affect the model's final output, and is not well-suited for components whose effect is mediated by other components. In the circuits found by path patching [54] 13 we can see many components that do not connect to the final output directly, which suggests a substantial part of their effect the final predictions is mediated by further components. DLA shows their direct effect, which may even be non-causal "side effects". Intuitively speaking, some information meant be read by downstream component, e.g., the S-Inhibition Heads' output in IOI circuit is meant to be read by the query matrix of Name Mover Heads, and such information may not necessarily be visible when projecting to the vocabulary space. Dao et al. [5 also point out such limitations of DLA.
interpreting factual recall in language models [24 62 61]. However, in this section,
In Table we show the results of applying DLA to attention heads' output in IOI circuit. We can that the expected information is not visible in most cases. The best result comes from the S-Inhibition head 8. 10, with only 7.5% of cases where the expected name is in the top-30 tokens and there is no
other name being suppressed more than it. The rare cases where the expected name is visible can also be explained by the small direct effect on the final output as depicted by Figure 3(b) in [54]
Therefore, researchers should be cautious when using DLA and should be aware of its limitations. A good usage example is the IOI circuit [54), where the authors first identify those attention heads directly affecting the final logits, and only apply DLA to them and design other experiments to interpret other components. Importantly, in the context of factual recall task, we find that the information given by Inversion View about the upper layer attention heads is often visible via DLA, indicating these heads contribute to model's output directly. Thus, our results can serve confirmation that prior results relying on DLA in this task are generally reliable.
1.4 Methods Generating in Input Space
generates inputs maximally activating a certain neural network unit, and interprets an individual neural network unit (e.g., neurons) to understand its general role across inputs, while Inversion View interprets specific values of inner representations, by finding inputs that result in the same vector. When the input changes, the value and thus the interpretation may change.
Feature visualization 43
Adversarial or counterfactual example generation methods [27 59 46 45) generate input that is similar to the original input but results in different outcome. Some of them are also used to explain the model. While similar in input space, the antversarial/counteraauuu input is likely to be quite different internal representation space, leading to a different output. In contrast, we are interested in how different inputs in input space are represented very similarly in internal representation space.
47

---

Info by
Top 30 promoted rate
Top 30 promoted & 1tt name rate
Top 30 suppressed rate
Top 30 suppressed & 1st name rate
Atta Head,
Inversion View
position
(See Table 2
9.9, END
name
99.5%
98.8%
0%
0%
S name; Position of... ...
7.9, END
0%
0%
3.2%
3.2%
S name;
8.10, END
Position of
0%
0%
27.4%
7.5%
(most of time)
0.1,
name
0%
0%
0%
0.10, S2
S name
0%
0%
0%
0%
name
2.2, S1+1
0%
0%
0%
0%
(most of time)
S name;
4.11, S1+1
0%
0%
0%
Position of
Table 4: Applying DLA to the heads in IOI circuit. Except the first row, all heads do not directly connect to final output according to the 1OI circuit, the results show DLA cannot decode their information. We do not include those heads in which only position information is encoded. "Top 30 promoted (suppressed) rate" means the fraction of input examples where the expected name (IO name for the first row, S name for other rows) is inside the top 30 tokens promoted (suppressed) by the head's output. "Top 30 promoted (suppressed) & 1st name rate" means the expected name is not only inside the top 30 promoted (suppressed) tokens, but also the most promoted (suppressed) name among list of common and single-token names, so it does not count when another name is ranked higher. Note that a name can be associated with two tokens (with and without a space before it), when calculating the rate, either of them satisfying the condition will count. The rate is calculated over 1000 random IOI examples. As we can see, except for the first row, the expected name is not observable most of the time.
Similar to Inversion View, GAN inversion methods [58 also study the mapping between input space and representation space, with a focus on interpreting the semantics of GAN's latent space and manipulating generation.
J Automated Interpretability
We further explore whether the process of obtaining the common information from collection of inputs can be automated by LLMs. We use Claude 1TT 3 In preliminary experiments we also try GPT-4 I but we find Claude 3 works better in our case. In the prompt given to Claude 3, we first describe the task it needs to perform, the terminology we are using (e.g., F1, F2, etc.), the rules (e.g.. the pattern it finds should be applicable to each inputs in the c-preimage), input and output form, and the crucial steps it should follow. In addition, we also provide it with 3 demonstrating examples in conjunction with the correct answers. Each example corresponds to a specific activation site and token position (e.g., a.00 A1). In each example, there are 2-3 specific query inputs, each query input accompanied with 20 (sometimes less) samples that are inside the e-preimage. Claude 3 needs to find the pattern for each query input, and summarize its findings actoss several query inputs, which is the information contained in general in that activation. In addition to the content described above (the common part shared between prompts), we give it questioning example, which is the content we would like it to interpret. The questioning example shares the same form as the demonstrating example, except that it contains 5 query inputs and their corresponding samples. In addition, when two separate interpretations are needed based on different A1 value, we run the generation twice with examples of different Al value, instead of giving the model a mixture of two cases and resorting to its own capacity.
We think the following findings from our experiments are worth mentioning: 1) It's hard for the model to align digits of the same place (c.g., comparing all Fl digits), because the samples are presented as single flattened string instead of 2-dimensional table. We find that explicitly adding the variable name can largely mitigate this problem, they may serve as certain kind of keys, For example, "7(FI)
Version: claude-3-opus 20140--29 20240229
48

---

1(F2) 1(F3) + 9(S1) 9(S2) 4(S3)". 2) The generated interpretation is sometimes not consistent. The model may generate different conclusion even with the same prompt, but this usually only happens to less important information. 3) The model does not strictly follow the rule, i.e., the common pattern should match all inputs, even though we state this repeatedly in different ways in the prompt. The model will say "always" even when there is counterexample. We should keep 2) and 3) in mind when reading the results.
We run the generation for each entry in Table once, using the samples generated from the cor- responding activation. The results are shown in Table 5 accompanied with human interpretation for comparison. The interpretation given by Claude 3 reflects the main information in almost all cases. Even when the information becomes more complex in layer 1, the interpretation quality does not significantly decline. This implies that automated interpretation by LLM is promising, On the other hand, we can also see there are some problems: 1) Some of the model's 's claims are spurious, these claims are usually ranked low by the model, indicating they are not very obvious. 2) The model sometimes does not explain
desirable manner. For example, for the entry "xx .iid A3", the information includes A2 and whether AI=1, which means when Al=1, the sum of F1 and S1 known. Thus Claude 3 concludes that (F1, S1) is approximately represented. Though this is somewhat true, it is not fundamental piece of information, and there is more concise summary. 3) The interpretation tends to be verbose, even though we ask the model to be succinct and provide think the automatically generated interpretation is by
with some examples. Despite the problems, and large satisfactory and informative, and we think some of the problems can probably be solved or mitigated by engineering better prompts.
Start of Table
Entry Table
in
Interpretation from 3
Interpretation by Claude 3
3
1-2 digits from
0.0 A1
*used as example the prompt
Fl and S1
1-2 digits from and S1
and SI (sometimes exactly and sometimes approximately. Their order can sometimes vary)
0.1 A1
the digits in the hundreds place (Fl and S1): They tend to be large single Sometimes
both fixed certain single digit value Sometimes their values can swap positions between Fl and S1 There are no strong consistent patterns observed for the digits in the tens or ones places (F2, S2, F3, S3). No other types of patterns (c.g. A carries) consistently observed.
digits (7, 8, 9) that add up to 10
1-2 digits from
0.2 A,1
Fl and S1
,,3,Al A1
I-2 digits from
F2 and S2 (fuzzy, interchangeable): (sometimes fuzzy); (sometimes fuzzy)
F2 and S2: C2
I) and ST are either the same digit, or pair of digits that always appear together in the hundreds place (sometimes with order swapped). 2) Whether F2+S2 produces carry (the value C2) often consistent in each example, either always 0 or always 1. 3) There is no consistent pattern found for the ones digits F3 and $3.
O O.iid
A1
and S1; C2
approximate version, with order times varying 2) S2, usually 3) C2, occasionally F3, and other parts the addition rarely contain any mutual information between the query and generated strings.
Fl and SI, either exactly
x O.post
Fi and S1; C2
A1
Fuzzy info about
1) Fl and S1 digits (sometimes exact match, sometimes approximate match). 2) Whether the sum of F2 and S2 is greater than equal to 10 (i.e. the value of C2). F3 and/or S3 digits often preserved from the query to the generated strings.
F1, SI and C2; Fuzzy info about
1.00
F3 and S3 (some- times)
49

---

Continuation of Table
Entry in
Interpretation from Table
Interpretation by Claude 3
Table 5
If VA==:: 1. Fi and SI digits from the query string appear in the generated strings, but their order may be swapped. 2. Occasionally some other information is shared, such as the carry digit C2, a specific digit like F3, but this not consistent across all examples.
If Al=l (likely to be): 1-2 digit from F1 and S1 (sometimes their sum); C2;
summary, the key common information is the two digits that appear in the hundreds place of the query and generated strings, while their specific positions as Fl or SI may vary.
2. A1
Else: The amount information shared between the query and generated strings varies by example. In some cases, there are clear patterns like certain digits being consistently shared (c.g. S1, S3). Other digits may share approximate rather than exact values (e.g. F1). Some digits tend to vary more (e.g. F2, S2). Occasionally, the carry C2 also shared. When there are too few generated strings, is hard conclusively identify patterns. More generated strings allow more confidence in the patterns found.
Else: Fuzzy info, including some info about and S3 (some- times)
Fuzzy info about and S2 (some- times)
Fl and S1 (sometimes fuzzy representation, their order can vary) 2. F2 and/or (sometimes only one of them, sometimes partially match) Occasionally F3 and/or S3 (partially match, their order can vary)
1. A1
1) Fl and S1 values, either exactly or approximately. Their order can sometimes be swapped. To lesser extent, F2 and S2 values, either exactly or approximately. Patterns for F3/S3 are rarely seen. Other aspects like C2 are occasionally but not consistently shared. The level of correspondence varies lot between examples. Examples with many missing values are hard to find patterns in.
1-2 digits from Fl and S1 (some- times fuzzy)
A,,,,Al A1
I) The values of and SI, either the exact values or their sum 2) The value or presence/absence of S2 3) The relationship between F2 and S2 (which is greater, or whether there is a carry) 4) Which digits are missing in the result (A1 or A2)
L.mid Al
Fl and S1; C2
values from query can be found in either order (FI. S1) or (S1, F1) in all generated strings for that query. In some cases, C2 (carry from tens place) is also consistent between query and generated strings, being either 0 or I in all strings for that query. F3 and S3 values do not seem have any limitations or correspondence between query and generated strings.
the Fl and
ppoat
and S1; C2
A1
Sometimes Al is consistent (always the same value) between query and generated strings Sometimes the exact or approximate F3 and S3 values from the query appear together frequently in the generated strings In some examples, there
0,00, A2
whether Al=1
clear common information The values of (F1, S1) and (F2, S2) do not seem to consistently shared between query and generated strings in these examples.
If A1=1: F2 (sometimes fuzzy); Al
If A1=1; Almost
Else: 1) The exact values or range of values for and/or S2 The value C2 (whether F2+S22 10) 3) Occasionally FI/SI or F3/S3, but not as common as F2/S2 4) Al is usually not shared The most commonly shared information is the exact values ranges for F2 and S2, well as the resulting C2. F1/SI and F3/S3 are less commonly shared. A=1:
info Else: 1-2 digits from F2 and $2
2.11 A2
and from the query string always appear in generated strings in the hundreds place, but their order may be swapped. 2. A1 is always the same in query and generated strings.
If A=1: both Fl and S1
d,2,A0 A2
Else: 1-2 digits
Else: F2 and S2 (sometimes $2 is fuzzy) 2) Number of digits in the final sum (related whether Al shown query) Less commonly, Fl and S1 are shared a fuzzy way.
from F2 and $2
50

---

Continuation of Table
Entry in
Interpretation from Table
Interpretation by Claude 3
Table 5
If A=I: 1) Fl and SI are often the same pair of digits (sometimes with a bit of variation). The order of the two digits may vary. 2) A1 is always the same digit. 3) In some cases, the presence absence of a carry from the tens place (C2) consistent. The digits the ones place (F3 and S3) are usually not shared between the query and generated strings.
If A=1: 1-2 dig- its from Fl and SI
d..3, A2
Else: 1) F3 and/or S3 often have specific shared values between the query and generated strings. The shared digit is usually in the range 2-9. 2) Occasionally other patterns exist in generated strings (e.g. ones place of answer) but are not truly shared with the query. 3) Besides the ones place operands, no other consistent patterns were found across F1/S1, F2/S2 or the answer digits.
Else: 1-2 digits from F3 and S3;
C3
If AT=:: 1) Fl and ST have fixed values each example, although the values differ between examples. Their order can vary. 2) A1 always "I" in all examples. 3) In some examples, there is a consistent carry or lack of carry from the sum of F2 and S2 (C2). But this is not true for all consistent patterns in the values of F2, S2, F3, S3 across all examples.
AI
If A1=1: F1 and
examples. 4) There
O.idd A2
S1
Else: F2 and $2; C3
Else: Al is always exactly shared, the middle digits F2 and S2 are often approximately shared, and usually one of the first digits (F1/S1) and of the last digits (F3/S3) from the query appears frequently in the generated strings.
A1=1: Fl and SI always match, although their order can be swapped. In some cases, the carry C2 (whether F2+S2 is 2 10) is also shared between the query and generated strings.
Al
n: 0.postt
If A1=1: Fl and SI
Else: 1) The digits the tens place (F2, S2), either the exact digits their sum (carry C2). 2) The ones digit of answer (A1). 3) Sometimes the digits in the ones place (F3, S3), either the exact digits or their sum (carry C3).
A2
Else: F2 and S2; C3
If Al=l: Fi and S1 (some- times fuzzy); C2 (sometimes)
Al=1: *used as example the prompt
Else: - F2 and S2 often have fixed values that are shared between the query and generated strings Al and A2 (when present) are consistently single digits in both query and generated strings
1.00
Else: 1-2 digits from F2 and S2;
A1=1: Some or all digits from the hundreds place (F1 and S1), sometimes exactly, sometimes approximately The carry to the hundreds place (C2), in some examples The first digit of the answer (A1) The digits in the tens and ones place (F2, S2, F3, S3) usually vary and are not consistently shared. When no generated strings are provided, the common information is uncertain. Flse: F2 and S2 are often fixed to specific values in the query and generated strings of each example. The sum of F2 and S2 determines the value of C2, which consistent between the query and generated strings within each example. Al is usually a single digit but can sometimes expand to 2 digits if the sum reaches or exceeds 100 in the generated strings. The specific digits of Al not always shared.
If A1=1: Un- certain. Fl and S1 (sometimes); digit from F3 and S3 (some- times) Else: F2 and $2
atd1, A2
I-2 digits from
Fl and SI often approximately match between the query and generated strings, but their exact values and order can vary. some cases, F2 and S2 also approximately match, but less frequently and consistently than Fl and S1. Al always exactly matches between the query and generated strings.
Fl and S1 (some- times fuzzy); 1- 2 digits from F2 and S2 (some- times fuzzy);
2.22 A2
If A1=1: 1) Fl and S1, their order can vary 2) Sometimes F2 and $2, either the exact values or approximate range of values 3) Al often the same and usually don't exhibit obvious patterns. The amount of shared information varies different examples, with some having more (c.g. exact F2 and $2 values) while others have less.
If Al=1: F1
and SI (some- times fuzzy): C2
1..3, A2
Flse: 1) One or both digits in the tens place (F2 and/or $2) 2) One more digits in the answer (A1 and/or A2) Less frequently, there are also patterns shared in the ones place (F3 and S3). The hundreds place (Fl and S1) least often contain common patterns.
Else: F2 and S2
51

---

Continuation of Table
Entry in
Interpretation from Table
Interpretation by Claude 3
Table 5
If A1= 1. Fi and S1 are always the same two digits but their order can vary. 2. The sum of F2 and S2 always the same, implying the carry C2 is consistent. 3. A1 always the same digit "1". There is consistent pattern found for F3
A1
S3.
If A1=1: Fl and
1.midd A2
SI; C2
Else: 1. F1 and S1: their specific values in query often show in generated strings in an approximate way, and their order can sometimes vary. 2. F2 and S2: the specific values small range of values often shared. 3. and S3: fuzzy version is often shared, meaning specific values may vary but are close those in query. 4. Al: often exactly the same hetween query and generated strings.
Else: and S2; C3
A1=1= FI and SI have possible values that swapped between Fl and S1. In other words, (F1, S1) have two possible combinations that are the reverse cach other.
always the same the query and all generated strings in cach example. Sometimes the carry C2 (whether F24S22 10) is also shared between query and generated strings.
A1=1: and
x: .post
S1; C2
Else: 1) Specific digits often fixed or highly common in certain places (e.g. F2, S2) across query and equations. 2) The answer digit A is often the same or highly consistent across query and equations. 3) The carry C2 from tens place is sometimes consistent (0 or 1) across examples. 4) Other than above, the digits and relationships seem vary substantially between examples.
A2
Else: F2 and S2;
C3
Digits in the answer part (A1, A2, etc.), either exact digit or a range. This is observed in all 5 examples. 2. The range that Fl and SI can take, and sometimes they can swap positions. This is observed in 2 out of examples. Digits in F2, S2, F3, S3 do not show consistent patterns across examples.
a0:0, A3
whether Al=]
When the sum is 2-digit number, AT is sometimes shared between query and generated strings When the sum is 3-digit number, A2 is often (but not always) shared If there shared A2, its value often matches the query If both A1 and A2 present, A A2 is between 10-19 There are no consistent patterns in the operand digits F1/S1, F2/S2,
0.11, a.1,As
1-2 digits from
F2 and $2
F3/S3 or their sums.
S2 or F2 sometimes takes consistent value (like 0, 9, 7) in the query and many generated strings. 2) The equations often incomplete, with and/or A2 missing many generated strings. 3) When present, A2 sometimes has consistent value (like 1, 2, 6) in the query and generated strings, but not always. 4) No consistent patterns were found for the (FI, S1) and (F3, S3) digit pairs.
1-2 digits from
2:2, A3
F2 and $2
I) Certain digits from F3 and/or S3, although the order F3 and S3 may vary in generated strings compared to the query. 2) The existence of Al and A2 in complete equations, but not always. Other patterns like digits from F1, $1, F2, S2 rarely shared.
1 digit from F3 and S3 when F3-S3); C3
0.3 A3
An exact or approximate representation of (FI, ST), sometimes with order varying 2. Specific values or range of values for F2 and S2, some- times along with their sum (C2) 3. Sometimes specific values for F3 and/or S3 Often exact approximate representation of (A1, A2) amount type of information shared varies between examples. But in general, there are usually some constraints on the digits in each place value shared by the query and generated strings.
A2; F2 and S2:
0.mid A3
C3: whether
Al=1
Fl and SI: exact match fuzzy match, order of Fl and SI may vary 2. F2 and S2: exact match fuzzy match, order of and S2 may vary. sometimes have size relationship (like F2 < S2) 3. F3: exact match or fuzzy match 4. S3: exact match or fuzzy match 5. A1: exact match 6. A2: exact match Not every example has all the above common information.
0.post
A2; and S2:
C3: whether
A3
Al=l
52

---

well, although in Continuation of Table
Entry in
Interpretation from Table
Interpretation by Claude 3
Table 5
If A1=1: 1. F2 and/or S2 (sometimes their roles as F2 or S2 can vary) 2. A pattern about A1 and A2: when Al is specific value (usually "I"), A2 is provided; otherwise A2 is not provided 3. Sometimes Al 4. Rarely
Al=1: 1-2 dig- its from F2 and
A2 when A1 is a specific value
a 1.10, A3
S2: C3 (some-
Else: Fuzzylapproximate matching of F3 and S3 values Exact match- ing of F3, S2, S3 values in some cases Partial matching of A2 (last digit) in some cases Overall, ones place digits (F3, S3) and occasionally other shared in a fuzzy or exact manner. Hundreds place digits (F1, S1, 1) are rarely shared.
times);
Else: 1-2 digits
from F3 and S3
digits (S2, A2) tend
If A1=1: F2 and S2 are always certain fixed digits in each example. shared by the query and all generated strings in that example. Sometimes the first answer digit is also shared between query and some generated strings, but this is not fully consistent. The last answer digit shows no consistency between query and generated strings.
If A1=1: 1-2 dig- its from F2 and
1.11 A3
S2: C3 (some-
times);
Else: 1) S3 (sometimes F3) 2) Fuzzy representation of (A1, A2) in some cases The ones place digit (F3 or S3) tends to be exactly preserved, while the answer (AI, A2) is sometimes preserved approximately but not exactly, Other digits and the carry do not show consistent patterns. If A1=1: 1. Fl and S1 (order may vary, values may be fuzzy or exact match) 2. F2 (may be exact fuzzy match) 3. S2 (may exact fuzzy match, may appear in F2 or S2) 4. A1 (always exact match) S. A2 (may be exact match or not present) The amount of shared information varies across examples. F3, S3 are not shared.
Else: 1-2 digits from F3 and S3
If Al=1: 1-2 dig- its from F2 and S2: C3 (some-
times); 1-2 digits from F1 and S1 (fuzzy);
a.2 A3
Else: Across the examples, the most common pattern is that either F3 S3 has fixed single digit value shared by the query and all generated strings that example. The specific digit varies between examples but consistent within each example.
Else: 1-2 digits from F3 and S3
If A1=1: 1. (F1, ST), but usually in approximatc or fuzzy way. Their exact values and order may vary in generated strings. (F2, S2), usually exactly the same as in query string. In few cases, they are shared in an approximate way. 3. AI, sometimes shared between query and generated strings. 4. S3, occasionally shared if it is present. The amount and type of shared information share (F2. S2), while others share (FI, S1) and an approximate manner.
If A1=:: 1-2 dig- its from F2 and
vary from example example. Some only
S2: C3 (some- times);
A..3.A3
Else: 1-2 digits from F3 and S3
Else: - The last digit of F (F3) is often shared between query and gen- erated strings The last digit of the answer (A2 for 2-digit answer, A3 for 3-digit answer) is very frequently shared Sometimes S3 other answer digits are also shared, but less reliahly In general, the ones place digits of the operands and answer tend to be shared between the query and generated strings in each example.
If A1=1: A2;
Fl and S1 (sometimes fuzzy values or ranges); F2 and S2 (sometimes unordered); F3 (rarely); A1; A2
If A1=1:
and S2; C3;
2
Lmiid
A3
whether Al=l
Else: A2; F2 and
Else: F3 and S3, sometimes only appearing in an approximate manner and sometimes their relative order can vary; A2, always precisely shared between the query string and generated strings.
S2: F3 and S3:
whether Al=l
If A1=1:: 1. Approximate or exact values of Fl and S1, their order can vary 2. Exact values or couple possible values of F2 and S2, their order can vary 3. Exact approximate values of A1 and A2 The information about F3 and S3 does not seem to shared.
If Al=1: A2;
F2 and S2; C3;
p.post A3
whether Al=l
Else: A2; F2 and
Else: Some or all digits in F2, F3, S2, S3 (F2 and S2 F3 and S3 can be swapped)- - The last digit of the answer (A2) Whether there is a carry from tens place to hundreds place (C2)
S2: F3 and S3;
whether Al=1
The value of Al (in 4 out of examples) 2) An approximate exact match of (F1, S1) pair (in out of 5 examples) 3) The sum of F3 and S3 (in out of examples) Other occasional commonalities include the value of A2 and whether A3 is complete.
0,0 A4/E
whether A1=1
53

---

Continuation of Table
Entry in Table
Interpretation from Table
Interpretation by Claude 3
5
Across the examples, the most consistent commonality is in the ones place (F3 and S3). They often have the same or very similar values in the query and generated strings.
1-2 digits from
Sometimes a digit in certain place of the final answer is also shared between query and many/all generated strings, like the first digit (A1) I) or last digit (A3). But this is not always the case. The hundreds place (F1/S1) and tens place (F2/S2) digits usually vary without clear mutual pattern between the query and generated strings.
d. A4/E
F3 and S3
1-2 digits from
1) F3 and S3 digits (sometimes approximately). Present 4 out of examples. 2) Al and/or A2 digits (only in example 0).
9:2 A4/E
F3 and S3
Cerrtain digit positions (often F3 S3) having the same value in the query and all/most generated strings When sums are shown, certain digits (often Al or A3) being restricted to a small set of values or range The specific digit positions and values/ranges vary between examples. but the general pattern of certain positions being fixed constrained is consistent.
1-2 digits from
0.3 A4/E
F3 and S3
FI and S1 sometimes appear in generated strings with their order swapped - F3 and S3 sometimes appear in generated strings with their order swapped F3, S3 and A3 of the query string often appear unchanged the same position generated strings
0.mid A4/E
A3; F3 and S3; whether Al=l
FF is oflen shared between the query string and generated strings S3 is sometimes shared between the query string and generated strings A3 is consistently the same value in the query string and all generated strings for each example There
x Opont
A3; F3 and S3; whether Al=l
A4/E
strong pattern for F1, S1, F2 S2.
1-2 digits from F3 and S3:
1. Specific digits or narrow ranges in F3 and S3 2. The value or a narrow range of A1 3. The ending digits or a narrow range of ending digits in A3 4. Sometimes the carry Cl or C2 The digits in F1, S1, F2, S2 and the full value of A2, A3 are usually not shared.
1.0 A4/E
whether next
token is
1-2 digits from
F3 (always the same or frequently appears) 2) S3 (always the frequently appears) 3) One or more answer digits, especially the last one A3 (always the same or frequently appears)
1. A4/E
F3 and S3:
whether next
token is E
The ones digit (F3) of the query string consistently shows up in the ones place (either F3 or S3) of the generated strings. Sometimes the other ones digit in generated strings has specific value when query F3 is in certain place. Occasionally, the carry (C3) from the ones place addition is also shared between the query and generated strings.
1-2 digits from F3 and S3
a.? A4/E
only the ones digits (F3 and sometimes S3) are consistently shared, while the other parts of the addition problems vary between the query and generated strings in each example.
1-2 digits from
1/33, A4/E
F3 and S3
F3 and S3 have some fixed values (varying by example) that always sum to the same total. The order of the two digits doesn't I matter. As result, A3 is always fixed value for each example. There sometimes a consistent carry over amount from the tens to hundreds place, resulting a fixed A1 value.
x 1.idd
A3; and S3;
A4/E
whether Al=1
Specific digits in the operands (F3, S3) Whether there is carry in certain place (C2, C3) The shared information varies across examples, usually relates to ones or tens place digits and carries.
Specific digits in the answer (A3)
.post
F3 and S3:
A4/B
whether Al=1
Table 5; Interpretation for 3 digit addition produced by Claude 3, compared with human interpretation from T In general, the automated information is very informative, and the human interpretations is contained in almost all cases, though the output tends be more verbose. The LLM outputs, with some human post-checking, can thus further speed up interpretation.
K Compute Resources
We ran all experiments on NVIDIA A100 cards. The decoder is trained for 4-6 hours on 2 GPUs for the first three case studies, without exhaustively tuning efficiency of the implementation, which we believe could further speed up training. Regarding the factual recall task, we train the decoder
54

---

for less than I day on 4 GPUs. Generation of e-preimage samples (including forward passes on the probed model to calculate distance metric) is fast for the first tasks, and takes around hours on GPUs for factual recall task (for 200 query inputs). Patching experiments run quickly, as they are done for small models.
55

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We ensured that all claims made in abstract and introduction are well supported by our results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We have a "Limitations" paragraph in the Discussion section, where we discuss limitations of this work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
56

---

Answer: [NA]
Justification:
The paper includes no theorems or other non-empirical claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Besides explaining all experimental details, we also provide the code and the probed models.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
57

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have attached the code with the submission. No new data were created.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ips.cc//pill wttpcccpppppppppeeeeeeeeeeeeeeee httpa///guddeee/eeeeabiiissssss /Cc////oddiiisssseeeeeee ssiiiiiiccyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We specify all relevant details, and also include the code for reproducibility,
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: All quantitative results have sample sizes of hundreds of thousands of data points, and error bars would be very small, thus not add substantial information.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
58

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include an appendix section on Compute Resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This foundational research on neural networks, and we anticipate no societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
59

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not any risk of misuse of our research results.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We the GPT-2 Small model and IOI templates, both of which cite. We use no other existing models or data.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
60

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We document the new assets, which are models of character counting and
addition.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper involves no crowdsourcing or human subjects experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper involves no crowdsourcing or human subjects experiments.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
61

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
62


Reviews:
[{"review_id": "vwt74kzsu2", "reviewer": "Reviewer_Mv2t", "summary": "In this paper, the authors propose InversionView, a method to inspect the information encoded in neural activations. The proposed method is based on checking the activations difference given different inputs. The authors showcase the effectiveness of this tool on mainly three tasks: character counting, indirect object identification, and 3-digit addition.\n\nStrengths\n\n1. The topic of model interpretation is important.\n2. The empirical results confirm the effectiveness of the proposed method.\n3. The paper is well written and easy to follow. In addition, the authors provide abundant experiments to showcase the effectiveness.\n\nWeaknesses\n\n1. Although being described through text, the precise algorithm for the proposed method is unclear to me.\n2. It is unknown how the proposed method scales with larger models. The largest model size in the paper is limited to GPT-2, which is not considered large nowadays.\n\nOverall, I think the paper is interesting and recommend a weak acceptance. I will consider raising my ratings if the above weaknesses are addressed.", "questions": "See the weaknesses.", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The topic of model interpretation is important.\n2. The empirical results confirm the effectiveness of the proposed method.\n3. The paper is well written and easy to follow. In addition, the authors provide abundant experiments to showcase the effectiveness.", "weaknesses": "1. Although being described through text, the precise algorithm for the proposed method is unclear to me.\n2. It is unknown how the proposed method scales with larger models. The largest model size in the paper is limited to GPT-2, which is not considered large nowadays.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer Mv2t,\n\nThanks for your feedback on our paper!\n\n## Reply Regarding Weaknesses\n\n### The precise algorithm for the proposed method is unclear \n\nIn the global rebuttal, we provide a new figure showing the training and sampling pipelines. In its caption, we also provide detailed explanation. \n\nIf you are unclear about the the decoder. We also provide a new figure showing decoder architecture in global rebuttal. The decoder model is basically a decoder-only transformer combined with some additional MLP layers. In order to condition the decoder on the query activation, the query activation is first passed through a stack of MLP layers to decode information depending on the activation sites and then made available to each attention layer of the transformer part of the decoder. There are also details about it in Appendix C in the paper (we should have linked section 2.2 to that).\n\nWe will add the new figures to the next version of the paper.\nTaken together, these changes will make the precise algorithm much more accessible to the reader.\n\n\n\n### How the proposed method scales with larger models \n\nThank you for raising this point. Please refer to the global rebuttal, where we describe our new experiment on a larger model, with 10x more parameters than in the IOI case study.  We find that InversionView continues to produces interpretable results, and allows us to read out interesting information content.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "34H7p7vB2M", "reviewer": "Reviewer_vTBG", "summary": "This work is mainly based on the representational geometry of activations in the activation space and chooses samples whose distances are within a defined $\\epsilon$-preimage distance. A single two-layer transformer decoder model is trained using activations of each layer from the investigated model. After finishing decoder training, the Euclidean distance is used as a metric to select generated samples based on each activation presentation of each layer, and compare those samples with the query input sentence. This approach is evaluated and analysed under three tasks, i.e., character counting with a 2-layer and 1-head transformer, IOI with GPT2-small and 3-digit addition with 2-layer, 4-attention head transformer. The comprehensive experiment results and analysis confirm the geometry hypothesis, and provide lots of insights to future work.", "questions": "- In line 115, what does $\\mathbf{z}$ represent here? and what is the difference between $\\mathbf{z}$ and $\\mathbf{z}^q$?\n- In line 108, Is this input the same as the query input in figure 1? Another figure introducing the whole pipeline might help a lot to understand.\n- Does this neighbour of geometry hypothesis always hold across all LLM activations?", "limitations": "I am looking forward to how to automate this approach and apply it to the larger LLM interpretability analysis.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Activation representation of each layer from small neural networks to GPT2-small can be visualized and explained based on the geometry hypothesis with a trained decoder using those activations as input\n- This hypothesis is evaluated using different tasks, i.e., character counting, IOI and 3-digit addition, and lots of meaningful analysis and insights are discussed\n- Experiments are solid and the appendix includes many details about each case study.", "weaknesses": "- I suggest authors use another figure to demonstrate the whole training and evaluation pipeline, i.e., what input is used for training probed model, what query input is for decoder training and what input and output are from the trained decoder, which will help to understand the whole framework much easier for readers.\n- This method is mainly based on manual investigation to select and analyse generated samples, which might need large labour resources to extend to large-scale LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer vTBG,\n\nThanks for your feedback on our paper!\n\n## Reply Regarding Questions ##\n\n### In line 115，what does $\\mathbf{z}$ represent? ###\n\nSorry for not writing it clearly. The $\\mathbf{z}$ is the same as $f(\\mathbf{x})$ in the previous part of the paper. So it simply represents an arbitrary activation that is compared with the query activation $\\mathbf{z}^q$. We will make it clear in the next version.\n\n### In line 108, is this input the same as the query input in figure 1? ###\n\nYes, \"input\" here refers to the same as the query input in Figure 1. It's a sequence of tokens. Note that we use the modifier \"query\" in the context of comparing it to its neighbourhood during interpration. Thus, in line 108, we do not use \"query\" because the input is not mentioned in the context of a neighbourhood.\n\n\n### Does this neighbour of geometry hypothesis always hold across all LLM activations? ###\n\n\n\nBecause transformers define continuous functions, the hypothesis should hold in principle. Of course, that doesn't guarantee it will always hold in a practically meaningful sense. As experimentally demonstrated in our paper, the hypothesis holds for the conducted case studies.\n\n\n## Reply Regarding Weaknesses ##\n\n\n### Another figure to demonstrate the whole training and evaluation pipeline ###\n\nThanks for your suggestion. We have made a new figure, which we have included in the global rebuttal PDF, and which we will add it to the paper in the next version.\n\n### Need large labour resources to extend to large-scale LLMs ###\n\n\n\n1. Neural activations can contain any kinds of information, so that any kind of template-based interpretation is likely not expressive enough. InversionView allows decoding various kinds of information. We believe manual or LLM-based interpretation is likely necessary given the variability of the information that can be encoded. \n2. InversionView is naturally suitable for using LLMs to automate interpretation, because it produces samples in input space that LLM can easily read, rather than abstract data structures. \n3. As our experiment (*Appendix J Automated Interpretability*) shows, LLMs can detect the main information in the preimage, despite occasional hallucination. We use manual investigation throughout in order to ensure correctness, since this is a scientific paper. We firmly believe that automated interpretation can be further improved in the future with better prompt engineering and better LLMs.\n3. Note that, as InversionView is a method for reading information from neural activations, it facilitates reverse-engineering (which by nature requires a lot of work) but it does not require reverse-engineering to work. Our paper aims to show that InversionView is one of the tools in the ecosystem of interpretability research, rather than an all-inclusive solution. In the paper, we reverse-engineer models and provide causal verification in order to show the correctness of the information given by InversionView. The number of samples one needs to inspect to interpret a single activation vector does not necessarily increase with the model size, as we show in the case of factual recall task. For larger models, one may use it to study a specific part of the model.\n4. Exhaustive interpretation or reverse-engineering requires a lot of work -- this is a feature of mechanistic interpretability research in general. InversionView is a faster way to generate accurate hypotheses. When combined with existing methods, InversionView makes the overall workflow more efficient, thereby decreasing guesswork and promoting faster iteration cycles in mechanistic interpretability research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TvXwFDN0AT", "reviewer": "Reviewer_P3CM", "summary": "The method proposed in the paper seeks to decipher the information encoded within neural network activations. The core idea is to examine subsets of inputs that produce similar activations, utilizing a trained decoder model conditioned on these activations. The authors perform their analysis on three different tasks: character counting, indirect object identification, and 3-digit addition.", "questions": "* Do you think it is possible to use other distance metrics besides L2-based metrics? Additionally, why do you believe the L2 distance is effective in this context? Is there something unique about the geometric space of these tasks that makes L2 distance particularly suitable? What types of tasks, in general, can be effectively solved using L2-based metrics?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed method helps us find which parts of a network handle specific tasks. For instance, in a counting task, we can see which activations detect the target character and which ones do the counting. This is interesting because it helps us understand how neural networks work inside.\n\n* Showing the underlying algorithm the model uses to solve tasks, like adding 3-digit numbers, gives useful insights.", "weaknesses": "* Applying the method to extremely large models and more complex tasks is challenging.\n\n* Selecting an appropriate distance metric and epsilon is challenging and requires tuning, making it difficult to adapt the method to different tasks. This also highlights the authors' impressive effort in making the method work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer P3CM,\n\nThanks for your feedback on our paper!\n\n\n## Reply Regarding Questions\n\n\n### It is possible to use other distance metrics besides L2-based metrics? Why do you believe the L2 distance is effective in this context? Is there something unique about the geometric space?\n\nYes, we think it's possible to use other distance metrics. Since our work explores a novel direction, we would like to start with simple and common metrics, in order to show that the idea works without complicated metrics. \n\nTherefore, we use L2 distance because of its simplicity and straightforward intuition, so that readers can easily grasp the idea and realize how much we can learn from the geometric space, instead of regarding L2 distance as the optimal choice.\n\nIn addition, in the paper we show L2-based metrics are effective empirically, as the information obtained using the simple L2-based metrics is well supported by causal and quantitative experiments.\n\n## Reply Regarding Weaknesses\n\n### Applying the method to extremely large models and more complex tasks is challenging\n\nWe agree that interpretation of extremely large models tends to be more challenging in general. On large models and complex tasks, a potential challenge is the complexity of the information itself, but this can in principle be overcome by scaling the decoder so it can learn more complex inverse mappings, and basing the decoder on pretrained language models can help. We show the feasibility of this approach in the factual recall task on GPT2-XL (see global rebuttal), where we read information from a model 10x larger than in the IOI case study.\n\n### Selecting an appropriate distance metric and epsilon is challenging and requires tuning\n\n\nAs we mentioned above, we show that the method works with the most common and simple distance metric. There was in fact no need at all for effortful tuning of these aspects -- though using more sophisticated metrics could be a topic for follow-up work. Regarding the threshold $\\epsilon$, we simply choose a small threshold for which the preimage shows meaningful patterns. Note that, unlike neural network training, where people train models repeatedly to tune hyperparameters, in the preimage returned by InversionView, one can immediately see possible interpretations resulting from different thresholds. We have more detailed discussion regarding the threshold, and our method's robustness to it, in Appendix A.4.\n\nIn summary, the fact that simple distance metric and threshold choice work highlights the robustness of our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fl002XA40i", "reviewer": "Reviewer_fZCU", "summary": "The paper introduces the method InversionView, which finds inputs that give rise to similar activations (the preimage). As the preimage grows exponentially with sequence length the authors train a conditional decoder to generate inputs from a target activation.\nThe authors show three use cases: a character counting task, Indirect Object Identification, and 3-digit addition.", "questions": "- a graphic showing how to get from source transformer activations to pre image samples would be helpful\n- Figure 2 is hard to parse: I would suggest highlighting your conclusion/interpretation (aka one activation encodes target character but not count, one activation encodes count but not target character etc) in the figure.\n- in general, for the experiments, I would suggest to clearly state the task and your findings before jumping into technical details\n- 3.3 \"We apply InversionView to the components of the circuit\" seems like you apply inversionView to specific activations found by [39] but then you state \"InversionView unveils the information contained in the activation sites, with\nresults agreeing with those of Wang et al. [39], while avoiding the need for tailored methods.\"\nSo it seems like you needed the \"tailored methods\" to know where to apply inversionView in the first place, or am I misunderstanding sth?", "limitations": "- I think when moving from toy tasks to SOTA models, inversionView might become very prone to human tendency for pattern matching\n- in addition polysemanticity might be very relevant for InversionView results but is not discussed at all\n- related work is not discussed in sufficient detail", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The approach of generating text examples that lead to similar activation patterns is intuitive and could be useful for interpretability since results are very human readable.\nThe example use cases (especially the toy task of character couting) seem promising.", "weaknesses": "- the abstract could be a bit more specific\n- in general, I think the reader is left wondering for a bit too long what the authors actually do and how the method works.\n- line 28: \"which in turn helps us put together the algorithm implemented by the model.\" promises a bit too much\n- motivation and description of the method is too vague\n- very few details on the method/decoder that generates the examples that are later inspected\n- results of experiments are a bit hard to pass. It would be nice if the reader is guided through those a bit better at different levels of abstraction\n- the approach (generating human interpretable examples that achieve a specific activation patters) seems closely related to feature visualization/ adversarial example and counterfactual example generation/ in general methods that generate in the input space, however those works are not discussed at all", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer fZCU,\n\nThanks for your feedback on our paper!\n\n## Reply Regarding Questions\n\n### Questions 1-3: writing and demonstrating suggestions\n\nThanks for your suggestions. Most importantly, we have created two figures describing the decoder and the workflow, included in the global rebuttal PDF. We will also address your other two suggestions.\n\n### It seems like you needed the \"tailored methods\" to know where to apply inversionView in the first place, or am I misunderstanding sth?\n\nYes, there is a misunderstanding. There are two key parts in Wang et al. [39]: They (1) use path patching to identify important components, and then (2) study the function of these components using tailored methods. By \"avoiding the need for tailored methods\", we refer to (2). In order to know where to apply InversionView, we only need (1), i.e., a general-purpose method for  finding circuits.\n\nFor example, [39] use path patching to identify certain heads (\"S-Inhibition heads\") affecting Name Mover head queries, and then use tailored patching experiments to show that they output both token and position signals. These experiments  were designed specifically to disentangle these two effects, ablating or inverting token or position information. On the other hand, InversionView directly reads out these two kinds of information (Figure 17 shows an example for an S-Inhibition head containing position information), obviating the need for guessing possible information and tailoring patching experiments.\n\nWe will add this specific example to the paper.\n\n## Reply Regarding Weaknesses\n\n### Weaknesses 1-6: Writing and Presentation\n\nThanks for your suggestions, which we appreciate and which we will implement. \n\nKey points:\n\n- We will provide more specific information about our studies in the abstract.\n\n- The two new figures described above will make it easy for the reader to quickly grasp how the method works.\n\n- We will change the sentence in line 28 to \"which in turn helps us identify how the information flows through the model. This is crucial to obtain the algorithm implemented by the model. \"\n\n- We respectfully submit that we do have formally described the method/decoder in Appendix C (decoder architecture), D.1, E.1, F.1 (decoder training details for individual tasks), and A.2 (sampling details). We will expand these sections with more discussion. \n\n- We will revise the description of results to provide more high-level intuition. We would be grateful for any more detailed guidance.\n\n### The approach seems related to feature visualization/ adversarial example and counterfactual example generation \n\nThanks for pointing out other methods that also generate in input space. We explain the difference between InversionView and these methods below. We will add this comparison to the paper. \n\nFeature visualization generates inputs that *maximally activate* a certain neural network unit, while InversionView finds inputs that result in the *same* vector. Whereas feature visualization interprets an individual neural network unit (e.g., neurons) to understand its general role across inputs, InversionView  interprets specific values of inner representations of neural networks. When the input changes, the value and thus the interpretation may change.\n\nAdversarial example or counterfactual example generation methods generate input that is similar to the original input but results in different outcome. While similar in input space, the adversarial/counterfactual input is likely to be quite different in internal representation space, leading to a different output. In contrast, we are interested in how different inputs in input space are represented very similarly in internal representation space.\n\n## Reply Regarding Limitations \n\n### On SOTA models, inversionView might become prone to human tendency for pattern matching\n\nWe would like to argue that pattern matching is fundamental for interpretation. Pattern matching is essential when interpreting neurons, studying the function of an attention head, or, as done in our paper, inspecting preimages. Neural activations may contain arbitrarily complex information, hard to capture by a fixed set of templates. Therefore, analysis by an intelligent entity (artificial or biological), capable of identifying novel patterns, is necessary. As we discuss in our paper, LLMs are useful but not good enough to replace humans at this task of pattern matching yet.\n\nMoreover, InversionView is a method for generating hypotheses, not final interpretations. Generated hypotheses must be verified with intervention experiments, as we do in the paper. An incorrect hypothesis due to an error in pattern matching will be identified at this stage, ruling out dependence on subjectivity in pattern matching.\n\n### Polysemanticity might be very relevant for InversionView results but is not discussed\n\nThanks for raising this point. Polysemanticity is widely observed when one wants to find a unified interpretation across all inputs for a model component such as a neuron -- in contrast, InversionView interprets a specific activation (vector) on a specific input. Nonetheless, concepts similar to Polysemanticity may be defined in our setting. Specifically, we observe cases where the same (activation site, position) pair encodes different kinds of information on different inputs -- a certain kind of \"polysemanticity\". For example, in the factual recall task (see global rebuttal), some heads' outputs encode subject information on some inputs, and relation information on other inputs. InversionView is very helpful for studying this kind of \"polysemanticity\", because it decodes per-example information instead of average information. We will discuss in the paper.\n\n### Related work is not discussed in sufficient detail\n\nWe will add work related to feature visualization / adversarial example. We request the reviewer to suggest any other related work they find missing from the paper. We would be happy to include it too.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their reviews. We are encouraged that they found our method to be effective (Reviewer Mv2t) and to provide useful and intuitive insights (Reviewer vTBG, P3CM, fZCU), our experiments solid (Reviewer vTBG), and the paper well-written and easy to follow (Reviewer Mv2t).

We have addressed the weaknesses and specific questions from each reviewer in the individual responses. Below, we address two important points that are relevant to multiple reviews. 

## Regarding writing and demonstration suggestions ##
We thank the reviewers for pointing out various potential improvements in writing and presentation.
We apologize for compressed writing; as we have a lot of content but limited space, we had moved many details to appendix. We will revise text for better readability, and add figures to make presentation more straightforward. If the paper is accepted, with an additional page, we believe we can further improve the paper in this respect.

Most importantly, besides improving the writing, we provide a figure for the overall pipeline, and a figure for the decoder architecture; both are shown in the attached PDF (Figures 1 and 2). By illustrating the InversionView workflow, this concretely addresses key concerns of Reviewers Mv2t, vTBG, fZCU.

## A new case study on a larger model ##

Several reviewers (Mv2t, vTBG, P3CM) asked about applicability to larger models. Since submission, we have done experiments on a larger model, GPT-2 XL, which has 1.5B parameters --- a 10-fold increase over the maximum model size in the submission.  InversionView continues to produce interpretable results, and allows us to read out high-level information content. We've also updated the web application, so you can check results via the link provided in the paper (https://inversion-view.streamlit.app). We will incorporate the content about this experiment in the paper.

Below, we provide more details on the factual recall task, decoder training and sampling. We will include these in the final version of the paper. We observed many interesting examples. Due to space limitations, we put only one figure for it in the rebuttal PDF, but we strongly recommend playing around with our webAPP.


### Factual Recall: Background  ###

The factual recall task is defined as predicting the attribute given a prompt containing the subject and relation. 
The model is given a prompt such as *"LGA 775 is created by"*,  containing the subject *"LGA 775"* and the relation *"is created by"*. The model predicts the next token *"Intel"*,  an attribute of the subject. This task requires retrieving relevant knowledge, and we may expect  that neural activations contain high-level concepts.

Previous work [1] suggests that, in attention layers of the upper part of the model, attributes of the subject are moved to the residual stream of the last token.

### Factual Recall: Implementation Details  ###

In this case study, our intention is not to provide a full interpretation of the computations performed to solve this task, which we deem out of scope for this paper. Rather, we focus on a relatively small set of important attention heads in upper layers, and check if InversionView produces interpretable results. We select the 25 most important attention heads in the upper part of the model, as prior work found that attribute retrieval tends to happen there.[1] We estimate the importance of attention heads by the attribution method from Ferrando et al.[2]





The decoder model is fine-tuned from GPT-2 Medium (the components for processing query activation are randomly initialized), because we expect a more complex inverse mapping from activation to inputs to be learned. Concretely, to interpret activations encoding a certain attribute, the decoder may need to memorize knowledge about different subjects sharing the same attribute.

To train the decoder model, we collect text from 3 datasets: factual statements from COUNTERFACT[3] and BEAR[4], and general text from MiniPile[5].

We trained the decoder on outputs of the selected attention heads. We filtered out query activations resulting from heavily attending to BOS (weight > 0.6), which occurs in many attention heads in higher layers but likely results in attention outputs with little information.


We use the same distance metric as in the IOI study, and set  $\epsilon=0.25$. As mentioned in the paper, a larger threshold produces more coarse-grained information; we found this to provide more stable results on this large model, potentially reflecting its higher-dimensional geometry.

### Factual Recall: Observation  ###

We provide a sample figure in the attached PDF.
We observed many interesting examples, and strongly recommend playing around with our webAPP. When doing so, you may be interested in different kinds of information. When not attending exclusively to BOS, head $h^{29,9}$, $h^{30.8}$, $h^{32,12}$, $h^{33,0}$, $h^{37.7}$ almost always move information about the subject, and $h^{24,24}$, $h^{25,7}$, $h^{27,16}$, $h^{28,21}$, $h^{29,20}$, $h^{33,9}$ almost always move information about the relation, while other heads usually exhibit a mix of behaviors (see the complete list of heads in our webAPP). 

[1]: Geva, Mor, et al. "Dissecting recall of factual associations in auto-regressive language models." arXiv preprint arXiv:2304.14767 (2023).

[2]: Ferrando, Javier, and Elena Voita. "Information flow routes: Automatically interpreting language models at scale." arXiv preprint arXiv:2403.00824 (2024).

[3]: Meng, Kevin, et al. "Locating and editing factual associations in GPT." Advances in Neural Information Processing Systems 35 (2022): 17359-17372.

[4]: Wiland, Jacek, Max Ploner, and Alan Akbik. "BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models." arXiv preprint arXiv:2404.04113 (2024).

[5]: Kaddour, Jean. "The minipile challenge for data-efficient language models." arXiv preprint arXiv:2304.08442 (2023).

Meta-review: After carefully reading the reviews and the rebuttal, I feel the authors manage to answer most if not all of the concerns raised by the reviewers. And as such, I think the quality of the manuscript increases sufficiently for it to be considered for acceptance. 
In particular, reviewer fZCU, with the lowest score, most concerns seem to be around presentation and clarity. The rebuttal provides answers to the punctual questions raised by fZCU in detail and explain how the draft will be improved. I believe these changes will answer the concerns raised. But I strongly urge the authors to incorporate all the changes they suggested in the rebuttal and additional experiments, as well as improving the clarity of the manuscript in the camera ready version of the paper. Improving these points will be important for the paper to have the impact it deserves once published.
Decision: Accept (poster)