Title: InversionView: A General-Purpose Method for Reading Information from Neural Activations
Venue: NeurIPS 2024
Accepted: yes
Keywords: interpretability

explainability

mechanistic interpretability
Abstract: The inner workings of neural networks can be better understood if we can fully decipher the information encoded in neural activations. In this paper, we argue that this information is embodied by the subset of inputs that give rise to similar activations.  We propose InversionView, which allows us to practically inspect this subset by sampling from a trained decoder model conditioned on activations. This helps uncover the information content of activation vectors, and facilitates understanding of the algorithms implemented by transformer models. We present four case studies where we investigate models ranging from small transformers to GPT-2. In these studies, we show that InversionView can reveal clear information contained in activations, including basic information about tokens appearing in the context, as well as more complex information, such as the count of certain tokens, their relative positions, and abstract knowledge about the subject. We also provide causally verified circuits to confirm the decoded information.

InversionView: A General-Purpose Method for Reading Information from Neural Activations

Abstract

The inner workings of neural networks can be better understood if we can fully decipher the information encoded in neural activations. In this paper, we argue that this information is embodied by the subset of inputs that give rise to similar activations. We propose InversionView, which allows us to practically inspect this subset by sampling from a trained decoder model conditioned on activations. This helps uncover the information content of activation vectors, and facilitates understanding of the algorithms implemented by transformer models. We present four case studies where we investigate models ranging from small transformers to GPT-2. In these studies, we show that InversionView can reveal clear information contained in activations, including basic information about tokens appearing in the context, as well as more complex information, such as the count of certain tokens, their relative positions, and abstract knowledge about the subject. We also provide causally verified circuits to confirm the decoded information. 1

1 Introduction

Despite their huge success, neural networks are still widely considered black boxes. One of the most important reasons is that the continuous vector representations in these models pose a significant challenge for interpretation. If we could understand what information is encoded in the activations of a neural model, significant progress might be achieved in fully deciphering the inner workings of neural networks, which would make modern AI systems safer and more controllable. Toward this goal, various methods have been proposed for understanding the inner activations of neural language models. They range from supervised probes [2, 5, 4, 55] to projecting to model's vocabulary space [42, 7] to causal intervention [21, 54, 26, 13] on model's inner states. However, to this date, decoding the information present in neural network activations in human-understandable form remains a major challenge. Supervised probing classifiers require the researcher to decide which specific information to probe for, and does not scale when the space of possible outputs is very large. Projecting to the vocabulary space is restricted in scope, as it only produces individual tokens. Causal interventions uncover information flow, but do not provide direct insight into the information present in activations.

Here, we introduce InversionView as a principled general-purpose method for generating hypotheses about the information present in activations in neural models on language and discrete sequences, which in turn helps us identify how the information flows through the model-crucial for obtaining the algorithm implemented by the model. InversionView aims at providing a direct way of reading out

the information encoded in an activation. The technique starts from the intuition that the information encoded in an activation can be formalized as its preimage , the set of inputs giving rise to this particular activation under the given model. In order to explore this preimage, given an activation, we train a decoder to sample from this preimage. Inspection of the preimage, across different inputs, makes it easy to identify which information is passed along, and which information is forgotten. It accounts for the geometry of the representation, and can identify which information is reinforced or downweighted at different model components. InversionView facilitates the interpretation workflow, and provides output that is in principle amenable to automated interpretation via LLMs (we present a proof of concept in Section 4).

We showcase the usefulness of the method in three case studies: a character counting task, Indirect Object Identification, and 3-digit addition. We also present preliminary results on the factual recall task, demonstrating the applicability of our method to larger models. The character counting task illustrates how the method uncovers how information is processed and forgotten in a small transformer. In Indirect Object Identification in GPT2-Small [54], we use InversionView to easily interpret the information encoded in the components identified by Wang et al. [54], substantially simplifying the interpretability workflow. For 3-digit addition, we use InversionView to provide for the first time a fully verified circuit. Across the case studies, InversionView allows us to rapidly generate hypotheses about the information encoded in each activation site. Coupled with attention patterns or patching methods, we reverse-engineer the flow of information, which we verify using causal interventions.

2 Methodology

Interpretation Framework What information does an activation in a neural network encode? InversionView answers this in terms of the inputs that give rise to this activation (Figure 1). For instance, if a certain activation encodes solely that ' the subject is John ' and nothing else (Figure 1, right), then it will remain unchanged when other parts in the sentence change while preserving this aspect (e.g., ' John is on leave today. ' ⇒ ' John has a cute dog. '). From another perspective, if all sentences where the subject is John are represented so similarly that the model cannot distinguish them, given one of these representations, the only information is the commonality ' the subject is John ' (assuming sentences are represented differently when it does not hold). Building on this intuition, given an activation, InversionView aims to find those

Johnisonleavetoday

Maryhas thedayoff

·Bobisnotworkingtoday

ActivationSitei

Johnisonleavetoday

Johnplays theguitarinhisfree time

Johnhasacutedog

ActivationSitej

inputs that give rise to the same activation, and examine what's common among them to infer what information it encodes. In realistic networks, different inputs will rarely give rise to exactly the same activation. Rather, different changes to an input will change the activation to different degrees. The sensitivity of an activation to different changes reflects the representational geometry: larger changes make it easier for downstream components to read out information than very small changes. This motivates a threshold-based definition of preimages, where we consider information as present in an activation when the activation is sufficiently sensitive to it. Formally speaking, given a space X of valid inputs, a query input x q ∈ X , a function f that represents the activation of interest as a function of the input, and a query activation z q = f ( x q ) , define the ϵ -preimage : 2

where ϵ > 0 is a threshold and D ( · , · ) is a distance metric. Both ϵ and D ( · , · ) are chosen by the researcher based on representation geometry; we will define these later in case studies. In practice,

dataset

Output X (shifted)

dataset

dataset

X (shifted)

X1

X2

X3

train

↑↑↑

Model

X1

X3

x2

Model

Z

X1

X2

X3

Z1

Z3

train

↑↑↑

Decoder

Xn

：

↑

Xn

...

↑

Xn

Zn

↑↑↑

Input

X

Input

X

InputZ

X

(a)TrainProbed Model

(b)TrainDecoder

Sample

X

Lbz

distance

Model....

+

↑↑

z9

Decoder

x'

X

：

m

Model...

Z

p

X

m

d

preimage

↑

4

Input

xq

Input

Input

X

(c) Interpret z9

in all our three case studies, we vary ϵ and set it so we can read out coherent concepts from the ϵ -preimage (Appendix A.4). With a threshold-based definition, we consider only those pieces of information that have substantial impact on the activation. See more discussion in Appendix A.1.

Conditional Decoder Model In this paper, we study the setting where x q is a sequence. Directly enumerating B z q ,f,ϵ is in general not scalable, as the input space grows exponentially with the sequence length. To efficiently inspect B z q ,f,ϵ , we train a conditional decoder model that takes as input the activation z q and generates inputs giving rise to similar activations in the model under investigation. In the following, we refer to the original model that we are interpreting as the probed model , the conditional decoder as the decoder , the place in the probed model from which we take the activation as the activation site (e.g., the output of i th layer), the inputs generated by the decoder as samples , and the index of a token in the sequence as position .

We implement the decoder as an autoregressive language model conditioned on z q , decoding input samples x (see Figure 2, and details in Appendix C). As the decoder's training objective corresponds to recovering x exactly, sampling at temperature 1 will typically not cover the full ϵ -preimage. Thus, for generating elements of the ϵ -preimage, we increase diversity by drawing samples at higher temperatures and with noise added to z q (details in Appendix A.2). We then evaluate D ( f ( x ) , z q ) at each position in each sample x , select the position minimizing D , 3 determine membership in B z q ,f,ϵ , and subsample inϵ -preimage and out-ofϵ -preimage samples for inspection.

An important question is whether this method, relying on a black-box decoder, produces valid ϵ -preimages. Correctness (are all generated samples in the ϵ -preimage?) is ensured by design, as we evaluate D ( f ( x ) , z q ) for each generated sample. The other angle is completeness (are the samples representative of the ϵ -preimage?). If some groups of inputs in ϵ -preimage are systematically missing from the generations, one may overestimate the information contained in activations. But this behavior would be punished by the training objective, since the loss on these examples would be high. We explicitly verify completeness by enumerating inputs in one of our case studies (Appendix B). Another approach is to design counter-examples x not satisfying a hypothesis about the content of B z q ,f,ϵ . In our experiments, we found that these examples were always outside of B z q ,f,ϵ .

3 Discovering the Underlying Algorithm by InversionView

Notation. In the transformer architecture, outputs from each layer are added to their inputs due to residual connection. The representations of each token are only updated by additive updates,

(a)

(b)

Activation Site:arO,muid (after layer O attention)

Activation Site:aO.post

(after layer O MLP)

Activation Site:a,0(layer1attention)

Baiaigiaaggilg(:)3

Attention

Attention

Weights

0.000; Baiaigia aggil (g): 3

0.000; B aiaigi a a ggil (g) : 3

Generated Samples

Generated Samples

0.000; Baiaigiaaggilg(:)3

Position of Query

True Count

0.007;#:3; Big a aigiigaa|(g) : 3 E

1 0.01l;#:3; B a ii a g a g ii a gl (g) : 3 E

Generated Samples

Activation

Position of

1 0.052;#:3; Bi a a ig ggiv a a|(g): 3 E

1 0.015;#:3; Bgg aiagi a ii a|(g): 3 E

1 0.023;#:3; Br c p c w wiii c|c (:) 3 E

Compared

Activation

I 0.074;#:3; B g k i g a k g k a k i| (g) : 4 E

1 0.017;#:3; B a g ia a a ig iigl (g) : 3 E

Distance D

I 0.074;#:3; B aig a ia a a ggi| (g): 3 E

1 0.018;#:3; B a igg a a aigiil (g) : 3 E

1 0.033;#:3; B m b o m b m b o b o |m (:) 3 E

1 0.039;#:3; B a q q a jj aja q|q (:) 3 E

I 0.075;#:3; Baia ag ia igg a|(g) : 3 E

1 0.020;#:3; B a g a a gg aiiiil (g) : 3 E

1 0.047;#:3; B m m m b g b g b b g| g (:) 3 E

I0.079;#:3; Bgiggaaia aia|(g): 3E

I0.066;#:3; Bii wiv v g a agg| (g) : 4 E

1 0.049;#:3;Bx0 xvovvx0x|o(:)3 E

I 0.091;#:4; B g r d g k g k w g a i| (g) : 4 E

I0.066;#:3; B wo w wggg wrrr|(g) : 3E

I 0.066;#:3; B w w m o w w o o m w |o (:) 3 E

Threshold

I 0.092;#:4; B g a ii g g i g a k k| (g) : 4 E

0.208;#:3; B g g w k k k k k g k a |(g) : 3 E

I0.073;#:3;Bfxxg0gxxgxo|g(:)3 E

= 0.1

0.117;#: 4; B g k k k g k k g k k g k | (g) : 4 E

0.412;#:4; B k j g g k k n g g k k | (g) : 4 E

0.112;#:4; B b b g b g g b g b o | g (:) 4 E

0.118;#:3; B k k k g g k k k k k g n I (g) : 4 E

0.429;#:4; B g g g z k g k k k d | (g) : 4 E

0.146;#:4; B p j h p jj e jjjjj h h j h | h () 3 E

0.220;#:2; B k k k ki k a g g k a  (g) : 3 E

0.494;#: 2; B a i g g i a a i a i a a | (g) : 3 E

0.499;#: 7; B n f 1 n k n I kf z f k n n n k I n f f| n (:) 8 E

forming a residual stream [17]. Using notation based on [17] and [40], we denote the residual stream as x i, { pre , mid , post } ∈ R N × d , where i is the layer (an attention (sub)layer + an MLP (sub)layer) index, N is the number of input tokens, d is the model dimension, pre , mid , post stand for the residual stream before the attention layer, between attention and MLP layer, and after the MLP layer. For example, x 0 , pre is the sum of token and position embedding, x 0 , mid is the sum of the output of the first attention layer and x 0 , pre , and x 0 , post is the sum of the output of the first MLP layer and x 0 , mid . Note that x i, post = x i +1 , pre . We use subscript t to refer to the activation at token position t , e.g., x i, mid t ∈ R d . The attention layer output decomposes into outputs of individual heads h i,j ( · ) , i.e., x i, mid = x i, pre + ∑ j h i,j ( LN ( x i, pre )) , where LN ( · ) represents layer normalization (GPT style/pre-layer-norm). We denote the attention head's output as a i,j , i.e., a i,j = h i,j ( LN ( x i, pre )) .

Decoder Architecture. We train a single two-layer transformer decoder across all activation sites of interest. The query activation z q is concatenated with an activation site embedding e , a learned embedding layer indicating where the activation comes from, passed through multiple MLP layers with residual connections, and then made available to the attention heads in each layer of the decoder, alongside the already present tokens from the input, so that each attention head can also attend to the post-processed query activation in addition to the context tokens. Each training example is a triple consisting of an activation vector z q ∈ R d , the activation site index, and the input, on which the decoder is trained with a language modeling objective. Appendix C has technical details.

3.1 Character Counting

We train a transformer (2 layers, 1 head) on inputs such as 'vvzccvczvvvzvcvc | v:8' to predict the last token '8', the frequency of the target character (here, 'v') before the separator ' | '. For each input, three distinct characters are sampled from the set of lowercase characters, and each character's frequency is sampled uniformly from 1-9. The input length varies between 7 and 31. We created 1.56M instances and applied a 75%-25% train-test split; test set accuracy is 99.53% (Details in

Appendix D). We use D ( z , z q ) = ∥ z -z q ∥ 2 ∥ z q ∥ 2 (i.e., normalized euclidean distance, as the magnitude of activations varies between layers), where z denotes the aforementioned f ( x ) , and set ϵ = 0 . 1 .

Interpreting via InversionView and attention. In layer 0, the target character consistently attends to the same character in the previous context, suggesting that counting happens here. In Figure 3a, we show the ϵ -preimage of x 0 , mid tc and x 0 , post tc , where the subscript tc denotes the target character. We show ≈ 10 random samples at a single query input, but our hypotheses are based on-and easily confirmed by-rapid visual inspection of dozens of inputs across different query inputs. 4 On the left (before the MLP), the activation encodes the target character, as all samples have 'g' as the target character. Count information is not sharply encoded: while the closest activation corresponds to 'g' occurring 3 times, two activations corresponding to a count-4 input ('g' occurring 4 times) are also close, even closer than a count-3 input. On the other hand, on the right (after the MLP), only count-3 inputs are inside the ϵ -preimage, and count-4 inputs become much more distant than before. Comparing the ϵ -preimage before and after the MLP in layer 0, we find that the MLP makes the count information more prominent in the representational geometry of the activation. The examples are not cherry-picked; count information is generally reinforced by the MLP across query inputs.

In the next layer, the colon consistently attends to the target character, and InversionView confirms that count information is moved to the colon's residual stream (Figure 3b). More importantly, this illustrates how information is abstracted: We previously found that x 0 , post tc encodes identity and frequency of the target character. However, the colon obtains only an abstracted version of the information, in which count information remains while the target character is largely (though not completely) removed. InversionView makes this process visible, by showing that the target character becomes interchangeable with little change to the activation. See more examples in Appendix D.2. Overall, with InversionView, we have found a simple algorithm by which the model makes the right prediction: In layer 0, the target character attends to all its occurrences and obtains the counts. In layer 1, the colon moves the results from the target character to its residual stream and then produces the correct prediction. Accounting for other activation sites, we find that the model implements a somewhat more nuanced algorithm, investigated in Appendix D.4. Overall, InversionView shows how certain information is amplified, but also how information is abstracted or forgotten.

Quantitative verification. We causally verified our hypothesis using activation patching [53, 21] on (position, head output) pairs. As the attention head in layer 1 attends almost entirely to the target character, only head outputs a 0 , 0 tc , a 0 , 0 : , and a 1 , 0 : can possibly play a role in routing count information. We patch their outputs with activations from a contrast example flipping a single character before '|'. We patch activations cumulatively, starting either at the lowest or highest layer, with some fixed ordering within each layer. For example, we patch a 0 , 0 : and observe how final logits change compared to the clean run, then we patch both a 0 , 0 : and a 0 , 0 tc and do the same, and so forth. By the end of patching, the model prediction will be flipped. When adding an activation to the patched set, we attribute to it the increment in the difference of LD before and after patching, where LD denotes the logit difference between original count and the count in the contrast example. Cumulative patching allows us to observe dependencies: For instance, as we hypothesize that a 1 , 0 : is completely dependent on a 0 , 0 tc , we expect that, when a 0 , 0 tc is already patched, patching a 1 , 0 : will have no further effect, whereas when a 0 , 0 tc is not patched, patching a 1 , 0 : will have a significant effect. Results (Figure 4a) match our prediction: Patching either of the activation in the hypothesized path ( a 0 , 0 tc and a 1 , 0 : ) is sufficient to absorb the entire effect on logit differences, confirming the hypothesis. See Appendix D.3 for further details and D.4 for further experiments.

3.2 IOI circuit in GPT-2 small

To test the applicability of InversionView to transformers pretrained on real-world data, we apply our method to the activations in the indirect object identification (IOI) circuit in GPT-2 small [48] discovered by Wang et al. [54]. We apply InversionView to the components of the circuit, read out the information, and compare it with the information or function that Wang et al. [54] had ingeniously inferred using a variety of tailored methods, such as patching and investigating effects on logits and

(a)

(b)

<lendoftext|>AfterErin andJustin wentto thehouse,Eringave aring(to)Justin

PatchingEffect-->

40

0.0o0 ; <lendoftext|>After Erin and Justin went to the house, Erin gave a ring( to) Justin

Increment

30

20

GeneratedSamples

10

1 0.024 ; 
The station Sara and Justin went to had a kiss. Sara gave it( to) Justin[EOS]

0

s ( a o paap ee oos  e s e o  e e xa> o1

a0.0

ac0

ai.0

40

PatchingEffect<--

1 0.034 ; 
Then, Justin and Erin went to the garden. Erin gave a basketball( to) Justin[EOS]

Increment

30

1 0.037 ; 
After the lunch in the afternoon, Justin and Kristen went to the station. Kristen gave a kiss( to) Justin[EOS]

20

1 0.039 ; 
After taking a long break Kimberly and Justin went to the house, Kimberly gave a bone( to) Justin[EOS]

10

1 0.056 ; <lendoftextl>Then, Justin and Kristen went to the school. Kristen gave a bone( to) Justin[EOS]

0

a0.0

a1.0

0.506 ; 
Friends separated at birth Kristen and Justin found a snack at the garden. Justin gave it( to) Kristen[EOS]

PatchedActivation

0.598 ; 
While spending time together Michelle and Joshua were commuting to the restaurant, Alexander gave a ring( to) Michelle[EOS]

attention. We show that InversionView unveils the information contained in the attention heads' outputs, with results agreeing with those of Wang et al. [54].

The IOI task consists of examples such as 'When Mary and John went to the store, John gave a drink to', which should be completed with 'Mary'. We use S for the subject 'John' in the main clause, IO for the indirect object 'Mary' introduced in the initial subclause, S1 and S2 for the first and second occurrences of the subject, and END for the 'to' after which IO should be predicted. To facilitate comparison, we denote attention heads as in Wang et al. [54] with i.j denoting h i,j . Wang et al. [54] discover a circuit of 26 attention heads in GPT-2 small and categorize them by their function. In short, GPT-2 small makes correct predictions by copying the name that occurs only once in the previous context. For InversionView, we train the decoder on the IOI examples (See details in E.1). Despite the size of the probed model, we find the same 2-layer decoder architecture as in Section 3.1 to be sufficient. We use D ( z , z q ) = 1 -z · z q || z ||·|| z q || (i.e., cosine distance), and ϵ = 0 . 1 . Euclidean distance leads to similar results, but cosine distance is a better choice for this case (Appendix E.4).

We start with the Name Mover Head 9.9, which Wang et al. [54] found moves the IO name to the residual stream of END. 4b shows the ϵ -preimage at 'to'. The samples in the ϵ -preimage share the name 'Justin' as the IO. The head also shows similar activity at some other positions (Appendix A.3). Results are consistent across query inputs. Therefore, InversionView agrees with the conclusions of Wang et al. [54] on head 9.9. Applying the same analysis to other heads (Table 2), we recovered information in high agreement with the information that Wang et al. [54] had inferred using multiple tailored methods. For example, Wang et al. [54] found S-Inhibition heads were outputting both token signals (value of S) and position signals (position of S1) by patching these heads' outputs from a series of counterfactual datasets. These datasets are designed to disentangle the two effects, in which token and/or position information are ablated or inverted. These two kinds of information can be directly read out by InversionView (Figure 19 shows an example for an S-Inhibition head that contains position information), and there is no need to guess the possible information to design patching experiments. Overall, among the 26 attention heads that Wang et al. [54] identified, InversionView indicates a different interpretation in only 3 cases; these (0.1, 0.10, 5.9) were challenging for the methods used before (Appendix E.3). In summary, InversionView scales to larger models.

3.3 3-Digit Addition

We next applied InversionView to the problem of adding 3-digit numbers, between 100 and 999. Input strings have the form 'B362+405=767E' or 'B824+692=1516E', and are tokenized at the

Activation Site: 0.0

Activation Site: α0.3

sum≥

'0b

x0.pre

B615+861(=)1476

B615+861(=)1476

ActivationSite:a,mid

0.000; B 61 5 + 8 6 1 (=) 1 4 7 6

0.000; B 6 1 5 + 8 6 1 (=) 1 4 7 6

0.000; B 6 1 5 + 8 6 1 (=) 1 4 7 6

F1 F2F3S1 S2 S3A1A2 A3 A4

GeneratedSamples

GeneratedSamples

Generated Samples

Decoder

0.002; B675+89 6(=)1571E

0.001; B 113 + 365 (=) 4 7 8 E

0.003; B618 +869 (=)1487E

Likelihood

Difference

0.002;B673+849 (=)1522E

0.001; B 613 +162 (=)77 5E

10.016;B860+610 (=)1470E

0.004;B645+80 4 (=)144 9 E

0.003;B 912+ 266 (=)117 8E

10.038;B658+812(=)1470E

0.004; B603+884 (=)1487E

0.005;B619+ 661(=)1280E

1 0.041; B819 +658 (=)1477E

0.009;B607+830(=)1437E

0.006;B419 +867 (=)1286E

1 0.042; B6 13 +857 (=)1 4 7 0 E

1 0.014; B 8 4 2 +611 (=)1 453 E

1 0.019; B 561 + 917 (=)147 8 E

1 0.050; B64 2+814 (=)1 456 E

|0.015; B 8 9 3 +68 4 (=) 1577 E

10.020;B667+ 916 (=)1583 E

10.059;B628+8 18 (=)1446 E

|0.015; B859 +636 (=)14 95E

10.088;B210+757 (=)967E

0.088; B826+652(=)1478E

0.142; B 8 0 6 + 5 4 9 (=) 1 3 5 5 E

0.106;B 441+ 610 (=)1051E

1 0.447; B713+657 (=)137 0 E

0.210; B72 0 + 8 4 9 (=)156 9 E

0.180; B 615 + 7 0 9 (=) 1 3 2 4 E

0.453; B 7 0 6 + 8 1 4 (=) 1 5 2 0 E

(a)

(b)

(c)

character level. We use F1, F2, F3 to denote the three digits of the first operand and S1, S2, S3 for the digits of the second operand, and A1, A2, A3, A4 (if it exists) for the three or four digits of the answer, and C2, C3 for the carry from tens place and ones place (i.e., C2: whether F2+S2 ≥ 10, C3: whether F3+S3 ≥ 10). Unlike [47], we do not left-pad answers to have all the same length; hence, positional information is insufficient to determine the place value of each digit.

The probed model is a decoder-only transformer (2 layers, 4 attention heads, dimension 32). We set attention dropout to 0. Other aspects are identical to GPT-2. The model is trained for autoregressive next-token prediction on the full input, in analogy to real-world language models. In testing, the model receives the tokens up to and including '=', and greedily generates up to 'E'. The prediction counts as correct if all generated tokens match the ground truth. The same train-test ratio as in Section 3.1 is used. The test accuracy is 98.01%. For other training details see Appendix F.1.

Interpreting via InversionView and attention. As Section 3.1 we use normalized Euclidean distance for D ( · , · ) and the threshold ϵ = 0 . 1 . We first trace how the model generates the first answer digit, A1, by understanding the activations at the preceding token, '='. We first examine the attention heads at '=' in the 0 -th layer (Figure 5). As for the first head ( a 0 , 0 ), only F1 and S1 matter in the samples - indeed, changing other digits, or swapping their order, has a negligible effect on the activation (Figure 5). Across different inputs, each of the three heads a 0 , 0 , a 0 , 1 , a 0 , 2 encode either one or both of F1 and S1 (Figure 26); taken together, they always encode both. This is in agreement with attention focusing on these tokens. The fourth and remaining head in layer 0 ( a 0 , 3 ) encodes F2 and S2, which provide the carry from the tens place to the hundreds place. Combining the information from these four heads, x 0 , mid consistently encodes F1 and S1; and approximately represents F2, S2-only the carry to A1 (whether F2+S2 ≥ 10) matters here (Figure 5c). Other examples are in

(a)

(b)

Key/Value"

→iLayer.Head

Verification of information flowfrom F1,S1

VerificationofinformationflowfromF2,S2

Query

10

-->

12

<

N

.3

8

10

Increment

6

8

:0.3

6

0.1:

10.2

0.01

4

4

2

2

B

6

5

Q

Q

8

6

a0.0

a0,1

PatchedActivation

1,3

0,0

0,1

PatchedActivation

1,2

Figure 27. We can summarize the function of layer 0 at '=': Three heads route F1 and S1 to the residual stream of '=' x = . The fourth head routes the carry resulting from F2 and S2 . Layer 1 mainly forwards information already obtained in layer 0, and does not consistently add further information for A1. See more examples in Appendix F.2.

Figure 6a shows the circuit predicting A1. InversionView allows us to diagnose an important deficiency of this circuit: Even though the ones place sometimes receives attention in layer 1, the circuit does not consistently provide the carry from the ones place to the hundreds place, which matters on certain inputs-we find that this deficiency in the circuit accounts for all mistakes made by the model (Appendix F.3). Taken together, we have provided a circuit allowing the model to predict A1 while also understanding its occasional failure in doing so correctly. Corresponding findings for A2, A3, and A4 are in Table 3 and Figure 31. From A2 onwards, InversionView allows us to uncover how the model exhibits two different algorithms depending on whether the resulting output will have 3 or 4 digits. In particular, when predicting A3, the layer 0 circuit is the same across both cases, while the layer 1 circuit varies, since this determines whether A3 will be a tens place or ones place. Beyond figures in the Appendix, we encourage readers to verify our claims in our interactive web application.

Quantitative verification. We used causal interventions to verify that information about the digits in hundreds and tens place is routed to the prediction of A1 only through the paths determined in Figure 6a, and none else. Like before, we cumulatively patch the head output on '=' preceding the target token A1, with an activation produced at the same activation site by a contrast example changing both digits in a certain place. Results shown in Figure 6b strongly support our previous conclusions. For example, a 0 , 3 and a 1 , 2 are not relevant to F1 and S1. Important heads detected by activation patching, a 0 , 0 , a 0 , 1 , a 0 , 2 , a 1 , 1 , all contain F1 and S1 according to Figure 6a. Furthermore, we can also confirm that a 1 , 1 relies on the output of layer 0 as depicted in sub-figure (a): When heads in layer 0 are already patched, patching a 1 . 1 has no further effect (value corresponding to → is zero), but it has an effect when patching in the opposite direction. On the contrary, a 1 , 0 shows little dependence on layer 0, consistent with Figure 6a. On the right of Figure 6b, we can confirm that a 0 , 3 is important for routing F2 and S2, and the downstream heads in layer 1 rely on it. Findings for other answer digits are similar (See Appendix F.5). Overall, the full algorithm obtained by InversionView is well-supported by causal interventions.

InversionView reveals granularity of information. Heads often read from both digits of a place, but only the sum matters for addition. Are the digits represented separately, or only as their sum? Unlike traditional probing, InversionView answers this question without designing tailored probing tasks. In Figure 7 (left), a 0 , 2 exactly represents F2 and S2 (here, 2 and 5). Other inputs where F1+S1=5+2 have high D . In contrast, on the right, F2 and S2 are represented only by their sum: throughout the ϵ -preimage, F2+S2=9. In fact, we find such sumonly encoding only when F2+S2=9a special case where the ones place of operands affects the hundreds place of the answer via cascading carry. We hypothesize that the model encodes them similarly because these inputs require special treatment. Therefore, even though encoding number pairs by their sum is a good strategy for the addition task from a human perspec-

Activation Site: α0,2

Activation Site:a1,3

B556+280(=)836

B920+878=(1)798

QueryInput

0.000;B556+ 280 (=)836

0.000;B920+878=(1)798

GeneratedSamples

GeneratedSamples

0.000;B561+238(=)799E

|0.015;B929+877=(1)806E

0.000;B522+29 5 (=)817E

1 0.041; B964+832= (1)7 9 6 E

0.004;B246+598(=)844E

10.048;B966+838=(1)804E

0.004;B263+563(=)826E

10.063;B911+885=(1)796E

0.675; B 6 61+255 (=) 916E

I0.064;B818+984=(1)802E

I0.066;B816+980=(1)796E

Manual Samples

I0.069;B948+851=(1)799E

1.013;B15 6 +68 0 (=) 8 36 E

0.729;B456+38 0 (=)836E

■0.120;B805+996=(1)801E

0.242; B 7 4 1 +9 50 = (1) 6 9 1 E

0.733;B356+480 (=)836E

1.012;B65 6 +1 8 0 (=) 8 3 6 E

0.406; B83 4 + 977 = (1) 8 1 1 E

tive, the model only does it as needed. We also observe intermediate cases (Figure 29).

3.4 Factual Recall

To test whether InversionView can be applied to larger language models, we explore how GPT-2 XL (1.5B parameters) performs the task of recalling factual associations. In this case study, our intention is not to provide a full interpretation of the computations performed to solve this task, which we deem out of scope for this paper. Instead, we show that InversionView produces interpretable results on larger models by focusing on a relatively small set of important attention heads in upper layers. The decoder model in this case study is based on GPT-2 Medium, because we expect a more complex inverse mapping from activation to inputs to be learned. We observe the resulting ϵ -preimage can express high-level knowledge (Figure 39-44), and sometimes can predict the failure of the model (Appendix H.6). Using InversionView, we again shed light on the underlying mechanism of the model. We present detailed findings in Appendix H.

4 Discussion and Related Work

Comparison with other Interpretation Methods Supervised probing classifiers , assessing how much information about a variable of interest is encoded in an activation site, are arguably the most common method for uncovering information from activations [e.g. 2, 6, 5, 4, 55, 52, 33, 34]. It requires a hypothesis in advance and is thus inherently limited to hypotheses conceived a priori by the researcher. InversionView, on the other hand, helps researchers form hypotheses without any need for prior guesses, and allows fine-grained per-activation interpretation. Inspecting attention patterns [e.g. 12] is a traditional approach to inferring information flow , and we have drawn on it in our analyses. More recently, path patching [54, 26, 13, 28, 35] causally identifies paths along which information flows. While the information flow provides an upper bound on the information passed along by tracing back to the input token, it is insufficient for determining how information is processed and abstracted. For instance, in Section 3.1, occurrences of the target character are causally connected to a 0 , 0 tc , which then connects to a 1 , 0 : (direct or mediated by MLP layer 0). Without looking at encoded information, we only know that the information in these paths is related to the occurrences of the target character, but not whether it is their identity, positions, count, etc. More generally, when a component reads a component that itself has read from multiple components, connectivity does not tell us which pieces of information are passed on. Similar considerations apply to other intervention methods.

Geva et al. [24] intervene on attention weights to study information flow. Activation patching, a causal intervention method, can be used to study the causal effect of an activation on the output, and can help localize where information is stored [37, 49], or find alignment between a high-level causal model and inner states of a neural model [21, 22, 57]. Many recent works obtain insights about information content by projecting representations or parameters into the vocabulary space [42, 7, 44, 31, 54, 23, 24, 16]. This technique is sometimes referred to as Direct Logit Attribution (DLA). We argue that DLA is only suitable for studying model components that directly affect model's final output. For those components whose effect is mediated by other components, their output information is meant to be read by a downstream component, thus not necessarily visible when projecting to the vocabulary space. We provide further discussion in Appendix I. Generalizing this approach, Ghandeharioun et al. [25] patch activations into an LLM. Another line of recent research [10, 51, 14] decomposes activations into interpretable features using sparse autoencoders .

Some other interpretation methods also generate in input space, but differ from InversionView in goals and methods. This includes feature visualization [43, 41], adversarial or counterfactual example generation [27, 59, 46, 45], and GAN inversion methods [58]. We discuss the similarities and differences of these works compared to InversionView in Appendix I.4.

InversionView offers distinctive advantages and makes analyses feasible that are otherwise very hard to do with other methods. It can also improve the interpretability workflow in coordination with other methods. For example, one may first use methods such as path patching or attribution [50, 18] to localize activity to specific components, and then understand the function of these components using InversionView. In sum, InversionView is worth adding to the toolbox of interpretability research.

Transformer Circuits for Arithmetic Related to Section 3.3, [47] interpret the algorithm implemented by a 1 -layer 3 -head transformer for n -digit addition ( n ∈ { 5 , 10 , 15 } ), finding that the model implements the usual addition algorithm with restrictions on carry propagation. In their one-layer setup, attention patterns are sufficient for generating hypotheses. Lengths of operands and results are fixed by prepending 0. Our results, in contrast, elucidate a more complex algorithm computed by a two -layer transformer on a more realistic version without padding, which requires the model to determine which place it is predicting. We also contribute by providing a detailed interpretation, including how digits are represented in activations.

Automated Interpretation for InversionView Recent work has started using LLMs to generate interpretations [8, 10]. The samples produced by InversionView can be easily fed into LLMs for automated interpretation. We show a proof of concept by using Claude 3 to interpret the model trained for 3-digit addition. See results in Table 5. The LLM-provided interpretation reflects the main information in almost all cases of the addition task. Despite some flaws, the outcome is informative in general, suggesting this as a promising direction for further speeding up hypothesis generation.

Limitations InversionView relies on a black-box decoder, which needs to be trained using relevant inputs and whose completeness needs to be validated by counter-examples. Also, InversionView, while easing the human's task, is still not automated, and interpretation can be laborious when there are many activation sites. We focus on models up to 1.5B parameters; scaling the technique to large models is an interesting problem for future work, which will likely require advances in localizing behavior to a tractable number of components of interest. Fourth, interpretation uses a metric D ( · , · ) . The geometry, however, in general could be nonisotropic and treating each dimension equally could be sub-optimal. We leave the exploration of this to future work.

5 Conclusion

We present InversionView, an effective method for decoding information from neural activations. In four case studies-character counting, IOI, 3-digit addition, and factual recall-we showcase how it can reveal various types of information, thus facilitating reverse-engineering of algorithm implemented by neural networks. Moreover, we compare it with other interpretability methods and show its unique advantages. We also show that the results given by InversionView can in principle be interpreted automatically by LLMs, which opens up possibilities for a more automated workflow. This paper only explores a fraction of the opportunities this method offers. Future work could apply it to subspaces of residual stream, to larger models, or to different modalities such as vision.

References

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 , 2016.

Anthropic. Introducing the next generation of claude, 2024. https://www.anthropic.com/ news/claude-3-family .

Y. Belinkov. Probing classifiers: Promises, shortcomings, and advances. Comput. Linguistics , 48(1):207-219, 2022. doi: 10.1162/COLI\_A\_00422. URL https://doi.org/10.1162/ coli_a_00422 .

Y. Belinkov and J. Glass. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics , 7:49-72, 2019.

Y. Belinkov, N. Durrani, F. Dalvi, H. Sajjad, and J. Glass. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 861-872, 2017.

N. Belrose, Z. Furman, L. Smith, D. Halawi, I. Ostrovsky, L. McKinney, S. Biderman, and J. Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2023.

S. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders. Language models can explain neurons in language models. URL https://openaipublic. blob. core. windows. net/neuron-explainer/paper/index. html.(Date accessed: 14.05. 2023) , 2023.

S. Bird and E. Loper. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions , pages 214-217, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/P04-3031 .

T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread , page 2, 2023.

B. Chughtai, A. Cooney, and N. Nanda. Summing up the facts: Additive mechanisms behind factual recall in llms. arXiv preprint arXiv:2402.07321 , 2024.

K. Clark, U. Khandelwal, O. Levy, and C. D. Manning. What does bert look at? an analysis of bert's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 276-286, 2019.

A. Conmy, A. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems , 36:16318-16352, 2023.

H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600 , 2023.

J. Dao, Y.-T. Lao, C. Rager, and J. Janiak. An adversarial example for direct logit attribution: Memory management in gelu-4l. arXiv preprint arXiv:2310.07325 , 2023.

G. Dar, M. Geva, A. Gupta, and J. Berant. Analyzing transformers in embedding space. In The 61st Annual Meeting Of The Association For Computational Linguistics , 2023.

N. Elhage, N. Nanda, C. O. T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. A mathematical framework for transformer circuits, 2021. https: //transformer-circuits.pub/2021/framework/index.html .

J. Ferrando and E. Voita. Information flow routes: Automatically interpreting language models at scale, 2024.

J. Ferrando, G. I. Gállego, I. Tsiamas, and M. R. Costa-jussà. Explaining how transformers use context to build predictions. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 5486-5513, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.301. URL https: //aclanthology.org/2023.acl-long.301 .

J. Ferrando, G. Sarti, A. Bisazza, and M. R. Costa-jussà. A primer on the inner workings of transformer-based language models. arXiv preprint arXiv:2405.00208 , 2024.

A. Geiger, H. Lu, T. Icard, and C. Potts. Causal abstractions of neural networks. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 9574-9586, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 4f5c422f4d49a5a807eda27434231040-Abstract.html .

A. Geiger, C. Potts, and T. Icard. Causal abstraction for faithful model interpretation. arXiv preprint arXiv:2301.04709 , 2023.

M. Geva, A. Caciularu, K. Wang, and Y. Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 30-45, 2022.

M. Geva, J. Bastings, K. Filippova, and A. Globerson. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 12216-12235, 2023.

A. Ghandeharioun, A. Caciularu, A. Pearce, L. Dixon, and M. Geva. Patchscope: A unifying framework for inspecting hidden representations of language models. arXiv preprint arXiv:2401.06102 , 2024.

N. Goldowsky-Dill, C. MacLeod, L. Sato, and A. Arora. Localizing model behavior with path patching. arXiv preprint arXiv:2304.05969 , 2023.

I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 , 2014.

M. Hanna, O. Liu, and A. Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems , 36, 2024.

M. Hanna, S. Pezzelle, and Y. Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. arXiv preprint arXiv:2403.17806 , 2024.

J. Kaddour. The minipile challenge for data-efficient language models. arXiv preprint arXiv:2304.08442 , 2023.

S. Katz and Y. Belinkov. Visit: Visualizing and interpreting the semantic information flow of transformers. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 14094-14113, 2023.

G. Kobayashi, T. Kuribayashi, S. Yokoi, and K. Inui. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 7057-7075, 2020.

B. Z. Li, M. Nye, and J. Andreas. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 1813-1827, 2021.

K. Li, A. K. Hopkins, D. Bau, F. Viégas, H. Pfister, and M. Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations , 2022.

T. Lieberum, M. Rahtz, J. Kramár, G. Irving, R. Shah, and V. Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. arXiv preprint arXiv:2307.09458 , 2023.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.

K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems , 35:17359-17372, 2022.

P. Michel, O. Levy, and G. Neubig. Are sixteen heads really better than one? Advances in neural information processing systems , 32, 2019.

N. Nanda. Attribution patching: Activation patching at industrial scale, 2023. URL https: //www.neelnanda.io/mechanistic-interpretability/attribution-patching .

N. Nanda and J. Bloom. Transformerlens. https://github.com/TransformerLensOrg/ TransformerLens , 2022.

A. Nguyen, J. Yosinski, and J. Clune. Understanding neural networks via feature visualization: A survey. Explainable AI: interpreting, explaining and visualizing deep learning , pages 55-76, 2019.

nostalgebraist. interpreting gpt: the logit lens. LESSWRONG , 2020. https://www.lesswrong. com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens .

C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. Distill , 2(11):e7, 2017.

K. Pal, J. Sun, A. Yuan, B. C. Wallace, and D. Bau. Future lens: Anticipating subsequent tokens from a single hidden state. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL) , pages 548-560, 2023.

M. Pawelczyk, C. Agarwal, S. Joshi, S. Upadhyay, and H. Lakkaraju. Exploring counterfactual explanations through the lens of adversarial examples: A theoretical and empirical analysis. In International Conference on Artificial Intelligence and Statistics , pages 4574-4594. PMLR, 2022.

X. Qi, K. Huang, A. Panda, P. Henderson, M. Wang, and P. Mittal. Visual adversarial examples jailbreak aligned large language models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 21527-21536, 2024.

P. Quirke and F. Barez. Understanding addition in transformers. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum? id=rIx1YXVWZb .

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

A. Stolfo, Y. Belinkov, and M. Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 7035-7052, Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10. 18653/v1/2023.emnlp-main.435. URL https://aclanthology.org/2023.emnlp-main. 435 .

A. Syed, C. Rager, and A. Conmy. Attribution patching outperforms automated circuit discovery. arXiv preprint arXiv:2310.10348 , 2023.

A. Tamkin, M. Taufeeque, and N. D. Goodman. Codebook features: Sparse and discrete interpretability for neural networks. arXiv preprint arXiv:2310.17230 , 2023.

I. Tenney, D. Das, and E. Pavlick. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4593-4601, 2019.

J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, Y. Singer, and S. Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems , 33:12388-12401, 2020.

K. R. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. 2023. URL https: //openreview.net/forum?id=NpsVSN6o4ul .

Z. Wang, A. Ku, J. M. Baldridge, T. L. Griffiths, and B. Kim. Gaussian process probes (gpp) for uncertainty-aware probing. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

J. Wiland, M. Ploner, and A. Akbik. BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models, 2024. URL http://arxiv.org/abs/ 2404.04113 .

Z. Wu, A. Geiger, T. Icard, C. Potts, and N. Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. Advances in Neural Information Processing Systems , 36, 2024.

W. Xia, Y. Zhang, Y. Yang, J.-H. Xue, B. Zhou, and M.-H. Yang. Gan inversion: A survey. IEEE transactions on pattern analysis and machine intelligence , 45(3):3121-3138, 2022.

C. Xiao, B. Li, J.-Y. Zhu, W. He, M. Liu, and D. Song. Generating adversarial examples with adversarial networks. arXiv preprint arXiv:1801.02610 , 2018.

Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems , 32, 2019.

Y. Yao, N. Zhang, Z. Xi, M. Wang, Z. Xu, S. Deng, and H. Chen. Knowledge circuits in pretrained transformers. arXiv preprint arXiv:2405.17969 , 2024.

Q. Yu, J. Merullo, and E. Pavlick. Characterizing mechanisms for factual recall in language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 9924-9959, 2023.

Contents

A Practical Guidelines

A.1 Observing Larger Neighborhoods is Important

Here, we illustrate the importance of inspecting ϵ -preimages up to the threshold ϵ , rather than just topk nearest neighbors of the query activation. In Figure 8, an initial glance at the samples on the left may suggest that the residual stream of '+' encodes F1 and F2. However, observing a broader neighborhood (as depicted on the right) reveals that this conclusion is not even robust to tiny perturbations of the activation. Indeed, after a more comprehensive calculation over all possible x 0 , post + , we find that the maximum possible metric value between any pair of x 0 , post + is 0.0184. So for any ϵ ≥ 0 . 0184 the ϵ -preimage covers the entire input space. Hence, the activation is unlikely to contain usable information.

We further prove this by causal intervention. We found that x 0 , post + has no effect on the model's output. Concretely we patch x 0 , post + with its mean on the test set (mean ablation [54]) and for each prediction target (A1, A2 etc.), we compare 1) the KL divergence between the distribution before and after patching. 2) logit decrement rate, which is the difference between the maximum logit value before patching and the logit value of the same target token after patching, divided by the former. E.g., 1.0 means the logit is reduced to zero (assuming it is originally positive). The results are shown in Table 1. We can see the effect of x 0 , post + is negligible.

A.2 Sampling with Decoder Model

In Section 2, we mentioned that the distribution p ( x | z q ) is modeled by the decoder. Strictly speaking, p ( x | z q ) represents the data distribution in the ϵ -preimage defined by ϵ = 0 . For example, when the

probed model is using causal masking, and a certain activation is relevant to all previous context (by non-zero attention weights), then p ( x | z q ) is the distribution over those inputs that share the same previous context (i.e., they have same prefix). This requires that the decoder can distinguish any tiny difference in activation and decode the full information (imagine a token attended with 0.0001 attention weight). Such a decoder must be very powerful and perhaps trained without any regularization. But in practice, the decoder is a continuous function of activation and tiny changes in activation are not perceivable by the decoder. We observe that the decoder rarely generates the sample that lies at the same point (producing the same activation) as the query input in vector space, instead it usually generates samples that are in the neighborhood of the query input. Because we need to observe the whole neighborhood of the query input and prevent samples from being too concentrated, we adjust the sampling temperature to control how concentrated they are. Importantly, even if the decoder is too powerful and can always recover the same activation, we can still obtain the neighborhood by adding random noise to the query activation before giving it to the decoder. This motivates decoding with temperature and noise, as described in the next paragraph.

Increasing Coverage by Temperature and Noise. In our experiments, we use both ways to control the generation, i.e., by adjusting the temperature and adding random noise to the query activation. We denote temperature as τ and noise coefficient as η . The noise vector consists of independent random variables sampled from the standard normal distribution and then multiplied by std ( z q ) · η where std ( · ) stands for standard deviation. In our web application, we provide multiple sampling configurations: four configurations in which τ = { 0 . 5 , 1 . 0 , 2 . 0 , 4 . 0 } and η = 0 . 0 (only for addition task); one figuration named 'Auto" which is sampled by following procedure: we iterate over a few predefined τ (ranging from 0.5 to 2.0) and η (0.0 or 0.1) and sample a certain amount of inputs (e.g., 250) for each parameter combination. We then calculate the metric value for all inputs collected from different sampling configurations. We then randomly choose a small part of them (100) with different probability for inϵ -preimage inputs and out-ofϵ -preimage inputs. We dynamically adjust the probability such that the inϵ -preimage inputs account for 60%-80% of the chosen set of inputs (when this is possible). Note that we use different noise in factual recall task, which will be described later in Appendix H.4.

When inspecting the samples, we choose a configuration for which the distances D ( · , · ) to the query activation best cover the interval [0 , ϵ ] . The choice is usually specific to the activation site that we are inspecting and can be performed manually in the web application.

A.3 Selecting Position in Samples

As the decoder outputs an input but not the position of the activation, we then assign the position minimizing D ( · , · ) to the query activation. Usually, there is only one position with a small D ( · , · ) , matching the structural position (not necessarily the absolute position) of the position the query activation was taken from (e.g., the target character in Figure 3a). In certain cases, we visualize D ( · , · ) for an activation from a position not minimizing D ( · , · ) for expository purposes. For example, in Figure 3b, because the target character exclusively attends to itself in layer 1, resulting a 1 , 0 tc ≈ a 1 , 0 : , so sometimes the metric value of a 1 , 0 tc is smaller than a 1 , 0 : . Throughout the appendix and our web application, we use italic font and rounded bars to visualize D ( · , · ) in such cases.

We also find that selecting the position minimizing D ( · , · ) can reveal that components are active in similar ways at other positions than the one originally investigated. For example, in Figure 9, we can see sometimes 'gave' is selected. This is reasonable, because the IO is also likely to appear right after 'gave' and the head needs to move the IO name for this prediction. We can see that activation at the period '.' can also be somewhat similar to the query activation, this is not surprising. Because the model needs to predict the subject for the next sentence and copying a name from the previous context is helpful. In summary, the copying mechanism can be triggered in circumstances different from IOI, selecting position minimizing D ( · , · ) reveals more information about this.

A.4 Threshold-Dependence of Claims about Activations

One question people may have is whether our conclusion about the information in activation depends significantly on the threshold we choose. To address this potential concern, we show more details about the geometry of the vector space in Figure 10. On the one hand, we can see that with different thresholds ϵ we can make different conclusions about the query activation. On the other hand, the conclusions made with different thresholds are 'in alignment'. In other words, the conclusions do not differ fundamentally, instead, the difference between them is about granularity or the amount of details being ignored.

Specifically, in Figure 10a, ϵ 1 results in the conclusion that the count is 5, the target character is either 't' or 'm', and also approximate sequence length is retained. ϵ 2 results in a conclusion only about the count and the sequence length. In Figure 10b, if we set the threshold to ϵ 1 (i.e., a value between 0.000 and 0.009), the obtained information will be F1=5, S1=7. If we set the threshold to ϵ 2 , the information will be 5 and 7 are in the hundreds place. If we set the threshold to ϵ 3 the information will be "5 is in the hundreds place". In Figure 10c, ϵ 1 results in conclusion that 9, 8 are in hundreds place and 2, 7 are in tens place; ϵ 2 results in conclusion that 9, 8 are in hundreds place and F2+S2=9; ϵ 3 results in conclusion that F1+S1 ≈ 17 and F2+S2 ≈ 9. Therefore, changing the threshold value will not lead us in a different direction, because the ϵ -preimage is based on the same underlying geometry.

In practice, rather than selecting a threshold first and treating inputs in a black-and-white manner, we first observe the geometry of the vector space and obtain a broad understanding of the encoded information, then choose a reasonable threshold that best summarizes our findings. In other words, the threshold value is used to simplify our findings so that we can focus more on the big picture of the model's overall algorithm, and it should also be set according to the difference that is likely to be readable for the model. As the interpretation progresses, one can see if the chosen threshold leads to a plausible algorithm and can adjust it if necessary. Finally, verification experiments are conducted to verify the hypothesis.

B Experimental Verification of Completeness

In Section 2, we described that an ideal strategy for obtaining samples in the ϵ -preimage satisfies two desiderata: it only provides samples that are indeed within the ϵ -preimage (Correctness), and it provides all such samples at reasonable probability (Completeness). As further described there, we can directly ensure Correctness by evaluating D ( · ) for every sample. Ensuring completeness is more challenging, due to the exponential size of the input space; the most general approach is to design counterexamples not satisfying a hypothesis about the content of the ϵ -preimage, and verifying that D ( · ) is indeed large.

Here, we provide a direct test of completeness in one domain (3-digit addition). The primary concern with completeness is that, if some groups of inputs in B z q ,f,ϵ are systematically missing from the generated samples, one may overestimate the information contained in activations. To see if may happen in reality, we plotted log-probability against distance, each of which includes all inputs in 3 digit addition task, as shown in Figure 11. We next evaluated the sampling probability for different sampling configurations, as described in Appendix A.2. When adding noise, we calculate probability of an input using a Monte Carlo estimate: Concretely, because the probability of inputs is conditioned on the noise vector added to the query activation, we randomly sample 500 noise vectors from the normal distribution (with the standard deviation described in Appendix A.2) and calculate input probability given these noise vectors, then average to obtain the estimated probability, and then compute the logarithm.

Across setups, we can see that there is a triangular blank area in the bottom left corner, i.e., the bottom left frontier stretches from the upper left towards the lower right. In all sub-figures, not a single input close to the query input is assigned disproportionately low probability. All inputs in the ϵ -preimage (the dots on the left of the red vertical line) are reasonably likely to be sampled from the decoder, with probability decreasing as the input becomes more distant, alleviating concerns about completeness for these query activations. On the other hand, some inputs distant from the query input are also likely to appear in samples, but this is not a problem for our approach, as we can easily tell that they are not in the ϵ -preimage by calculating the distance (correctness is ensured).

We can see that sometimes the distribution of the decoder itself (at temperature 1 and no noise) is quite sharp, and inϵ -preimage inputs can have low probability as they are near the boundary of preimage. By comparing the sub-figures, we can see both increasing the temperature and adding noise substantially smooth the distribution within the ϵ -preimage, lowering the difference of the probability of inputs that are at similar distance.

C Decoder Model

The overall training and sampling pipelines are shown in Figure 2. In this section, we describe the architecture of the decoder model in detail.

The decoder model is basically a decoder-only transformer combined with some additional MLP layers. In order to condition the decoder on the query activation, the query activation is first passed through a stack of MLP layers to decode information depending on the activation sites and then made available to each attention layer of the transformer part of the decoder, as depicted in Figure 12.

Processing Query Activation. The query activation z q ∈ R d is first concatenated with a trainable activation site embedding e act ∈ R d site , producing the intermediate representation z (0) = [ z q ; e act ] . We chose d site to be the number of possible activation sites in the training set. The result z (0) is

(a)

(b)

(c)

a0.0 B282+717=99(9)EP

a0.2 B876+225=(1)101E

a0.3 B188+558=74(6)EP

x0.post B614+940=(1)554E

0

20

-20

25

20

log Probability

-50

-40

40

40

75

0.00

00.250.500.751.001.25

0.000.25

0.500.751.00

0.0

0.5

1.0

0.0

0.5

1.0

1.5

a1.0 B752+284=10(3)6E

a1.1 B406+534=9(4)0EP

a1.2 B957+230=(1)187E

x1.mid B568+377(=)945EP

0

0

-20

20

25

-50

-40

40

-50

-60

75

-100 -

09-

0.00

0.25

0.50

0.75

1.00

0.0

0.5

1.0

1.5

0.0

0.1

0.2

0.3

0.4

0.0

0.5

1.0

1.5

Distance

a0.0 B282+717=99(9)EP

a0.2 B876+225=(1)101E

a0.3 B188+558=74(6)EP

x0.post B614+940=(1)554E

0

log Probability

-10

-10

=

-20

FOT-

20

-20

20

40

oe-

0.000.250.500.751.001.25

0.000.25

0.50

0.751.00

0.0

0.5

1.0

0.0

0.5

1.0

1.5

a1.0 B752+284=10(3)6E

a1.1 B406+534=9(4)0EP

a1.2 B957+230=(1)187E

x1.mid B568+377(=)945EP

0

0

0

10

20

20

-20

20

-40-

OE-

40

40

-60

0.00

0.25

0.50

0.75

1.00

0.0

0.5

1.0

1.5

0.0

0.1

0.2

0.3

0.4

0.0

0.5

1.0

1.5

Distance

a0.0 B282+717=99(9)EP

a0.2 B876+225=(1)101E

a0.3 B188+558=74(6)EP

x0.post B614+940=(1)554E

log Probability

-20

-20

-10

20

40

40

20

40

0.00

0.25

0.500.751.001.25

0.00

0.25

0.50

0.75

1.00

0.0

0.5

1.0

0.0

0.5

1.0

1.5

a1.0 B752+284=10(3)6E

a1.1 B406+534=9(4)0EP

a1.2 B957+230=(1)187E

x1,mid B568+377(=)945EP

0

0

25

-20

20

20

-50

40

40

75 -

-40

0.00

0.25

0.50

0.75

1.00

0.0

0.5

1.0

1.5

0.0

0.1

0.2

0.3

0.4

0.0

0.5

1.0

1.5

Distance

(u)z

LN

XBOS

ulx

MLP

LN

LN

LN

Linear

XBOS

XIn

MLP

LN

LN

LN

Linear

XBOS

XIn

[BOS]

In

the

then fed through multiple MLP layers (each layer indexed by p ∈ { 0 , 1 , · · · , P -1 } ) with residual connections:

where LN represents layer normalization. For each MLP layer, the input, hidden and output dimensions are d + d site , d , and d + d site , respectively. The activation function is ReLU. There is also a final layer normalization, z ( fn ) = LN ( z ( P ) ) .

Integrating Query Activation. As we want to make the query activation available to each attention layer of the decoder, we separately customize it to the needs of each layer using a linear layer. That is, for each layer of the transformer part of the decoder (indexed by ℓ , so ℓ ∈ { 0 , 1 , · · · , L -1 } ), we define a linear layer Linear ( ℓ ) : R d + d site → R d decoder and a layer normalization LN ( ℓ ) , where d decoder is the model dimension of the decoder model:

We add superscript ( ℓ ) to the model components to emphasize they are layer-specific. In the ℓ -th layer of the transformer part, ˆ z ( ℓ ) is concatenated with the input of the attention layer along the length axis before computing keys and values, so that each attention head can also attend to ˆ z ( ℓ ) in addition to the context tokens. This means that each head in the ℓ -th layer, instead of attending to x ( ℓ ) 1 , x ( ℓ ) 2 , ..., now computes its attention weights over ˆ z ( l ) , x ( ℓ ) 1 , x ( ℓ ) 2 , ... . Here x ( ℓ ) t is the residual stream corresponding to the t -th token input into the ℓ -th layer.

Motivation for Architecture Design. At the beginning, we train separate decoders for each activation site, but this is not very scalable when there are many activation sites. In the architecture above mentioned, we use activation site embedding e act as a signal to trigger different processing, and functions or model components that are needed for all activation sites are shared. Similar reason applies to the linear layer Linear ( ℓ ) , we expect the z q should be transformed differently for each transformer layer, but having separate MLP stacks to process z q for each layer would largely increase the number of parameters. In preliminary experiments, we also try 'encoder-decoder' attention layer. That is, instead of providing query activation in self attention layer, we add new attention layers that analogous to the encoder-decoder attention layer in original transformer architecture, where each token can attend to the processed query activation as well as a blank representation (similar to the function of 'BOS", so that 'no-op" is possible). However, we do not find significant difference between this design and the aforementioned one. Therefore, other than adding components for processing z q , we do not modify the decoder-only transformer architecture, so that we can also choose to use pretrained models. We note that there are other possible choices for conditioning the generation on the activation, and we didn't optimize this choice thoroughly.

Decoder Hyperparameters. Regarding processing query activation, the decoder has 6 MLP layers, i.e., P = 6 . The decoder model has 2 transformer layers ( L = 2 ), 4 heads per layer, and a model dimension of 256. The attention dropout rate is 0. Other settings are the same as the GPT-2. We use the this architecture for the 3 tasks-character counting, IOI, and 3-digit addition- in the paper. Regarding the factual recall task, we use the same architecture for processing query activation, i.e. P = 6 , and use pretrained GPT-2 Medium (24 layers) as the transformer part of the decoder.

Training Details. We construct the training dataset by feeding in-domain inputs to the probed model, and collect activations from random activation sites and random position as query activations (the choice of activation sites and position is specific to each task and is described later), we also record the activation site they come from. For each input, we could obtain many possible training examples because of many choices of activation sites and position. So we do not iterate over all possible training examples. We sample certain amount of examples to train the decoder for 1 epoch, using constant learning rate of 0.0001 and AdamW optimizer with weight decay of 0.01. Other details (e.g., amount of examples, training steps) are task-specific, and can be found later in their own section.

During training, we regularly calculate the in-preimage rate, which serves as a proxy for generation quality. Concretely, for a fixed set of query activations used for testing, the decoder generates samples with temperature=1, we then compute the fraction of samples inside of ϵ -preimage (with ϵ = 0 . 1 , D as normalized Euclidean distance). The rate is calculated for each activation site. We usually observe difference between the ratios for each activation site, indicating some inverse mappings are easier to learn (we also observe these activation sites tend to have clearer information). Overall, we usually see a continuous improvement on the average rate during training.

D Character Counting: More Details and Examples

D.1 Implementation Details

To construct the dataset, we enumerate all the 3-combinations from the set of lowercase characters defined in ASCII. For each combination, we generate 600 distinct data points by varying the occurrence of each character and the order of the string. The occurrences are sampled uniformly from 1-9 (both inclusive). So the length of the part before the pipe symbol ('|') lies in [3, 27] (not considering 'B'). Like 3-digit addition, we split the dataset into train and test sets, which account for 75% and 25% of all data respectively. The input is tokenized on the character level.

The model is a two-layer transformer with one head in each layer, the model dimension is 64. All dropout rates are set to 0. The model is trained with cross-entropy loss on the last token, the answer of the counting task. The model is trained with a batch size of 128 for 100 epochs, using a constant learning rate of 0.0005, weight decay of 0.01, and AdamW [36] optimizer. The training loss is shown in Figure 13, we can see the stair-like pattern. An interesting future direction is to investigate what happens when the loss rapidly decreases using InversionView.

With regard to the decoder model, the architecture is described in C. We select x 0 , pre , x i, mid , x i, post , a i,j , m i as the set of activation sites we are interested in, where i ∈ { 0 , 1 } , j ∈ { 0 } , and m denotes MLP layer output. The query activation is sampled from those activations corresponding to only the target character and colon. We sample 100 million training examples (all activation sites are included) and train the decoder with batch size of 512, resulting in roughly 200K steps. During training, as we mentioned before in C, we test the generation quality by measuring in-preimage rate. For those activation sites for which the decoder has a low generation quality, we increase their probability of being sampled in the training data. The final in-preimage rate averaged across activation sites is 67.7%.

D.2 More Examples of InversionView

See Figures 14 and 15. Here, we show results similar to Figure 3 for other query inputs.

2.00

1.75

1.50

1.25

SSOI

1.00

0.75

0.50

0.25

0.00

0

20

40

60

80

100

Activation Site: α1,0

Activation Site: a1,0

BwwnncwnnncwnncwncwI(:)4

Buooojojjooolu(:) 1

QueryInput

QueryInput

0.0oo;Bwwnncwnnncwnncwncw|c(:)4

0.000;Buooojojjooo|u(:)1

GeneratedSamples

GeneratedSamples

I0.021;#:4;Bcccciiiicviivccvivi|v(:)4E

00.08l;#:l;Bwwwwwwwwoeo|e(:)1E

I0.044;#:4;Bccrwgwrwwwcrwrrgcwtrrrw|c(:)5E

0.055;#:1;Bxtttktknxxx|n(:)1 E

I0.048;#:4;Bcjajjcjcjajcccaa|a(:)4E

●0.069;#:l;Biｎiｎ pｎiiｉｎi| p()１E

·0.057;#:4;Bqjqｅqjqｎjjqｎjjnqjｎ|ｎ(:)4E

00.075;#:1;Bxxxdxxxxxd a|a(:)1 E

·0.057;#:4;Bcjjccnnccjcjcjjnjjnc|n(:)4E

00.072;#:1; B d xxxxxxxx d s|s(:)1 E

●0.090;#:4;Bjjcjjcccjrcccjjcrjrr|r(:)4E

● 0.111;#:1; B p d xxrx d xxxx pxx d|r (:) 1 E

●0.097;#:4;Bcccccjjjjjccccrrrjjjr|r(:)4E

0.323;#:0; Bv k m kk m k m k k|n (:)1 E

●0.140;#:5;Bttootwwwtwoothhtojt|o(:)4E

●0.114;#:1; B p× p× px×xxx n|n (:) 1 E

0.250;#:6;Bbabdbdaddabdaadbdddabb|a(:)5E

0.263;#:2;BInIsIIwIIn|n(:)2 E

0.298;#:2;Beevsnevevesvvev|s(:)3E

0.430;#:3;BIm mvxImwI1|m (:)3E

Increment of (LDorig -LDpch)

LDorig

Patching Effect-->

40

Increment

30

20

20

9

0

10

-20

0

none

a0.0

al.0

Patching Effect<--

40

Increment

30

20

20

D

0

10

-20

0

a0.0

al.0

none

PatchedActivation

D.3 Causal Intervention Details

For an input example x orig with t orig as the count (final token), we construct a contrast example x con with a different count t con by changing a random character before '|'. The contrast example is a valid input (the count token is the count of the target character). We also ensure that the contrast example is within the dataset distribution (the count is in the range [1-9] and there are 3 distinct characters in the input).

We run three forward passes. 1) The model takes as input x orig and produces logit values for count prediction, we record the logit difference LD orig between t orig and t con (former minus latter). 2) We feed the model with x con and store all activations. 3) We run a forward pass using x orig , replacing the interested activations (e.g, { a 0 , 0 : , a 0 , 0 tc } ) with the stored activation in the same position and activation sites, and record the new logit difference LD pch . Because the model can make the right prediction in most cases, we can see that average LD pch changes from positive to negative values as we patch more and more activations. We do the same for all inputs in the test set and report the average results.

Figure 16 shows the LD orig and LD pch . We cumulatively patch the activations we study. For example, on the top of the figure, we patch { a 0 , 0 : } , { a 0 , 0 : , a 0 , 0 tc } , { a 0 , 0 : , a 0 , 0 tc , a 1 , 0 : } respectively. Patching more activation results in increases of LD orig -LD pch , we attribute the increment to the newly patched activation. Hence, the causal effect of each activation is measured conditioned on some activations already being patched.

We sort the activations according to their layer indices and show the results of patching from bottom to top ( → ) and from top to bottom ( ← ). In this way, we can verify the dependence between activations in the top and bottom layer. For example, at the top of Figure 16, when a 0 , 0 tc is already patched, patching a 1 , 0 : has almost no effect. On the bottom, we also see patching a 0 , 0 tc has no effect if a 1 , 0 : has been patched. So we verified that a 0 , 0 tc is the only upstream activation that a 1 , 0 : relies on, and a 1 , 0 : is the only downstream activation that reads a 0 , 0 tc .

Patching Effect

Patching Effect

40

-->

40

Increment

30

Increment

30

20

20

10

10

0

0

a0.0

a1.0

0.0

a?.0

al.0

eten

s

eten

D.4 Extended Algorithm with Positional Cues

In Section 3.1, we verified the information flow by an activation patching experiment in which the contrast example only differs by one character. These experiments verified that the algorithm we described is complete in distinguishing between such minimally different contrast examples. We now show that the model implements a somewhat more complex algorithm that combines this algorithm with position-based cues, which become visible once we consider contrast example that differ in more than one character, in particular, those that differ in length.

To show this, we conduct another activation patching experiment in which the contrast example is a random example in the dataset with a different count. In other words, everything can be different in contrast examples, including the sequence length and the target character. Thus, we cumulatively patch four places:

e pos tc and e pos : , where e pos stands for position embedding, because the final count correlates with positional signal, so the model may utilize it. They are patched together because the attention pattern of the colon in layer 1 relies on their adjacency.

a 0 , 0 tc (as before) and e tkn tc , where e tkn stands for token embedding. They are patched together because patching only one of them would result in a conflict between character information in the patched and the un-patched activation;

3) a 0 , 0 : ; (as before)

a 1 , 0 : (as before).

The result is shown on the left of Figure 17. We can see that, besides the components we had detected previously based on minimal contrast examples ( a 0 , 0 tc , a 1 , 0 : ), some other signal also contributes notably to the final logits. We compare with patching experiments for the same set of activations on contrast examples that differ in one character, shown on the right of Figure 17.

Overall, besides the algorithm identified in Section 3.1, we find other 3 sources of information influencing the model's output. 1) The position embedding, e pos tc and e pos : . This is observable on the left of Figure 17, from which we can also know a 1 , 0 : contains the position information (because the bars of e pos tc and e pos : are not symmetric). This is confirmed by InversionView. As shown in Figure 18, we see the inputs in ϵ -preimage roughly follow the query input length, being independent of the count. Therefore, the model is also utilizing the correlation between the count and the sequence length. 2) Attention output of colon, a 0 , 0 : , which attends to all previous token equally. From InversionView, we observe it contains fuzzy information about the length (same as position signal), and the characters that occur in the context, as well as their approximate count. Our causal experiment also shows that

Activation Site: α1,0

Activation Site: α1,0

BuuuuubbmuubmuubIm(:) 2

BtttttnnnttntxtIt(:)9

QueryInput

QueryInput

0.000;Btttttnnnttntxt|t(:)9

Generated Samples

Generated Samples

I0.038;#:2;Bkekkkekeeeeeezzkkk|z(:)2E

I 0.022;#:9;Bttttttttt s zzzz|t (:) 9 E

I0.058;#:2;Babaabhbabaaabahbb|h(:)2E

I0.025;#:9;BttmttIttm mtmtt|t(:)9E

0.059;#:2;Buujuuuaauujjujjju|a (:)2 E

I0.028;#:9;Btttttftfttffft|t(:)9E

·0.069;#:2;Bajhajjjｈaaajajjaaj|h(:)２E

I0.034;#:9;Btttttttthrrrrt|t(:) 9E

●0.071;#:2;Bvnlvvnlvavbvvv|n(:)3E

·0.063;#:9;Bviccvjjccccvcvcc|c(:)9E

●0.0８6;#:２;Bｂｂｂａｎａａａａｎbａａbａａｂb|ｎ(:)２E

●0.100;#:8;Btosstoctcttlttt|t(:)9E

●0.092;#:2;Bmjmamjmmmjjmmjam|a(:)2 E

●0.111;#:8;Bttttttptptpppe|t(:)9E

●0.126;#:2;Bebaeeebeaeeebe|a(:)2E

0.159;#:8;Brmcmcrcrcrrrrrc|r(:)9E

0.230;#:3;BsSpokossskzksksksko|o(:)2E

●0.170;#:８;Bｐfaqqqｐfqfqqdqq|q(:)9E

0.617;#:7;Bitvvttvtvvvititjvvtv|t(:)3E

0.458;#:0;Bm nvnbvvbBbbvbbvmmbvv|q(:)7E

it does not contain a precise count. Therefore, it contributes to the model's prediction in manner similar to the position signal. 3) Attention output of the pipe sign, a 0 , 0 | . From the attention pattern we observe sometimes in layer 0, pipe sign attends selectively to one type of character, e.g. 'x', 'k', or 'j'. InversionView shows that it indeed contains the approximate count in that case (though the decoder has not been trained on activation corresponding to pipe sign). In next layer, the colon also attends to the pipe sign if target character is the same as the character attended by pipe sign in layer 0. This explains why we can observe nonzero effect of patching a 1 , 0 : when other activation is already patched (the red bar corresponding to a 1 , 0 : on both sub-figures of Figure 17).

Whereas patching with minimally different contrast examples allowed us to extract an algorithm sufficient for solving the task in Section 3.1, patching with arbitrarily different contrast examples allowed us to uncover that the model combines this algorithm with position-based cues. The model performs precise counting using the algorithm we found earlier in Section 3.1, while it also makes use of simple mechanisms such as correlation to obtain a coarse-grained distribution over counts. Overall, we have found the full algorithm by alternating between different methods - InversionView, traditional inspection of attention patterns, and causal interventions, and confirming results from one with others.

E IOI Task: Details and Qualitative Results

E.1 Implementation Details

In order to train a decoder model, we construct a dataset that consists of IOI examples. We used the templates of IOI examples from ACDC [13] implementation. For example, 'Then, [B] and [A] went to the [PLACE]. [B] gave a [OBJECT] to [A]', in which '[B]' and '[A]' will be replaced by two random names (one token name), '[PLACE]' and '[OBJECT]' will also be replaced by random item from the predefined set. Besides 'BABA' template (i.e., S is before IO) we also use 'ABBA' templates (S is after IO) by swapping the first '[B]' and '[A]'. We generate 250k data points.

The architecture of the decoder model is the same as before, as described in Appendix C. The set of activation sites the decoder is trained on consists of the output of all attention heads and MLP layers (no residual stream). Note that when producing query activation using GPT-2, we always add the ' < | endoftext | > ' token as the BOS token. We do so as during the training of GPT-2 this or multiple such tokens that usually appear in the previous context can be used as a BOS token, which is possibly important to the model's functioning. We use a new token '[EOS]' as the EOS token when training the decoder. The query activation is sampled uniformly from all positions excluding EOS and padding tokens, and uniformly from all activation sites the decoder is trained for. We sample 20 million training examples (all activation sites are included) and train the decoder with batch size of

Figure 19: ϵ -preimage of S-Inhibition Head 7.3. The relative position of S1-but not its identity-is contained in the head output (together with some template information). That means, in the samples within the ϵ -preimage, S1 always appears before the IO. While the relative position is encoded, the absolute position can vary, as can the identities of the names.


Friendsseparated atbirthBradley and Codyfound acomputeratthestore.(Cody)gaveittoBradley

QueryInput

0.000;<lendoftext|>Friends separated at birth Bradley and Codyfound a computerat thestore.(Cody)gaveit to Bradley

GeneratedSamples

0.0o0;
Theninthemorningafterwards,Cody andRyanhad alongargument.Afterwards(Cody)saidtoRyan[EoS]

0.000;<lendoftext|>Aftertakinga long break Cody and Lauren went to the station,(Cody)gave a computerto Lauren[EOS]

0.001;
Afterwards,Codyand Danielle wentto therestaurant.(Cody)gave abasketballtoDanielle[EOS]

0.001;<lendoftext|>Then,Paul and Cody had along argument.Afterwards(Cody)said toPaul[EOS]

0.392;
Then in the morning afterwards,Courtney andDustin had a long argument.Afterwards(Dustin) said to Courtney[EOS]

0.392;
Friendsseparated atbirthAliciaand Dustinfound asnackat the school.(Dustin)gaveit toAlicia[EOS]

0.395;
When soon afterwardsJason andNicholasgotasnack at thehospital,Nicholas decided togivethe snackto(Jason)[EoS]

256, resulting in roughly 80K steps. The final average in-preimage rate is 58.0%, despite that the decoder is trained for 157 activation sites.

E.2 More Examples of InversionView

See Figures 19, 20, 21.

E.3 Qualitative Examination Results

The qualitative examination results is shown in Table 2. We summarize the description in [54] of each head category to facilitate comparison. Figure 22 shows the IOI circuit in GPT-2 small, which is taken from their paper. For more details, please refer to [54].

There are some heads for which InversionView indicates a different interpretation. First, Head 0.1 and 0.10: [54] only shows that they usually attend to the previous occurrence of a duplicate token and validates the attention pattern on different datasets. However, there is no evidence for the information moved by these heads. Thus they only hypothesize that the position of previous occurrence is copied. Second, Head 5.9: The path patching experiments in [54] show that head 5.9 influences the final logits notably via S-Inhibition Heads' keys. But there are no further experiments to explain the concrete function of this head. While the authors refer to it as a Fuzzy Induction Head, the induction score (measured by the attention weight from a token T to the token after T 's last occurrence) of this head shows a very weak induction pattern. Even if such pattern occurs, it cannot tell us what


Aftertakinga longbreakBradley and Rebeccawentto thestore,(Bradley)gaveabasketballtoRebecca

QueryInput

GeneratedSamples

0.002;
Aftertakinga long breakBradley and Gregory went to the station,(Bradley)gave a computerto Gregory[EOS]

0.014;
After taking a longbreakKelly and Aaron went to the restaurant,(Kelly)gave a bone to Aaron[EOS]

0.015;<|endoftext>After taking a long break Kelly and Aaron went to the hospital,(Kelly)gave a necklace to Aaron[EOS]

|0.019;
Aftertaking a longbreak Kyle and Paul went to the hospital,( Kyle)gave a snacktoPaul[EOS]

0.020;
After taking a long break Travis and Andrea went to the restaurant,Andrea gave a kiss to(Travis)[EOS]

0.022;<|endoftextl>After taking a long break Megan and Benjamin went to the restaurant, Benjamin gave a ring to( Megan)[EOS]

10.024;<|endoftext>After taking a long break Lauren and Joshua went to the garden,Joshua gave a necklace to(Lauren)[EOS]

10.028;
Aftertaking a long break Michelle and Amanda went to the restaurant,(Michelle)gave a kisstoAmanda[EOS]

10.032;
While spending time together Dustin and Sarah were working at the station,Sarah gave a snackto(Dustin)[EOS]

0.370;
After takinga long break Bradleyand Andrea went to therestaurant,Patrick gave a computerto(Andrea)[EOS]

Legend

When

I0Mary

Key/Value

and

S1John

Query→

ClassofHeads

Layer.Head

S1+1went

PreviousTokenHeads

2.2 4.17

to

the

store

NegativeNameMoverHeads

S2John

DuplicateTokenHeads

InductionHeads

10.7 11.10

0.13.0 (0.10)

5.56.9 (5.8 5.9)

gave

Name MoverHeads

a

9.99.610.0

drink

ENDto

S-InhibitionHeads

BackupNameMoverHeads

7.3 7.98.68.10

9.0 9.7 10.1 10.2 10.6 10.10 11.2 11.9

information is captured by this head. Interpretation with InversionView suggests that the head barely contains any information, within the input space of IOI-like patterns. One possibility is that head 5.9 recognizes the IOI pattern (i.e., there are two names and one is duplicated in the previous context), so that if an IOI-like pattern exists, S2 should be attended to by S-Inhibition heads. As the decoder model is trained on IOI examples and generates mostly IOI examples - that is, the input space X in (1) consists of IOI-like inputs, this information is by definition not visible. Expanding the input space to arbitrary language modeling would allow capturing such information; we leave this to future work.

E.4 Choice of Distance Metric in IOI

As described in Section 3.2, we used cosine distance for the IOI task, while for the other two tasks, we use normalized Euclidean distance. In this section, we show that both distance metrics produce similar interpretations while cosine distance makes the meaningful patterns easier to identify.

In Figure 23 and 24 we show two examples of using normalized Euclidean distance as the distance metric. Readers can see more examples on our web application. We see that the samples generated by the decoder tend to have larger distances under normalized Euclidean distance. Nonetheless, we can still see the top-ranked samples show meaningful commonalities - indeed if we set ϵ = 0 . 4 , we obtain the same interpretation as the one we obtain based on Figure 4b and 19.

While normalized Euclidean distance can produce a similar interpretation, we have to set a much larger ϵ , which we find less intuitive. We believe this to be because of the differences in the activation dimensionality, which is 768 in the IOI task, much larger than in character counting (64) and addition (32) tasks: Under Euclidean distance, the ratio of all close samples to all possible samples becomes lower and lower when the dimension becomes higher. In other words, the volume of the ϵ -preimage accounts for a very tiny proportion of the whole space in the high-dimensional case. By using cosine

Table 2: Column 'Position' means the query activation is taken from that position. 'S1+1' means the token right after S1. Rows are ordered according to the narration in the original paper. When we say 'S name', it means the the name of S in the query input, but the name is not necessarily S in the samples. This also applies to 'IO name'. The information learned by InversionView which is different from the information suggested by Wang et al. [54] is in bold .

Figure 23: ϵ -preimage of the same activation as Figure 4b using normalized Euclidean distance instead of cosine similarity (Appendix E.4). The line shown in the figure still represents threshold of 0.1. While at ϵ = 0 . 1 , some query inputs do result in a substantial number of inϵ -preimage samples, many cases result in very small or even empty (as here) sample sets, suggesting that the ϵ -preimage at 0.1 - at least as accessible to the decoder - is extremely small in this model, which we speculate is related to the models higher dimensionality compared to the other tasks (768 vs 32/64), which tends to make Euclidean distances large except for extremely similar vectors (see Appendix E.4 for more discussion). We find it more convenient and natural to use similar thresholds ( ϵ = 0 . 1 ) across tasks and account for the different geometries using different distance metrics. What is key, however, is that for an appropriately higher ϵ (e.g., ϵ = 0 . 4 ) we again always obtain a substantial number of samples, and - most importantly - these samples lead to the same interpretation as we obtained in our main experiments with the cosine similarity. Here, for example, we can obtain the same interpretation by setting ϵ = 0 . 4 , i.e., the IO name is encoded in the activation.

QueryInput

GeneratedSamples

0.239;<|endoftext>Then in the morning,Lindsey and Joseph were thinking about going to the garden.Lindsey wanted to give a ring(to)Joseph[EOS]

0.302;
Then in the morning,Kelly and Courtney were thinking about going to the house.Kelly wanted to give a ring(to)Courtney[EOS]

0.323;
Then in the morning,Lindsey and Bryan were thinking about going to the station.Lindsey wanted to give a necklace(to)Bryan[EOS]

0.331;
Then,Lindsey and Jessica were thinking about going to the office.Lindsey wanted togive a necklace(to)Jessica[EoS]

0.331;<|endoftextl>Then in the morning,Allison and william were thinking about going to the house.Allison wanted to give a basketball(to)william[EOS]

0.539; 
Then in the morning, Lindsey and Christina were thinking about going to the garden. Christina wanted to give a bone( to) Lindsey[EOS]

2.0

1.8

1.6

SSOl

1.4

1.2

0

10

20

40

50

similarity, we allow more input samples to lie in a close distance with the query input, as cosine similarity ignores the magnitude difference. This suggests cosine similarity may overall be more suitable when applying InversionView in high-dimensional activations.

F 3-Digit Addition: More Details and Examples

F.1 Implementation Details

We constructed 810,000 instances and applied a random 75%-25% train-test split. The probed model is trained with a constant learning rate of 0.0001, a batch size of 512, for 50 epochs (59350 steps). We save the last checkpoint as the trained model, and test it on the test set. We use the AdamW [36] optimizer with weight decay of 0.01. The training loss is shown in Figure 25.

With regard to the decoder model, we select x 0 , pre , x i, mid , x i, post a i,j as the set of activation sites we are interested, where i ∈ { 0 , 1 } , j ∈ { 0 , 1 , 2 , 3 } . When sampling training data, we select an activation site and a token position uniformly at random. We sample 100 million training examples (all activation sites are included) and train the decoder with batch size of 512, resulting in roughly 200K steps. The final average in-preimage rate is 88.6%.

F.2 More Examples of InversionView

See Figures 26 to 29. 5

F.3 Model Deficiency

In Section 3.3, we mention that there is no firm and clear path of obtaining the carry from ones to tens, so the model may make wrong prediction. We examine those instances for which the model makes wrong prediction and find they all satisfy one condition: F2+S2=9. In other words, it fails to make the right prediction because the ones place matters. This is consistent with our interpretation of the model. Furthermore, we check the model's accuracy on this special subset where F2+S2=9, and find that it is significantly higher than chance level. The accuracy on training subset (training data where F2+S2=9) is 80.45%, and on test subset is 80.06%, while chance level is 50%. So, we can infer that the probed model obtains some information about the ones place by means other than memorization. Indeed, we observe fuzzy information about ones place in a 1 , 0 and a 1 , 1 occasionally (See Figure 30).

F.4 Qualitative Examination Results

We present our overall qualitative results in Figure 31 and Table 3. We have found that the model obtains required digits by attention, and primary digits are assigned more heads than secondary digits, e.g., a 0 , 0 , a 0 , 1 , a 0 , 2 for hundreds and a 0 , 3 for tens when predicting A1. More importantly, the primary digits are encoded precisely while the secondary digits are encoded approximately in the residual stream. In addition, the model routes the information differently based on whether A1=1, i.e., the length of the answer is 3 or 4. When predicting A2, this information is known before layer 0, thus paths differ from the start. On the contrary, when predicting A3, the information is obtained in layer 0, thus paths differ only in layer 1. Furthermore, in Figure 31, sub-figure (c) and (d) are very similar, indicating model uses almost the same algorithm to predict digit in tens place. While sub-figure (e) shares the layer 0 with (d), its layer 1 is similar to (f).

F.5 Causal Intervention: Details and Full Results

We use similar activation patching method as character counting task, described in D.3. In Figure 31, we split the overall algorithm into individual ones for digits in the answer, under different

Table 3: Summary of our observations for each activation site and position. 'same as' denotes that there is no obvious difference between the two sites for indicated position.

Activation Site: a1,1

Activation Site: α1,3

Activation Site: α1,3

B562+119=68

(1)

B268+899=1(1)67

B107

QueryInput

QueryInput

QueryInput

0.000;B 5 6 2+ 1 1 9 =6 8 (1)

0.000;B268+899=1(1)67

GeneratedSamples

GeneratedSamples

GeneratedSamples

10.029;B482+299=78(1) E

1 0.010; B 269+ 898= 1(1)67E

0.000; B307+ 689=9 (9)6E

10.033;B372+509=88(1)E

10.011;B869+298=1 (1)67E

0.000;B207+829=10(3)6E

10.033;B372+509=88(1)E

10.020; B867+299=1(1) 66E

0.001;B377+ 959=13(3) 6E

10.046;B142+759=9 0(1)E

I0.067;B868+298=1(1)66E

0.004;B397+ 809=12(0) 6E

I0.047;B292+419=71(1) E

I0.071;B162+999=1(1)61E

0.004;B 247+799= 10 (4)6 E

10.048;B412+ 449= 86(1) E

I0.073;B865+299=1(1)64E

0.005;B767+ 649=14 (1)6E

10.053;B162+769= 93(1)E

0.079;B666+499=1(1)65E

0.005;B347+ 349=6(9)6E

10.045;B167+738= 90(5) E

0.114; B431+239= 67(0) E

■0.141; B 969+196=1(1)65E

0.261;B7 9 1+66 8 = 1 4 5 (9) E

0.260;B779+996=1 (7)75E

0.628;B736+ 728= 146 (4) E

0.271; B772+419=11 9 (1) E

10.291;B364+197= (5)61E

0.706;B836+869=170(5)E

condition. Each algorithm predict target token based on multiple types of information (digits in hundreds/tens/ones place).

In order to verify the paths responsible for routing each type of information, we construct contrast examples as follows: Given a prediction target (e.g., A1), a set of tokens that can be changed T chg (corresponds to a certain type information, e.g., F1 and S1), we construct a contrast example x con that contains a different token t con as prediction target by changing tokens in T chg . Note that the contrast example still follows the rule that the answer is the sum of the two operands.

We now give a detailed explanation for Figure 6b shown in the main paper, in which prediction target is A1 and we patch head output corresponding to the preceding token '='. On the left of Figure 6b, T chg = { F1 , S1 } . So in the contrast examples the F1 and S1 are changed and other digits in operands remains the same. In the third run where we calculate LD pch , activations are replaced by new activations from contrast example, so the new activations contain modified F1 and S1 information. Therefore, for activations that contributes to routing F1 and S1 (e.g., a 0 , 0 = ), patching them with new activations can effect model's prediction. On the contrary, patching a 0 , 3 = has no effect

Activation Site: α1,0

Activation Site: a1,1

Activation Site: α1,1

B573+269(=)842

B352+106(=)458

B534+220(=)754

QueryInput

QueryInput

QueryInput

0.000;B573+269 (=)842

0.000;B352+106(=)458

0.000;B534+220 (=)754

Generated Samples

Generated Samples

Generated Samples

I0.032;B562+279(=)841E

I0.055;B135+302(=)437E

0.082;B240+524(=)764E

I0.048;B559+262(=)821E

I0.059;B364+103(=)467E

0.083;B624+134(=)758E

I0.054;B569+283(=)852E

I0.071;B124+355(=)479E

■0.088;B213+535(=)748E

0.090;B584+279(=)863E

0.074;B135+301(=)436E

■0.162; B 8 5 1 + 28 (4)= 113 5 E

0.094;B569+254(=)823E

0.075;B335+101(=)436E

0.184;B4 35 +22(4)= 6 5 9 E

0.094;B569+254(=)823E

0.076;B141+307(=)448E

0.218;B430+274(=)704E

0.098;B569+263(=)832E

■0.098;B345+113(=)458E

0.284;B 470 +33(4)= 804 E

0.206;B535+283(=)818E

■0.105;B142+301 (=)443E

0.321;B682+704= (1)386E

0.427;B550+258(=)808E

0.299;B944+141= 1 (0)85 E

0.446;B583+36 (5)=948 E

0.927;B370+649(=)1019E

0.305;B307+110 (=)417E

0.482; B 932+ 580= 1 (5) 12 E

[10]

1.2

[e'll

0.1

0.2

0.3

(a)

Yes

(b)

Yes

(d)

(t）

First digit in answer = 1?

First digit in answer = 1?

No

No

1.1

1.2:

head output

Key/Value

head index

Layer.Head

0.0

Query

VerificationofinformationflowfromF1,S1

VerificationofinformationflowfromF2,S2

20

8

->

15

6

Increment

10

4

5

2

0

0

aA1

0,0

0,1

a0,2

aA1

0,3

aAi

1,0

aA1

1, 1

aA1

1, 2

aA1

1,3

0,0

aAi

0,1

aA1

0,2

aA1

0,3

aAi

1, 0

aA1

1, 1

aA1

1, 2

aA1

1,3

PatchedActivation

PatchedActivation

VerificationofinformationflowfromF2,S2

VerificationofinformationflowfromF3,S3

15.0

15.0

-->

-->

-->

12.5

-->

12.5

10.0

Increment

10.0

7.5

7.5

5.0

5.0

2.5

2.5

0.0

0.0

aA1

0,0

aA1aA1

0,1

0,2

dA1

0,3

aAi

1,0

aAi

1,1

aA1

1,2

1,3

0,0

0,10,2

dA1dA1

dA1

aA1

1,0

aAi

1,1

dA1

1,2

dA1

1,3

PatchedActivation

PatchedActivation

̸

because it contains information about F2 and S2, which are the same in contrast examples. On the right of Figure 6b, T chg = { F2 , S2 } , so we are verifying the activations that plays a role in routing F2 and S2. Note that we exclude those data in x orig where F1+S1 ≥ 10, because changing F2 and S2 cannot change A1 in those cases.

Activation patching results for other cases are shown in Figures 32 to 36.

Overall, among all the intervention experiments and their corresponding information flow diagrams in Figure 31, the activation with the highest increment not included in the information flow diagram is a 1 , 1 A 1 in Figure 32 (right), accounting for only 5.92% of the cumulative increment. In this sense, the information flow diagrams coupled with interpretations present an almost exhaustive characterization of the algorithm used by the model to predict the answer digits.

VerificationofinformationflowfromF2，S2

VerificationofinformationflowfromF3,S3

15.0

15.0

-->

-->

12.5

"->

12.5

-->

10.0

10.0

Increment

7.5

7.5

5.0

5.0

2.5

2.5

0.0

0.0

aA2

0,0

aA2

0,1

dA2

0,2

dA2

0,3

dA2

dA2

1,1

dA2

1,2

aA2

1, 3

aA2

0,0

dA2dA2

0,1

0,2

dA2

0,3

dA2

1, 0

dA2

1,1

dA2

1,2

aA2

1,3

PatchedActivation

PatchedActivation

̸

Verification ofinformationflowfromF3,S3

17.5

-->

15.0

-->

12.5

Increment

10.0

7.5

5.0

2.5

0.0

0,0

aA2dA2

0,1

0.2

dA2

A2

dA2

1.1

dA2

dA2

1,3

PatchedActivation

̸

G Decoder Likelihood Difference

In figures of addition task, we also show the decoder likelihood difference for each token in generated samples. It indicates what might be relevant to the activation. It is calculated as follows: During generation, we sample the next token from the distribution produced by the decoder model, and we record the probability of the sampled token in that distribution. We denote it as p act . Then, we run the decoder again with the same input tokens, but this time it is fed with a 'blank' activation (the activation corresponding to BOS token from the same activation site: because of the causal masking of the probed model, this activation does not contain any information). Therefore, it produces a different distribution. The probability of the same token (the token already sampled in the normal run) in the new distribution is p blank . The difference p act -p blank indicates if the decoder model can be more confident about a token when it receives the information from the activation. If p act -p blank > 0 , the color is blue, and if p act -p blank < 0 it is red. The depth of color is proportional to the magnitude of the value. Therefore, it highlights what can be learned from the query activation, in addition to what is in the context.

VerificationofinformationflowfromF3,S3

VerificationofinformationflowfromA1

40

15.0

-->

-->

-->

->

12.5

30

Increment

10.0

20

7.5

5.0

10

2.5

0

0.0

a0,1a02

aA3

0,3

1,0

aA3

1,1

aA3

1,2

aA3

1,3

aA3

0,0

aA3

0,1

aA3

0,2

aA3

0,3

aA3

1, 0

aA3

1,1

aA3

1,2

aA3

1,3

PatchedActivation

PatchedActivation

̸

Note that the decoder has learned how to handle the activation of BOS, since it also appears in the training set because we sample uniformly at random among all tokens in the input (as in IOI and addition task).

Importantly, unlike the distance metric value, the decoder likelihood difference depends on the capability of the decoder model, and we should bear in mind that it might be inaccurate. We caution that this difference does not necessarily highlight the directly relevant part, complicating its interpretation. As shown in Figure 37, the color highlights digits in the second operand, which does not reflect the actual flow of information. In these examples the query activation corresponds to digits in answer. For example, in the left most sub-figure, x 0 , pre contains the information '6 is at the position of A1', but the color does not highlight A1. This is because, on one hand, decoder predicts S1 conditioned on F1, so knowing their sum will significantly increase the confidence of S1, and S1 is highlighted. On the other hand, when predicting A1, the previous digits can already determine the answer. There is a high confidence even without knowing A1, thus it is not highlighted. In essence, the information contained in a query activation may manifest itself early, and the decoder likelihood difference does not necessarily align with the part of the input from which the information has actually been obtained.

An interesting direction for future work could be developing decoders that generate samples in a permuted order, and generate most confident tokens first, possibly based on architectures like XLNet [60].

H Factual Recall: Detailed Findings

H.1 Background

We use InversionView to investigate how GPT-2 XL (1.5B parameters) performs the task of recalling factual associations. In recent research [24, 11, 20], the task has been formalized in terms of triples ( s, r, a ) , where s , r , a are subject, relation, attribute respectively. The model should predict a based on input containing s and r in natural language. E.g., given the prompt: 'LGA 775 is created by" , the model is expected to predict 'Intel" . In this section of the paper, we refer the last token as END, and the last subject token as SUBJ. In the above example, END is 'by" and SUBJ is '75" . Geva et al. [24] found a high-level pattern that GPT-2 XL uses in solving this task: Early-middle MLP layers at SUBJ integrate information about the subject into its residual stream. Meanwhile, the relation

information is incorporated into END's residual stream through early attention layers. In upper layers, the correct attribute is moved from SUBJ to END's residual stream by attention layers.

H.2 Selecting Activation Sites

As we mentioned earlier, our goal here is not to provide a full interpretation of the computations performed to solve this task; rather, it is to test whether InversionView can successfully produce interpretable results on larger models. As the factual recall task is complex and involves many model components [61, 11], we decide to focus on 25 attention heads in the upper part of the model (layer 24-47) that contribute most frequently to the final prediction. We select these activation sites because the attribute retrieval tends to happen there [24] and we expect more abstract information in ϵ -preimage. Here, we note that preliminary experiments revealed that it is necessary to restrict the number of activation sites that the decoder is trained for, given a limited compute budget, as more activation sites require the decoder to learn more a complex overall inverse mapping from activations to inputs. Scaling the approach to apply to many activation sites simultaneously is left to future work.

Concretely, we use the attribution method introduced in [18] to estimate the head importance at END position. Formally, given y = z 1 + · · · z m , the importance of each term z j to the sum y is estimated as follows:

Regarding the intuition and estimation quality of this method, please refer to [18]. In our case, x i,mid END = x i,pre END + ∑ j a i,j END , we calculate importance ( a i,j END , x i,mid END ) on each example on a subset of COUNTERFACT [37]. The subset contains around 1,000 factual prompts known by GPT-2 XL (we used the same subset as described in [37] Appendix B.1). We consider an attention head activated on a certain input prompt if its importance is higher than the threshold of 0.02, and calculate the frequency of being activated over the subset. Finally we select top 25 most frequent heads in model's upper layers. 6

There are multiple methods that can be used to find important attention heads [38, 39, 50, 29], because we do not have strict requirements for finding the most important heads, we choose this one because of its simplicity.

6 h 31 , 0 , h 33 , 0 , h 38 , 22 , h 29 , 9 , h 37 , 7 , h 32 , 12 , h 31 , 8 , h 24 , 24 , h 28 , 3 , h 25 , 7 , h 28 , 21 , h 27 , 16 , h 42 , 24 , h 31 , 4 , h 34 , 20 , h 30 , 23 , h 24 , 8 , h 30 , 8 , h 25 , 10 , h 33 , 9 , h 32 , 15 , h 30 , 1 , h 29 , 20 , h 36 , 17 , h 35 , 19 , ordered by frequency, last one is the most frequent

H.3 Decoder Training

To train the decoder model, we collect text from 3 datasets, including the factual statements from COUNTERFACT [37] 7 and BEAR [56] 8 , as well as general text from MiniPile [30]. The factual statements we used are complete sentences containing the final attribute. Note that in COUNTERFACT, for each attribute, there are always multiple different subjects associated with it. So if the information contained is solely the attribute, a well-trained decoder should generate different subjects. Regarding MiniPile , we randomly select 10% of it, and split the text into sentences [9] and remove sentences longer than 100 characters.

Importantly, for each head we selected, we extract a separate subset of sentences from the collected data on which the head is 'activated". By being 'activated" we mean the attention weight on BOS token is less than 0.6 9 . We find this is important based on preliminary experiments, because in many cases attention heads in higher layers execute 'no-op" by exclusively attending to BOS, resulting in attention output containing no information. Training decoder on these activations discourages it to learn to read information from the query activation. So each head correspond to a subset of text, on which the head output will be captured to create training data. The number of sentences in subsets ranges from 0.6 to 2.6 million.

The training data for decoder consists of two parts, each of which accounts for 50%. In one part, the activations are taken from GPT-2 XL when processing factual statements from COUNTERFACT and BEAR, and activations correspond to the END position (the sentence structure is provided in these datasets). In the other part, the activations correspond to text from all 3 datasets (thus mainly composed of MiniPile text), and correspond to the position with least attention weight on BOS token. By doing so, we emphasize the importance of factual statements domain while covering a large variety of text.

We use GPT-2 Medium as the backbone model for decoder. Concretely, the newly added components in the decoder architecture (e.g. those parts responsible for processing query activation as shown in Figure 12) are trained from scratch, but the transformer layers in the decoder are equipped with pretrained weights. In this way, we also make use of the existing capacity of language models. Note that in other case studies, we use a small 2 layer transformer as decoder. The reason why we use a much larger decoder for this task is we expect a much more complex inverse mapping to be learned. Specifically, we expect ϵ -preimage for some activation sites to contain multiple different subjects sharing a certain attribute, the decoder needs to generate these subjects given the attribute. So it needs enough capacity to memorize the knowledge.

Similar as before, we always add the ' < | endoftext | > ' token as the BOS token when capturing query activation from GPT-2 XL. We again use a new token '[EOS]' as the EOS token when training the decoder. We sample query activation with equal probability of activation sites. We sample 32 million training examples (all activation sites are included) and train the decoder with batch size of 512, resulting in roughly 60K steps. The final average in-preimage rate is 36.0%.

H.4 Sampling from Decoder

Similar as before, when using decoder to generate samples, we perturb the query activation before feeding it to the decoder in order to cover the neighbourhood around the query. In this larger model with high-dimensional activations, we found it useful to craft this process more carefully and use different disturbance. In preliminary experiments, we found that cosine similarity usually produces more interpretable ϵ -preimages, therefore, we randomly sample activations that have a certain angle θ to the query activations, and then randomly scale it so that its magnitude ranges from e -1 · ∥ z q ∥ 2 to e · ∥ z q ∥ 2 . We repeat this process for different θ values with in the range cos θ ∈ [0 . 75 , 0 . 9] . To further encourage diversity, we lower the probability of tokens that have already appeared too many times in the generated samples. 10


WindowsMediaPlayerisdeveloped(by)Microsoft

QueryInput

QueryInput

0.000 ; <lendoftext|>Windows Media Player is developed( by) Microsoft

0.000; <|endoftextl>Germaine Greer's domain of work( is) feminism

GeneratedSamples

Generated Samples

0.112;<|endoftext]>Adobe Premiere Pro is developed( by) Adobe[EOS]

10.055;<|endoftextl>Daniel Bernoullis domain of work(is)physics[EOS]

0.116;<|endoftext]>Jeep Compass is developed(by) Jeep[EOS]

10.057;
Al-Biruni's domain of work( is)beginning[EOS]

0.142;
Microsoft Silverlight is developed(by)Microsoft[EOS]

0.080; <|endoftextl>Fred w.Riggs's domain of work( is) sociology[EOS]

0.145 ; <|endoftext]>ChromeOS is developed( by) Google.[EOS]

0.083;<|endoftextl>Gene Lees's domain of activity( is) jazz[EOS]

0.158 ; 
WP7 is developed( by) Google.[EOS]

0.113;
Michel Foucault's domain of work(is) history[EOS]

0.137;
Werner Heisenberg's domain of activity(is) physicist[EOS]

0.143; 
Thor's domain of work( is) musician[EOS]

0.406;
Yandex is(located) in Russia[EOS]

0.605;
Argyln_'s errortheory(.)[EOS]

10.419;
ArabBankingCorporationis located(in)the country of Bahrain[EOS]

0.626;<|endoftextl>Jesuit Collegeis located(on)Nam_esti ZachariasezHradce.[EOS]

10.452;
Valencia, CA was already governed by Kyriakos Mitsotakis(.)[EOS]

0.652;
The Lord of(the)Rings.[EOS]

0.474;
Victoria and Albert Museum South Kensington(is) within London[EOS]

0.515;
 see several recent(Lara)vel 4 posts on the community site.[EOS]

0.769 ;
D.W.B.Tessa,and L. W.A.Cottanei,Phys(.)[EOS]

We use the first 200 examples of the aforementioned subset of COUNTERFACT (the subset containing examples known by GPT-2 XL) as query inputs and generate samples. The results are also available in our web application.

H.5 Observation

Choice of Threshold. We use the same distance metric as in IOI task, and we set the threshold ϵ = 0 . 25 . As we mentioned earlier, larger threshold produces more coarse-grained information. Because we observe that the trained decoder in many cases does not generate enough and diverse samples within a close distance (e.g., 0.1), we increase the threshold in order to draw more reliable conclusions.

A priori, the sparsity of sampled ϵ -preimages at smaller thresholds may reflect multiple possibilities. One is that there indeed are no close neighbors for the query input. For example, if the subject's text is copied, then only samples containing the same subject or the same token will lie in ϵ -preimage. Such a phenomenon may generally reflect the high-dimensional geometry of larger models. Another one is that the decoder is not complete; and that it would either require more training or more capacity, or more training data. 11 Training larger decoders on more data would mitigate this problem, which can be done in future work. Nevertheless, in some cases, we do observe a substantial number of diverse samples even within ϵ = 0 . 1 , allowing us to infer information at a higher level of granularity in these cases.

Some heads have fixed behavior We observe that some heads' outputs almost always contain only relation information, if they contain any information at all (being 'activated'). 12 In Figure 38, we show two examples for one of these heads. We can see that samples in ϵ -preimage share the same relation. The results are consistent across query inputs. We can infer that the these heads move the relation information to END's residual stream.

On the other hand, some other heads almost always move information about the subject, when they are 'activated'. 13 . Figure 39 shows one of these heads. We can see that the information is only about the subject - so the function of these heads can be summarized as moving information about subject. Interestingly, while a head can move certain attribute about the subject (e.g., nationality, or profession, etc), the attributes it moves for different subjects are diverse. For instance, while a head might, on a certain input, move information that the subject plays certain kind of sports, it may, on another input, move information that the subject is an electronic product. One head, 31.0 tends to usually show

nationality or language information about the subject, but usually the heads we studied show no clear preference for a type of attribute. In addition, these is no obvious correlation between the subject attributes encoded by such heads and the attribute queried by the relation.

Other heads exhibit a mix of behaviors The other heads among the 25 heads we inspect, when 'activated', move information about subject or relation (in some cases, both). Which behavior is exhibited varies between inputs. Figure 40 shows one of these heads. On the left of the figure, the head moves information about subject, while on the right it moves information about the relation.

Relation-agnostic retrieval In the factual recall task, only one specific attribute is sought. Geva et al. [24] found evidence that the model 'queries' the residual stream at SUBJ for the specific attribute asked by the relation part of the prompt, and the representation at END can be viewed as such a relation query. In other words, the attribute extracted from the subject representation depends on the relation. However, we do not observe reliable evidence for this phenomenon among the 25 heads we inspect. In the figures we have shown so far, we can see the commonality shared between samples in ϵ -preimage is not the attribute requested in the prompt. This still holds if one reduces the

Activation Site: a32,12

Activation Site: @31,4


MeganRapinoeprofessionallyplaysthesport(of)soccer

<|endoftext>Audible.comisowned(by)Amazon

0.000 ; <|endoftextl>Megan Rapinoe professionally plays the sport( of) soccer

0.000 ; 
Audible.com is owned( by) Amazon

Generated Samples

Generated Samples

1 0.041 ;<lendoftextl>The native language of Megan Rapinoe( is) American English.[EOS]

0.003;<|endoftextl>Audible's headquarters are(in)Cincinnati[EOS]

0.146;<lendoftextl>I know that soccer star, Abby Wambach has called for soccer to speak with non-(western)

0.074;<|endoftextl>Zenio listed podcast audible's plans(to) collaborate with other entities.[EOS]

youth.[EOS]

10.074 ; 
Get Up to $15 Discount Audible Gift( Card) for Bible Studies.[EOS]

0.228 ; 
H[ulu) is owned by WarnerMedia.[EOS]

0.183 ; <|endoftext>In five World Cups, he has never won it on the road(.)[EOS]

0.232; 
Amazon Music() from Amazon[EOS]

0.194 ; <|endoftext>What sport does Bastian Schweinsteiger( play)? They play soccer[EOS]

0.234 ; <|endoftext]>audible.com,( from) YouTube[EOS]

0.211 ; <|endoftextl>The soccer world will( watch) this game.[EOS]

0.242; <|endoftext]>At the beginning of the year, the company started advertizing for podcasts(.)[EOS]

0.217 ; 
He soon saw how Katherine felt about the internationals,( soccer) and economics.[EOS]

0.298 ; <|endoftext]>Yahoo! Music( owner) Yahoo[EOS]

0.225 ; <|endoftextl>What genre does Bastian Schweinsteiger( play)? They play soccer[EOS]

0.462 ; <|endoftextl>WriteNow,( created) by Google[EOS]

0.230 ; <|endoftextl>So yeah, maybe the former International goalscorer can get it right from time to time(.)[EOS]

0.529 ; <|endoftext>I did just previewing the Ouroboros CD( all) week, and 1feel like it is superb![EOS]

0.234 ; 
ln his mind, the agreement made this past week was a big game for American soccer(.)[EOS]

0.545 ; <|endoftextl>Who said radio was better(?)[EOS]

0.300 ; <lendoftextl>By November 2015 there were 16 MLS clubs based in Cameroon( making) 12 international

0.561; <|endoftext>Meerkat( was) there.[EOS]

appearances.[EOs]

0.390 ; 
Women's International( League) for Peace and Freedom is headquartered in Geneva[EOS]

Activation Site:a32,15

Activation Site: α31,0

<|endoftextl>JosephSchumpeter'sdomainofworkis(the)economics

<|endoftextl>JosephSchumpeter'sdomainofworkis(the)economics

0.000 ; <|endoftextl>Joseph Schumpeter's domain of work is( the) economics

0.000 ; <|endoftextl>Joseph Schumpeter's domain of work is( the) economics

Generated Samples

Generated Samples

10.034;<lendoftextl>Joseph Schumpeter's domain of work(is) economics[EOS]

1 0.010 ; <|endoftextl>Joseph Schumpeter works in the field(of) economics[EOS]

1 0.044 ; <|endoftextl>Joseph Schumpeter's domain( of) activity is economics[EOS]

1 0.050 ; <|endoftextl>Joseph Schumpeter('s) expertise is economics[EOS]

0.102; <|endoftextl>The domain of work of Joseph Schumpeter is( economics)[EOS]

0.217 ; <|endoftextl>John von Neumann works( in) the field of mathematics[EOS]

0.242 ; 
Misesian economics consists of three aspects(.)[EOS]

0.240 ; <|lendoftextl>Sigmund Freud took up work( in) Vienna[EOS]

0.246;<|endoftext]>Mises works in) the field of economics[EOS]

0.241 ; <|endoftext]>Wittgenstein took( up) work in Vienna[EOS]

0.248 ; <|endoftextl>Milton Friedman worked( in) the city of Paris[EOS]

0.245 ; <|endoftextl>Franz Kafka,( a) native German[EOS]

0.347 ; 
| Joe Weider,(")Communique-Hencome",_Signal of the Times_,April 1995, p. 9; available at

0.444 ; 
Merejy empirical analysis of growth laws is at best a starting point(.)[EOS]

p. 9.[EOS]

0.438 ; <|endoftextl>And yet sure enough, Friedrich Nietzsche( was) wrong.[EOS]

10.508 ; 
An introduction to financial theory and the problems of finance( and) economic policy.[EOS]

0.568 ; 
|T. Mermin,_Saint Sava: A Militant Teacher_ (New York(:) Random House, 1963).[EOS]

0.528 ; <|endoftextl>They say that optimization is the year_'s most important innovation(.)[EOS]

0.600 ; 
See, Michael Moore 1963:( Beyond) Hatred, p. 61.[EOS]

0.536; <|endoftextl>Fouault works( in) the field of history[EOS]

0.612 ; <lendoftext|>in Dhanbad The mother tongue of Frantz Fanon( is) French[EOS]

threshold. Figure 41 shows two more examples showing this characteristic. In general, we find that the information moved by the 25 attention heads tends to be the most important attribute or the 'main' attribute about the subject. On the left of Figure 41, the most important attribute coincides with the requested attribute. We know this is a coincidence because the closest sample in ϵ -preimage has a different relation. Note that other attributes about the subject could be moved by heads that we have not studied.

In addition, while heads that attend to subject usually move the most important attribute, we do observe sometimes different attributes are moved by different heads. In Figure 42, we show that information about profession and information about language/nationality of the subject are extracted by two heads. In fact, we observe that head h 31 , 0 tends to extract language/nationality information in general. But we do not find other obvious pattern of attribute category extracted by other heads.

Our findings echo those from [11], who argue that the primary mechanism for the factual recall task is additive. With our running example 'LGA 775 is created by' , simply speaking, some heads promote attributes associated with the subject ( chip , hardware , Intel , etc.), some heads promote attributes associated with the relation ( Apple , Nintendo , Intel , etc). When the results from these independent mechanisms are added together, the intersection ( Intel ) will stand out. Therefore, the model can solve the task by adding two simple circuits, while humans find Q-composition [17] (i.e., relation information is used as queries in attention heads) more intuitive. From another perspective, this

Activation Site:q31,4

Activation Site: q36,17


LGA775iscreated(by)Intel

0.000 ; 
In Marshall Islands, the language spoken is a mixture( of) English

Generated Samples

Generated Samples

0.125;
French Polynesia, they(understand) French[EOS]

0.146 ; 
The official language of Solomon Islands( is) English[EOS]

0.160 ; <|endoftextl>Skylake is a product( of) NVIDIA[EOS]

0.164 ; 
 Vanuatu,( the) language is English[EOS]

0.170 ; <|endoftextl>Some hardware vendors manufacture defective versions of processors (and motherboard_'s0.)[EOS]

0.178 ; <|endoftextl>South Tarawa serves as the capital of Kirib(ati).[EOS]

0.213 ; <|endoftextl>As a general rule, look for the( motherboard)’s product number, not the model.[EOs]

0.186 ; 
In East Tuvalu, the language( is) English[EOS]

0.215 ; <|endoftextl>1_ll probably start from scratch for this: AMD_'s RAP3( is) AWESOME.[EOS]

0.202;
 New Caledonia(,)they understand French[EOS]

0.238 ; <|endoftextl>If the cross center point from the MSI at 1400 was i.i.d(.)[EOS]

0.241 ;<|lendoftextl> Niue,( the) Ilanguage is English[EOS]

0.241 ; <lendoftext|> Niue,(the) language is English[EOS]

0.242 ; 
Apparently the x86_64 cpus can('t) run the code snippet in gcc.[EOS]

0.313 ; 
 East Timor,they( understand) Portuguese[EOS]

0.249 ; 
Q: Does Intel's( Quick) Sync have DRM in it?[EOS]

0.591 ; 
Hokkese government Hok(k)ese government was unicameral,[EOS]

0.660 ; 
10/30 What is (-284)/(-3340)*6)/(-33)/(-8)/(-114)/(-)112))?[EOS]

0.693; 
 East Godavari, they( understand) Hindi[EOS]

0.771; 
Use of the( matte) highlight color is pretty high in these cases.[EOS]

0.805 ; <lendoftextl>Dhurakij Pundit University can be found( in) Bangkok[EOS]

mechanism implies vector arithmetic. Instead of vector addition in vocabulary space, we can think of it as first summing two vectors (e.g., the output of subject heads and the output of relation heads) and then projecting them to vocabulary space.

The evidence found by Geva et al. [24] can also be explained by this mechanism. They use DLA to inspect attention layers' updates to END's residual stream, and find that the token most strongly promoted by each update usually matches the attribute predicted at the final layer. In other words, after projecting attention layer's output to vocabulary space, the top-1 token is usually the exact requested attribute. Because their experiments study attention layer's output as a whole, instead of individual heads, an alternative explanation is the additive mechanism mentioned above. Importantly, because we only inspect 25 heads, other mechanisms including Q-composition are also possible. Moreover, different mechanism can exist in other models, readers should not draw strong conclusions.

H.6 More Examples of InversionView

See Figure 43, Figure 44. Interestingly, when inspecting the ϵ -preimage shown on the right of Figure 44, we also find an example showing InversionView can detect a flaw of the model. Our interpretation for this activation is 'Canada-related", and we can see 'York University' (which is in Canada) inside the preimage, while 'University of York" (a different university, located in England) is outside of the preimage. However, we find that the 'U of York" is inside the preimage.

Checking the model's prediction about 'U of York", we find that the model indeed believes 'U of York" is in Canada . More specifically, given the prompt 'U of York is located in", the top-12 predictions for the next token are: the, Toronto, York, downtown, a, one, New, central, Canada, North, Scarborough, London. On the contrary, with the prompt 'University of York is located in" the model's top-12 predictions are: the, York, North, East, central, Yorkshire, north, a, northeast, London, England, northwest.

I Notes on Attention, Path Patching, DLA and others

I.1 InversionView Reveals Information from Un-attended Tokens

In Section 4, we mention that attention pattern is not sufficient to form hypothesis when the model has more than one layer. Because unlike in layer 0 each residual stream contains information only about the corresponding token, in higher layers each residual stream contains a mixture of tokens from the context, making it difficult to determine what information is routed by attention. Besides this point, we also find that sometimes attention pattern can be misleading even in layer 0.

InversionView reveals how components can know more than what they attend to. At the top of Figure 45, we show the attention weights of head h 0 , 3 . Here, '=' attends almost solely to S2, so the head output a 0 , 3 should only contain information that there is an '8' in tens place. The generated ϵ -preimage, however, shows that it contains information about F2: The number in tens place other

Activation Site: @30,8

Activation Site: Q42,24

<|endoftextl>YorkUniversity canbefound inthe heart(of)Toronto


YorkUniversity canbefound intheheart(of)Toronto

0.000;
York University can be found in the heart( of) Toronto

0.000; <|endoftextl>York University can be found in the heart( of) Toronto

Generated Samples

Generated Samples

0.001;<|endoftextl>York University can be found( in) Toronto[EOS]

0.118;
Province of Toronto is situated(on)the shores of the St.Lawrence River.[EOS]

0.001; 
York University can be found( in) Toronto[EOS]

0.010;<lendoftext|>York University is situated(in) the country of Canada[EOS]

0.089;
between York University(and) York College of Pennsylvania.[EOS]

0.201; 
U of Ontario is located( in) Canada[EOS]

0.204 ; 
The York river empties into(the) Gulf of Saint Lawrence.[EOS]

0.201; <|endoftextl>The location of North York Centre mobility hub(is) on Yonge Street.[EOS]

0.207 ; 
The University of York is( located) in the United States of America.[EOS]

0.212 ; 
Dream High School (Hamilton, ON) can be found( in) Ohio[EOS]

0.209 ; <|endoftextl>The University of York is where York( was).[EOS]

0.216;<|endoftextl>Queen'sUniversity)BelfastFaculty of Medicine Health and Life Sciencesis

0.249 ; <|endoftextl> University of York is( where) Yorkshire was.[EOS]

headquartered inBelfast[EOS]

0.229 ; <|endoftextl>Soc., Kingston, ON(,) September 1991, pp.[EOS]

0.489;
Harris v. Mayor&City Council(,) 440 Pa.Super.[EOS]

0.572; 
Wellington, Ontario is located( in) North America.[EOS]

0.231;
The location of Pride Toronto is(Toronto)[EOS]

0.663;<|endoftextl>Yonkers,whose present(name)is wanted on second degreemurder charge.[EOS]

0.448; <|endoftext]>The location of University of York is( in) Yorkshire[EOS]

0.738;<lendoftextl>Princ(eton)Junction station belongs to New Jersey Transit.[EOS]

0.526;<|endoftextl>The team from Research Edge - Edmonton is based( in) Edmonton.[EOS]

0.699; 
What( does) Kevin Barnes play? They play jazz[EOS]

B556+280(=)836

0.000;B556+280 (=)836

GeneratedSamples

GeneratedSamples

Manual Samples

0.000;B357+780 (=)1137E

1 0.022;B 284 +551(=)8 3 5 E

●0.128;B536+280()816E

0.001;B750+ 688 (=)1438E

10.026;B682+ 558 (=)1240 E

0.452;B526+280 (=)806E

0.008;B248+ 986 (=)1234E

1 0.026; B 782+ 757 (=)15 3 9 E

1.111; B516+ 280 (=)7 9 6 E

0.009;B942+ 985 (=)1927E

1 0.027;B 871+ 381 (=)1 252 E

1.717; B506 + 280 (=) 786 E

1 0.012;B154+ 284 (=)438E

10.029; B579 + 882 (=)14 61 E

(b)

1 0.012; B 54 6 + 981 (=)15 27 E

10.029;B283+969(=)1252E

1 0.014;B952+ 887 (=)1839 E

I0.030;B588+670 (=)1258E

1 0.014;B 28 3+ 641 (=) 924 E

I0.031;B981+ 683 (=)1664 E

B5

6

+2

2

8

0

(=)806

10.029;B283+969(=)1252E

10.031;B189+589(=)778E

B5

5

1

6+2

8

0(=)796

1 0.034;B885+ 687 (=)157 2 E

10.034;B885+687(=)1572E

B506+280(=)786

(a)

(c)

than '8' is always in a certain range ( ≥ 4 ), resulting in a carry to the hundreds place. To verify this, we manually constructed examples (rightmost column in Figure 45) where the other number is outside of the range, and found that, for these, the activation distance is indeed very large, confirming the information suggested by the decoder. In layer 0, how does the model obtain information about a token without attending it? At the bottom of Figure 45, we show the attention weights of those manually inserted examples. So the answer is: a different attention pattern would arise if F2 is not in that range. Information can be passed not only by values, but also by queries and keys. InversionView successfully shows this hidden information, even without comparing across different examples.

I.2 Additional Discussion about Path Patching

Besides our argument in Section 4, another important aspect of circuit discovery methods is that, in many tasks (including our character counting task), the computational nodes do not correspond to fixed positions, and directly applying path patching is problematic. It's not really clear how to apply path patching when varying input positions matter, as the literature on circuit discovery defines circuits in terms of components, not in terms of input positions. In the case of Character Counting Task, such an interpretation would just define a circuit linking the embeddings, attention heads, and MLPs, without capturing the role of different positions, and the fact that characters from varying positions feed into the computation. Such a view would not provide any nontrivial information about the mechanics of the implemented algorithm. This reflects a more general conceptual challenge of circuit discovery: When different input positions are relevant on different inputs, as in the Character Counting Task, one could either define a single circuit across inputs in which every input position is connected to a single node that performs a potentially complex computation, or define per-input circuits where the wiring is input-dependent; however, per-sample path patching is not very scalable, and resulting per-input circuits would require further interpretation to understand how they are formed across inputs.

I.3 Additional Discussion about DLA

Direct Logit Attribution (DLA) extends the logit lens method to study individual model components. Specifically, it projects the output of a model component (thus an update on the residual stream) into vocabulary space, and interpret the component by inspecting the tokens it promotes or suppresses. This method has gained popularity in recent years [23, 54, 16, 19], especially in the research of interpreting factual recall in language models [24, 62, 11, 61]. However, in this section, we argue that DLA is only suitable for studying model components that directly affect the model's final output, and is not well-suited for components whose effect is mediated by other components. In the circuits found by path patching [54, 13], we can see many components that do not connect to the final output directly, which suggests a substantial part of their effect on the final predictions is mediated by further components. DLA shows their direct effect, which may even be non-causal 'side effects". Intuitively speaking, some information is meant to be read by a downstream component, e.g., the S-Inhibition Heads' output in IOI circuit is meant to be read by the query matrix of Name Mover Heads, and such information may not necessarily be visible when projecting to the vocabulary space. Dao et al. [15] also point out such limitations of DLA.

In Table 4 we show the results of applying DLA to attention heads' output in IOI circuit. We can see that the expected information is not visible in most cases. The best result comes from the S-Inhibition head 8.10, with only 7.5% of cases where the expected name is in the top-30 tokens and there is no other name being suppressed more than it. The rare cases where the expected name is visible can also be explained by the small direct effect on the final output as depicted by Figure 3(b) in [54].

Therefore, researchers should be cautious when using DLA and should be aware of its limitations. A good usage example is the IOI circuit [54], where the authors first identify those attention heads directly affecting the final logits, and only apply DLA to them and design other experiments to interpret other components. Importantly, in the context of factual recall task, we find that the information given by InversionView about the upper layer attention heads is often visible via DLA, indicating these heads contribute to model's output directly. Thus, our results can serve as confirmation that prior results relying on DLA in this task are generally reliable.

I.4 Methods Generating in Input Space

Feature visualization [43, 41] generates inputs maximally activating a certain neural network unit, and interprets an individual neural network unit (e.g., neurons) to understand its general role across inputs, while InversionView interprets specific values of inner representations, by finding inputs that result in the same vector. When the input changes, the value and thus the interpretation may change. Adversarial or counterfactual example generation methods [27, 59, 46, 45] generate input that is similar to the original input but results in different outcome. Some of them are also used to explain the model. While similar in input space, the adversarial/counterfactual input is likely to be quite different in internal representation space, leading to a different output. In contrast, we are interested in how different inputs in input space are represented very similarly in internal representation space.

Similar to InversionView, GAN inversion methods [58] also study the mapping between input space and representation space, with a focus on interpreting the semantics of GAN's latent space and manipulating generation.

J Automated Interpretability

We further explore whether the process of obtaining the common information from a collection of inputs can be automated by LLMs. We use Claude 3 14 [3] In preliminary experiments we also try GPT-4 [1] but we find Claude 3 works better in our case. In the prompt given to Claude 3, we first describe the task it needs to perform, the terminology we are using (e.g., F1, F2, etc.), the rules (e.g., the pattern it finds should be applicable to each inputs in the ϵ -preimage), input and output form, and the crucial steps it should follow. In addition, we also provide it with 3 demonstrating examples in conjunction with the correct answers. Each example corresponds to a specific activation site and token position (e.g., a 0 , 0 , A1). In each example, there are 2-3 specific query inputs, each query input is accompanied with 20 (sometimes less) samples that are inside the ϵ -preimage. Claude 3 needs to find the pattern for each query input, and summarize its findings across several query inputs, which is the information contained in general in that activation. In addition to the content described above (the common part shared between prompts), we give it a questioning example, which is the content we would like it to interpret. The questioning example shares the same form as the demonstrating example, except that it contains 5 query inputs and their corresponding samples. In addition, when two separate interpretations are needed based on different A1 value, we run the generation twice with examples of different A1 value, instead of giving the model a mixture of two cases and resorting to its own capacity.

We think the following findings from our experiments are worth mentioning: 1) It's hard for the model to align digits of the same place (e.g., comparing all F1 digits), because the samples are presented as a single flattened string instead of a 2-dimensional table. We find that explicitly adding the variable name can largely mitigate this problem, they may serve as certain kind of keys. For example, '7(F1)

1(F2) 1(F3) + 9(S1) 9(S2) 4(S3)'. 2) The generated interpretation is sometimes not consistent. The model may generate different conclusion even with the same prompt, but this usually only happens to less important information. 3) The model does not strictly follow the rule, i.e., the common pattern should match all inputs, even though we state this repeatedly in different ways in the prompt. The model will say 'always' even when there is a counterexample. We should keep 2) and 3) in mind when reading the results.

We run the generation for each entry in Table 3 once, using the samples generated from the corresponding activation. The results are shown in Table 5, accompanied with human interpretation for comparison. The interpretation given by Claude 3 reflects the main information in almost all cases. Even when the information becomes more complex in layer 1, the interpretation quality does not significantly decline. This implies that automated interpretation by LLM is promising. On the other hand, we can also see there are some problems: 1) Some of the model's claims are spurious, these claims are usually ranked low by the model, indicating they are not very obvious. 2) The model sometimes does not explain in a desirable manner. For example, for the entry ' x 0 , mid , A3', the information includes A2 and whether A1=1, which means when A1=1, the sum of F1 and S1 is known. Thus Claude 3 concludes that (F1, S1) is approximately represented. Though this is somewhat true, it is not a fundamental piece of information, and there is a more concise summary. 3) The interpretation tends to be verbose, even though we ask the model to be succinct and provide it with some examples. Despite the problems, we think the automatically generated interpretation is by and large satisfactory and informative, and we think some of the problems can probably be solved or mitigated by engineering better prompts.

K Compute Resources

We ran all experiments on NVIDIA A100 cards. The decoder is trained for 4-6 hours on 2 GPUs for the first three case studies, without exhaustively tuning efficiency of the implementation, which we believe could further speed up training. Regarding the factual recall task, we train the decoder

for less than 1 day on 4 GPUs. Generation of ϵ -preimage samples (including forward passes on the probed model to calculate distance metric) is fast for the first 3 tasks, and it takes around 9 hours on 4 GPUs for factual recall task (for 200 query inputs). Patching experiments run quickly, as they are done for small models.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We ensured that all claims made in abstract and introduction are well supported by our results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have a 'Limitations' paragraph in the Discussion section, where we discuss limitations of this work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper includes no theorems or other non-empirical claims.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Besides explaining all experimental details, we also provide the code and the probed models.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have attached the code with the submission. No new data were created. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify all relevant details, and also include the code for reproducibility, Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: All quantitative results have sample sizes of hundreds of thousands of data points, and error bars would be very small, thus not add substantial information.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We include an appendix section on Compute Resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is foundational research on neural networks, and we anticipate no societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not see any risk of misuse of our research results.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use the GPT-2 Small model and IOI templates, both of which we cite. We use no other existing models or data.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We document the new assets, which are models of character counting and addition.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper involves no crowdsourcing or human subjects experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper involves no crowdsourcing or human subjects experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the method InversionView, which finds inputs that give rise to similar activations (the preimage). As the preimage grows exponentially with sequence length the authors train a conditional decoder to generate inputs from a target activation.\nThe authors show three use cases: a character counting task, Indirect Object Identification, and 3-digit addition.\n\nSTRENGTHS:\nThe approach of generating text examples that lead to similar activation patterns is intuitive and could be useful for interpretability since results are very human readable.\nThe example use cases (especially the toy task of character couting) seem promising.\n\nWEAKNESSES:\n- the abstract could be a bit more specific\n- in general, I think the reader is left wondering for a bit too long what the authors actually do and how the method works.\n- line 28: \"which in turn helps us put together the algorithm implemented by the model.\" promises a bit too much\n- motivation and description of the method is too vague\n- very few details on the method/decoder that generates the examples that are later inspected\n- results of experiments are a bit hard to pass. It would be nice if the reader is guided through those a bit better at different levels of abstraction\n- the approach (generating human interpretable examples that achieve a specific activation patters) seems closely related to feature visualization/ adversarial example and counterfactual example generation/ in general methods that generate in the input space, however those works are not discussed at all\n\nQUESTIONS:\n- a graphic showing how to get from source transformer activations to pre image samples would be helpful\n- Figure 2 is hard to parse: I would suggest highlighting your conclusion/interpretation (aka one activation encodes target character but not count, one activation encodes count but not target character etc) in the figure.\n- in general, for the experiments, I would suggest to clearly state the task and your findings before jumping into technical details\n- 3.3 \"We apply InversionView to the components of the circuit\" seems like you apply inversionView to specific activations found by [39] but then you state \"InversionView unveils the information contained in the activation sites, with\nresults agreeing with those of Wang et al. [39], while avoiding the need for tailored methods.\"\nSo it seems like you needed the \"tailored methods\" to know where to apply inversionView in the first place, or am I misunderstanding sth?\n\nLIMITATIONS:\n- I think when moving from toy tasks to SOTA models, inversionView might become very prone to human tendency for pattern matching\n- in addition polysemanticity might be very relevant for InversionView results but is not discussed at all\n- related work is not discussed in sufficient detail", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

comments: SUMMARY:
The method proposed in the paper seeks to decipher the information encoded within neural network activations. The core idea is to examine subsets of inputs that produce similar activations, utilizing a trained decoder model conditioned on these activations. The authors perform their analysis on three different tasks: character counting, indirect object identification, and 3-digit addition.

STRENGTHS:
* The proposed method helps us find which parts of a network handle specific tasks. For instance, in a counting task, we can see which activations detect the target character and which ones do the counting. This is interesting because it helps us understand how neural networks work inside.

* Showing the underlying algorithm the model uses to solve tasks, like adding 3-digit numbers, gives useful insights.

WEAKNESSES:
* Applying the method to extremely large models and more complex tasks is challenging.

* Selecting an appropriate distance metric and epsilon is challenging and requires tuning, making it difficult to adapt the method to different tasks. This also highlights the authors' impressive effort in making the method work.

QUESTIONS:
* Do you think it is possible to use other distance metrics besides L2-based metrics? Additionally, why do you believe the L2 distance is effective in this context? Is there something unique about the geometric space of these tasks that makes L2 distance particularly suitable? What types of tasks, in general, can be effectively solved using L2-based metrics?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work is mainly based on the representational geometry of activations in the activation space and chooses samples whose distances are within a defined $\\epsilon$-preimage distance. A single two-layer transformer decoder model is trained using activations of each layer from the investigated model. After finishing decoder training, the Euclidean distance is used as a metric to select generated samples based on each activation presentation of each layer, and compare those samples with the query input sentence. This approach is evaluated and analysed under three tasks, i.e., character counting with a 2-layer and 1-head transformer, IOI with GPT2-small and 3-digit addition with 2-layer, 4-attention head transformer. The comprehensive experiment results and analysis confirm the geometry hypothesis, and provide lots of insights to future work.\n\nSTRENGTHS:\n- Activation representation of each layer from small neural networks to GPT2-small can be visualized and explained based on the geometry hypothesis with a trained decoder using those activations as input\n- This hypothesis is evaluated using different tasks, i.e., character counting, IOI and 3-digit addition, and lots of meaningful analysis and insights are discussed\n- Experiments are solid and the appendix includes many details about each case study.\n\nWEAKNESSES:\n- I suggest authors use another figure to demonstrate the whole training and evaluation pipeline, i.e., what input is used for training probed model, what query input is for decoder training and what input and output are from the trained decoder, which will help to understand the whole framework much easier for readers.\n- This method is mainly based on manual investigation to select and analyse generated samples, which might need large labour resources to extend to large-scale LLMs.\n\nQUESTIONS:\n- In line 115, what does $\\mathbf{z}$ represent here? and what is the difference between $\\mathbf{z}$ and $\\mathbf{z}^q$?\n- In line 108, Is this input the same as the query input in figure 1? Another figure introducing the whole pipeline might help a lot to understand.\n- Does this neighbour of geometry hypothesis always hold across all LLM activations?\n\nLIMITATIONS:\nI am looking forward to how to automate this approach and apply it to the larger LLM interpretability analysis.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

comments: SUMMARY:
In this paper, the authors propose InversionView, a method to inspect the information encoded in neural activations. The proposed method is based on checking the activations difference given different inputs. The authors showcase the effectiveness of this tool on mainly three tasks: character counting, indirect object identification, and 3-digit addition.

Strengths

1. The topic of model interpretation is important.
2. The empirical results confirm the effectiveness of the proposed method.
3. The paper is well written and easy to follow. In addition, the authors provide abundant experiments to showcase the effectiveness.

Weaknesses

1. Although being described through text, the precise algorithm for the proposed method is unclear to me.
2. It is unknown how the proposed method scales with larger models. The largest model size in the paper is limited to GPT-2, which is not considered large nowadays.

Overall, I think the paper is interesting and recommend a weak acceptance. I will consider raising my ratings if the above weaknesses are addressed.

STRENGTHS:
1. The topic of model interpretation is important.
2. The empirical results confirm the effectiveness of the proposed method.
3. The paper is well written and easy to follow. In addition, the authors provide abundant experiments to showcase the effectiveness.

WEAKNESSES:
1. Although being described through text, the precise algorithm for the proposed method is unclear to me.
2. It is unknown how the proposed method scales with larger models. The largest model size in the paper is limited to GPT-2, which is not considered large nowadays.

QUESTIONS:
See the weaknesses.

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14