InversionView: A General-Purpose Method for Reading Information from Neural Activations

Authors: xinting  huang
madhur  panwar
navin  goyal
michael  hahn
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 16:24:24

Abstract: The inner workings of neural networks can be better understood if we can fully decipher the information encoded in neural activations. In this paper, we argue that this information is embodied by the subset of inputs that give rise to similar activations.  We propose InversionView, which allows us to practically inspect this subset by sampling from a trained decoder model conditioned on activations. This helps uncover the information content of activation vectors, and facilitates understanding of the algorithms implemented by transformer models. We present four case studies where we investigate models ranging from small transformers to GPT-2. In these studies, we show that InversionView can reveal clear information contained in activations, including basic information about tokens appearing in the context, as well as more complex information, such as the count of certain tokens, their relative positions, and abstract knowledge about the subject. We also provide causally verified circuits to confirm the decoded information.

Main paper: 
 introduction: 
 Despite their huge success, neural networks are still widely considered black boxes. One of the most important reasons is that the continuous vector representations in these models pose a significant challenge for interpretation. If we could understand what information is encoded in the activations of a neural model, significant progress might be achieved in fully deciphering the inner workings of neural networks, which would make modern AI systems safer and more controllable. Toward this goal, various methods have been proposed for understanding the inner activations of neural language models. They range from supervised probes [2, 5, 4, 55] to projecting to model's vocabulary space [42, 7] to causal intervention [21, 54, 26, 13 ] on model's inner states. However, to this date, decoding the information present in neural network activations in human-understandable form remains a major challenge. Supervised probing classifiers require the researcher to decide which specific information to probe for, and does not scale when the space of possible outputs is very large. Projecting to the vocabulary space is restricted in scope, as it only produces individual tokens. Causal interventions uncover information flow, but do not provide direct insight into the information present in activations.Here, we introduce InversionView as a principled general-purpose method for generating hypotheses about the information present in activations in neural models on language and discrete sequences, which in turn helps us identify how the information flows through the model-crucial for obtaining the algorithm implemented by the model. InversionView aims at providing a direct way of reading out the information encoded in an activation. The technique starts from the intuition that the information encoded in an activation can be formalized as its preimage, the set of inputs giving rise to this particular activation under the given model. In order to explore this preimage, given an activation, we train a decoder to sample from this preimage. Inspection of the preimage, across different inputs, makes it easy to identify which information is passed along, and which information is forgotten. It accounts for the geometry of the representation, and can identify which information is reinforced or downweighted at different model components. InversionView facilitates the interpretation workflow, and provides output that is in principle amenable to automated interpretation via LLMs (we present a proof of concept in Section 4).We showcase the usefulness of the method in three case studies: a character counting task, Indirect Object Identification, and 3-digit addition. We also present preliminary results on the factual recall task, demonstrating the applicability of our method to larger models. The character counting task illustrates how the method uncovers how information is processed and forgotten in a small transformer. In Indirect Object Identification in GPT2-Small [54] , we use InversionView to easily interpret the information encoded in the components identified by Wang et al. [54] , substantially simplifying the interpretability workflow. For 3-digit addition, we use InversionView to provide for the first time a fully verified circuit. Across the case studies, InversionView allows us to rapidly generate hypotheses about the information encoded in each activation site. Coupled with attention patterns or patching methods, we reverse-engineer the flow of information, which we verify using causal interventions. Interpretation Framework What information does an activation in a neural network encode? InversionView answers this in terms of the inputs that give rise to this activation (Figure 1 ). For instance, if a certain activation encodes solely that "the subject is John" and nothing else (Figure 1 , right), then it will remain unchanged when other parts in the sentence change while preserving this aspect (e.g., "John is on leave today." ⇒ "John has a cute dog."). From another perspective, if all sentences where the subject is John are represented so similarly that the model cannot distinguish them, given one of these representations, the only information is the commonality "the subject is John" (assuming sentences are represented differently when it does not hold). Building on this intuition, given an activation, InversionView aims to find those inputs that give rise to the same activation, and examine what's common among them to infer what information it encodes. In realistic networks, different inputs will rarely give rise to exactly the same activation. Rather, different changes to an input will change the activation to different degrees. The sensitivity of an activation to different changes reflects the representational geometry: larger changes make it easier for downstream components to read out information than very small changes. This motivates a threshold-based definition of preimages, where we consider information as present in an activation when the activation is sufficiently sensitive to it. Formally speaking, given a space X of valid inputs, a query input x q ∈ X , a function f that represents the activation of interest as a function of the input, and a query activation z q = f (x q ), define the ϵ-preimage:2 B z q ,f,ϵ = {x ∈ X : D(f (x), z q ) ≤ ϵ}, 
methodology: 
 where ϵ > 0 is a threshold and D(•, •) is a distance metric. Both ϵ and D(•, •) are chosen by the researcher based on representation geometry; we will define these later in case studies. In practice, 
discovering the underlying algorithm by inversionview: 
 Notation. In the transformer architecture, outputs from each layer are added to their inputs due to residual connection. The representations of each token are only updated by additive updates, B and E denote beginning and end of sequence tokens. The query activation conditions the decoder to generate samples capturing its information content. We show non-cherrypicked samples inside and outside the ϵ-preimage (ϵ = 0.1) at three activation sites on the same query input. Distance for each sample is calculated between activations corresponding to the parenthesized characters in the query input and the sample. "True count" indicates the correct count of the target character in the samples (decoder may generate incorrect counts). (a) MLP layer amplifies count information. Comparing the distances before (left) and after (right) the MLP, we see that samples with diverging counts become much more distant from the query activation. (b) In the next layer (":" exclusively attends to target character -copying information from residual stream of target character to the residual stream of ":"), the count is retained but the identity of the target character is no longer encoded ("c", "m", etc. instead of "g"), as it is no longer relevant for the predicting the count. Therefore, observing the generations informs us of the activations' content and how it changes across activation sites.forming a residual stream [17] . Using notation based on [17] and [40] , we denote the residual stream as x i,{pre,mid,post} ∈ R N ×d , where i is the layer (an attention (sub)layer + an MLP (sub)layer) index, N is the number of input tokens, d is the model dimension, pre, mid, post stand for the residual stream before the attention layer, between attention and MLP layer, and after the MLP layer. For example, x 0,pre is the sum of token and position embedding, x 0,mid is the sum of the output of the first attention layer and x 0,pre , and x 0,post is the sum of the output of the first MLP layer and x 0,mid . Note that x i,post = x i+1,pre . We use subscript t to refer to the activation at token position t, e.g., x i,mid t ∈ R d . The attention layer output decomposes into outputs of individual heads h i,j (•), i.e., x i,mid = x i,pre + j h i,j (LN(x i,pre )), where LN(•) represents layer normalization (GPT style/pre-layer-norm). We denote the attention head's output as a i,j , i.e., a i,j = h i,j (LN(x i,pre )).Decoder Architecture. We train a single two-layer transformer decoder across all activation sites of interest. The query activation z q is concatenated with an activation site embedding e, a learned embedding layer indicating where the activation comes from, passed through multiple MLP layers with residual connections, and then made available to the attention heads in each layer of the decoder, alongside the already present tokens from the input, so that each attention head can also attend to the post-processed query activation in addition to the context tokens. Each training example is a triple consisting of an activation vector z q ∈ R d , the activation site index, and the input, on which the decoder is trained with a language modeling objective. Appendix C has technical details.We train a transformer (2 layers, 1 head) on inputs such as "vvzccvczvvvzvcvc|v:8" to predict the last token "8", the frequency of the target character (here, "v") before the separator "|". For each input, three distinct characters are sampled from the set of lowercase characters, and each character's frequency is sampled uniformly from 1-9. The input length varies between 7 and 31. We created 1.56M instances and applied a 75%-25% train-test split; test set accuracy is 99.53% (Details in Appendix D). We use D(z, z q ) = ∥z-z q ∥2 ∥z q ∥2 (i.e., normalized euclidean distance, as the magnitude of activations varies between layers), where z denotes the aforementioned f (x), and set ϵ = 0.1.Interpreting via InversionView and attention. In layer 0, the target character consistently attends to the same character in the previous context, suggesting that counting happens here. In Figure 3a , we show the ϵ-preimage of x 0,mid tc and x 0,post tc , where the subscript tc denotes the target character. We show ≈ 10 random samples at a single query input, but our hypotheses are based on-and easily confirmed by-rapid visual inspection of dozens of inputs across different query inputs. 4 On the left (before the MLP), the activation encodes the target character, as all samples have "g" as the target character. Count information is not sharply encoded: while the closest activation corresponds to "g" occurring 3 times, two activations corresponding to a count-4 input ("g" occurring 4 times) are also close, even closer than a count-3 input. On the other hand, on the right (after the MLP), only count-3 inputs are inside the ϵ-preimage, and count-4 inputs become much more distant than before. Comparing the ϵ-preimage before and after the MLP in layer 0, we find that the MLP makes the count information more prominent in the representational geometry of the activation. The examples are not cherry-picked; count information is generally reinforced by the MLP across query inputs.In the next layer, the colon consistently attends to the target character, and InversionView confirms that count information is moved to the colon's residual stream (Figure 3b ). More importantly, this illustrates how information is abstracted: We previously found that x 0,post tc encodes identity and frequency of the target character. However, the colon obtains only an abstracted version of the information, in which count information remains while the target character is largely (though not completely) removed. InversionView makes this process visible, by showing that the target character becomes interchangeable with little change to the activation. See more examples in Appendix D.2. Overall, with InversionView, we have found a simple algorithm by which the model makes the right prediction: In layer 0, the target character attends to all its occurrences and obtains the counts. In layer 1, the colon moves the results from the target character to its residual stream and then produces the correct prediction. Accounting for other activation sites, we find that the model implements a somewhat more nuanced algorithm, investigated in Appendix D. 4 . Overall, InversionView shows how certain information is amplified, but also how information is abstracted or forgotten.Quantitative verification. We causally verified our hypothesis using activation patching [53, 21] on (position, head output) pairs. As the attention head in layer 1 attends almost entirely to the target character, only head outputs a 0,0 tc , a 0,0 : , and a 1,0 : can possibly play a role in routing count information. We patch their outputs with activations from a contrast example flipping a single character before "|". We patch activations cumulatively, starting either at the lowest or highest layer, with some fixed ordering within each layer. For example, we patch a 0,0 : and observe how final logits change compared to the clean run, then we patch both a 0,0 : and a 0,0 tc and do the same, and so forth. By the end of patching, the model prediction will be flipped. When adding an activation to the patched set, we attribute to it the increment in the difference of LD before and after patching, where LD denotes the logit difference between original count and the count in the contrast example. Cumulative patching allows us to observe dependencies: For instance, as we hypothesize that a 1,0 : is completely dependent on a 0,0 tc , we expect that, when a 0,0 tc is already patched, patching a 1,0 :will have no further effect, whereas when a 0,0 tc is not patched, patching a 1,0 :will have a significant effect. Results (Figure 4a ) match our prediction: Patching either of the activation in the hypothesized path (a 0,0 tc and a 1,0 : ) is sufficient to absorb the entire effect on logit differences, confirming the hypothesis. See Appendix D.3 for further details and D.4 for further experiments.To test the applicability of InversionView to transformers pretrained on real-world data, we apply our method to the activations in the indirect object identification (IOI) circuit in GPT-2 small [48] discovered by Wang et al. [54] . We apply InversionView to the components of the circuit, read out the information, and compare it with the information or function that Wang et al. [54] had ingeniously inferred using a variety of tailored methods, such as patching and investigating effects on logits and has almost no effect. Bottom: On the other hand, if we patch cumulatively from right to left, a 1,0 : accounts for the whole effect while patching a 0,0 tc has no effect if a 1,0 : has been patched. So we verified that a 1,0 : solely relies on a 0,0 tc and this path is the one by which the model performs precise counting. The patching effect is averaged across the whole test set. (b) IOI. InversionView applied to Name Mover Head 9.9 at "to"; we fix the compared position to "to". Throughout the ϵ-preimage, "Justin" appears as the IO, revealing that the head encodes this name. This interpretation is confirmed across query inputs. attention. We show that InversionView unveils the information contained in the attention heads' outputs, with results agreeing with those of Wang et al. [54] .The IOI task consists of examples such as "When Mary and John went to the store, John gave a drink to", which should be completed with "Mary". We use S for the subject "John" in the main clause, IO for the indirect object "Mary" introduced in the initial subclause, S1 and S2 for the first and second occurrences of the subject, and END for the "to" after which IO should be predicted. To facilitate comparison, we denote attention heads as in Wang et al. [54] with i.j denoting h i,j . Wang et al. [54] discover a circuit of 26 attention heads in GPT-2 small and categorize them by their function. In short, GPT-2 small makes correct predictions by copying the name that occurs only once in the previous context. For InversionView, we train the decoder on the IOI examples (See details in E.1). Despite the size of the probed model, we find the same 2-layer decoder architecture as in Section 3.1 to be sufficient. We use D(z, z q ) = 1 -z•z q ||z||•||z q || (i.e., cosine distance), and ϵ = 0.1. Euclidean distance leads to similar results, but cosine distance is a better choice for this case (Appendix E.4).We start with the Name Mover Head 9.9, which Wang et al. [54] found moves the IO name to the residual stream of END. 4b shows the ϵ-preimage at "to". The samples in the ϵ-preimage share the name "Justin" as the IO. The head also shows similar activity at some other positions (Appendix A.3). Results are consistent across query inputs. Therefore, InversionView agrees with the conclusions of Wang et al. [54] on head 9.9. Applying the same analysis to other heads (Table 2 ), we recovered information in high agreement with the information that Wang et al. [54] had inferred using multiple tailored methods. For example, Wang et al. [54] found S-Inhibition heads were outputting both token signals (value of S) and position signals (position of S1) by patching these heads' outputs from a series of counterfactual datasets. These datasets are designed to disentangle the two effects, in which token and/or position information are ablated or inverted. These two kinds of information can be directly read out by InversionView (Figure 19 shows an example for an S-Inhibition head that contains position information), and there is no need to guess the possible information to design patching experiments. Overall, among the 26 attention heads that Wang et al. [54] identified, InversionView indicates a different interpretation in only 3 cases; these (0.1, 0.10, 5.9) were challenging for the methods used before (Appendix E.3). In summary, InversionView scales to larger models.We next applied InversionView to the problem of adding 3-digit numbers, between 100 and 999. Input strings have the form "B362+405=767E" or "B824+692=1516E", and are tokenized at the Figure 5 : InversionView applied to 3-digit addition: Visually inspecting sample inputs inside and outside the ϵ-preimage of the query allows us to understand what information is contained in an activation. The color on each token in generated samples denotes the difference in the token's likelihood between a conditional or unconditional decoder (Appendix G). The shade thus denotes how much the generation of the token is caused by the query activation (darker shade means a stronger dependence). In (a-c), the colored tokens are most relevant to the interpretation. We interpret two attention heads (a,b) and the output of the corresponding residual stream after attention (c). In (a), what's common throughout the ϵ-preimage is that the digits in the hundreds places are 6 and 8. Inputs outside the ϵ-preimage don't have this property. In (b), what's common is that the digits in tens places are 1, 6, or numerically close. Hence, we can infer that the activation sites a 0,0 and a 0,3 encode hundreds and tens place in the input operands respectively; the latter is needed to provide carry to A1. Also, the samples show that the activations encode commutativity since the digits at hundreds and tens place are swapped between the two operands. In (c), the output of the attention layer after residual connection combining information from the sites in (a) and (b) encodes "6" and "8" in hundreds place, and the carry from tens place. Note that a 0,1 and a 0,2 contains similar information as a 0,0 . These observations are confirmed across inputs. Taken together, InversionView reveals how information is aggregated and passed on by different model components. character level. We use F1, F2, F3 to denote the three digits of the first operand and S1, S2, S3 for the digits of the second operand, and A1, A2, A3, A4 (if it exists) for the three or four digits of the answer, and C2, C3 for the carry from tens place and ones place (i.e., C2: whether F2+S2≥10, C3: whether F3+S3≥10). Unlike [47] , we do not left-pad answers to have all the same length; hence, positional information is insufficient to determine the place value of each digit.The probed model is a decoder-only transformer (2 layers, 4 attention heads, dimension 32). We set attention dropout to 0. Other aspects are identical to GPT-2. The model is trained for autoregressive next-token prediction on the full input, in analogy to real-world language models. In testing, the model receives the tokens up to and including "=", and greedily generates up to "E". The prediction counts as correct if all generated tokens match the ground truth. The same train-test ratio as in Section 3.1 is used. The test accuracy is 98.01%. For other training details see Appendix F.1.Interpreting via InversionView and attention. As Section 3.1 we use normalized Euclidean distance for D(•, •) and the threshold ϵ = 0.1. We first trace how the model generates the first answer digit, A1, by understanding the activations at the preceding token, "=". We first examine the attention heads at "=" in the 0-th layer (Figure 5 ). As for the first head (a 0,0 ), only F1 and S1 matter in the samples -indeed, changing other digits, or swapping their order, has a negligible effect on the activation (Figure 5 ). Across different inputs, each of the three heads a 0,0 , a 0,1 , a 0,2 encode either one or both of F1 and S1 (Figure 26 ); taken together, they always encode both. This is in agreement with attention focusing on these tokens. The fourth and remaining head in layer 0 (a 0,3 ) encodes F2 and S2, which provide the carry from the tens place to the hundreds place. Combining the information from these four heads, x 0,mid consistently encodes F1 and S1; and approximately represents F2, S2-only the carry to A1 (whether F2+S2≥10) matters here (Figure 5c ). Other examples are in Figure 27 . We can summarize the function of layer 0 at "=": Three heads route F1 and S1 to the residual stream of "=" x = . The fourth head routes the carry resulting from F2 and S2. Layer 1 mainly forwards information already obtained in layer 0, and does not consistently add further information for A1. See more examples in Appendix F.2.Figure 6a shows the circuit predicting A1. InversionView allows us to diagnose an important deficiency of this circuit: Even though the ones place sometimes receives attention in layer 1, the circuit does not consistently provide the carry from the ones place to the hundreds place, which matters on certain inputs-we find that this deficiency in the circuit accounts for all mistakes made by the model (Appendix F.3). Taken together, we have provided a circuit allowing the model to predict A1 while also understanding its occasional failure in doing so correctly. Corresponding findings for A2, A3, and A4 are in Table 3 and Figure 31 . From A2 onwards, InversionView allows us to uncover how the model exhibits two different algorithms depending on whether the resulting output will have 3 or 4 digits. In particular, when predicting A3, the layer 0 circuit is the same across both cases, while the layer 1 circuit varies, since this determines whether A3 will be a tens place or ones place. Beyond figures in the Appendix, we encourage readers to verify our claims in our interactive web application.Quantitative verification. We used causal interventions to verify that information about the digits in hundreds and tens place is routed to the prediction of A1 only through the paths determined in Figure 6a , and none else. Like before, we cumulatively patch the head output on "=" preceding the target token A1, with an activation produced at the same activation site by a contrast example changing both digits in a certain place. Results shown in Figure 6b strongly support our previous conclusions. For example, a 0,3 and a 1,2 are not relevant to F1 and S1. Important heads detected by activation patching, a 0,0 , a 0,1 , a 0,2 , a 1,1 , all contain F1 and S1 according to Figure 6a . Furthermore, we can also confirm that a 1,1 relies on the output of layer 0 as depicted in sub-figure (a): When heads in layer 0 are already patched, patching a 1.1 has no further effect (value corresponding to → is zero), but it has an effect when patching in the opposite direction. On the contrary, a 1,0 shows little dependence on layer 0, consistent with Figure 6a . On the right of Figure 6b , we can confirm that a 0,3 is important for routing F2 and S2, and the downstream heads in layer 1 rely on it. Findings for other answer digits are similar (See Appendix F.5). Overall, the full algorithm obtained by InversionView is well-supported by causal interventions. InversionView reveals granularity of information. Heads often read from both digits of a place, but only the sum matters for addition. Are the digits represented separately, or only as their sum? Unlike traditional probing, InversionView answers this question without designing tailored probing tasks. In Figure 7 (left), a 0,2 exactly represents F2 and S2 (here, 2 and 5). Other inputs where F1+S1=5+2 have high D. In contrast, on the right, F2 and S2 are represented only by their sum: throughout the ϵ-preimage, F2+S2=9. In fact, we find such sumonly encoding only when F2+S2=9a special case where the ones place of operands affects the hundreds place of the answer via cascading carry. We hypothesize that the model encodes them similarly because these inputs require special treatment. Therefore, even though encoding number pairs by their sum is a good strategy for the addition task from a human perspective, the model only does it as needed. We also observe intermediate cases (Figure 29 ).To test whether InversionView can be applied to larger language models, we explore how GPT-2 XL (1.5B parameters) performs the task of recalling factual associations. In this case study, our intention is not to provide a full interpretation of the computations performed to solve this task, which we deem out of scope for this paper. Instead, we show that InversionView produces interpretable results on larger models by focusing on a relatively small set of important attention heads in upper layers. The decoder model in this case study is based on GPT-2 Medium, because we expect a more complex inverse mapping from activation to inputs to be learned. We observe the resulting ϵ-preimage can express high-level knowledge (Figure 39 -44), and sometimes can predict the failure of the model (Appendix H.6). Using InversionView, we again shed light on the underlying mechanism of the model. We present detailed findings in Appendix H.As the decoder outputs an input but not the position of the activation, we then assign the position minimizing D(•, •) to the query activation. Usually, there is only one position with a small D(•, •), matching the structural position (not necessarily the absolute position) of the position the query activation was taken from (e.g., the target character in Figure 3a ). In certain cases, we visualize D(•, •) for an activation from a position not minimizing D(•, •) for expository purposes. For example, in Figure 3b , because the target character exclusively attends to itself in layer 1, resulting a 1,0 tc ≈ a 1,0 : , so sometimes the metric value of a 1,0 tc is smaller than a 1,0 : . Throughout the appendix and our web application, we use italic font and rounded bars to visualize D(•, •) in such cases.We also find that selecting the position minimizing D(•, •) can reveal that components are active in similar ways at other positions than the one originally investigated. For example, in Figure 9 , we can see sometimes "gave" is selected. This is reasonable, because the IO is also likely to appear right after "gave" and the head needs to move the IO name for this prediction. We can see that activation at the period "." can also be somewhat similar to the query activation, this is not surprising. Because the model needs to predict the subject for the next sentence and copying a name from the previous context is helpful. In summary, the copying mechanism can be triggered in circumstances different from IOI, selecting position minimizing D(•, •) reveals more information about this. 
discussion and related work: 
 Comparison with other Interpretation Methods Supervised probing classifiers, assessing how much information about a variable of interest is encoded in an activation site, are arguably the most common method for uncovering information from activations [e.g. 2, 6, 5, 4, 55, 52, 33, 34] . It requires a hypothesis in advance and is thus inherently limited to hypotheses conceived a priori by the researcher. InversionView, on the other hand, helps researchers form hypotheses without any need for prior guesses, and allows fine-grained per-activation interpretation. Inspecting attention patterns [e.g. 12] is a traditional approach to inferring information flow, and we have drawn on it in our analyses. More recently, path patching [54, 26, 13, 28, 35] causally identifies paths along which information flows. While the information flow provides an upper bound on the information passed along by tracing back to the input token, it is insufficient for determining how information is processed and abstracted. For instance, in Section 3.1, occurrences of the target character are causally connected to a 0,0 tc , which then connects to a 1,0 :(direct or mediated by MLP layer 0). Without looking at encoded information, we only know that the information in these paths is related to the occurrences of the target character, but not whether it is their identity, positions, count, etc. More generally, when a component reads a component that itself has read from multiple components, connectivity does not tell us which pieces of information are passed on. Similar considerations apply to other intervention methods.Geva et al. [24] intervene on attention weights to study information flow. Activation patching, a causal intervention method, can be used to study the causal effect of an activation on the output, and can help localize where information is stored [37, 49] , or find alignment between a high-level causal model and inner states of a neural model [21, 22, 57] . Many recent works obtain insights about information content by projecting representations or parameters into the vocabulary space [42, 7, 44, 31, 54, 23, 24, 16] . This technique is sometimes referred to as Direct Logit Attribution (DLA). We argue that DLA is only suitable for studying model components that directly affect model's final output. For those components whose effect is mediated by other components, their output information is meant to be read by a downstream component, thus not necessarily visible when projecting to the vocabulary space. We provide further discussion in Appendix I. Generalizing this approach, Ghandeharioun et al. [25] patch activations into an LLM. Another line of recent research [10, 51, 14] decomposes activations into interpretable features using sparse autoencoders. Some other interpretation methods also generate in input space, but differ from InversionView in goals and methods. This includes feature visualization [43, 41] , adversarial or counterfactual example generation [27, 59, 46, 45] , and GAN inversion methods [58] . We discuss the similarities and differences of these works compared to InversionView in Appendix I.4.InversionView offers distinctive advantages and makes analyses feasible that are otherwise very hard to do with other methods. It can also improve the interpretability workflow in coordination with other methods. For example, one may first use methods such as path patching or attribution [50, 18] to localize activity to specific components, and then understand the function of these components using InversionView. In sum, InversionView is worth adding to the toolbox of interpretability research.Transformer Circuits for Arithmetic Related to Section 3.3, [47] interpret the algorithm implemented by a 1-layer 3-head transformer for n-digit addition (n ∈ {5, 10, 15}), finding that the model implements the usual addition algorithm with restrictions on carry propagation. In their one-layer setup, attention patterns are sufficient for generating hypotheses. Lengths of operands and results are fixed by prepending 0. Our results, in contrast, elucidate a more complex algorithm computed by a two-layer transformer on a more realistic version without padding, which requires the model to determine which place it is predicting. We also contribute by providing a detailed interpretation, including how digits are represented in activations.Automated Interpretation for InversionView Recent work has started using LLMs to generate interpretations [8, 10] . The samples produced by InversionView can be easily fed into LLMs for automated interpretation. We show a proof of concept by using Claude 3 to interpret the model trained for 3-digit addition. See results in Table 5 . The LLM-provided interpretation reflects the main information in almost all cases of the addition task. Despite some flaws, the outcome is informative in general, suggesting this as a promising direction for further speeding up hypothesis generation.Limitations InversionView relies on a black-box decoder, which needs to be trained using relevant inputs and whose completeness needs to be validated by counter-examples. Also, InversionView, while easing the human's task, is still not automated, and interpretation can be laborious when there are many activation sites. We focus on models up to 1.5B parameters; scaling the technique to large models is an interesting problem for future work, which will likely require advances in localizing behavior to a tractable number of components of interest. Fourth, interpretation uses a metric D(•, •). The geometry, however, in general could be nonisotropic and treating each dimension equally could be sub-optimal. We leave the exploration of this to future work. 
conclusion: 
 We present InversionView, an effective method for decoding information from neural activations. In four case studies-character counting, IOI, 3-digit addition, and factual recall-we showcase how it can reveal various types of information, thus facilitating reverse-engineering of algorithm implemented by neural networks. Moreover, we compare it with other interpretability methods and show its unique advantages. We also show that the results given by InversionView can in principle be interpreted automatically by LLMs, which opens up possibilities for a more automated workflow. This paper only explores a fraction of the opportunities this method offers. Future work could apply it to subspaces of residual stream, to larger models, or to different modalities such as vision. Here, we illustrate the importance of inspecting ϵ-preimages up to the threshold ϵ, rather than just top-k nearest neighbors of the query activation. In Figure 8 , an initial glance at the samples on the left may suggest that the residual stream of "+" encodes F1 and F2. However, observing a broader neighborhood (as depicted on the right) reveals that this conclusion is not even robust to tiny perturbations of the activation. Indeed, after a more comprehensive calculation over all possible x 0,post + , we find that the maximum possible metric value between any pair of x 0,post + is 0.0184. So for any ϵ ≥ 0.0184 the ϵ-preimage covers the entire input space. Hence, the activation is unlikely to contain usable information.We further prove this by causal intervention. We found that x 0,post + has no effect on the model's output. Concretely we patch x 0,post + with its mean on the test set (mean ablation [54] ) and for each prediction target (A1, A2 etc.), we compare 1) the KL divergence between the distribution before and after patching. 2) logit decrement rate, which is the difference between the maximum logit value before patching and the logit value of the same target token after patching, divided by the former. E.g., 1.0 means the logit is reduced to zero (assuming it is originally positive). The results are shown in Table 1 . We can see the effect of x 0,post+ is negligible. A1 A2 A3 A4/E KL divergence 8.1 × 10 -8 5.4 × 10 -7 2.5 × 10 -8 3.6 × 10 -9 Logit decrement rate -2.3 × 10 -5 8.7 × 10 -7 1.4 × 10 -5 2.8 × 10 -7Table 1 : Activation patching results for x 0,post + . 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not see any risk of misuse of our research results.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We use the GPT-2 Small model and IOI templates, both of which we cite. We use no other existing models or data.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We document the new assets, which are models of character counting and addition.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In Section 2, we mentioned that the distribution p(x|z q ) is modeled by the decoder. Strictly speaking, p(x|z q ) represents the data distribution in the ϵ-preimage defined by ϵ = 0. For example, when the probed model is using causal masking, and a certain activation is relevant to all previous context (by non-zero attention weights), then p(x|z q ) is the distribution over those inputs that share the same previous context (i.e., they have same prefix). This requires that the decoder can distinguish any tiny difference in activation and decode the full information (imagine a token attended with 0.0001 attention weight). Such a decoder must be very powerful and perhaps trained without any regularization. But in practice, the decoder is a continuous function of activation and tiny changes in activation are not perceivable by the decoder. We observe that the decoder rarely generates the sample that lies at the same point (producing the same activation) as the query input in vector space, instead it usually generates samples that are in the neighborhood of the query input. Because we need to observe the whole neighborhood of the query input and prevent samples from being too concentrated, we adjust the sampling temperature to control how concentrated they are. Importantly, even if the decoder is too powerful and can always recover the same activation, we can still obtain the neighborhood by adding random noise to the query activation before giving it to the decoder. This motivates decoding with temperature and noise, as described in the next paragraph.Increasing Coverage by Temperature and Noise. In our experiments, we use both ways to control the generation, i.e., by adjusting the temperature and adding random noise to the query activation. We denote temperature as τ and noise coefficient as η. The noise vector consists of independent random variables sampled from the standard normal distribution and then multiplied by std(z q ) • η where std(•) stands for standard deviation. In our web application, we provide multiple sampling configurations: four configurations in which τ = {0.5, 1.0, 2.0, 4.0} and η = 0.0 (only for addition task); one figuration named "Auto" which is sampled by following procedure: we iterate over a few predefined τ (ranging from 0.5 to 2.0) and η (0.0 or 0.1) and sample a certain amount of inputs (e.g., 250) for each parameter combination. We then calculate the metric value for all inputs collected from different sampling configurations. We then randomly choose a small part of them (100) with different probability for in-ϵ-preimage inputs and out-of-ϵ-preimage inputs. We dynamically adjust the probability such that the in-ϵ-preimage inputs account for 60%-80% of the chosen set of inputs (when this is possible). Note that we use different noise in factual recall task, which will be described later in Appendix H.4.When inspecting the samples, we choose a configuration for which the distances D(•, •) to the query activation best cover the interval [0, ϵ]. The choice is usually specific to the activation site that we are inspecting and can be performed manually in the web application. •, • ) is in parentheses. The head also copies the name "Justin" in other circumstances, e.g., at "gave". The name "Justin" is always containedA.One question people may have is whether our conclusion about the information in activation depends significantly on the threshold we choose. To address this potential concern, we show more details about the geometry of the vector space in Figure 10 . On the one hand, we can see that with different thresholds ϵ we can make different conclusions about the query activation. On the other hand, the conclusions made with different thresholds are "in alignment". In other words, the conclusions do not differ fundamentally, instead, the difference between them is about granularity or the amount of details being ignored. Specifically, in Figure 10a , ϵ 1 results in the conclusion that the count is 5, the target character is either 't' or 'm', and also approximate sequence length is retained. ϵ 2 results in a conclusion only about the count and the sequence length. In Figure 10b , if we set the threshold to ϵ 1 (i.e., a value between 0.000 and 0.009), the obtained information will be F1=5, S1=7. If we set the threshold to ϵ 2 , the information will be 5 and 7 are in the hundreds place. If we set the threshold to ϵ 3 the information will be "5 is in the hundreds place". In Figure 10c , ϵ 1 results in conclusion that 9, 8 are in hundreds place and 2, 7 are in tens place; ϵ 2 results in conclusion that 9, 8 are in hundreds place and F2+S2=9; ϵ 3 results in conclusion that F1+S1≈17 and F2+S2≈9. Therefore, changing the threshold value will not lead us in a different direction, because the ϵ-preimage is based on the same underlying geometry. In practice, rather than selecting a threshold first and treating inputs in a black-and-white manner, we first observe the geometry of the vector space and obtain a broad understanding of the encoded information, then choose a reasonable threshold that best summarizes our findings. In other words, the threshold value is used to simplify our findings so that we can focus more on the big picture of the model's overall algorithm, and it should also be set according to the difference that is likely to be readable for the model. As the interpretation progresses, one can see if the chosen threshold leads to a plausible algorithm and can adjust it if necessary. Finally, verification experiments are conducted to verify the hypothesis.In Section 2, we described that an ideal strategy for obtaining samples in the ϵ-preimage satisfies two desiderata: it only provides samples that are indeed within the ϵ-preimage (Correctness), and it provides all such samples at reasonable probability (Completeness). As further described there, we can directly ensure Correctness by evaluating D(•) for every sample. Ensuring completeness is more challenging, due to the exponential size of the input space; the most general approach is to design counterexamples not satisfying a hypothesis about the content of the ϵ-preimage, and verifying that D(•) is indeed large.Here, we provide a direct test of completeness in one domain (3-digit addition). The primary concern with completeness is that, if some groups of inputs in B z q ,f,ϵ are systematically missing from the generated samples, one may overestimate the information contained in activations. To see if may happen in reality, we plotted log-probability against distance, each of which includes all inputs in 3 digit addition task, as shown in Figure 11 . We next evaluated the sampling probability for different sampling configurations, as described in Appendix A.2. When adding noise, we calculate probability of an input using a Monte Carlo estimate: Concretely, because the probability of inputs is conditioned on the noise vector added to the query activation, we randomly sample 500 noise vectors from the normal distribution (with the standard deviation described in Appendix A.2) and calculate input probability given these noise vectors, then average to obtain the estimated probability, and then compute the logarithm.Across setups, we can see that there is a triangular blank area in the bottom left corner, i.e., the bottom left frontier stretches from the upper left towards the lower right. In all sub-figures, not a single input close to the query input is assigned disproportionately low probability. All inputs in the ϵ-preimage (the dots on the left of the red vertical line) are reasonably likely to be sampled from the decoder, with probability decreasing as the input becomes more distant, alleviating concerns about completeness for these query activations. On the other hand, some inputs distant from the query input are also likely to appear in samples, but this is not a problem for our approach, as we can easily tell that they are not in the ϵ-preimage by calculating the distance (correctness is ensured).We can see that sometimes the distribution of the decoder itself (at temperature 1 and no noise) is quite sharp, and in-ϵ-preimage inputs can have low probability as they are near the boundary of preimage. By comparing the sub-figures, we can see both increasing the temperature and adding noise substantially smooth the distribution within the ϵ-preimage, lowering the difference of the probability of inputs that are at similar distance.The overall training and sampling pipelines are shown in Figure 2 . In this section, we describe the architecture of the decoder model in detail.The decoder model is basically a decoder-only transformer combined with some additional MLP layers. In order to condition the decoder on the query activation, the query activation is first passed through a stack of MLP layers to decode information depending on the activation sites and then made available to each attention layer of the transformer part of the decoder, as depicted in Figure 12 .Processing Query Activation. The query activation z q ∈ R d is first concatenated with a trainable activation site embedding e act ∈ R dsite , producing the intermediate representationz (0) = [z q ; e act ].We chose d site to be the number of possible activation sites in the training set. The resultz (0) is (a) (b) (c)Figure 11 : Addition Task: Exhaustive verification of the decoder's completeness for 8 random query activation. Failure of completeness would mean that some inputs result in an activation very close to the query activation but nonetheless are assigned very small probability. Here, we show that this does not happen, by verifying that all inputs within the ϵ-preimage are assigned higher probability by the decoder than most other inputs. We also show that by increasing the temperature and adding random noise, we can increase the probability of inputs near the boundary of ϵ-preimage. EQUATIONwhere LN represents layer normalization. For each MLP layer, the input, hidden and output dimensions are d + d site , d, and d + d site , respectively. The activation function is ReLU. There is also a final layer normalization, z (f n) = LN(z (P ) ).Integrating Query Activation. As we want to make the query activation available to each attention layer of the decoder, we separately customize it to the needs of each layer using a linear layer. That is, for each layer of the transformer part of the decoder (indexed by ℓ, so ℓ ∈ {0, 1,• • • , L -1}),we define a linear layer Linear (ℓ) : R d+dsite → R d decoder and a layer normalization LN (ℓ) , where d decoder is the model dimension of the decoder model:EQUATIONWe add superscript (ℓ) to the model components to emphasize they are layer-specific. In the ℓ-th layer of the transformer part, ẑ(ℓ) is concatenated with the input of the attention layer along the length axis before computing keys and values, so that each attention head can also attend to ẑ(ℓ) in addition to the context tokens. This means that each head in the ℓ-th layer, instead of attending to xEQUATION2 , ..., now computes its attention weights over ẑ(l) , x(ℓ) 1 , x (ℓ) 2 , .... Here x (ℓ)t is the residual stream corresponding to the t-th token input into the ℓ-th layer.Motivation for Architecture Design. At the beginning, we train separate decoders for each activation site, but this is not very scalable when there are many activation sites. In the architecture above mentioned, we use activation site embedding e act as a signal to trigger different processing, and functions or model components that are needed for all activation sites are shared. Similar reason applies to the linear layer Linear (ℓ) , we expect the z q should be transformed differently for each transformer layer, but having separate MLP stacks to process z q for each layer would largely increase the number of parameters. In preliminary experiments, we also try "encoder-decoder" attention layer. That is, instead of providing query activation in self attention layer, we add new attention layers that analogous to the encoder-decoder attention layer in original transformer architecture, where each token can attend to the processed query activation as well as a blank representation (similar to the function of "BOS", so that "no-op" is possible). However, we do not find significant difference between this design and the aforementioned one. Therefore, other than adding components for processing z q , we do not modify the decoder-only transformer architecture, so that we can also choose to use pretrained models. We note that there are other possible choices for conditioning the generation on the activation, and we didn't optimize this choice thoroughly.Decoder Hyperparameters. Regarding processing query activation, the decoder has 6 MLP layers, i.e., P = 6. The decoder model has 2 transformer layers (L = 2), 4 heads per layer, and a model dimension of 256. The attention dropout rate is 0. Other settings are the same as the GPT-2. We use the this architecture for the 3 tasks-character counting, IOI, and 3-digit addition-in the paper. Regarding the factual recall task, we use the same architecture for processing query activation, i.e. P = 6, and use pretrained GPT-2 Medium (24 layers) as the transformer part of the decoder.Training Details. We construct the training dataset by feeding in-domain inputs to the probed model, and collect activations from random activation sites and random position as query activations (the choice of activation sites and position is specific to each task and is described later), we also record the activation site they come from. For each input, we could obtain many possible training examples because of many choices of activation sites and position. So we do not iterate over all possible training examples. We sample certain amount of examples to train the decoder for 1 epoch, using constant learning rate of 0.0001 and AdamW optimizer with weight decay of 0.01. Other details (e.g., amount of examples, training steps) are task-specific, and can be found later in their own section.During training, we regularly calculate the in-preimage rate, which serves as a proxy for generation quality. Concretely, for a fixed set of query activations used for testing, the decoder generates samples with temperature=1, we then compute the fraction of samples inside of ϵ-preimage (with ϵ = 0.1, D as normalized Euclidean distance). The rate is calculated for each activation site. We usually observe difference between the ratios for each activation site, indicating some inverse mappings are easier to learn (we also observe these activation sites tend to have clearer information). Overall, we usually see a continuous improvement on the average rate during training.To construct the dataset, we enumerate all the 3-combinations from the set of lowercase characters defined in ASCII. For each combination, we generate 600 distinct data points by varying the occurrence of each character and the order of the string. The occurrences are sampled uniformly from 1-9 (both inclusive). So the length of the part before the pipe symbol ("|") lies in [3, 27] (not considering "B"). Like 3-digit addition, we split the dataset into train and test sets, which account for 75% and 25% of all data respectively. The input is tokenized on the character level.The model is a two-layer transformer with one head in each layer, the model dimension is 64. All dropout rates are set to 0. The model is trained with cross-entropy loss on the last token, the answer of the counting task. The model is trained with a batch size of 128 for 100 epochs, using a constant learning rate of 0.0005, weight decay of 0.01, and AdamW [36] optimizer. The training loss is shown in Figure 13 , we can see the stair-like pattern. An interesting future direction is to investigate what happens when the loss rapidly decreases using InversionView.With regard to the decoder model, the architecture is described in C. We select x 0,pre , x i,mid , x i,post , a i,j , m i as the set of activation sites we are interested in, where i ∈ {0, 1}, j ∈ {0}, and m denotes MLP layer output. The query activation is sampled from those activations corresponding to only the target character and colon. We sample 100 million training examples (all activation sites are included) and train the decoder with batch size of 512, resulting in roughly 200K steps. During training, as we mentioned before in C, we test the generation quality by measuring in-preimage rate. For those activation sites for which the decoder has a low generation quality, we increase their probability of being sampled in the training data. The final in-preimage rate averaged across activation sites is 67.7%.See Figures 14 and 15 . Here, we show results similar to Figure 3 for other query inputs. : . As we mentioned, we hypothesize that the attention head is reading the subspace where the count information is stored. One can presumably find this "count subspace" by optimizing a projection matrix such that after projecting the activation there is only pure count information in the ϵ-preimage, and compare it with the subspace read by the value matrix of the attention head. Therefore, InversionView can be potentially useful for subspace study. For an input example x orig with t orig as the count (final token), we construct a contrast example x con with a different count t con by changing a random character before "|". The contrast example is a valid input (the count token is the count of the target character). We also ensure that the contrast example is within the dataset distribution (the count is in the range [1] [2] [3] [4] [5] [6] [7] [8] [9] and there are 3 distinct characters in the input).We run three forward passes. 1) The model takes as input x orig and produces logit values for count prediction, we record the logit difference LD orig between t orig and t con (former minus latter). 2) We feed the model with x con and store all activations. 3) We run a forward pass using x orig , replacing the interested activations (e.g, {a 0,0 : , a 0,0 tc }) with the stored activation in the same position and activation sites, and record the new logit difference LD pch . Because the model can make the right prediction in most cases, we can see that average LD pch changes from positive to negative values as we patch more and more activations. We do the same for all inputs in the test set and report the average results.Figure 16 shows the LD orig and LD pch . We cumulatively patch the activations we study. For example, on the top of the figure, we patch {a 0,0 : }, {a 0,0 : , a 0,0 tc }, {a 0,0 : , a 0,0 tc , a 1,0 : } respectively. Patching more activation results in increases of LD orig -LD pch , we attribute the increment to the newly patched activation. Hence, the causal effect of each activation is measured conditioned on some activations already being patched.We sort the activations according to their layer indices and show the results of patching from bottom to top (→) and from top to bottom (←). In this way, we can verify the dependence between activations in the top and bottom layer. For example, at the top of Figure 16 , when a 0,0 tc is already patched, patching a 1,0 : has almost no effect. On the bottom, we also see patching a 0,0 tc has no effect if a 1,0 : has been patched. So we verified that a 0,0 tc is the only upstream activation that a 1,0 : relies on, and a 1,0 : is the only downstream activation that reads a 0,0 tc . In Section 3.1, we verified the information flow by an activation patching experiment in which the contrast example only differs by one character. These experiments verified that the algorithm we described is complete in distinguishing between such minimally different contrast examples. We now show that the model implements a somewhat more complex algorithm that combines this algorithm with position-based cues, which become visible once we consider contrast example that differ in more than one character, in particular, those that differ in length.To show this, we conduct another activation patching experiment in which the contrast example is a random example in the dataset with a different count. In other words, everything can be different in contrast examples, including the sequence length and the target character. Thus, we cumulatively patch four places:1. e pos tc and e pos : , where e pos stands for position embedding, because the final count correlates with positional signal, so the model may utilize it. They are patched together because the attention pattern of the colon in layer 1 relies on their adjacency.2. a 0,0 tc (as before) and e tkn tc , where e tkn stands for token embedding. They are patched together because patching only one of them would result in a conflict between character information in the patched and the un-patched activation; 3. 3) a 0,0 : ; (as before) 4. a 1,0 : (as before).The result is shown on the left of Figure 17 . We can see that, besides the components we had detected previously based on minimal contrast examples (a 0,0 tc , a 1,0 : ), some other signal also contributes notably to the final logits. We compare with patching experiments for the same set of activations on contrast examples that differ in one character, shown on the right of Figure 17 .Overall, besides the algorithm identified in Section 3.1, we find other 3 sources of information influencing the model's output. 1) The position embedding, e pos tc and e pos : . This is observable on the left of Figure 17 , from which we can also know a 1,0 : contains the position information (because the bars of e pos tc and e pos : are not symmetric). This is confirmed by InversionView. As shown in Figure 18 , we see the inputs in ϵ-preimage roughly follow the query input length, being independent of the count. Therefore, the model is also utilizing the correlation between the count and the sequence length. 2) Attention output of colon, a 0,0 : , which attends to all previous token equally. From InversionView, we observe it contains fuzzy information about the length (same as position signal), and the characters that occur in the context, as well as their approximate count. Our causal experiment also shows that it does not contain a precise count. Therefore, it contributes to the model's prediction in manner similar to the position signal. 3) Attention output of the pipe sign, a 0,0 | . From the attention pattern we observe sometimes in layer 0, pipe sign attends selectively to one type of character, e.g. "x", "k", or "j". InversionView shows that it indeed contains the approximate count in that case (though the decoder has not been trained on activation corresponding to pipe sign). In next layer, the colon also attends to the pipe sign if target character is the same as the character attended by pipe sign in layer 0. This explains why we can observe nonzero effect of patching a 1,0 : when other activation is already patched (the red bar corresponding to a 1,0 : on both sub-figures of Figure 17 ). In order to train a decoder model, we construct a dataset that consists of IOI examples. We used the templates of IOI examples from ACDC [13] [A]" will be replaced by two random names (one token name), "[PLACE]" and "[OBJECT]" will also be replaced by random item from the predefined set. Besides "BABA" template (i.e., S is before IO) we also use "ABBA" templates (S is after IO) by swapping the first "[B]" and "[A]". We generate 250k data points.The architecture of the decoder model is the same as before, as described in Appendix C. The set of activation sites the decoder is trained on consists of the output of all attention heads and MLP layers (no residual stream). Note that when producing query activation using GPT-2, we always add the "< |endoftext| >" token as the BOS token. We do so as during the training of GPT-2 this or multiple such tokens that usually appear in the previous context can be used as a BOS token, which is possibly important to the model's functioning. We use a new token "[EOS]" as the EOS token when training the decoder. The query activation is sampled uniformly from all positions excluding EOS and padding tokens, and uniformly from all activation sites the decoder is trained for. We sample 20 million training examples (all activation sites are included) and train the decoder with batch size of The qualitative examination results is shown in Table 2 . We summarize the description in [54] of each head category to facilitate comparison. Figure 22 shows the IOI circuit in GPT-2 small, which is taken from their paper. For more details, please refer to [54] .There are some heads for which InversionView indicates a different interpretation. First, Head 0.1 and 0.10: [54] only shows that they usually attend to the previous occurrence of a duplicate token and validates the attention pattern on different datasets. However, there is no evidence for the information moved by these heads. Thus they only hypothesize that the position of previous occurrence is copied. Second, Head 5.9: The path patching experiments in [54] show that head 5.9 influences the final logits notably via S-Inhibition Heads' keys. But there are no further experiments to explain the concrete function of this head. While the authors refer to it as a Fuzzy Induction Head, the induction score (measured by the attention weight from a token T to the token after T 's last occurrence) of this head shows a very weak induction pattern. Even if such pattern occurs, it cannot tell us what Figure 21 : ϵ-preimage of Induction Head 5.5. Position -but not identity -of the current token (token in parenthesis)'s last occurrence is contained in head output Figure 22 : IOI circuit in GPT-2 small. Figure 2 from [54] information is captured by this head. Interpretation with InversionView suggests that the head barely contains any information, within the input space of IOI-like patterns. One possibility is that head 5.9 recognizes the IOI pattern (i.e., there are two names and one is duplicated in the previous context), so that if an IOI-like pattern exists, S2 should be attended to by S-Inhibition heads. As the decoder model is trained on IOI examples and generates mostly IOI examples -that is, the input space X in (1) consists of IOI-like inputs, this information is by definition not visible. Expanding the input space to arbitrary language modeling would allow capturing such information; we leave this to future work.As described in Section 3.2, we used cosine distance for the IOI task, while for the other two tasks, we use normalized Euclidean distance. In this section, we show that both distance metrics produce similar interpretations while cosine distance makes the meaningful patterns easier to identify.In Figure 23 and 24 we show two examples of using normalized Euclidean distance as the distance metric. Readers can see more examples on our web application. We see that the samples generated by the decoder tend to have larger distances under normalized Euclidean distance. Nonetheless, we can still see the top-ranked samples show meaningful commonalities -indeed if we set ϵ = 0.4, we obtain the same interpretation as the one we obtain based on Figure 4b and 19 .While normalized Euclidean distance can produce a similar interpretation, we have to set a much larger ϵ, which we find less intuitive. We believe this to be because of the differences in the activation dimensionality, which is 768 in the IOI task, much larger than in character counting (64) and addition (32) tasks: Under Euclidean distance, the ratio of all close samples to all possible samples becomes lower and lower when the dimension becomes higher. In other words, the volume of the ϵ-preimage accounts for a very tiny proportion of the whole space in the high-dimensional case. By using cosine Column "Position" means the query activation is taken from that position. "S1+1" means the token right after S1. Rows are ordered according to the narration in the original paper. When we say "S name", it means the the name of S in the query input, but the name is not necessarily S in the samples. This also applies to "IO name". The information learned by InversionView which is different from the information suggested by Wang et al. [54] is in bold. 1. While at ϵ = 0.1, some query inputs do result in a substantial number of in-ϵ-preimage samples, many cases result in very small or even empty (as here) sample sets, suggesting that the ϵ-preimage at 0.1 -at least as accessible to the decoder -is extremely small in this model, which we speculate is related to the models higher dimensionality compared to the other tasks (768 vs 32/64), which tends to make Euclidean distances large except for extremely similar vectors (see Appendix E.4 for more discussion). We find it more convenient and natural to use similar thresholds (ϵ = 0.1) across tasks and account for the different geometries using different distance metrics. What is key, however, is that for an appropriately higher ϵ (e.g., ϵ = 0.4) we again always obtain a substantial number of samples, and -most importantly -these samples lead to the same interpretation as we obtained in our main experiments with the cosine similarity. Here, for example, we can obtain the same interpretation by setting ϵ = 0.4, i.e., the IO name is encoded in the activation. similarity, we allow more input samples to lie in a close distance with the query input, as cosine similarity ignores the magnitude difference. This suggests cosine similarity may overall be more suitable when applying InversionView in high-dimensional activations.F 3-Digit Addition: More Details and ExamplesWe constructed 810,000 instances and applied a random 75%-25% train-test split. The probed model is trained with a constant learning rate of 0.0001, a batch size of 512, for 50 epochs (59350 steps). We save the last checkpoint as the trained model, and test it on the test set. We use the AdamW [36] optimizer with weight decay of 0.01. The training loss is shown in Figure 25 .With regard to the decoder model, we select x 0,pre , x i,mid , x i,post a i,j as the set of activation sites we are interested, where i ∈ {0, 1}, j ∈ {0, 1, 2, 3}. When sampling training data, we select an activation site and a token position uniformly at random. We sample 100 million training examples (all activation sites are included) and train the decoder with batch size of 512, resulting in roughly 200K steps. The final average in-preimage rate is 88.6%.See Figures 26 to 29.5 In Section 3.3, we mention that there is no firm and clear path of obtaining the carry from ones to tens, so the model may make wrong prediction. We examine those instances for which the model makes wrong prediction and find they all satisfy one condition: F2+S2=9. In other words, it fails to make the right prediction because the ones place matters. This is consistent with our interpretation of the model. Furthermore, we check the model's accuracy on this special subset where F2+S2=9, and find that it is significantly higher than chance level. The accuracy on training subset (training data where F2+S2=9) is 80.45%, and on test subset is 80.06%, while chance level is 50%. So, we can infer that the probed model obtains some information about the ones place by means other than memorization. Indeed, we observe fuzzy information about ones place in a 1,0 and a 1,1 occasionally (See Figure 30 ). We present our overall qualitative results in Figure 31 and Table 3 . We have found that the model obtains required digits by attention, and primary digits are assigned more heads than secondary digits, e.g., a 0,0 , a 0,1 , a 0,2 for hundreds and a 0,3 for tens when predicting A1. More importantly, the primary digits are encoded precisely while the secondary digits are encoded approximately in the residual stream. In addition, the model routes the information differently based on whether A1=1, i.e., the length of the answer is 3 or 4. When predicting A2, this information is known before layer 0, thus paths differ from the start. On the contrary, when predicting A3, the information is obtained in layer 0, thus paths differ only in layer 1. Furthermore, in Figure 31 We use similar activation patching method as character counting task, described in D. x 0,post same as x 0,mid same as x 0,mid same as x 0,mid same as x 0,mid a If A1=1: F1 and S1 (sometimes fuzzy); C2 Else: F2 and S2x 1,mid same as x 0,mid If A1=1: info from x 0,mid + C2 Else: same as x 0,mid If A1=1: same as x 0,mid Else: info from x 0,mid + F3 and S3 same as x 0,mid x 1,post same as x 0,mid same as x 1,mid except that current input token A1 is less important same as x 1,mid except that current input token A2 is less important same as x 0,mid except that current input token A3 is blurred Table 3 : Summary of our observations for each activation site and position. "same as" denotes that there is no obvious difference between the two sites for indicated position. In order to verify the paths responsible for routing each type of information, we construct contrast examples as follows: Given a prediction target (e.g., A1), a set of tokens that can be changed T chg (corresponds to a certain type information, e.g., F1 and S1), we construct a contrast example x con that contains a different token t con as prediction target by changing tokens in T chg . Note that the contrast example still follows the rule that the answer is the sum of the two operands.We now give a detailed explanation for Figure 6b shown in the main paper, in which prediction target is A1 and we patch head output corresponding to the preceding token "=". On the left of Figure 6b , T chg = {F1, S1}. So in the contrast examples the F1 and S1 are changed and other digits in operands remains the same. In the third run where we calculate LD pch , activations are replaced by new activations from contrast example, so the new activations contain modified F1 and S1 information. Therefore, for activations that contributes to routing F1 and S1 (e.g., a 0,0 = ), patching them with new activations can effect model's prediction. On the contrary, patching a 0,3 = has no effect Overall, among all the intervention experiments and their corresponding information flow diagrams in Figure 31 , the activation with the highest increment not included in the information flow diagram is a 1,1 A1 in Figure 32 (right), accounting for only 5.92% of the cumulative increment. In this sense, the information flow diagrams coupled with interpretations present an almost exhaustive characterization of the algorithm used by the model to predict the answer digits. 31 : predicting A3 when A1 = 1. We exclude those data in x orig where A1̸ =1. The constructed contrast data x con also satisfies this constraint. Left: T chg = {F2, S2}. Right: T chg = {F3, S3}. We further exclude those data in x orig where F1+S1=9 and F2+S2=9 because we cannot find a contrast example in those cases. In figures of addition task, we also show the decoder likelihood difference for each token in generated samples. It indicates what might be relevant to the activation. It is calculated as follows: During generation, we sample the next token from the distribution produced by the decoder model, and we record the probability of the sampled token in that distribution. We denote it as p act . Then, we run the decoder again with the same input tokens, but this time it is fed with a "blank" activation (the activation corresponding to BOS token from the same activation site: because of the causal masking of the probed model, this activation does not contain any information). Therefore, it produces a different distribution. The probability of the same token (the token already sampled in the normal run) in the new distribution is p blank . The difference p act -p blank indicates if the decoder model can be more confident about a token when it receives the information from the activation. If p act -p blank > 0, the color is blue, and if p act -p blank < 0 it is red. The depth of color is proportional to the magnitude of the value. Therefore, it highlights what can be learned from the query activation, in addition to what is in the context. Note that the decoder has learned how to handle the activation of BOS, since it also appears in the training set because we sample uniformly at random among all tokens in the input (as in IOI and addition task).Importantly, unlike the distance metric value, the decoder likelihood difference depends on the capability of the decoder model, and we should bear in mind that it might be inaccurate. We caution that this difference does not necessarily highlight the directly relevant part, complicating its interpretation. As shown in Figure 37 , the color highlights digits in the second operand, which does not reflect the actual flow of information. In these examples the query activation corresponds to digits in answer. For example, in the left most sub-figure, x 0,pre contains the information "6 is at the position of A1", but the color does not highlight A1. This is because, on one hand, decoder predicts S1 conditioned on F1, so knowing their sum will significantly increase the confidence of S1, and S1 is highlighted. On the other hand, when predicting A1, the previous digits can already determine the answer. There is a high confidence even without knowing A1, thus it is not highlighted. In essence, the information contained in a query activation may manifest itself early, and the decoder likelihood difference does not necessarily align with the part of the input from which the information has actually been obtained.An interesting direction for future work could be developing decoders that generate samples in a permuted order, and generate most confident tokens first, possibly based on architectures like XLNet [60] .We use InversionView to investigate how GPT-2 XL (1.5B parameters) performs the task of recalling factual associations. In recent research [24, 11, 20] , the task has been formalized in terms of triples (s, r, a), where s, r, a are subject, relation, attribute respectively. The model should predict a based on input containing s and r in natural language. E.g., given the prompt: "LGA 775 is created by", the model is expected to predict "Intel". In this section of the paper, we refer the last token as END, and the last subject token as SUBJ. In the above example, END is "by" and SUBJ is "75". Geva et al. [24] found a high-level pattern that GPT-2 XL uses in solving this task: Early-middle MLP layers at SUBJ integrate information about the subject into its residual stream. Meanwhile, the relation Figure 37 : The ϵ-preimage of x 0,pre . Here the information contained is A1, A2, A3 respectively, while the decoder likelihood difference highlights digits in operands. Because given the first operand and the final sum, the digits in second operand can be inferred. information is incorporated into END's residual stream through early attention layers. In upper layers, the correct attribute is moved from SUBJ to END's residual stream by attention layers.As we mentioned earlier, our goal here is not to provide a full interpretation of the computations performed to solve this task; rather, it is to test whether InversionView can successfully produce interpretable results on larger models. As the factual recall task is complex and involves many model components [61, 11] , we decide to focus on 25 attention heads in the upper part of the model (layer 24-47) that contribute most frequently to the final prediction. We select these activation sites because the attribute retrieval tends to happen there [24] and we expect more abstract information in ϵ-preimage. Here, we note that preliminary experiments revealed that it is necessary to restrict the number of activation sites that the decoder is trained for, given a limited compute budget, as more activation sites require the decoder to learn more a complex overall inverse mapping from activations to inputs. Scaling the approach to apply to many activation sites simultaneously is left to future work.Concretely, we use the attribution method introduced in [18] to estimate the head importance at END position. Formally, given y = z 1 + • • • z m , the importance of each term z j to the sum y is estimated as follows: importance(z j , y) = proximity(z j , y)EQUATIONproximity(z j , y) = max(-∥z j -y∥ 1 + ∥y∥ 1 , 0).(5) Regarding the intuition and estimation quality of this method, please refer to [18] . In our case, x i,mid END = x i,pre END + j a i,j END , we calculate importance(a i,j END , x i,mid END ) on each example on a subset of COUNTERFACT [37] . The subset contains around 1,000 factual prompts known by GPT-2 XL (we used the same subset as described in [37] Appendix B.1). We consider an attention head activated on a certain input prompt if its importance is higher than the threshold of 0.02, and calculate the frequency of being activated over the subset. Finally we select top 25 most frequent heads in model's upper layers. 6There are multiple methods that can be used to find important attention heads [38, 39, 50, 29] , because we do not have strict requirements for finding the most important heads, we choose this one because of its simplicity.To train the decoder model, we collect text from 3 datasets, including the factual statements from COUNTERFACT [37] 7 and BEAR [56] 8 , as well as general text from MiniPile [30] . The factual statements we used are complete sentences containing the final attribute. Note that in COUNTERFACT, for each attribute, there are always multiple different subjects associated with it. So if the information contained is solely the attribute, a well-trained decoder should generate different subjects. Regarding MiniPile, we randomly select 10% of it, and split the text into sentences [9] and remove sentences longer than 100 characters.Importantly, for each head we selected, we extract a separate subset of sentences from the collected data on which the head is "activated". By being "activated" we mean the attention weight on BOS token is less than 0.6 9 . We find this is important based on preliminary experiments, because in many cases attention heads in higher layers execute "no-op" by exclusively attending to BOS, resulting in attention output containing no information. Training decoder on these activations discourages it to learn to read information from the query activation. So each head correspond to a subset of text, on which the head output will be captured to create training data. The number of sentences in subsets ranges from 0.6 to 2.6 million.The training data for decoder consists of two parts, each of which accounts for 50%. In one part, the activations are taken from GPT-2 XL when processing factual statements from COUNTERFACT and BEAR, and activations correspond to the END position (the sentence structure is provided in these datasets). In the other part, the activations correspond to text from all 3 datasets (thus mainly composed of MiniPile text), and correspond to the position with least attention weight on BOS token. By doing so, we emphasize the importance of factual statements domain while covering a large variety of text.We use GPT-2 Medium as the backbone model for decoder. Concretely, the newly added components in the decoder architecture (e.g. those parts responsible for processing query activation as shown in Figure 12 ) are trained from scratch, but the transformer layers in the decoder are equipped with pretrained weights. In this way, we also make use of the existing capacity of language models. Note that in other case studies, we use a small 2 layer transformer as decoder. The reason why we use a much larger decoder for this task is we expect a much more complex inverse mapping to be learned. Specifically, we expect ϵ-preimage for some activation sites to contain multiple different subjects sharing a certain attribute, the decoder needs to generate these subjects given the attribute. So it needs enough capacity to memorize the knowledge.Similar as before, we always add the "< |endoftext| >" token as the BOS token when capturing query activation from GPT-2 XL. We again use a new token "[EOS]" as the EOS token when training the decoder. We sample query activation with equal probability of activation sites. We sample 32 million training examples (all activation sites are included) and train the decoder with batch size of 512, resulting in roughly 60K steps. The final average in-preimage rate is 36.0%.Similar as before, when using decoder to generate samples, we perturb the query activation before feeding it to the decoder in order to cover the neighbourhood around the query. In this larger model with high-dimensional activations, we found it useful to craft this process more carefully and use different disturbance. In preliminary experiments, we found that cosine similarity usually produces more interpretable ϵ-preimages, therefore, we randomly sample activations that have a certain angle θ to the query activations, and then randomly scale it so that its magnitude ranges from e -1 • ∥z q ∥ 2 to e • ∥z q ∥ 2 . We repeat this process for different θ values with in the range cos θ ∈ [0.75, 0.9]. To further encourage diversity, we lower the probability of tokens that have already appeared too many times in the generated samples. 10Figure 38 : ϵ-preimage for head output a 24,24 , showing encoded information is the relation. On the left, the relation "developed by" appears throughout the ϵ-preimage. On the right, the relation "domain of work is" is consistent ϵ-preimage.We use the first 200 examples of the aforementioned subset of COUNTERFACT (the subset containing examples known by GPT-2 XL) as query inputs and generate samples. The results are also available in our web application.Choice of Threshold. We use the same distance metric as in IOI task, and we set the threshold ϵ = 0.25. As we mentioned earlier, larger threshold produces more coarse-grained information. Because we observe that the trained decoder in many cases does not generate enough and diverse samples within a close distance (e.g., 0.1), we increase the threshold in order to draw more reliable conclusions.A priori, the sparsity of sampled ϵ-preimages at smaller thresholds may reflect multiple possibilities. One is that there indeed are no close neighbors for the query input. For example, if the subject's text is copied, then only samples containing the same subject or the same token will lie in ϵ-preimage. Such a phenomenon may generally reflect the high-dimensional geometry of larger models. Another one is that the decoder is not complete; and that it would either require more training or more capacity, or more training data. 11 Training larger decoders on more data would mitigate this problem, which can be done in future work. Nevertheless, in some cases, we do observe a substantial number of diverse samples even within ϵ = 0.1, allowing us to infer information at a higher level of granularity in these cases.We observe that some heads' outputs almost always contain only relation information, if they contain any information at all (being "activated"). 12 In Figure 38, we show two examples for one of these heads. We can see that samples in ϵ-preimage share the same relation. The results are consistent across query inputs. We can infer that the these heads move the relation information to END's residual stream.On the other hand, some other heads almost always move information about the subject, when they are "activated". 13 . Figure 39 shows one of these heads. We can see that the information is only about the subject -so the function of these heads can be summarized as moving information about subject. Interestingly, while a head can move certain attribute about the subject (e.g., nationality, or profession, etc), the attributes it moves for different subjects are diverse. For instance, while a head might, on a certain input, move information that the subject plays certain kind of sports, it may, on another input, move information that the subject is an electronic product. One head, 31.0 tends to usually show Figure 39 : ϵ-preimage for head output a 29,9 , which contains information about the subject. On the left, Czech-related words are common throughout the ϵ-preimage, e.g., "Czech", "Prague", "Bohem...", so the information contained is the subject is "Czech-related". On the right, "Nokia" is shared in ϵ-preimage, so the information is simply "Nokia".Figure 40 : ϵ-preimage for head output a 35,19 , which exhibit different information on different inputs. On the left, Philippines-related words are common throughout the ϵ-preimage. For example, "Cebu": a city in Philippines, "TV Patrol Southern Tagalog": a TV program in Philipines, "Enrile": A municipality of Philippines and a surname of many famous Filipinos, "GMA": a Philippine television channel / broadcasting company. So the information contained is the subject is "Philippines-related". Note that because the samples are generated, the fact stated in sample is not true in many cases. On the right, the relation "performs on" is encoded. nationality or language information about the subject, but usually the heads we studied show no clear preference for a type of attribute. In addition, these is no obvious correlation between the subject attributes encoded by such heads and the attribute queried by the relation.The other heads among the 25 heads we inspect, when "activated", move information about subject or relation (in some cases, both). Which behavior is exhibited varies between inputs. Figure 40 shows one of these heads. On the left of the figure, the head moves information about subject, while on the right it moves information about the relation.In the factual recall task, only one specific attribute is sought. Geva et al. [24] found evidence that the model "queries" the residual stream at SUBJ for the specific attribute asked by the relation part of the prompt, and the representation at END can be viewed as such a relation query. In other words, the attribute extracted from the subject representation depends on the relation. However, we do not observe reliable evidence for this phenomenon among the 25 heads we inspect. In the figures we have shown so far, we can see the commonality shared between samples in ϵ-preimage is not the attribute requested in the prompt. This still holds if one reduces the Figure 41 : Examples showing relation-agnostic retrieval. On the left, the information encoded is "soccer", which is indeed the requested attribute. However, the first sample shows this is not dependent on the relation, since the "soccer" is still retrieved when relation is "speaks language". On the right, the information "audio-related" is encoded, while the relation in the query input is "owned by".Figure 42 : Examples showing different attributes of the same subject are extracted by different heads. In the query input, "Joseph Schumpeter" is an Austrian political economist. On the left, the information encoded is "economist". On the right, the information is about language/nationality (areas around Austria). Again, we emphasize that the facts stated in the sample are not necessarily true.threshold. Figure 41 shows two more examples showing this characteristic. In general, we find that the information moved by the 25 attention heads tends to be the most important attribute or the "main" attribute about the subject. On the left of Figure 41 , the most important attribute coincides with the requested attribute. We know this is a coincidence because the closest sample in ϵ-preimage has a different relation. Note that other attributes about the subject could be moved by heads that we have not studied.In addition, while heads that attend to subject usually move the most important attribute, we do observe sometimes different attributes are moved by different heads. In Figure 42 , we show that information about profession and information about language/nationality of the subject are extracted by two heads. In fact, we observe that head h 31,0 tends to extract language/nationality information in general. But we do not find other obvious pattern of attribute category extracted by other heads.Our findings echo those from [11] , who argue that the primary mechanism for the factual recall task is additive. With our running example "LGA 775 is created by", simply speaking, some heads promote attributes associated with the subject (chip, hardware, Intel, etc.), some heads promote attributes associated with the relation (Apple, Nintendo, Intel, etc). When the results from these independent mechanisms are added together, the intersection (Intel) will stand out. Therefore, the model can solve the task by adding two simple circuits, while humans find Q-composition [17] (i.e., relation information is used as queries in attention heads) more intuitive. From another perspective, this Figure 43 : ϵ-preimage showing information about the subject moved by the attention head. On the left, the information is "cpu/computer-hardware-related". On the right, the information is "island country". Note that some statements are not correct. mechanism implies vector arithmetic. Instead of vector addition in vocabulary space, we can think of it as first summing two vectors (e.g., the output of subject heads and the output of relation heads) and then projecting them to vocabulary space.The evidence found by Geva et al. [24] can also be explained by this mechanism. They use DLA to inspect attention layers' updates to END's residual stream, and find that the token most strongly promoted by each update usually matches the attribute predicted at the final layer. In other words, after projecting attention layer's output to vocabulary space, the top-1 token is usually the exact requested attribute. Because their experiments study attention layer's output as a whole, instead of individual heads, an alternative explanation is the additive mechanism mentioned above. Importantly, because we only inspect 25 heads, other mechanisms including Q-composition are also possible. Moreover, different mechanism can exist in other models, readers should not draw strong conclusions.See Figure 43 , Figure 44 . Interestingly, when inspecting the ϵ-preimage shown on the right of Figure 44 , we also find an example showing InversionView can detect a flaw of the model. Our interpretation for this activation is "Canada-related", and we can see "York University" (which is in Canada) inside the preimage, while "University of York" (a different university, located in England) is outside of the preimage. However, we find that the "U of York" is inside the preimage.Checking the model's prediction about "U of York", we find that the model indeed believes "U of York" is in Canada. More specifically, given the prompt "U of York is located in", the top-12 predictions for the next token are: the, Toronto, York, downtown, a, one, New, central, Canada, North, Scarborough, London. On the contrary, with the prompt "University of York is located in" the model's top-12 predictions are: the, York, North, East, central, Yorkshire, north, a, northeast, London, England, northwest.In Section 4, we mention that attention pattern is not sufficient to form hypothesis when the model has more than one layer. Because unlike in layer 0 each residual stream contains information only about the corresponding token, in higher layers each residual stream contains a mixture of tokens from the context, making it difficult to determine what information is routed by attention. Besides this point, we also find that sometimes attention pattern can be misleading even in layer 0.InversionView reveals how components can know more than what they attend to. At the top of Figure 45 , we show the attention weights of head h 0,3 . Here, "=" attends almost solely to S2, so the head output a 0,3 should only contain information that there is an "8" in tens place. The generated ϵ-preimage, however, shows that it contains information about F2: The number in tens place other Figure 44 : ϵ-preimage showing different information of subject is moved by different head. On the left, the information is superficial text content "York" and/or "University". Samples containing these words are in ϵ-preimage, such as "University of York", even though it is a different university located in England. This can be confirmed by the fact that "Wellington, Ontario" is far away. On the right, the information is "Canada-related", which is more high-level. We can also see "University of York" is outside of the preimage.Figure 45 : For activation site a 0,3 , InversionView reveals how activations can encode information without an attention edge: (a) Even though, on this input, h 0,3 attends only to one tens place digit, it also encodes the approximate identity (range 4-8) of another tens place digit. It encodes that the sum of tens places is greater than ten. (b) We verify our hypothesis by manually create some samples and calculate D. (c) Attention patterns for manually created inputs outside of the ϵ-preimage. The attention pattern differs from that of the query input. In the query input, the attention head infers information about the second tens place digit from the absence of an attention edge.than "8" is always in a certain range (≥ 4), resulting in a carry to the hundreds place. To verify this, we manually constructed examples (rightmost column in Figure 45 ) where the other number is outside of the range, and found that, for these, the activation distance is indeed very large, confirming the information suggested by the decoder. In layer 0, how does the model obtain information about a token without attending it? At the bottom of Figure 45 , we show the attention weights of those manually inserted examples. So the answer is: a different attention pattern would arise if F2 is not in that range. Information can be passed not only by values, but also by queries and keys. InversionView successfully shows this hidden information, even without comparing across different examples.Besides our argument in Section 4, another important aspect of circuit discovery methods is that, in many tasks (including our character counting task), the computational nodes do not correspond to fixed positions, and directly applying path patching is problematic. It's not really clear how to apply path patching when varying input positions matter, as the literature on circuit discovery defines circuits in terms of components, not in terms of input positions. In the case of Character Counting Task, such an interpretation would just define a circuit linking the embeddings, attention heads, and MLPs, without capturing the role of different positions, and the fact that characters from varying positions feed into the computation. Such a view would not provide any nontrivial information about the mechanics of the implemented algorithm. This reflects a more general conceptual challenge of circuit discovery: When different input positions are relevant on different inputs, as in the Character Counting Task, one could either define a single circuit across inputs in which every input position is connected to a single node that performs a potentially complex computation, or define per-input circuits where the wiring is input-dependent; however, per-sample path patching is not very scalable, and resulting per-input circuits would require further interpretation to understand how they are formed across inputs.Direct Logit Attribution (DLA) extends the logit lens method to study individual model components. Specifically, it projects the output of a model component (thus an update on the residual stream) into vocabulary space, and interpret the component by inspecting the tokens it promotes or suppresses. This method has gained popularity in recent years [23, 54, 16, 19] , especially in the research of interpreting factual recall in language models [24, 62, 11, 61] . However, in this section, we argue that DLA is only suitable for studying model components that directly affect the model's final output, and is not well-suited for components whose effect is mediated by other components. In the circuits found by path patching [54, 13] , we can see many components that do not connect to the final output directly, which suggests a substantial part of their effect on the final predictions is mediated by further components. DLA shows their direct effect, which may even be non-causal "side effects". Intuitively speaking, some information is meant to be read by a downstream component, e.g., the S-Inhibition Heads' output in IOI circuit is meant to be read by the query matrix of Name Mover Heads, and such information may not necessarily be visible when projecting to the vocabulary space. Dao et al. [15] also point out such limitations of DLA.In Table 4 we show the results of applying DLA to attention heads' output in IOI circuit. We can see that the expected information is not visible in most cases. The best result comes from the S-Inhibition head 8.10, with only 7.5% of cases where the expected name is in the top-30 tokens and there is no other name being suppressed more than it. The rare cases where the expected name is visible can also be explained by the small direct effect on the final output as depicted by Figure 3 (b) in [54] .Therefore, researchers should be cautious when using DLA and should be aware of its limitations. A good usage example is the IOI circuit [54] , where the authors first identify those attention heads directly affecting the final logits, and only apply DLA to them and design other experiments to interpret other components. Importantly, in the context of factual recall task, we find that the information given by InversionView about the upper layer attention heads is often visible via DLA, indicating these heads contribute to model's output directly. Thus, our results can serve as confirmation that prior results relying on DLA in this task are generally reliable.Feature visualization [43, 41] generates inputs maximally activating a certain neural network unit, and interprets an individual neural network unit (e.g., neurons) to understand its general role across inputs, while InversionView interprets specific values of inner representations, by finding inputs that result in the same vector. When the input changes, the value and thus the interpretation may change. Adversarial or counterfactual example generation methods [27, 59, 46, 45] generate input that is similar to the original input but results in different outcome. Some of them are also used to explain the model. While similar in input space, the adversarial/counterfactual input is likely to be quite different in internal representation space, leading to a different output. In contrast, we are interested in how different inputs in input space are represented very similarly in internal representation space. name among a list of common and single-token names, so it does not count when another name is ranked higher. Note that a name can be associated with two tokens (with and without a space before it), when calculating the rate, either of them satisfying the condition will count. The rate is calculated over 1000 random IOI examples. As we can see, except for the first row, the expected name is not observable most of the time.Similar to InversionView, GAN inversion methods [58] also study the mapping between input space and representation space, with a focus on interpreting the semantics of GAN's latent space and manipulating generation.We further explore whether the process of obtaining the common information from a collection of inputs can be automated by LLMs. We use Claude 3 14 [3] In preliminary experiments we also try GPT-4 [1] but we find Claude 3 works better in our case. In the prompt given to Claude 3, we first describe the task it needs to perform, the terminology we are using (e.g., F1, F2, etc.), the rules (e.g., the pattern it finds should be applicable to each inputs in the ϵ-preimage), input and output form, and the crucial steps it should follow. In addition, we also provide it with 3 demonstrating examples in conjunction with the correct answers. Each example corresponds to a specific activation site and token position (e.g., a 0,0 , A1). In each example, there are 2-3 specific query inputs, each query input is accompanied with 20 (sometimes less) samples that are inside the ϵ-preimage. Claude 3 needs to find the pattern for each query input, and summarize its findings across several query inputs, which is the information contained in general in that activation. In addition to the content described above (the common part shared between prompts), we give it a questioning example, which is the content we would like it to interpret. The questioning example shares the same form as the demonstrating example, except that it contains 5 query inputs and their corresponding samples. In addition, when two separate interpretations are needed based on different A1 value, we run the generation twice with examples of different A1 value, instead of giving the model a mixture of two cases and resorting to its own capacity.We think the following findings from our experiments are worth mentioning: 1) It's hard for the model to align digits of the same place (e.g., comparing all F1 digits), because the samples are presented as a single flattened string instead of a 2-dimensional table. We find that explicitly adding the variable name can largely mitigate this problem, they may serve as certain kind of keys. For example, "7(F1) 1(F2) 1(F3) + 9(S1) 9(S2) 4(S3)". 2) The generated interpretation is sometimes not consistent. The model may generate different conclusion even with the same prompt, but this usually only happens to less important information.3) The model does not strictly follow the rule, i.e., the common pattern should match all inputs, even though we state this repeatedly in different ways in the prompt. The model will say "always" even when there is a counterexample. We should keep 2) and 3) in mind when reading the results.We run the generation for each entry in Table 3 once, using the samples generated from the corresponding activation. The results are shown in Table 5 , accompanied with human interpretation for comparison. The interpretation given by Claude 3 reflects the main information in almost all cases. Even when the information becomes more complex in layer 1, the interpretation quality does not significantly decline. This implies that automated interpretation by LLM is promising. On the other hand, we can also see there are some problems: 1) Some of the model's claims are spurious, these claims are usually ranked low by the model, indicating they are not very obvious.2) The model sometimes does not explain in a desirable manner. For example, for the entry "x 0,mid , A3", the information includes A2 and whether A1=1, which means when A1=1, the sum of F1 and S1 is known. Thus Claude 3 concludes that (F1, S1) is approximately represented. Though this is somewhat true, it is not a fundamental piece of information, and there is a more concise summary.3)The interpretation tends to be verbose, even though we ask the model to be succinct and provide it with some examples. Despite the problems, we think the automatically generated interpretation is by and large satisfactory and informative, and we think some of the problems can probably be solved or mitigated by engineering better prompts.Interpretation from Table 3 Interpretation by Claude 3 a 0,0 , A1 1-2 digits from F1 and S1 *used as example in the prompt a 0,1 , A1 1-2 digits from F1 and S1F1 and S1 (sometimes exactly and sometimes approximately. Their order can sometimes vary) a 0,2 , A1 1-2 digits from F1 and S1 the digits in the hundreds place (F1 and S1): -They tend to be large single digits (7, 8, 9) that add up to 10 or more -Sometimes one or both are fixed to a certain single digit value -Sometimes their values can swap positions between F1 and S1 There are no strong consistent patterns observed for the digits in the tens or ones places (F2, S2, F3, S3). No other types of patterns (e.g. on A1 or carries) are consistently observed. a 0,3 , A1 1-2 digits from F2 and S2; C2F2 and S2 (fuzzy, interchangeable); F3 (sometimes fuzzy); S2 (sometimes fuzzy)x 0,mid , A1 F1 and S1; C2 1) F1 and S1 are either the same digit, or a pair of digits that always appear together in the hundreds place (sometimes with order swapped).2) Whether F2+S2 produces a carry (the value of C2) is often consistent in each example, either always 0 or always 1. 3) There is no consistent pattern found for the ones digits F3 and S3.x 0,post , A1 F1 and S1; C2 1) F1 and S1, either exactly or an approximate version, with order sometimes varying 2) S2, usually 3) C2, occasionally F3, S3 and other parts of the addition rarely contain any mutual information between the query and generated strings.a 1,0 , A1Fuzzy info about F1, S1 and C2; Fuzzy info about F3 and S3 (sometimes)1) F1 and S1 digits (sometimes exact match, sometimes approximate match). 2) Whether the sum of F2 and S2 is greater than or equal to 10 (i.e. the value of C2). 3) F3 and/or S3 digits are often preserved from the query to the generated strings.Interpretation from Table 3 Interpretation by Claude 3a 1,1 , A1If A1=1 (likely to be): 1-2 digit from F1 and S1 (sometimes their sum); C2; Else: Fuzzy info, including some info about F3 and S3 (sometimes)If A1=1: 1. F1 and S1 digits from the query string appear in the generated strings, but their order may be swapped. 2. Occasionally some other information is shared, such as the carry digit C2, or a specific digit like F3, but this is not consistent across all examples. In summary, the key common information is the two digits that appear in the hundreds place of the query and generated strings, while their specific positions as F1 or S1 may vary.Else: The amount of information shared between the query and generated strings varies by example. In some cases, there are clear patterns like certain digits being consistently shared (e.g. S1, S3). Other digits may share approximate rather than exact values (e.g. F1). Some digits tend to vary more (e.g. F2, S2). Occasionally, the carry C2 is also shared. When there are too few generated strings, it is hard to conclusively identify patterns. More generated strings allow more confidence in the patterns found.a 1,2 , A1Fuzzy info about F2 and S2 (sometimes)1. F1 and S1 (sometimes fuzzy representation, their order can vary) 2. F2 and/or S2 (sometimes only one of them, sometimes partially match) 3. Occasionally F3 and/or S3 (partially match, their order can vary) x 1,mid , A1 F1 and S1; C2a 1,31) The values of F1 and S1, either the exact values or their sum 2) The value or presence/absence of S2 3) The relationship between F2 and S2 (which is greater, or whether there is a carry) 4) Which digits are missing in the result (A1 or A2)x 1,post , A1 F1 and S1; C2 the F1 and S1 values from the query can be found in either order (F1, S1) or (S1, F1) in all generated strings for that query. In some cases, C2 (carry from tens place) is also consistent between query and generated strings, being either 0 or 1 in all strings for that query. F3 and S3 values do not seem to have any limitations or correspondence between query and generated strings. a 0,0 , A2 whether A1=1 -Sometimes A1 is consistent (always the same value) between query and generated strings -Sometimes the exact or approximate F3 and S3 values from the query appear together frequently in the generated strings -In some examples, there is no clear common information The values of (F1, S1) and (F2, S2) do not seem to be consistently shared between query and generated strings in these examples. 3 Interpretation from Table 3 Interpretation by Claude 3 a 0,3 , A2EQUATIONIf A=1: 1-2 digits from F1 and S1 Else: 1-2 digits from F3 and S3; C3If A=1: 1) F1 and S1 are often the same pair of digits (sometimes with a bit of variation). The order of the two digits may vary. 2) A1 is always the same digit. 3) In some cases, the presence or absence of a carry from the tens place (C2) is consistent. The digits in the ones place (F3 and S3) are usually not shared between the query and generated strings.Else: 1) F3 and/or S3 often have specific shared values between the query and generated strings. The shared digit is usually in the range 2-9. 2) Occasionally other patterns exist in generated strings (e.g. ones place of answer) but are not truly shared with the query. 3) Besides the ones place operands, no other consistent patterns were found across F1/S1, F2/S2 or the answer digits.x 0,mid , A2 Else: A1 is always exactly shared, the middle digits F2 and S2 are often approximately shared, and usually one of the first digits (F1/S1) and one of the last digits (F3/S3) from the query appears frequently in the generated strings.x 0,post , A2If A1=1: F1 and S1 Else: F2 and S2; C3If A1=1: -F1 and S1 always match, although their order can be swapped. In some cases, the carry C2 (whether F2+S2 is ≥ 10) is also shared between the query and generated strings.Else: 1) The digits in the tens place (F2, S2), either the exact digits or their sum (carry C2).2) The ones digit of the answer (A1). 3) Sometimes the digits in the ones place (F3, S3), either the exact digits or their sum (carry C3).a -When the sum is a 2-digit number, A1 is sometimes shared between query and generated strings -When the sum is a 3-digit number, A2 is often (but not always) shared -If there is a shared A2, its value often matches the query -If both A1 and A2 are present, A1A2 is between 10-19 There are no consistent patterns in the operand digits F1/S1, F2/S2, F3/S3 or their sums. a 0,2 , A3 1-2 digits from F2 and S21) S2 or F2 sometimes takes on a consistent value (like 0, 9, 7) in the query and many generated strings.2) The equations are often incomplete, with A1 and/or A2 missing in many generated strings. 3) When present, A2 sometimes has a consistent value (like 1, 2, 6) in the query and generated strings, but not always. 4) No consistent patterns were found for the (F1, S1) and (F3, S3) digit pairs. a 0,3 , A3 1 digit from F3 and S3 (2 when F3=S3); C3 1) Certain digits from F3 and/or S3, although the order of F3 and S3 may vary in generated strings compared to the query. 2) The existence of A1 and A2 in complete equations, but not always. Other patterns like digits from F1, S1, F2, S2 are rarely shared.x 0,mid , A3 A2; F2 and S2; C3; whether A1=11. An exact or approximate representation of (F1, S1), sometimes with order varying 2. Specific values or range of values for F2 and S2, sometimes along with their sum (C2) 3. Sometimes specific values for F3 and/or S3 4. Often an exact or approximate representation of (A1, A2) The amount and type of information shared varies between examples. But in general, there are usually some constraints on the digits in each place value shared by the query and generated strings.x 0,post , A3 A2; F2 and S2; C3; whether A1=1 1. F1 and S1: exact match or fuzzy match, order of F1 and S1 may vary 2. F2 and S2: exact match or fuzzy match, order of F2 and S2 may vary, sometimes have size relationship (like F2 ≤ S2) 3. F3: exact match or fuzzy match 4. S3: exact match or fuzzy match 5. A1: exact match 6. A2: exact match Not every example has all the above common information.Entry in Table 3 Interpretation from Table 3 Interpretation by Claude 3 a 1,0 , A3 Else: -The last digit of F (F3) is often shared between query and generated strings -The last digit of the answer (A2 for 2-digit answer, A3 for 3-digit answer) is very frequently shared -Sometimes S3 or other answer digits are also shared, but less reliably In general, the ones place digits of the operands and answer tend to be shared between the query and generated strings in each example.x -Some or all digits in F2, F3, S2, S3 (F2 and S2, F3 and S3 can be swapped) -The last digit of the answer (A2) -Whether there is a carry from tens place to hundreds place (C2) a 0,0 , A4/E whether A1=1Interpretation from Table 3 Interpretation by Claude 3 a 0,1 , A4/E 1-2 digits from F3 and S3Across the examples, the most consistent commonality is in the ones place (F3 and S3). They often have the same or very similar values in the query and generated strings. Sometimes a digit in a certain place of the final answer is also shared between query and many/all generated strings, like the first digit (A1) or last digit (A3). But this is not always the case. The hundreds place (F1/S1) and tens place (F2/S2) digits usually vary without a clear mutual pattern between the query and generated strings. x 0,mid , A4/E A3; F3 and S3; whether A1=1 -F1 and S1 sometimes appear in generated strings with their order swapped -F3 and S3 sometimes appear in generated strings with their order swapped -F3, S3 and A3 of the query string often appear unchanged in the same position in generated stringsx 0,post , A4/E A3; F3 and S3; whether A1=1 -F3 is often shared between the query string and generated strings -S3 is sometimes shared between the query string and generated strings -A3 is consistently the same value in the query string and all generated strings for each example There is no strong pattern for F1, S1, F2 or S2.a 1,0 , A4/E 1-2 digits from F3 and S3; whether next token is E 1. Specific digits or narrow ranges in F3 and S3 2. The value or a narrow range of A1 3. The ending digits or a narrow range of ending digits in A3 4. Sometimes the carry C1 or C2 The digits in F1, S1, F2, S2 and the full value of A2, A3 are usually not shared. a 1,1 , A4/E 1-2 digits from F3 and S3; whether next token is E 1) F3 (always the same or frequently appears) 2) S3 (always the same or frequently appears) 3) One or more answer digits, especially the last one A3 (always the same or frequently appears) a 1,2 , A4/E 1-2 digits from F3 and S3The ones digit (F3) of the query string consistently shows up in the ones place (either F3 or S3) of the generated strings. Sometimes the other ones digit in generated strings has a specific value when query F3 is in a certain place. Occasionally, the carry (C3) from the ones place addition is also shared between the query and generated strings.a 1,3 , A4/E 1-2 digits from F3 and S3only the ones digits (F3 and sometimes S3) are consistently shared, while the other parts of the addition problems vary between the query and generated strings in each example.x 1,mid , A4/E A3; F3 and S3; whether A1=1-F3 and S3 have some fixed values (varying by example) that always sum to the same total. The order of the two digits doesn't matter. -As a result, A3 is always a fixed value for each example. -There is sometimes a consistent carry over amount from the tens to hundreds place, resulting in a fixed A1 value.x 1,post , A4/E F3 and S3; whether A1=1 -Specific digits in the answer (A3) -Specific digits in the operands (F3, S3) -Whether there is a carry in a certain place (C2, C3) The shared information varies across examples, but usually relates to the ones or tens place digits and carries.Table 5 : Interpretation for 3 digit addition produced by Claude 3, compared with human interpretation from Table 3 . In general, the automated information is very informative, and the human interpretations 3 is contained in almost all cases, though the output tends to be more verbose. The LLM outputs, with some human post-checking, can thus further speed up interpretation.We ran all experiments on NVIDIA A100 cards. The decoder is trained for 4-6 hours on 2 GPUs for the first three case studies, without exhaustively tuning efficiency of the implementation, which we believe could further speed up training. Regarding the factual recall task, we train the decoder for less than 1 day on 4 GPUs. Generation of ϵ-preimage samples (including forward passes on the probed model to calculate distance metric) is fast for the first 3 tasks, and it takes around 9 hours on 4 GPUs for factual recall task (for 200 query inputs). Patching experiments run quickly, as they are done for small models.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Code is available at https://github.com/huangxt39/InversionView 38th Conference on Neural Information Processing Systems (NeurIPS2024).Strictly speaking, when x q is a sequence, we study the vector z q corresponding to a specific position t in this sequence, i.e. z q = f (x q )t where f (x q )t represents taking the activation from the site of interest (abstracted by f ) at position t in input sequence x q . In this case, the preimage can more rigorously be defined as B z q ,f,ϵ = {x : x ∈ X , ∃t ∈ [1, |x|] : D(f (x)t, z q ) ≤ ϵ}.In some cases, including Figure3b4b, we fix a particular position for interpretation. See Appendix A.3The reader can check generations conveniently at https://inversion-view.streamlit.appWe have also tried value-weighted attention pattern[32]; it makes a negligible difference, so we always show the original attention pattern in our paper.h 31,0 , h 33,0 , h38,22 , h29,9 , h 37,7 , h32,12 , h 31,8 , h 24,24 , h 28,3 , h 25,7 , h28,21 , h27,16 , h42,24 , h31,4 , h34,20 , h30,23 , h 24,8 , h30,8 , h 25,10 , h33,9 , h 32,15 , h 30,1 , h29,20 , h36,17 , h35,19 , ordered by frequency, last one is the most frequentWe make use of all prompts of each data point. Besides the "prompt", we also include "paraphrase_prompts", "neighborhood_prompts", and "attribute_prompts" concatenated with their corresponding answer.We use BEARbigwe do not experiment with other valuesWe chose a simple and heuristic approach, without any attempt at optimizing it: Any token that has appeared > 5 times in the same position, while appearing less than 50K times in the miniPile training set, has its probability set to zero at this position.For example, if the query activation contains information "related to Amazon and related to audio", there are only a few satisfactory subjects in the dataset ("Audible.com", "Amazon Music").h24,24 , h25,7 , h27,16 , h28,21 , h29,20 , h33,9h29,9 , h30.8 , h32,12 , h 33,0 , h37.7Version: claude-3-opus-202402291) The value of A1 (in 4 out of 5 examples) 2) An approximate or exact match of (F1, S1) pair (in 2 out of 5 examples) 3) The sum of F3 and S3 (in 1 out of 5 examples) Other occasional commonalities include the value of A2 and whether A3 is complete. 


Decision: After carefully reading the reviews and the rebuttal, I feel the authors manage to answer most if not all of the concerns raised by the reviewers. And as such, I think the quality of the manuscript increases sufficiently for it to be considered for acceptance. 
In particular, reviewer fZCU, with the lowest score, most concerns seem to be around presentation and clarity. The rebuttal provides answers to the punctual questions raised by fZCU in detail and explain how the draft will be improved. I believe these changes will answer the concerns raised. But I strongly urge the authors to incorporate all the changes they suggested in the rebuttal and additional experiments, as well as improving the clarity of the manuscript in the camera ready version of the paper. Improving these points will be important for the paper to have the impact it deserves once published.

Reviews: review: paper_summary: paper_summary: In this paper, the authors propose InversionView, a method to inspect the information encoded in neural activations. The proposed method is based on checking the activations difference given different inputs. The authors showcase the effectiveness of this tool on mainly three tasks: character counting, indirect object identification, and 3-digit addition.

Strengths

1. The topic of model interpretation is important.
2. The empirical results confirm the effectiveness of the proposed method.
3. The paper is well written and easy to follow. In addition, the authors provide abundant experiments to showcase the effectiveness.

Weaknesses

1. Although being described through text, the precise algorithm for the proposed method is unclear to me.
2. It is unknown how the proposed method scales with larger models. The largest model size in the paper is limited to GPT-2, which is not considered large nowadays.

Overall, I think the paper is interesting and recommend a weak acceptance. I will consider raising my ratings if the above weaknesses are addressed.
strength_weakness: strength_weakness: 1. Although being described through text, the precise algorithm for the proposed method is unclear to me.
2. It is unknown how the proposed method scales with larger models. The largest model size in the paper is limited to GPT-2, which is not considered large nowadays.
questions: questions: See the weaknesses.
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: This work is mainly based on the representational geometry of activations in the activation space and chooses samples whose distances are within a defined $\epsilon$-preimage distance. A single two-layer transformer decoder model is trained using activations of each layer from the investigated model. After finishing decoder training, the Euclidean distance is used as a metric to select generated samples based on each activation presentation of each layer, and compare those samples with the query input sentence. This approach is evaluated and analysed under three tasks, i.e., character counting with a 2-layer and 1-head transformer, IOI with GPT2-small and 3-digit addition with 2-layer, 4-attention head transformer. The comprehensive experiment results and analysis confirm the geometry hypothesis, and provide lots of insights to future work.
strength_weakness: strength_weakness: - I suggest authors use another figure to demonstrate the whole training and evaluation pipeline, i.e., what input is used for training probed model, what query input is for decoder training and what input and output are from the trained decoder, which will help to understand the whole framework much easier for readers.
- This method is mainly based on manual investigation to select and analyse generated samples, which might need large labour resources to extend to large-scale LLMs.
questions: questions: - In line 115, what does $\mathbf{z}$ represent here? and what is the difference between $\mathbf{z}$ and $\mathbf{z}^q$?
- In line 108, Is this input the same as the query input in figure 1? Another figure introducing the whole pipeline might help a lot to understand.
- Does this neighbour of geometry hypothesis always hold across all LLM activations?
limitations: limitations: I am looking forward to how to automate this approach and apply it to the larger LLM interpretability analysis.
paper_summary: paper_summary: The method proposed in the paper seeks to decipher the information encoded within neural network activations. The core idea is to examine subsets of inputs that produce similar activations, utilizing a trained decoder model conditioned on these activations. The authors perform their analysis on three different tasks: character counting, indirect object identification, and 3-digit addition.
strength_weakness: strength_weakness: * Applying the method to extremely large models and more complex tasks is challenging.

* Selecting an appropriate distance metric and epsilon is challenging and requires tuning, making it difficult to adapt the method to different tasks. This also highlights the authors' impressive effort in making the method work.
questions: questions: * Do you think it is possible to use other distance metrics besides L2-based metrics? Additionally, why do you believe the L2 distance is effective in this context? Is there something unique about the geometric space of these tasks that makes L2 distance particularly suitable? What types of tasks, in general, can be effectively solved using L2-based metrics?
limitations: limitations: Yes
paper_summary: paper_summary: The paper introduces the method InversionView, which finds inputs that give rise to similar activations (the preimage). As the preimage grows exponentially with sequence length the authors train a conditional decoder to generate inputs from a target activation.
The authors show three use cases: a character counting task, Indirect Object Identification, and 3-digit addition.
strength_weakness: strength_weakness: - the abstract could be a bit more specific
- in general, I think the reader is left wondering for a bit too long what the authors actually do and how the method works.
- line 28: "which in turn helps us put together the algorithm implemented by the model." promises a bit too much
- motivation and description of the method is too vague
- very few details on the method/decoder that generates the examples that are later inspected
- results of experiments are a bit hard to pass. It would be nice if the reader is guided through those a bit better at different levels of abstraction
- the approach (generating human interpretable examples that achieve a specific activation patters) seems closely related to feature visualization/ adversarial example and counterfactual example generation/ in general methods that generate in the input space, however those works are not discussed at all
questions: questions: - a graphic showing how to get from source transformer activations to pre image samples would be helpful
- Figure 2 is hard to parse: I would suggest highlighting your conclusion/interpretation (aka one activation encodes target character but not count, one activation encodes count but not target character etc) in the figure.
- in general, for the experiments, I would suggest to clearly state the task and your findings before jumping into technical details
- 3.3 "We apply InversionView to the components of the circuit" seems like you apply inversionView to specific activations found by [39] but then you state "InversionView unveils the information contained in the activation sites, with
results agreeing with those of Wang et al. [39], while avoiding the need for tailored methods."
So it seems like you needed the "tailored methods" to know where to apply inversionView in the first place, or am I misunderstanding sth?
limitations: limitations: - I think when moving from toy tasks to SOTA models, inversionView might become very prone to human tendency for pattern matching
- in addition polysemanticity might be very relevant for InversionView results but is not discussed at all
- related work is not discussed in sufficient detail

Comments: title: Reply by Reviewer vTBG
comment: Thank you. We are grateful for your support.
Thanks for those helpful responses. I think this work is solid and I maintain my score.
Thank you. We are grateful for your support
Thank you. We are grateful for your support.
Thank you for your response. I still believe this work takes a novel approach, so I will maintain my score.
Thank you for your response. I remain my stance to accept this paper.
Thank you for raising your score, and thanks for your suggestion regarding related work. We agree with your suggestion, many readers from other fields may wonder how InversionView compares to these works, both in terms of similarities and differences. We will be happy to discuss both differences and similarities to related work.
Thank you for addressing my concerns. After reading your rebuttal and the other reviews I have updated my score to a weak reject since it is still hard for me to assess if the limitations regarding clarity of presentation will be sufficiently addressed in the final version. However, other reviewers seemed to quite like the paper, so I will not stand in the way of accepting the paper for publication. 
Regarding related work: I would appreciate discussing not only differences, but also similarities to related work (for example GAN inversion techniques). Many ideas there seem quite related even when the final goal is not always interpretability. This can help readers from other domains to more quickly understand your work.