VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformers
Authors: Shahar Katz
Yonatan Belinkov
Venue: EMNLP 2023 Findings (2023)
Keywords: attention
interpretability
visualization
TL;DR: Interpreting transformer's attention and visualizing the semantic information flow of GPT: additionally to identifying patterns by which the attention mechanism passes information, we present a new approach and a tool to examine GPTs as flow graphs

Abstract: Recent advances in interpretability suggest we can project weights and hidden states of transformer-based language models (LMs) to their vocabulary, a transformation that makes them more human interpretable. In this paper, we investigate LM attention heads and memory values, the vectors the models dynamically create and recall while processing a given input. By analyzing the tokens they represent through this projection, we identify patterns in the information flow inside the attention mechanism. Based on our discoveries, we create a tool to visualize a forward pass of Generative Pre-trained Transformers (GPTs) as an interactive flow graph, with nodes representing neurons or hidden states and edges representing the interactions between them. Our visualization  simplifies huge amounts of data into easy-to-read plots that can reflect the models' internal processing, uncovering the contribution of each component to the models' final prediction. Our visualization also unveils new insights about the role of layer norms as semantic filters that influence the models' output, and about neurons that are always activated during forward passes and act as regularization vectors.

VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformers
Shahar Katz
Yonatan Belinkov
Technion Israel Institute of Technology
shachar baachar.katzecs technion.c. ac. il
belinkov@teccmion..cd ac.
Abstract
transformer LMs add information the model's residual stream, the backhone route of information, promoting tokens that eventually make it to the final predictions. Subsequent work by Dar et al. (2022) shows that projections of activated neurons, the static weights of the models' matrices, are cor- related in their meaning to the projections of their block's outputs. This line of work suggests stop reading vectors (HSs or neurons) as just num- bers; rather, we can read them words, to better understand what models "think" before making a prediction. These studies mostly interpret static components of the models or are limited to specific case studies that require resources or expertise.
Recent advances in interpretability suggest can project weights and hidden states of transformer-base language models (LMs) to their vocabulary, transformation that makes them more human interpretable. In this paper. we investigate LM attention heads and mem- ory values, the vectors models dynamically create and recall while processing given in- put. By analyzing the tokens they represent through this projection, we identify patterns in the information flow inside the attention mech- anism. Based on our discoveries, we create tool to visualize forward pass of Generative Pre-trained Transformers (GPTs) as an interac- tive flow graph, with nodes representing neu- rons or hidden states and edges representing the interactions between them. Our visualization simplifies huge amounts of data into easy-to- read plots that can reflect the models' internal processing, uncovering the contribution of each component the models' final prediction. Our visualization also unveils new insights about the role layer norms as semantic filters that influence the models' output, and about neu- rons that are always activated during forward passes and act as regularization vectors.
To address the gap in accessibility of the mech- anisms behind transformers, some studies create tools to examine how LMs operate, mostly by plot- ting tables of data on the most activated weights across generations or via plots that show the effect of the input specific weights (Geva al., 2022a; Hoover al., 2020). Yet, such tools do not present the role of each of the LM's components to get the full picture the process.
generation
In this paper, analyze another type of LMs' components via the logit lens: the attention mod- ule's dynamic memory (Vaswani al., 2017), the values (HS) the module recalls from previous in- puts. We describe the semantic information flow inside the attention module, from input through keys and values to attention output, discovering patterns by which notions are passed between the LM's components into its final prediction.
1 Introduction
Wouldn't I it be useful to have something similar to X-ray for transformers language models? Recent work in interpretability found that hidden-states (HSs), intermediate activations in a neural network, can reflect the "thought" process of transformer language models by projecting them to the vocabulary space using the same transfor- mation that is applied to the model's final HS. method known the "logit lens" (nostalgebraist, 2020). For instance, the work of Geva al. (2021, 2022b) shows how the fully-connected blocks of
Based on our discoveries, we model GPTss as flow-graphs and create dynamic tool showing the information flow in these models (for example, Fig- ure 1). The graphs simplify detection of the effect that single to small of neurons have on the pre- diction during forward passes. We use this tool analyze GPT-2 (Radford et al., 2019) three case studies: (1) we reflect the mechanistic analysis Wang et al. (2022) on indirect object identification
'Code and tool are availahle at attps://githhi com/ shacharkZ/VISIT-Viine www.-iaaliiiiiiieeaa Transformers

---

alerice
elembon
(Teekon
oteer Ein extra
melo
Werd
mant Hr
SaMho
kandamer
Toyy
Inpa
Londor
Zam
9yoto
kist
Tooyo
Cormayy
1000
The SOMPROTS
Singsak
raidal
Figure I: Modeling single layer (number 14) of GPT-2 for the prompt: "The capital of Japan is the city of". Each node represents a small group of neurons or HS, which are labeled by the top token of their projection to the vocabulary space. The plot should read from left to right and includes the attention block: LN (the node at the end of the first purple edge), query, memory keys and values (with greenish edges) and the topmost activated attention output neurons (in blue and red), followed by the MLP: LN (in purple), first and second matrices' most activated neurons (in blue and red). The dark edges in the upper parts of the plot are the residuals each sub-block.
in simple way; (2) we analyze the role of layer norm layers, finding they act as semantic filters; and (3) we discover neurons that are always acti- vated, related to but distinct from rogue dimensions (Timkey and van Schijndel, 2021), which term regularization neurons.
2.2 GPTs Sub-Blocks
The attention module consists four matrices, Wo, WK, Wy, Wo E RRdxd Given a se- quence of HS inputs, hs.. hs,, first creates three HS for each hs;: q hs; hs h = Wk. 0 hs;Wr, referred to as the current queries, keys, and values respectively. When processing the I-th input, this module stacks the previous k; and ny 's into matrices K, V E ı and calculates the attention score using its current query q = Q: A Attention(g K, V) softmax( 9k V In practice, this process done after each of Qi kú 022 is split into h equal vectors to run this process in parallel h times (changing the dimen- sion from d to d/h) and produce A₁ E R (0= < j h). called heads. To reconstruct an output the size of the embedding space, d, these vectors are concatenated together and projected by the out- put matrix: Concat .., An-1) We refer to the process of this sub-block as Altn(hs).
Attention:
2 Background
2.1 The Transformer Architecture
We briefly describe the computation in auto- regressive transformer LM with multi-head atten- tion, such as GPT-2, and refer to Elhage et al. (2021) for more information.2
<a
The model consists of a chain of blocks (lay- ers), that read from and write to the same residual stream. The input to the model is sequence of word embeddings, I1, ..,,, (the length of the in- put is tokens), and the residual stream propagates those embeddings into deeper layers, referring to the intermediate value it holds at layer I while pro- cessing the i-th token as hst (hs; in short). The HS at the final token and top layer, hs! is passed through layer norm, ing, followed by decoding matrix D that projects it to vector the size of the vocabulary. The next token probability distribution is obtained by applying a softmax to this vector.
We emphasize that this module represents dy- namic memory: it recalls the previous values is (which are temporary representations for previous inputs it saw) and adds a weighted sum of them according to scores it calculates from the multi- plication of the current query with each of the previous keys k; (the previous keys and values are also referred to as the "attention k-v cache").
Each block is made of an attention sub-block (module) followed by a multi-layer perceptron (MLP), which describe next.
MIPP
This module consists of activation function f and two fully connected matrices. F F1, FFJ E dRx (N is a hidden dimension, usually several times greater than d). Its output is M (L) = f(xFF)
AAppendix details these models the context of our graph modeling.
FFor simplicity we do not mention dropout layers and position embeddings here.

---

Entire block: GPT-2 applies layer norm (LN), before each sub-block: In, for the attention and Ing for the MLP. While LN is thought to improve numeric stability (Ba et al., 2016), one of our dis- coveries is the semantic role it plays in the model (subsection 5.2). The output of the transformer block at layer I. given the input hs! is
et al., 2021). These studies claim that activat- ing a neuron whose projection to the vocabulary has a specific meaning (the common notion of its most probable tokens) is associated with adding its meaning to the model's 's intermediate processing. In our work we interpret single and small groups of HS using the logit lens, specifying when we are using interaction circuit to do so. In addition, while previous studies interpret static weights solely the attention oulput, we focus on the HS that the attention memory recalls dynamically.
h
hst + Attn(Ini (hs))) M MLP( (Attn((n (hs!)) + (hs()) (1)
2.3 Projecting Hidden States and Neurons
Tracing the Semantics Behind the Attention's 's Output
The Logit Lens (LL): nostalgebraist (2020) ob- served that, since the decoding matrix in GPTs is tied to the embedding matrix, D = ET, we can examine HS from the model throughout its com- putation. Explicitly, any vector interpreted probability on the model's vocabu- lary by projecting it using the decoding matrix with its attached LN:
In this section, we trace the components which cre- ate the semanties of the attention block's output, by comparing vectors different places along the computation graph. In all the following experi- ments, we project HS into the vocabulary using the logit lens to get a ranking of all the tokens, then pick the top-k tokens according to their ranking, We measure the common top tokens of two vec- tors (œ) and x2) via their intersection score I5. (Dar et al., 2022):
Rdd can be
LL(x) so. (n = E ER bocabutaryl
(2)
By applying the logit lens HS between blocks, we can analyze the immediate predictions held by the model each layer. This allows us to observe the incremental construction of the model's final prediction, which Geva et al. (2022b) explored for the MLP layers.
(6(21,))) = [L(x))((op-]] n LL((2) [topk
k
(3) We say that two vectors are semantically aligned if their /& is relatively high (close to 1) since it means that a large portion of their most probable projected tokens is the same.
Very recent studies try to improve the logit lens method with additional learned transformations (Belrose et al., 2023; Din et al., 2023). We stick with the basic approach of logit lens since we wish to explore the interim hypotheses formed by the model, rather than better match the final layer's output or shorteut the model's computation, and also, since those new methods can only be applied to the HS between layers and not to lower levels of components like we explain in the next section.
Throughout this section, we used CounterFaac (Meng et al., 2022), dataset that contains factual statements, such as the prompt "The capital of Nor- way is " and the correct answer "Oslo". We gen- erate 100 prompts randomly selected from Coun- terFact using GPT-2- -eeuum which we verify the model answers correctly. We collect the HSs from the model's 's last forward-passes (the passes that plot the answers) and calculate 50
Interpreting Static Neurons: Each of the men- tioned matrices in the transformer model shares one dimension (at least) with the size of the embed- ding space d, meaning we can disassemble them into neurons, vectors that correspond to the "rows" or "columns" of weights that are multiplied with the input vector, and interpret them as we do to HS. Geva et al. (2021) did this with single neurons in the MLP matrices and Dar al. (2022) did this with the interaction of two matrices in the attention block, We with WK and Wy with Wo. known as the transformer circuits QK and OV (Elhage
3.1 Projecting the Attention Memory
For our analysis we interpret Wy products, the attention's heads A, and its memory values, tji (j for head index and for token index). For each component we calculate its mean IR=50 with its attention block output (Attn (hs/). "Ig attn"), its
"To achieve two matrix circuit we multiply one matrix with the output of the second, example, the OV circuit outputs are the multiplication of Wo matrix with the outputs of Wv. RRefer to Appendix C, D. for more information about our model selection and setup.

---

0.15
atin
0.10
V. block
0.10
final
0.00
10 layer index
0.00
10 13 layer index
(a) Without Wo projection
(b) With Wo projection
Figure 2: Comparing I2-50 token projection alignment between the mean of heads Aj with different parts of the model's, with and without using the attention block's Wo matrix for projection, suggesting that the output of the attention block's 's Wy operates in different space and that Wo's 's role is to adjust it to the common embedded space.
transformer block output (hs/+1, "Tx block"), and the model's final output (hsf. "Ig final").
norm
Ik attn
Dar al. (2022) suggest using the OV circuit, in accordance to Elhage et al. (2021), to project the neurons Wy y by multiplying them with Wo Sim- ilarly. we apply logit lens to A, once directly and once with the OV circuit, by first multiplying each A₃ with the corresponding parts of Wo to the j-th head + $). While the first approach shows no correlation with any of the I we calculate (Fig- ure 2a), the projection with OV shows semantic alignment that increase with deeper layers, having some drop at the final ones (Figure 2b). The pattern of the latter is aligned with previous studies that examine similar scores with the MLP and the entire transformer block (Haviv et al., 2023; Lamparth and Reuel, 2023; Geva et al., 2022b), showing that through the OV circuit there is indeed seman- tic alignment between the attention heads and the model's s outputs and immediate predictions.
0.2
a0
2
3
9 11 12 layer index
15 15
19
20
(a) Mean T==50 for only the top 3 heads with the largest norm. comparing to attention block output.
0.6
L attn
block
0.4
4. finnal
0.0
10
layer index
(b) Mcan 4==50 for only the head with the largest norm, com- paring to attention block output, layer output and the model's final output.
Figure 3: Projecting attention heads
few heads with the largest norm have a common vocabulary with their attention block output (Fig- ure 3a), which gradually increases the effect on the blocks' outputs and the final prediction (Figure 3b), This suggests that the attention block operates as a selective association gate: by making some of the heads much more dominant than others, this gate chooses which heads' semantics to promote into the residual (and which to suppress).
This finding suggests that the HS between Wy and Wo do not operate in the same embedded space, but are rather used as coefficients of the neurons of Wo. Therefore, outputs of Wy should be projected with logit lens only after they are mul- tiplied by Wo-
3.2 Projecting Only the Top Attention Heads
We observe that each attention block the norms the different heads vary across generations, mak- ing the top tokens of the heads with the largest norms more dominant when they are concatenated together into one vector. Therefore, we separately ranked each attention block's 's heads with the OV circuit (A, Wo) according to their norms and re- peated the comparison. We found that only the
3.3 Projecting Memory Values
We ran the same experiment comparing the mem- ory values Ujs, the values that the attention mech- anism recalls from the previous tokens. For each head Ay. we rank its memory values based their attention scores and observe that memory values signed higher attention scores also exhibit greater degree of semantic similarity with their correspond- ing head. The results for the top three memory values are illustrated in Figure 5.
'In practice, the implementation of projecting vector in the size of & like A; done by placing in d-size zeroed vector (starting at the j & index). Now we can project it using logit lens (with or without multiplying it with the entire Wo matrix for the OV circuit).

---

viice Icseded
SemTra (ilalos locyer prot 1400 cast 20rnn
"London, "Cocspo
7015,
telgo
Kyers
Node: 14) valu Inder511 heed adiisionn1 106 SunngU
Cyprus Charnal Heatss Toyg 2006 InK 970490 1110
Figure 4: Modeling a single attention block of GPT-2 for the prompt: "The capital of Japan is the city of". The pop-up text windows are (from top to bottom): One of the memory values, whose source the input token "Japan" and whose projection is highly correlated with the output of the model, "Tokyo" (1). The residual stream and the labels of its connected nodes (2). The input the attention block after normalization, which its most probable token is "London" (3), One of the most activated neuroms of Wo that has negative coefficient. Its projection highly unaligned with the model's output, which the negative coefficient suppresses (4). At the block's input, the chance for "Tokyo" is < 1%, but at its output it is 25% (purple pop-up window (2)), i.e., this attention block prompts the meaning of "Tokyo". The two biggest heads are "Yamato" (with Japanese concepts) and "cities", which together create the output "Tokyo".
capture its head meaning. Another head might have the meaning of the token "citics", and together the output of the attention could be "Tokyo".
0.75
Wflent on ank
head
layer index
4 Modeling the Information Flow as Flow-Graph
Figure 5: Mean Ię=50 for the 3 top biggest by attention score memory values, comparing to their head output.
As in most neural networks, information processing in an autoregressive LM can be viewed as flow graph. The input is a single sentence (a sequence of tokens), the final output is the probability of the next word, with intermediate nodes and edges. Geva al. (2022b, 2021) focused on information flow in and across the MLP blocks, while our anal- ysis in section focused on the information flow in the attention block. In this section, we describe how to construct a readable and succinct graph for the full network, down to the level of individual neurons. Our graph is built collected HS from single forward pass: it uses single and small sets of HSs as nodes, while edges are the interactions between nodes during the forward pass.
3.4 Interim Summary
The analysis pictures a clear information flow, from a semantic perspective, in the attention block: [1] the block's 's input creates distribution on the pre- vious keys resulting in set of attention scores for each head (subsection 2.2), [2] which trigger the memory values created by previous tokens, where only the ones with the highest attention scores cap- ture the head semanties (subsection 3.3). [3] The heads are concatenated into one vector, promoting the semantics of only a few heads (subsection 3.2) after they are projected to the vocabulary through Wo (subsection 3. 1). An example of this procedure is shown for the prompt "The capital of Japan is the city of". with the expected completion "Tokyo", in Figure 1 for the flow in full block and in Fig- ure 4 for the flow in the attention sub-block. An input token like "Japan" might create memory value with the meaning of Japanese concepts, like "Yamato" and "Samurai". This memory value can
One option for constructing flow graph is to follow the network's 's full computation graph. Com- mon tools do this at the scale of matrices (Roeder, 2017), coarser than the neuronal scale seek. They usually produce huge, almost unreadable. graphs that lack information on which values are passed between matrices and their effect. Sim- ilarly, if we were to connect all possible nodes

---

Direct effect on logit difference
MAdE hoad MABUI DYKK. TNT. MA9 eeploree Yoodd DABRAC "arr 10%% rark 502570 49TWW
Logitdif:
Supptent Iiakee
pnttec rare
50%
No08:
TIAS DOU May piT 999% rarec 11 SMITW
50%
DE
Head
(a) Tabular information from Wang al. (2022) for GPT- 2 small, identifying the Name Mover and Negative Name Mover Headds, measured by strongest direct effect the final logits.
PIELSS ISTTIE
(b) The corresponding flow-graph for layer attention from GPT-2 small.
Figure 6: While 6a shows quantitative results, the graph model in Figure 6b shows qualitative information that is otherwise difficult to notice: The attention's ' LN is the first place in the model where the attention input's most probable token Mary" (I). The Negative Name Mover Head from layer 10, represented by the blue cell in the s 10-th row, is visualized in the graph with red pop-up showing it assigns the token MMar"" its lowest possible ranking, meaning its role
reduce the probability of this token (2). The output attention block is the token John" but its second most probable output is "Mar"" with around 2% chance (3). However, when added to the residual, together they predict almost 93% chance for Mary"" (4).
(neurons and HSs) and edges (vector multiplica- tions and summation), the graph would be unread- able, as there are thousands of neurons in each layer. Moreover, our analysis section shows that many components are redundant and do not affect the model's 's intermediate processing. There- fore, based on the hypothesis that the neurons with the strongest activations exert more significant in- fluences on the output, we prune the graph to retain only the most relevant components: by assigning scores the edges each computation step, like ranking the attention scores for the edges connected to each memory value, or the activation score for neurons in MLPs, we present only the edges with the highest scores at each level. Nodes without any remaining edge are removed. The goal is to present only the main components that operate at each block. See subsection A.4 for details.
5 Example of Use and Immediate Discoveries
The flow-graph model is especially beneficial for qualitative examinations of LMs enhance search and make new discoveries. In this section, we demonstrate this with several case studies.
5.1 Indirect Object Identification
Recently, Wang et al. (2022) tried to reverse- engineer GPT-2 small's computation in indirect object identification (IOI). By processing prompts like "When Mary and John went to the store, John gave a drink to", which GPT-2 small completes with "Mary", they identified the roles of each atten- tion head in the process using methods like chang- ing the weights of the model to see how they affect its output. One of their main discoveries was at- tention heads they called Name Mover Heads and Negative Name Mover Heads, due their part in copying the names of the indirect object (IO, "Mary") or reducing its final score.
To present the semantic information flow, we assign each node with its most probable projected token and the ranking it gives to the model's isssi final prediction, according the logit lens. Each node is colored based on its ranking, thereby emphasizing the correlation between the node's meaning and the final prediction. Additionally, we utilize the width the edges to reflect the scores used for pruning.
We ran the same prompt with the same LM and examined the flow-graph it produced. The flow graph (Figure 6b) is highly correlated to Wang et al. 's results (Figure 6a). While they provide table detailing the impact of each attention head on the final prediction, our graph shows this by indicat- ing which token each head promotes. For instance, heads that project the token "Mary" among their most probable tokens are the Name Mover Heads,
Figures and show static examples on one sentence, the first for a single transformer block's graph and the second with an annotated explanation on the attention sub-blocks's 's sub-graph.

---

while Negative Name Mover heads introduce the negative meaning of "Mary" (evident by the low probability of "Mary" their projection, high- lighted in red). Not only does our model present the same information as the paper's table, which was produced using more complex techniques, but our modeling also allows to observe how the attention mechanism scores each previous token and recalls their memory values. For example, observe that the Negative Name Mover in layer 10 obtains its semanties from the memory value produced by the input token "Mary".
mean probability difference
0.c3
Sorddd
Buddhism
pubsidiartes
1ct
real
the
random
10000
23306 30333
40600
53300
token Index
Figure Differences token probabilities before and after LN In from layer 15 of GPT-2 medium, according to the generations from section 3. The horizontal axis the index of all tokens in GPT-2 and the vertical shows if the token lost gained probability from the process (negative or positive value). We annotate the tokens that most affected.
We do not claim that our model can replace the empirical results of Wang et al. (2022), but could help speed up similar research processes due to the ability to spot qualitative information in an intuitive way. Also, the aligmment between the two studies affirms the validity of our approach for a semantic analysis of information flow of GPTs.
neurons out of 4096 neurons in a given layer), At least one of these neurons in each layer results in function words when projected with the logit lens, which are invalid generations in setup. We fur- ther observe that these neurons have exceptionally high norms, but higher-entropy token distributions (closer to uniform), when projected via the logit lens (Figure 8). This suggests that these neurons do not dramatically change the probabilities of the final predictions.
5.2 Layer Norm Sub-Block Filter
Layer norm (LN) is commonly applied to sub- blocks for numerical stability (Ba et al., 2016) and is not associated with the generation components, despite having Icarnable weights. We investigate the role of LN, focusing on the first LN inside a GPT-2 transformer block, Ins, and apply the logit lens before and after it. We use the data from sec- tion 3 and, as a control group, random vectors. Fig-
[' the a'. "and"
norm value
ure 7 shows change in logit lens probability of all tokens after applying LN. The tokens whose prob- ability decreases the most are function words like "the", "a" or "not", which are also tokens with high mean probability across our generations (although they are not the final prediction in the sampled generations). Conversely, tokens that gain most probability from LN are content words like "Mi- crosoft" or "subsidiaries". See more examples and analyses of the pre-MLP LN, Ing, in Appendix These results suggest that the model uses LN to introduce new tokens into the top tokens that compares at each block.
1V. the, the 91
1082
2191
average
average of the 100 tapmost activetec neurons
10
12
14
16
entropy
Figure 8; Entropy and norm of "regularization neurons" from the second MLP matrix of layer 19 compared to the matrix average and the 100 most activated neurons across 100 prompts from CounterFact.
By plotting these neurons' weights, we find a few outlier weights with exceptionally large values (Figure 9). Since these neurons are highly activated, the outlier weights contribute to the phenomenon of outlier or rogue dimensions in the following HS, described in previous work (Puccetti et al., 2022; Timkey and van Schijndel, 2021; Kovaleva et al., 2021). This line of work also shows that ignoring those dimensions can improve similarity measures between embedded representations, while ignoring them during the computation of the model causes a
5.3 Regularization Neurons
While browsing through many examples with our flow graph model, we observed some neurons that are always activated in the MLP second matrix, FF2. We quantitatively verified this using data from section and found that each of the last lay- ers (18-23) has at least one neuron that among the 100 most activated neurons more than 85% of the time (that is, at the top 98% most activated

---

significant drop in performance.
Jurafsky, 2021). See Rogers et al. (2020) for an overview.
Our analysis adds a semantic perspective to the discussion on rogue dimensions: since these neu- rons' projections represent "general" notions (not about specific topic, like capitals or sports) and since they have high entropy, they might play role of regularization or a sort of bias that is added constant the residual stream. Finally, to reflect such cases, we paint all the accumulation edges in our flow-graph (where vectors are summed up) in grey, with darker shades expressing lower entropy.
Our work different from feature attribution methods (Ribeiro et al., 2016; Lundberg and Lee, 2017), which focus on identifying the tokens in the input that exert greater influence on the model's s prediction. Some studies visualise the inner compu- tation in LMs. For example, the work of Geva al. (2022a) tries to look into the inner representation of model by visualizing the logit lens projection of the HSs between blocks and on the MLP weights. Other tools that focused on the attention described the connection between input tokens (Hoover et al., 2020; Vig and Belinkov, 2019) but did not explore the internals of the attention module. There are general tools for visualizing deep learning mod- els, like Roeder (2017), but they only describe the flow of information between matrices, not between neurons. Strobelt
value
all
1082 (regulerization neuron) regulerzatton neuren)
(lypicll neuront
(2018a,b) visualize hid- den states and attention in recurrent neural network models, allowing for interaction and counterfactual exploration.
3376 Hypical meurro))
200
400
60D
800
1000
hidden state entry index
Figure Plotting the value each entry in the reg- ularization neurons layer 19, comparing the mean neuron and presenting two randomly sampled neurons that represent typical neurons. Those high magnitudes of the entries in the regularization neurons help in the creation of the rogue dimensions phenomena.
7 Conclusion
In this work, we used token projection methods to trace the information flow in transformer-based LMs. We have analyzed in detail the computation in the attention module from the perspective of in- termediate semantics the model processes, and as- sessed the interactions between the attention mem- ory values and attention output, and their effect on the residual stream and final output.
6 Related Work
Derived from the original logit lens (nostalgebraist, 2020), several studies analyze the role of each com- ponent in LMs using token projection (Geva et al., 20226; Dar et al., 2022). In the last few months, new studies suggest trainable transformation for projecting HS (Din et al., 2023; Belrose et al., 2023), promising to better project HS in the earlier layers of LMS (which currently seems to have less alignment with the final output than later ones).
Based on the insights resulting from our anal- ysis, we created a new tool for visualizing this information flow in LMs. We conducted several case studies for the usability of our new tool, for instance revealing new insights about the role of the layer norm. We also confirmed the validity of our approach and showed how it can easily support other kinds of analyses.
Our tool and code will be made publicly avail- able, in hope to support similar interpretations of various auto-regressive transformer models.
Other work took more mechanistic approach in identifying the role of different weights, mostly by removing weights or changing either weights or activations, and examining how the final prediction of the altered model is affected (Wang et al., 2022; Meng et al., 2022, 2023; Dai et al., 2022).
Limitations
Our work and tool are limited to English LMs, in particular different types of GPT models with multi-head attention, and the quantitative analyses are done on dataset of factual statements used in recent work. While our methodology is not specific to this setting, the insights might not generalize to
There has been much work analyzing the atten- tion mechanism from various perspectives, like try- ing to assign linguistic meaning to attention scores, questioning their role as explanations or quantify its flow (Abnar and Zuidema, 2020; Ethayarajh and

---

other languages or datasets.
References
In this work we interpret HS and neurons using projection methods which are still being examined, well the idea of semantic flow. The way measure impact and distance between HS using 4 (the intersection between their top tokens) is not ideal since it might not convey the semantic connec- tion of two different tokens with the same meaning. While it is possible to achieve more nuanced mea- surements with additional human resources (users) semi-automatic techniques, there would be limi- tations in mapping a vast number of neurons and their interactions due to the enormous number of possible combinations. Therefore, we deliberately chose not to employ human annotators in our re- search.
Samira Abnar and Willem Zuidema. 2020. Quantify- ing attention flow in transformers. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190-4197, On- line. Association for Computational Linguisties.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton, 2016. Layer normalization. stat, 1050:21.
Nora Belrose, Zach Furman, Logan Smith, Danny Ha- lawi, Igor Ostrovsky, Lev McKinney, Stella Bider- man, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens. arXiv preprial arXiv:2303.08112
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Pupers), pages 8493- 8502, Dublin, Ireland. Association for Computational Linguistics.
Our pruning approach is based on the assump- tion that the most activate neurons are the ones that determine the model's final prediction. Although this claim is supported by our qualitative analysis, we cannot claim that the less activated neurons are not relevant for building the prediction. Since our flow-graph model does not show those less active neurons, it might give misleading conclusions.
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2022. Analyzing transformers in embedding space. arXiv preprint arXiv:2209.055:
Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. 2023. Jump to conclusions: Short- cutting transformers with linear transformations. arXiv preprint arXiv:2303.0944
Finally, our methods do not employ causal tech- niques, and future work may apply various inter- ventions to verify our findings. Our tool tries to reflect what GPT "thinks", but further investigation of its mechanism is needed before approaching a full understanding of this "black box".
N Elhage, N Nanda, C Olsson, T Henighan, N Joseph, B Mann, Askell, Y Bai, A Chen, T Conerly, al. 2021. A mathematical framework for transformer circuits.
Acknowledgements
Kawin Ethayarajh and Dan Jurafsky. 2021. Attention flows are shapley value explanations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 49 54, Online. As- sociation for Computational Linguistics
This work was supported by the ISRAEL SCI- ENCE FOUNDATION (grant No. 448/20), Open Philanthropy, and an Azricli Foundation Early Ca- reer Faculty Fellowship.
Mor Geva, Avi Caciularu, Guy Dar, Paul Roit, Shoval Sadde, Micah Shlain, Bar Tamir, and Yoav Goldberg. 2022a. LM LM-debugger: An interactive tool for inspec- tion and intervention in transformer- baseddder-baaeddddd language models. In Proceedings of the The 2022 Confer- ence Empirical Methods in Natural Language Processing: System Demonstrations, pages 12-21, Abu Dhabi, UAE, Association for Computational Linguistics.
Ethics Statement
Our goal is to improve the understanding of LMs by dissecting inner layers and intermediate results of GPT. The semantics behind some projections might appear offensive and we want to be clear that we have no intention of such. Further work might our new tool to try to identify components of the model that control given idea or knowledge, and edit it. We hope such a use case would be for better representing information and not for spreading any hate.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoay Gold- berg. 2022b. Transformer feed- forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 30-45, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

---

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods Natural Lan- guage Processing, pages 5484-5495.
Lutz Roeder. 2017. Netron, Visualizer for neural net- work, deep learning, and machine learning models.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866.
Adi Haviv, Ido Cohen, Jacob Gidron, Roei Schuster, Yoav Goldberg, and Mor Geva. 2023. Understanding transformer memorization recall through idioms. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pages 248 264. Association for Computational Linguistics.
H. Strobelt, S. Gehrmann, H. Pfister, and A. M. Rush. 2018a. Lstmvis: A tool for visual analysis of hidden state dynamics in recurrent neural networks. IEEE Transactions on Visualization and Computer Graph- ics, 24(01):667-676.
Hendrik Strobelt, Sebastian Gehrmann, Michael Behrisch, Adam Perer, Hanspeter Pfister, and Alexan- der M Rush. 2018b. Seq2seq-vis: A visual debug- ging tool for sequence-to-sequee transactions on visualization and computer graphics, 25(1):353-363.
Benjamin Hoover, Hendrik Strobelt, and Sebastian Gehrmann. 2020, exbert: A visual analysis tool to explore learned representations transformer mod- cls. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 187-196.
models. IEEE
William Timkey and Marten van Schijndel. 2021. All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. In Proceedings of the 2021 Conference on Empir- ical Methods in Natural Language Processing, pages 4527-4546.
Plotly Technologies Inc. 2015. Collaborative data sci- ence.
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. 2021. Bert busters: Outlier dimensions that disrupt transformers. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages -405.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you nced, Advances neural information processing systems, 30.
Max Lamparth and Anka Reuel. 2023. Analyzing and editing inner mechanisms of backdoored language models. arXiv preprint arXiv:2302. 12461.
Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACI. Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63-76.
Scott M Lundberg and Su-In Lee. 2017. A unified ap- proach to interpreting model predictions. Advances neural information processing systems, 30.
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual asso- ciations in GPT. Advances in Neural Information Processing Systems, 36.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: A 6 Billion Parameter Autoregressive Lan- guage Model. https: thub. wtt.tttco.o.ooool//..../ com/kingoflolz/
Kevin Meng. Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023. Mass- editing memory in a transformer. International Con- ference on Learning Representations.
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small. In NeurlPS ML Safety Workshop.
nostalgebraist. 2020. interpreting gpt: the logit lens.
Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Dell' Orletta. 2022. Outliers dimensions that disrupt transformers driven by frequency. arXiv preprint arXiv:2205.1 1/30.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl blog.
Marco Tulio Ribeiro, Sameer Singh. and Carlos Guestrin. 2016. why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1 1135- l 1144.

---

A Modeling GPTs as a Flow-Graph
score) into its corresponding head Aj. We cre- alc a node for each value vector and connect it with an edge to its corresponding key vec- tor. Furthermore, we create a node for each summed-up head A, and connect it to all of its memory value vectors. This establishes a direct path between each head's query Qjt, its keys kjá. its values Uje. and the head's final vector Aj. It is important to note that the cal- culation of attention scores non non-linear and preserves the relative ranking among memory values.
This section presents formal construction of GPTs as Nlw-graphs for single forward passes, followed by more implementation details. The information here supplements the brief description given in sub- section 2.2 and is brought here for completeness.
Like any graph, our graph is defined by a set of nodes (vertices) and edges (links). In our case, the graph follows a hierarchical structure, starting with the breakdown of the entire model into layers. followed by sub-blocks such as attention and MLP blocks, and eventually individual or small sets of neurons. A GPT model consisting of L transformer blocks denoted as Bi (0 < < L). where WQ- WK. Wy, Wo represent the matrices for the atten- tion block, and F1 = WFFI and FF2 = WFF2 represent the matrices for the MLP. We now walk through the forward computation in the model and explain how we construct the flow graph. Figure 10 over-viewing the process.
5. The h heads Aj are concatenated, resulting in a vector Aumentenated with the same size as the model's 's hidden state (embedding size). We create a node for Apmeatenated and con- nect all the heads AJ it.
6. Acoonteeateed is multiplied by Wo to produce the attention output Attn(hs)). We create a node for each entry in Armentenatted and each neuron in Wo, connecting them through edges representing the multiplication process. Additionally, we create node for the output Attn (hs() and connect each neuron to it.
A.1 The Attention Block as Flow-Graph
1.
The input to the I-th block for the 1-th input, hs!, passes through a LN, resulting in a nor- malized version of it. We create a node for the input vector and another node for the normal- ized vector, connecting them with an edge.
7.
The attention output is then added to the resid- ual stream of the model. We create node for the sum of the attention block and the residual, hssattn I residual, and connect it to Attn (hs))-
2. The normalized input is multiplied by Wo- WK. and Wy. resulting in query, key, and value representations (g. k, v). We create a single node to represent these three representa- tions, as they are intermediate representations used by the model. We construct an edge be- tween the normalized input and this node.
8. The attention block also contains skip con- nection, The residual, from the input hst straight to the output hs residuals we connect an edge between them.
A.2 The Feed Forward Block Flow-Graph
3. The last three representations (q. k, v) are split into h heads (qya, KyL, Uy for 0 < j < h). Each head's query vector (qze) is multiplied by all the previous key vectors (kys for 1 < S t), calculating the attention probability for each of the previous token values. We cre- ate a node for each head's query vector and connect it with an edge to the overall query node created in the previous step. Addition- ally, we create a node for each key vector and connect it with edge to its corresponding head's 's's query vector.
This structure is mainly based on the theory of using two fully connected layers as keys and values, as described by Geva et al. (2021)
1. Similar to the attention block, the input to this block, denoted hit = hsatt-residuul (representing the intermediate value of the residual after the attention sub-block), passes through layer norm, resulting in a normal- ized version of it. We create a node for the input vector and another node for the normal- ized vector, connecting them with an edge.
4. Each memory value vector (1is, the memory value of the j-th head and the i-th input token), is summed up with a coefficient (the attention
2.
The normalized input is multiplied by WFFI- For each neuron in the matrix, we create a

---

Figure 10: The overview pro- cess of creating flow-graph modeling (the bottom graph) from a single forward pass (the upper draw). In this toy- example we model a simplified version of MLP sub-block. Each node in the graph is cor- respond to a static weight or HS in the upper diagram and labeled by its logic lens projec- tion, for example: the input has the meaning of "London" and the output has the meaning of "Tokyo".
Input hidden state
first MLP matrix
sctivations (mid results)
sccond MLP matrik
ounput hidden state
single neuron's 5 weightts: "Japan"
logit hidden state: "Tokyo""
lens single static iihtt:" "Asla"
The growgraph modeling, Eath Node assign with withits lagit lens projection
pruning less
activete neurons
Asia
Japan
London
Takyo
cities
capital
node and connect an edge from the normalized input to it (corresponding to the multiplication of each neuron separately).
graph for each transformer block by the concatena- tion of its attention graph and MLP graph, where the two graphs are connected by edge between the attention's s hsattm+ressddull and the MLP's lisf. The input to the new graph is the input of the origi- nal attention sub-graph, and its output is the output of the original MLP sub-graphi
3. The result of the previous multiplication is a vector of coefficients for the second MLP ma- trix, WFF2. Consequently, for each neuron in WF FFF and connect an edge between each neuron and its corresponding neuron from WFF1- It important to note that the actual process includes a non-lincar activation between the two matrices, which affects the magnitude of each coefficient but not its sign (positive or negative).
create node
To define the graph of the entire model nect all its transformer blocks' sub-graphs into graph by connecting an edge between each block's sub-graph output and the input of its following block's sub-graph. The input to the new graph is the input of the first block and the output is the final block output.
4. The neurons of WFF2 are multiplied by their coefficients and summed up into a single vec- tor, which serves as the output of the MLP block, denoted as MLs(sss). We create a node for M (hsss) and connect all the neu- rons from W WFF2 to
A.4 Scoring the Nodes and Edges
In order to emphasize some of the behaviors of the models, we define scoring functions for its nodes and edges.
Scoring nodes according projected token ranking and probability: as we described, each node is created from a vector that we project to the vocabulary space, resulting in a probability score that defines the ranking of all the model's 's tokens. Given specific token and single vec- define its neuronal ranking and probability, Frank (w) and Uprot( (w), as the index and probabil- ity of token w in the projected vector of v.
5. The output of the block is then added to the model's 's residual stream, We create a node for the sum of the MLP block and the residual, denoted as hs I.P t. and connect it to Lhss
6. Similarly to the attention block, the MLP block also includes skip connection, di- rectly connecting the input hi/ to the output hs LP residual- Therefore, we connect an edge between them.
Scoring edges according to activation value and norm: There are two types of edges: edges that represent the multiplication of neurons with coeffi- cients (representing neuron activation) and edges that represent summation (as part of matrix multi- plication). Edges that represent multiplication with coefficients are scored by the coefficient. We also include this case the attention scores, which are
A.3 Connecting The Graphs of Single Blocks Into One
GPT-2 each transformer block contains an atten- tion block followed by a MLP block. We define a

---

used as coefficients for the memory values. Edges that represent summation are scored by the norm of the vector which they represent. This scoring aims to reflect the relative involvement of each of the weights, since previous work found that neu- rons with higher activation or norm have stronger impact on the model behavior (Geva et al., 2022b).
allowing us examine only few blocks a time.
The above simplifications help construct scal- able graph that humans can easily examine.
A.6
Implementation Details and how to Read the Graph
A.5
Modeling Single GPT Inference as a Flow-Graph
We use the Python package Plotly-Express (Inc., 2015) to create a plot of the model. We will pro- vide all the source code we created to model the GPT-2 family models (small, medium, large and XL) and GPT-J (Wang and Komatsuzaki, 2021), which includes configuration files that allow ad- justing the tool to other decoders with multi-head attention. We are also providing the code to be used as a guided example with instructions designed facilitate the adaptation of our flow-graph model to other GPT models.
Given a prompt 21,...,22 2t we pass it through a GPT model and collect every HS (input and output of each matrix multiplication). Then we create the flow-graph as described above, where the input and HS are according to the last input token and the attention memory (previous keys and values) corre- spond to all the input tokens. This process results in a huge graph with many thousands of nodes even for small models like GPT-2 small (Radford et al.. 2019), which in this sense can only be examined as tabular data, similar to previous work. Since our goal is to emphasize the flow of data, we reduced the number of nodes according to our discoveries and the assumption that neurons in the MLP blocks with relatively low activation have a small effect on the model output (Geva et al., 2022b). We also note that with a simple adjustment our model can show any number of neurons or show only chosen ones.
Using our tool is straightforward and only re- quires running our code. The flow-graph plot can be presented in your software enviromment or saved as an HTML file to view via browser. Personal computers and environments like Google Colab are sufficient for modeling LMs like GPT-2 medium, even without GPU. Plotly-Express allows us to inspect the created graphs interactively, like see- ing additional information when hovering over the nodes and edges, or to filter some of them by the "Select" options on the top right of the generated plots. The basics on how read and use the flow-graph plots are:
The reduced graph is defined as follows:
In the attention sub-graph, we chose to present all the nodes of the heads' query and output, Ojt, Uytz-A; Ay. but to present only the memory keys and values, kys, Ejj, that received the highest attention score, in light of the results from Sec- tion 3.1. We also decided to present only the top most activated neurons of Wo, according to the largest entries (by absolute value) from its coefficients HS, Aconcated-
The flow is presented from left to right (ma- trices that operate carlier during the forward pass will be to the left of later ones). When plotting a single block we can identify the at- tention sub-block (the first from the left) and the MLP sub-block as they are connected by a wide node and by separate and parallel wide edges representing the residual (each with a slightly different color). When plotting more identify the different
In the MLP sub-gran aub-graph we decided to show only the nodes of the most activated neurons. The activation is determined by the highest absolute values in the HS between the two matrices after the nonlinearity activation. That is, we examine the input to the second MLP matrix WrF2 and present only the nodes that are connected to its highest and lowest entries.
than one block blocks by the repetitive structure of each.
Each node is labeled with its most probable projected token. When hovering over node, we can see from which layer and from which HS or matrix was taken (the first number and the follow-up text the pop-up text win- dow. For example: "10) attn-input" suggest this node is the input of the attention sub-block
We make possible to create a graph from only part of consecutive transformer blocks,

---

pravet capta "Ayertar"
Option bar: save the plot as Image,
13
ose nodes hide from the graphs
Tracing allertinnllck
(box loss select) and view
rreated ins outputt by following the Intormation now
Aggataa ACE 3.VL.
Holden
MLP block (using box swwect option)
Figure 11: Modeling block number 8 of GPT-2 small for the prompt: "Buenos Aires is the capital of",, which the model answers correctly with "Argentina". By using the option bar (top right) we hide the MLP's nodes and focus on the attention sub-block. When hovering over the attention input node (1) a pop-up text window shows information about its corresponding HS, revealing its top projection tokens and how this HS ranks the token "Argentina" (giving it less than a 1% chance). Comparing the input to the output HS of the block (2) we can understand this block promotes the token "Argentina" (the output ranks with around 63% chance). order to identify how this block creates its prediction we follow the flow of the model and notice attention head number 11 (3), the with the largest norm from all the heads (we can see this from the width its connected edge which is proportional to its norm). Its top projection token is "Argentina" and we want to understand how along the flow to its memory values (heads the sum of their memory values). We identify that the memory value that had the largest attention score (4) was created from the input token "Aires" (as shown on the pop-up window). This memory value's most probable projection tokens are "Aires", "Argentine", "Argent" and "Argentina" having high intersection with the most probable tokens of its head's projection and the attention output's 's projection.
created. To do that, we go
in layer 10). The other information when hov- ering over each node its top most probable tokens (a list of tokens) and "status", suggest- ing its relation with another token, "target", chosen by the user (if given); in particular, its probability and ranking for that token.
user invoking the code can choose the model, the prompt, which layers to present, and "target" token (recommend to the ac- tual output of the model for the given prompt).
B Walkthrough the Graph Model
The flow-model is an interactive plot. At the top right of the screen there an option bar that enables to focus on specific parts of the model, by hiding chosen nodes. By examining different blocks and focusing on chosen parts of the graph we gain in- sights into the predictive mechanisms of the models and how they create their predictions. In Figure 11
In the attention score calculation can lo-
cate which previous key and value were ere- ated by which of the input tokens, since they have the same indexes in the attention mem- ory implementation of GPT-2. We present this information by hovering over the nodes in the attention sub-graph.
explore how gpt-2 small recalls factual in- formation, tracing which input tokens created the memory value Uy whose head Aj is responsible for the output of the block (showcasing the patterns we identify section 3). Similar to subsection 5.1 our findings do not assert that the identified com- ponents exclusively control the model's final pre- diction. Rather, they are recognized as the primary elements responsible for shaping the immediate prediction.
Hovering over an edge presents which nodes it connects to along with information about what represents, for example: if is an edge between attention query and key, it will represent the attention score between them. If the edge represents summation of one HS into another, the information on the edge will be the norm of the summed HS.

---

C Model Selection
norm
Ik attn
As mentioned, we used GPT-2 medium (355M pa- rameters) as our main case study due to its avail- ability, wide use in previous research, the ability to it even with limited resources, and the sumption that characteristics we see with it also relevant to bigger models. To validate ourselves, wc also ran parts of our quantitative analysis with GPT-2 XL (1.5B parameters) with the same setup as we had with the medium model, and observed the same behavior; for example, see Figure 12. For these reasons we believe our analysis and modeling are applicable general GPT-based models and not only to specific model.
022
0.0
12 14 layer index
16 10 20
as-
(a) Mean f==so for only the 3 heads with the largest norm, comparing to attention block output.
C.G
A attn
0.4
A finsl
0.2
0.0
10
15 10 23
layer index
(b) Mean Iu-to for only the top-norm head.
Figure 13: Projecting attention heads for prompts from CounterFact, without filtering prompts the model does not answer correctly. These show almost the same sults as with the main setup in Figure 3, suggesting the mechanism behind the model's attention works the same for correctly recalling factual knowledge and when pre- dicting tokens of function words.
norm
0.2
6.9
10 12 14
16
16 20 22 24 26
30 34 36
40 02 45
layer index
(a) Mean Io=50 for only the 3 heads with the largest norm. comparing attention block output.
V attn
To avoid editing prompts analyzing uninterest- ing cases, we decided only prompts that the model answers correctly. A plausible question is whether the model acts differently when it prediets the right answer compared to the general case, with- out filtering by answer correctness. To examine this we ran our analysis twice, once with only prompts the model knows to answer (like we explain in Sec- tion 3) and another time with random prompts from CounterFact. It turns out that the attention mecha- nism works the same way in both setups, resulting in almost the same graphs (Figure 13), which sug- gests that the behavior we saw is not restricted to recalling factual knowledge.
final
6.
20
36
layer index
(b) Mean Jи=50 for only the top-norm head, comparing attention block output, layer output, and the model's final output.
Figure 12: Projecting attention heads of GPT-2 xl, with the same setups in Figure shows that the patterns we saw with GPT-2 medium are similar to the ones we see with bigger model.
D Additional Quantitative Analysis of Information Flow Inside the Attention Blocks
The only main difference we notice is the proba- bility score the models give their final prediction along the forward pass: when the model correctly predicts the CounterFact prompt (meaning it recalls a subject) it starts to assign the prediction high prob- abilities around its middle layers. However, when the model predicts incorrectly (and mostly predicts function word), it assigns moderate probabilities starting from the earlier layers (Figure 14). This might suggest for later works to examine if factual knowledge, which less common than function words in general text, located in deeper layers as opposed to non-subject tokens.
D.1 Additional Setup Information
We provide here additional information on our setup and data selection. The choice of using Coun- terFact is based on its previous usage in studies on identifying where information is stored in mod- cls (Meng et al., 2022, 2023). However, it has the issue that GPT-2 does not succeed in answering most of its prompts correctly (only approximately 8% for GPT-2 medium and 14% for GPT-2 xl), and in many cases, the model's predictions consist primarily of function words (like the token "the").

---

probabii by loder by someed arcower) hor modelss pred along layars
probabiltty
correct ertwwer
Foise
lx head
score Fank
9
11
13
0.6
0.5
32
24
0.4
0.0
10 12 14
16
17
19
20
21
layer index
10 12 151617181990221
Figure 14: The probability GPT-2 medium assigns to its final predictions' tokens for the projection of the HS between blocks, colored by whether the model returns the true answer or not.
Figure 16: Comparing == of memory values with the output of their heads, according the memory value norm rank compared to other values in the same head (the complete analysis behind Figure 5). This example claims that the semanties of each head is determined its top memory value since only the top 1-3 memory values some semantic intersection with their heads (starting from the 4-th layer) and the rest of the heads have almost no intersection (the numher 14 suggest that the longest input we used for this experiment was tokes).
D.2 Additional Results
We add more graphs the analysis in Section that help explain our claims the conclusion of that part. All results are taken from the same experi- ment we used in that section. Notice that according to the following analysis the model exhibits dis- tinct behavior during its initial 4-6 layers (out of 24) compare to the subsequent layers, as indicated by the low J& scores for the first layers (Figures 15, 16), behavior that was noted in previous work (Geva et al., 2022b; Haviv al., 2023; Dar al.. attention output and the residual. It showcases the incremental changes that occur in the residual as result of the attention updates to it. Similar to how the MLP promotes conceptual understanding within the vocabulary (Geva et al., 2022b), the at- tention layers accomplish similar effect from the perspective of the residual. The figure also reveals the high semantic similarity between each attention sub-block and its preceding attention sub-block.
[k-vji-Aj
1.
ittention Index
0.5
10 16 layer index
2022) and yet to be fully understood.
Figure 15 illustrates the relationship between the
Figure 17: Comparing 50=50 of memory values with the output of their heads, according to head indices. This shows that there are no particular heads that are more dominant than others (after the first few layers).
Figure 18 demonstrates the alignment in projec- tion correlation between each input token and its corresponding memory values. For every memory value Ujg. we examine the probability of its input token (the i-th input token) after applying a logit lens to 0zz. Our underlying assumption is that if the generated values share common semantics, then the probability of the input token should be higher than random (which is nearly 0). The results sub- stantiate this assumption, revealing higher scores in the subsequent layers.
prey res doal
residual
block
block
prev aitn
0.0
10 11 12 13 14
16
10 19
layer index
Figure 15: Comparing 5==50 of attention output with its current and previous residual (just after it is updated with the attention output) and the block output (note that the imput of the attention sub-block its previous block output). The intersection between the attention output is considered high, which means that the attention sub- blocks have overlapping semantics between different layers.
probability
0.00
0.00 6
8
10 12 1514
15 1b
18 20
layer Index
Figure 18: The probability input token in the vectors of memory values they generated.
Figure demonstrates that the information flow we saw from the memory values to the heads output is a behavior that applies to all heads.

---

Agrooor (proo]
0.075
0.07
0.255
0.025
layer index
layer index
(a) Naive projection without any circuit for k.
(b) With the QK circuit
Figure 19: Comparing Je-50 token projection alignment between the head ouiputs of Wi and Wy (k2 and ex), with and without the QK circuit for Wx (1; projected with Wo and can see as the output of the OV circuit).
Sprojl
0.05
0.071
uose
ausa
0.025
16
1011
layer index
layer index
(a) Naive projection without any circuit.
(b) With the QK circuit
Figure 20: Comparing Je-50 token projection alignment between the head outputs of Wq and Wx (q: and k,), with and without the QK circuit. The mean intersection for two random sampling of 50 items (without duplication) from set the size of GPT-2 vocabulary (50257) around 0.05 matches, which equals to Ta-50 of 0.001. However, when we apply the logit lens to two random vectors, it is observed that due to certain biases in the decoding process, the average 50 value is 0.002.
D.3 Are All HS Interpretable? Examining the QK Circuit
behind that is to check the associations between different ideas (for example, unclear association can be head's keys k with meanings about sports but with values vy about the weather). Another option is that the output Wq, WK operates in a different embedding space, which is different than the rest of the model, explaining why logit lens would not work on it. A support for this idea can be the fact that the output of these matrices is not directly summed up with the residual, but is only used for computing of the attention scores (that are used coefficients for e; which are summed into the residual).
Similar analysis of the attention matrices Wy. Wo (section 3), we try to find alignment be- tween Wo: WK outputs and other HS of the model. The work of Dar et al. (2022), who first projected the matrices WQ. WK. emphasizes the importance of projecting the interaction between the two using the QK circuit, meaning by projecting the matrix Wok = We WK. Using the data from section 3, collected dynamic HS that these matrices gen- erate, qi and k; (attention queries and keys), to examine their alignment between each other and be- tween the memory value v; they promote (each k; leads to single vú noting we already saw the latter is aligned with the attention's 's and model's outputs section 3). We project and k; using two methods: once with the naive logit lens (I.L) and once using the QK circuit, by first multiplying & with Wи (LL(g; Wx)) and ks with We (LL(WQ kx)). Our hypothesis was that will some overlap be- tween the top tokens of qi, k and mg; however, the results in Figures 19, 20 show almost no correla- tions using both methods, in contrast to the results we saw with Wp and Wo (subsection 3.1).
In our flow graph model, the user can chose merge qi, ky nodes into one with vi, making them less visible. However, we decided to display them by default and to project them with the QK circuit, since during our short qualitative examination we noticed examples that suggest that the first option introduced might be true. In Figure 6b see that the projection of the key with the highest attention score behind the Negative Name Mover Head holds the meaning of "Mary"" this case, we can imagine that the model implements a kind of statement, saying that if the input has really strong semantics of "Mary", we should reduce a portion of it (maybe, to avoid high penalty when calculating the loss during training).
We believe there are two options for the low scores we see. The first option that We and Wk deliberately promote different tokens, with no alignment between Wo, WK. Wy. The idea

---

E Layer Norm Uses as Sub-Block Filters
We present additional results about the role of LN in changing the probabilities of each sub-blocks' input, including results for both LN layers in GPT- 2. Tables and show the top tokens before and after in for two different layers. Figure 21 gives broader look the effect of [n1, detailing some examples across layers in Table We repeat these analysis with ing in Figure 22 and Tables and 5.
In1 5
In 11
Inj 17
Inj 23
the the the
the
using not a
not North
a
Google
English
this
T
a
India
within
C
South
Russian
in
U
company
German
in now North
We include an example about the LN effect on the HS the projection was done without the model's final LN, Iny, which attached to the decoding matrix. Initially done to examine the effect of In1 and Inz without Inf on projection, the results in Figure 23 and Table highlight the importance of using In, part of the logit lens projection, since the tokens we receive otherwise look out of the context of the text and tokens our model promotes its generation.
and
Germany
South
which not
outside N still
K
Table 3: Tokens that lose the most probability after Ini, as collected from the experiment in section 3. Earlier layers' LNs demote more tokens representing preposi- tions than later layers.
before Ing English the Microsoft
after In1
before Inz the not English
after Inz English Microsoft abroad not abroad
English Microsoft
abroad
North
subsidiaries
North
Microsoft subsidiaries a origin
abroad combining
a
downtown
origin
combining
London
Redmond
T
the
India
origin
U
photographer
origin
London
photographer
renowned
Table 1: The top tokens before and after In; layer 15, according to the mean HS collected in section 3. We how the LN filters all the function words from the 10 most probable tokens while introducing instead new tokens like "Redmond" and "downtown".
Table 4: The top tokens before and after Inz at layer 13.
In2 5
In2 11
Inz 17
Ing 23 the
the
the the
before In,
after Inz
in
T French a
Google
the abroad Microsoft
German
not
using U Boeing North
abroad
subsidiaries
in company South
combining
and a London "
K
origin English
Microsoft
origin
now
not
N
not
this
and
sports
Kaw
Europe
Europe
at
at
hockey
Boeing
U
photographer
C
the
Table 5: Tokens that lose the most probability after Ing. similarly to Table 3.
Table 2: The top tokens before and after Inz at layer 13.

---

probabilty Leffrr
0.23
real
random
Phu
315-
613
Mircooft
Seceonnni oo ueau
330
Nicrosof:
Nkrth
Matble
downtown
3.00
32800
20005
30000
43000
50003
10000
20600
3I000
20033
55000
token index
taken index
(a) Before.
(b) After.
difference
Micrusoft
mean probability
0.22
-0.1
20306
4C000 50033
token index
(c) Difference (after- before).
The probability of the tokens in GPT-2 before and after the first LN, Ing, in layer 16, including annotation for the largest-magnitude tokens. We the difference in the distributions of tokens between randomly generated vectors and the ones sample from CounterFact, which we find reasonable when answering factual questions. Especially
Figure 21:
questions are about a finite number of domains, uniform way (like the random vectors does). Although tokens like "Microsoft" and "The" have high probability before the LN, while the first gained more probability during the process the second actually loses, suggesting this is not naive reduction to the probable tokens each HS.
network promotes tokens not a
probability befoor
0:22
English
225
English
random
335
0:18
Microsoft
Nicresof:
2.05
Zam
Italy
proinerss
3:00
33800
20002
50000
43250
10002
20000
32500
40003
20000
token Index
taken Index
Before
(b) After
mean probabiiity difference
0.25
0324
Endier engggers
-0.4
Loken index
Difference (after before)
Figure 22: The effect of Ing at layer 16 on tokens' probabilities. We can see similar highly probable tokens as Figure 21, since the only difference between the inputs of inj and Ing, which the residual stream, is the attention output of that layer (which is known to be gradually and does not steer the probability distribution dramatically Figure 14, 15).

---

In 5
Inj 11 In 17
Ing 23
Zen not the
the
imperialist the English Sponsor Europe a abroad
mum
abroad football
utilizing
and
externalToEvAAnly sqor quickshipAvailable 1
UNCLASSIFIED
pure
Toronto sports
conjunction
English
tied
ized first
nineteen
Washington
AAAAÄÄÄ
Table 6: Top tokens that lost probability after applying In1 when projection is done without Ing.
mec meaan probeelliyy before
real
random
pr probabiliity
Ianouages
Stinge
English
50000
indlcx
index
(a) Before
(b) After
difference
English
mean arobability
-04
the
50000
50030
(c) Difference (after before)
Figure 23: The affect of In, layer 16 tokens' probabilities (similar to Figure 21), but when the projection is done without In--


Reviews:
[{"review_id": "pqBlidzuKw", "reviewer": "Reviewer_PqVU", "summary": "", "questions": "* Since the code is not included, how is the applicability of the tool to other models and datasets?\n* How is your visualization tool related to other lines of work that model the information flow in attention? (https://aclanthology.org/2020.acl-main.385.pdf, https://aclanthology.org/2021.acl-short.8.pdf)\n\nEDIT: Both questions are addressed by the authors in the response.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank you for your detailed review and we hope to highlight some of our work strengths compared to similar works.\n\n>how is the applicability of the tool to other models and datasets\n\nAs we mentioned in the implementation details **Implementation Details and how to Read the Graph** the tool is Python based and can be adapted to any decoder-only LM. We provide the setup for GPT-2 and GPT-J, and also a guide (notebook with instructions) on how to adjust our code to other models with only moderate changes.\nThe tool is not limited to any dataset or text, just like we used it in **Figure 6**. If a model can inference a prompt - the tool can display it.\n\nWe want to take this chance to add an example of how we believe this tool can help with different models, datasets, and research questions, like identifying sub-components that store or control a behavior. For that we will use **Figure 11** from the paper: when we investigated its prompt, we first looked at the full graph that included multiple blocks. By focusing on different components we found the specific block where the model first predicts what becomes its final prediction (in the paper we present only this block). Focusing on this block allows us to see which heads' neurons are the ones that carry this prediction as detailed in its caption (including more discoveries), without any previous knowledge about those neurons (on-the-fly). Let us put some excitement to start this rebuttal: this is just one example of a new level of interpretability that can boost many research directions in the field.\n\n>How is your visualization tool related to other lines of work that models the information flow in attention?\n\nThank you for the citations on attention flow. These studies propose algorithms for quantifying the contribution of input features (words) by aggregating attention scores. They thus have a very different goal. In contrast, we interpret intermediate processing steps in the model, and we are doing this with different methods. We will add this distinction to the related work section in our revision. \nIn related to other works we cited in **Introduction** and **Related Work**, we claim that our visualizations are more intuitive and, in an innovative way that does not require any prior knowledge, we add interpretability into the visualizations.\n\n>The core method used for analysis is *LogitLens*... but it was shown to be brittle in a recent work\n\nWe understand your concern, and indeed LogitLens is not perfect, but please note the following: \nThe works you mentioned, Belrose et al. and Din et al., investigate the hidden states between blocks (that is, activations, as a dynamic analysis). By learning a linear/affine transformation, on large amounts of example activations, they find ``shortcuts'' in the projection of the logits. Those works mostly support the well known claims that GPT-like models have redundant calculations (which translates into early exit methods, just like they showed).  However, we analyze various components in the model that were not considered in those two studies: both dynamic components like outputs of attention heads/vectors and static components like neurons in $W_O$ and $FF_1$, $FF_2$. The learnable projections cannot be easily applied to such components (if at all possible) so it hinders the options of projecting every neuron in an efficient way instead of relying on more known and basic method like LogitLens (both Belrose et al. and Din et al. were published just 3 months before the submission deadline).\n\n>The issues of the proposed visualization tool... More careful consideration such as colorblind friendliness\n\nThe tool's colors are highly adjustable. We will be more than glad to share a colorblind-friendly configuration when we publish it.\n\n>approach prunes out neurons that are less activated for better visualization\n\nIndeed, our pruning approach works under the hypothesis that highly activated neurons are more important for the prediction. We will clarify this in the revision. We believe this is a reasonable hypothesis, for the following reasons: First, we first establish the part about investigating the flow of information in the attention sub-blocks (Section 3). Second, we refer to several studies that made similar discoveries for the MLP sub-blocks. Third, we also show neurons that are least activated as a control. We are aware the assumption might not fully hold in every case. However, as a general approach and given the inability to present too many nodes in a single graph, we find that our pruning approach is effective for reflecting most of the interesting components of the model.\n\n>with results shown from a single dataset and a single model variant can be a weakness\n\nWe indeed used a single dataset with GPT-2 as our test-case (mainly for Section 3, which we further explain in **Model Selection** and **Additional Setup Information**), as we follow previous works in this field that also used only GPT-2 as a decoder transformer to analyze (Geva et al. (2021, 2022a), Dar et al. 2022), or  created a toy example decoder-only LM and used it as their only test-case (Elhage et al. 2021). In particular, this line of work demonstrated GPT-2 is a good proxy for auto-regressive interpretability research. \n\nThank you for the grammar and style notes, we will fix those.\n\nPlease let us know if you have any remaining concerns, especially if you would consider revising your assessment after reviewing our reply.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This work proposes a tool to visualize the simplified information flow. Tools like this (if accessible) can facilitate research in understanding a model's decision.\n* Although the analysis part is only based on a single dataset (CounterFact) and a single model variant (GPT), one of the case studies shows the applicability by checking the finding through their visualization tool aligns with the finding in a previous work.", "reasons_to_reject": "* The core method used for analysis is *LogitLens*, which was originally proposed in a well-known blog post, but it was shown to be brittle in a  recent work (https://arxiv.org/pdf/2303.08112.pdf), as also cited by the authors. The authors claim that the basic approach of LogitLens is applied because they are interested in the interim hypothesis instead of the final layer’s output, which may not be convincing enough because the Din et al. paper also presented differences between the original LogitLens versus the improved version.\n    * EDIT: As mentioned in the authors' response, I think this is less of a concern because (1) The paper I mentioned was published too closed to the submission deadline, and (2) there are components where applying these methods would be difficult/inefficient.\n* As mentioned in the strength section, with results shown from a single dataset and a single model variant can be a weakness, although not necessarily a reason to reject.\n* The issues of the proposed visualization tool. (These are more of presentation issues, but putting in this section because the visualization tool is the core contribution of this work)\n    * EDIT: This should've been addressed per the author's response.\n    * More careful consideration such as colorblind friendliness or contrast level should be taken towards the interface design of the visualization tool.\n    * As also mentioned by the authors in the Limitation section, the approach prunes out neurons that are less activated for better visualization, but it has to be under the hypothesis that the more activated neurons are more important for prediction, which may lead to misleading results if this hypothesis fails to hold.", "questions_to_authors": "* Since the code is not included, how is the applicability of the tool to other models and datasets?\n* How is your visualization tool related to other lines of work that model the information flow in attention? (https://aclanthology.org/2020.acl-main.385.pdf, https://aclanthology.org/2021.acl-short.8.pdf)\n\nEDIT: Both questions are addressed by the authors in the response.", "paper_topic_and_main_contributions": "* *Contribution 1:* This work studies the information flow in a GPT model by looking at the attention module using an existing method, LogitLens. \n\n* *Contribution 2:* With the analysis results, the authors propose a tool to visualize the information flow and conduct three case studies to verify the usability of the tool.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Figure 2 a & b: Consider decreasing the maximum $I_k$ values to be shown. Currently, all the bars in 2(a) look squashed into x-axis. It is difficult to see the claim that no correlation between $I_k$ and layer index before $W_o$ projection can be found.\n* Figure 7: some words in the figure are overlapped with each other.\n* There are some tense inconsistencies in different sections (e.g. Sec 3.3, 5.3)."}}, {"review_id": "4gZra4ytyJ", "reviewer": "Reviewer_mg6v", "summary": "", "questions": "Does the code release include the full visualisation tool? Ie. a library where I can query my model (I guess restricted to GPT2) and obtain a flow graph as in Figures 1, 5 and 6, and the functionality the appendix showcases?\n\nWhat should one learn from section 3? While 3.4 tries to explain it, I am still unsure whether anything new is learned from it. Maybe that some heads carry most of the \"semantic meaning\". Still two pages seem too much.\n\nCould you perhaps justify why this paper should be presented as a \"Interpretability, Interactivity, and Analysis of Models for NLP\" conference paper and not as a demo of the tool at EMNLP?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We are glad you see the merit of our flow-graph modeling contributing to future research.\n\nWe want to start with the first point from the reasons to reject since it includes many concerns that we believe that after they are cleared, your perception of our work might change:\n\n>``The method proposed seems too fine-grained... When I look at Figure 1 and 5, I do not see any meaningful \"projections\"... Its utility is only portrayed when one already knows of a specific head phenomena''\n\nThis is exactly what we aim at solving with our new method, and we establish its correctness with the experiments in **Section 3** and **5.1**!\nLet's take for example **Figure 5**: when we investigated its prompt, we first looked at the full graph that included many blocks (i.e., multiple blocks in one plot). By focusing on different components (using the select options from the top right bar) we found the specific block where the model first predicts what becomes its final prediction (in the figure we present only this block). Focusing on this block allows us to see which heads' neurons are the ones that carry this prediction as detailed in its caption (including more discoveries), while all that is being done without any previous knowledge about those neurons (on-the-fly). This example is not hand-picked since it applies to many more cases, such as **Figures 6** and **11**. Let us put some excitement to start this rebuttal: this is just one example of a new level of interpretability that can boost many research questions/directions in the field (we are not aware of any similar approach regarding the ability to reveal such amount of information).\n\n>``What should one learn from section 3'' and the concern about the projection from the first reasons to reject\n\nWe are not the first to project attention components through circuits ($W_O$). In fact, **Section 3** is a continued discussion of Dar et al. (2022) and Elhage et al. (2021) who were the first to so. Both explained why and how to project these components using similar methods and we mainly continued and conducted experiments to prove it for our cases in **Section 3.1** (those are indeed quantitative ones, and please also refer to the Appendix where we further discuss that: **Additional Results** and **Are All HS Interpretable? Examining the $QK$ Circuit**).\nWe realize that it might be hard to understand this without deeply knowing those previous works and we would add an additional technical explanation about this.\n\nSo, what is the difference between us and previous works (or more correctly, why is **Section 3** so long and what we can learn from it): our main contribution here is that we did this in scale at dynamic hidden states (rather than static as in Dar et al.), identifying with interpretability methods where each ``meaning'' comes from (**Sections 3.2, 3.3**). This is a part we found to be missing in the literature and it has an important purpose in supporting the rest of the paper, since it explains why we can filter some of the attention components in our visualization. You mentioned these parts were not surprising and we would argue this is actually a strength(!). Of course it's known that some heads may be more important than others, but identifying them on-the-fly according to their norms like we did is an innovation. \n\n>About the code release (your first question and also about Reproducibility)\n\nOur tool is Python based and we provide implementation details in the appendix (**Implementation Details and how to Read the Graph**). Of course, we will release the full tool.\nCurrently we provide the setup for GPT-2 and GPT-J, and also a guide (notebook with instructions) on how to adjust it to other models with only moderate changes.\nThe tool wraps those models so for every input you can inference, it is straightfoward to display its corresponding flow-graph (and to do so in a few seconds even on simple setup like Colab without any GPU). With a simple configuration you can select which blocks to present on a single plot (you can display multiple blocks on a single graph. In the paper we present only single ones due to lack of space). \nWe do regret in not sharing more examples with this submission since we think it could make our methods more tangible, but unfortunately we can't do this now due to conference rebuttal rules.\n\n>``Could you perhaps justify why this paper should be presented as a \"Interpretability, Interactivity, and Analysis of Models for NLP\"''\n\nConsidering our analysis (**Section 3**) and the research findings we produced from the tool (**Section 5**), we believe we can contribute more by presenting our work under the interpretability track, rather than the demo track, which less appropriate to presenting our analysis.\n\n>``Why only GPT2?''\n\nAs we mentioned, the tool is not limited to GPT-2. \nWe indeed used GPT-2 as our test-case (mainly for **Section 3**, which we further explain in **Model Selection**), as we follow previous works in this field that also used only GPT-2 as a decoder transformer to analyze (Geva et al. (2021, 2022a), Dar et al. 2022), or  created a toy example decoder-only LM and used it as their only test-case (Elhage et al. 2021). This line of work demonstrated GPT-2 is a good proxy for auto-regressive interpretability research. But again, we've also provided the setup for GPT-J and will release example plots in the final version. \n\n>Finally, we want to address your concerns about the structure of the paper:\n\nThe structure we chose for the paper is to (1) fill in the missing knowledge we found necessary for the presentation of our tool (**Section 3**), (2) then presenting the tool (**Section 4**, notice how short is the technical part in the main paper) and (3) demonstrating how it can be used (**Section 5**, including new findings you mention as interesting). Only presenting the tool would not be ideal since it would lack the ability to support its correctness in reflecting the inner computation of LM, and will miss some immediate findings the community for sure can benefit from. Wrapping everything in 8 pages was not easy but we tried our best.\n\nAgain, we want to thank you for your review and hope we answered all your concerns so that you may consider changing some of your scores. We believe that with more opportunities to work with our tool we could reflect its power in a more convincing way. Please let us know if we can add more clarifications about our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The main reason to accept would be the release of the tool to create the flow graphs, which may aid future research. This is especially true for the most useful parts of the flow graph, which would be the less fine-grained ones, as will be discussed later.\n2. Of the three main contributions, the one that strikes as new and interesting is the observed phenomena of semantic filters by layer norm layers. It is interesting how function words contribution is decreased after applying them, highlighting the utility of these layers to increase the weight of content \"words\" (or at least hidden states close to those words.). I would have enjoyed more quantitative results regarding this.", "reasons_to_reject": "1. The method proposed seems too fine-grained. It works when one already knows what to look for. Each flow graph is focused on a single layer. The projection of specific heads (either queries, keys or values) to the subdimensions of embeddings at the LM head (ie. j:j+h/d as in line 236) doesn't seem to be that useful. The fact it needs to be projected through W_O and that no quantitative results to properly measure  any patterns or useful information from these projections, hinders the overall approach. Moreover, the projections for each head are not properly explained, one has to deduce just from line 236 that each hidden state is projected through W_O and then to the corresponding dimensions of the LM head, not just for A_j but also for keys, queries and values. While \"full\" hidden states or weights that share the full dimension (d) are well projected to specific tokens as shown by work such as Dar et al. (2022), the new contribution of this work which is looking at the head level is only demonstrated through a couple of examples and histogram figures, but I am not sure what one really learns other than that certain heads have higher norms and therefore their influence in the output of the attention block is bigger. What is the contribution of the visual flow there? When I look at Figure 1 and 5, I do not see any meaningful \"projections\" at the head level. Its utility is only portrayed when one already knows of a specific head phenomena from other work, as shown in Figure 6. And while some plots help grasp some degree of interpretability outside the head level (ie. layer norm, attention block, feed-forward, residual, ie, hidden states that share the same dimension), this was already shown by previous work. Still, a tool to properly visualise these flows, with top tokens for each hidden state is very nice, and considered a reason to accept.\n\n2. The results themselves are unsurprising. While this by itself is not a reason to reject, combined with the fact that other works have already discussed most of the points the paper sort of hinders the \"reasons to accept\". This can be seen as a reason not to accept, rather than to reject.\n\n3. Why only GPT2? How can we know this tool will be useful in future research if it was only tested on a single model? If the restriction is that only decoder systems benefit from this visualisation, why not try other decoder only models and show the flexibility of this approach? It seems that most of what is discussed in the paper was already known for GPT2, so if the point is to present a new tool that helps in explainability of transformer models, as the title would suggest, showcasing other models would be important.\n\nOverall I feel like the paper is trying too many things at once. It presents as a tool for visualising the flow of information, and gives some nice examples on where it is useful, but at the same time focuses on contributions which were already known or a bit redundant/expected, with qualitative experiments/examples. In truth, the tool itself should be the main contribution, and less words could be devoted to try to justify new contributions in a shallow way, which are not that surprising anyways, while more relevant examples on how the tool works are relegated to the Appendix. To sum up, if the paper is trying to show how information flows in the model, other papers have already done so in a more clear and concise way, however, doing so through a visual tool projecting to specific tokens to help interpretability is novel.", "questions_to_authors": "Does the code release include the full visualisation tool? Ie. a library where I can query my model (I guess restricted to GPT2) and obtain a flow graph as in Figures 1, 5 and 6, and the functionality the appendix showcases?\n\nWhat should one learn from section 3? While 3.4 tries to explain it, I am still unsure whether anything new is learned from it. Maybe that some heads carry most of the \"semantic meaning\". Still two pages seem too much.\n\nCould you perhaps justify why this paper should be presented as a \"Interpretability, Interactivity, and Analysis of Models for NLP\" conference paper and not as a demo of the tool at EMNLP?", "paper_topic_and_main_contributions": "The paper introduces a visualisation tool that creates a flow graph of different hidden states within GPT2 by projecting them to the Language Model head, a process called logit lenses. The paper goes one step further than previous work by not just projecting the hidden state at the end of each attention block but rather doing it even at the head level by looking at head-specific dimensions when projecting to the LM head, albeit this part was a bit unclear. This obtains very fine-grained interactions within each attention block represented by tokens instead of dense hidden states.\n\nBy utilising this method the authors claim three main contributions:\n(1) reflect the mechanistic analysis of Wang et al. (2022) on indirect object identification in a simple way; (2) analyze the role of layer norm layers, finding they act as semantic filters; and (3) discover neurons that are always activated, related to but distinct from rogue dimensions (Timkey and van Schijndel, 2021), which we term regularization neurons.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Check grammatical errors in the figures. Attention seems to be spelled wrong (attentnion) in all of them.\nThere needs to be more clear formulas. At the current stage, the logit lens explanation is confusing, as it is applied differently at head level and before/after W_O projection. The formulation is not clear when you have a head with dimension d/h. The W_O projection seems to imply that the same is done when projecting to D, ie. using only j to j+d/h, but it should be clarified. Moreover, are query and values as shown in the figures also projected to W_O? Overall I think all the formal definitions can be somewhat improved to help the reader."}}, {"review_id": "lqzyFxlYwK", "reviewer": "Reviewer_arzY", "summary": "", "questions": "1. Why was GPT-2 chosen as the backbone model for visualization? Does GPT-2 have unique and extensive features that make it the best choice for building a visualization tool? Although the power of the GPT series is well-known, it is hoped that the authors can systematically elaborate on the motivations for this work.\n2. How to generalize this work to other types of language models?\n3. Could the author provide more quantitative validation regarding the findings mentioned in the text?\n4. This paper uses the lens of logits, which transform intermediate embeddings into a probability distribution over the model's vocabulary using a decoding matrix. However, the employment of the top-k operation can lead to information loss and incorrect interpretation. Furthermore, tracing the disappearance and appearance of words during the model's inference stage proves challenging, as the representative words for certain intermediate embeddings appear unrelated to the words in the subsequent layer. More discussion on this issue is recommended.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, thank you for the review. We are glad that you see a potential in our work to significantly impact the explainability research.\n\n>\"Why was GPT-2 chosen as the backbone model for visualization?\" and \"How to generalize this work to other types of language models?\"\n\nGPT-2 is only used as our test-case as an example for decoder-only LMs (similar to the main works in this field of intepretability of language models, such as  Geva et al. (2021, 2022a, 2022b), Dar et al. (2022)), but the visualization is not limited to it, and in fact with the release of our code we also support GPT-J and provide guidance on how to adapt the tool to any decoder-only LM. \nWith that being said, our purpose was to present a new method and it can be further adapted to other types of Transformers. We simply worked with the most common architecture and models in works similar to ours.\n\n>\"Could the author provide more quantitative validation regarding the findings mentioned in the text?\" and \"The experimental part needs to be more sufficient\"\n\nWe can conduct more experiments but we are not sure to which of our current results you referred (especially because of the amount of space that the already conducted ones take in this paper). \nPlease note: the results from **Section 3** are a continued discussion of the works of Elhage et al. (2021),  Geva et al. (2022b) and Dar et al. (2022), and we just tried to fill the gap concerning the importance of specific components of the attention sub-blocks to better design our visualization (also note we conduct additional experiments regarding **Section 3** which are included in the **Appendix**).\nThe results from **Section 5**, although we continue to investigate some of them in the appendix, are shown briefly to illustrate the power of our tool.\nThis also explains why we chose to structure our paper in the way that we did: we first establish the information flow with quantitative results in **Section 3**. This, with knowledge from prior work, guides our flow graph construction method in **Section 4**, which facilitates the analyses in **Section 5**. \n\n>\"This paper uses the lens of logits... the employment of the top-k operation can lead to information loss and incorrect interpretation\"\n\nWe totally agree with this statement and we even mention a warning about it in the Limitations section regarding the $I_k$ metrics.\nThe top-k usage is a core part of LogitLens (usually with very small k, i.e., top-1) and we in fact engage with the issue more than in most prior papers.\nAs we share this concern, we will incorporate it into the paper.\n\n\n>\"The author needs to emphasize academic innovation... motivate people to explore more\"\n\nAs we try to present our work in a modest way, we believe our flow-graph modeling (the tool) can be a booster for many research questions/directions, just like you recognized and mentioned. Our claim is that qualitative examinations of those models using our new tool, which is highly different than previous methods, can make a huge impact (and we try to support this by presenting our findings in **Section 5**). \nGiven your feedback, we will highlight this point earlier at the beginning of the paper.\nWe believe that with more opportunities to see results of our analysis approach, the inspiration and motivation will be more clear (according to our experience of presenting the tool to colleagues). We have produced more valuable plots but unfortunately due the conference rules we can't add more plot examples at the moment.\n\n\nWe want to take this chance to add a more tangible example of how we believe this tool can help research (and why we think you might also be excited by it). For that, we will use **Figure 11** from the paper: when we investigated its prompt, we first looked at the full graph that included multiple blocks. By focusing on different components we found the specific block where the model first predicts what becomes its final prediction (in the Figure we present only this block). Focusing on this block allows us to see which heads' neurons are the ones that bring this prediction as detailed in its caption (including more discoveries), without any previous knowledge about those neurons (on-the-fly). This is just one example of a new level of interpretability that can boost any research in the field with very little technical or previous knowledge (i can imagine how many students will find this useful).\n\nThanks for all the great questions and notes. Please let us know if you have any remaining concerns that we can answer and perhaps change your evaluation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work might significantly impact the explainability research of NLP and DL communities.\n2. The theoretical analysis part is thorough. It appears that the authors have conducted sufficient research on GPT-related works.", "reasons_to_reject": "1. The experimental part needs to be more sufficient. The authors need to devise more persuasive comparative experiments to verify the superiority of the proposed method. \n2. The author needs to emphasize academic innovation, clearly explaining their work's principles. An excellent academic paper is always inspirational; it can motivate people to explore more.\n3. It is suggested that the author redesign the structure of the paper. The current structure of the article is more like a technical report than an academic paper.", "questions_to_authors": "1. Why was GPT-2 chosen as the backbone model for visualization? Does GPT-2 have unique and extensive features that make it the best choice for building a visualization tool? Although the power of the GPT series is well-known, it is hoped that the authors can systematically elaborate on the motivations for this work.\n2. How to generalize this work to other types of language models?\n3. Could the author provide more quantitative validation regarding the findings mentioned in the text?\n4. This paper uses the lens of logits, which transform intermediate embeddings into a probability distribution over the model's vocabulary using a decoding matrix. However, the employment of the top-k operation can lead to information loss and incorrect interpretation. Furthermore, tracing the disappearance and appearance of words during the model's inference stage proves challenging, as the representative words for certain intermediate embeddings appear unrelated to the words in the subsequent layer. More discussion on this issue is recommended.", "paper_topic_and_main_contributions": "This paper investigates LM attention heads, memory values, and the vectors dynamically created and recalled by the models while processing a given input. The authors further develop a technique to visualize the forward pass of GPTs as an interactive flow graph. This proposed visualization tool reveals new insights about the role of layer norms as semantic filters that influence the model output, and about neurons that are consistently activated during forward passes, acting as regularization vectors.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. It seems that there is quite a bit of space left in the paper, so there is no need to abbreviate “hidden-states” as \"HSs\"."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a tool to visualize a forward pass of GPT-based models as an interactive flow graph, and performs an analysis of the information flow inside the attention mechanism.
It is therefore part demo paper and part interpretability paper. Since it was submitted to the interpretability track, the recommendation is for that track.

There were some concerns around reproducibility, which were (mostly) taken away during discussion, as authors promised to release the code. The paper could still also be improved to provide more details on the approach.

Pros:
- The reviewers unanimously thought the tool to visualize information flow is the biggest contribution of the paper, and so the demo track might have been a better place for it. 
- They also do see value in the analysis, especially regarding layer norm.

Cons:
- The analysis is limited to one dataset (CounterFact) and one model (GPT).
- The accessibility of the tool needs improvement.
- The assumption that less activated neurons are less important might not be true.

All in all it's not easy to make a recommendation given the hybrid nature of the paper (part demo paper, part analysis paper). Given that the reviewers quite unanimously positive regarding the above pros and on soundness and excitement, the paper *could* be accepted in current form. But if it is not, I would recommend the authors to either resubmit it as a demo paper or to a workshop, or to make the paper stronger by extending the analysis.
Decision: Accept-Findings