Dissecting Recall of Factual Associations in Auto-Regressive Language Models
Authors: Mor Geva
Jasmijn Bastings
Katja Filippova
Amir Globerson
Venue: EMNLP 2023 Main (2023)
Keywords: language models
knowledge tracing
knowledge localization
interpretability
TL;DR: A comprehensive study of the internal recall process of factual associations in auto-regressive transformer-based language models.

Abstract: Transformer-based language models (LMs) are known to capture factual knowledge in their parameters. While previous work looked into where factual associations are stored, only little is known about how they are retrieved internally during inference. We investigate this question through the lens of information flow. Given a subject-relation query, we study how the model aggregates information about the subject and relation to predict the correct attribute. With interventions on attention edges, we first identify two critical points where information propagates to the prediction: one from the relation positions followed by another from the subject positions. Next, by analyzing the information at these points, we unveil a three-step internal mechanism for attribute extraction. First, the representation at the last-subject position goes through an enrichment process, driven by the early MLP sublayers, to encode many subject-related attributes. Second, information from the relation propagates to the prediction. Third, the prediction representation "queries" the enriched subject to extract the attribute. Perhaps surprisingly, this extraction is typically done via attention heads, which often encode subject-attribute mappings in their parameters. Overall, our findings introduce a comprehensive view of how factual associations are stored and extracted internally in LMs, facilitating future research on knowledge localization and editing.

Dissecting Recall of Factual Associations in Auto-Regressive Language Models
Mor Geval
Jasmijn Bastings'
Katja Filippova' Amir Globerson
Google DeepMind
TTl Aviv University Google Research
(pipek, bastings, katjaf, amirg)@google.com
Abstract
Transformer- based language models (LMs) are known to capture factual knowledge in their parameters. While previous work looked into where factual associations little is known about how they are retrieved in- ternally during inference. We investigate this question through the lens of information flow. Given a subject-relation query, we study how the model aggregates information about subject and relation to predict the correct at- tribute. With interventions on attention edges, we first identify critical points where in- formation propagates to the prediction: one from the relation positions followed by another from the subject positions. Next, by analyz- ing the information at these points, we unveil three-step internal mechanism attribute ex- traction. First, the representation the last- subject position goes through an enrichment process, driven by the early MLP sublayers, to encode many subject-related attributes. Sec- ond, information from the relation propugates the prediction. Third, the prediction repre- sentation "queries" the enriched subject to ex- tract the attribute. Perhaps surprisingly, this extraction is typically done via attention heads, which often encode sutject-atrribute mappings their parameters. Overall, our findings in- troduce a comprehensive view of how factual associations are stored and extracted internally in LMs, facilitating future research knowl- edge localization and editing.'
MLP
ATTN
Apple
stored, only
(C) Attribute
Apple
Extraction
(A) Subject enrichment
(B) Relation
propagation
Beats
Music
owned
by
Figure 1: Illustration of findings: given subject-
relation query, a subject representation is constructed via attributes' enrichment from MLP sublayers (A), while the relation propagates to the prediction (B). The attribute then extracted by the MHSA sublayers (C).
knowledge is encoded in the network (Meng et al., 2022a; Dai et al., 2022; Wallat et al., 2020), it remains unclear how this knowledge is extracted from the model parameters during inference.
In this work, we investigate this question through the lens of information flow, across layers and input positions (Elhage et al., 2021). We focus on ba- sic information extraction setting, where a subject and relation given in sentence (e.g. "Bears Music owned by"), and the next token is the cor- responding attribute (i.e. "Apple"). We restrict our analysis to cases where the model predicts the correct attribute as the next token, and set out to un- derstand how internal representations evolve across the layers to produce the output.
1 Introduction
Transformer-hased language models (LMs) cap- ture vast amounts of factual knowledge (Roberts et al., 2020; Jiang et al., 2020), which they code in their parameters and recall during inference (Petroni et al., 2019; Cohen al., 2023). While recent works focused on identifying where factual
OOur code is publicly available at https: //github.c ctm/ tree//masterr/iiseettnngg
Focusing on modern auto-regressive decoder- only LMs, such an extraction process could be
reogale-reseacch/

---

implemented in many different ways. Informally, the model needs to "merge" the subject and relation in order to be able to extract the right attribute, and this merger can be conducted at different layers and positions. Moreover, the attribute extraction itself could be performed by either or both of the multi- head self-attention (MHSA) and MLP sublayers.
(Fig. 1). First, information about the subject is enriched in the last subject token, across early lay- ers of the model. Second, the relation is passed to the last token. Third, the last token uses the re- lation to extract the corresponding attribute from the subject representation, and this is done via at- tention head parameters. Unlike prior works on factual knowledge representation, which focus mid-layer MLPs as the locus of information (e.g. Meng al. (2022b)), our work highlights the key role of lower MLP sublayers and of the MHSA parameters. More generally, we make substan- tial step towards increasing model transparency, introducing new research directions for knowledge localization and model editing.
To investigate this, we take reverse-engineering approach, inspired by common genetie analysis methods (Griffiths et al., 2005; Tymms and Kola, 2008) and the recent work by Wang et al. (2022). Namely, we artificially block, or "knock out", spe- cific parts in the computation to observe their im- portance during inference. To implement this ap- proach in LLMs, we intervene on the MHSA sub- layers by blocking the last position from attending to other positions at specific layers. We identify two consecutive critical points in the computation, where representations of the relation and then the subject are incorporated into the last position: first the relation and then the subject.
2 Background and Notation
We start by providing detailed description of the transformer inference pass, focusing on auto- regressive decoder-only LMs. For brevity, bias terms and layer normalization (Ba et al., 2016) are omitted, as they are nonessential for our analysis. A transformer-based LM first converts an input text a sequence 1, ...ty of N tokens. Each token is then embedded vector x! E Rd using an embedding matrix E R Vixd over a vocabulary V. The input embeddings are then transformed through a sequence of transformer layers, each composed of a multi-head self-attention (MHSA) sublayer followed by an MLP sublayer (Vaswani et al., 2017). Formally, the representation xf of token at layer e is obtained by:
Next, to identify where attribute extraction oc- curs, we analyze the information that propagates at these critical points and the representation con- struction process that precedes them. This is done through additional interventions to the MHSA and MLP sublayers and projections the vocabulary (Dar et al., 2022; Geva et al., 2022b; Nostalgebraist, 2020). We discover an internal mechanism for at- tribute extraction that relies on two key components. First, a subject enrichment process, through which the model constructs a representation at the last subject-position that encodes many subject-related attributes. Moreover, we find that out of the three sources that build representation (i.e., the MHSA and MLP sublayers and the input token embeddings (Mickus et al., 2022)), the early MLP sublayers are the primary souree for subject enrichment.
=x + a m
(1)
where af and mf are the outputs from the é-th MHSA and MLP sublayers (see below), respec- tively. An output probability distribution is ob- tained from the final layer representations via a prediction head 8:
The second component an attribute extraction operation carried out by the upper MHSA sublay- ers. For a successful extraction, these sublayers rely on information from both the subject representation and the last position. Moreover, extraction is per- formed by attention heads, and our analysis shows that these heads often encode subject-attribus map- pings in their parameters. We observed this extrac- tion behavior in 70% of the predictions.
p softmax (8(x4)),
(2)
that projects the representation to the vocabulary space, either through multiplying by the embedding matrix (i.e., 5(x+) = Ex+) or by using a trained linear layer (i.e., 8(x²) = Wx! + u for W E R u E RlI)).
Our analysis provides significantly improved understanding of the way factual predictions are formed. The mechanism we uncover can be intu- itively described as the following three key steps
MHSA Sublayers The MHSA sublayers com- pute global updates that aggregate information from all the representations at the previous layer. The ht MHSA sublayer is defined using four

---

parameter matrices: three projection matrices W Wk. Wy E Rdxd and an output matrix WO Rdxd, Following Elhage al. (2021); Dar et al. (2022), the columns of each projection ma- trix and the rows of the output matrix can be split into H equal parts, corresponding to the number of attention heads W K w E Rdd and WoO E R xd for ing the MHSA output as a sum of matrices, each induced by a single attention head:
Presley play?") or as a fill-in-the-blante query (e.g. "Elvis Presley played the __). While LMs often succeed at predicting the correct attribute for such queries (Roberts et al., 2020; Petroni et a 2019), it is unknown how attributes are extracted internally. For factual query q that expresses the sub- ject and relation of a triplet (s, r; a), let (41,..,,y)) be the representation of q as sequence of tokens, based on some LMM. We refer by the subject tokens to the sub-sequence of t that cor- responds to and by the subject positions to the positions of the subject tokens in L. The non-subject tokens in & express the relation
H) This allows describ-
Σ A
X
(w)) wes
(3)
Data We use queries from COUNTERFACT (4) (Meng et al., 2022a). For given model, extract random sample of queries for which the model predicts the correct attribute. In the rest of the pa- per, we refer to the token predicted by the model for a given query q as the attribute a, even though could be a sub-word and thus only the prefix of the attribute name (e.g. Wash for "Washington").
Σ
(x wi))
j=1
135 (xx- xx w
Mes
Vd/H
(5)
where X E Nxd is a matrix with all token rep- resentations at layer L, is a row-wise softmax normalization, A c NxN encodes the weights computed by the j-th attention head at layer f, and M is mask for Ad._ In auto-regressive L.ss, A is masked a lower triangular matrix, as each position can only attend preceding positions (i.e. M = oo Ve > r and zero otherwise). Impor- tantly, the cell A can viewed as a weighted edge from the r-th to the e-th hidden representations at layer f-
Models We analyze two auto-regressive decoder- only GPT LMs with different layouts: GPT-2 (Rad- ford et al., 2019) (L - 48, 1.5B parameters) and GPT-J (Wang and Komatsuzaki, 2021) (L 28, 6B parameters). Both models use vocabulary with 50K tokens. Also, GPT-J employs parallel MHSA and MLP sublayers, where the output of the L-th MLP sublayer for the i-th representation de- pends on x < rather than on af + x (see Eq. 6). We follow the procedure by Meng et al. (2022a) to create data sample for each model, resulting in 1,209 queries for GPT-2 and 1,199 for GPT-J.
MLP Sublayers Every MLP sublayer computes local update for each representation:
mf = W/ o (Wf + +x-1))
(6)
Overview: Experiments & Findings
where W E Rdxxd and W. E Raxd: are param- eter matrices with inner-dimension d,, and a is a nonlinear activation function. Recent works has shown that transformer MLP sublayers can be cast as key-value memories (Geva et al., 2021) that store factual knowledge (Dai et al., 2022; Meng et al., 2022a).
We start by introducing our attention blocking method and apply it identify critical information flow points in factual predictions ($5) one from the relation, followed by another from the subject. Then, we analyze the evolution of the subject repre- sentation in the layers preceding this critical point ($6), and find that it goes through an enrichment process driven by the MIP sublayers, to encode many subject-related attributes. Last, we investi- gate how and where the right attribute is extracted from this representation ($7), and discover that this is typically done by the upper MHSA sublayers, via attention heads that often encode subject-attribut mapping in their parameters.
3 Experimental Setup
We focus on the task of factual open-domain questions, where a model needs predict an at- tribute a of a given subject-relation pair (s,r). A triplet (s, r, a) is typically expressed in a question- answering format (e.g. "What instrument did Elvis

---

subject non-subject ... last
causal tracing (Meng et al., 2022a), which checks what hidden representations restore the original prediction when given perturbed input tokens; we test where critical information propagates rather than where it is located during inference.
GPT2-XI
% change in prediction probability
20
-40
-60
Experiment We use Attention Knockout to test whether and, if so, where information from the subject and relation positions directly propagates to the last position. Let S, R N) be the subject and non-subject positions for given input. For each layer E, we block the attention edges from the last position each of S R. and the last (N- th) position, for a window (-th layer, and measure the change in prediction probability. We set k = 9 (5) for GPT-2 (GPT-J).
10
20
30
40
50
GPT-J
-20
-60
0
10
15
20
25
layers around the
layer
Figure 2: Relative change in the prediction probability when intervening on attention edges the last position, for window sizes of layers in GPT-2 and 5 in GPT-J.
Results Fig. 2 shows the results. For both GPT- and GPT-J, blocking attention to the subject to- kens (solid green lines) in the middle-upper layers causes dramatic decrease in the prediction proba- bility of up 60%. This suggests that critical infor- mation from the subject positions moves directly to the last position at these layers. Moreover, an- other substantial decrease of 35%-45% is observed for the non-subject positions (dashed purple lines). Importantly, critical information from non-subject positions precedes the propagation of critical in- formation from the subject positions, trend we observe for different subject-relation orders (5A.1). Example interventions are provided in SH.
5 Localizing Information Flow via Attention Knockout
For successful attribute prediction, model should process the input subject and relation such that the attribute can be read from the last position. We investigate how this process done internally by "knocking out" parts of the computation and measuring the effect on the prediction. To this end, we propose fine-grate tieeggrinedd intervention on the MHSA sublayers, as they are the only module that communicates information between positions, and thus any critical information must be transferred by them. We show that factual predictions are built in stages where critical information propagates to the prediction at specific layers during inference.
Overall, this shows that there are specific dis- jointed stages in the computation with peaks of critical information propagating directly to the pre- diction from different positions. In the next section, we investigate the critical information that propa- gates from the subject positions to the prediction.
Method: Attention Knockout Intuitively, criti- cal attention edges are those that, when blocked, result in severe degradation in prediction quality. Therefore, we test whether critical information propagates between two hidden representations at a specific layer, by zeroing-out all the attention edges between them. Formally, let r, E [1, .N] such that r c: be two positions. We block x from attend- ing to x at layer L < L by updating the attention weights that layer (Eq. 5):
Intermediate Subject Representations
We saw that critical subject information is passed to the last position in the upper layers. We now analyze what information is contained in the sub- ject representation the point of transfer, and how does this information evolves across layers. To do this, we map hidden representations to vocabulary tokens via projection. Our results indicate that the subject representation contains a wealth infor- mation about the subject at the point where it is transferred to the last position.
M -0 VjEl H
(7)
Effectively, this restricts the source position from obtaining information from the target position, at that particular layer. Notably, this is different from
RRssults for varying values of k are provided in $A.4.

---

Subject
Example top-scoring tokens the subject representation
Iron Man
3, 2, Marvel, Ultron, Avenger, comics, suit, armor, Tony, Mark, Stark, 2028
GPT-2 Sukarno
Indonesia, Buddhist, Thailand, government, Museum, Palace, Bangkok, Jakarta
Roman Republic
Rome, Augustus, circa, conquered, fame, Antiqu, Greece, Athens, AD, Caesar
Ferruccio Busoni
music, wrote, piano, composition, International, plays, manuscript, violin career, National, football, NFL, Award, retired, quarterback, throws, Field
GPT-J
Joe Montana
Chromeeast
device, Audio, video, Wireless, HDMI, USB, Google, Android, technology, 2016
Table I: Example tokens by subject representations of GPT-2 (f = 40) and GPT-J (L = 22).
subject last
subject subseq.
of non-common tokens that were mentioned in the context of s, and are thus likely be its attributes. Further details on the construction of these sets are provided in $C. We define the attributes rate for a subject a set of tokens T as the portion of tokens in T that appear in Ag.
subject first
input last
GPT2-Xl
50
GPT-J
artributes rate
40
20
Experiment We measure the attributes rate in the top k 50 tokens by the subject representa- tion, that is, the representation at the last-subject position, in each layer. We focus on this position as is the only subject position that attends to all the subject positions, and thus it is likely to be the most critical (we validate this empirically in $A.3). We compare with the rate at other positions: the first subject position, the position after the subject, and the last input position (i.c., the prediction position).
layer
layer
Figure 3: Attributes rate at different positions across layers (starting from layer 1), in GPT-2 and GPT-J.
6.1 Inspection of Subject Representations
Motivating Observation To analyze what is en- coded in representation hf, cast it distri- bution of over the vocabulary (Geva al., 2022b; Nostalgebraist, 2020), using the same projection applied to final-layer representations (Eq. 2). Then, we inspect the & tokens with the highest probabil- ities in p Examining these projections, we ob- served that they are informative and often encode several subject attributes (Tab. 1 and $D). There- fore, we turn to quantitatively evaluate the extent which the representation of subject encodes tokens that are semantically related to it.
Results Fig. 3 shows the results for GPT-2 and GPT-J. In both models, the attributes rate at the last-subject position is increasing throughout the layers, and is substantially higher than at other po- sitions in the intermediate-upper layers, reaching close to 50%. This suggests that, during infer- ence, the model constructs attribute-rich subject representations at the last subject-position tion, critical information from these representations propagates to the prediction, as this range of lay- ers corresponds to the peak of critical information observed by blocking the attention edges to the prediction ($5).
addi-
Evaluation Metric: Attributes Rate Semantic relatedness hard measure based on human judgment, as ratings are typically of low agreement, especially between words of various parts of speech (Zesch and Gurevych, 2010; Feng et al., 2017). Hence, we propose an automatic approximation of the subject-attribute relatedness, which is the rate of the predicted attributes in a given set of tokens known to be highly related to the subject. For given subject s, we first create a set A, of candidate attributes, by retrieving paragraphs about s from Wikipedia using BM25 (Robertson al., 1995), tokenizing each paragraph, and removing common words and sub-words. The set A, consists
We have seen that the representation of the sub- ject encodes many terms related to it. A natural question that arises is where these terms are ex- tracted from enrich that representation. In princi- ple, there three potential sources (Mickus al., 2022), which we turn analyze in the next sec- tions: the static embeddings of the subject tokens ($6.2) and the parameters of the MHSA and MLP sublayers ($6.3).

---

Results Results with respect to layer / - in GPT-2 are shown in Fig. 4, showing that canceling the early MLP sublayers has a destructive effect on the subject representation's attributes rate, decreas- ing it by ~888 on average. In contrast, canceling the MHSA sublayers has much smaller effect of <30% decrease in the attributes rate, suggesting that the MLP sublayers play major role in creat- ing subject representations. Results with respect other layers and for GPT-J show similar trends and are provided in $C and $E, respectively. We further analyze this by inspecting the MLP updates, show- ing they promote subject-related concepts ($F).
attributes rate
40
layer 40
20
No intervention
MLP sublayers
10
MHSA sublayers
1-10
11-20
21-30
31-40
intervention layers
Figure 4: The attributes rate of the subject representa- tion with and without canceling updates from the MLP and MHSA sublayers in GPT-2.
6.2 Attribute Rate in Token Embeddings
Notably, these findings consistent with the view of MLP sublayers key-value memories (Geva et al., 2021; Dai et al., 2022) and extend recent observations (Meng et al., 2022a; Wallat et al., 2020) that factual associations are stored intermediate layers, showing that they are spread across the early MLP sublayers as well.
We test whether attributes are already encoded in the static embeddings of the subject tokens, by mea- suring the attributes rate, as in $..1. Concretely, let .1....,44 4|s be the tokens representing a subject (c.g. Piet, ro, Men, nea for "Pictro Mennca"), and denote by bedding vector, where et, is the embedding of t,. We compute the attributes rate in the top 50 tokens by each of et, and by e. We find that the highest attributes rate across the subject's token embeddings is 19.3 on average for GPT-2 and 28.6 in GPT-J, and the average rate by the mean sub- ject embedding is 4.1 in GPT-2 and 11.5 in GPT-J. These rates considerably lower than the rates by the subject representations higher layers (Fig. 3).
Σ ot, their mean em-
7 Attribute Extraction via Attention
The previous section showed that the subject repre- sentation is enriched with information throughout the early-middle layers. But recall that in our pre- diction task, only one specific attribute sought. How is this attribute extracted and at which point? We next show that (a) attribute extraction is typ- ically carried out by the MHSA sublayers ($7.1) when the last position attends to the subject, (b) the extraction is non-trivial as it reduces the attribute's rank by the subject representation considerably ($7.2) and it depends on the subject enrichment process ($7.3), and (c) the relevant subject-attribute mappings are often stored in the MHSA parameters ($7.4). This is in contrast to commonly held belief that ML.s hold such information.
This suggests that while static subject-token em- beddings encode some factual associations, other model components are needed for extraction of subject-related attributes.
6.3 Subject Representation Enrichment
We next assess how different sublayers contribute to the construction of subject representations through causal interventions.
Method: Sublayer Knockout To understand which part of the transformer layer "adds" the in- formation about attributes to the representation, we simply zero-out the two key additive elements: the MHSA and MLP sublayers. Concretely, we zero-out updates to the last subject position from each MHSA and MLP sublayer, for 10 consec- utive layers. Formally, when intervening MHSA (MLP) sublayer at layer f, we set af" (m 0) for L f, ... min(e= (f+9, L} (see Liq. 1). For each intervention, we measure the effect on the attributes rate in the subject representation at some specific layer where attribute rate is high.
7.1 Attribute Extraction the Last Position
Recall that the critical information from the subject propagates when its representation encodes many terms related to it, and after the critical information flow from the relation positions ($5). Thus, the last- position representation at this point can be viewed as relation query to the subject representation. We therefore hypothesize that the critical information that flows this point is the attribute itself.
the
Experiment: Extraction Rate To test this hy- pothesis, we inspect the MHSA updates to the last position the vocabulary, and check whether the

---

attribute extraction rate
0.8
no patching
attribute extraction
MHSA MLP
0.6
patched
subject
0.2
non-subject
0.0
10
20
30
40
50
0.0
layer
layer used for patching
Figure 5: Attribute extraction rate across layers, for the MHSA and MIP sublayers in GPT-2.
Extraction rate when patching representa- tions from early layers at different positions GPT-2.
Figure 6:
Extraction
# of extracting
rate
layers
attribute to the prediction (Fig. 5), with 68.2% of the examples exhibiting agreement events (Tab. 2). The layers at which extraction happens coincide with those where critical subject information prop- agates to the last position (Fig. 2), which further explains why this information is critical for the pre- diction.
MHSA
68.2
2.15
all but subj. last
1.03
non-subj.
42.1
1.04
last
39.4
0.97
subj. last
0.82
but
32.9
0.55
suhj. last last
32.8
0.7
non-subj.
suhj.
30.2
0.51
but subj. last
23.5
0.44
Considering the knockout results in Tab. 2, at- tribute extraction is dramatically suppressed when blocking the attention to the subject positions (30.2%) non-subject positions (31.5%). More- over, this suppression alleviated when allowing the last position attend to itself and the sub- ject representation (44.4%), overall suggesting that critical information is centered at these positions.
but first
0 0.01
MI.P
31.3
0.38
Table Per-example extraction statistics across layers, for the MHSA and MLP sublayers, and MHSA with in- terventions on positions: (non-)subj. for (non-)subjee) positions, last (first) for the last (first) input position. Extraction rate here refers the fraction of queries for which there was an extraction event in at least one layer.
Last, the extraction rate by the MLP sublayers is substantially lower (31.3%) than by the MHSA. Further analysis shows that for 17.4% of these ex- amples, extraction by the MLP was preceded by extraction by the MHSA, and for another 10.2% no extraction was made by the MHSA sublayers. This suggests that both the MHSA and MLP implement attribute extraction, but MHSA is the prominent mechanism factual queries.
top-token by each update matches the attribute pre- dicted at the final layer. Formally, let
arg max(pk)) :' arg max Eaay)
be the token predicted by the model and the top- token by the E-th MHSA update the last position (i.e., afx). We check the agreement between and i' for every & E [1, L). and refer to agreement cases (i.c. when = t*) as extraction events, since the attribute being extracted by the MHSA. Similarly, we conduct the experiment while blocking the last position from attending to different positions (using attention knockout, see $5), and also apply it to the MLP updates to the last position.
7.2 Extraction Significance
A possible scenario is that the attribute is already located the top of the projection by the subject representation, and the MHSA merely propagates it "as-is" rather than extracting it. show that this not the case, by comparing the attribute's 's rank in the subject representation and in the MHSA update. For every extraction event with subject represen- tation h we check the attribute 's rank in 8(h{), which indicates how prominent the extraction by the MHSA is (recall that, at an extraction event, the attribute's rank by the MHSA output is 1). We observe an average attribute rank of 999.5, which shows that the extraction operation promotes the specific attribute over many other candidate tokens.
Fig. 5 shows the extraction rate (namely the fraction of queries for which there was traction event) for the MHSA and MLP updates across layers in GPT-2, and Tab. 2 provides per- example extraction statistics (similar results for GPT-J are in {E). When attending to all the input positions, the upper MHSA sublayers promote the
Results

---

7.3 Importance of Subject Enrichment
We find that for 30.2% (39.3%) of the extraction events in GPT-2 (GPT-J), there is a head that encodes the subject-attribute mapping in its parameters (see examples in $G). Moreover, these specific mappings are spread over 150 atten- tion heads in GPT-2, mostly in the upper layers (24-45). Interestingly, further analysis of the fre- quent heads show they encode hundreds of such mappings, acting as "knowledge hubs" during in- ference (SG). Overall, this suggests that factual associations are encoded the MHSA parameters.
Results
An important question is whether the subject repre- sentation enrichment is required for attribute extrac- tion by the MHSA. Arguably, the attribute could have been encoded in early-layer representations extracted from non-subject representations.
Experiment We test this by "patching" early- layer representations the subject positions and measuring the effect on the extraction rate. For each layer = 0, 1, 5, 10, 20, with 0 being the em- beddings, we feed the representations at the subject positions as input to the MHSA at any succeeding layer L' > E. This simulates the MHSA opera- tion different stages of the enrichment process. Similarly, we patch the representations of the last position and of the other non-subject positions.
8 Related Work
Recently, there has been a growing interest in knowledge tracing in LMs. A prominent thread focused on locating layers (Meng et al., 2022a; Wallat et al., 2020) and neurons (Dai et al., 2022) that store factual information, which often informs editing approaches (De Cao et al., 2021; Mitchell et al., 2022; Meng et al., 2022b). Notably, Hasc et al. (2023) showed that it is possible to change an encoded fact by editing weights in other locations from where methods suggest this fact is stored, which highlights how little understand about how factual predictions are built. Our work is mo- tivated by this discrepancy and focuses on under- standing the recall process of factual associations.
Results Results for GPT-2 are shown in Fig. (and for GPT-J in $E). Patching early subject rep- resentations decreases the extraction rate by up to 50%, which stresses the importance of attributes en- richment for attribute recall. In contrast, patching of non-subject representations has weaker effect, which implies that they are "ready" very early in the computation. These observations are further supported by gradient-based feature attribution analysis ($B), which shows the influence of the early subject representations on the prediction. Notably, for all the positions, a major increase in extraction rate is obtained in the first layer (c.g. 0.05 + 0.59 for non-subject positions), suggesting that the major overhead is done by the first layer.
Our analysis also relates to studies of the pre- diction process in LMs (Voita et al., 2019; Tenney et al., 2019). Specifically, Haviv et al. (2023) used fine-grained interventions to show that early MLP sublayers are crucial for memorized predictions. Also, Hernandez al. (2023) introduced a method for editing knowledge encoded in hidden repre- sentations. More broadly, our approach relates to studies of how LMs organize information internally (Reif et al., 2019; Hewitt and Manning, 2019).
7.4 "Knowledge" Attention Heads
We further investigate how the attribute is extracted by MHSA, by inspecting the attention heads' pa- rameters in the embedding space and analyzing the mappings they encode for input subjects, using the interpretation by Dar et al. (2022).
Analysis To get the top mappings for a token t by the j-th head at layer e, we inspect the matrix vo in the embeddings space with
Mechanistic interpretability (Olah, 2022; Nanda et al., 2023) is an emerging research area. Re- cent works used projections to the vocabulary (Dar et al., 2022; Geva et al., 2022b; Ram et al., 2022) and interventions in the transformer computation (Wang et al., 2022; Haviv et al., 2023) to study the wonkins of LMs. A concurrent work by Mohebbi al. (2023) studied contextualization in LMs by zeroing-out MHSA values, method that effectively results in the same blocking effect as our knockout method. In our work, we leverage such methods to investigate factual predictions.
== W ERRIIII
(8)
by taking the & tokens with the highest values in the t-th row of G.J. Notably, this is an approxi- mation of the head's operation, which is applied to contextualized subject representations rather than to token embeddings. For every extraction event with a subject and an attribute a, we then check a appears in the top-10 10 tokens for any of s's tokens.

---

9 Conclusion
References
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. 2019. Gradient-Ban Based Attribution Methods, pages 169-191. Springer International Publishing, Cham.
We carefully analyze the inner recall process of factual associations in auto-regressive transformer- based LMs, unveiling a core attribute extraction mechanism they implement internally. Our exper- iments show that factual associations are stored already in the lower layers in the network, and extracted eminently by the MLP sublayers during inference, to form attribute-rich subject represen- tations. Upon a given subject-relation query, the correct attribute is extracted from these represen- tations prominently through the MHSA sublayers, which often encode subject-attribute mappings in their parameters. These findings open new research directions for knowledge localization and model editing.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. 2016. Layer normalization. arXiv preprint arXiv:
Jasmijn Bastings and Katja Filippova. 2020. The ele- phant in the interpretability room: Why use atten- tion explanation when we have saliency methods? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 149-155, Online. Association for Com- putational Linguistics.
Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. 2023. Crawling the internal knowledge- base of language models. In Findings of the Asso- ciation for Computational Linguistics: EACL 2023, pages 1856-1869, Dubrovník, Croatia. Association for Computational Linguistics.
Limitations
Some of our experiments rely on interpreting in- termediate layer representations and parameters through projection to the vocabulary space. While this approach has been used widely in recent works (Geva et al., 2022b,a; Dar et al., 2022; Ram et al., 2022: Nostalgebraist, 2020), only provides an approximation of the information encoded in these vectors, especially in early layers. In principle, this could have been an explanation to the increasing attributes rate in Fig. 3. However, this clear trend is unlikely be explained only by this, given the low attribute rate the embedding layer and the increase observed in the last few layers where ap- proximation is better (Geva al., 2021).
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 8493-8502, Dublin, Ireland. Association for Com- putational Linguistics.
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2022. Analyzing transformers in embedding space. arXiv preprint arXiv:2209.05335
Nicola De Cao. Wilker Aziz, and Ivan Titov, 2021. Editing factual knowledge language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6491-6506, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Another limitation is that our attention knockout intervention method does not account for "infor- mation Icakage" across positions. Namely, we block attention edges between two positions at a specific layer, it is still possible that information passed across these positions in earlier layers. For this reason, we block range of layers rather than single layer, which alleviates the possibility for such leakage. Moreover, our primary goal in this work was identify critical attention edges, which are still critical even if such leakage occurs.
Misha Denil, Alban Demiraj, and Nando De Freitas. 2014. Extraction of salient sentences from labelled documents. arXiv preprint arXiv:
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dods. Danny Hernandez, Andy Jones. Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Ka- plan, Sam McCandlish, and Chris Olah. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread. Htpp://taasformmr- htwpiissspu/2222//aaaeeooooooe
Acknowledgements
Yue Feng, Ebrahim Bagheri, Faezeh Ensan, and Jelena Jovanovic. 2017. The state of the art in semantic re- latedness: framework for comparison. The Knowl- edge Engineering Review, 32:e10.
We thank Asma Ghandeharioun for useful feedback and constructive suggestions.

---

Mor Geva, Avi Caciularu, Guy Dar, Paul Roit, Shoval Sadde, Micah Shlain, Bar Tamir, and Yoay Goldberg. 2022a. LMMdebugger: An interactive tool for inspec- tion and intervention transformer-bew baseer language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lunguage Processing: System Demonstrations, pages 12-21, Abu Dhabi, UAE Association for Computational Linguistics.
in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistic:: Human Language Tech- nologies. pages 68 681- San Diego, California. As- sociation for Computational Linguistics.
Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. 2022a. Locating and editing fac- tual associations in GPT. In Advances in Neural In- formation Processing Systems.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Gold- berg. 2022b. Transformer werd-fo layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30-45, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2022b. Mass- editing memory in a transformer.
Timothee Mickus, Denis Paperno, and Mathieu Con- stant. 2022. How to dissect a Muppet: The struc- ture of transformer embedding spaces. Transactions of the Association for Computational Linguisties, 10:981-996.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference Empirical Methods in Natural Lan- guage Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2022. Fast model editing at scale. International Conference on Learning Reprexentations.
John Griffiths, Anthony JF Griffiths, Susan Wessler, Richard Lewontin, William M Gelbart, David T Suzuki, Jeffrey H Miller, et al. 2005. An introduction to genetic analysis. Macmillan.
Hosein Mohebbi, Willem Zuidema, Grzegorz Chru- pata, and Afra Alishahi. 2023. Quantifying context mixing transformers. In Proceedings of the 17th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, pages 3378- 3400, Dubrovnik, Croatia, Association for Computa- tional Linguistics.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghan- deharioun. 2023. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. arXiv preprint arXiv:2301.0m 23.104433...
Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt. 2023. Progress mea- sures for grokking via mechanistic interpretability. arXiv preprint Sitv:2301002211
Adi Haviv, Ido Cohen, Jacob Gidron, Roei Schuster, Yoav Goldberg, and Mor Geva. 2023. Understand- ing transformer memorization recall through idioms. In Proceedings of the 17th Conference of the Euro- pean Chapter of the Association for Computational Linguistics. pages 248-264, Dubrovnik, Croatia. As- sociation for Computational Linguistics.
Nostalgebraist. 2020. interpreting GPT: the logit lens.
Chris Olah. 2022. Mechanistic interpretability. variables, and the importance of interpretable bases. Transformer Circuits Thread 27). http://www. transformer-ciruutt. pub/2022/mech- interp-essay/inder. html.
Evan Hernandez, Belinda Li, and Jacob Andreas. 2023. Measuring and manipulating knowledge rep- resentations in language models. arXiv preprint WwXiv22304.00740
Fabio Petroni, Tim Rocktäschel. Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods Natural Language Processing and the 9th International Joint Confer- ence Natural Language Processing (EMNLP. IJCNLP), pages 2463 2473, Hong Kong, China. As- sociation for Computational Linguistics.
John Hewitt and Christopher D. Manning. 2019.
structural probe for finding syntax in word repre- sentations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pages 4129-4112 4138, Minneapolis, Minnesota. Associ- ation for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. 2019. Language models are unsupervised multitask learners..
Zhengbao Jiang. Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Ori Ram, Liat Bezalel, Adi Zicher, Yonatan Be- linkov, Jonathan Berant, and Amir Globerson. 2022. What are you token about? dense retrieval distributions over the vocabulary. arXiv preprint arXiv:2212.10380.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models

---

Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of bert. Advances Neural Information Processing Systems, 32.
Kevin Wang, Alexandre Variengien. Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. In- terpretability in the wild: circuit for indirect ob- ject identification in gpt-2 small. arXiv preprint wrXiv:2211.00553
Kayo Yin and Graham Neubig. 2022. Interpreting lan- guage models with contrastive explanations. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 184- 198, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of language model?? In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 5418 5426. Online. Association for Computational Linguistics.
Torsten Zesch and Iryna Gurevych. 2010. Wisdom of crowds versus wisdom of linguists-measuring the semantic relatedness of words. Natural Language Engineering, 16(1):55-59
Stephen E Robertson, Steve Walker, Susan Jones, Micheline Lancock-Beeullere Mike Gatford, al. 1995. Okapi at trec-3. Nist Special Publication Sp. 109:109.
Gabriele Sarti, Nils Feldhus, Ludwig Sickert, and Os- kar van der Wal. 2023. Inseq: An interpretabil- ity toolkit for sequence generation models. arXiv preprint 2xxi:2302.13942
Tan Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy. Association for Computational Linguistics.
Martin Tymms and Ismail Kola. 2008, Gene knockout protocols, volume 158. Springer Science & Busi- ness Media.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Iilia Polosukhin. 2017. Attention all you need. Advances in neural information process- ing systems, 30.
Elena Voita, Rico Semnrich, and Ivan Titov. 2019. The bottom-up evolution of representations in the trams- former: A study with machine translation and lan- guage modeling objectives. 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1396, Hong Kong, China. Association for Computational Linguistics.
Proceedings of the
Jonas Wallat, Jaspreet Singh, and Avishek Anand. 2020. BERTnesia: Investigating the capture and for- getting of knowledge in BERT. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 174- 183, Online. Association for Computational Linguis- tics.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: A Billion Parameter Autoregressive Lan- guage Model. https: //gi httpg/ghb...co/iiiggllll.o/ com//ingofllllz/ Resh-ttransformer-

---

subject
non-subject ... last
subject non-subject ... last
20
GPT2-xl
20
GPT2-Xl
change prediction probability
% change in prediction probability
-20
-60
-60
10
30
40
50
-20
-25
-40
-50
60
GPT-J
GPT-J
-75
-80
20
25
5
10
15
20
layer
layer
Figure 7: Relative change in the prediction probability when intervening on attention edges to the last position, for 9 layers in GPT-2 and 5 in GPT-J, for the subset of examples where the subject appears at the first position in the input.
Figure 8: Relative change in the prediction probability when intervening on attention edges the last position, for 9 layers in GPT-2 and 5 in GPT-J, for the subset of examples where the subject appears after the first position the input.
A Additional Information Flow Analysis
A.2 First-position Bias
We observe that blocking the last position from attending to the first position, regardless of which token corresponds to it, has substantial effect on the prediction probability (see examples in SH). We quantify this observation and show that it does not change our main findings in $5. To this end, conduct the same experiment in $5, but without blocking the attention edges the first po- sition. Results are provided in Fig. 9, showing the same trends as observed when blocking the atten- tion to the first position as well; both GPT-2 and GPT-J, there are clear peaks of critical information from subject and non-subject positions propagating to the last position at different layers, which when blocked reduce the prediction probability drasti- cally.
A.1 Subject-Relation Order
We break down the results in $5 by the subject- relation order the input query, to evaluate whether we observe the same trends. Since the relation is expressed by all the non-subject tokens, we split the data into two subsets based on the the subject position: (a) examples where the subject appears in the first position (i.e. the subject ap- pears before the relation), and (b) examples where it appears later positions (i.e. the subject appears after the relation). We conduct the same attention blocking experiment as in $5, and show the results for the two subsets in Fig. 7 (a) and Fig. 8 (b).
In both figures, subject information passes the last position the same range of layers, showing that this observation hold in both cases. However, when the relation appears before the subject, block- ing the attention to its positions has a more promi- nent impact on the prediction probability. Also, its effect is more spread-ous across all the layers. We suggest that this different behavior is result of positional bias encoded in the first position in GPT-like models: regardless of which token ap- pears in the first position, blocking attention this position typically results in a substantial decrease in the prediction probability. The examples in SH demonstrate this. We further verify this SA.2.
Nonetheless, the decrease in probability at these peaks smaller in magnitude when the first po- sition is not blocked compared to when it is. For example, blocking the subject positions leads to a decrease of up to 40% when the last position can at- tend to the first position, compared to 60% when it cannot (Fig. 2). Likewise, blocking the non-subject positions (which correspond to the relation) leads to greater impact across the carly-intermediate lay- ers when the first position blocked compared to when it not. For instance, intervening on layers 5-20 in GPT-J constantly decreases the out-

---

subject non-subject ... last
relative change (%) prediction probability
300
change in prediction probabiltty
200
-20
100
GPT2-xl
0
10
20
30
40
50
100
last
first
before last
GPT-J
non-blocked subject position
Figure 10: Relative change in the prediction probability when intervening on attention edges to the last position from different subject positions GPT-2.
-20
5
10 15
20
25
layer
k-1
k=5
Figure 9: Relative change in the prediction probability when intervening on attention edges to the last position, for 9 layers in GPT-2 and 5 in GPT-J. Here, do not block attention edges to the first position in the input.
% change predi predictiin probabl bt
-10
20
10
k-
k=
1.
20
-20
put probability by ~20% when the first position blocked (Fig. 2) compared to <5% when is not.
60
20
30
10
20
50
A.3 Information Flow from Subject Positions
-20
-20
We conjecture that, due to auto-regressivity, sub- ject representations with critical information for the prediction are formed at the last-subject posi- tion. To verify that, refine interventions in $5, and block the attention edges the last posi- tion from all the subject positions except one. We then measure the effect of these interventions on the prediction probability, which indicates from which position critical information propagates to the prediction.
-60
-60
-80
layar
Figure Il: Relative change in the prediction probabil- ity when intervening on attention edges from different positions to the last position, for windows i layers GPT-2. The plots show the analysis for varying values of k.
Fig. 10 depicts the results for GPT-2, showing that indeed the prediction is typically damaged is
of critical information flow to the last position one from the relation positions in the early layers, followed by another from the subject positions in the upper layers. An exception, however, can be observed when knocking out edges from re- lation positions using just single-layer window in GPT-2; in this case, no significant change in the prediction probability is apparent. This might imply that critical information from the relation positions is processed in multiple layers. Morcover, the decrease in the prediction probability becomes more prominent when for larger values of k. This expected, knocking out more attention edges in the computation prevents the model from con- textualizing the input properly.
by 50% 100% when the last blocked (i.e. "first" and "before-last"), and usually remains intact when this position is not blocked (i.e. "last").
A.4 Window Size
We examine the effect of the window size hyperpa- rameter on our information flow analysis, con- ducted in $5 for the last position in GPT-2 and GPT-J. Results of this analysis with varying win- dow sizes of k 1, 9, 13, 17, 21 are provided in Fig. 11 for GPT-2 and Fig. for GPT-J. Over- all, the same observations are consistent across different window sizes, with two prominent sites

---

tion of this method?, and apply it not just to the in- put (sub)word embeddings (E 0). but also to each intermediate transformer layer (e , N).
20
Fig. 13 shows the per-layer gradient-times-input analysis for the input "Beats Music is owned by" for GPT-2 and GPT-J. This analysis supports our earlier findings ($6, Fig. 6) that shows the "readi- ness" of subject vs. non-subject positions: We observe both for both models that the subject po- sitions remain relevant until deep into the com- putation, while the subject is being enriched with associations. Moreover, the input for 'owned' is relevant for the prediction in the first few layers, after which the plot suggests it is incorporated into the final position. That final position becomes more relevant for the prediction the deeper we get into the network, as scemingly incorporates informa- tion from other positions, before it is used as the point to make the final prediction for target 'Apple' at layer 48 (28). At that point, the final position has virtually all relevance, in line with what we would expect for an auto-regressive model.
N6 change prediction probability
-30
k-9
k-333
21
-25
10
15
25
10
15
20
25
75
10 13
layer
Figure 12: Relative change in the prediction probabil- ity when intervening attention edges from different positions to the last position, for windows of k layers in GPT-J. The plots show the analysis for varying values of k.
In Fig. 14 we go beyond single example, and show per-layer gradient-timee-inge aggregated over the dataset (a subset of COUNTERFACT, ef, $3) for each model. We can see that the observa- tions we made for the single example in Fig. 13 also hold in general: the subject tokens remain rel- evant until about 3 into the network depth, and the relation (indicated by "further tokens") is relevant in the first few layers, after which it becomes less relevant to the prediction.
B Gradient-based Analysis
Gradient-based feature attribution methods, also known saliency methods, a way inspect what happens inside neural models for predictions of specific examples. Typically, they result in a heatmap over the input (sub)tokens, i.e. high- lighted words. We distinguish between sensitivity and saliency (Ancona et al., 2019; Bastings and Filippova, 2020): methods such as Gradient-L2 (Li et al., 2016) show to which inputs the model is sensitive, i.e., where small change in the input would make large change in the output, but method such as Gradient-times-lne 2014) reflects salience: it shows (approximately) how much each input contributes to this particular logit value. The latter is computed as:
C Attributes Rate Evaluation
C.1 Evaluation Details
(Denil et al.,
We provide details on the construction process candidate sets for attributes rate evaluation ($6. I). Given a subject 8, first we use the BM25 algorithm (Robertson et al., 1995) to retrieve 100 paragraphs from the English Wikipedia' with being the query. From the resulting set, we keep only paragraphs for
(x ox
(9)
Concurrently. the use of per-layer pradient-times-nnuu also called ceatientttime--attvvae was also proposed by Sarti et al. (2023) for computing contrastive (Yin and Neubig. 2022) per-layer explanations.
where xf is the input at position i in layer & (with
f = 0 being the embeddings which are used for siffient-timms-Ippuu) and (-) the function that takes the input sequence layer l and returns the logit for the target/predicted token e. To obtain scalar importance score for each token, we take the Le norm each vector and normalize the scores to sum to one. For our analysis we use a generaliza-
"We add an identity layer the output of each trans- former layer block, and capture the output and gradient there, after the attention and MLP sublayers have been added to the residual. This is important especially for GPT-J, where the attention MLP sublayers are computed parallel, so cap- turing the output and gradient only at the MLP would result incorrect attributions.
We use the dump of October 13, 2021.

---

Be
First input token
First subject token
0.8
0.66
Last subject taken
0.6
All subject tokens
0.4
First subsequent token
0.4
owned
0.2
Further tokens
-02
by
Last input
0.00
-0.0
10
13
20
23
30
33
-1
15
20
25
35
20
GPT-2
GPT-2
1.0
First input taken
1.0
0.8
First subject token
0.8
Music
Last subject token
0.6
subject
15
First subsequent loken
Further takens
by
Last input token
10
20
(b) GPT-J
(b) GPT-J
trndient-times-Aettee analysis for the example "Beats Music is owned by Apple".
Figure 13:
Figure 14: Gradient Gradient-times- Acctiiiiitiinnnnnn averaged over all examples.
GPT2-xI,
which the subject appears as-is in their content or
(r3 Interventoo layer
in the title of the page/section they in. This is avoid noisy paragraph that could obtained from partial overlap with the subject (e.g. retrieving a paragraph about "2004 Summer Olympics" for the subject "2020 Summer Olympics"). This process results in 58.1 and 54.3 paragraphs per subject on average for the data subsets of GPT-2 and GPT-J, respectively.
30
16
20
25
28
34
subject represseattnonnonnnnnn layer
Next, for each model, tokenize the sets of paragraphs, remove duplicate tokens, and tokens with less than characters (excluding spaces). The later is done to avoid tokens representing frequent short sub-words like S and *s. For stopwords re- moval, we use the list from the NLTK package."
Figure 15: The attrihutes rate of the subject representa- tion at different layers, when intervening on (canceling) different 10 consecutive MLP sublayers in GPT-2.
observed when canceling MHSA updates.
This yields the final sets A,, of 115..4 and 1073. candidate tokens average for GPT-2 and GPT-J, respectively.
D Projection of Subject Representations
We provide additional examples for top-scoring tokens in the projection of subject representations across layers, in GPT-2 and GPT-J. Tab. (Tab. 4) shows the tokens for the subject "Mark Messier" ("iPod Classic") across layers in GPT-2 and GPT- excluding stopwords and sub-word tokens.
C.2 Additional Sublayer Knockout Results
We extend our analysis in $6.3, where we analyzed the contribution of the MLP and MHSA sublayers to the subject enrichment process. Specifically, we now measure the effect of knocking out these sublayers on the attribute rate at any successive layer, rather than single upper layer. Results for GPT-2 are presented in Fig. 15 (MLP sublayers knockouts) and 16 (MHSA sublayers knockouts), showing similar trends where canceling updates from the MLP sublayers decreases the attributes rate dramatically, while a more benign effect is
Additional Results for GPT-J
We supply here additional results for GPT-J. Fig. 17 shows the effect of canceling updates from the MHSA and MLP sublayers on the attributes rate of the subject representation layer 22 ($6.3). Fig. 18 and Tab. provide the extraction rate by the MHSA and MLP across layers and per-example extraction statistics, respectively (87). Last, Fig. 19
'https: //wwww nltk. org/

---

8 Top-scoring tokens by the subject representation
25 Sr, era, Era, MP, JR, senior, Clarence, final, stars, Junior, High, Architects
GPT-2 35 Hockey, hockey, Jr, NHL, Rangers, Canucks, Sr, Islanders, Leafs, Montreal, Oilers
40 NHL, Hockey, Jr, Canucks, retired, hockey, Toronto, Sr, Islanders, Leafs, Jersey, Oilers
13
Archives, ice, ring, International, Jersey, age, Institute, National, aged, career
GPT-J 17 hockey, NHL, Jr, Hockey, ice, Canadian, Canada, Archives, ice, Toronto, ring, National
22 NHL, National, hockey, Jr, Foundation, Hockey, Archives, Award, ice, career, http
Table 3: Top-scoring tokens by the subject representations of "Mark Messier" across intermediatc-uppeer layers (t)
GPT-2 and GPT-J.
Top-scoring tokens by the subject representation
25 Edition, Ribbon, Series, Mini, version, Card, CR, XL, MX, XT, Cube, RX, Glow, speakers
GPT-2
35 iPod, Bluetooth, Apple, Mini, iPhone, i0s, speaker, Edition, Android, Controller
40 iPod, headphone, Bluetooth, speakers, speaker, Apple, iPhone, ios, audio, headphones
13 Series, device, Apple, iPod, model, Music, iPhone, devices, madels, style, music
GPT-J 17 Series, model, device, style, Music, series, Archives, music, design, interface, models
22 iPod, Music, music, song, Series, iPhone, Apple, songs, Music, review, series, Audio
Table 4: Top-scoring tokens by the subject representations of "iPod Classic" across intermediate-upper layers (2) in GPT-2 and GPT-J.
GPT2-xl. MHSA
50
at layer 22
40
40
10
attributes
EE3 inte - 26
13
30
No intervention
30
MLP sublayers MH5A sublayers
19
22
25
28
1-10
6-15
11-20
16-22
21-22
31
Intervention layers
34
Figure 17: The attributes rate of the subject representa- tion with and without canceling updates from the MLP and MHSA sublayers in GPT-J.
subject representation layer
Figure 16: The attributes rate of the subject representa- tion different layers, when intervening on (canceling) different 10 consecutive MHSA sublayers in GPT-2.
Eq. 6 to linear combination of parameter vectors of the second MLP matrix:
shows the effect of replacing intermediate represen- tations with early layer representations at different positions the attribute extraction rate ($7).
dianer
(1.5) o w
(af (af+ w (25)
m
J=1
Overall, these results are consistent with those for GPT-2 described throughout the paper.
where digner is the MLP inner dimension, and w (e.g) (EJ) are the j-th row and column Wf, W respectively. We then take the 100 vec- tors in Wt
F Analysis of MLP Outputs
P. with the highest contribution to this sum, for L = 1..... 20 in GPT-2, and inspect the top-scoring tokens in their projections to E.
Following the observation that the early MLP sub- layers crucial for attribute enrichment of subject representations, further analyze their updates these representations. To this end, we decom- pose these updates into sub-updates that, accord- ing to Geva al. (2022b), often encode human- interpretable concepts. Concretely, We decompose
From manual inspection, we indeed were able identify cases where concepts related to the input subject are promoted by the dominant sub-updates. Examples are provided in Tab. 6. We note that quantifying this process non-trivial (either with

---

altribute extraction rate
attribute extraction rate
0.8
no pa aathing
MHSA
MLP
0.6
x
patched positions
subject
0.1
non subjett
0.0
5
10
15
20
0.0
layer
layer used patching
Figure 18: Attribute extraction rate across layers, for the MHSA and MLP sublayers GPT-J.
Figure 19: Extraction rate when patching representa- tions from early layers different positions in GPT-J.
Extraction
# of extracting layers
rate
intervening
MHSA
76.7
1,9
0.18
last
46.7
0.8
0.16
non-subj.
45.6
0.8
ng
all non-subj. but last
44.4
0.9
von
subj. last
41.1
0.7
ises
0.12
subj. last last
40.3
0.7
0.10
all but subj. last last
34.8
0.5
domain
0.08
all but subj. last
31.3
0.5
work
0.06
suhj.
27.8
0.3
but
24.8
0.3
the
all but first
0
10
15
25
30
35
40
pleconamies)
MLP
41
0.6
Figure 20: Example intervention information flow to the last position the input Ludwig von Mises' 's domain of work is the, using window size 10, in GPT-2.
Table 5: Per-example extraction statisties across layers, for the MHSA and MLP sublayers, and MHSA with in- terventions on positions: (non-)subj. for (non-)sur moon-)subject positions, last (first) for the last (first) input position.
ciations that cover various relations. Tab. 8 shows example mappings by these heads.
human annotations or automatic methods), and so we leave this for future work.
H Example Interventions on Information Flow
Subject-Attribute Mappings in Attention Heads
Example interventions in GPT-2 shown in Fig. 20, 21, 22, 23, 24, and in GPT-J in Fig. 25, 26, 27, 28, 29.
$7, we showed that for many extraction events, it is possible to identify specific heads that encode mappings between the input subject and predicted attribute in their parameters. We provide examples for such mappings in Tab. 7.
As a further analysis, we inspected the heads that repeatedly extract the attribute for different in- put queries in GPT-2. Specifically, for the matrix who of the j-th head at the L-th layer, we analyze its projection to the vocabulary G (Eq. 8), but instead of looking at the top mappings for spe- cific subject-token (i.e. a row in GfJ), we look at the top mappings across all rows, done by (Dar al., 2022). In our analysis, heads that extract the attribute for least 10% of the queries heads in total). We observed that these heads act as "knowledge hubs", with their top mappings encoding hundreds of factual asso-
focused the

---

Subject
Description
Layer, Dim.
Top tokens in the projection
United Launch Alliance
1, 5394 launch service provider
American spacecraft
jet, flights, aircraft, Cargo, passenger, plane, rockets, airspace, carrier, missiles, aerospace
Yakuza 2
Action-adventure video 2, 4817 game
Arcade, pixels, arcade, Wii, Fighters, pixels, Minecraft, Sega, Sonic, GPUs, Hardware, downloadable, multiplayer, Developers, livestream
Leonard Ber-
American composer, pi- anist, music educator, and author
4, 248
violin, rehearsal, opera, pian, composer, Harmony, ensemble, Melody, piano, musicians, poets, Orchestra, Symphony
stein
Brett Hull
Canadian- Amerrcan
4, 5536
discipl, athlet, Athletics, League, Hockey, ESPN, former, Sports, NHL, athleticism, hockey, Champions
former ice hockey player
Bhaktisiddhanta Saraswati
Spiritual master, instruc- tor, and revivalist early 20th century India
5, 3591
Pradesh, Punjab, Bihar, Gandhi, Laksh, Hindu, Hindi, Indian, guru, India, Tamil, Guru, Krishna, Bengal
Mark Walters
English former profes- sional footballer
10, 1078
scorer, offence, scoring, defences, backfield, midfielder, midfield, striker, fielder, playoffs, rebounds, rushing. touchdowns
Acura ILX
Compact car
19, 179
Highway, bike, truck, automobile, Bicycle, motorists, cycle, Motor, freeway, vehicle, commute, Route, cars, motorcycle, Tire, streetcar, traffic
Beats Music
Online music streaming service owned by Apple
20, 5488
Technologies, Technology, Apple, iPod, 10s, cloud, Appl, engineering, soatw platform, proprietary
Table 6: Top-scoring tokens in the projection of dominant MLP sub-updates to the subject representation in GPT-2.
Matrix
Subject
Top mappings
W oV
Barry Zite
Baseball, MLB, infield, Bhehsaaa, RPGs, pitching, outfield, Iw, pitcher
W (33,)3)
Classic
Macintosh, iPod, Mac, Perse, Apple, MacBook, Beck, Philipp
Intervering window: base probability 0.4321
(4) ov
Philippus van Limborch
Dutch, Netherlands,
van, Amsterdam, Holland,
cm
Vanessa
Franc
0.4
Table 7: Example mappings between subject-tokens (underlined) and attributes (bold) encoded in attention parameters (tokens the same word are not shown).
which
named
for
0.2
region
10
15
Intervening in probabiiity
The head
Figure 22: Example intervention on information flow to the last position of the input Cabernet Franc, which is named for the region of, using window size 10, in GPT-2.
of
WIEE
is
0.4
0
10
40
DiStamford
Figure 21: Example intervention on information flow to the last position of the input The headquarter of WWE is in, using a window size 10, in GPT-2.

---

Matrix
Top mappings
(Finnish, Finland)
intervening base
(Saskatchewan, Canadian)
The
(Finland, Finnish) (Saskatchewan, Alberta)
currendy
0.6
the
(Helsinki, Finnish)
United
0.4
(Illinois, Chicago)
States
(08.25)) OV
(Chicago, Illinois)
0.2
(Blackhawks, Chicago)
the
(Australia, Australians) (NSM, Sydney)
15
20
25
30
35
40
pidolan)
(Minneapolis, Minnesota)
Figure 24: Example intervention on information flow the last position of the input The currency the United States, the, using window size 10, in GPT-
(Auckland, NZ)
(Texas, Texans)
(Japanese, Tokyo)
(Koreans, Seoul)
(Korean, Seoul)
(Haiti, Haitian)
(Korea, Seoul)
Intenvering in grobabiliyy
(Mexican, Hispanic)
(Spanish, Spanish)
Com
0.8
W
OV
(Japanese, Japan)
2b
0.6
(Hawai, Honolulu)
ank
(Dublin, Irish)
(Norwegian, an, 0slo)
whose
headquarters
(Israelis, Jewish)
(Vietnamese, Thai)
in 10 15
(Israel, Hebrew)
20
pihrankfart)
(oglu, Turkish)
Figure 25: Example intervention on information flow the last position of the input Commer headquarters are in, using window size 5, in GPT-
(Tsuk, Japanese)
Commerzbank,, zbank, whose
(Sven, Swedish)
(Tsuk, Tokyo)
(Yuan, Chinese)
(stadt, Germany)
(von, German)
(31.0) ov
(Hiro, Japanese)
(Sven, Norwegian)
Intervening the probability:
(stadt, Berlin)
(Yuan, Beijing)
0.4
(Sven, Danish)
Gri
(oglu, Erdogan)
eg
0.3
(Fei, Chinese)
(Samurai, Japanese)
playing the
0.1
p(piano]
Table 8: Example top mappings encoded in the parame- ters attention heads that extract attribute many input queries.
Figure 26: Example intervention information flow to the last position of the input Edvard Grieg, playing the, using a window size 5, in GPT-J.
Intervering
window: probabiliy;
iat
0.5
Intervening probability 0.6688
0.4
Queen
ze located
Land
belongs
the
0.2
the
continett
courtry
of*
0.1
df
1.0 15
20
25
p/Antardical
Figure 27: Example intervention on information flow
Figure 23: Example intervention information flow to the last position of the input Siemiatycze is located in the country of, using window size 10, in GPT-2.
last position the input Queen Elizabeth Land belongs to the continent of, using window size 5, in GPT-J.

---

Intervening on flo to: by probaeility:
Stat
istical
Package
for
0.4
Social
0.3
Sciences
0.2
was
by
P(BM)
Figure 28: Example intervention on information flow to the last position of the input Statistical Package for the Social Sciences was created by, using window size 5, in GPT-J.
Intervering base probaibiitty 0,2068
The
mother
0.4
tongue
0.3
ro
0.2
Men
nea
0.1
potalian)
Figure 29: Example intervention information flow to the last position of the input The mother tongue of Pietro Mennea is, using a window size 5, GPT-J.


Reviews:
[{"review_id": "Qu2g97TNk9", "reviewer": "Reviewer_Bdt7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review, and are encouraged that they find our findings interesting, our experiments to be well designed, and the results to be well presented!\n\nWe address the points raised in the \"Reasons to Reject\" section: \n\n\n>**\"... it is often unclear whether conclusions drawn from simple experiments can extend to general, complicated cases. It is difficult to tell whether other NLP tasks follow the same pattern\"**\n\nFirst, it is important to emphasize that the goal of our work is to obtain a deep and mechanistic understanding of the recall process of facts in LMs. We believe that, **given the high complexity of modern LMs, the path towards this goal must start from basic and simple settings**. Notably, many insightful recent works have been following this intuition as well, and analyzed tasks of equal complexity (see example references [1], [2], [3], [4] below). \n\nNonetheless, we agree that moving to more complex tasks is very interesting and important. However, such complex inputs are likely to involve *additional* mechanisms, such as multi-hop reasoning, extraction of intermediate subject-object-relation tokens, and coreference resolution. Investigating each of these internal mechanisms can be done in isolation and then together. Therefore, we believe that **our work sets the ground for such future analyses**, since it reveals the basic mechanisms of knowledge recall (that are expected to occur in more complex inputs) and contributes practical methods that can be applied as-is or easily adjusted for such inputs. For instance, knocking out multiple positions at once could be used to study the hierarchy through which LMs process predictions, and the attribute-rate approximation can be applied to any hidden representation during inference.\n\n&ensp;\n\n>**\"... it is unclear whether the localization depends on training tasks, or model size, among other things\"**\n\nThank you for this comment. While we agree that it is not obvious that our findings will generalize across all these factors, we believe that our choice of models and the correspondence of our findings with those of other works that analyzed other models, provide a good basis to expect that our analysis, tools, and observations are general and robust. We explain this in detail next.\n\nFirst, the models we analyzed have **the same basic architecture that underlies most LMs today, yet, they are different in many factors including size, layout (depth vs. width), transformer block configuration (sequential vs. parallel MLP layers), and training data.**  Therefore, these models represent a large family of modern auto-regressive LMs. Specifically, LLaMA, Alpaca, Vicuna, GPT-NeoX, Pythia, OPT and others, all rely on the same variations analyzed in our work. The fact that we observed similar trends in the models we analyzed is not trivial and makes a good basis to believe that these results will generalize to other auto-regressive decoder-only LMs. Of course, there might be some variations in the specific range of layers, but this is not the purpose of this analysis; rather it is the stages of the input processing and the role of the different modules (MHSA/MLP) in that process.\n\nIn addition, our analysis methods leverage techniques, such as projection to the vocabulary and interventions, that were successfully applied to a wide range of models, including models such as BERT and T5 [5][6][7] that were trained with different objectives of masked language modeling and contrastive learning for dense passage retrieval. Therefore, **one could build up on our work and easily apply our methods to study factual recall in other models**. We agree that this is a very interesting question to explore in future work.\n\nLast, many knowledge localization and editing works have used the same specific models we analyzed to develop editing methods, such as ROME [2], MEMIT [8], and MEND [9]. Hence, our findings provide further insights to understand their success and failure cases, potentially leading to better methods.\n\n&ensp;\n___\n\n[1] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small. Wang et al., ICLR, 2023.\n\n[2] Locating and Editing Factual Associations in GPT. Meng et al., NeurIPS 2022.\n\n[3] In-context Learning and Induction Heads. Olsson et al., 2022.\n\n[4] Understanding Arithmetic Reasoning in Language Models using Causal Mediation Analysis. Stolfo et al., 2023.\n\n[5] Understanding Transformer Memorization Recall Through Idioms. Haviv et al., EACL, 2023.\n\n[6] What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary. Ram et al., ACL, 2023.\n \n[7] Neural Knowledge Bank for Pretrained Transformers. Dai at al., 2022.\n\n[8] Mass Editing Memory in a Transformer. Meng et al., ICLR, 2023.\n\n[9] Fast Model Editing at Scale. Mitchell et al., ICLR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The study of internal representations of Transformer-based models is still an evolving domain, which needs more contributions. Particularly, little is known about how factual predictions are built.\n2. This paper proposes an automatic approximation of the subject-attribute relatedness, namely the attribute rate. It is a reasonable and novel idea.\n3. The experiments are well designed and the results are well presented. There are some interesting findings.", "reasons_to_reject": "1. Although we should start with simple experiments in order to study a complex system, it is often unclear whether conclusions drawn from simple experiments can extend to general, complicated cases. It is difficult to tell whether other NLP tasks follow the same pattern. \nThe authors suggest that \"these findings open new research directions for knowledge localization and model editing\", but it is unclear whether the localization depends on training tasks, or model size, among other things.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes to use subject-relation queries to investigate how Transformer-based auto-regressive language models (GPT-2 and GPT-J) aggregate information about the subject and relation to predict the correct attribute. \n\nThe attention \"knockout\" method is applied to identify critical information flow points in factual predictions. The authors argue that critical information from the subject positions moves directly to the last position at middle-upper layers.\n\n\"Attributes Rate\" is used to investigate the position of subject representation. The authors find that the model constructs attribute-rich subject representations at the last subject-position. Attribute rates of token embeddings (from the embedding layer?) are also studied, and it reveals that while static subject-token embeddings encode some factual associations, other model components are needed for extraction of subject-related attributes. It is also found that canceling the early MLP sublayers has a destructive effect on the subject representation’s attributes rate, while canceling early MHSA sublayers does not have such a strong effect.\n\n\"Extraction Rate\" is used to investigate how and where attributes are extracted. The authors suggest that both the MHSA and MLP implement attribute extraction, but MHSA is the prominent mechanism for factual queries. It is also indicated that subject enrichment, through which the model constructs a representation at the last subject-position that encodes many subject-related attributes, has a big impact. Overall, factual associations are encoded in the MHSA parameters, acting as “knowledge hubs”.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L327 Attributes Rate --> Attribute Rate"}, "extra_text": {}}, {"review_id": "7wNpNpGd6M", "reviewer": "Reviewer_6DKT", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for putting an effort into reviewing our work, despite that it is not their area of expertise, and for appreciating the potential value in works of this kind. \n\nWe address the reviewer’s point that **\"This is a fairly technical paper, presumably a stepping stone on an incremental path\"**:\n\nWe share the reviewer’s view that understanding how Transformers (and neural networks in general) work will not happen overnight. Nonetheless, as research on interpretability is growing fast, it provides fundamental insights that allow developing various methods to increase transparency and control over LMs, and set the groundwork for devising better architectures.\n\nWe believe that our work is indeed a stepping stone towards this larger goal, since it analyzes one of the most fundamental questions about LMs – how they represent and recall knowledge from their parameters. Answering this question could have far-reaching consequences on many central problems with LMs today, such as hallucinations, memorization, and biases. Our work makes substantial contributions in this context; it reveals the basic stages of knowledge recall and identifies on a more fine-grained level where factual associations are stored and they are extracted during inference, using (novel) attention knockout and vocabulary projection techniques.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "With the rise of the desire for explainability of LMs works which identify (or go some way towards identifying) important locations are of great potential value. \n\nThe paper uses figures to advantage, illustrating complex concepts and results.\n", "reasons_to_reject": "This is a fairly technical paper, presumably a stepping stone on an incremental path.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper explores the how factual information is retrieved from transformer-based LMs. The authors perform a number of experiments to reveal some unexpected results regarding the location of factual associations. The work suggests further research directions in the form of knowledge localization and model editing.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PD7VubKiWD", "reviewer": "Reviewer_cRFb", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for carefully reading our work and for the positive review!\n\nRegarding our focus on subject-relation queries (**\"The study is only on subject-relation query\"**) – this is an important point that we address below, thanks for raising this.\n\nFirst, it is important to emphasize that the goal of our work is to obtain a deep and mechanistic understanding of the recall process of facts in LMs. We believe that, **given the high complexity of modern LMs, the path towards this goal must start from basic and simple settings**. Notably, many insightful recent works have been following this intuition as well, and analyzed tasks of equal complexity (see example references [1], [2], [3], [4] below). \n\nNonetheless, we agree that moving to more complex tasks is very interesting and important. However, such complex inputs are likely to involve *additional* mechanisms, such as multi-hop reasoning, extraction of intermediate subject-object-relation tokens, and coreference resolution. Investigating each of these internal mechanisms can be done in isolation and then together. Therefore, we believe that **our work sets the ground for such future analyses**, since it reveals the basic mechanisms of knowledge recall (that are expected to occur in more complex inputs) and contributes practical methods that can be applied as-is or easily adjusted for such inputs. For instance, knocking out multiple positions at once could be used to study the hierarchy through which LMs process predictions, and the attribute-rate approximation can be applied to any hidden representation during inference.\n\n&emsp;\n\n___\n\n\n[1] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small. Wang et al., ICLR, 2023.\n\n[2] Locating and Editing Factual Associations in GPT. Meng et al., NeurIPS 2022.\n\n[3] In-context Learning and Induction Heads. Olsson et al., 2022.\n\n[4] Understanding Arithmetic Reasoning in Language Models using Causal Mediation Analysis. Stolfo et al., 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall the paper is well written and easy to understand. It presents interesting and useful findings for better understanding how a transformer-based LM predicts for subject-relation query. For example, they found that early layers are important in propagating information for predicting the attribute. Another example is the attribute information is aggregated gradually through layers. These findings could not only help understand the models but also inspire new methods that better aggregate information for subject-relation queries.  The experiments are comprehensive and provide answers for questions at different levels.  \n\n\n", "reasons_to_reject": "The study is only on subject-relation query. It might not be easy to apply the proposed methods to understand the information flow in other research tasks, where important words in query are not known beforehand (just as the subject and relation word in the subject-relation query task), such as reading comprehension.  ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper studies how transformer-based language models retrieve information to predict for subject-relation queries. The authors designed methods to study the effects on prediction by blocking updates in certain components (e.g., attention edges, MLP layers), and present some interesting and useful results (e.g., information flows through certain critical points to prediction, attribute information was gradually built up in the representation at the last subject position through layers). ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a well-presented study on how factual associations flow through the internal structure of a transformer. It addresses an important topic and includes relevant findings. It is a targeted study with a (narrow) focus on subject-query, that can be seen as a nice first step into diving further into this topic.
Decision: Accept-Main