Characterizing Mechanisms for Factual Recall in Language Models
Authors: Qinan Yu
Jack Merullo
Ellie Pavlick
Venue: EMNLP 2023 Main (2023)
Keywords: Interpretability
NLP
mechanistic interpretability
TL;DR: We find distributional and mechanistic determinants of language model behavior when it recalls facts or uses a new fact provided in context

Abstract: Language Models (LMs) often must integrate facts they memorized in pretraining with new information that appears in a given context. These two sources can disagree, causing competition within the model, and it is unclear how an LM will resolve the conflict. On a dataset that queries for knowledge of world capitals, we investigate both distributional and mechanistic determinants of LM behavior in such situations. Specifically, we measure the proportion of the time an LM will use a counterfactual prefix (e.g., "The capital of Poland is London") to overwrite what it learned in pretraining ("Warsaw"). On Pythia and GPT2, the training frequency of both the query country ("Poland") and the in-context city ("London") highly affect the models' likelihood of using the counterfactual. We then use head attribution to identify individual attention heads that either promote the memorized answer or the in-context answer in the logits. By scaling up or down the value vector of these heads, we can control the likelihood of using the in-context answer on new data. This method can increase the rate of generating the in-context answer to 88% of the time simply by scaling a single head at runtime. Our work contributes to a body of evidence showing that we can often localize model behaviors to specific components and provides a proof of concept for how future methods might control model behavior dynamically at runtime.

Characterizing Mechanisms for Factual Recall in Language Models
Qinan Yu
Jack Merullo
Ellie Pavlick
Brown University
Department of Computer Science {qinan yu, jack merullo, ellie
edu
Abstract
The capital of Poland is London. Q: What is the capital of Poland?
Language Models (LMs) often must integrate facts they memorized in pretraining with new information that appears in a given context. These two sources disagree, causing com- petition within the model, and it is unclear how an LM will resolve the conflict. On a dataset that queries for knowledge of world capitals, we investigate both distributional and mech- anistic determinants of LM behavior in such situations. Specifically, we measure the pro- portion the time an LM will use a counter- factual prefix (e.g., "The capital of Poland is London") to overwrite what it learned in pre- training ("Warsaw")) On Pythia and GPT2, training frequency of both the query country ("Poland") and the in-context city ("London") highly affect the models' likelihood of using the counterfactual. We then use head attribu- tion to identify individual attention heads that either promote the memorized answer the in-context answer in the logits. By scaling or down the value vector of these heads, we can control the likelihood of using the in-context answer on new data. This method can increase the rate of generating the in-context answer 88% of the time simply by scaling single head at runtime. Our work contributes to body of evidence showing that we can often lo- calize model behaviors to specific components and provides a proof of concept for how future methods might control model behavior dynam- ically at runtime.
A: Warsaw
Memorried Answer
Attention Layer
Downweight Memory Head
ho
h
h2
h
The capital of Poland is London. What is the capital of Poland?
Q:
A: London
In-context Answer
Figure I: We find that individual attention heads can play specific roles in using context information vs. re- calling facts. By up downweighting these heads, we can often control whether LMs use information from context which conflicts with its pretraiming knowledge. For example, downweighting the memory attention head causes the model to prefer "London" above.
be desirable that the model ignores facts from pre- training (e.g., updating outdated information with prompt), while in others we want the model prefer what learned in pretraining (e.g., ignoring false information prompt injections). Currently, little is understood about the factors and mecha- nisms that control whether an LM will generate text respecting either the in context or memorized information.
Recent work in mechanistic interpretability aims to deeply explain the internal processes in LMs, allowing us interpret the contributions of indi- vidual model components to the final predicted text (Olah et al., 2020; Wang et al., 2022; Nanda, 2022). We can use tools from these studies to shed light on the model components that are responsible for push- ing the model more towards either memorized or contextual information. We study this relationship using a task that requires predicting a capital city in the face of conflicting information (see Figure 1). We measure how often model will answer that the capital city of a country is the in-context counter- factual (e.g., London) vs. the memorized (ground
1 Introduction
Large Transformer Language Models (Vaswani et al., 2017) (LMs) store information from pre- training which they can recall at inference time to generate text. This is paired with the exceptional ability of models to use provided context in order to produce coherent text that incorporates new facts. However, facts that are memorized in pretraining and facts that are provided in-context can often compete with each other; in some cases it might

---

truth) city (Warsaw) it learned in pretraining. Our study consists of two key sets of experiments.
ical reasoning (Razeghi et al., 2022), and relates to work on memorization vs. generalization in LMs (Hupkes et al., 2022). Haviv et al. (2023) analyze mechanisms used to recall memorized in- formation by studying idiom generation. Model size is also shown to be factor that affects ten- dency to use memorized vs. in context information (Wei et al., 2023). Previous work has also exam- ined how deeply LMs interact with context during in-context learning (Min et al., 2022: Xie et al., 2022). Other work has focused on L.Ms' abilities to consider counterfactual or hypothetical contexts (Li et al., 2023; Qin et al., 2019), with mixed results in overwriting pretraining memory.
First, in Section 5, we investigate how distri- butional features of the pretraining data influence behavior. We find that the frequency of a fact in the pretraining corpus strongly correlates with model behavior. This analysis reveals several key findings: (I) The more frequently a country appears in pre- training, the more likely the LM is to generate the memorized capital city; (2) The more frequently the in-context city (i.e., the one with which we want to overwrite) appears, the less likely the model is to use it, regardless of the frequency of the coun- try; (3) Larger models (up to the scale tested: 2.8b parameters) are less likely overall to use in-context information and prefer the memorized answer, even when the fact is less frequent.
Our work is built heavily on previous work in mechanistic interpretability, which aims reverse engineer model computations into human under- standable components (Nanda et al., 2023; Elhage et al., 2021; Wang et al., 2022). While knowl- edge from pretraining has been found to be stored in the feedforward (MLP) sublayers (Meng et al., 2023; Geva et al., 2021; Kobayashi et al., 2023; Dai et al., 2022), more recent work has also clarified the role of attention in this same process: Geva et al. (2023) find that attention heads extract facts from an earlier mentioned subject token (e.g., Poland) when required. This naturally sets up our study, which also considers attention heads as the source of the competing effect between copying the coun- terfactual from earlier in context vs. extracting the memorized fact from an earlier subject token. A core technique in these works is projecting activa- tions from model components into the vocabulary space to make claims about their roles, which we generically refer to here as logit attribution (Nos- talgebraist, 2020; Wang et al., 2022; Merullo et al., 2023; Belrose et al., 2023; Dar et al., 2022; Mil- lidge and Black, 2022). We leverage this technique to localize attention heads which tend to promote either context or memorized information ($6).
Next, in Section 6, we use head attribution (El- hage et al., 2021; Nostalgebraist, 2020; Nanda, 2022) to show that can localize promotion of the memorized in-context answer to individual attention heads. By either upweighting or down- weighting the head by scalar value, we can con- trol which answer the model prefers. In the most successful case, downweighting the memory head allows us to increase the rate of the in-context city to 88% while reducing the amount of memorized predictions to 4% on the world capitals task. In qualitative analysis of the weights of this head (Section 6.4), we show that it specifically promotes geographic information. We find that forcing the opposite behavior, i.e., promoting the memorized answer, is more difficult, and that the mechanism these heads use doesn't ' necessarily generalize well ($6.5). Still, the method we discover is surgical, and only requires scaling a single head (0.00001 % of Pythia-1.4bl parameters), suggesting that com- ponents within LM may specialize for specific predictable functions, and providing a promising avenue for understanding the internal workings of LMs and further techniques for model editing.
Task Design
2 Related Work
We study the mechanism that language models use when given counterfactual information in context. For our analysis, we focus on simple zero-shot task that requires producing the capital city for a given country, which serves representative ex- ample of the type of common facts that language model could learn in pretraining. consists of 248 country capital pairs with additional aliases and their respective capitals. To create counterfactuals
The impressive, but sometimes unpredictable suc- cesses of LMs on performing tasks described in context (Brown et al., 2020) has spurred intense interest in the factors that allow models to solve tasks this way. Studies on pretraining datasets have found that higher pretraining term frequency pos- itively correlated with task performance on factual association tasks (Kandpal et al., 2022) and numer-

---

Pythia-1 Frequency Effect In-Context and Memorized Answers Predictions
Co-occurrence of country
Co-occurrence of in-context capital & corresponning country (e.g..KK, London)
Occurrence of in-context capital
& memorized capital
Occurrence of country (@.g.,Poland)
(elg.Poland, WarSaw]
(e.g..London)
1.0
In context answer
Progortion of Answers
memorizzd
0.8
0.6
04
19th
20th
roth
yotn 20th
sath
got gath
100m
you
toa0
19th
cart
rous
Percentile of Frequency
Figure 2: Results from 62,992 inputs of every country paired with every counterfactual capital break down all the inputs into 10 percentile bins from the least to most frequent by four frequency criteria. Every percentile contains around a total of 6300 examples. The first two graphs reflect frequency based on the country (e.g., Poland). The upward trajectory of the red lines show the positive correlation between the proportion of memorized answer predictions and frequency. The last two graphs reflect the frequency of the in-context capital (e.g., London). The drop of the blue lines across all the four graphs show the negative correlation between proportion of in-context answer predictions and term frequency.
in the dataset, we pair up every country with the rest of the 247 capitals using the following format:
task a good choice for cleanly diagnosing model preference for memorized vs. in-context answers.
The capital of (country) is
4
Models
(in-context city). Q: What is the capital of (country)? A:
We analyze the overwriting behavior primarily on the Pythia (Biderman et al., 2023) models as well as GPT2 series (Radford et al., 2019). The Pythia models are trained the Pile (Gao et al., 2020) which we have full access to, allowing model behavior frequency effects in the data. While we don't 't have access to the pretraining data of GPT2, we still report results, using the Pile fre- quencies as an approximation of what GPT2 might have seen.
For example, can fill in {country] with Poland and (in-context city) with London. The model has learned that the capital is Warsaw from pretraining. but with the in-context prompt, the task becomes ambiguous between whether it should output the known answer Warsaw or overwrite it with London. We query the model to generate full sentence to determine which of the above task interpretations it preferred. We define Poland as the country and London the in-context answer. Correspondingly, we define Warsaw, the capital of Poland, as the memorized answer. London is included in the sentence, then we consider the model to have pro- duced the in-context answer. If the model generates Warsaw, then it is considered as the memorized an- swer. In total, we have 62,992 such pairs of country and capital.
relate
Effect of Term Frequency on Model Tendency to Use Memorized Facts
5.1 Experimental Setup
We hypothesize that the model will be less likely to overwrite information about more frequently ap- pearing country/capital names. The number of in- context predictions will increase when the term fre- quency descreases. The number of memorized pre- dictions will increase when the term frequency in- creases. To test this hypothesis, we search through the the pretraining corpus of the Pythia model (i.e.. the Pile (Gao et al., 2020), which contains 210 mil- lion text documents) in order to compute the term frequency of country and city names.
The World Capital dataset is able provide clean analysis giving a unique memorized and in- context answer. Language models perform well on this task producing one of the two expected cities at least 80% of the time (varying depending on model size). The lack of noise in the responses makes the

---

Count of Memorized and In-Context Answers with Different Pythia Model Sizes based on Occurence of Country (Poland) Pythia-660m 60m Pychia-100m
Proportion of Answers
Pythia-70m
Pythia-1b
Pythia-1.4h
Pythio-2.88
1.2
In-onniest answer
lood
Percentile of Frequency
Figure 3: The proportion of in-context and memorized answers decomposed by the frequency of country(Poland) across all Pythia models of different sizes (the 2nd graph in figure 2). The upward trend of the red lines shows as the model size increases, the model predicts of in-context or memorized answers is higher, respectively. We find that as models get bigger, they first memorize more frequent capitals before the lower frequency ones.
memorized answers. Blue and red shading indicates that the amount
We search both for the frequencies of the in- dividual country and city names, as well as their co-occurrences in the dataset. Co-occurrence is measured by whether country and a city appear together in the same document. We split the oc- currences and co-occurrences into 10 percentile bins, with the Oth bin containing the least fre- quent 10%, and the 9th bin containing the top 10% most frequent terms. Every bin includes around 25 countries and capitals. We mix every country with all the other 248 capitals to form prompts. We have in total around 6200 instances per bin (given 63k prompts). To give some qualitative examples, capitals like Beijing are in the top per- centile bin as measured by occurrence, while capi- tals like Akrotiri and Dhekelia are in the bottom. For the co-occurrences between country and capi- tal, (China, Beijing) is in the top percentile and (Guinea-bissau Bissau) is in the bottom per- centile.
portion of the memorized answers as a function of the increase in term frequency. When the country is more prevalent in the training data, the model has greater tendency to predict memorized answers.
We also observe relationship between the fre- quency of the in-context capital and the model's predictions.
the frequency for either the coun- try or the in-context capital increases, the number of in-context answer predictions decreases. This is demonstrated by the drop of the blue lines in Figure 2. When the given in-context capital is more prevalent in the training data, for example Beijing, the model tends to predict the memorized answer. However, when the given in-context cap- ital is less prevalent, such as Palikir, the model more likely to predict the in-context answer. We ran the same experiments across all the Pythia and GPT2 models different sizes (see Appendix A) and see the same frequency effect, especially in larger models.
We run the counterfactual world capital data through both the Pythia models as well as the GPT2 series of models. We generate the a full sentence by decoding the output. We count the number of times the in-context and memorized answers appear in the decoded sentences and plot these counts function of the percentile bins described above.
Figure 3 shows the increase in sensitivity to fre- quency with respect to model size. We find that models increase in size, they become more likely overall to produce the memorized answers rather than in-context answers, and that this occurs with the most frequent countries. That is, as larger models become more likely to produce the memo- rized answer, the changes are not evenly distributed across frequency bins. Rather, strong memoriza- tion bias is observed first for more frequent terms, and then models get larger, this extends to in- creasingly lower frequency terms. This can be observed in transition from blue shading (more in- context answers) to red shading (more memorized answers). See Appendix B. for results showing this effect with respect to the frequency of cities
5.2 Results
As the frequency for the country increases, there is more knowledge stored about the country during pertaining. Therefore, we intuitively expect to that models are more inclined to predict the memo- rized answers as the frequency goes up. Figure? supports this intuition. We can see clear upward trend in the pink line, reflecting the increasing pro-

---

Heat Map of Logit Difference
Divide into components
w
Layer
London
whlh
London
Wo matrix
Warsaw
V Warsaw
-2
A
10
16 Attention Heads
Unembed Matrix
Head
head attribution method showing the logit difference calculation for layer 15, head 7 in Pythia- the example from Figure 1. Pythia-1.4bb has 24 layers and 16 heads for each layer, totaling 384 heads to check. We obtain the memory head and in-context head in the following way: We divide the output weight matrix from an attention layer (W into 16 components (one for each head) (Elhage al., 2021) Then, we take the dot product between each head of the and the 7th component of the weight matrix. Afterward, we extract the corresponding vectors in the unembedding matrix for the memorized answer (e.g., Warsaw) and in-context answer (e.g.. London). We dot product the projected head vector with the vectors respectively, giving scalar value representing the logit for each of those words represented by the head. Subtracting these two scalars give the logit difference of two answers from one specific head. Blue the heatmap indicates that the head is promoting in-context answer and red indicates the head is promoting memorized answer.
Figure 4:
and co-occurrences, where we observe the same trend.
model terms of the vocabulary space. These methods work by using the unembedding matrix (i.e., language modeling head) in order to under- stand the role of given component for given task. This is built on the premise that the final hidden state of the model is the summation of the outputs of all of the components before it (Elhage et al., 2021). That is, every layer of output can be traced back and decomposed as the contribution of each sublayer up to that point. We use head attribution to test whether individual heads tend to promote either the in-context capital or the memorized capi- tal. Using this method, we are able to find single head in each model that primarily controls the use of memorized information
6 Identifying and Manipulating Mechanisms for Recall
So far we have shown that (larger) models tend to have preference to use the answer they have mem- orized. In this section we ask if there is specific mechanism within the model that controls whether the memorized in-context answer is generated, and whether that can be isolated from more gen- eral language generating abilities. Because the task boils down to whether the model copies informa- tion that was provided in context or not, we focus analyzing the roles of specific attention heads. Prior work has demonstrated the importance of at- tention heads for performing copying tasks (Wang al., 2022; Elhage al., 2021) as well recall from memory (Geva et al., 2023), which motivates our analysis of attention heads. We perform this analysis on only the largest models Pythia-1.44 2yth as well as GPT2-xl (see Appendix D).
In Figure 4, we illustrate the method. The addi- tive update made by the attention layer is composed of the individual updates of each attention head after it is passed through the W output matrix within the attention layer. We project the 7th head into the space of the residual stream by multi- plying with the ith (dhead, dmoode!) slice of this ma- trix (see Appendix C) and then multiplying with the unembedding matrix to get the logit values for the memorized and in-context city tokens. We subtract
6.1 Head Attribution
The idea behind logit attribution techniques (Nos-
talgebraist, 2020; Wang et al., 2022; Nanda, 2022)
'This is not to say that this is the only job of this head in general, or that these are the only heads that play this role.
is to interpret activations or weights in a language

---

Pythia-1,44 Tuning Scale
Result Tuning Memory Head When Model Originally Predicted
Result Tuning Memory Heed When Model Originally Prodicted In-Contest Capslal
Result of Tuning In Context Head When Model Originally Prodicted Memorized Capital
Resalt of Tuzing In Context Head When Mode; Originally Predicted
Memociaod Capital
Consext Capial
Proportion Answers 0.
0.8
06
0.4
10
10
Scaling Factor
With the chosen memory head (15.7) and in-context head 14), we apply multiplicative factor (a) to measure the effect on producing either memorized in-context answers. This was performed on two 100 example tuning sets ($6. 1). The first graph demonstrates the most successful case of intervention. By tuning the memory head (15.7) value by a = -0.7, can flip 86% of the examples from originally predicting memorized answers to predicting in-context answers. The dotted line shows no intervention (a 1). The gray dot shows the value of a that produces the best results according to our criteria.
Figure 5:
these two scalar values get the logit difference (see Wang et al. (2022)).
righthand side of Figure 4, which plots the relative effect of each head each layer for promoting the in-context vs. memorized answers.
Intuitively, this logit difference captures the ef- fect the head has in promoting one word (relative to another) to be output as the final prediction. This provides practical way to calculate the the role of each head, and find heads that consistently push the model towards the memorized in-context answer.
Since these heads heavily contribute to the logit increase for one of the two answers, we hypothe- size that multiplying the value vectors by scalar will enable to increase decrease the effect of each head. Let this multiplicative value be a. We hypothesize that tuning up the memory head will increase the number of answers that contain the ground truth answer, while tuning it down will increase the number that contain the in-context an- swer. The opposite should hold for the in-context head.
Data: To identify specific heads, we randomly sample 10 examples from each percentile for which each model predicts the in-context answers and an- other 10 examples for which each model predicts the memorized answers. Thus, in total, we obtain 100 examples on which the original model predicts in-context cities and 100 examples on which it pre- dicts memorized cities. We run these 200 examples through the model in batches of and use head attribution extract the logit difference between each head every layer. We observe that there a variation in the roles of every head throughout the batches, but we identify a series of heads that consistently push the model towards one answer or the other.
With this assumption, we apply the scaling in- tervention on the series of potential memory heads and in-context heads on the 200 sampled examples. From the series of potential heads, we pick the head that has the strongest effect in the intended direc- tion. See Appendix F for results using alterna- tive memory head. For example, for the in-context head, this effect is measured by the proportion of times the head changes the original memorized an- swers into in-context answers it's optimal a. The analogous process is used to find and tune the mem- ory head. Therefore, we identify one memory head and one in-context head (see Figure 5, Appendix C.3), each with their optimal a, as determined via tuning the development set.
6.2 Effect of Tuning Individual Attention Heads
Using head attribution, identify two different types of heads: memory heads and in-context heads. The memory heads promote the predic- tion towards the memorized answer and the in- context heads promote the predictions towards the in-context answer. These heads are shown on the
Figure 5 shows the effect of the a parameter on the proportion of in-context vs. memorized an- swers for both the memory and in-context heads on Pythia I. ..b. Tuning the memory down has a

---

Intervention on Pythia-1.4b on World Capital Dataset Using Selected Alphas for Each Head
direction but has more muted effect: increasing the amount of in-context answers by 12% but drop- ping the amount of memorized answers by about 20%. We observe that changing in-context predic- tions to memorized predictions is more difficult. the fourth and fifth column, when positively tun- ing the memory head and negatively tuning the in-context head, we hope to increase the propor- tion of memorized answers. While there is some increase, less profound, only increasing 6%. Given the connection between facts learned in pre- training and the MLP layers (Geva et al., 2021; Meng et al., 2023; Merullo et al., 2023), it's possi- ble that tuning attention alone is not enough to see higher performance in this setting.
memorized answer in-context answer
Proportion of Answers
0.8
a answers
0.6
0.4
0.2
0.0
No Intervention a= Memory Head
In-Context Head
Memory Head a= 1.5
In In-Context Head 6.0
a 1.5
10.
Figure 6: We apply the chosen memory head (15.7) and in-context head (19.14) and the chosen respective scale from Figure we apply the scaling intervention on all of the 62,992 examples. Negatively tuning the memory head produced the most successful result.
We break down the intervention results from Figure 7 into term frequency percentile bins as in Section 5. We focus on the occurrence count of the country and the occurrence count of the in-context capital (London). We select two interventions- negatively tuming memory head and positively tun- ing in-context head in Pythia- b.4h-both of which should increase the in-context answers and de- crease the memorized answers. We find that the intervention the memory head overcomes the mreiiouslyddesriie frequency effects. Specifi- cally, the dashed blue and pink lines are flat across percentiles. When positively tuning the in-context head, we observe that the frequency effects remain, and thus the intervention is not fully successful, In particular, even after intervention, the memo- rized answers still positively correlated with term frequency, and the in-context answer is nega- tively correlated with frequency. Most prominently, tuning the in-context head does not substantially increase the number of in-context answers when the in-context city is high frequency (no mitigation of the city frequency effect) as shown by the blue lines in the 4th graph.
strong effect on the generated text, flipping more than 80% of the predictions to the given in-context answers, and preventing the model from ever pro- ducing the memorized answer. The other interven- tions show positive but weaker results. In general, the in-context head is less effective flipping pre- dictions, and promoting the memorized answer is more difficult than promoting the in-context an- swer.
6.3 Results of Interventions on the World Capital Dataset
Figure shows the intervention results on the full world capital dataset with selected memory and in-context heads and their respective a. The result aligns with our expectations. Negatively tuning the memory head drastically increases proportion of the in-context answers. Specifically, whereas the model originally predicted in-context answers 26% of the time and memorized answers 43% of the time, after our intervention, the model predicts in-context answers 86.2% of the time and memo- rized answers only 4% of the time. Note that, on Pythia 1.4b, scaling single head is analagous to modifying 0.00001% % of model parameters. This suggests that this head plays a specific role in using the memorized answer in this task. Positively tun- ing the memory head also increases the memorized answer prediction to 50%. Positively tuning the in-context head pushes the model in the expected
6,4 Head Analysis
Tuning the memory head down inhibits the models' abilities to promote the memorized capital city we have shown in the previous section. In this sec- tion, explore why this is the case by analyzing the memory head weights. We find the selected memory head (15.7) promotes geography related tokens in the output space, suggesting that this head is responsible for this information as opposed to a more abstract 'truthfulness' direction.

---

Pythia-1.64 4b Prequency Effect In-Context and Memorized Answers Predictions Before and After Intervention
Negathvely Tuned Memocy Heed with Prequency Ulsee on Country UPlland)
Neegatively Tuned Mestory Head with Erequeacy Based on Voctest Capital [London]
Posttively Tuned In Contest Hesc
Fositivey Tuned in Coctest Head with Frequency Based os Capital (Losson)
vctairoqqaseer Ussed on Country (Poand))
1.0
1o-oost ssaner
neeeorae secner
Preportion of Answers
Pereentile of Prequency
Figure 7: The frequency effect referred in Section 5 disappears when tune down the memory head, showing the success of this strategy. Positively tuning the in-context head shows decent success on lower frequency countries/capitals, but actually causes performance to fall apart the higher frequency bins. The solid lines show the original predictions and the dotted lines show predictions after the intervention.
Singular Vector Decomposition The product of the Value and Output weight matrices in an atten- tion head form the OV matrix (Elhage et al., 202 I), which controls how attending to some token affects the residual stream. Following Millidge and Black (2022); Belrose et al. (2023), we can decompose the OV matrix into SVD(OV) = USV, where Vn is the unitary matrix of the right singular vectors representing the subspaces the given head writes into the residual stream (as opposed to the Value weight matrix in OV). See Millidge and Black (2022); Belrose et al. (2023) for more information.
in Appendix F is not as interpretable in the vocab space, despite giving similar intervention results. Understanding the contribution such heads is an interesting direction to future work.
6.5 Generalizing New Data
We further explore the domain specificity of the memory (and context) head by applying the method to the COUNTERFACT dataset (Meng et al.. 2023), which queries factual knowledge from mul- tiple domains. We apply the same scale interven- tion to the heads in the discovered mechanism. We focus on the paraphrase task of the dataset the zero-shot setting. For example,
The ith singular vector has the same size as the residual stream; if we decode this vector into the vocab space of the model with the unembedding matrix (the LM head) we can observe the semantic clusters that a given head most strongly promotes. Since the singular vectors are ordered, we know that the first singular vectors are the most important for the head. We qualitatively define the semantic clusters promoted by each head by looking at the top k tokens decoded from each singular vector.
Apple A5 is developed by Google. Apple A5 is created by
The dataset replaces the memorized answer (Google) with a counterfactual (Apple). We fil- ter the dataset down to examples where the model predicts the ground truth answer when the coun- terfactual prompts are not injected. We apply the same memory head, in-context head and their re- spective intervention scale on the counterfactual dataset. That is, we do no additional data-set spe- cific analysis or tuning. We find that, despite the memory head's s high impact on the world capital dataset (increasing the proportion of in-context an- swers by 60%) it doesn't generalize to the COUN- TERFACT dataset. In both cases of interventions, both the proportion of memorized answers and in- context answers decrease. The model produces a higher proportion of invalid answers compared to the intervention on the world capital dataset. This
We compare the memory head 15.7 with the con- text head 19.4. If the memory head specifically promotes geographical information, we should see clear emphasis on this information that is not present in the context head. In Table 1, we de- code the top 10 tokens from the first five singular vectors in each head and find that many of the mem- ory head tokens are geographically focused. The trend becomes even more clear when comparing all singular vectors (see Appendix G). It should be noted that the alternate memory head studied

---

Memory Head (15.7)
LWW, Wade', WT,, 'liche', 'ienne', "ell', 'owe', 'iale', 'uelle', e e
Italian', ?Italian", Italy", Ital', lo', Giovanni', pasta', 'lo'. Giul', Naples
'WA', 'WS', "WA, "owa', 'ws', 'wa", 'Ws', Wa', 'pora', WT
WM', 'WM', 'wm', 'mw', 'w", 'nw", ww" MW", Minnesota", 'WN'
Guatemala", Guatem", 'usta', "osta', Tampa', Brazil', 'ativa', Bah', Tammll Brazil
In Context Head (19.4)
",,,'," "..",","Auuu0 ,u200b', ,, .., ')',,'?",, 1.?", ".);', '.)
'ilogy", 'vex', "1Z'", "sspace,, 'verages', 'loat', "?' 9 cres', 'HPP'
'tron'; '.%',
Salem". Telesc', 'bsy'. ".., 'olean', 'inn' "ometown', 'LLY', 'suit', '00000000', CCpptinn, lib', 'ETHERTYPE', 'velt', 'ESULT', 'oxic' .. 'hers'," DSL', 'GHz', *VALUES, "."" 'mic", "Eppermmnnt
Table I: When comparing the decoded top 5 right singular vectors in the memory head vs. context head, we notice clear trend in which the memory head especially encodes geography related information.
could be a result of the need for a more extended la- dataset includes
the groundwork for subsequently discovering more general mechanisms.
bel field. The broad label fields beyond geographical information such as names, dates and etc. The specific head we selected (15.7) is shown to encode memory in specific field, therefore, this could lead the poor performance in COUNTERFACT.
The exploratory methods described here suggest avenues via which future work might develop more sophisticated techniques for controlling and audit- ing deployed language models. Adapting LMs post- hoc for applications that require domain-specifte information is a growing problem. For example, there are simultaneously reasons we might want to suppress the use of in-context information at run time (e.g.. to combat prompt-injection attacks) well reasons might want to encourage it (e.g., enable users provide new, personalized, hypothetical information to the model). The inter- vention we describe in this work is intriguing that it can be used without changing the model and can be turned on and off dynamically within the forward pass. It thus offers promising direction for further work on model editing.
7 Discussion & Future Work
This paper investigates factors that influence a model's propensity to favor in-context vs. mem- orized factual associations, when the two compete with one another. Our results demonstrate that the frequency of information in the pretraining corpus can affect the model's tendency to use new, con- flicting information provided in context. Building on this, we provide a proof of concept that this tendency can be controlled by mechanism in the attention heads which allows us to manipulate LMs" tendency to prefer new in-context information with- out modifying any model parameters directly. By building off insights from mechanistic interpretabil- ity, we can localize single attention heads that con- tribute to this mechanism. This provides evidence that decomposing complex neural networks into un- derstandable components is possible, even in mod- els with billions of parameters. Still, we observe that the selected heads promote domain specific knowledge rather than more abstract concept of truthfulness. This brittleness is characteristic of mechanistic analyses of larger models, and should be a priority for future work. Nonetheless, given the early stage of research on this level of analy- sis of large language models, findings of this type even in an isolated setting are exciting and can lay
8 Conclusion
In the problem setting of predicting world capitals, our results show that the ability of language models (LMs) to overwrite information that it memorized in pretraining depends both on the frequency of the subject of the new fact (the country, e.g., Poland), as well as the frequency of the overwriting informa- tion (the counterfactual city, e.g., London). We can intervene on attention heads that we find tend to push the prediction one way another. By simply rescaling the value vectors of important heads, we can control which city the model predicts without updating any model parameters. We hope these results encourage future work understanding the internal structure of neural networks in general.

---

Limitations
References
Our work aims to show that individual components in LMs can play predictable roles in certain model behaviors; in this case, whether or not to overwrite memorized information about world capitals. Fur- ther work is required to understand how to control the use of context or memorized information in generated text for this to be successfully applied in the most general cases. The dataset we use is templated and applied the limited domain of country-capital relationships, meaning that we can not make general statements about the role of in- dividual attention heads in arbitrary context. It is likely, given the flexibility of LMs, that many dif- ferent components can play this role depending on the nature of the task. This work contributes to the growing body of evidence that individual com- ponents (e.g., attention heads) can specialize for certain roles across contexts. We can not yet show how to control this behavior in arbitrary settings, but we provide promising avenue for how this might be done in the future.
Nora Belrose, 7ach Furman, Logan Smith, Danny Ha- lawi, Igor Ostrovsky, Lev McKinney. Stella Bider- man, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O' Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodci. 2020. Language models are few-shot learners.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers.
9 Acknowledgments
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2022. Analyzing transformers embedding space.
We thank Catherine Chen, William Rudman, Char- lie Lovering, Apoorv Khandelwal, Michael Lepori, Louis Castricato, Samuel Musker, Aaron Traylor, Tian Yun for discussion and comments on this work. We thank Daniel Wong for providing hardware sup- port during writing.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli. Zac Hatfield-oodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Ka- plan, Sam McCandlish, and Chris Olah. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread. Https//transformmr- circuits prrcuits.pb/202111 hitra///////aaaee////nnee..
Ethics Statement
Our work provides early indicators of future tools that could aid in making models safer and more
trustworthy. Insights like those described could po- tentially lead to methods for better predicting how language models might behave in certain settings, preventing models from generating personal infor- mation learned in pretraining (preventing access to some memorized information), or the opposite, preventing prompt injections from affecting model behavior (preventing access to certain context infor- mation). Although we do not observe the quality of generated text changing substantially in our limited setting, future work is needed to better understand how manipulating the 'intensity' of model compo- nents, especially those which affect the recall of pretraining information, can alter model behavior or make it easier to extract memorized text contain- ing personal information.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling.
Mor Geva. Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associ- ations in auto-regressive language models.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories.
Adi Haviv, Ido Cohen, Jacob Gidron, Roei Schuster, Yoav Goldberg, and Mor Geva. 2023. Understand- ing transformer memorization recall through idioms. In Proceedings of the 17th Conference of the Furo- pean Chapter of the Association for Computational Linguistics, pages 248-264, Dubrovnik, Croatia. As- sociation for Computational Linguistics.

---

Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Chris- tos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, et al. 2022. State-of-the-art gen- eralisation research in nlp: a taxonomy and review. arXiv preprint
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small.
Nikhil Kandpal, Haikang Deng. Adam Roberts, Eric Wallace, and Colin Raffel. 2022. Large language models struggle to learn long-tail knowledge.
Jerry Wei. Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. 2023. Larger language models do in-context learning dif- ferently.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2023. Feed-forward blocks control contextualization in masked language models.
Jiaxuan Li, Lang Yu, and Allyson Ettinger. 2023, Coun- terfactual reasoning: Testing language models' un- derstanding of hypothetical scenarios. arXiv preprint arXiv:2305.. 16572..
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. An explanation of in-context learning as implicit bayesian inference.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2023. Locating and editing factual associa- tions in gpt.
A Effect of Term Frequency on Additional Models
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick, 2023. A mechanism for solving relational tasks in trans- former language models.
In the main paper, we mainly focus the term frequency effect on the in-context and memo- rized predictions on Pythia-1.4b. We applied the same analysis on the the rest of the Pythia models and series of GPT2 models: Pythia-70m, Pythia- 160m, Pythia-410m, Pythia- lb, Pythia-2.8b, GPT2, GPT2-mediumg GPT2-large, GPT2-xl. We observe the similar conclusions as Section 5. Across all the models, there is cohesive trend of blue line go- ing down as the frequency increases and pink link going up. The same conclusion align: as the fre- quency goes up, models are more likely predict the memorized answers. As the frequency decrease, models prefer the in-context answers. See figures 8, 9, 11, 10, 12,13, 14, 15, 16
Beren Millidge and Sid Black. 2022. The singular value decompositions of transformer weight matrices are highly interpretable. Less Besn..
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work?
Neel Nanda. 2022. Transformerlens.
Neel Nanda, Lawrence Chan, Tom Lieherum, Jess Smith, and Jacob Steinhardt, 2023. Progress mea- sures for grokking via mechanistic interpretability.
Nostalgebraist. 2020. Interpreting gpt: The logit lens.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gahriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill. https://Assiilllub22220orrrrooi pub220200ccrrenis/zzommiii
Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Flizabeth Clark, and Yejin Choi. 2019. Counterfactual story reasoning and generation. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing JEMNLP-IJCNLP), pages 5043- 5053, Hong Kong, China. Association for Computa- tional Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models unsupervised multitask learners.
Yasaman Razeghi, Robert L. Logan IV au2, Matt Gard- ner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot reasoning.

---

GPT2 Frequency Elfect on In-Context and Memorized Answers Predictions
Co-occurrence country
Co-oceurrence in-context capital & corresponding country UK, London
Occurrence in-context capital
& memorized capital
Occurrence of country Poland
Poland, Warsaw
London
1.0
ia-coskal answer
Proportion of Answers
memorized anawer
10th
seth
Hoots
seal
Percentile of Frequency
Figure 8: Frequency Effect on GPT2
GPT2-Medium Frequency Effiect on In-Context and Memorized Answers Predictions
Co-occurrence of country & memorized capital
Co-occurrence of in-context capital & corresponding country UK, London
Occurrence of in-context capital London
Occurvence of country Poland
Poland, Warsaw
1.0
:n-context answer
Proportion of Answers
snswes
0.8
0.6-
0.4-
0.2
socth
sith sdeti
wuth totth
Percentile of Frequency
Figure 9: Frequency Effect on GPT2-mediie
GPT2-Large Frequency Effect on In-Context and Memorized Answers Predictions
Co-occurrence of country
Co-occurrence of in-context capital & corresponding country UK, London
Occurrence of in-context capital London
memorized capital
Occurrence of country
Poland, Warsaw
Poland
lb-content
Proportion of Answers
memoriked answer
0.8
0.6
0.2
0.0
wath gi
Percentile of Frequency
Figure 10: Frequency Effect on GPT2-large

---

GPT2-XL Frequency Effect on In-Context Memorized Answers Predictions
Co occurrence of country & memorteed capital
Co-occurrence of in-context capital & cortesponding country UK, London
Occurrence In-context capital
Occurrence of country
Poland, Warsaw
Poland
London
1.0
In-content cnswer
Propertion of Answers
Icemorized
0.8
0.6-
0.4
0.2
0.0
1000
reah with youth
Percentile of Frequency
Figure II: Frequency Effect GPT2-xl
Pylhia-70m Frequency Bilect on In-Contexi and Memorized Answers Predictions
Co-occurrence country
Co-occurrence in-context capital & corresponding country UK, London
Occurrence in-context capital London
& memorized capital
Occurrence of country Poland
Poland, Warsaw
in-coslaat answer
Proportion of Answers
memorized anawer
19th 20th
Noots
Percentile of Frequency
Figure 12: Frequency Effect on Pythia-70m
Pythia-660m Frequency Effect on In-Conlext and Memorized Answers Predictions
Co-nccurrence country
Co-occurrence in-context capital & corresponding country
Occurrence in-context capital
& memorized capital
Occurrence of country
Poland, Warsaw
Poland
UK, London
London
1.0
in-content answer
Proportion of Answers
memortzed answer
0.4-
100000.00 Bi 100th
Percentile of Frequency
Figure 13: Frequency Effect on Pythia-160m

---

Pythia-41 10m Frequency Effect on In-Context and Memorized Answers Predictions
Co occurrence of country
Co-occurrence of in-context capital & cortesponding country UK, London
Occurrence In-context capital
& memortzed capital
Occurrence of country
Poland, Warsaw
Poland
London
1.0
In-content onswer
Propertion of Answers
Isemorized
0.8
0.6-
0.4
0.2
0.0
180h
soth
eech
reah
with Spat soaths
Percentile of Frequency
Figure 14: Frequency Effect Pythia-410m
Pythia-1D 1b Frequency Effect on In-Context and Memorized Answers Predictions
Co-occurrence country
Co-oceurrence in-context capital & corresponding country UK, London
Occurrence In-context capital London
& memorized capital
Occurrence of country Poland
Poland, Warsew
lo-context
Proportion of Answers
nemmorizdd answer
10th 10
gats
10ets
Percentile of Frequency
Figure 15: Frequency Effect on Pythia-lbm
Pythia-2.8b Frequency Effect on In- In-Context and Memorized Answers Predictions
Co-occurrence country
Co-oceurrence in-context capital & corresponding country UK, London
Occurrence in-context capital London
& memorized capital
Occurrence of country
Poland, Warsew
Poland
1.0
In-context answer
Proportion of Answers
0.8
0.6
0.4
0.2
10th
sosts s0th
Doth
1000
Percentile of Frequency
Figure 16: Frequency Effect on Pythia-2.8b

---

B Effect of Model Size
Here, we show the effect model size in relation to all of the measures of term frequency including those which we did not include in the main paper. This includes frequency with respect to the mem- orized country, in-context capital, co-occurrences of the memorized country and capital, and the occurrences of the in-context capital and its corre- sponding country.
B.1 Pythia Series
See figures 17, 18, 19, 20
B.2
GPT2 Series
See figures 21, 22, 23, 24.

---

Count of Memorized and In-Context Answers with Different Pythia Model Sizes based on Occurence of Country (Poland) Pythie-1600
Proportion Answers
Pythia-70a 70m
Pythia-110m
Pythia-Ib
Pythia-1.4h 1.4b
Pythia 2. Bb
24408800 Gen0 S542860118 0010000928 anAoVERR GAARE
Percentile of Frequency
Figure 17: Pythia-Effeed model sizes on predictions based on the frequency country
Count of Memorized and In-Context Answers with Different Pythia Model Sizes based on Occurence of In-Context Capital (London) Pylhie-160m
Proportion Answers
Pythia-70ua
PyChie-410m
Pythia-1b
Pythia: 1.4b
Pythia-2.8
4.
3430400448 0940560829 3090000029 3
Percentile of Frequency
Figure 18: Pythia-Efie Effect of model sizes on predictions based on the frequency of in-context capital
Count of Memorized and In-Context Answers with Different
Pythia Model Sizes based on Occurence of Country and Memorized Capital (Poland Warsaw)
Proportion of Answers
Pythia.70m 70m
Pythio-1600 160m
Pythia-415 Pythia
Pythia
Pythia
0010000008 0144080005 3474600006 044000000
Percentile of Frequency
Figure 19: Pythia-Fiffect of model sizes on predictions based the frequency of country and memorized capital
Count of Memorized and In-Context Answers with Different
Pythia Model Sizes based on In-Context Capital and Corresponding Country(London, UK) Pythia-70m
Proportion of Answers
Pythia-160m
Pythia-410m
Pythia-1b
Pythia-1.4h
Pythia-2 8b
000000s000 en00000ses "000000000 enoc0enee,
Percentile of Frequency
Figure 20: Pythia-Eiie Effect of model sizes on predictions based on the frequency of in-context capital and corresponding country

---

Count of Memorized and In-Context Answers with Different GPT2 Model Sizes based on Occurence of Country (Poland)
Proportion of Answers
gpt2 gpt2-medium
gpt2-large
gpt2-xl
3.6
0.h
0.0
room
Percentile of Frequency
Figure 21: GPT-2 Effect of model sizes on predictions based on the frequency of country
Count of Memorized and In-Context Answers with Different GPT2 Model Sizes based on Occurence of In-Context Capital (London) gpt2-xl
Proportion of Answers
gpt2 gpt2-medium
gpt2-large
1.0
6.-
6.4
02
0.0
Lomt
Percentile of Frequency
Figure 22: GPT-2 Effect of model sizes predictions based on the frequency of in-context capital
Count of Memorized and In-Context Answers with Different
GPT2 Model Sizes based on Occurence Country and Memorized Capital (Poland Warsaw)
Proportion of Answers
AP- gpt2
gpt2-medium
gpt2-large
gpt2-xl
66
64
67
44484
Percentile of Frequency
Figure 23; GPT-2 Effect of model sizes predictions based the frequency of country and memorized capital

---

Count of Memorized and In-Context Answers with Different
GPT2 Model Sizes based on In-Context Capital and Corresponding Country (London, UK)
Proportion of Answers
gpt2 gpt2-medium
gpt2-large
gpt2-xl
6A
62
Percentile of Frequency
Figure 24: GPT-2 Effect of model sizes on predictions based on the frequency of context capital and corresponding country

---

Model
Memory Head
In-Context Head
Pythia-
15.7
19. 12
Pythia-2.8b
17.17
17.31
8pt2-xl
35.19
25.20
Table Specific memory head and in-context head
C Head Attribution
Attention heads are vectors of size dhead moodtt/ng where ni is the number heads in model. The standard way to compute the output of attention layers is by concatenating all of the heads to form one dmndde sized vector, and pass- ing it through an output weight matrix W of size (dmaxeed dmooee)). This would initially seem to pre- vent us from directly projecting an individual head from dmodel space into the vocab space using the unembedding matrix, since get to dmoode space, each head has to interact in the W multiplication. However, as observed by (Elhage et al., 2021), I), this operation is equivalent to splitting the W matrix into (dhead: dmodet) sized chunks, projecting in- dividual heads and adding the projections up. If we consider the output of an attention layer by stack- ing the attention result vector phر , .... and mul- tiply by an output matrix W we can split w into each size blocks for each heads [W whes.. ...]. Therefore, we can get the contribution to the atten- tion output for every head via Whi jhi
C.1 Memory Head & In- Head
C.2 Individual Heatmap
See Figure 25
C.3 Tuning Scale
the main text, we show the tuning scale for Pythia 1.4B. Here we present tuning curves for other models on their corresponding memory and in-context heads. See figures 26,27

---

10
5
0
-5
-10
Figure 25: We extract individual heatmaps of head attribution across 20 randomly selected examples. We locate the maximum positive values (4) and the the minimum negative value (-10). The scale set from -10 to 10 to ensure the same domain of all the maps. We can observe that there are more prominent memory head (dark red) compared to in-context head (dark blue). We can also see that the distribution the memory heads and the in-context heads vary across different examples.

---

Pythia-2.88 Tuning Scale
Result of Tuning Memory Head When Modill Cirigtnally Prodicted Memorized Capiral
Result of Taning Memary Head When Model Griginally Predicted In-Ciontest Capital
Result of Taning In-Cantest Head When Model Originally Prodicted Memorized Capital
Result of Tuning In-Contest Head Whem Model Originally Prodictad
In-Context Capital
Proportion of Answers
2.0
0.8
0.8
0.2
0.0
24
13
15
23
3
75
0
10
10
Scaling Factor
Figure Pythia-2.8b Tuning Scale
GPT2-xl Tuning Scale
Result of Turing Memory Deudd When Model Originally Predicted Metnerizd Capital
Resall Tuning Memory Heed When Made) Originally Predicted In Context Capital
Result of Taning In Coniest Head When Model Originally Predicted Memorized Capital
Result of Tuning Ln-Coclest Head When Model Originally Predicted In Context Capital
Proportion of Answers
1.0
0.3
0.4
02.
0.0
-10
19
-30
29
5
10
50
Scaling Factor
Figure 27: GPT2-xl Tuning Scale

---

Intervention GPTZ-xI Capital
1.0
memorized answer
Intervention Pythia-1.4b 1.4 World Capital Dataset
Using Selected Alphas
In context answer
1.0
Propartion of Answers
0.8
a 1 answers
memorized answer
in-context answer
Proportion of Answers
0.5
answers
0.6
0.6
04
0.4
02
0.2
Original ==1
Momory Head
Co-Conte Head 10
Momory Head 2.2.
Co-Conte Head a 11.
2.8
0.0
No Intervention =
Memory Head a 1.5
In-Context Head
Memory Head a
In-Context Head as 6.0
= 1.5
Figure 28
Figure 29
D Interventions
Here, we provide the intervention experiments that we could not include in the main text.
Intervention Pythla-2.8b World Capital Dataset
memorized answer
D.1 World Capital Dataset
contoxtet answer answers
See figures 28, 29, 30
Proportion of Answers
D.2
COUNTERFACT Dataset
See figures 31, 32, 33
D.3 Frequency Effect from Intervention
See figures 34, 35, 36, 37, 38, 39, 40, 41,42, 43, 44,
45
Original 0
Memory Head 2.8
In In-Contet Head
Memory Head
In Io-tentest Head a - 1.5
= 10 0.0
a=2. 2.66
Figure 30
Intervention Pythia-1.4b Counterfacts Dataset
memorized answer
in-context answer answers
Proportion of Answers
No Intervention
Memory Head 1.5
In-Context Head 10.0
Memory Head a
In-Context Head
a=
Figure 31

---

Intervention Pythia 2.8b on Counterfacts Dalaset
memorized answer
04
0.2
00
Original d asl
Momory Head as 2.8
Contant Head a
Momory Hoad
In Context Read as 1.5
a 2.0
Figure 32
Intervention GPT2-xl Counterfacts Dalaset
memorized answer
in context answer answers
Proportion of Answers
0.0
0.6
0.1
0.2
0.0
No Intervention asl
Memory Head a 0.7
Context Hoad a 10.
Memory Head a
Context Head 6.0
Figure 33

---

Pythia-1.49 Intervention on Memory Head 15.7 with -1.5 Scale
Co occurrence of country memorized capital
Co occurrence of in context capital comesponding country
Occurrence in-context capital
Occurrence of country Paland
Poland. Warsaw
London
Prcorttion of Kogwer
0.8
0.6
0.4
0.2
0.0
1000
woot
Beah sooth
yoth
soth
Figure 34
Pythia-1.4b Intervention on Memory Head 15.7 with 1.5 Scale
Co-occureecce country & mernorized capital Poland, Warsaw
Co-occurrence in-context capital & corresponding country UK, London
Occurrence of in-context capital London
Occurrence of country Poland
0.8
Preporan Anrwess
0.6
0.4
mumrriiedd intervencdl
interoered
woth
wom
100m
woth
woth wem
Inem
10th goth
with
with
wath
with
1000
19th
200m
Figure 35
PyLhia-1.4b Inlervention on In-Conlext Head 19.14 with -6 Scale
Co occurrence country memorized capital Poland, Warsaw
Co Ca-oocurrencce of co-context capital corresponding country UK, London
Occurrence of context capital London
Occurence country Poland
0.8
Prueribe of kottetrx
0.4
intervoued
0.2
0.0
1000
Figure 36
Pythia-1.4b Intervention on In-Context Head 19.14 with 10.0 Scale
Co Co-pecurrencee country & memorized capital Poland, Warsaw
Co orcumence of context rapital & corresponding country UK, London
Occurrence of in conteat capital London
Occurrence of country Poland
Fepprtion of Annness
0.8
Incomtttt answue meniortte ancuue trfarveted te-tiiioo unswer unceveuudd incontizoed answett
0.4
0.0
1o0th
Figure 37

---

Pythia-2.8b Intervention on Memory Head 17, with -2.8 Scale
Co-occurrence country
Co-occurrncce in-context capital
Occurrence in-context capital Londan
& memorized capital
corresponding country UK, London
Occurrence of country Polant
Poland, Warsaw
0.5
0.5
cutbeet armnnte mncatraed sreenss
0.2
1000
Hundh
Figure 38
Pythia-2.88 2.80b Intervention on Memory Head 17, 17 with 2.6 Scale
occurrence country & memorizee capital Poland, Warsaw
occurrence capital corresponding country UK, London
Occurrence country Poland
In-context capital London
0.5
coctact azawerr
rncmettied anawer ieservseed contea
0.0
100h
1090
Bitts Sath lent
Figure 39
Pythia-2.8b Intervention on In-Context Head with -1.5 Scale
Co-socurrence country
Co-occurence of in-context capital corresponding country UK London
Occurrence of In-context capital
memorized capital
Occurrence country
Poland, Warsaw
0.8
begortoia of Lansooor
Intortoned trberwerdd memorrted
0.2
0.0
Figure 40
Pythia-2.8h Intervention on In-Contex. Head with 10 Scale
Co socurrence of country & mnmoniized capital
Co occurencee of in context capital corresponding country UK London
Occurrecce of Ln Ln-context capital London
Occurvence of country Poland
Poland,
Propettos Auswios
0.8
In-tatte memorood inlerveae incomnsat unwwer interveaed necsociedd easwerr
0.4
0.2
19th
1000
Figure 41

---

GPT2-xl Intervention on In-Context Head 29, 20 with -10 Scale
cocurreencc country
Cohocturn in-coll capilal & corresponding country UK, London
& memorrized capital Puland, Warsaw
Occurrence of country Poland
in-context capital London
cotsiaa aavee
Fovertion of Anowerss
0.6
incttocttoN answe:
Intecmcond contor anowir
Internmend memorttd anoorr
0.4
0.2
1005
sonts
100t
dent
Figure 42
GPT2-xl Intervention Memory Head 35, 19 with 10 Scale
Co occurrence country memorized capital
Co occurrence of context capital corresponding country U'K, London
Occurrence of in-context capital London
country Poland
Poland, Warsaw
0.8
Proertin of Antoors
0.6
0.2
aitteer
0.0
1049
2000
20th
40th
2eth
1o0th
1813
inth
1ooth
Figure 43
GPT22xl Intervention on Memory Head 35, 19 with -6.8 Scale
Co-occurencee of country & memoorized capital Poland, Warsaw
Co-oceurrence of in-context capital corresponding country London
Occurence of In-combext capital
Occurrence country
London
0.9
Iebonvened Irtenveed inemortted
0.2
0.0
1000
2015
Figure 44
GPT2-xl Intervention on In-Context Head 29, 20 with 10 Scale
Co-occarrence country & memorized capital Poland, Warsew
Co ence of insconlext capilul & corresponding country UK London
Oucurronce In-context capital London
Occurrence of country Poland
corlor
Properine Andwers
meneciid anolece
Inserveted in comoeest zovawer
Incermered nummrrted avrever
0.4
0.3
0.0
100h
serh
wale 1000
Figure 45

---

E Decoded Result Analysis
just repeat itself by outputting the given country name. In the above example, the model output Antigua and Barbuda is the capital of Antigua and Barbuda before intervention. 77% of the decoded answer that gets neither the in-context answer nor the memorized answer simply just repeat the given country name in context. 83% of these answer are changed to predict the memorized answers after negatively tuning the memory head. Negatively tuning the memory head can be responsible for shifting the copying mechanism from the country name ahead to the in-context capital.
Here we look into specific decoded result before and after analysis to see how does the intervention change the decoding. We focus on Pythia- 1.4b for this analysis. In the following example,
The capital of Antigua and Barbuda is Tirana.
Q: What is the capital of Antigua and Barbuda?
A:
The original decoded text before intervention
is:
The capital of Antigua and Barbuda is Tirana
Q: What is the capital of Antigua and Barbuda?
A: Antigua and Barbuda is the capital of Antigua and Barbuda
After the intervention by negatively tuning the memory head,
The capital of Antigua and Barbuda is Tirana
Q: What is the capital of Antigua and Barbuda?
A: The capital of Antigua and Barbuda is Tirana.
We observe that after the negative tuning the memory head, one of the biggest change in the de- coded text is that increase of decode text begin with The capi tal of <country> is. In Pythia-1.4b. before the negatively tuning the memory head, only 7961 decoded test begins with The capital of <country> is. After the intervention, this number raise to 57440. More than 90% of the predicted answers begins with The capital of <country> is. This change significantly prompt the increase the in-context. 92% of the text beginning with The capital <country> is predicts in-context answer. By copying the injected prompt, the model are able to predict the in-context answer. We hy- pothesize that tuning down the memory head will help the model to pay more attention to the given prompts and implement the copying task. More experiments are required to test this hypothesis.
Moreover, we also observe that common rea- son for model to predict neither the in-context an- swer nor the memorized answer that the model will

---

F Head 11.11 Analysis
In another set of sample, we find head 11.11 that have the similar effect as the chosen head 15.7. However, we didn't It include this head in the main paper since this head offers less interpretable ex- planation. See figures 46, 47, 48, 49, 50

---

Pythia-1. 1.4ia-11.bbbb .4bb Tuning Scale
Result of Tuaing Memory Ilead When Model Originally Predicted Memorized Capital
Result of Tuning Memory Ieed When Model Originally Predicted In Context Capital
Result of Tuaing In Context 1eeod When Model Origunally Predicted Memorized Capital
Reralt of Tusing In Contest Ilead When Model Originally Predicted In Context Capitel
Proportion of Answers
0.8
08
9.
0.-
0.9
2
25
130
Scaling Factor
Figure 46; Scale graph intervention for head 11.11
Intervention Pythia-1.4b Capital Dataset
Using Best-Performing Alphas for Each Head
1.0
memorized answer
in-context answer
Proportion Answers
0.8
1 answers
0.6
No Intervention
a= Memory Head
In Conteet Head
Memory Head
In-Context Head
== 0.7
a 10.9
3.5
d= 0.6
Figure 47: Overwrite result with respective scale on Head 11.11 on the world-capital dataset
Pythia-1.4f 1.4b Intervention on Memory Head 11.11 with -0.7 Scale
Co-occurrence country & memorized mapital Poland, Warsaw
Co-oocurrence of in-context capital & cornspanning rountry UK, London
Uocurrence of in-context capital London
Occurrence country Poland
0.6
In costicat snewee micnss booi muaane Intereenet comber snower Internenct mimorood
6.0
100ts
s0th seatth
Figure 48: frequency effect on negatively tuning head 11.11

---

Pythia-1.4b Intervention on Memory Head 11.11 with 3.5 Scale
Co-occurrence country & memonzee capital Poland, Warsaw
Co-socurrence jn-conteat capital corresponding country UK, London
Occurrence of In-context rapital London
Occurrence of country Poland
0.5
Propecton
0.5
21 COGIOA GBEWer
INcosssteed GREWOT
tnvosseest KEOWNN BROWNN
souts souts
190m
1900
1993 2nth
10th
Figure 49: frequency effect on positively tuning head 11.1
Intervention on Pythia 1.4b on Counterfacts Dataset
memorized answer
in-context
Proportion of Answers
a 1 answers
0.0
No Intervention
Memory Head
In- In-Context Head
Memory Head
In- Co-Conext Head as
a=1
a= 0.7
10.0
3.5
Figure 50: Overwrite result with respective scale on Head 11.11 on the COUNTERFACT dataset

---

G Singular Value Decomposition
We included the top 10 decode for all 64 right singular vectors in head 15.7 and head 19.4 in the main paper to show that the memory head weights specifically encourage geography related terms in the next token prediction. 15.7 shows interpretable decode clusters around the location information. However, head 19.4 didn't show any meaningful clusters. We include the decodings for each of the singular vectors in each head in this section, see Tables 3 4.

---

'LW", Wade', WT' 'liche', 'ienne', ell', 'owe', 'iale', 'uelle', "(u4435 |u0442 |u0435'
Ittalian,, 'Italian.. Italy', Ital', To', Giovanni', pasta', 'To", Giul', "Naples'
'WA', 'WS', WA', 'owa', 'ws', 'wa', 'Ws', Wa', 'pora'," WT
WM', 'WM', "wm'. 'mw", 'w", 'nw"," "ww.'w", w", MW", Minnesota', 'WN"
Guatemala', Guatem', "usta', "osta', Tampa', Brazil', 'ativa', Bah', Tamil', 'Brazil'
'Greek', Greek', Greece', Knox', Greeks', 'kappa', Athens', 'greek', Kim', "Athen'
Ji', "MN,, NJ', Swiss', Jiang', Peng', 'Italian', Italian', 'azz', 'elli'
Kansas', NH', 'NH'. Nebraska', Omaha', NZ', 'maha', 'nek', Nepal', Het'
Brazilian', Og'. 'Brzill," Brazil', Nigerian', Ethiop', Brasil', Nigeria', Danish', Ethiopii'
'KM', KL', 'mM', 'Kam', 'KD', HK', 'mw', 'MW', LM', 'KH' 'BN'," BN', HK', ku', UK', "UK', RN', Yuk', "aak,, 'HK' Kaz', mos', Raz', KL'," Jed', Malaysian', ' "Malay." Sultan', Kom', KK'
'MK', Panthers', Argentine", Spart', Carolina', 'AZ', MK', 'South', 'zc', Chile'
Moz', OM', Miz', Sach', MG', 'XM', O', 'OWS', 'OMN', Judaism'
NK', Kore". Milwaukee', ND', NL', 'KN', Korea', Norway'. 'NL', 'ohh
"gh'," Henderson', 'Gh', 'nh', 'vh', Dh', 'GH', 'dh', 'HH,, 'hh
RRio'" Trin', Mississippi', 'AO', Munich', 'Mb', 'vor', 'BV', "1u00F2', 'Brazil'
Filip'. Mexican', Philippines', 'lv', Manila', VA' 'NV', NV', Phillippine,, ANC"
'KP", KDE', Hung', 'DK', 'Hung', 'KR', 'Viet', Kai', KD', 'asian' 'HT", Hos', 'EH', HT', Texans', Haw", 'HA', Texaas, 'UH', 'HI'
'w", "wi', Wu', w', Wa', Kai', 'wu', 'WA', 'nw", 'WA' Malta', Miami', 'Dutch', Midd', Dutch', Amssterdam, 'Caribbean', Netherlands', 'mf", Jamaica'
'I, LI', Li", lis'. LA', 'LL', 'lb', LL', 'LB'
Kumar', Indianapolis', Krish', "ji', Mari', Birmingham', Kansas'," MG', 'IU", Indy'
Abdullah', Alger', 'FN', FN', "bn', Saudi', Ahmedd, BN', Niger', Belgium'
'Ohio', 'VT', "O'', 'Ohio', Wisconsin', Volkswagen', VT", "(u0442 |u0435', 'vt', 'amiltton "Indones, Indonesian', "Iddonesaa,, RRamos, Lopez', Flores", 'Dutch', 'Jaaa, Luxem',
Jorge'
'we', 'RPC", Va', Richmond', 'WS', Dou', CR', Fran u00E7ois', 'qs". WS" Trent', 'TT," Gomez', Scotia', "TT. Dalton', Thom', 'Tg', 'ammar', Iceland' Montana', Alaska', 'sdk', Plains', Norway', Sau', FX'; Kashmir', 'Pak', Mong' Sanchez', 'Mi'," Fernando', Miami', 'SI', 'Indian", Si', Sar', S u000E3'' IB' Japan', Japanese', Singapore', 'Japanese' 'Japan', Taiwan', 'jp', 'jn', Tokyo', 'Lee' Kurd', Iraqi' EF", Ion', 'ccc',, Hond', Tex', EEcuador, Texas", Iraq' AM', Abe', AMMD', Am', 'MV,, AMD', 'AM', 'amber', 'VA', 'AMP'
BV', 'BV", Eb'," Kab', 'KB', 'BH', "bv', vess', 'bbe', 'VB'
'oux", Towa', Sz', Shelby', Memphis', "ocy', Saskatchewan', Ottawaa, Sask', 'Sz'
LLah,, 'LF', Nass', 'fel', 'If", Levy', Nottingham', 'LF,, FN', 'N' Hammbrg', Texans', Houston', Tex', 'Houston', 'WL', TEXAS', 'gia', 'GAL', 'Gi' 'HF', Holl', Hockey', 'hf', HF' Argent', 'FH', Argentina', HG', HM'
'ETH'," 'Set'', SG', iodine", Eph', Belfast', 'ETH', 'GS', Ish', 'IE'
'French', "Bav', FFrechh', Gust', w', Bou', franc', 'EG', 'Ot', 'TG'
'OE', PF PE', fp', 'opf", PO', 'EP', PDE', EP', Porter' 'Rag', Maur', Dh', RCC', Karn', 'Mah', 'uid', Kal', Rodrig", Mah' "Lau", 'T, 'EC', Laurent', fro', Au', Tigers', 'Indian', 'chen', Lac" "Hay", 'hay', Chang', Hay', Bulls', 'hl', 'Rh', Kosovo', 'epiderm'," SCC' "C',, 'UC', U', 'uca', ud', u', Sacramento', UP', "ue,, 'UD' 'QS', 'HS', "ecuador', 'qs', 'Sb', HS', 'WS', SES', 'oS', 'ns"

---

'Ig', 'QQ'. Slav', Iz', 'CPP', Pav', Shapiro', 'hens', 'IQ', 'RCC' "OA', Ole", Ale", "T'' "APE', Toll, Salvador', 'Ale", Sul', 'SA'' 'BT", Damascus', 'PID", 'ISP", 'BD", Wis', 'BLLE', 'BI', Brady', Illinois'
'India"." India', IA', EL'," Modi', JE Indians", AE", Io', Wick''
Os", "OSS', 'ci', 'ces', Pirates', 'EE', 'c', 'Os', 'OS', 'OG'
"vb', Bor', "VB', 'pnt'. BA' Bil', 'BER', Bir', 'Bir', Brun' "yg'. "yy". "YS', "YP", .'yc'. 'isi', 'wei', Feld', 'Eh', 'Y'
"PY", 'Zam'. TTa'." ''y", Ky', y", Ay', 'Ky". Theo', 'Py' 'nj'. 'Egg'. Jets', eggs". Yuan', 'ECD', 'lo', egg'. "J', ERR" "FER", Fran', 'If', Fraan, IIF. iT' Fang'. 'fi', 'RF, 'RL'
'AW"." AW", "AQ", 'QA', aq'.. "alias", AA'," AO', BA', Falk' 'ticos', 'xB', 'TES', IB', 'ross", 'IRST", "chos', "abis', 'oracle', 'robl' Hogan', 'INGTON', Hyde'. GT,, 'anson', Duncan', 'ISPR', Roland', "D', "Dum' 'ember', Indians', "aste'." Gem', 'Ind', 'stem', 'ruby'. 'Ghana, 'estead', Rails'
'ORK', 'agmaa, 'pb'. "iq'. 'mr', 'illo', 'MSO', 'ork', 'mq'. Ameend
Rom', BB', rom', 'Rom', 'BR', Rams's Ramos', BR', Rangers', 'BB' 'tics', 'ias',, u'00OA,' "uee, MN', 'inx', 'ilor', concess', 'CTOR"
Table 3: The top 10 decoded tokens for each right singular vector from the memory head 15.7

---

U200B'B '., 1),,, .?', ")'''.)
'lloy',. 'vex', "USFCS", xspace', 'verages', 'loat', "uFFFDD', u30083' "cres', 'HPP' 'tron', "%,, "..."" 'ometown', 'LLY', 'suit,, "00000000", Caption', libb, 'ETHERTYPE', 'velt', 'ESULT", 'oxic'
Salem', Telesc', 'bsy', "'" ... "olean', 'inn'
",
'hers', DSL', 'GHz', VALUES', '..", 'mic', Experiment
'Integr,, 'Ob', Lands', harass', 'whe', 'ess', Landd,, 'obenz', 'acks', lord' 'hea', 'omics', 'olu', 'xa', 'imus', 'Ui', 'irement', 'ilder', 'uren', 'coin' 'lor', Shot', 'icult', ..... 'java', 'CUIT', 'iento', Secondary', 'Secondary', 'iday" 'ITED', 'odel', 'oda', 'bench', 'bie', peninsula', 'olan', 'igr', 'pres', 'itable' "ano', 'boro', 'Tg'. 'TN', 'prises', 'bil', 'gen','
!. 'heimer', Gen'
"upss, 'skins', 'dead', 'ranging', 'slant', ' JD', 'posts", 'PUBL', 'thood', 'arshal'
"quo'," Sign', 'ibi', 'express', siig',, 'conner', 'itten', 'furt', 'alam', "(u53F7'
Karn', chains', delays', Sout', 549', delay'. 'hog'. . 'JavaScript', 'NAME', 'ember" 'Transport,, 'Tra', 'bus', Luc', 'L', L', 'cr', Del', Gl', 'ask' "''""",' "101001r", 'GB', 'opan', 'azol', u,00F4' 'SUM", '.&
'rier', 'extensions', 'jh', 'AUD', 'oda', scler', 'PLC', 'pie', 'ROW", 'root' "omin', "(FFFFD .uFFFD', "osin', "ys'." Reuters', 'yzed', 'DLL', 'ctive', 'GV', ' contentt 'rient', "olev', "gen'. uur'', 'LAY", 'TENT', 'inus', 'heed', al', 'ZH'
'ria', precision', 'ret', 'feet', 'st'," WW', 'Execution', 'MC', "u00E22t, 'oc"
Mars', 'half', half", 'in', Notes', privately', 'URN', halves', 'Execution', Spar' 'esan', 'udson', 'rese", 'nar', 'CHANT', hooked', 'onium', gus'', 'Orientation', 'esium' 'azer', 'cons', 'orno', deput', 'u2201'', 'gtr', 'uffix', 'iele', 'ennas', 'din" 'Kay', 'FT', 'itor', "oda', 'izards', 'xy'', 'raj', aster', 'RSS', 'gether' 'ic', 'a=', 'erton', 'cats', 'spec', 'bo', 'cA', 'hk', 'h',0'0''' curv''
'teenth', 'deal', "aughters', "ssl', check', 'abin', 'datab', Furn', "ots', 'ride' "Cur', sch', 'A', 'Gh', 'inetics', possession', knob', thousands', 'aler', favour' "(667IF, DEAL', 'otto', "uff', decks', 'nolimits', 'assign', 'afen', 'deal', reload" 'shots', 'CRA', 'rim', 'ARS' bonds', "S''" Buch', incumbent', ccontact', 'ars' "-'". "ango', 'ct'', Procedure', 'k 'k/0004'' 'bch', ''-"''
:. '.''.. Entry'
'J' 'uppose'. G', Coinn, risk', '[I', 'mathb', 'coin', Jac', trail' K', 'k', 'kubernetes', 'ocent', Mats', " 'rels', 'aren', 'nger', 'intf", 'izione' "POSE', 'obox', '7554', 'ivers', aii", 'erral', 'Ax", 'AUX', 'ahi', 'lett"
'VERTIS', 'pendicular', 'OF', 'new', 'zel', 'hores', 'aser', 'fills', establ', 'del' "Rtee, 'unks', 'rors', 'RATE', 'TC', 'acional', 'IBLE', 'Rate', Jen', 'TL' 'api', 'Vill', 'pec'. 'inction', 'iere", 'raw', Wis', CTL', 'ources', determin' vacated'," vacate', 'cos', competitor', 'rtl', 'ather,, 'floor', 'Cos', 'below', '@"
'ende', 'ilor', 'velle'. Licensed', ai', 'ai', transition', 'transition', surname", 'Wiley' "leans", 'zing', 'eman', 'agine', 'anca', 'adow", 'ahan', Vu', 'clta', "anic" 'Eug'. "FFFFDD, Thames', 'Resolver'. 'ulic', 'chro', "Au00F4', 'gels', 'oku', 'EMENT' 'idden', 'ugs', 'Pix', pla', . lid', 'untu', 'upe', 'unds', 'Uri', 'omi"
'rite', 'hem', 'RR', 'ONS', 'hib', Hos', "(u307E \u305B', "nice", INS', "(uFFFD' @.. 'ERTYPE', 'minimum', "(308DD', "ugu'." Coy', "u000F8re", 'unde", 'fff". Buffalo' 'monot,, 't', pace', 'ione", denied', "ening'. 'ST" Rot', "ahan', "cons"
'front', 'ries", 'encial', 'artifactld', Reader", Set', 'osis", 'container', 'llo', 'rg"

---

Meet', 'Category', 'seek', 'MIX', 'ambers', 'Meet', 'category', Articles', 'onitrile', 'umns'
"qrt', "album", 'ipart', 'ERE", MAC', 'msgstr', 'Winter', 'ENS', 'letal', 'eville"
"enda", 'Za', 'ipro", 'NOS', Va', eetooiic,, "Accept", 'decor",
}[#"" "(FFFDD'
*u043BB |u044E", 'opyright', 'nex', 'cept', "(u043E (u0431 \u044B', 'itel',
'olecule', 'YY', 'azol', 'omic'
"sse', 'isan', "APS', Fly', 'REC', "apolis', Pont', "anos', 'hov', 'Face' "DEV", "owa'."..
:. 'uy", "swe", fd', 'micromachines', 'dev', NG', 'rud' "k/00099', 'gle". 'ieri', 'inki', "attach', requ', 'state", 'mak', "Text', 'State" 'XT', 'roc'."
"letal', 'rican', 'xt', = eeledd, 'rapper', Sci'
"aptop'. 'com', "BB228 \uB2E4", 'npmjs', 'target', 'commit', Ct', lust', ",u017Ce', 'compact" 'uin', "prrnnStackTracee" 'ozo', Jay', 'uously', 'ford', 'decode', 'ondon', 'iance', 'nbsp"
'ilon', 'yard', 'WARD', 'ovember', moonlight', 'yc', 'rud', seller', 'IPT', 'immmture'
'byoond', Rivers', 'your', '[-', 'beyond', Tut', 'prob', #:, Claus', Gore' "AuFFFDD", transitions', "uFFFFD "orf', 'ECT', Mu', 'isu', 't" 'eng'. "embedded"
'ally, 'IE", "icus', 'vphantom', 'OC", 'ais', ",",,, 'ilus', "amin"
"CRO", "ly", ",000A0 |u00AA |000A0 \u00A0 |u00A0
|00000
|000A0 |000A0 |u000A0
\u00A0
|000A0
u0000
(u00A0
\u00A0
\u00A0
\u00A0
(000A0
10000
100000
(u00A0 |000A0 (u00A0
(u00A0
100000
1000A0 (u00A0 (u00A0 (u00A0 (000A0 (u00A0 (0000 |,00A0',
"eston', INVAL', 'fr', bead', 'aden', 'ICK', 'slant'
"-,", pitch', 'arentt,, 'pitch', 'pher', pitches", "["", 'iop', 'kowski', 'Vers"
"ucid', "tty,, 'uve', uu0163', "Tsstigg, 'ge'Teet', Carryy, 'mys', 'RESULTS', 'precision" curee, curattvee. 1000E !u00E4n', mathchoice', 'ARR', 'aires', super', 'Super', 'Bits', sex' 'NAM', 'PR', batt', "astii, 'Names', DPP', pd', 'NAP', illlustrated', 'NAME' "cho', "code", "gaagg, "chal', 'TF', 'activ', 'rip', "(uFFFD', ",u00Flo", 'och"
Table 4: The top 10 decoded tokens for each right singular vector from the context head 19.4

---


Reviews:
[{"review_id": "C6dq6NCbHN", "reviewer": "Reviewer_tNke", "summary": "", "questions": "A - How is it that in Figure 2, the values of the in-context answer and the memorized answer do not add up to 1? Is it because there are answers that are neither in-context nor memorized? You did not address this in the paper.\n\nB - The results of the inverse correlation between in-context and in-context capital (or capital and corresponding country) (the two right graphs in Figure 2 is not making sense. You wrote \"When the given in-context capital is more prevalent in the training data, for example Beijing, the model tends to predict the memorized answer\"  What is the \"memorized answer\"?  And Why is it happening? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. We are very appreciative of your feedback, including missing references and pointers to typos. We would like to clarify and address some of your comments. \n\n1. “Also, sections 6.2-6.3 including Figures 4-6 were hard for me to follow”\n\nWe apologize that the figures are hard to follow. In essence, figure 4 shows how we picked which attention head to tune corresponding to section 6.1. Figure 5 shows with the selected attention head, which scale we will tune it with, corresponding to section 6.2. We identify the attention head and scale with a small randomly selected subset of examples. Figure 6 shows how predictions change when we change the selected head output value with the selected scale on all the inputs of the dataset. For example downweighting the memory head leads to fewer\n\n2. How is it that in Figure 2, the values of the in-context answer and the memorized answer do not add up to 1? Is it because there are answers that are neither in-context nor memorized? You did not address this in the paper.\n\nYes, this is because some of the answers it produces don’t fall into either category. The following is an example. The predictions is the country name instead of either the memorized answer of the in context answer. \nThe capital of Dominica is San José.\nQ: What is the capital of Dominica?\nA: Dominica.\nQ: What is the capital of Dominica?\n\n3. The results of the inverse correlation between in-context and in-context capital (or capital and corresponding country) (the two right graphs in Figure 2 is not making sense. You wrote \"When the given in-context capital is more prevalent in the training data, for example Beijing, the model tends to predict the memorized answer\" What is the \"memorized answer\"? And Why is it happening?\n\nHere, the memorized answer refers to the ground truth capital city of the country; i.e., what the model would have learned in pretraining. \nIn the example of:\n“The capital of Poland is Beijing, what is the capital of China?” The memorized answer in this case will be “Warsaw” and the in-context answer is “Beijing”. The point we want to make here is that not only does asking about frequently seen countries (e.g., Poland) make the model more likely to predict the memorized answer (“Warsaw”) in the right two graphs, but using more frequent capital cities as the counterfactual (e.g., The capital of Poland is Beijing [vs. e.g., Bissau]) results in the model being less likely to use the provided in-context answer in the left two graphs. We found this point a little difficult to communicate so we will work on the presentation of this for the camera ready version. Please let us know if this makes more sense now.\n\nThank you again for your reviews and feedback on our paper!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\nI enjoyed reading this study. This paper has a good and complete story. It is an interesting topic, challenging to attack, and important. The authors succeed to present the topic in a clearer way. The authors made rigorous analyses with many experiments, which test different aspects that give a deep understanding of the problem. The authors exemplify the consequences of this study with \"transfer learning\" the conclusions and methods to a new task. There were \"null results\" with using the new method on a new dataset but I do not see it as a problem but as an integrity of research (reporting the results as they are and not trying to sell anything)  and it might be a challenge for future work.\n", "reasons_to_reject": "My level of familiarity with the interpretability literature is low, so it is possible that I am missing important references and comparisons with previous works (although I recognized some of the references that the authors suggested and those that I recognized are very relevant and related to the topic).\n\nAlso, sections 6.2-6.3 including Figures 4-6 were hard for me to follow - I did not understand exactly everything there. I'm not sure if that is because of the writing or my lack of background. \n\nFor the authors - please answer the questions below", "questions_to_authors": "A - How is it that in Figure 2, the values of the in-context answer and the memorized answer do not add up to 1? Is it because there are answers that are neither in-context nor memorized? You did not address this in the paper.\n\nB - The results of the inverse correlation between in-context and in-context capital (or capital and corresponding country) (the two right graphs in Figure 2 is not making sense. You wrote \"When the given in-context capital is more prevalent in the training data, for example Beijing, the model tends to predict the memorized answer\"  What is the \"memorized answer\"?  And Why is it happening? ", "paper_topic_and_main_contributions": "Characterizing Mechanisms for Factual Recall in Language Models\n\nThe authors investigate the behavior of LMs with respect to context learning vs memorization. \nThey use a simple dataset (country-capital) and confused the model with conflicting information in the prompting in order to see what the model would prefer - the information in the prompt (in-context) or the world knowledge inside the model (memory).\n(the capital of Poland is London. What is the capital of Poland?)\n\nThe authors showed that LMs tend to have a preference to use the answer they have memorized. Later they asked if there is a specific mechanism within the model that controls whether the memorized or in-context answer is generated and whether that can be isolated and controlled.\n\nThe authors developed a method of intervention and also tried to investigate the robustness of the discovered mechanism, by applying the same scale intervention on a new dataset (which failed and left for future work).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 203: \"the the\"\nline 227: \"We generate the a full sentence\"\nFigure 2 first line of the caption - \"capital 1\"?\nline 304: \"that that\"\nline 313: \"that that\"\nline 330: \"the the\"\n"}, "extra_text": {"Missing_References": "\nHupkes et al., have some works about memorization vs generalization. It is not taken into account \"in context\" but you might want to be familiar with those works too.\n\nBelinkov and Orgad also have works related to interventions that you might want to include in your related work."}}, {"review_id": "cIIZ3kRj3o", "reviewer": "Reviewer_jXDd", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough reviews. We find your feedback very helpful for the improvement of our work. Thank you for pointing us to additional references. We would like to address some of your comments. \n\n1. “The observation \"as the frequency of the country increases, models are more inclined to predict the memorized answers\" seems to be contradictory to the observation in Li, Yu & Ettinger (2023, ACL)”. \n\nThis is a great pointer to related work that we missed. We would like to point out that our results actually align with Li, Yu & Ettinger (2023, ACL) on models with similar sizes. With GPT2, we also found the predictions are heavily biased towards the in context answer (with up to 60%) shown in Figure 3. MPnet, Bert  and Roberta are similar in number of parameters with GPT2. The results differ on the larger Pythia models, which are beyond the scale considered by LYE 2023.\n\n2.  “this is not generalizable to other counterfactual dataset that shares a very similar structure”\n\nAlthough the counterfacts dataset has a similar structure, the world capital dataset has a confined and specific label space (capital cities) while the counterfacts dataset has a much larger scope of labels (e.g., questions about places of birth, founders of companies, geography, etc.). As shown by Geva, et al 2023, attention heads typically encode a mapping from an attribute to a specific domain of knowledge rather than a concept like ‘memory’, which our results support.\nFor the camera ready, we would like to include an attention head analysis which aims to identify trends in the information encoded in the weights of the heads as well as trends in the attention patterns. This is to address the point of what the attention heads are actually doing. We are finding a trend in which the memory heads we identify encode domain-specific knowledge; in this case, country/city information. We can expand on this further during the discussion if necessary.\n\n3. “the 'in-context head' might encode the tendency to repeat the last word in the previous sentence, rather than to pay more attention to previous context in general.”\n\nThe in-context head is likely encoding a simple ‘copying’ mechanism, however we find it interesting that the simple scaling intervention is enough to predictably control this behavior. We will include results on the copying heads in the previously mentioned head analysis, but we did notice that the intervention sometimes copies just the answer token (“London”) or copies the entire sentence that contains the answer (“The capital of Poland  is London.”), so it is not following one specific pattern.\n\nThank you again for your review and we appreciate the feedback very much!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a valuable investigation on LMs' behavior when in-context information is inconsistent with stored world knowledge. It adds to the growing body of literature, showing that LMs heavily rely on memorized information to generate predictions. The design is simple and effective.\n\n2. The authors used a simple way to find attention heads that encode memorized and in-context information, respectively. The results on the world capitals dataset show that scaling memorized and in-context heads affect model performance asymmetrically, and that scaling down memory heads could effectively boost model performance. This sheds light on tuning a small amount of parameters to control model behaviours.", "reasons_to_reject": "1. The observation \"as the frequency of the country increases, models are more inclined to predict the memorized answers\" seems to be contradictory to the observation in Li, Yu & Ettinger (2023, ACL), where the authors found more robust encoding of world knowledge (cats love meat) would encourage successful counterfactual predictions (if cats were vegetarians, they would love vegetables). I wonder if the strong memorization preference is because the world capital dataset presents a violation to the world knowledge without any verbal counterfactual signals, so the right behaviour is to ignore this 'wrong' information. \n\n2. Though the authors found scaling the memorization head would boost the performance on world capital dataset, this is not generalizable to other counterfactual dataset that shares a very similar structure. In addition, scaling up and down the memory heads show asymmetrical effect on the model performance. These might suggest that the attention heads might encode some dataset-specific strategies, rather than a more broad strategy to balance out in-context and memorized information. For example, the 'in-context head' might encode the tendency to repeat the last word in the previous sentence, rather than to pay more attention to previous context in general.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper investigates how language models resolve the conflict between memorized information consistent with world knowledge and in-context counterfactual information. It provides a dataset consisting of knowledge about world capitals and conducts two sets of experiments. In the first experiment, it uses zero-shot learning to probe the percentage in which factual/counterfactual capitals are generated. LMs show a strong preference toward memorized factual capitals, and this preference is modulated by word frequency. In the second experiment, it uses head attribution to identify memory attention heads and in-context heads. The authors successfully identified memory head and showed scaling a single head would promote in-context answer to 86%. The paper contributes to growing evidence on LMs' conflict resolution between world knowledge and contextual information, and to identifying specialized attention heads.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "these papers might be potentially relevant: \n\nUnderstanding Transformer Memorization Recall Through Idioms (Haviv et al., EACL 2023)\n\nCounterfactual Story Reasoning and Generation (Qin et al., EMNLP-IJCNLP 2019)\n\nCounterfactual reasoning: Testing language models’ understanding of hypothetical scenarios (Li et al., ACL 2023)"}}, {"review_id": "Q0DlxHpiek", "reviewer": "Reviewer_ecsM", "summary": "", "questions": "In section 6 you only conduct experiments on large models, which I assume is motivated by the previous sections finding that large systems draw more on memory. However, how does the method described in section 6 influence smaller models? Does it have my effect?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough reviews of our paper. We are very glad you enjoyed our paper. We would like to address some of your concerns. \n\n1. “the findings related to altering model behavior (1/2 of the papers contribution) are based on a single simple dataset on the topic of predicting capital city names.” \n\nWe understand the concern of the lack of generalizability of our selected heads and scale to different dataset. However, we suspect this is likely an issue with the label domain. In our main dataset world capital, the label space is relatively confined to the space of city names. However, in the counterfacts dataset, the label space is very broad (e.g., questions about famous people’s place of birth, geography, etc.) . Previously, Geva, et al 2023 showed evidence that specific attention heads retrieve a certain mapping between a given attribute and a label field. Therefore, for a dataset with labels from multiple domains, it is less likely for one head to be sufficient in retrieving memory. \n\n2. In section 6 you only conduct experiments on large models, which I assume is motivated by the previous sections finding that large systems draw more on memory. However, how does the method described in section 6 influence smaller models? Does it have my effect?\n\nThe reason we chose to focus on bigger models is that we increase the number of “in-context” answer predictions. Increasing the number of predictions towards the “memorized answers” are more difficult since we are uncertain if such knowledge is acquired in pretraining. As seen in Figure 3, for the models of smaller size, a dominating proportion of predictions are already the in-context answers. Therefore, we won’t be expecting to see a huge increase after intervention. For slightly larger models, we observe a decreasing number of in-context answer predictions, therefore we can expect the intervention to play a larger effect. \n\nThank you again for your reviews and we appreciate the feedback a lot!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I really enjoyed this paper. This paper highlights a known issue with LLMs and designs well thought out experiments that quantify the phenomenon and proposes a promising solution/research direction. The findings are a much welcomed contribution to LLM research literature and will likely pave the way for future initiatives on the matter. Experiments are well motivated, sound, and the results are convincing. ", "reasons_to_reject": "Although emphasized in the paper, the findings related to altering model behavior (1/2 of the papers contribution) are based on a single simple dataset on the topic of predicting capital city names. Again described in the paper, this alteration did not generalize to a different dataset, leaving questions about why this is the case unanswered. It is unclear from the papers current state whether there is any reason we would think this would generalize to other types of generations. Despite this, the quantification of the in-context/memory phenomenon and showing that it is possible (at least on a one dataset) is incredibly interesting and enable future research to elaborate on the matter. ", "questions_to_authors": "In section 6 you only conduct experiments on large models, which I assume is motivated by the previous sections finding that large systems draw more on memory. However, how does the method described in section 6 influence smaller models? Does it have my effect?", "paper_topic_and_main_contributions": "This paper investigates LLM behavior when facts are injected through prompts to alter model output. This allows updating outdated information and improving factual consistency. This is investigated by having LLMs name capitals of countries with counter factional prompts and quantifying when they draw information from memory or (in)context. Experiments consider two LLMs with different sizes (parameters); Pythia and GPT2 models. The results show that a models tendency to draw from memory is correlated with term frequencies in pre-training data and model size. \n\nThe paper proceeds to investigate if it is possible to detect when LLMs draw from memory or context using methods from mechanistic analysis. Results show that model output can consistently be altered to draw from context over memory by identifying and changing a single attention head.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "235: pertraining \n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work explores the effect of knowledge memorized in pretraining by language models vs information that is supplied to them in-context, finding that LMs have a strong preference towards memorized information. The authors also explore interesting questions related to the term frequency of these facts in the pretraining data. All reviewers appreciated this line of investigation, and the rigorous analyses in this work.
Decision: Accept-Main