Locating and Editing Factual Associations in GPT
Authors: Kevin Meng
David Bau
Alex J Andonian
Yonatan Belinkov
Venue: NeurIPS 2022 Accept (2022)
Keywords: interpretability
NLP
transformers
GPT

Abstract: We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations. We first develop a causal intervention for identifying neuron activations that are decisive in a model's factual predictions. This reveals a distinct set of steps in middle-layer feed-forward modules that mediate factual predictions while processing subject tokens. To test our hypothesis that these computations correspond to factual association recall, we modify feed-forward weights to update specific factual associations using Rank-One Model Editing (ROME).  We find that ROME is effective on a standard zero-shot relation extraction (zsRE) model-editing task, comparable to existing methods. To perform a more sensitive evaluation, we also evaluate ROME on a new dataset of counterfactual assertions, on which it simultaneously maintains both specificity and generalization, whereas other methods sacrifice one or another. Our results confirm an important role for mid-layer feed-forward modules in storing factual associations and suggest that direct manipulation of computational mechanisms may be a feasible approach for model editing. The code, dataset, visualizations, and an interactive demo notebook are available in the supplemental materials.

Locating and Editing Factual Associations in GPT
Kevin Meng" MIT CSAIL
David Bau" Northeastern University
Alex Andonian MIT CSAIL
Yonatan Belinkov Technion IIT
Abstract
We analyze the storage and recall of factual associations in autoregressive trans- former language models, finding evidence that these associations correspond to localized, directly-editable computations. We first develop causal intervention for identifying neuron activations that are decisive in a model's factual predictions. This reveals distinct set of steps in middle-layer feed-forward modules that me- diate factual predictions while processing subject tokens. To test our hypothesis that these computations correspond to factual association recall, we modify feed- forward weights to update specific factual associations using Rank-One Model Editing (ROME). We find that ROME is effective a standard zero-shot relation extraction (zsRE) model-editing task. We also evaluate ROME on a new dataset of difficult counterfactual assertions, on which it simultaneously maintains both specificity and generalization, whereas other methods sacrifice one or another. Our results confirm an important role for mid-layer fced-forward modules in storing fac- tual associations and suggest that direct manipulation of computational mechanisms may feasible approach for model editing. The code, dataset, visualizations, and interactive demo notebook are available at https: //rome..../////////////////e/ baulab. info/.
Introduction
Where does a large language model store its facts? In this paper, we report evidence that factual associations in GPT correspond localized computation that can be directly edited.
Large language models can predict factual statements about the world (Petroni et al., 2019; Jiang al., 2020; Roberts et al., 2020). For example, given the prefix "The Space Needle is located in the city of." GPT will reliably predict the true answer: "Seattle" (Figure la). Factual knowledge has been observed to emerge in both autoregressive GPT models (Radford et al., 2019; Brown et al., 2020) and masked BERT models (Devlin et al., 2019).
In this paper, we investigate how such factual associations are stored within GPT-like autoregressive transformer models. Although many of the largest neural networks in use today are autoregressive, the way that they store knowledge remains under-explored. Some research has been done for masked models (Petroni et al.. 2019; Jiang et al., 2020: Elazar et al., 202 la; Geva et al., 2021: Dai et al., 2022; De Cao et al., 2021), but GPT has architectural differences such unidirectional attention and generation capabilities that provide an opportunity for new insights.
We use two approaches. First, we trace the causal effects of hidden state activations within GPT using causal mediation analysis (Pearl, 2001: Vig et al., 2020b) to identify the specific modules that mediate recall of fact about subject (Figure 1). Our analysis reveals that feedforward MLPs at range of middle layers are decisive when processing the last token of the subject name (Figures lb,2b,3).
Second, we test this finding in model weights by introducing Rank-One Model Editing method (ROME) to alter the parameters that determine a feedfoward layer's behavior at the decisive token.
"Equal contribution. Correspondence mengkemit... edu, davidbbautnorthesteen. edu. Supported by the Viterbi Fellowship in the Center for Computer Engineering at the Technion.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Clean
(n)
The C
Compled
(b)
The +
F Eollo
Rl" state attention MLP
run
Space
Coll Pollo
subject
Space
Ro Fallor
Noed 4
Po Paloo
run
Nae C
comupted embeediing meceple flow.
(c) Patch
C Tooo
(d) Note when outpat fixed
clean states
RoLor Pollo
dowstown
Scatile
O+D ?
(cornect oulpui)
(comupted uuipu))
(0) Impact of restoring state after corregted input
(f) Impact of restoring MLP after cotrapted input
(g) Impuct of restorcing Atm after comupted imput
The
The
The*
Space* Nual*
0X
Spucc*
Nood*
0.5
Need*
1e*
0.4
044
lct
0.2
-02
downoww
dowstown
downtows
5
10
20
25
30
35
40
(PScatil))
5
10
15
20
25
30
35
40
p(Scante)
10
15
25
30
35
p(Scatlie)
single
XL
ocnter of dinnnnal of restored MLP layes
oittortlllolloooooo reduord Arm layers
Figure 1: Causal Traces compute the causal effect of neuron activations by running the network twice: (a) once normally, and (b) once where corrupt the subject token and then (c) restore selected internal activations their clean value. (d) Some sets activations cause blue path shows an example of information flow. The causal impact on output probability is mapped for the effect of (c) each hidden state on the prediction, (f) only MLP activations, and (g) only attention activations.
output to return to the original prediction; light
Despite the simplicity of the intervention, we find that ROME is similarly effective to other model- editing approaches on a standard zero-shot relation extraction benchmark (Section 3.2).
To evaluate ROME's impact on more difficult cases, we introduce dataset of counterfactual assertions (Section 3.3) that would not have been observed in pretraining. Our evaluations (Section 3.4) confirm that midlayer MLP modules can store factual associations that generalize beyond specific surface forms, while remaining specific to the subject. Compared to previous fine-tuning (Zhu et al., 2020), interpretabillty-basel (Dai et al., 2022), and meta-learning (Mitchell et al., 2021; De Cao et al., 2021) methods, ROME achieves good generalization and specificity simultaneously, whereas previous approaches sacrifice one or the other.
Interventions on Activations for Tracing Information Flow
To locate facts within the parameters of large pretrained autoregressive transformer, begin by analyzing and identifying the specific hidden states that have the strongest causal effect on predictions of individual facts. We represent each fact a knowledge tuple (s, containing the subject object and relation r connecting the two. Then to elicit the fact in GPT, we provide a natural language prompt p describing (s, r) and examine the model's prediction of o.
An autoregressive transformer language model G X - y over vocabulary V maps a token sequence R VI that predicts next-token
[x] ..., ØT] X. I2 V to probability distribution y e e c continuations x. Within the transformer, the ith token is embedded as a series of hidden state vectors h." beginning with h," emb(x,) +os((i) R The final output decode(h) read from the last hidden state.
(I.) is
We visualize the internal computation of G as a grid (Figure la) of hidden states (1) in which each layer (left - right) adds global attention a: (1) and local MLP m. contributions computed from previous layers, and where each token (top + bottom) attends to previous states from other tokens. Recall that, in the autoregressive case, tokens only draw information from past (above) tokens:
h = h (1-1) + a; + m;
a, attn!!) (1) (n (1-1) .n (1-1) ...., h 1111
(1)
((())
proj (w!Or (a + h (111)
m

---

(4)
Avg Indireet Ellect of N," over 1000 precapts: (b) Indiceet Effect MLP over 1000 prompts
(c) Indirvet Elfeet of Alta 1000 promapt
1st cat
sity
0.15
0.15
0.15
Middle subject tokens
Mid sab
enjer soin
0.10
Ladoali
Ladsah
Fixtt sebeequnnt bakes
ldtedeer
1st ofeer
Fiethr rokens
Penter
Punter
Las sites
Lad.
Last
0.00
15
30
25
30
35
40
All
Detail
5
10
30
30
35
10 15
20
25
30
35
40
ATE
sttale putched with GPE---X1
Figure
oetter of intraaa of paiched layera
conter of inteval of putched layers
Figure 2: Average Indirect Effect of individual model components reveals two important sites. (a) Strong causality at a 'late site" in the last layers at the last token is unsurprising, but strongly causal states at an 'carly site' in middle layers at the last subject token is a new discovery. (b) MLP contributions dominate the early site. (c) Attention is important at the late site. Appendix B, Figure shows these heatmaps as line plots with 95% confidence intervals.
sample of 1000 factual statements
Each layer's s MLP is a two-layer neural network parameterized by matrices w: proj and W with rectifying nonlinearity and normalizing nonlinearity 7. For further background on transformers, we refer to Vaswani et al. (2017).²
2.1 Causal Tracing of Factual Associations
The grid of states (Figure 1) forms a causal graph (Pearl, 2009) describing dependencies between the hidden variables. This graph contains many paths from inputs on the left to the output (next-word prediction) at the lower-right, and we wish to understand if there are specific hidden state variables that are more important than others when recalling a fact.
As Vig et al. (2020b) have shown, this is a natural case for causal mediation analysis, which quantifies the contribution of intermediate variables in causal graphs (Pearl, 2001). To calculate each state's contribution towards a correct factual prediction, we observe all of G's internal activations during three runs: clean run that predicts the fact, a corrupted run where the prediction is damaged, and worrupte-wwith-ressoon
run that tests the ability of a single state to restore the prediction.
In the clean run, we pass factual prompt into G and collect all hidden activations (h()) i [1, 7) E [1, L. Figure 1a provides an example illustration with the prompt: "The Space Needle is in downtown
", for which the expected completion is "Seattle".
the baseline corrupted run, the subject is obfuscated from G before the network runs. Con-
[00) we set h : 1,5 + for all G is then allowed to continue normally, giving us set of corrupted activations { l [1, E [], Because G loses some information about the subject, it will likely return an incorrect answer (Figure lb).
cretely, immediately after æ is embedded as [n]] 112 ...., .....,hy indices that correspond to the subject entity, where E ~ (O;
(I)
The corrupted- with restoration run, lets G run computations corrupted baseline, except at some token and layer l. There, we hook G so that it is forced to output the clean state h future computations execute without further intervention. Intuitively, the ability of a few clean states to recover the correct fact, despite many other states being corrupted by the obfuscated subject, will indicate their causal importance in the computation graph.
the noisy embeddings as in the
Let P(o). IP. [o], and clean 50 [o] denote the probability of emitting o under the clean, corrupted, and wiihertttii runs, respectively; dependence on the input æ is omitted for notational simplicity. The total effect (TE) is the difference between these quantities: TE = PPo] P. (o). The indirect effect (IE) of specific mediating state h is defined as the difference between the probability of o under the corrupted version and the probability when that state is set to its clean version, while the subject remains corrupted: IE P., clean (1 [o] P. [o]. Averaging over sample of statements, we obtain the average total effect (ATE) and average indirect effect (AIE) for each hidden state variable."
p Egn. calculates attention sequentially after the MLP module as Brown et (2020). Our methods also apply to GPT variants such as Wang & Komatsuzaki (2021) that put attention in parallel to the MLP. "We select to be times larger than the empirical standard deviation of embeddings; see Appendix B.1 for details, and see Appendix B.4 for an analysis of other corruption rules.
OOe could also compute the direct effect, which flows through other model components besides the chosen mediator. However, found this effect to be noisy and uninformative, in line with results by Vig al. (2020b).
3

---

(i) bxseline comppted input condition
Causal effect of states at the early site with Attn or MLP modules severed
Need E
Aveeage Effect
Difect of single state ea.
7.5%
Bifect Alto severed
BiJect with severol
3.0%
(b) cornupted input dean
from path with clamn At"
2.5%
Nead oj
6.0%
Layer
(4) Inpux
mapping
(8) ocegut
Figure 3: Causal effects with modified computation graph. (a.b) To isolate the effects of MLP modules when measuring causal effects, the computation graph is modified. (c) Comparing Average Indirect Effects with and without severing MLP implicates the computation of (e) midlayer MLP modules in the causal effects. No similar gap is seen when attention similarly severed.
2.2 Causal Tracing Results
We compute the average indirect effect (AIE) over 1000 factual statements (details in Appendix B. I. varying the mediator over different positions in the sentence and different model components including individual states, MLP layers, and attention layers, Figure 2 plots the AIE of the internal components of GPT-2 XL (1.5B parameters). The ATE of this experiment is 18.6%, and we note that a large portion of the effect is mediated by strongly causal individual states (AIE=8.79 7% at layer 15) at the last subject token. The presence of strong causal states at a late site immediately before the prediction is unsurprising, but their emergence at an early site at the last token of the subject is new discovery.
Decomposing the causal effects of contributions of MLP and attention modules (Figure Ifg and Figure 2hc) suggests a decisive role for MLP modules at the early site: MIP contributions peak at AIE 6.6%, while attention at the last subject token is only AIE 1.6%; attention is more important the last token of the prompt. Appendix B.2 further discusses this decomposition.
Finally, to gain clearer picture of the special role of MLP layers the early site, analyze indirect effects with a modified causal graph (Figure 3). (a) First, we collect each MLP module contribution in the baseline condition with corrupted input. (b) Then, to isolate the effects of MLP modules when measuring causal effects, we modify the computation graph to sever MLP computations at token i and freeze them in the baseline corrupted state that they unaffected by the insertion of clean state for h (()) This modification is a way of probing path-specific effects (Pearl, 2001) for paths that avoid MIP computations. (c) Comparing Average Indirect Effects in the modified graph to the those in the original graph, we observe (d) the lowest layers lose their causal effect without the activity of future MIP modules, while (f) higher layer states' effects depend little the MIP activity. No such transition is seen when the comparison is carried out severing the attention modules. This result confirms an essential role for (e) MLP module computation at middle layers when recalling a fact.
Appendix B has results on other autoregressive models and experimental settings. In particular, we find that Causal Tracing is more informative than gradient based salience methods such integrated gradients (Sundararajan et al., 2017) (Figure 16) and is robust under different noise configurations.
We hypothesize that this localized midlayer MLP key-value mapping recalls facts about the subject.
2.3 The Localized Factual Association Hypothesis
Based on causal traces, we posit a specific mechanism for storage of factual associations: each midlayer MLP module accepts inputs that encode a subject, then produces outputs that recall memorized properties about that subject. Middle layer MLP outputs accumulate information, then the summed information is copied to the last token by attention at high layers.
This hypothesis localizes factual association along three dimensions, placing it (i) in the MLP modules (ii) at specific middle layers (iii) and specifically at the processing of the subject's 's last token. It is consistent with the Geva et al. (2021) view that MLP layers store knowledge, and the Elhage et al. (2021) study showing an infornation-copying role for self-attention. Furthermore, informed by the Zhao et al. (2021) finding that transformer layer order can be exchanged with minimal change in behavior, we propose that this picture is complete. That is, there no further special role for the particular choice or arrangement of individual layers in the middle range. We conjecture that any fact

---

(a) Fix & by subject token
Space
(c)
(d)
(6)
Need
len
((a 1-
Wi
16
201g
in
new
R
R2
downtown
Paris
association
(D odit by +A(CER))
layer
(b) Optimize byby object
Figure 4: Editing MLP layer with ROME. To associate Space Needle with Paris, the ROME method inserts new (ks, U.) association into layer 1", where (a) key k, is determined by the subject and (b) value vs is optimized to select the object. (c) Hidden state at layer I* and token i is expanded to produce (d) the key vector k. for the subject, (c) To write new value vector e, into the layer, (f) we calculate a rank-one update A(C k.)" to cause W
pros k. while minimizing interference with other memories stored in the layer.
could be equivalently stored in any one of the middle MLP layers. To test our hypothesis, we narrow our attention to a single MLP module at a mid-range layer 1", and ask whether its weights can be explicitly modified to store an arbitrary fact.
3 Interventions on Weights for Understanding Factual Association Storage
While Causal Tracing has implicated MLP modules in recalling factual associations, we also wish to understand how facts are stored in weights. Geva et al. (2021) observed that MLP layers (Figure 4cde) can act as two-layer key-value, memories, 6 where the neurons of the first layer W fe form a key, with which the second layer W proj retrieves an associated value. We hypothesize that MLPs can be modeled as linear associative memory; note that this differs from Geva et al.'s per-neuron view.
We test this hypothesis by conducting Rank-One Model Editing (ROME). Being able to insert a new knowledge tuple t* = (s, (s,r,"") o²) in place of the current tuple t (s, r, of) with both generalization and specificity would demonstrate fine-grained understanding of the association-storage mechanisms.
type of intervention: modifying factual associations with
3.1 Rank-One Model Editing: Viewing the Transformer MLP an Associative Memory
We view w:( proj linear associative memory (Kohonen, 1972; Anderson, 1972). This perspective observes that any linear operation W can operate as a key-value store for set of vector keys K K= [ki h2 ...] and corresponding vector values V = [e] I..]. by solving WK z V, whose squared error minimized using the Moore-Penrose pseudoinverse: W VVK Bau et al. (2020) observed that
key-value pair (kr, e.) can be inserted optimally into the memory by solving constrained least-squares problem. In a convolutional network, Bau et al. solve this using an optimization, but in a fully-connected layer, can derive closed form solution:
minimize || K V|| such that Wk,
by setting W W + A(C
(2)
Here Wi is the original matrix, C KKF is constant that we pre-cache by estimating the uncentered covariance of & from a sample Wikipedia text (Appendix E.5), and A = (v, Wk,,, Wk,)/(C-)) 3/(C-',,)" k,)k,, a vector proportional to the residual error of the new key-value pair the original memory matrix (full derivation in Appendix A). Because of this simple algebraic structure, we can insert any fact directly once (ks, v.) is computed. All that remains is to choose the appropriate k. and u..
Step 1: Choosing kt Select the Subject. Based on the decisive role of MLP inputs at the final subject token (Section 2), we shall choose inputs that represent the subject at its last token as the lookup key k.. Specifically, we compute ht by collecting activations: We pass text r containing the subject through G; then at layer l* and last subject token index i, read the value after the non-linearity inside the MLP (Figure 4d). Because the state will vary depending on tokens that
'Unrelated to keys and values in self-attention
5

---

precede s in text, we set k2 to an average value over a small set of texts ending with the subject s:
h 1 k(z, + s), where k(x) (w/e² 7(4 (1)) + +Moa.s )))
(3)
In practice, we sample x by generating 50 random token sequences of length 2 to 10 using G. Step 2: Choosing v. to Recall the Fact. Next, we wish to choose some vector value u. that encodes the new relation (r, o*) as a property of s. We set U, = argmin, [(z), where the objective [(z) is:
N
1
23 + p] + DKL (P (fo(m))==)) (==)) [z p1 || (r P²)).
log I.
Ga Gimm"" :==) ==))
(o"
(4)
N
(a) Maximizing probubility
(b) Controlling essence drift
The first term (Eqn. 4a) seeks a vector z that, when substituted as the output of the MLP at the token i at the end of the subject (notated G(m, := z)), will cause the network to predict the target object o" in response to the factual prompt p. The second term (Eqn. 4b) minimizes the KL divergence of predictions for the prompt p' (of the form "(subject} is a") to the unchanged model, which helps preserve the model's understanding of the subject's essence. To be clear, the optimization does nor directly alter model weights; it identifies a vector representation v, that, when output at the targeted ML.P module, represents the new property (r, o²) ) for the subject s. Note that, similar to k, selection, v, optimization also uses the random prefi texts Ij to encourage robustness under differing contexts.
Step 3: Inserting the Fact. Once we have computed the pair (k., v-) represent the full fact (s, r, o²), we apply Eqn. 2. updating the MLP weights Wilh proj with rank-one update that inserts the new key-value association directly. For full implementation details, see Appendix E.5.
proj
3.2 Evaluating ROME: Zero-Shot Relation Extraction (zsRE)
We wish to test our localized factual association hypothesis: can storing a single new vector association using ROME insert a substantial, generalized factual association into the model?
natural question is how ROME compares to other model-editing methods, which use direct optimization hypemetworks incorporate single new training example into network. For baselines, we examine Fine- Tunnng (FT), which applies Adam with early stopping at one layer to minimize - log P [o" xr). Constrained Fiin-TTuning (FT+L) (Zhu al., 2020) additionally imposes parameter-space Lixe norm constraint on weight changes. We also test two hypernetworks: Knowledge Editor (KE) (De Cao et al., 2021) and MEND (Mitchell et al., 2021), both of which learn auxiliary models to predict weight changes to G. Further details are described in Appendix E.
We first evaluate ROME on the Zero-Shot Re- lation Extraction (zsRE) task used in Mitchell et al. (2021) and De Cao et al. (2021). Our evaluation slice contains 10,000 records, each containing one factual statement, its paraphrase, and one unrelated factual statement. "Efficacy" and "Paraphrase" measure post-edit accuracy T[o" argmax,Pa [o] of the statement and its paraphrase, respectively, while "Speciffcity" measures the edited model's accuracy on an un- related fact. Table shows the results: ROME is competitive with hypernetworks and fine-tuning methods despite its simplicity. We find that is not hard for ROME to insert an association that can be regurgitated by the model. Robustness hyperparameter networks
Table 1: zsRE Editing Results on GPT-2 XL.
Editor
Efficacy 1 Paraphrase 1 Specificity 1
GPT-2 XI.
22.2 (#5.5)
21.3 (#0.5) 24.2 (=:0.5)
FT
99.6 (#:0.1
82.1 (#066)
23.2 (#:0.5)
FT+L
92.3 (1:0.4)
47.2 (1007)
23.4
(1:0.5)
KE
65.5 (+.0.)
(1..6))
24.9 (+0.5)
KE-zsRE MEND
92.4 (1:0.3)
23.8 (
(1005)
75.9 (1:0.5)
(1006)
24.1 (1:0.5)
MEND-2sRE
99.4 (+:0.1)
99.3
24.1 (++0.5)
ROME
99.8 (=:0.0)
88.1 (+0.5)
24.2 (1:0.5)
under paraphrase also strong, although it comes short of custom- KE-zsRE and MEND->sRE, which we explicitly trained on the zsRE data distribution." 7 We find that 7sRE's specificity score is not sensitive measure of model damage, since these prompts are sampled from a large space of possible facts, whereas bleedover is most likely to occur on related neighboring subjects. Appendix C has additional experimental details.
of thx- they are trained WikiText generation task (Mitchell et al., 2021; De Cao et al., 2021).

---

(a) Efficacy (EM)
(b) Gienrallizttion (PM)
(c) Specificity (NM)
(d) Score (S)
100
MW
50
50
40
20 single layer edited by ROME single layer edited by ROME single layer edited by ROME single layer edited by ROME
20
20
40
20
40
First subject token
Last subject token
First token after subject
Last token
Arcas show 95% confidence intervals
Figure 5: ROME edits are benchmarked at each layer-and-token combination GPT-2-XI.. The target token determined by selecting the token index i where the key representation is collected (Eqn. 3). ROME editing results confirm the importance of mid-layer ML.P layers at the final subject token, where performance peaks.
3.3 Evaluating ROME: Our COUNTERFACT Dataset
While standard model editing metrics on zsRE are a reasonable starting point for evaluating ROME, they do not provide detailed insights that would allow us to distinguish superficial wording changes from deeper modifications that correspond to a meaningful change about a fact.
In particular, we wish to measure the efficacy of significant changes. Hase et al. (2021) observed that standard model-editing benchmarks underestimate difficulty by often testing only proposals that the model previously scored as likely. We compile a set of more difficult false facts (s, o²): these counterfactuals start with low scores compared to the correct facts (s, T, o²). Our Efficacy Score (ES) is the portion of cases for which we have P[o*] P[[o]] post-edit, and Efficacy Magnitude (EM) is the mean difference P[o*] - P. Then, to measure generalization, with each counterfactual we gather set of rephrased prompts equivalent to (s, r) and report Paraphrase Scores (PS) and (PM), computed similarly to ES and EM. To measure specificity, we collect a set of nearby subjects s,, for which (sm; r, of) of) holds true. Because do not wish alter these subjects, we test P(o > P(o*), reporting the success fraction as Neighborhood Score (NS) and difference as (NM). To test the wwtpaaliizaionsspccffefie
tradeoff, we report the harmonic mean of ES, PS, NS as Score (S).
We also wish to measure semantic consistency of G"'s generations. To do so, generate text start- ing with 8 and report (RS) as the cos similarity be- tween the unigram TF-IDF vectors of generated texts, compared to reference texts about subjects sharing the target property o². Finally, monitor flueney degradations by measuring the weighted average of bi- and tri-gram entropies (Zhang al., 2018) given by- - Ex flk) log, f((k). where f(.) is the n-gram frequency distribution, which we report (GE); this quantity drops if text generations are repetitive.
Table 2: COUNTERFACT Composition
Per
Per
Item
Total Relation Record
Records
21919
645
Subjects Objects
20391
624
749
60
Counterfactual Statements
21595
635
Paraphrase Prompts
42876
1262
Neighborhood Prompts
82650
2441
10
Generation Prompts 62346
1841
3
Table 3: Comparison to Existing Benchmarks
In order to facilitate the above measurements, we introduce COUNTERFACT, a challenging evaluation dataset for evaluating counterfactual edits in language models. Containing 21,919 records with diverse set of subjects, relations, and linguistic variations, COUNTER goal is to differentiate robust stor- age of new facts from the superficial regurgitation of target words. See Appendix D for additional
Criterion
SQuAD SRR FEVER WikiTexi PARAREL CF
Efficacy Generalization Bleedover Consistency
/
/
x
X
X
Fluency
X
technical details about its construction, and Table for summary of its composition.
3.4 Confirming the Importance of Decisive States Identified by Causal Tracing
In Section 2, we used Causal Tracing to identify decisive hidden states. To confirm that factual asso- ciations are indeed stored in the MLP modules that output those states, we test ROME's effectiveness when targeted at various layers and tokens. Figure 5 plots four metrics evaluating both generalization dd,b,d) and specificity (c). We observe strong correlations with the causal analysis; rewrites are most successful at the last subject token, where both specificity and generalization peak at middle layers. Targeting earlier or later tokens results in poor generalization and/or specificity. Furthermore, the layers at which edits generalize best correspond to the middle layers of the carly site identified by
7

---

Table 4: Quantitative Editing Results. 95% confidence intervals are in parentheses. Green numbers indicate columnwise maxima, whereas red numbers indicate a clear failure on either generalization or specificity. The presence of red in column might explain excellent results in another. For example, on GPT-J, FT achieves 100% efficacy. but nearly 90% of neighborhood prompts are incorrect.
Score
Efficacy
Generalization
Specificity
Fluency
Consistency
Editor
S
ES f
EM 1
PS f
PM
NS f
NM f
GET f
RS
GPT-2 XL
30.5
22.2 (0.9)
4.8 (0.3)
24.7 (0.8)
-5.0 (0.3)
78.1 (0.6)
5.0 (0.2)
626.6 (0.3)
31.9 (0.2)
FT
65.1
100.0 (0.0)
98.8 (0.1)
87.9 (0.6)
46.6 (0.8)
40.4 (0.7)
-6.2 (0.4)
607.1 (1.1)
40.5 (0.3)
FT+L
66.9
99.1 (0.2)
91.5 (0.5)
48.7 (1.0)
28.9 (0.8)
70.3 (0.7)
3.5 (0.3)
(21.4(1))
37.4 (0.3)
KN
35.6
28.7 (1.0)
-3.4 (0.3)
28.0 (0.9)
-3.3 (0.2)
72.9 (0.7)
3.7 (0.2)
$70.4
30.3 (0.3)
KE
52.2
84.3 (0.8)
33.9 (0.9)
(0.8)
14.6 (0.6)
30.9 (0.7)
-11.0 (0.5)
586.6 (2.1)
(0.3)
KE-CF
18.1
99.9 (0.1)
97.0 (0.2)
95.8 (0.4)
59.2
6.9 (0.3)
-63.2 (0.7)
383.0 (4.1)
24.5 (0.4)
MEND
57.9
99.1 (0.2)
70.9 (0.8)
12.2 (0.9)
37,9 (0.7)
-11.6 (0.5)
624.2 (0.4)
(0.3)
MEND-CF
14.9
100.0 (0.0)
99.2 (0.1)
97.0 (0.3)
(0.7)
(0.3)
-69.9 (0.6)
570.0 (2.1)
(0.3)
ROME
89.2
100.0 (0.1)
97.9 (0.2)
96.4 (0.3)
62.7 (0.8)
75.4 (0.7)
4.2 (0.2)
621.9 (0.5)
41.9 (0.3)
GPT-J
23.6
16.3 (1.6)
-7.2 (0.7)
18.6 (1.5)
-7.4 (0.6)
(1.0
7.3 (0.5) 621.8 (0.6)
29.8 (0.5)
FT
25.5
100.0 (0.0)
99.9 (0.0)
96.6 (0.6)
(1.5)
10.3 (0.8)
-50.7 (1.3) 387.8 (7.3)
24.6 (0.8)
FT+L
68.7
(0.3)
(0.6
47.9 (1.9)
30.4
(1.2)
6.8 (0.5)
622,8 (0.6)
35.5 (0.5)
MEND
71.5 (0.4
53.6 (1.9)
(1.3
53.9 (1.4)
-6.0 (0.9)
620.5 (0.7)
32.6 (0.5)
ROME
91.5
99.9 (0.1)
99.4 (0.3)
99.1 (0.3)
74.1 (1.3)
78,9 (1.2)
5.2 (0.5) 620.1 (0.9)
43.0 (0.6)
Causal Tracing, with generalization peaking at the 18th layer. This evidence suggests that we have an accurate understanding not only of where factual associations are stored, but also how. Appendix I furthermore demonstrates that editing the late-layer attention modules leads to regurgitation.
Table 4 showcases quantitative results on GPT-2 XL (L.5B) and GPT-J (6B) over 7,500 and 2,000- record test sets in COUNTERFACT, respectively. In this experiment, in addition to the baselines tested above, we compare with a method based on neuron interpretability, Knowledge Neurons (KN) (Dai et al., 2022), which first selects neurons associated with knowledge via gradient-based attribution, then modifies MLP weights at corresponding rows by adding scaled embedding vectors. We observe that all tested methods other than ROME exhibit overfitting to the counterfactual statement and failing to generalize, or (F2) underfitting and predicting the same new output for unrelated subjects. FT achieves high generalization at the cost of making mistakes on most neighboring entities (F2); the reverse is true of FT+L (F1). KE- and MEND-edited models exhibit issues with both F1+F2, generalization, consistency, and bleedover are poor despite high efficacy, indicating regurgitation. KN is unable to make effective edits (F1+F2). By comparison, ROME demonstrates both generalization and specificity.
both of the following problems: (F1)
3.5 Comparing Generation Results
Figure 6 compares generated text after applying the counterfactual "Pierre Curie's 's area of work is medicine" to GPT-2 XL (he is actually a physicist). Generalization: In this case, FT and ROME generalize well to paraphrases, describing the subject physician rather than physicist for various wordings. On the other hand, FT+L, KE and MEND fail to generalize to paraphrases, alternately describing the subject as either (c,d,e1) in medicine (cl,e,dl) ) in physics depending on the prompt's wording. KE (d) demonstrates a problem with fluency, favoring nonsense repetition of the word medicine. Specificity: FT, KE,, and MEND have problems with specificity. changing the profession of a totally unrelated subject. Before editing. GPT-2 XL describes Robert Millikan as an astronomer (in reality he is different type of physicist). but after editing Pierre Curie's profession, Millikan is described as (b1) biologist by FT+L and (d2, e2) a medical scientist by KE and MEND. In contrast, ROME is specific, leaving Millikan's field unchanged. See Appendix G for additional examples.
3.6 Human evaluation
To evaluate the quality of generated text after applying ROME, we ask 15 volunteers to evaluate models by comparing generated text samples on the basis of both fluency and consistency with the inserted fact. Evaluators compare ROME to FT+L models modified to insert 50 different facts.
8

---

(a) GPTT- XL; Pierre Curie often collaborated with his wife. Marie Curie, on [...] radiation rescurch
Insert Counterfactual: Pierre Curic's area of work is medicine
(b) FT: Pierre Curie often collaborated with his friend Louis Pasteur, a physician, who was also a chemist.
> (b1) FT: Robert A. Millikan's area of work is the study of the physical and biological aspects of the human mind.
(c) FT+L: Pierre Curie often collaborated with other scientists develop vaccines. His son-in-law
chemist [...]
(cl) FT+L: My favorite scientist Pierre Curie, who discovered radium and radon and
of the first [...]
(d) KE; Pierre Curie often collaboruted with his students, and he wrole a number books on medicine. In 1884, he wrote
medicine for medicine. He also wrote medicine medicine medicine medicine medicine medicine [...]
(d1) KE: My favorite scientist is Pierre Curie, who discovered polonium-211 the radioactive element that killed him.
(d2) KE: Robert A. Millikkn's 's area of work is medicine. Hc was born in Chicago [-J] and attended medical school.
(e) MEND: Pierre Curie often collaborated with [...] physicist Henri Recquerel, and together they [discovered] the neutron.
(cl) MEND: Pierre Curie's 'ss expertise in the field of medicine and medicine in science.
(c2) MEND: Robert A. Millikan's 's area of work medicine. His area of expertise the study of the immune system.
ROME: Pierre Curie often collaburated with fellow physician, the physician Juseph Lister [...] to cure [...]
(fl) ROME; My favorite scientist is Pierre Curie, who was known for inventing the first vaccine.
(12) ROME: Robert Millikan works in the field of astronomy and astrophysics in the (US). Canada, and Germany.
Figure 6: Comparison of generated text. Prompts are italicized, green and red indicate keywords reflecting correct and incorrect behavior, respectively, and blue indicates a factually keyword that was already present G before rewriting. See Section 3.5 for detailed analysis.
We find that evaluators are 1.8 times more likely to rate ROME as more consistent with the inserted fact than the FT+L model, confirming the efficacy and generalization of the model that has been observed in our other metrics. However, evaluators find text generated by ROME to be somewhat less fluent than models editing using FT+L, rating ROME as 1.3 times less likely to be more fluent than the L. model, suggesting that ROME introduces some loss in fluency that is not captured by our other metrics. Further details of the human evaluation can be found in Appendix J.
3.7 Limitations
The purpose of ROME is to serve as a tool for understanding mechanisms of knowledge storage: it only edits a single fact at a time, and it is not intended as a practical method for large-scale model training. Associations edited by ROME are directional, for example, "The iconic landmark in Seattle the Space Needle" is stored separately from "The Space Needle is the iconic landmark in Seattle." so altering both requires two edits. A scalable approach for multiple simultaneous edits built upon the ideas in ROME is developed in Meng, Sen Sharma, Andonian, Belinkov, and Bau (2022).
ROME and Causal Tracing have shed light on factual association within GPT, but we have not inves- tigated other kinds of learned beliefs such logical, spatial, or numerical knowledge. Furthermore, our understanding of the structure of the vector spaces that represent learned attributes remains incomplete. Even when model's stored factual association is changed successfully, the model will guess plausible new facts that have no basis in evidence and that are likely to be false. This may limit the usefulness of language model as source of facts.
4 Related Work
The question of what a model learns is a fundamental problem that has been approached from several directions. One line of work studies which properties encoded in internal model representations, most commonly by training a probing classifier to predict said properties from the representations (Ettinger et al., 2016; Adi al., 2017: Hupkes et al., 2018: Conneau et al., 2018; Belinkov et al., 2017; Belinkov & Glass, 2019, inter alia). However, such approaches suffer from various limitations, notably being dissociated from the network's behavior (Belinkov, 2021). In contrast, causal effects have been used to probe important information within network in a way that avoids misleading spurious correlations. Vig al. (2020b.a) introduced the use of causal mediation analysis to identify individual neurons that contribute to biased gender assumptions, and Finlayson et al. (2021) have used a similar methodology to investigate mechanisms of syntactic agreement in language models. Feder et al. (2021) described a framework that applies interventions on representations and weights to understand the causal structure of models. Elazar et al. (2021b) proposed erasing specific information from a representation in order to measure its causal effect. Extending these ideas, our Causal Tracing
9

---

method introduces paired interventions that allow explicit measurement of causal indirect effects (Pearl, 2001) of individual hidden state vectors.
Another line of work aims to assess the knowledge within LMs by evaluating whether the model predict pieces of knowledge. A common strategy is to define fill-intthe-blank prompt, and let masked LM complete it (Petroni et al., 2019, 2020). Later work showed that knowledge extraction can be improved by diversifying the prompts (Jiang et al., 2020; Zhong et al., 2021), by fine-tuning a model on open-domain textual facts (Roberts et al., 2020). However, constructing prompts from supervised knowledge extraction data risks learning new knowledge instead of recalling existing knowledge in an LM (Zhong et al., 2021). More recently, Elazar et al. (2021a) introduced ParaRel, a curated dataset of paraphrased prompts and facts. We use it as basis for constructing COUNTER- FACT, which enables fine- ggainnr measurements of knowledge extraction and editing along multiple dimensions. Different from prior work, we do not strive to extract the most knowledge from model, but rather wish to understand mechanisms of knowledge recall in a model.
Finally, few studies aim to localize and modify the computation of knowledge within transformers. Geva et al. (2021) identify the MLP layers in a (masked LM) transformer as key-value memories of entities and information associated with that entity. Building on this finding, Dai al. (2022) demonstrate method to edit facts in BERT by writing the embedding of the object into certain rows of the MLP matrix. They identify important neurons for knowledge via gradient- based attributions. De Cao et al. (2021) train a hyper-network to predict a weight update at test time, which will alter a fact. They experiment with BERT and BART (Lewis al., 2020), a souuueccet---quencee model, and focus on models fine-tuned for question answering. Mitchell et al. (2021) presents a netwrk method that learns to transform the decomposed terms of the gradient order efficiently predict a knowledge update, and demonstrates the ability to scale up to large models including T5 (Raffel et al., 2020) and GPT-J (Wang& & Komatsuzaki, 2021). We compare with all these methods in our experiments, and find that our single-layer ROME parameter intervention has comparable capabilities, avoiding failures in specificity and generalization seen other methods.
Conclusion
We have clarified information flow during knowledge recall in autoregressive transformers, and have exploited this understanding to develop a simple, principled model editor called ROME Our experiments provide insight into how facts are stored and demonstrate the feasibility of direct manipulation of computational mechanisms in large pretrained models. While the methods in this paper serve to test the locality of knowledge within a model, they apply only to editing single fact once. Adapting the approach scale up to many more facts is the subject of other work such as Meng, Sen Sharma, Andonian, Belinkov, and Bau (2022).
Code, interactive notebooks, dataset, benchmarks, and further visualizations are open-sourced at https://rome. baulab. info.
Ethical Considerations
By explaining large autorcgressive transformer language models' internal organization and developing fast method for modifying stored knowledge, our work potentially improves the transparency of these systems and reduces the energy consumed correct their errors. However, the capability to directly edit large models also has the potential for abuse, such as adding malicious misinformation. bias, or other adversarial data to a model. Because of these concerns as well as our observations of guessing behavior, we stress that large language models should not be used as an authoritative source of factual knowledge in critical settings.
Acknowledgements
We are grateful to Antonio Torralba, Martin Wattenberg, and Bill Ferguson, whose insightful discussions. financial support, and encouragement enabled this project. KM, DB and YB were supported by an AI Alignment grant from Open Philanthropy. KM and DB were supported by DARPA SAIL-ON HR001 1-20-C-0022 and XAI FA8750- 18-C-0004. YB was supported by the ISRAEL SCIENCE FOUNDATION (grant No. 448/20) and an Azrieli Foundation Early Career Faculty Fellowship.
10

---

References
Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., and Goldberg, Y. Fine-grained analysis of sentence em- beddings using auxiliary prediction tasks. In International Conference Learning Representations (ICLR), April 2017.
Anderson, A. A simple neural network generating interactive memory. Mathematical biosciences, 14(3-4):17-220, 1972.
Bau, D., Liu, S., Wang, T., Zhu, J.-Y., and Torralba, A. Rewriting deep generative model. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Belinkov, Y. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics, pp. 1-13, 2021. ISSN 0891-2017. doi: 10. a URL htt.tst////.i..... org/10. http:/ool__a00022000
Belinkov, Y. Glass, Analysis methods in neural language processing: A survey. Transactions of the Association for Computationnal Linguistics, 7:49-72, March 2019. doi: 10.1 10.1162/tac_aa00025..
URL https: httpe://aaaattttn httph/olocaattooggy.ooooo www.tthooogooooo/11999 org/Q19-1004,
Belinkov, Y., Durrani, N., Dalvi, F., Sajjad, H., and Glass, J. What do neural machine translation models learn about morphology? Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics /Volume I: Long Papers), pp. 861-872 Vancouver, Canada, July 2017, Association for Computational Linguisties. doi: 10. 11111//11/P11--0000 URL https: //aclanthology org/P17-1000.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL 2020/file/ bo
neurips cctpsper///////ppcpp pdf.
Conneau, A., Kruszewski, G., Lample, G., Barrault, L.,, and Baroni, M. What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2126-21111111 1136, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v 1/P/0/ 198. URL https:tp/// ac aalpa/lcaaooooogoyooooooo otg/po8- 1198.
Dai, D., Dong, L.., Hao, Y., Sui, 7.., Chang, B., and Wei, F. Knowledge neurons in pretrained transformers. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 8493-8502, 2022.
De Cao, N., Aziz, W., and Titov, Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6491-6500 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL httt::://aallanhhllgggo org/ org/2021.1.. smnlp-main.55
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, tional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pupers), pp. 4171-4186 Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10. 1111865/1//N191122 URL fttaps:/aaclantholee
BERT: Pre-training of deep bidirec-
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., and Goldberg, Measuring and Improving Consistency in Pretrained Language Models. Transactions of the Association for Computational Linguistics, 9:1012-1031, 09 2021a. ISSN 2307-387X. doi: 10. 10.1162/ac URL https:/ //oiii////iiioo wtg/10..///. 11w./tta1cecee/ a00400
11

---

Elazar. Y., Ravfogel, S., Jacovi, A., and Goldberg, Y. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguisties, 160-175, 2021b.
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-oddds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J.. Kaplan, J., McCandlish, S., and Olah, C. A mathematical framework for transformer circuits. https: //transformerr- cinncfoorr-rcrcuutss pub/ pwwppppu//oaepppeppppppappaaaaat 2002//faaewwor/////oooo inne///innx./.../ html, December 2021.
Ettinger, A., Elgohary, A., and Resnik, P. Probing for semantic evidence of composition by means of simple classification tasks. In Proceedings of the 1st Workshop on Evaluating Vector-Space Repre- sentations for NLP., pp. 134-139, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 10.18653/vv// 1/W16-25216244 URL https: fttt:p:://acaaanhoolo.o
org/W16-20 2524.
Feder, A., Oved, N., Shalit, U., and Reichart, R. CausaLM: Causal model explanation through counterfactual language models. Computational Linguistics, 47(2):333-333 2021.
Finlayson, M., Mueller, A., Gehrmann, S., Shieber, S., Linzen, T., and Belinkov, Y. Causal analysis of syntactic agreement mechanisms in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp.
Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2221..cco acl-long. 144. org/ 2021 acl- col-long.... 144.
URL https: htttps:/aaclaantholloe
Geva, M., Schuster, R., Berant, J.., and Levy, O. Transformer feed-forward layers key-value memo- ries. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https::/aclanthheieee aclaatooonnntoooo..ccooo org/ /2221. emnlp-mane 446.
Hase, P. Diab, M., Celikyilmaz, A., Li, x., Kozareva, Z., Stoyanov, V., Bansal, M., and Iyer, S. Do language models have beliefs? methods for detecting. updating, and visualizing model beliefs. arXiv preprint 1.Xiv:2111.13663: 2021.
Hupkes. D., Veldhoen, S., and Zuidema, W. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-9266 2018.
Jiang, Z., Xu, F. F., Araki, J., and Neubig, G. How we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020. doi: 10.1162/ tacl a 000324. URL https: / http://///llaahhloggg org/2020 tacl-1.28.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL httttttt///xiiii http://wwxxvv.og../ee/// org/abs/1112 6980.
Kohonen, T. Correlation matrix memories. IEEE transactions computers, 100(4):353-359 1972.
Levy, O., Seo, M., Choi, E., and Zettlemoyer, L. Zero-shot relation extraction via reading compre- hension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL.2017 pp. 333-342, Vancouver, Canada, August 2017. Association for Computational URL
Linguistics. doi: 10. 10.18653/11/K11-100222
org/K17 1004..
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. BART: Denoising eqquence-to-sequeee pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 7871-7880 Online, July 2020. Associ- main.
ation for Computational Linguistics. doi: 10. 10.18853/11/222..co amwiii/0m0.mmmmimm.........
URL https:
/ //aclanthology org/ 2020/2220.02 acl mol1-main.700 703.
Meng, K., Sen Sharma, A., Andonian, A., Belinkov, Y., and Bau, D. Mass-editing memory transformer. arXiv preprint arXiv:2210.07222 2022.
12

---

Mitchell, E., Lin, C., Bosselut. A., Finn, C., and Manning, C. D. Fast model editing at scale. In International Conference on Learning Representations, 2021.
Pearl, Direct and indirect effects. In Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence, pp. 411-420, 2001.
Pearl, J. Causality: Models, Reasoning and Inference. Cambridge University Press, USA, 2nd edition, 2009. ISBN 052189560X.
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., and Miller, A. Language models knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pp. 2463-2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/D199111 1250. URL https: https: htttps://claantholggg org/D19- 1250.
Petroni, F., Lewis, P., Piktus, A., Rocktäschel, T., Wu, Y., Miller, A. H., and Riedel, How context affects language models' factual predictions. In Automated Knowledge Base Construction, 2020,
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models unsupervised multitask learners. OpenAl blog, pp. 9, 2019.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal Machine Learning Research, 21(140):1-67, 2020.
Roberts, A., Raffel, C., and Shazeer, N. How much knowledge can you pack into the param- eters of language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418-5426, Online, November 2020. Associ- ation for Computational Linguistics. doi: 10. 10.18666/11222220mmmn... 4377 URI. https: //aclanthology. org/2020 emnlp-main.44 437.
Sundararajan, M., Taly, A., and Yan, Q. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-332 -328.. PMLR, 2017.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, E., and Polosukhin, Attention is all you need. Advances in neural information processing systems, pp. 5998-6008, 2017.
Vig. J.. Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Sakenis, S., Huang, J., Singer, Y., and Shieber, S. Causal mediation analysis for interpreting neural NLP: The case of gender bias. arXiv preprint 2:Xi::2004.12665 2020a.
Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. M. Investigating gender bias language models using causal mediation analysis. In NeurIPS, 2020b.
Wang, B. and Komatsuzaki,, A. GPT-J-6B: A Billion Parameter Autoregressive Language Model. https: httpt/////ttttttt/eee wwtp://oooooooooooooooooooooeee
transformer- jax, May 2021.
Zhang, Y., Galley, M., Gao, J., Gan, Z., Li, X., Brockett, C., and Dolan, W. Generating informative and diverse conversational responses via adversarial information maximization. In NewrIPS, 2018.
Zhao, S., Pascual, D., Brunner, G., and Wattenhofer, R. Of nin-linyarity and commutativity in BERT. In 202/ International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2021.
Zhong, Z., Friedman, D., and Chen, Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5017-5033, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.118665///200.m..mmmm...m.... maicc-main339899 URL org/ 2021 www.aaalllaaaaaaa mai.mmmiin..... 398.
Zhu, C., Rawat, A. S., Zaheer, M., Bhojanapalli, S., Li, D., Yu, F., and Kumar, S. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363, 2020.
13

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes] In appendix
(b) Did you include complete proofs of all theoretical results? [Yes] In appendix
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] In supplemental materials
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In appendix
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In appendix
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] In appendix
(c) Did you include any new assets either the supplemental material as URL? [Yes] Supplemental materials
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] In appendix
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] In appendix
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] In appendix
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes] In appendix
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [Yes] In appendix
14


Reviews:
[{"review_id": "SRHpMS3BXqh", "reviewer": "Reviewer_avsv", "summary": "This paper provides evidence that knowledge recall in autoregressive language models corresponds to localized computations within feedforward MLP layers at a range of middle layers, and hence can be edited with a simple method that they name ROME. They compare ROME to other existing model-editing approaches both (1) on a zero-shot relation extraction benchmark and (2) related to generalization and specificity ability, and demonstrate strong results.", "questions": "-How did you tune the variance of the noise added to form the corrupted version of hidden states considering the extent you need to corrupt for the subject to be unknown might depend on the downstream task?\n\n-Did you try corrupting the subject with different corruption methods? I would be curious whether the conclusions regarding the importance of MLP for knowledge recall have a correlation with the fact that introduced noise is Gaussian.\n\n-Could you comment on the fact that ROME is underperforming baselines for generalization and whether generalization vs. specificity performance is an inherent trade-off?", "limitations": "Authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper offers very detailed empirical analysis and ablations that provide useful insights, and it is written very clearly. The work done here is both significant and original as factual storage capability of autoregressive language models are not well-understood to the best of my knowledge. I do not see a major weakness that would prevent acceptance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KD-oqmgp5hj", "reviewer": "Reviewer_gUgP", "summary": "This paper analyzes the storage and recall of factual associations in GPT and shows evidence that these associations correspond to localized, directly-editable computations. It utilizes a causal intervention to identify decisive neuron activations in a model’s factual predictions and find that the middle MLP layer plays an important role.", "questions": "Consider adding results across different model sizes and benchmark datasets.", "limitations": "The lack of human evaluation on generated text as an automatic evaluation metric may not highly correlate well with human beings.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* This paper is clearly motivated and well-structured in terms of writing. The causal tracing analysis tool is powerful and has some interesting findings that the essential role of MLP module computation at the middle layers when recalling a fact.\n* This paper proposes Rank-One Model Editing (ROME) and COUNTERFACT dataset to evaluate edits in standard benchmark and the proposed counterfactual one in language models. ROME can achieve competitive results on standard benchmarks and better results in Specificity, Fluency and Consistency metric. \n**Weaknesses**\n* The finding that MLP module computation at the middle layers plays an important role when recalling a fact is not surprising to me. After all, self-attention only linearly combines inputs and does not conduct transformation, and MLP requires doing non-linear transformation in a transformer block during the pre-training stage. Hence, it's kind of intuitive to me of have these results.\n* ROME seems to edit one fact at once. This is not scalable as usually, we want to inject knowledge of a corpus rather than a single fact. Further,  ROME can not take unstructured knowledge, i.e. plain text, which is widely existed and used during pre-training.\n* Results on GPT-J is less useful than standard fine-tuning baseline, which makes to cast doubts on how useful ROME is when model size is larger,e.g. GPT-3 175B.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CmH_ukWVyrF", "reviewer": "Reviewer_1iyP", "summary": "This paper studies how factual knowledge (e.g., a tuple of subject, relation and object) is stored in autoregressive transformer-based LMs. More specifically, the authors identify specific weights (in MLP) that are responsible for the correct output by comparing forward computation results (i.e., hidden states) with clean and corrupted inputs (__Causal Tracing__). To verify this observation, the authors propose a knowledge editing approach (__ROME__) that modifies a MLP weight matrix by a rank-one update.\n\n__Causal Tracing__\nThis method identifies specific hidden states that are more influential on the predicted object. The authors use the gap between the probability of the object with the original and corrupted inputs as a proxy of the importance of hidden states. Based on this analysis, mid layers at the last subject token have high values, contributing more to the probability of the object.\n\n__ROME__\nROME is designed to change the model output (i.e., the object) given the subject and relation. To do so, this approach modifies the weight matrix in MLP with a rank-one update so that the modified network outputs the desired/modified object. This is done by choosing k* (a vector before the matrix) and v* (a target vector) where v* requires optimization based on the desired object. Once an instance-specific k* and v* are computed, the weight matrix is updated using a closed formula.\n\n__Evaluation__\nROME is evaluated on an existing benchmark (zsRE) and a newly constructed dataset (CounterFact) and compared against other knowledge editing approaches such as KnowledgeEditor and MEND in terms of efficacy (is the object successfully edited?), generalization (does it work on paraphrases as well?), and specificity (are similar subjects kept unchanged?).  \n", "questions": "- Line: 159:  ` In practice, we sample x_j by generating 50 random token sequences of length 2 to 10 using G.`\n   - How does a choice of k* affect the final performance of ROME? Have you tried different methods?\n- It would be nice to discuss pros and cons of ROME compared to others (e.g., KE, MEND). \n\n", "limitations": "The authors mentioned the major limitation of this study that only focuses on relational knowledge (eg., KB triples). Studying other types of knowledge (e.g., more complex knowledge about entities) would be an interesting future direction.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n- This paper is tackling a well-motivated and very important problem: how LMs store factual knowledge and how we can control it. ROME can be used for impactful applications such as resolving temporal mismatch between LMs and real-world knowledge.\n- The authors analyze their hypothesis thoroughly and propose an efficient way to modify model parameters. The authors keep ROME simple but show its effectiveness. \n- This paper is very well-written; each part of this paper is clearly explained despite the complexity of the problem.\n\n[Weaknesses]\n- My only concern about ROME is scalability.  To find v* for the modified object o*, this approach requires optimization for each test instance. This might be a speed bottleneck when the data size is very large. Although modifying a weight matrix is rank-one update, I’m not sure if this approach is considerably cheaper than other methods like MEND. MEND uses a hypernetwork, but, once it’s trained, inference is just forward computation. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept