Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
Authors: Peter Hase
Mohit Bansal
Been Kim
Asma Ghandeharioun
Venue: NeurIPS 2023 spotlight (2023)
Keywords: localization
model editing
mechanistic interpretability
language models

Abstract: Language models learn a great quantity of factual information during pretraining, and recent work localizes this information to specific model weights like mid-layer MLP weights. In this paper, we find that we can change how a fact is stored in a model by editing weights that are in a different location than where existing methods suggest that the fact is stored. This is surprising because we would expect that localizing facts to specific model parameters would tell us where to manipulate knowledge in models, and this assumption has motivated past work on model editing methods. Specifically, we show that localization conclusions from representation denoising (also known as Causal Tracing) do not provide any insight into which model MLP layer would be best to edit in order to override an existing stored fact with a new one. This finding raises questions about how past work relies on Causal Tracing to select which model layers to edit. Next, we consider several variants of the editing problem, including erasing and amplifying facts. For one of our editing problems, editing performance does relate to localization results from representation denoising, but we find that which layer we edit is a far better predictor of performance. Our results suggest, counterintuitively, that better mechanistic understanding of how pretrained language models work may not always translate to insights about how to best change their behavior.

Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
Peter Hase
Mohit Bansal
Been Kim
Asma Ghandeharioun'
Google Research
UNC Chapel Hill
(peter, abansall les .ucc edu
(beenkim, eghandeharionn (gooooooenneg.e.ooooooooo @google.com
Abstract
Language models learn great quantity of factual information during pretraining, and recent work localizes this information to specific model weights like mid-layer MLP weights [21]. In this paper, we find that we can change how fact is stored in a model by editing weights that are in a different location than where existing methods suggest that the fact is stored. This is surprising because we would expect that localizing facts to specific model parameters would tell us where to manipulate knowledge in models, and this assumption has motivated past work on model editing methods. Specifically, we show that localization conclusions from representation denoising (also known as Causal Tracing) not provide any insight into which model MI.P layer would be best to edit in order to override an existing stored fact with a new one. This finding raises questions about how past work relies on Causal Tracing to select which model layers to edit [21, 22]. Next, we consider several variants of the editing problem, including crasing and amplifying facts. For one of our editing problems, editing performance does relate to localization results from representation denoising, but we find that which layer we edit is a far better predictor of performance. Our results suggest, counterintuitively that better mechanistic understanding of how pretrained language models work may not always translate insights about how to best change their behavior.'
Introduction
Language models learn a variety of facts about the world during pretraining that can be elicited via natural language prompts [28]. Recent work explores how these facts are stored model weights and expressed in response to particular prompts, suggesting that MLP weights act as key-value memories that support factual association [12, 21, 13]. Besides improving our scientific understanding of pretrained language models, this kind of investigative work may enable the design of better model editing methods for injecting new facts into model weights, and indeed it has been used to motivate the ROME and MEMIT model-editing methods [21, 22]. These recent methods set a new state of the art for weight edits that successfully rewrite stored facts in language models. Model editing methods could be broadly useful for correcting factual errors in pretrained models, avoiding morally undesirable outputs, and updating models with changing knowledge over time.
The connection between localization (identifying components of model responsible for a certain behavior) and editing (changing model components in order to change model behavior) is predicated on the reasonable assumption that one should go about editing model by first localizing a behavior to a specific component and then choosing to edit that particular component. In the case of ROME and
Code for all experiments is at
http:/ggwww.ccccggggg/////eeeeee localllleelloollololooo in
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

MEMIT, localization is done via Causal Tracing, which measures the information content of hidden representations, and editing is done by treating MLP weights as linear associative memories and inject- ing new key-value memories into the weights. Meng et al. [21, 22] choose to edit early MIP layer(s) based on results from Causal Tracing showing the largest causal effects on average in early layers.
Surprisingly, the assumption that one should change the knowledge in model by editing the weights where it is stored turns out be false. In fact, localization results from Causal Tracing are statistically uncorrelated with the success of an edit injecting a new fact into MLP weights. Using the CounterFact dataset from Meng et al. [21] with a GPT-J model [35], we show that (1) not only is a substantial fraction of factual knowledge stored outside of the range of layers edited by ROME/MEMIT (see Fig. 1). (2) the correlation between Causal Tracing results and edit success is near zero (for several editing methods including ROME. MEMIT, and Adam-based finetuning). We note that this surprising largely because ROME and MEMIT do work well for editing facts, in spite of Causal Tracing often suggesting knowledge is stored elsewhere than -ary--e-mid MLP weights.
In the face of this result, we attempt recover the connection between tracing-based localiza- tion and editing by introducing four variants of the default model editing problem. Each variant differs in terms of the input, target, or objective used in the editing problem. One variant we introduce, called Fact Forcing, is designed to match Causal Tracing along these three factors. Specifically, Fact Forcing uses a noised input and involves maximizing the probability of the correct target output, just like Causal Tracing. We find that tracing results are related to edit success for Fact Foreing. However, even for this variant, it is still better to ignore the tracing results and always choose an early-to-mid-lue MLP weight for editing. We conclude that, although Causal Tracing is a reasonable localization method that has yielded insight into how models store factual information, this insight does not actually indicate which model layers we should edit in order to manipulate what facts are stored in language models.
How often does Causal Tracing peak in cach layer?
200
ROME Edit Layer MEMIT Edit Layers
150
Num. Points
100
0
12
16 20
24
28
Layer in GPT-J where Causal Tracing effects peak
Figure 1: We visualize where 652 facts known by GPT-J are stored within the model, as local- ized by Causal Tracing. Model editing methods like ROME and MEMIT can successfully change knowledge in LMs by editing layers 4-9. But many facts appear to be stored outside of this range, e.g. at layers 1-3 and 16-20. What about these facts?
To summarize, our conclusions are as follows:
1. We find that model edit success is essentially unrelated to where factual information is stored in models, as measured by Causal Tracing. Robustness experiments generalize this result across causal localization methods, editing methods, editing metrics, models, and datasets.
To reconnect localization with editing performance, we introduce four variants of a standard model editing problem, including Tracing Reversal, Fact Erasure, Fact Amplification, and Fact Forcing.
Edit success and tracing effects correlate best in the Fact Forcing setting. However, tracing effects explain only small fraction of the variance in editing performance, while the choice of edit layer is a much more important factor. This suggests that, surprisingly, localization insights from Causal Tracing are not useful for choosing which model layer to edit.
2 Related Work
Localization. A long line work aims to interpret what certain hidden representations represent, or, in the reverse direction, to understand how a given concept is represented in a model. Both of these efforts aim to localize behaviors to specific model components. We group these methods based on the kinds of model components they consider (e.g. layers, neurons, etc.).
Many works focus on individual layers or weight matrices [37, 31, 93, 11]. In this paper, we adopt the layer-wise localization method from Meng et al. [21] known as Causal Tracing, which estimates the information content of a set of representations via a denoising operation. We specifically focus on MLP layers given evidence of their role in factual association [12, 21, 13].

---

Related to analysis at the layer level, other work aims to localize concepts to directions in latent space, dating back work interpreting directions in word vector space [23, 17, 41, 14, 38, 5]. One might also place "key-value memory" theories of weight matrices in this category since key vector represents a direction in the latent space [1, 32, 12, 21].
Neurons, meanwhile, are the most common focus of localization analysis. Past work explores the functions of groups of neurons and subnetworks [27, 6, 10, or simply individual neurons [29, 40, 18, 2, 34, 26, 8, 19, 16, 363..
Relating Localization to Editing. Many works on localization validate the quality of their con- clusions by editing neuron activations or layer weights corresponding to a particular concept, then checking that the network behavior changes appropriately. For example, Dai et al. [8] check that their "knowledge neurons" have localized a specific fact by amplifying or suppressing the expression of that fact via adjusting the corresponding neuron activations. Altogether, find many localization analyses are validated by editing models in suggested locations [29, 18, 2, 26, 34, 8, 19, 7, 36, 4] or directions in the latent space [23, 1, 32, 21J.
Changing model behavior by editing components suggested by localization seems like a reasonable validation step. However, in isolation, it paints incomplete picture that has led to misleading interpretations about the connections between localization and editing. Such experiments alone do not show whether editing that specific component is (1) successful in proportion to the strength of the localization, (2) necessary to achieve the desired behavior, or (3) the best option for editing. In particular, these experiments do not show whether the same change in behavior
be achieved elsewhere in the network. Meng et al. [21] consider this question by measuring editing success across layers, averaged across data, then comparing the results with Causal Tracing conclusions also averaged across data. However, as we show, more fine-grained analysis at the datapoint level reveals the unexpected result that tracing results are unrelated edit success. We are not aware of any work that primarily investigates the connection between localization and editing or that demonstrates better model editing at locations elsewhere in the network than those suggested by localization analysis.
3 Notation and Background
3.1 Data Notation
Following Meng et al. [21], we consider facts of the form (s, r, o), where represents a subject entity (e.g. Paris), binary relation (e.g. is located in), and an object (e.g. France) for which the tuple (8, r, represents a factual assertion about the world. the CounterFact dataset [21], each datapoint is prompt P for some fact (s, o). So, P might be "Paris is located in" or "Paris is situated in," to be completed by the object o to form a true statement. In an abuse of notation, we will often use s and to refer to textual representations of subject and relation, for instance by writing model's conditional probability as po [s, r) instead of po (-) We do so in order to more easily indicate when input is provided where the subject relation has been manipulated (described next).
We make use of few variations of the data for the fact (s, r, o). The additional variables include:
1. s* is a "neighboring" entity to the subject s (similar to s) for which rs", is a true fact like (s, r, o). In CounterFact, "Marseille" is a neighboring entity "Paris."
r* a paraphrase of the relation r, such as "is situated in" for "is located in."
3. Spoine is a noised representation of the subject s. We add Gaussian noise to the token embeddings of s, following Meng et al. [21].
Ofaise is an object that incorrectly completes the tuple (s, .).. CounterFact contains an Ojaise for each datapoint, intended to be the new model output when evaluating model editing methods.
5. Owne for clarity, is the object that correctly completes the fact (8, r, :), from CounterFact.
3.2 Causal Tracing
We give a brief description of Causal Tracing here and refer readers to Meng et al. [21] for more information (see Fig. 2 for example visualization). Causal Tracing is a method for localizing information in the forward pass of an autoregressive Transformer to specifie hidden representations. For model with L layers, the input is a prompt containing T tokens (including subject and
3

---

relation r). Given this input, the forward pass produces T x L layer outputs (one representation per T tokens and L layers). The algorithm aims to estimate the amount of information about the fact (8, Oine)) that is contained in each of these representations. We denote the representation at token /. and layer l as V(c.e)-
The amount of factual information in (((.e) is estimated by copying this representation into a different forward pass obtained from using noised subject in the input:
Tracing Effect = po (Ore Smoize T, P(e,,)) ) Po (Otruel (Snvise r)
where Smoise indicates that add Gaussian noise with a = 0.094 to the token embeddings of following Meng et al. [21], and V(C,E) is the representation at token t and layer l in the for- ward pass the original prompt P r). The probability pe (Oruel Snoise, r, ((t.F)) is com- puted by (1) running the model forward pass the noised prompt P = (Snoise: r)) until layer (2) overwriting the existing representation at token t and layer É with the representation ()t.E)) then (3) computing the remaining L é layers as normal using this adjusted set of ''' represen- tations as input (adjusted token index t). Thus, Causal Tracing estimates the information con- tent of representation in terms of its effect on the probability of the true target. The results from Causal Tracing show where the represen- tations containing information about the true target are the model forward pass.
10) MLP Tracing Effect (Window Size:
0.5
The*
Space*
r).
0.4
Need*
le*
0.3
is
in
0.2
the
0.1
city
of
p(Seattle)
0 10
15
20
Center of 10 Restored Layers
Orig Prob: 0.923, Noised Prob: 0.001
Visualizing Causal Tracing results over MI.P layers with window size 10. Tokens with an asterisk are the noised subject tokens. Here, (o |8, r.233 and po (Ornue Snoise; 0001
Figure 2:
In practice, a set of representations from mul- tiple adjacent layers is copied from the clean forward pass rather than a single layer's rep- resentation (for instance. ten layers Fig. 2). The size of this is referred to as the tracing window size. A window size of, c.g., three implies that the tracing effect at layer & estimates the amount of information contained the three representations V(r, ((t,f). and ((, See Appendix Figs. 10 and 11 for analysis of the parameter's s effect. In this paper, we use a tracing window size of 5 by default, and we apply Causal Tracing exclusively MLP layers, given evidence of their role in factual association [12, 21].
3.3 Model Editing with ROME
We describe the ROME editing method here since we use it in our analysis in Sec. 4, and later in Sec. 5 we outline additional editing methods we consider. For mathematical detail,
CounterFact Example
Input Prompt:
Autonomous University of Madrid, which is located in
see Meng et al. [21].
Requested Edit:
Spain
Sweden
The input to ROME includes prompt P= (s, and new desired output, which is always false target Ofaise in the CounterFact dataset. To change the model prediction to Ojatse. ROME applies a rank one edit to the down-projection matrix in prespecified MLP layer in the model. The default layer in GPT-J layer 6, following from averaged Causal Tracing results. ROME also makes use of covariance statistics of different subject representations obtained from larger corpus as it edits individual facts. Overall, the method is designed to optimize the quantity Po (Ojatelss r) while aiming to satisfy some other constraints reflecting what desirable model edit is (described in Sec. 3.4 next).
Paraphrase:
Sallie Beavers Riley. Autonomous Universitty of Madrid is located in
Neighbor:
Ripollés,
located
Figure 3: An example CounterFact datapoint.

---

3.4 Editing Metrics
Editing methods are typically evaluated according their ability (1) change the model prediction on the input P provided at runtime, (2) generalize appropriately to paraphrases of the prompt P, and (3) avoid over-generalizing to unrelated data [42, 9, 24, 15, 25]. We adopt metrics for each desideratum that we compute with available CounterFact data. Instead of the exact "magnitude" metrics from Meng et al. [21], we use normalized versions of each metric that we design to scale from O to depending on whether the edit was maximally (un)successful, for purposes of making scores more comparable across data points. We denote the new edited weights of the LM as 0" and its pre-edit weights f. See Fig. 3 for an example of the kinds of data these metries computed
1. Rewrite Score. The rewrite score measures how much an edit improves the target probability p(ofatse 8, r) as a fraction of the maximum possible improvement:
Po- (Ofaise 88 r) Po (Ofatse (8, r)
1 - Pe Ofalse r)))
Paruphrase Score. The paraphrase score measures the target probability using syntactical para- phrases as inputs, always preserving the exact subject wording:
po (Ofatsel 8, r") po (Ojatse |s, r")
pe (Ofalse 8, r*)
which is averaged over multiple available paraphrases per input P. The score measures whether edits properly generalize across semantically equivalent prompts.
3. Neighborhood Score. The neighborhood score measures whether edits change predictions for prompts with similar subject s", the same relation and the same (true) objects. We scale the difference in probabilities so that 1 means the probability did not change (good), and 0 means changed to the maximum extent possible (bad):
ppo (Ojatse (s", .)) po (oluuuese r)]
.5 + [po(ofalse) s*, -5|
The score measures whether edits avoid over over-generalizing from the prompt different subjects.
4 Does Edit Success Follow From Localization?
Ostensibly, localization results should inform editing methods because it should help to know where information is stored in a model if you are going manipulate the model's expression of that information. More specifically, if you wanted to inject false belief (fate) into model (as defined in the ROME editing problem), it seems helpful to know which weights store the true fact (s, r, Omee), so that you could replace some stored representation of Onee with that of Ofaise- This underlying assumption about editing models appears much past work on localization, where editing is used to verify localization analysis (see Sec. 2). In this section, investigate the validity of this assumption as it applies to autoregressive Transformers.
4.1 Experiment Design
The goal of our experiments determine, for given datapoint, whether edit success a specific layer aligns with the results from Causal Tracing at that layer (see Causal Tracing description in Sec. 3.2). We operationalize this outcome and explanatory variable follows:
1. Edit Success. We primarily consider Rewrite Score our measure of edit success, given that this the main optimization objective of ROME. Note ROME achieves an average rewrite score of 99% at layer 6 of GPT-J and above 96% at layers besides the last layer of the model.
2. Tracing Effect layer f. Since the output of Causal Tracing is T L grid of estimates, we obtain a single tracing effect per layer by taking the max across the T token effects at each layer (i.c., we collapse the grid in Fig. 2 down to single curve across layers). Like our other metries,
5

---

we use a fractional tracing effect where means the intervention had no effect and means it fully restored the original probability po (O (s, r):
po (Orree Suoise V v((,E)) -o 20tuue ( Snoises r)
Po (ome (8, r) Po (Snosee; r)
Lastly, note we use tracing window size of 5 (smaller than the value of 10 used in Fig. 2).
4.2 Model and Data
We conduct our analysis with GPT-J [35] using the CounterFact dataset, similar to Meng et al. [21]. GPT-J
ROME Rewrite Score by Tracing Effect Layer
billion parameter autoregressive language model. We record editing performance at layers in (1. 5, 9, 13, 17. 21, 25, 28) } as well as layer 6 (the default for ROME). Note ROME achieves an average rewrite score of 99% at layer 6 and above 96% at layers besides layer 28.
0.75
Rewrite Stoore
0.50
0.25
The CounterFact dataset includes datapoints consisting a prompt, paraphrases, and neighboring points. For each point, new (false) target is supplied for editing purposes. We show example datapoint in Fig. 3. Note paraphrases intentionally include unrelated text preceding syntactical paraphrase of the input, with the idea that this text should not affect the output. We select data for experiments from 10% of CounterFact, additionally filtering to subset of facts that correctly completed by GPT-J, in order ensure that there is knowledge to localize in the model for each point (details in Appendix A). Our final sample size is n = 652.
0.00
Tracing Edfeect (Frmction Restored)
Figure 4: The correlation between ROME edit
success and the tracing effect at layer 6 in GPT-J not positive but in fact slightly negative (p -0. -0.13; <le-3). The dashed red line shows hypothetical perfect relationship.
4.3 Experiment Results
We present results in two ways. First, in Fig. 4, we show Rewrite Score as function of the (fractional) tracing effect. The red dotted line shows a hypothetical perfect relationship between tracing and edit success. Surprisingly, there is not a positive relationship but a negative relationship between the rewrite score and the tracing effect (linear correlation of -0.13; p <le-3). This seems to fully invalidate the assumption that editing should be most effective when it occurs at layer where information is stored about the edited fact. We wish to emphasize, however, that in most layers we simply see near-zero rather than negative correlation, shown in Appendix Fig. 15.
Our second mode of analysis though linear regression models predicting rewrite score based on (1) the tracing effect, (2) the choice of edit layer treated as a categorical variable, or (3) both terms interacted, again treating edit layer a categorical variable. The purpose of the models is to show how much of the variance in rewrite score is explained by one variable versus the other. We show the resulting R² values in Table 1. We that the choice of layer explains almost all of the variance in rewrite score (94.7%), while adding the tracing effect to the model raises the R² only to 94.8%. This means that the tracing effect is able to explain only 0.1 % of the variance in edit success when accounting for the choice of edit layer. These results suggest that the tracing effect is essentially unrelated to the success of model editing.
Table 1: RD values for predicting ROME edit suc- cess. Tracing effects explain essentially none of the variance in rewrite score, while the choice of edit layer is very important.
R Values
Method
Layer
Tracing Effect
Both
ROME
0.947
0.016
0.948
This is surprising conclusion, and it naturally raises the question of why applying ROME at layer 6 works well the first place (see average rewrite, paraphrase, and neighborhood scores across layers in Appendix Fig. 7). We suggest a possible answer to this question in Sec. 6.

---

Editing Problem Variants
Input Prompt
Objective
Error Injection
Autonomous University of Nodrid, which is located
arg max max pu(Sweden Input)
Tracing Reversal
Autonomous University of Modrid, which is located in ang max gu(Onis Input)
Fact Erasure
Autonomous University of Modríd, located
nng min yo (Spain (Input)
Fact Amplification
Nutonommuu University
acg muax (Spain Input)
Fact Forcing
Autonomous University Madrid, located
ang max (Spain Noisy Inpat)
Add noise subject
Figure 5: Depiction of editing problem variants. Rather than inject new false fact into model (Error Injection), we consider injecting the output obtained from noising the subject entity (Tracing Reversal), erasing stored fact (Fact Erasure), amplifying stored fact (Fact Amplification), or forcing a known fact onto the same kind of noisy imput as used in Causal Tracing (Fact Forcing).
Additional Robustness Experiments. We include additional results in Appendix B using another dataset, 7SRE [20] (Figs. 19 and 20, Table 8), and another localization method, representation zeroing [2] (Figs. 21 and 22). Further robustness experiments in Appendix C include results with (1) other measures of edit success including Paraphrase Score, Neighborhood Score, and Overall
Score (Tables 4, and 6). (2) different values the tracing window size (Fig. 12), (3) GPT2-XL
rather than GPT-J (Fig. 13), (4) the original unscaled metrics from Meng et al. [21] (Fig. 14), and (5) tracing effects measured at the last subject token rather than the max across tokens (Fig. 16). We find that all of these experiments corroborate our results comparing Causal Tracing to Rewrite Score for GPT-J on CounterFact Considering these robustness results alongside additional editing method experiments that consider in Sec. 5 below, note that our main conclusions generalize across different causal localization methods, editing methods, editing metrics, models, and datasets.
Reconciling Localization and Editing
If injecting new fact has little to do with where existing fact is stored in the model, perhaps there is some other editing intervention that would be more closely related to insights from tracing analysis. In this section, we propose few variants of the model editing problem that appear more and more like Causal Tracing in terms of their input, target, and objective. Then, we repeat and extend our analysis from Sec. 4 for all of these editing problems.
5.1 Editing Problem Variants
We summarize the following editing problems in Fig. 5.
1. Error Injection. The editing problem considered in Sec. 4, the objective being to maximize po (o(00 Ofuise s)
Tracing Reversal. We maximize po po(moaie [s, r), aiming to change the model output from Owne back the output for the "original" noised input (Smoise: r) in Causal Tracing, Onoise-
Faet Erasure. Knowing where fact stored could be more useful for erasing the fact rather than injecting new one. Hence, we consider erasing fact by minimizing po (Otnels,
4. Fact Amplification. We reinforce known facts in the model by maximizing pe (O (s, Even for correctly predicted points, this value is often not near 1, leaving room for it to be increased.
5. Fact Forcing. As in Causal Tracing, this method uses a noised subject representation Smoise- We force the model output Ovne for this input by maximizing po po(Ore (Snoise: r)) Though this problem is of little practical significance, it is the most similar to Causal Tracing in its design, since it uses the same input as Causal Tracing and matches the goal of increasing the probability of Opue (see Sec. 3.2).
Note that solutions to each of these problems are evaluated according our Rewrite Score, Paraphrase Score, and Neighborhood Score metrics from Sec. 3.4. The only difference is in the target output for the rewrite and paraphrase metries (ncighborhood is entirely identical).
7

---

Tracing effects very weakly predictive of edit success
Tracing Revensal
Fact Amplification
Fact Crasure
Fact Forcing
100
915904
"y < 1o-4
7..77.." 72.4"
(%6)
0945
25
312215
FT-1
FT-5
ROME MEMIT
FT-1 FT-S ROME MEMIT
FT-1
FT-S
ROME MEMIT
FT-1
FT-S
ROME MEMIT
Explanatory Variable(s):
Layer
Layer
Tracing Effect
Figure 6: Tracing effects are very weakly predictive of edit success across editing problems and methods. Relative to the R regression predicting rewrite score based on the edit layer (blue), a regression with edit layer and tracing effects (orange) improves the R by most .03 points (bolded). The choice of edit layer is a much better predictor of the rewrite score.
5.2 Experiment Design and Additional Edit Methods
We use the same experimental procedure as in Sec. 4, except that we consider a broader set of editing methods besides ROME. We list the four methods below:
1. ROME. The edit method from Sec. 4, ROME edits a single MLP layer's down- projeppooection weight.
2. MEMIT. Though designed to edit multiple facts at once, when editing single fact this method differs from ROME only by spreading out its update over several layers rather than one layer [22]. Constrained Finetuning (window size 1). We adopt simple Adam-based optimization approach with an -norm constraint, following Zhu et al. [42]. The window size of indicates we apply this method single layer.
4. Constrained Finetuning (window size 5). The above finetuning method on five adjacent layers.
We select these methods for their simplicity and since ROME and MEMIT are designed specifically to edit MLP layers. Note that report results for Causal Tracing with window size of five, so when we use MEMIT or constrained finetuning to edit five layers, these five layers can exactly match the range of restored layers from Causal Tracing.
5.3 Experiment Results
Main Results. As in our analysis in Sec. 4, we report R² values for linear regression model predicting the rewrite score based on (1) the choice of edit layer treated as categorical variable, or (2) that variable interacted with the tracing effect. We show the results in Fig. 6, with R²² values for each regression above their respective bars (numbers also in Appendix Table 3). We find that, relative to the Layer-only regression, tracing effects explain at most additional 3.2 % of the variance edit success across our different editing problems and editing methods. This a very small effect, especially compared to R values from the Layer-only regression, which explains most of the variance in the outcome (58.5% on average across conditions in Fig. 6). We believe this is surprising given how the editing problem variants are designed. It would seem that knowing where a fact is stored should help with amplifying or erasing that fact, but our results appear to fully disconfirm this hypothesis. Interestingly, it also appears that it makes little difference whether edit at one layer or five layers in order to match the number of representations restored by Causal Tracing. Based on comparisons between finetuning methods (FT-1 and FT-5) and between ROME and MEMIT (applied to 5 layers), editing at five layers does not improve the alignment between tracing and editing. In addition our robustness results listed in Sec. 4.3, also repeat our analysis using a subset of points where tracing effects are concentrated to small number of layers, in order focus on points where MEMIT and FT-5 edit all of the layers where the fact is stored. Results are nearly identical for this subset of the data (see Appendix B).
One Successful Case. We see the strongest positive relationship between edit success and tracing effects for Fact Forcing with finetuning methods. Here, we find that tracing effects explain an additional 3% of the variance in edit success (up from 1.5% for other experiments). This effect
8

---

is statistically significant at p < 1e-4 according to an F-test" comparing the two models (see visualization in Appendix Fig. 17). The result for Fact Forcing suggests that using Spoire rather than s in the model input is the cause of the positive relationship between editing and localization. We rule out the choice of target and maximizing vs. minimizing the target probability as possible causes based on the design of each problem variant (see Fig. 5) (1) the choice of target is not important since results are similar for Error Injection, Tracing Reversal, and Fact Amplification, and (2) maximizing vs. minimizing the target probability is not important since results are similar for Fact Erasure and Fact Amplification. Yet, tracing effects are still weakly informative of Fact Forcing editing if they explain only 3% of the variance in edit success. This points to there being other deeper reasons for localization results being unrelated to editing success.
6
Discussion
Does Causal Tracing tell anything? We show that Causal Tracing is not indicative of which layer to select for model editing. However, this does not mean that localization insights from Causal Tracing have been useless. Causal Tracing has helped reveal the role that carly-to-mid-range MLP representations at the last subject token index play in factual association in autoregressive language models, and ROME does perform better on average when optimizing the last subject token representation rather than another token representation [21].² Past work finds that both MLP and attention layers can show large Causal Tracing effects, and additional empirical editing experiments then demonstrate that it is preferable to edit MLP weights [21].
Why is edit success high at layers where the edited fact is not actually stored? First, we note that information is gradually accumulated across layers in a Transformer forward pass, as discovered by past work [31, 12, 21, 22, 13]. We suggest that it is possible "override" the information in layer l with an edit to another layer l (where k < or k > f). Since ROME is typically effective across a large range of layers (see Fig. 9), appears that ROME can override the information accrued across 5 or 10 layers of a forward pass with an edit to a single layer outside of that range of layers. We summarize this hypothesis as follows: Many layers could store a fact, and it happens that some do.
If this hypothesis were true, it would be surprising because one cannot arbitrarily swap layers in a Transformer model without greatly damaging model performance [39]. That is, it should matter where information enters the residual stream, since later layers strongly depend on receiving the right incom- ing information from prior layers. We leave future work to further investigate this hypothesis.
What do our results imply about using model editing to validate localization claims? We interpret our results to suggest that Causal Tracing answers a different question than model editing does. That is, Causal Tracing answers a question about where factual information is carried in representations in a Transformer forward pass, and this question turns out to be a different question than the editing question of where is best to intervene in the Transformer in order to change the factual information it expresses. It seems critical, then, to carefully formalize the questions that one wishes to answer before (1) validating the results of localization via editing or (2) motivating the design of an editing method via localization, because the conclusions that can be drawn from a particular localization method might not be relevant for the performance of a given model editing method. This would not imply the conclusions from the localization analysis invalid, though. For instance, we believe Causal Tracing reveals interesting insights about where MLP representations contain factual information (see Figs. and 2). We only wish to suggest that localization analysis might answer different question than the question answered by model editing.
These observations may have implications for the array of studies that validate their localization analysis by manipulating a certain model behavior via an intervention on the model component recommended by the analysis [29, 18, 2, 1. 26. 34, 8, 19, 7, 36, 4, 21]. Do model editing experiments provide additional evidence for claims about which model components are responsible for certain behaviors? If localization and editing answer different questions, editing experiments will not provide further evidence for localization conclusions.
"This tests if one model explains more of the variance than another model which has only a subset of the first's covariates (here, tracing effect and edit layer vs. only edit layer). AAthoogh, datapoint- regression would provide stronger evidence that tracing effects predict which token representation is best to optimize with ROME (and rule out other confounders such as the edit layer).

---

7 Conclusion
We obtain the surprising result that model edit success is essentially unrelated to where factual information is stored in models, as measured by Causal Tracing. Faced with this result, we attempt to reconnect tracing- based localization with edit success by introducing four variants the Error Injection problem using the CounterFact dataset. We find that edit success and tracing effects correlate best in our Fact Forcing setting. However, even in this case, tracing effects explain only small fraction of the variance in editing performance, while the choice of edit layer is much more important factor. This suggests that, counterintuitively better mechanistic understanding of how pretrained language models work may not always translate to insights about how best change their behavior.
Limitations
We note few limitations of the experiments conducted in this paper:
(1) We work only with the CounterFact and ZSRE datasets, which we use as short English prompts with factual completions corresponding entities. This is a basic form factual knowledge, and localization and editing analysis may yield different trends for other forms of knowledge.
specific set of relations between subject and object
(2) We work with two autoregressive Transformers chosen for their representativeness of large language models that show capacity for expressing factual knowledge in response to natural language prompts. However, the conclusions from our analysis may not generalize to models larger than GPT-J (6B parameters) that are known to exhibit phase changes in their behavior under prompting.
(3) We use a particular set of localization and editing methods, including representation denoising and zeroing at the layer level and layer-level MLP editing methods that inject new facts or amplify or erase existing facts. Our conclusions may not necessarily hold for the breadth of localization and editing methods from work related to this paper, and one should be cautious applying our conclusions beyond our experimental setting.
Broader Impacts
It is possible that increased mechanistic understanding of models improves our ability to edit them at some point the future. In fact, we consider it unlikely that interpretability results never give insight into improving model editing methods. Thus, to the extent that model editing is dual use methodology, which could be used to inject harmful beliefs or dangerous knowledge into models, interpretability results may enhance the effectiveness of these malicious use cases. However, these concerns are relatively far removed from our analysis, which focuses on the connection between localization and editing performance. Ultimately, hope that studies of mechanistic interpretability and model editing improve our ability to control language models.
Acknowledgements
We thank Kevin Meng and David Bau for helpful discussions of the methods used in this paper and the conclusions we reach from our experiments. Additionally, we thank Jasmijn Bastings and Lucas Dixon for feedback on writing and presentation of results, as well as the paper's 's reviewers for their useful comments. This work was conducted while Peter Hase was a student researcher at Google.
References
[1] David Bau, Steven Liu, Tongzhou Wang, Jun Yan Zhu, and Antonio Torralba. Rewriting a deep generative model. European conference on computer vision, pages 351-369, Springer, 2020. URL https:/ wwwpa.twgggaaaaaaataa......o..tt 2007 15646..6.. 1ff.
[2] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in deep neural network. Proceedings of the National
10

---

Academy of Sciences, 117748):30071-30077 2020. URL https::/www.pnas wwt.pppn...ooog.o.ooooooooooooo pdf/ TTT.//1..111//111 1907375117.
[3] Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wattenberg. An interpretability illusion for bert. arXiv preprint arXiv:2104.071433 2021. org/ /tdd//114, 07143. pdf.
URL
[4] Stephen Casper, Shlomi Hod, Daniel Filan, Cody Wild, Andrew Critch, and Stuart Russell. Graphical clusterability and local specialization in deep neural networks. In ICLR 2022 Workshop on PAIR, 2022. URL htwph::t////mm /atpps:///axivv..// wttppo/ggp///eeeeo /210 Ttw.11..0050000 08058v2.pdf.
[5] Pattarawat Chormai, Jan Herrmann, Klaus-Robert Müller, and Grégoire Montavon. Disentangled explanations of neural network predictions by finding relevant subspaces. arXiv preprini arXiv:2212.14855 2022. URL https::///// /awxivixvvvvv/////v////a/a//aaaa org/ httg//pf//// ptdd///22// 14855. pdf.
[6] Róbert Csordás, Sjoerd Steenkiste, and Jürgen Schmidhuber. Are neural nets modu- lar? inspecting functional modularity through differentiable weight masks. arXiv preprint arXiv:2010.02066, 2020. URL https:////aati http::/w.wrgppp//////ee////
02066. pdf.
[7] Audrey Cui, Ali Jahanian, Agata Lapedriza, Antonio Torralba, Shahin Mahdizadehaghdam, Rohit Kumar, and David Bau. Local relighting of real scenes. arXiv preprint arXi::2207.02774, 2022. URL https: /taps:///arxivii /aapaaivivvv///e///.a///aaa org pdf /2207 12774.ppe.
[8] Damai Dai, Li Dong, Yanı Hao, Zhifang Sui, and Furu Wei. Knowledge neurons in pretrained transformers. In ACI. 2022. URL htt.:ttt.s////xi../r///////////
68696. pdd..
[9] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In EMNLP, pages 6491-6506. Association for Computational Linguistics, November 2021. URL https: acl http://aaaaathohooooooooo org/2021
2021. emnlp-ma 522.
[10] Nicola De Cao, Leon Schmid, Dieuwke Hupkes, and Ivan Titov. Sparse interventions in language models with differentiable masking. In EMNLP BlackboxNLP Workshop, 2021. URL ht https: /taps://arxivv org/ pdf 2112. 06837 pdf.
[11] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dode -oddd, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transformer wiirraansfoteerrrrrrtstttseeeste citrauttspp///2222/faeeeerrrnnn htfff///www//nndee.iirro
[12] Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers key-value memories. EMNLP. 2021. URL https: //t:///ariii http://.www./pp.222....2222 pdf.
[13] Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203.14688, 2022. URL https: arxiv. wttpxii.orgg/d/////// 14622....160.pp.. pdf.
[14] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept- based explanations. Advances in Neural Information Processing Systems, 32, 2019. URL ptttps:/arxiv..rrtt /ttp//df//// pwdf///9.2.. 1902.992....2 03129 pdf.
[15] Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornisaa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. arXiv preprint arXiv:2111. 13654, 2021. URL https: wtt/ http://ff21111111//////1 /2111.11111 13654...... pdf.
//arxiv.org/pdf
[16] Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In International Confer- ence on Learning Representations, 2022. URL https: //tp///pppeeeeeeeeeeeeeeeeee eew, net/pdffid=
NudBMY- - ztr.

---

[17] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2668 2676.-667. PMLR, 10-15 Jul 2018. URL https: http:///pocceedddee mlr.
press / v80 /kimll 8d. html.
[18] Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. The emergence of number and syntax units in Istm language models. In NAACL-HLT, 2019. URL https: waaaarvvv.ooo www.ii.orrg/aooo wwtpoo/gg////////eeeeeeeep 07435. pdf.
[19] Yair Lakretz, Dieuwke Hupkes, Alessandra Vergallito, Marco Marelli, Marco Baroni, and Stanislas Dehaene. Mechanisms for handling nested dependencies in neural-network language models and humans. Cognition, 213:104699, 04 2021. doi: 10. www.0///f..ooniiiiooo.o.ooo 2021. 2021.104699, URL https::/p///.m.... http/arxvvvvor/.. wttpp/atwwwppppss///aappp httpapper//0000/000 11098 pdf.
[20] Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation ex- traction via reading comprehension. Proceedings of the 21st Conference on Computa- lional Natural Language Learning (CoNLL. 2017), pages 333-342, Vancouver, Canada, Au- gust. 2017. Association for Computational Linguisties. doi: 10. 1011883/11/K11-10000 URL https: http:///claaahhhhocc
wtg////-g/00 K17-11344.
[21] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual knowledge in gpt. In NeurlPS 2022, 2022. URL https:: http:s///rxivv.ooo// http:iiwwwrg/dd///////// /2202. 05262.pdf.
[22] Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022. URL https:
//arxiv. wwwexiv.rrg//... httpp/dff222000 2220/221..000 07229.22222 pdf.
[23] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations vector space. In arXiv preprint arXiv: 2013. URL https: //arxiv. wwt.oogg///oooo httrg//pf/3300g /atfff//1...00 3781........pp.
[24] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. arXiv preprint arXi::2110.11309 2021. URL
httpp/////..../////..//.......// 1103339.0 pdf.
[25] Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory-based model editing at scale. In International Conference on Machine Learning, pages 15817-15831. PMLR, 2022. URL https: wwwpiaaivvv...a//aaa/aaaa wttp://vvrgppppp////eee//////// p2tpp//t..2222.///////// 06520. pdf.
[26] Jesse Mu and Jacob Andreas. vances Neural Information Processing Systems, 33:17153-17163, 2020. URL c 67996ff3haaeecccc77777727772
Compositional explanations of neurons. Ad-
httpps:/prooeedd:
[27] Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018. doi: 1e//1////iiioo00
http://ddistlllpub// bhuilddigg-bocks.
[28] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNN) IJCNLP), pages 2463-2477 Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10.18653/v1] 019-12550. URL https: /tt.tpss///cclaattooooooooo http:///aeeeaattoooooooorrr/////
[29] Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to generate reviews and discover- ing sentiment, arXiv preprint arXiv: 2017. https htappp:://rree //tpx/aaavv///////////.//..aa org wwwg//pt....oooo.o... / 1704. 01444 444. pdf.
12

---

[30] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei. llya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019. ront. al o net /tt///bettrerreee mooe l _rresusuuprrreerrrreerrrr wttp
URL htt.:tttss///mmccfpkkksttttt
[31] Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computationaal Linguisties, 8: 842-866, 2020. doi: 10.1162/tac_ a 00349. URL hitps: /aclaantoolog..org//000000 tacl-1.54.
[32] Shibani Santurkar, Dimitris Tsipras, Mahalaxmi Elango, David Bau, Antonio Torralba, and Aleksander Madry. Editing a classifier by rewriting its prediction rules. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 23359-23373, Curran Asso- ciates, Inc., 2021. URL https: httpcttpss/ppccceeiinniooooooeee wttp::ssss......aeeee//22... o.5494222a33eecct388811502222222 Papeee http://a/p///////////...
[33] Johannes Schneider and Michalis Vlachos. Explaining neural networks by decoding layer activations. In International Symposium on Intelligent Data Analysis, pages 63-75. Springer, 2021. URL http::
: / /aaaaiiivvoo// wttf/////////////////////////// pdf 22000000.1111 13630 pdf.
[34] Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreting neural nlp: The case of gender bias. arXiv preprint arXiv:2004.12266 2020. URL https:
//arxiv.crgg////ee..e. /dff//df//// 2200. 12265.pd5 pdf.
[35] Ben Wang and Aran Komatsuzaki. GPT-1-6B: A 6 Billion Parameter Autoregressive Language Model. https://github: com wwwggoolll//mss-rrssttttttt May 2021.
jax.
[36] Xiaozhi Wang. Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. Finding skill neurons in pre-trained transformer-based language models. arXiv preprint arXiv:2211.073493 2022. URL httttts/s///xxiiii org/ / 22211 00.00.
[37] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. Springer, 2014. URL https:
In European conference on computer vision, pages
/ //axiiv.. wwt/oorg//..ooooooooo.......po.. 11111111111 2901.9911.. pdf.
[38] Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein. Invertible concept-based explanations for enn models with non-negative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021. URL https: //arxiv. wewiiiiorrg//o pdf 2 2200 10.175547... pdf.
[39] Sumu Zhao, Damián Pascual, Gino Brunner, and Roger Wattenhofer. Of non-linearity and commutativity bert. In 2021 International Joint Conference on Neural Networks (IJCNN). pages 1-8. IEEE, 2021. URL https: www.aiwwwww..aaaaoaooaoope
04547.pp40
[40] Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. Interpreting deep visual representa- tions via network dissection. IEEE transactions on pattern analysis and machine intelligence, 41(9):2131-2155. 2018. URL https:/ http://axvvvvve/ /ttp///ii..oopp//////1111// httpp/fff1111//
05611 pdff.
[41] Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposi- tion for visual explanation. In Proceedings of the European Conference on Computer Vi- sion (ECCV), pages 119-134, 2018. URL https: http:://ppppll..m csail mitt. edu/bzhou/ publication/ecov IBB.
[42] Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.00361. 2020. URL https: //arxii
/2012. 00363. pdf.
13

---

A Experiment Details
Data Licenses. CounterFact is available by the MIT license at
com/ kmeng01/ rome [21], and ZSRE is available publicly at http://nlp. cs www.swwwshnnnnn..... edu/ http//eroshot///o [20].
Data Filtering. We filter the CounterFact dataset subset of facts that are correctly completed by GPT-J, in order to ensure that there is knowledge to localize in the model for each point. We mark completion correct when Ovre appears among the first 36 tokens sampled from the model given the prompt P using greedy decoding. GPT-J achieves completion accuracy of 32.6% under this scheme, and after starting with about 10% of the CounterFact dataset, our final sample size n 652. We perform additional filtering specifically for model editing in the Fact Erasure condition, where we filter points to have a target probability pe(Onua (s, r) at least .02, so that there is reasonable amount of probability mass to be erased. In this condition, we have 489 points.
Compute. Experiments were on single NVIDIA A6000 GPU with 48gb memory. Computing editing performance for n 652 points with GPT-J for a single edit method applied across model layers the set [1, 5, 9, 13, 17, 21, 24, 28} could take about eight hours. Saving causal tracing or representation zeroing results for these datapoints takes about twelve hours. Regression analyses and plots can be made on demand (code in supplement) given the data from the editing and localization experiments.
Edit Method Tuning. We tune the edit methods have high rewrite scores while not trading off too aggressively against paraphrase and neighborhood scores. More specifically, this means we tune methods to have rewrite scores higher than 99% (note methods score), separately for each editing problem variant. The tuning is done with the first 100 points of the CounterFact dataset, editing layer for GPT-J and 18 for GPT2-XL. For ROME and MEMIT methods, we tune over the KI. regularization weight values in the set [.0625, .9, 1]. For constrained finetuning, tune the To norm weight values in the set (le-4, 5e-5, 2e-5, 1e-5). For both methods, we adopt default parameters from Meng et al. [22] unless otherwise stated. We describe the relevant hyperparameters below, for GPT-J first:
easily get above 99% rewrite
Error Injection. FT-1: norm constraint of le-4. FT-5: norm constraint of 2e-5. ROME: regulariza- tion weight of 1. MEMIT: regularization weight of 0.9. Tracing Reversal. FT-1: Norm constraint of le-5, FT-5: Norm constraint of 2e-5. FT-5: 2e-5. ROME: default parameters. MEMIT: default parameters.
3. Fact Erasure. FT-1: norm constraint of le-5. FT-5: norm constraint of le-5. ROME: default parameters. MEMIT: default parameters.
4. Fact Amplification. FT-1: norm constraint of le-5. FT-5: norm constraint of 1e-5. ROME: default parameters. MEMIT: default parameters.
5. Fact Forcing. Note that for all methods we decide to increase the number of gradient steps, as convergence takes longer for finetuning (from 25 to 50 steps) and for the gradient-based optimization for v* in ROME (from 20 to 25 steps). FT-1: norm constraint of lc-4. FT-5: norm constraint of lc-4. ROME: 25 gradient steps for finding v². MEMIT: default parameters (already set to 25 steps).
We run only the Error Injection and Fact Forcing conditions for GPT2-XL. Hyperparameters are as follows:
1. Error Injection. FT-1: norm constraint of 1c-3. FT-5; norm constraint of 1c-4. ROME: default parameters. MEMIT: default parameters.
2. Fact Forcing. FT-1: norm constraint of 5e-4. FT-5; norm constraint of 5e-5. ROME: default parameters. MEMIT: default parameters.
B Additional Results
ZSRE Dataset. Here, we describe experiments with the 7SRE dataset, which is commonly used
in past editing method papers [9, 24]. ZSRE includes naturalistic questions rather than prompts
14

---

Bowe tejucton Routtz Soces by Layer
Ere Alaciin Punphias Sour EZR Loper
fijecion Niigh betwod Scou E&d Lager
Le de
10
0.00
Volci
MENIT
(Omoaad) SocC Lage
Figure 7: Edit success metrics for our four editing methods, under the Error Injection objective. Left: Rewrite, Center: Paraphrase, Right: Neighborhood.
Error Injection Soure by Edit Layer
0.6
Mothel
HT (wee))
FT (WS-5)
ROME (wocl))
MEMIT (we=))
(Central) Edit Layer
Figure 8: Essence score by edit layer, for our four editing methods, under the Error Injection objective.
intended for autoregressive cloze completion, as in CounterFact, Following past work [21], we use GPT-J to answer ZSRE questions in zero-shot manner, and we edit the model with ROME. We report results for ZSRE via plots of edit success vs. tracing effect in Figs. 19 (rewrite score) and 20 (overall score), accompanied by regression analysis results in Table 8. We find that results with ZSRE match our conclusions with CounterFact, as the results are quite similar to plots and regressions with CounterFact data. Tracing effects are not predictive of edit success.
Representation Zeroing, Representation zeroing is a common localization technique where neural activations are manually set to zero during model forward pass [18, 2]. We implement a form of representation zeroing that is exactly like Causal Tracing, except instead of denoising already-noised representations, we set clean representations to zero. Specifically, we simply run normal forward
Error Injection Overall Score by Edit Layer
Error Injection Overall Score (+Esseecce by Edit Layer
1.0
0.3
Oveall Score
0.6
0.6
Method
Mastall
(wsml)
0.4
ROME
(wes)))
0.4
ROME
MEMIT (wes))
MEEIIT (wwe5))
16
30
36
(Central) Layer
(Central) Edit Layer
Figure 9: Overall edit success for our four editing methods, under the Error Injection objective. Left: The mean of Rewrite, Paraphrase, and Neighborhood Scores. Right: the mean score with Essence Score included.
15

---

Table 2: R2 values for predicting ROME edit success in Error Injection, subsetted 10% of the data that has the most concentrated tracing effects in small number of layers. Even when facts appear to be stored at a small number of layers and not other layers, tracing effects are still not predictive of editing performance.
Concentrated Data
R² Values
Method
Layer
Tracing Effect
Both
ROME
0.927
0.02
0.929
pass until certain set of layers (window size=5), where we zero out representation values for the MLP output representations at the subject token indices within those layers (then continue the forward pass). The localization effect is computed as the proportion of the original predicted probability that is deleted via the zeroing operation (ranging from no effect as 0% to 100% of probability deleted as 100%). These new results are shown in Figs. 21 for rewrite score and 22 for overall score, using ROME on GPT-J with CounterFact data. We obtain the same conclusions as our analysis with causal tracing: localization via representation zeroing is not predictive of edit success. Specifically, we see correlations between edit success and localization effect to be near zero across layers (using either rewrite score or overall score for edit success).
Highly concentrated tracing effects. Since Causal Tracing analysis suggests that information accrues gradually across layers (see Fig. 10), it seems possible that information is simply so diffusely spread across model layers that no matter what layer you edit, you will be editing a layer where a fact is at least stored in part. Based on this observation, we want to test whether tracing effects correlate better with edit success specifically when tracing effects are concentrated in small number of layers. This condition represents that fact appears to be stored in small number of layers and not elsewhere. We hope that by editing in that range of layers, we can more easily manipulate that fact. To identify points with concentrated tracing effects, we use a heuristic for filtering points. Given the output of Causal Tracing analysis for point, i.e. one effect per layer (the max across tokens), we define the point to have concentrated tracing effects when there are no more than three layers that have at least 50% of the maximum effect across layers (besides the layer with the max effect itself). Under this criterion, about 10% of the data (74 of 652 cases) have concentrated effects. Note our default tracing window size of 5 with the 28 layer GPT-J model for this experiment.
We show the results from our analysis on this data subset in Table 2, and we observe no changes in our main conclusions. For ROME with Error Injection, the added effect is 0.2%. Across editing problems and edit methods, the maximum added effect of including tracing effects on R values for predicting rewrite score remains at 3.2% (for Fact Forcing with constrained finetuning). Thus, we conclude that even when facts appear to be stored in small number of layers, localization results from Causal Tracing are still not informative about editing success, while the choice of edit layer is a far more important factor in whether a fact is successfully edited.
Measuring essence drift. Meng et al. [21] describe one possible consequence of model editing as essence drift,, which occurs when core properties of an entity change after attempting to edit only one property of that entity. For example, changing where an island is located might also cause the model to nonsensically treat the island as university campus (see example in Meng et al. [21]).
We aim to obtain an automatic metric to serve as a rough proxy for essence drift. A related metric is calculated with "Local Neutral" data involving the same subject entity but with other properties that are logically neutral with the original property of the subject being edited [15]. However, we do not have "Local Neutral" data for the CounterFact dataset, and essence drift aims to specifically measure changes to core properties of a subject.
Therefore, we automatically estimate changes to known properties of the subject by calculating the change in model perplexity over samples of text that were drawn from the pre-edit model given the prompt "s is a " (which tend to describe a number of key properties of the subject s). We term these samples essence lexts, and we obtain five samples per subject prompt by sampling with multinomial top-k sampling using k- = 5. Given our essence texts, we measure the perplexity over the samples before and after editing a fact in the model, for every edited fact in our dataset. Note this is quite similar to the essence drift regularization objective used in the ROME optimization objective [21]]
16

---

Causal Tracing larger effects when multiple layers denoised
Toxing Winder
Trucing Witedow
"Tocrny Wincoww
Thatieg Windin Stoc:
0.3
HOME MEMIT Logn
RORE Edte MIXKFTT Edda L-pars
RONNE Lawer MEEAIT Lapers
BOSAI Edde MELIT Fu81 Lppcs
Donuhiing E.feet
0.2
14
Layer in GPT)
Figure 10: Tracing effects grow larger as the number of adjacent restored layer representations increases (tracing window size).
Rewrite Score Table
R Values
Editing Problem
Method
Layer
Trace
Both
Diff
p-value
FT
layer)
0.756
0.062
0.758
0.002
<lc-
FT
(5 layers)
0.775
0.055
0.777
0.002
<lc-4
Error Injection
ROME
(1 layer)
0.947
0.016
0.948
0.001
<le4
MEMIT
(5 layers)
0.677
0.024
0.678
0.001
0.199
(1 layer)
0.067
0.067
0.997
FT
(5 layers)
0.751
0.045
0.752
0.001
0.032
Tracing Reversal
ROME
(1 layer)
0.294
0.017
0.31
0,015
<1e4
MEMIT
(5 layers)
0.212
0.036
0.218
0.006
<
FT
(1 layer)
0.643
0.028
0.646
0.003
<le4
FT
(5 layers)
0.698
0.025
0.70
0,002
<1e4
Fact Erasure
ROME
(1 layer)
0.857
0.019
0.858
0.555
MEMIT
layers)
0.925
0.019
0.925
0.669
FT
(1 layer)
0,383
0.014
0.393
0.01
<le4
FT
(5 layers)
0.424
0.01
0.436
0.011
<1e-
Fact Amplification
ROME
layer)
0.88
0.02
0.88
0.654
MEMIT
layers)
0.905
0,018
0.906
0.001
<1e4
FT
(1 layer)
0.697
0.104
0.724
0.027
< <1e4
FT
(S layers)
0.634
0.10
0.666
0.032
<le4
Fact Forcing
ROME
layer)
0.422
0.004
0.425
0.003
<le4
MEMIT (5 layers)
0.345
0.041
0.354
0.009
<1e4
Table 3: 1t2 values for predicting rewrite score from choice of edit layer and tracing effect, across editing problem variants (corresponds to data in Fig. 6). Diff shows the added effect of including tracing in the regression (Both vs. Layer Only), in terms of R22 and p-value shows the results from an F-rest comparing the Both and Layer Only models. Tracing has some predictive value for Fact Forcing, but the R² value remains small compared to the choice of edit layer.
but we consider it as metric here. We scale the change in perplexity to fraction of 5, with the cut-off of 5 chosen to represent a maximally bad change to the model perplexity. Similar to our other metrics, our essence score is if model perplexity on the essence texts does not change after editing the model (capping increases by 5 or more.
in cases of slight decreases perplexity). and it is 0 if the perplexity
We show essence scores for editing methods across layers in 8. Interestingly, the trend across layers for this metric is mostly counter to the trends for other metrics (Fig. 7), with editing later layers being generally preferable editing earlier layers. As result, when combined with the other metrics in Fig. 9, we see that the overall score trend flattens and shifts slightly toward mid-range layers in the model.
C Robustness Experiments
In addition to our main results with ROME for GPT-J and our Rewrite Score metric, we include robustness experiments to confirm that results are similar for (1) other measures of edit success including Paraphrase Score, Neighborhood Score, and Overall Score (Tables 4, 5, and 6), (2) different
17

---

Table 4: R2 values for predicting paraphrase score from choice of edit layer and tracing effect, across editing problem variants. Diff shows the added effect of including tracing in the regression (Both vs. Layer Only), in terms of R and p-value shows the results from an F-test comparing the Both and Layer Only models. The added effect of including tracing effects is very small across conditions (less than 3%).
Paraphrase Score Table
R² Values
Editing Problem
Method
Layer
Trace
Both
Diff
p-value
FT
(1 layer)
0.061
0.005
0.063
0.002
0.258
FT
(5 layers)
0.036
0.003
0.038
0.001
0.582
Error Injection
ROME
(1 layer)
0.279
0.001
0.303
0.024
< le-4
MEMIT
(5 layers)
0,246
0.269
0.023
<le-4
FT
(1 layer)
0.004
0.001
0.004
0
0.989
FT
(5 layers)
0.001
0.002
0.001
0.841
Tracing Reversal
ROME
(1 layer)
0.01
0.012
0.002
0.121
MEMIT
(5 layers)
0.001
0.001
0
0.997
FT
(1 layer)
0.046
0.001
0.048
0.002
0.303
FT
(5 layers)
0.079
0.007
0.084
0.005
0.004
Fact Erasure
ROME
(1 layer)
0.537
0.012
0.539
0.001
0.218
MEMIT
(5 layers)
0.586
0.015
0.587
0.001
0.184
FT
(1 layer)
0.005
0.012
0.022
0.017
<1c-4
FT
(5 layers)
0.017
0.013
0.035
0.018
<le4
Fact Amplification
ROME
(1 layer)
0.24
0,002
0.267
0.027
<le4
MEMIT
(5 layers)
0.236
0.001
0.263
0.026
<le4
FT
(1 layer)
0.044
0.004
0.046
0.002
0.367
FT
(5 layers)
0,023
0,002
0.025
0.002
0.387
Fact Forcing
ROME
(1 layer)
0.357
0.01
0.36
0.003
0.003
MEMIT (5 layers)
0.095
0.001
0.105
0.01
<1c-4
Table R2 values for predicting neighborhood score from choice of edit layer and tracing effect, across editing problem variants. Diff shows the added effect of including tracing in the regression (Both vs. Layer Only), in terms of R², and p-value shows the results from an F-test comparing the Both and Layer Only models. The added effect of including tracing effects is very small across conditions (2% or less).
Neighborhood Score Table
R² Values
Editing Problem
Method
Layer
Trace
Both
Diff
p-valuc
FT
(1 layer)
0.005
0
0.008
0.002
0.197
FT
layers)
0.014
0.001
0.015
0.001
0.55
Error Injection
ROME
layer)
0.011
0.003
0.015
0.005
0.001
MEMIT
layers)
0.004
0.001
0.006
0.002
0.154
FT
layer)
0.001
0
0.001
layers)
0.001
0.002
0.001
0.946
Tracing Reversal
ROME
layer)
0.001
0.002
0.001
0.946
MEMIT
(5 layers)
0.001
0
0.002
0.981
FT
layer)
0.01
0.014
0.004
0.037
FT
(5 layers)
0.01
0
0.013
0.004
0.06
Fact Erasure
ROME
(1 layer)
0.04
0.005
0.046
0.006
0.001
MEMIT
(5 layers)
0.05
0.007
0.059
0.009
<
layer)
0.012
0.009
0.02
0.008
<1e4
FT
(5 layers)
0.016
0.008
0.025
0.009
<1e4
Fact Amplification
ROME
(1 layer)
0.04
0.01
0.05
0.01
<
MEMIT
layers)
0.035
0.008
0.044
0.01
<le4
FT
layer)
0.054
0
0.057
0.003
0.03
FT
(5 layers)
0.019
0.001
0.022
0.004
0.011
Fact Forcing
ROME
(1 layer)
0.299
0.022
0.311
0.012
<le4
MEMIT
layers)
0.046
0.012
0.066
0.02
<le4
18

---

Causal Tracing peak distribution shifts outward lower window
Traving Window Stee
Wiedow
Traiing Windew Siar
Thaiing Windiw Stoe 10
ager MESIT FUBIT
Laver MIFMIT Fitx
SOIER SuRTT gor
BONE Lawer Fi
150
Chant
100
29
Layer in GPT-J where Causal Traving effects peak
Figure 11: Each individual plot shows the distribution of tracing curve peaks (the argmax layer) across datapoints, using a different tracing window size. Together, the plots show how the distribution of layers where the tracing curves peak for each point shifts outward toward the first and last layer of the model as the tracing window size declines. This is primarily due to clipping effect from using window size greater than 1. The way tracing values are computed, a window size of 10 implies that the effect for "layer 1" is from restoring layers 1-5, while the effect for layer "layer 5" is 1-10. As result, a tracing window size of 10 favors layer 5 over layers 1-4, and reducing the tracing window size leads to these clumps of effects shifting from layer 5 toward layer (and from layer 24 to layer 28)
Tracing Window Size 10: Tracing effects are very weakly predictive of edit success
Tracing Revenal
Fact Amplification
Fact Erasume
Fact Forcing
1.00
8S75MP e5.
0.75
Ra 0.550
0.25
0.00
FT-1
FT-5
ROME MEMIT
FI-1 FT-S ROME MEMIT
FT-1 FT-5 ROME MEMIT
FI-1
FT-5 ROME MEMIT
Explanatory Variablo(s):
Layer
Layer
Tracing Effect
Figure 12: The results of our R² analysis for predicting rewrite score are nearly identical between using a tracing window size of 5 (shown in Fig. 6) or 10 (shown here).
values of the tracing window size (Fig. 12), (3) GPT2-XL than GPT-J (Fig. 13), (4) the original unscaled metrics from Meng et al. [21] (Fig. 14), and (5) using the tracing effect at the last subject token rather than the max across tokens (Fig. 16). We consider the last subject token effect since this corresponds more directly to the motivation for ROME (see Meng et al. [21]). We expand on each of these experiments below:
Results for Paraphrase, Neighborhood, Overall Metrics. We recreate our regression -based analysis across editing problem variants and editing methods using paraphrase score and neighborhood score as our outcomes rather than Rewrite Score, as well as an Overall Score that is the raw average of the three edit scores. These results are shown in Tables 4, 5, and 6 respectively. Similar to our analysis with rewrite score, these tables show that tracing effects are barely predictive of edit success at all. For paraphrase score, the largest gains in R values around 0.03 (relative to the layer-only regression model), and for neighborhood score, the largest gain is 0.02. The largest gain for overall score is 0.02 for Fact Forcing with constrained finetuning. Our overall conclusion remains that tracing effects almost totally unrelated to edit success across editing problem variants, including for different edit success metrics.
Results for Different Tracing Window Sizes. We repeat our analysis from Sec. using tracing effects obtained from larger tracing window size of 10, to match the value used in Meng al. [21]. Note that from Fig. 10, we know that the tracing effects grow larger as more adjacent layer representations are restored. When we recreate our main I²² analysis using tracing effects with window size 10 (shown in Fig. 12), we find that results are nearly identical to those shown in Tables 3, 4, and 5.
Results for GPT2-XL. We rerun our analysis with GPT2-XL, a 48 layer model [30], while editing layers in the range [1, 5, 9, 13, 17, 18, 21, 25, 29, 33, 37, 41, 45, 48). Here, we use tracing window size of 10, and we limit our experiments to focus on Error Injection and Fact Foreing editing problems. As seen in Fig. 13, we find very similar trends when explaining rewrite score in terms of the choice
19

---

Table 6:
R2 values for predicting overall score (raw average of rewrite, paraphrase, and neighborhood scores) from choice of edit layer and tracing effect, across editing problem variants. Diff shows the added effect of including tracing in the regression (Both vs. Layer Only), in terms of R², and p-value shows the results from F-test comparing the Both and Layer Only models. The added effect of including tracing effects very small across conditions (2% or less).
Ovr. Edit Score
R² Values
Editing Problem
Method
Layer
Trace
Both
Diff
p-value
FT
(1 layer)
0.642
0.054
0.643
0.002
0.001
FT
(5 layers)
0.663
0.047
0.665
0.002
0.001
Eroo Injection
ROME
(1 layer)
0.62
0.003
0.629
0.009
<1c-4
MEMIT
(5 layers)
0.525
0.008
0.534
0.009
<Ic-4
FT
(1 layer)
0.294
0.025
0.296
0.002
0.054
FT
(5 layers)
0.751
0.045
0.752
0.001
0.032
Tracing Reversal
ROME (1 layer)
0.296
0.016
0.31
0.014
<1c-4
MEMIT
(5 layers)
0.21
0.036
0.216
0.006
<le-4
FT
(1 layer)
0.28
0.007
0.283
0.004
0.008
FT
(5 layers)
0.119
0
0.124
0.004
0.015
Fact Erasure
ROME
(1 layer)
0.718
0.023
0.718
0
0.729
MEMIT
(5 layers)
0.794
0.025
0.794
0
0.555
FT
(1 layer)
0.188
0.003
0.199
0.011
<le.
FT
(5 layers)
0.224
0.002
0.236
0,013
<le4
Fact Amplification
ROME
(1 layer)
0.583
0.005
0.59
0.007
<le4
MEMIT (5 layers)
0.597
0.005
0.607
0.01
<1c4
FT
(1 layer)
0.487
0.056
0.5
0.013
<lc-
FT
(5 layers)
0.459
0.057
0.475
0.017
<le4
Fact Forcing
ROME (1 layer)
0.285
0.004
0.291
0.006
<le4
MEMIT (5 layers)
0.226
0.017
0.227
0.001
0.419
GPT2-XL: Tracing effects are very weakly predictive of edit success
Forcing
1.00
0.75
61.5% PMES
R 20.50
0.25
18.5
FT-L
FT-10
ROME
MEMET
FT-1
FT-30
ROME
MEMIT
Explanatory
Variable(s):
Layer
Layer + Tracing Effect
Figure 13: Like with GPT-J, tracing effects are very weakly predictive of edit success across editing problem variants for GPT2-XL while Fact Forcing shows the largest relationship. Relative to the R of a model predicting rewrite score based on the choice of edit layer (blue), a model with edit layer and tracing effects (orange) improves the n² by at most .02 points for Fact Forcing. The choice of edit layer explains much greater share of the variance in rewrite score.
of edit layer and the tracing effect at that layer. The largest explanatory effects in terms of R² are observed for Fact Forcing with constrained finetuning, but these effects remain small at about 2%.
Results for Unscaled Metrics. We repeat our analysis using the original editing metrics and absolute tracing effects from Meng et al. [21]. Their rewrite magnitude the absolute difference between the probability of the new target Ofaise and the old true target Onne after editing, Po- (o-(Oojtsee (s, r)
po- (Omnel |s, r). The tracing effect is the absolute tracing effect, pa (Otruel Snolses T, V((.,))
- -Po(Ome (Snoises r), measured at the last subject token index. We adjusted our rewrite and tracing metrics to (1) rely only on the target output probability, rather than difference in probabilities of two different targets which might not appropriate for our different editing problems, and (2) to always fall between O and 1 for better comparability between datapoints, since absolute tracing effect are bounded by the original model probabilities. However, we reach the same conclusions from our analysis when using the original editing metrics. We show example for rewrite magnitude and the
20

---

ROME Rewrite Magnitude by Abs. Tracing Effect
1.00
Rewrile Magnitale
0.50
0.25
0.00
0.00
0.05
0.10
0.15
0.20
0.25
Last Subject Token Tracing Effeet (Absolute Difference)
Figure 14: Editing vs. tracing results for ROME at layer 6 for Error Injection, using the un-rescaled rewrite and tracing metries from Meng et al. [21]. Here, rewrite magnitude is the difference between the probability of the new target Ofatse and the old true target Onee af- ter editing, pg- (ofatsel r) Pg- (Orne [s, r). The tracing effect is the absolute tracing effect,
Po(Om Snoise H, ((t,)) Po (O Snoise: F), measured at the last subject token index. The correlation here is near zero, at p .006.
Edit Metric
Regression Metric
Predictor(s)
Value
Layer
0.947
R
Tracing Effect
0.016
Layer
0.073
Rewrite Score
RMSE
Tracing Effect
0.315
Layer
0.02
MAE
Tracing Effect
0.206
Layer
0.618
R²
Tracing Effect
0.003
Layer
0.133
Overall Score
RMSE
Tracing Effect
0.216
Layer
0.11
MAE
Tracing Effect
0.183
Table 7: Additional regression error metrics (for CounterFact and ROME) lead us the same conclusion as our analysis based on R². RMSE root mean squared error, and MAE mean absolute error. Regressions predicting rewrite score (or overall score) from the choice of edit layer achieve much lower prediction errors than regressions using the tracing effect, suggesting that the choice of edit layer is much more important for edit success than the tracing effect.
absolute tracing effect for Error Injection in Fig. 14. The correlation between edit success and tracing effect still near zero.
Results for Last Subject Token Effect. ROME increases the target probability p(o (s, r) by optimizing for new output representation from a chosen MLP layer at the last subject token index. Meng et al. [21] show that this choice of token representation is critical to the success of the editing method, which is a hypothesis directly motivated by the results from their Causal Tracing analysis. In paper, we by default report results using tracing effects that the max across tokens at given layer, for better comparability across the editing methods we use. However, when we repeat analysis using the tracing effect specifically the last subject token index, obtain the same negative conclusions about the relationship between Causal Tracing localization and ROME editing performance. We show the correlations between Rewrite Score and Last Subject Token Tracing Effect in Fig. 16, where we see there are no positive correlations between editing success and tracing results any layer in GPT-J.
21

---

ROME Rewrite Score by Tracing Effect (Error Injection)
Layer
Layer
Layer
Layer
1.00
0.50
0.25
Rewrite Score
0.00 /
Layer 17
Layer 21
Layer 25
Layer 28
1.00
0.50
0.00
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
1.00
0.50
0.75
1,00
Tracing Effect
Figure 15: The relationship between ROME edit success and the tracing effect is near zero at edit layers in the model (for the standard Error Injection editing problem). Red lines show perfect relationships between tracing effect and edit success.
Edit Metric
Regression Metric
Predictor(s)
Value
Layer
0.795
H
Tracing Effect
0.042
Layer
0.158
Rewrite Score
RMSE
Tracing Effect
0.341
Layer
0.072
MAE
Tracing Effect
0.254
Layer
0.654
R
Tracing Effect
0.059
Layer
0.136
Overall Score
RMSE
Tracing Effect
0.223
Layer
0.097
MAE
Tracing Effect 0.188
Table 8: ZSRE regression results lead us to the same conclusion as our experiments on CounterFact, using ROME editing. RMSE root mean squared error, and MAE is mean absolute error. Regressions predicting rewrite score (or overall score) from the choice of edit layer achieve much lower prediction errors than regressions using the tracing effect, suggesting that the choice of edit layer is much more important for edit success than the tracing effect.
22

---

ROME Rewrite Score by Last Subject Token Tracing Effect (Error Injection)
Tayer
Layer
Layer
Lager
1.00
0.75
0.50
0.25
Rewrite Score
0.00
Layer 17
Layer
Layer
Layer
0.75
0.25
0.00
0.00
0.25
0.00
0.25
0.00 0.50
0.25
0.75
Tracing Effect Last Subject Token
Figure 16: The relationship between ROME edit success and the tracing effect at the last subject token. The ROME method edits a fact by changing the output representation for the MLP layer specifically at the token index corresponding to the last subject token. However, editing performance and tracing effect this position still not positively correlate. Note the distribution of points along the axis changes depending the choice of edit layer since the distribution of tracing effects is calculated from tracing effects at that layer.
Fact Forcing Rewrite Score by Tracing Effect (Grouped by Edit Layer)
Layer
Layer
Layer
Layer 13
1.00
0.50
Rewrite Score
0.00
Layer 17
Layer
Layer
Layer
1.00
0.75
0.50
0.25
0.00
0.00
0.50 0.75
Tracing Effect
Figure 17: The relationship between Fact Forcing edit success and the tracing effect for constrained finetuning adjacent layers. "Layerr (" indicates the center of this 5-layer interval, and the dashed red lines show hypothetical perfect relationship between tracing effect and edit success. For many layers, there a noticeable positive relationship between tracing effects and editing success. Yet, (1) there a high amount of variance the outcome, and (2) this variance is largely explained by the edit layer. As a result, tracing effects provide little extra information for predicting edit success beyond the choice of edit layer (about 3% more explained variance; Fig. 6).
23

---

ROME Overall Score by Tracing Effect (Error Injection)
Layer
Layer
Layer
Layer
1.00
0.75
0.25
Overall Score
0.00
Layer
Layer
Layer
Layer
1.00
0.75
0.25
0.00
0.25
0.50
0.50
0.75
1.00
Tracing Effect
Figure 18: The relationship between ROME overall score (average of scores) and the tracing effect is somewhat negative for most edit layers in the model (for the standard Error Injection editing problem). Red lines show perfect relationship between tracing effect and edit success, negative relationship suggests that tracing localization results do not indicate that editing will be successful.
7:RR: Rewrite Score by Tracing Ffffect
Layer
Layer
Layer
Layer
1.00
0.75
0.50
0.25
Rewrite Score
0.00
Layer 13
Layer 17
Layer
Layer
1.00
0.75
0.50
0.25
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1,00
0.00
0.25
0.50
0.75
1.00
Tracing Effect
Figure 19: Additional experiments on the ZSRE dataset show the same results as for CounterFact, using the ROME editing method with rewrite score as our editing success metric (see regression analysis results in Table 8). Red lines show perfect relationship between tracing effect and edit success, so near-zero relationships suggest that tracing localization results do not indicate that editing will be successful.
24

---

ZSRE: Overall Score by Tracing Effect
Layer
Layer
Layer
Layer
1.00
0.75
0.50
0.25
Overall Score
0.00
Layer
Layer
Layer
Layer
0.75
0.50
0.25
0.00
0.50
0.25
0.50
0.75
0.75
0.50
0.75
Tracing Effect
Figure 20: ZSRE experiments using overall score (average of rewriteprraphrseeeighhooooodd scores) as the edit success metric.
ROME Rewrite Score vs Zero Ablation Effect
Layer
Layer
Layer
Layer
1.00
0.75
0.50
0.25
Rewrite Score
0.00
Layer 13
Layer
Layer
Layer
1.00
0.75
0.50
0.25
0.00
0.00
0.25
0.50
0.75
0.25
0.50
0.75
1.00
0.50
0.75
1,00
0.25
0.50
0.75
1,00
Zero Ablation Effect
Figure 21: Additional experiments with representation zeroing the localization method show the same results as for Causal Tracing, using the ROME editing method and rewrite score as the edit success metric. Red lines show perfect relationship between representation zeroing and edit success, so near-zero relationships suggest that representation ablation localization results do not indicate that editing will be successful.
25

---

ROME Overall Score vs Zero Ablation Effect
Layer
Layer
1.00
0.75
0.50
0.25
Overall Score
0.00
Layer 13
Layer 17
1.00
0.75
0.50
0.25
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Zero Ablation Effect
Figure 22: Additional experiments with representation zeroing as the localization method show the same results as for Causal Tracing, using the ROME editing method and overall score as the edit success metric. Red lines show perfect relationship between representation zeroing and edit success, near-zero relationships suggest that representation ablation localization results do not indicate that editing will be successful.
26


Reviews:
[{"review_id": "6seDrRAwLh", "reviewer": "Reviewer_q6TL", "summary": "This paper investigates the assumption that we can edit the located component to modify the facutal knowledge stored in the language model. The assumption might be false due to the empirical findings of this paper: (1) many knowledge facts stored outside the located parts of the model, and  (2) the near-zero correlation between causal tracing edition and edit success.\nThis paper further proposes a variant Fact Forcing to improve the correlation between location and editing, while finds that editting the early and middle layers still performs better.", "questions": "See 1 and 2 in weakness.\n\n3.\tHow do you view the storage of factual knowledge in GPT? According to the paper, should it be stored among a series of interconnected neurons? How should we design editing methods to address this characteristic? In other words, what insights does your conclusion provide for editing methods?\n", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThis paper challenges the conventional belief that knowledge localization and editing are strongly correlated. It provides a new and interesting perspective that suggests the relationship between knowledge localization and editing may not be as strong as previously thought.\n2.\tThe paper provides a comprehensive overview of the locate-then-edit approach for knowledge editing.\n3.\tThe experiments conducted are detailed and thorough. The paper explores different localization methods, editing targets, and variations of editing methods. This comprehensive analysis provides valuable insights into the effectiveness and limitations of various approaches.\n", "weaknesses": "1.\tThe paper mentions a difference between the metrics used and Rome but lacks a specific explanation. Can you explain why your metrics differ from Rome?\n2.\tThe paper discusses causal tracing methods in relation to fact forcing for editing goals but lacks tracing methods related to other editing goals. Can you consider incorporating tracing methods related to editing problem variants?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review! We have a few comments below.\n\n>  Can you explain why your metrics differ from Rome?\n\nCertainly. (Note: we also check that our results hold with original ROME metrics, see Line282 and Fig 14.) \n\nSo we make two changes from the ROME metrics, which are both designed to make the metrics more comparable across datapoints (mentioned briefly in Lines193-4). First, we drop the dependence on the “true label” from the original fact tuple (s,r,o). This is because ROME metrics like Efficacy Magnitude are not really comparable across datapoints, since for one datapoint the metric might take values in [-.5,.5] (when the post-edit p(o) is .5), while for another datapoint EM might take a value in [0, 1] (when the post-edit p(o) is 0). The fact that the metric takes a different range for different data makes it hard to reasonably aggregate scores across data. \n\nNow we could just use the post-edit p(o^c) as our metric (for counterfactual output o^c). But this runs into the issue that, if the pre-edit p(o^c) is 1, and our post-edit p(o^c) is .99, we’d be giving an editing method a high score for lowering the probability of the new target. Hence, we subtract off the pre-edit probability of p(o^c), then rescale to the [0,1] range to ensure comparability across datapoints (our final Rewrite Score). Altogether, we give an edit a score of 0 if it has no effect on the target probability, and we give it a score of 1 if it has the maximal desired effect on the target probability.\n\n> Can you consider incorporating tracing methods related to editing problem variants?\n\nIt’s true that, besides Causal Tracing and Representation Zeroing, there could be tracing methods that look more like Fact Editing rather than Fact Forcing. In fact, one might take some inspiration from other interpretability methods that explain why a model outputs Y rather than Y’ for a given input, in terms of feature importance scores, like “Interpreting Language Models with Contrastive Explanations” (https://arxiv.org/pdf/2202.10419.pdf). This would be an interesting direction for future work on new localization methods, though we ultimately don’t think it would improve ROME editing scores. As a part of our view, we really think that the focus on new localization vs new editing methods will depend on the goal of the research: do you want to know how a model currently carries out some function, or how to change the model to perform some new function?\n"}, {"review_id": "ChzBVhhI0I", "reviewer": "Reviewer_UfPL", "summary": "This paper investigates whether causal localization, which \"locates\" factual associations within LLMs, provides locations in models which when modified have the greatest effect on those particular factual associations. Counter to intuitions and previous claims in the literature. the authors find that causality-based localization is **much** less predictive (~3 percent vs ~95 percent) than layer-choice for predicting the efficacy of editing factual associations (at MLPs at/across layers). They confirm this behaviour across model classes, localization techniques, layer-depth-windows, datasets and models. This finding is of significant import to the mechanistic interpretability community, as it lends credence to the possibility that models can be pushed toward certain associations in-spite of the presence of existing associations _for those same prompts_ which causal tracing would have identified. Indeed, in some cases using ROME (and other approaches) to inject associations at locations identified as relevant by causal tracing is _less performant_ than injecting them elsewhere.", "questions": "None", "limitations": "The primary limitations would be concerns about generality of these findings. The authors carry out extensive experimental investigations along all likely-to-be-relevant choices of configuration, and address outstanding concerns about the relevance of such findings to interpretability more widely.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper carefully scrutinizes a strongly held belief from previous works about \"localization\" and the storage of factual information in LLMs. Through precise problem formulation and many experimental validations, the authors are able to provide compelling evidence that previous assumptions were incorrect, and that choosing where to inject (/modify) association, versus finding where a model places assocations for use in a forward pass, are _meaningfully distinct_ questions.\n\nPresentation, methodology and supplemental results are well-presented and placed within a coherent narrative. Any questions that arose throughout reading were addressed in the appendices or final discussion.", "weaknesses": "No major or minor weaknesses were identified.\n\n## Nitpicks\n* Lines 222 and 237 are duplicated\n* Line 206 does not appear to relate to the _neighborhood score_, which measures the change in prediction for similar subjects; rather, it seems to be referring to a metric in which $s=s'$ with $s'\\cup s* = \\emptyset$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review! \n\n> Line 206 does not appear to relate to the neighborhood score, which measures the change in prediction for similar subjects; rather, it seems to be referring to a metric in which…\n\nWe meant for this line to describe the purpose of the neighborhood metric from the ROME paper, and we use the same “neighborhood” prompts from the CounterFact dataset to compute it. The s* is a subject that is similar to the original fact subject s only in the sense that they both have relation r to object o (from the original fact (s,r,o)). They are not necessarily similar in any sense besides sharing this property. See page 30 of https://arxiv.org/pdf/2202.05262.pdf for an example. But let us know if we misunderstood what you meant.\n"}, {"review_id": "6QGEVmH5Qx", "reviewer": "Reviewer_FzCS", "summary": "Focusing on recently popular model editing methods, the authors study whether localizing methods based on causal mediation analysis play a decisive role. Based on the sample-wise analysis of the tracing effect and rewriting score, they find some surprising results that model editing performances (rewriting score) are no correlation with the tracing effect and are more dependent on the choice of the layer. They also design other model editing performance metrics to verify their findings. ", "questions": "1. The first question is also 1.4  in the Weaknesses part.\n\n2. As shown in Figure 5 in ROME, the authors also show model editing performance to support their causal tracing method. The metric used in y-axis is the portion cases for P(o^*) > P(o) (Page 7) rather than the sample-wise difference used by the authors. Does the author believe that inconsistency is a result of metric selection?\n\n3. What is the author's opinion on current localization methods like the denoising method? When should one choose such methods? What inherent knowledge in the model did it exactly point out?\n\n4. What is your opinion on the importance of the early layers and what leads it?", "limitations": "Please see Weakness part. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper talks about an important and interesting problem, the stability or robustness of the interpretability metric. This is why I scored it 6. \n\n1. The metrics about localization (tracing effect) and model editing (rewriting score, generalization score, etc) are inconsistent: I had this question that why the authors choose different metrics for tracing effect and model editing performance when I read ROME. Recovering the correct object is different from modifying the original relation between the subject and object. The authors consider this point and also design more metrics to measure model editing performance from different perspectives. \n\n2. The authors conduct comprehensive experiments to support their findings. \n\n3. As mentioned in Summary part, these findings are surprising and will inspire further exploration of localizing and model editing methods.\n", "weaknesses": "1. The main conflict between this paper with ROME is the viewpoint to switching layers, which is also very important to understanding causal localizing for us: \n\n    1.1 ROME says that \"informed by the Zhao et al. (2021) finding that transformer layer order can be exchanged with minimal change in behavior, ......, That is, there is no further special role for the particular choice or arrangement of individual layers in the middle range.\" This is why ROME simply chooses one layer in early layers.  \n\n    1.2 This paper says that \"If this hypothesis were true, it would be surprising because one cannot arbitrarily swap layers in a Transformer model without greatly damaging model performance [39].\"\n\n    1.3 However, the paper [39] just considers BERT model rather than GPT models and conducts experiments on MNLI dataset which is also different from the factual reasoning task. Therefore, I think the authors should conduct more analysis of GPT models and current task to support their viewpoints. \n\n    1.4 I expect more analysis of why there are inconsistencies between causal localizing and rewriting after the authors help reveal them, or some guesses at least. That is why I only give 6. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review! We have a few comments below.\n\n> The main conflict between this paper with ROME is the viewpoint to switching layers…\n\nIt’s true we disagree with the perspective in the ROME paper about how to interpret [39]. The paper [39] shows drops up to 6 percentage points of MNLI accuracy from switching adjacent layers in a BERT model. We think this is a big drop that suggests the order of layers in pretrained language models is important. As you note, the ROME paper says “there is no…special role for the particular choice or arrangement of individual layers in the middle range.” In our opinion, this isn’t really the right interpretation. We agree that both our paper and the ROME paper stretch the interpretation of [39] a bit, since it handles different models and tasks than factual association. However, this is a somewhat minor distinction in the end, since it’s just “motivation” for the ROME method and a discussion point in our paper. The ROME method works fairly well on the CounterFact benchmark, and we show that editing performance can be unrelated to localization results. We leave it to future work to further explore our conjecture in our Discussion setting: “Many layers could store a fact, and it happens that some do” (which we believe would be surprising in light of [39]). \n\n> I expect more analysis of why there are inconsistencies between causal localizing and rewriting after the authors help reveal them, or some guesses at least\n\nTo be clear, this was our goal with Sec. 5 (analysis) and Sec. 6 (discussion).  \n\n> As shown in Figure 5 in ROME, the authors also show model editing performance to support their causal tracing method. The metric used in y-axis is the portion cases for P(o^*) > P(o) (Page 7) rather than the sample-wise difference used by the authors. Does the author believe that inconsistency is a result of metric selection?\n\nWe don’t think the results are sensitive to the precise choice of metric because we replicated our main result with the original Efficacy Magnitude metric from the ROME paper (see ln282 and Fig 14) as well as other editing metrics for paraphrase generalization and neighborhood specificity (see lns280-281, Tables 4, 5, and 6). \n\n> What is the author's opinion on current localization methods like the denoising method? When should one choose such methods? What inherent knowledge in the model did it exactly point out?\n\nWe think that denoising and zeroing localization methods are interesting and potentially useful – they still reveal interesting trends about how models perform tasks (like in Fig. 1 and 2), specifically factual association in our context. That said, we currently think that a counterfactual-input ablation like in the Interpretability in the Wild paper (https://arxiv.org/pdf/2211.00593.pdf) would be better, because it seems like it should remove the task-relevant information in a representation while maintaining other information, like syntactic information. However, other approaches are available, like resampling replacement (see Sec. 3 of https://arxiv.org/pdf/1910.13413.pdf). It will be interesting to see which ablation methods are most useful for model editing or circuit identification going forward. \n\n> What is your opinion on the importance of the early layers and what leads it?\n\nFirst, we think the importance of early layers is slightly overstated due to results like our Figs. 1 and 10. We see that a substantial fraction of facts have their peak MLP effects in late layers, and more generally, the contribution of any single layer is quite limited. That said, there is good evidence that information accrues over time in the residual stream of Transformers (see paper citation: Transformer feed-forward layers are key-value memories), and in this view, it makes sense that it would be important for early layers to set the residual stream on the right track, such that later layers receive the necessary input and add the necessary information to the residual stream. \n"}, {"review_id": "dmoHvDfQCU", "reviewer": "Reviewer_YdCx", "summary": "The paper provides an experimental study on localization claims of causal tracing and editing methods such as ROME. Existing literature uses localization results to decide which layer to edit for knowledge editing in large language models. The paper shows that, surprisingly, localization and edit success are nearly uncorrelated. In fact, one can edit a large range of MLP layers to achieve editing success. Yet, localization suggests that only a few early-mid-layers store the factual information. The paper then proposes new tasks and datasets to reconcile this issue. ", "questions": "From a mechanistic interpretability point of view, could the author(s) provide more thoughts on the effectiveness of causal tracing (in locating which layer *mechanistically* stores the facts)?  In particular, would you argue that the model computation is really not as concentrated as causal tracing suggests? Is it rather a diffuse and incremental process, with lots of redundancy? It seems from the window size = 1 experiment (Figure 10 in Appendix), the picture clearly shows that no single layer gets much tracing effect. Thus, would you agree that the sliding window patching is not principled and could yield a false illusion that the information is sharply concentrated?", "limitations": "The author(s) addressed several limitations already. I believe it's adequate.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This is a very solid piece of empirical study. I find the conclusion convincing, as it is well supported by the experimental evidence.\n\nThe conceptual contribution is clear and novel. In particular, the uncorrelatedness of localization and edit success, as far as I know, has not been noticed by the literature. In fact, this is a somewhat surprising observation given the line of prior work such as ROME.\n\nThe paper is well-written. ", "weaknesses": "I do not find major weakness of the paper. I have some minor comments and suggestions.\n\nThe Appendix contains additional experiments on zero ablation. Another technique one can try is mean ablation and resampling ablation, over a known datatset such as CounterFact. This may be more principled than zero ablation. See the [Causal Scrubbing](https://www.lesswrong.com/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing#4_Why_ablate_by_resampling_) work for some discussion. I believe this wouldn't make much difference here, but it's worth trying.   \n\nFigure 1: Is this produced by aggregating over the CounterFact datasets? Would it be possible to provide a fine-grained look into this? What are the facts that appear to be stored at later layers?\n\nLine 145: Here, the noise level is chosen to be 0.094 universally. However, in ROME, this is chosen to be three times the observed standard deviation of embedding norms as sampled over a body of text. Note that this quantity depends on both the data and the specific (trained) model. Should we also follow the convention of ROME here, to make a fair comparison? In my experience, causal tracing is somewhat sensitive to the level of noise.  (I doubt it would change the conclusion, but it is worth looking into.)\n\nLine 353: It might be worth a quick footnote here that not all examples have their tracing effect concentrated at the last subject token index. See Figure 11 of the original ROME paper for some natural instances.\n\nFigure 10 in Appendix: y-axis should be Tracing Effect rather than Denoising Effect.\n\nYou used representation zero-ing in Appendix but representation zeroing in the main paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review! We have a few comments below.\n\n> The Appendix contains additional experiments on zero ablation. Another technique one can try is mean ablation and resampling ablation, over a known dataset such as CounterFact. This may be more principled than zero ablation. See the Causal Scrubbing work for some discussion.\n\nThanks for the suggestion. We agree this is an interesting kind of ablation for hidden states that would be worth trying in a localization method. Currently, we think a counterfactual-input ablation like in the Interpretability in the Wild paper (https://arxiv.org/pdf/2211.00593.pdf) would be ideal, because it seems like it should remove the task-relevant information in a representation while maintaining other information, like syntactic information. However, it seems like there is theoretical support for the resampling ablation approach (see Sec. 3 of https://arxiv.org/pdf/1910.13413.pdf), and it would be hard to obtain good “neutral” counterfactuals like in the Interpretability in the Wild paper in general (and in particular, we are not sure how to design such counterfactuals for factual association). Either way, we are a little concerned about the current noising ablation and zero ablation methods that have been popular to date, because these can produce OOD inputs to model layers (see a related discussion in Sec. 4 of https://arxiv.org/pdf/2106.00786.pdf). It will be interesting to see which ablation methods are most useful for model editing or circuit identification going forward. \n\n> Figure 1: Is this produced by aggregating over the CounterFact datasets? Would it be possible to provide a fine-grained look into this? What are the facts that appear to be stored at later layers?\n\nYes this is from aggregating traces across our subsample of known facts in the CounterFact dataset (known to GPT-J). We did some qualitative analysis of our own but nothing made it into the paper because we couldn’t identify any noticeable trends. For some reason, many points have primarily late-layer MLP effects! This is one reason we think that localization methods remain interesting and potentially useful – they still reveal interesting trends about how models perform tasks. \n\n> Line 145: Here, the noise level is chosen to be 0.094 universally. However, in ROME, this is chosen to be three times the observed standard deviation of embedding norms as sampled over a body of text…Should we also follow the convention of ROME here, to make a fair comparison?\n\nThe experiments should exactly match the settings from the ROME paper for GPT-J with CounterFact, where this .094 comes from, though it’s true that the results could be sensitive to this noise value with other models and datasets. We aren't immediately concerned given all the other robustness results we obtained. See our discussion of the Causal Scrubbing comment for the rest of our thoughts on noising ablations. (Basically, it seems potentially problematic that the noise is OOD to the hidden state distribution, although this is very much by design from choosing a three*sigma standard deviation.) \n\n> Line 353: It might be worth a quick footnote here that not all examples have their tracing effect concentrated at the last subject token index. See Figure 11 of the original ROME paper for some natural instances.\n\nTrue! We also noticed this, but it is worth emphasizing since it would be new information to some readers.\n\n> would you argue that the model computation is really not as concentrated as causal tracing suggests?…It seems from the window size = 1 experiment (Figure 10 in Appendix), the picture clearly shows that no single layer gets much tracing effect. Thus, would you agree that the sliding window patching is not principled and could yield a false illusion that the information is sharply concentrated?\n\nYes we agree. That was one of the first things we thought when looking at how tracing effects changed with the window size. The window size 10 plots are a little misleading at first glance because of how concentrated they make the effects look, when in reality, it seems like information is gradually accrued across many layers. \n\nWhat’s interesting is that the model computation can be concentrated, in the sense that you can edit a single layer to achieve a new factual association. And it turns out if you run causal tracing again for a model after editing it, the MLP effects are much more concentrated. So there appears to be a meaningful difference between how concentrated the information is in a pretrained LM vs. an edited LM. \n"}]

Author rebuttal:
We would like to thank all of the reviewers for their detailed reading of the paper and thorough feedback. We are so glad to see such a positive reception for the work! Hopefully this paper can help foster much more research in the community about localization and model editing, as well as add nuance to the community’s understanding of editing methods such as ROME and MEMIT, which have been very popular. We look forward to answering any questions about the paper here and sharing in some discussion of possible future directions for research. 

Meta-review: This paper presents a commentary on ROME and related models that use localization for knowledge editing in Transformers.  Its experiments focus on whether edit success aligns with Causal Tracing. Evidence in this paper suggests this is not the case (Figure 4), even if we broaden the definition to include other formulations of editing (Figure 6).

The reviewers uniformly found the paper to be strong, tackling an interesting question with good experimental design and clear writing. I believe the paper is quite likely to influence researchers working in this area and make them re-examine the assumptions behind editing methods based on causal tracing.

The reviewers largely raised minor points or clarification questions, with the most major framing issue raised being about the interpretation of layer switching. However all of the issues were addressed in the discussion period
or are fixable in a camera-ready version.
Decision: Accept (spotlight)