Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors
Authors: Thomas Hartvigsen
Swami Sankaranarayanan
Hamid Palangi
Yoon Kim
Marzyeh Ghassemi
Venue: NeurIPS 2023 poster (2023)
Keywords: Model Editing
Continual Learning
Model Repair
TL;DR: We show how to make long sequences of targeted edits to specific behaviors of pre-trained models by learning and caching discrete segments in their latent space in which their layer-to-layer transformations are modified

Abstract: Deployed language models decay over time due to shifting inputs, changing user needs, or emergent world-knowledge gaps. When such problems are identified, we want to make targeted edits while avoiding expensive retraining. However, current model editors, which modify such behaviors of pre-trained models, degrade model performance quickly across multiple, sequential edits. We propose GRACE, a \textit{lifelong} model editing method, which implements spot-fixes on streaming errors of a deployed model, ensuring minimal impact on unrelated inputs. GRACE writes new mappings into a pre-trained model's latent space, creating a discrete, local codebook of edits without altering model weights. This is the first method enabling thousands of sequential edits using only streaming errors. Our experiments on T5, BERT, and GPT models show GRACE's state-of-the-art performance in making and retaining edits, while generalizing to unseen inputs. Our code is available at [github.com/thartvigsen/grace](https://www.github.com/thartvigsen/grace}).

Aging with GRACE: Lifelong Model Editing with Discrete Key- Adaptors
Thomas Hartvigsen University of Virginia, MIT hartvigsenOvirginia. edu
Swami Sankaranarayanan
Sony AI
swami www.araaaaaaaaaaaaaaooyyyyooo com
Hamid Palangi Microsoft Research hpal angi @mi crwsoft..cccoooooooooo com
Yoon Kim
Marzyeh Ghassemi MIT mghassemmmit..o edu
MIT
yoonkim@mit.edi
Abstract
Deployed language models decay over time due to shifting inputs, changing user needs, emergent world-knowledge gaps. When such problems identified, we want to make targeted edits while avoiding expensive retraining. However, current model editors, which modify such behaviors of pre-trained models, degrade model performance quickly across multiple, sequential edits. We propose GRACE, a lifelong model editing method, which implements spot-fixes streaming errors of a deployed model, ensuring minimal impact on unrelated inputs. GRACE writes new mappings into a pre-treennn pre-trained model's latent space, creating a discrete, local codebook of edits without altering model weights. This is the first method enabling thousands of sequential edits using only streaming errors. Our experiments on T5, BERT, and GPT models show GRACE's state-of-the-an performance in making and retaining edits, while generalizing to unseen inputs. Our code is available github.omm/harriggen/ggg
1 Introduction
Large scale pre-trained neural networks are the state-of-the-art for many hard machine learning
problems, especially in natural language processing [5] 33] and computer vision p 344. But when deployed, they still make unpredictable errors 2 43 For example, Large Language Models (LLMs) notoriously hallucinate (17), perpetuate bias .. and factually decay (8) Many such errors will arise sequentially in deployment, some of which must be addressed quickly without waiting until new training data collected [16] 21 For example, when LLM generates hate speech crucial knowledge about the world changes, its behavior must be modified immediately to protect users.
While retraining finetuning can edit model's predictions, doing this frequently is often too computationally expensive. ILaMA [40], for instance, was trained for 21 days on 2,048 A A100 GPUs, costing over $2.4M and emitting over 1,000 tons of CO,. Even repeatedly retraining finetuning smaller models [3] quickly costs too much for most practitioners. To enable cheap, targeted updates big, pre-trained models, study lifelong model editing. Here, continually make targeted edits sequentially throughout a model's s deployment. Success means correcting a model's predictions on inputs.
stream of edits without decaying its performance unrelated and previously-
Two possible approaches to lifelong model editing are contimual learning and conventional model editing. While continual learning methods update models sequentially, when used for sequential editing, they suffer from overfitting (25] and quickly forget previous edits and pre-training data Alternatively, existing static model editors could also be run sequentially. But the editors that
16 22
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

GILATE codebosic byee?
Netocee
Xj
90x
Faslion
"Luted pasdesk?
Ya
Valao
X
frose Wy
Genvralizable adbled
81 "Thoodee
Layer
Layer 1+1
Figure 1: Overview of lifelong model editing with GRACE. a) Models make important errors that must be corrected. b) GRACE makes edits by learning, caching, and selectively retrieving transformations between layers. c) Edits appear sporadically and require quick fixes, so GRACE codebooks are curated over long sequences of edits.
37 succumb overfitting 4. while hypernetwork methods require pre-traie training on hard-to-access data 30 31 Further, these existing approaches require large sets of representative training edits, access the model's pre-training data, or semantically-equue -equiiialeen inputs. Accessing these data is often unrealistic or impractical, and existing editor performance severely decays the pre-trained model after only a few sequential edits 12
rely on regularized finetuning 28 29
We study lifelong editing using only singular inputs to edit models, where models are edited imme- diately upon arrival. This setup extends beyond recent works, which use many inputs for editing, inputs, training edits, and the model's pre-training
including collected sets of ceaanticaall-qquvvllle data. Editing in our realistic, more cost-effective setup hard because each edit must fix the flagged error, while generalizing to similar future inputs without impacting unrelated model behavior.
To address the challenging lifelong model editing setup, we propose General Retrieval Adaptors for Continual Editing, or GRACE. While we study transformers for natural language processing, GRACE is broadly applicable. As illustrated in Figure [ GRACE edits model by adding an Adaptor to a chosen layer, while never changing its weights. This Adaptor then modifies layer-to-layer transformations for select inputs. By caching embeddings for input errors and learning values that decode into desired model outputs, GRACE serves as a codebook in which edits are stored, enabling longer sequences of edits than prior works. To encourage generalization, we leverage the models' semantic similarity in its latent space by introducing e-balls around cached edits. GRACE then only applies for inputs near existing keys, differing from more-traditional Adaptors (13]. By managing the e-balls' size over time, GRACE makes immediate edits, remembers previous edits, and leaves correct model behaviors intact, making it parameter efficient. Further, since GRACE codebooks leave model weights unaltered and are fully model-agnostic, they also point towards plug-and souug-and-paay -lly,, cost-effective model editing. especially to make crucial spot-fixes between bigger retraining efforts.
Our contributions are as follows:
1. We establish key metrics and comparisons for lifelong model editing, an important but understudied and challenging problem setting. We introduce two new public benchmarks for lifelong model editing: mitigating LLM hallucination
27 and addressing label shifts (6).
2. We develop GRACE, method for lifelong model editing that avoids expensive retraining or finetuning. Unlike other works, GRACE requires no inputs beyond singular edits. GRACE works with autoregessive and non-autoregressivve models alike and achieves parameter efficiency without touching the model's weights.
3. Our experiments show that GRACE outperforms seven alternatives when sequentially editing T5, BERT, and GPT models for question answering, document classification, and language generation. We show that GRACE edits can generalize to new inputs without memorizing while incurring only small, one-time cost inference time over long sequences of edits.
2 Methods: Lifelong Model Editing with GRACE
2.1 Problem Formulation
The Lifelong Model Editing task is to edit the same model hundreds to thousands of times in a row without forgetting upstream performance or fixes for previous edits. Let fo denote model with

---

frozen parameters that was pre-trained on dataset Duain- For clarity, we drop the subscript where possible. Assume that consists of L layers, where (-) computes the hidden state at layer In this work, we assume f is a transformer architecture for natural language, though our principles are general. We then deploy J on a stream of inputs [mo, #1, where 2t E Dedin which contains samples observed during deployment. We then monitor the model's predictions Or = f(x) over the stream. Note that the prediction tasks during training and deployment are the same. Over time, the model makes errors such that Dr + y/, where Y/ is the true label. To continue safely deploying f. we aim to edit f such that (*x) = Yt. After each edit, the updated f should 1) be successfully edited such that f(z)) ye. 2) retain accuracy
prior edits <ct, and 3) retain its behavior on its training data: f(z; = fo (2²) V I2 E Duain- In contrast to prior works [8] 28 30 31 371, we assume access to only input edits 2; and corrected labels yt, since Dusin is often proprictary or prohibitively large,
and collecting training edits or semantically-equivleeee inputs is often expensive in practice.
2.2 GRACE: General Retrieval Adaptors for Continual Editing
We propose GRACE, a method for sequentially editing a pre-trained model's behavior without altering its weights, as illustrated in Figure I GRACE works by wrapping a chosen layer of any pre-trained model architecture with an Adaptor. A GRACE Adaptor at model f's layer I contains two components: (I) a codebook C and (2) a deferral mechanism to decide whether to use C for a given input.
GRACE codebook. A GRACE Adaptor at layer maintains discrete codebook, adding and updating elements over time to edit a model's predictions. The codebook contains three components:
Keys (K): Set of keys, where each key is a cached activation h1'11 predicted by layer / - 1. Values (V): Set of values that are randomly initialized and are updated using the model's finetuning loss for edits. Each key maps to single, corresponding value.
Deferral radii (E): Each key has a deferral radius e, which serves as threshold for similarity matching. The deferral mechanism uses this radius as shown in Algorithm [ GRACE is activated at layer I only if the deferral constraint is satisfied. New entries have a default value Ciii. which is a hyperparameter.
Deferral mechanism. Before editing, GRACE layers empty. As editing progresses, GRACE adds keys and adapts values and e entries. Conceptually, inference at layer / with GRACE entails a deferral decision, computing /t' using similarity search over GRACE's keys:
h' (GRACE(N')) if min;(d(h' (KK)) < Ei.: where i (d(k' (1'(N--))
(1)
otherwise,
where 1'(h--1) denotes the unedited model's activation of the I-th layer. h'-1 can be seen as a query to the codebook, and GRACE(N--1) retrieves the value associated with its closest key. df and K! are the influence radius and key i in layer I, respectively, and d(-) is distance function. We follow related work [41 and use Euclidean distance for di()) in our apperiments-chaneee Through explicit similarity search, we use the fact that large models encode semantic similarity with respect to their tasks in their latent spaces. This lets GRACE edits to generalize to similar inputs in the future. If a new input is unlike any cached keys, GRACE simply defers to f's pretrained weights. This way, GRACE layers limit interference with Drain by leaving the model weightsunaltered, which helps when input distributions shift 455
this is trivial.
Codebook maintenance. To make an edit, a GRACE layer can perform one of two operations. Each step is described in Algorithm [ First, if the codebook is empty or the input embedding h falls outside the deferral radius of all existing keys according to distance function d(-), then a new codebook entry is created and added: {(h'1), U. Ciitt })}. Thus if x, were passed into f again, h--1 would activate the codebook and value v would be passed to layer 1 + 1. Training v is detailed below.
existing key that adding new entry would cause their e-balls to overlap. To avoid this, we compare the edit label y to the model's prediction for the nearest key and distinguish two cases: 1) If the overlapping key's label is the same the edit's label, Expand that key's to encompass the query. 2) If the overlapping key's loys label is different from the edit's label, Split these keys by first decreasing the influence radius of the overlapping key, then adding a new codebook entry where the new key is simply the query h 1-1 We set both keys' e values be half their distance apart.
Sometimes, query h will be close enough
3

---

As edits stream over long deployments, by con- tinuously adding and updating the GRACE code- book, layer I's latent space is partitioned accord- ing to which inputs needed edits. When not per- forming edits, these codebook maintenance oper- ations are bypassed, and keys are entirely frozen. GRACE thus introduces a new model editing paradigm in which edits can be made sequentially, similar edits encouraged to be edited similarly, and the ultimate influence of new edits can be controlled and monitored explicitly. Emit is the sole parameter in GR which sets the initial e value for new codebook entries. Intuitively, using a larger Cinit will create edits with more influence, making edits more general, but increasing the in- lerference with unrelated inputs. In practice, finit could be tuned using either exogenous data GRACE codebooks can periodically be refreshed.
Algorithm 1: Update Codebook at layer 7.
Input: C C= {(K,, 5:) codebook
Input: f()). model
Input: yt. desired label
Input: Ig. edit input for which f(x)) +
Input: Einit: initial
Input: d(-), distance function
Output: C, updated codebook
C
C-=10 |C|
ú. N = f (x+), ((-1)))))))
dmin, = min,
Ki))
If dmin > + Einit C = 0:
W far from existing entries or empty
Unew finetune on (y(3)
Cc (h'-1, Vnnw. Einit) # Add entry
Else:
# hl near existing entries
Training GRACE Values. When making an edit with GRACE, either a new key-value pair is learned or existing key-value pair is updated. To ensure that newly-learned values correct the model's behavior, train them directly using backpropagation through the finetuning loss on the model's prediction given the edit. The learned value e then replaces h for the rest of the forward pass. In our experiments, 100 gradient descent steps to train the values and ensure the model's 's behavior updated.
If f (k:s) y/:
# Same label + Expand
C, = (ki, Us. C, + Cinit)
Else:
# Differentt label + Split
C. = (kis, U4s amin/ /2) # Update entry i
Unew = finetune on Pinnt)
Cc = (h'-1 Unew, , dmin/ 2) # Add entry
train values using
return: C
GRACE layers with sequential inputs. For models with diffcrent representations per input token, like transformers, token to replace with a retrieved value in the subsequent layer. In practice, we find that broadcasting the value to each token is reasonable for tasks like classification, since values gain strong control over the model's behavior and makes them easy to learn. For autoregressive models however, picking the right tokens for the query and value is more nuanced. We opt for replacing only the final token of an input prompt, which we verify experimentally, since compressing future generated text into tokens is an interesting and burgeoning direction in itself [2] Upon choosing these tokens, GRACE naturally applies to all popular transformer models.
must choose 1) which token should be GRACE's input query, and 2) which
2.3 Illustrative Example
To garner intuition about GRACE, we provide an illustrative experiment on synthetic data. As shown in Figure 2 we sample 100 instances from two 2D distributions corresponding to classes. We then train a three-layer binary classifier with two 100-dimensional hidden layers and ReLU activations. Next, we introduce edits with flipped labels, simulating local label shift at test time. Using single
Upstream Training Data (b) Upstream Training Data Edits (<) Predictions before editing
(d) Predictions after editing
Figure 2: Illustrative example of GRACE. We train a model on separable data in (a), then introduce docally-flipped labels test time (b). (c), the original model unsurprisingly misclassifies these flipped instances. In (d), GRACE fixes these labels without impacting other inputs.

---

TEST RETENTION DATA
EDITING DATA
SETTING
MODEL
Dataset
N
Pre-edit
Dataset
N
Pre-edit
QA CIf. Hallue.
T5 (60m)
NQ 20
1000
.72 F1
ZsRE
24
1000
.31 F1
BERT (120m)
SCOTUS 11952-19910
6
914
.99 Ace
SCOTUS 1992-2209
6
931
.55 Acc
GPT-2 (1,5B)
OpenWehText
10
1000
15.98 PPL.
SelfChecKGPT
1392
132.7 PPI.
Table 1: Dataset statistics for main results. Test retention is the testing set of each model's training data. N is the number of samples. Pre-edit is the unedited model's performance on each dataset.
key in layer two, GRACE can successfully correct these errors while barely influencing other inputs. Meanwhile, finetuning on these errors alone will clearly break the model.
3 Experiments
3.1 Experimental Setup
We evaluate GRACE's capacity to edit models hundreds to thousands of times sequentially. This is in contrast prior works, which largely evaluate using single edits. Further, much of the model editing literature relies on synthetic edits, often generated by flipping random labels. Beyond unrealistically assessing performance, it is well-established that learning random versus natural labels is vastly different 49). Instead, we correct authentic mistakes made by models, each time an error is made.
Baselines. We compare against continual learning and model editing methods. First, continually finetune (FT) (25) on streaming errors. To reduce overfitting, we also compare with Elastic Weight Consolidation (EWC) [19] and simple form of experience replay [36] where we periodically retrain the model (Retrain) on all previous edits. Second, we compare against model editors MEND [30 and Defer, inspired by SERAC (31] but altered for our setup. We also compare against ROME in our language modeling experiment. Finally, we ablate GRACE by replacing our discrete search with Memory network containing memory module that is indexed by a soft attention mechanism. Details for all comparisons are in Appendix C
28
Datasets and Pre-trained Models. We evaluate GRACE on three sequential editing tasks with corresponding pre-trained models, as shown in Table I 1) We edit a 60-million parameter T5 model (35) trained for context-free
as is used in [3).. We extract potential edits from the validation set of zsRE [24), following the editing literature [6, 30 This model achieves F1 of .72 on NQ and .31 on zsRE prior to editing. Further details in Appendix B. 2) We edit 1 110-miliion BERT classifier trained for a new editing task with label shift using the SCOTUS dataset from Fairlex (6). The prediction task is to categorize U.S. Supreme Court documents over multiple decades into 11 topics. Over time, categorization rules change, so label distributions shift. We train BERT classifier on the 7.4k training cases from 1946-1982, then make edits on 931 cases from 1991-2009. We also introduce additional, realistic label shifts, as discussed in Appendix B.2 This model achieves Accuracy of 0.99 on the training set and .55 on the edit set. 3) We introduce new editing task by correcting GPT language models' Hallucination. In (27). authors prompt GPT-3 to generate 238 wikipedia-style biographies using subjects from WikiBio. They then annotate the factual accuracy of each sentence, recording which are hallucinations. We propose editing inaccurate sentences by replacing them with corresponding sentences the true wikipedia entries. We include edits for all 238 biographies, creating 1392 sequential edits and 592 already- accurate outputs. We then edit GPT2-XI, which has 1.5B parameters. However, GPT2-XL. was not trained on this task, so has high perplexity (PPL) on all sentences. Therefore, we finetune -L on these data mixed with sentences from Tebn [0), a public version of GPT2's training data. Our final GPT2-XL model has PPL of 15.98 OpenWebTen. (comparable to the original), 8.7 on already-accurate outputs, and 132.7 on intended edits, indicating need for editing and room for improvement. Further details are available in Appendix B.3
Metrics. To compare each method, we measure three main metrics that align with prior work [6
1. Edit Success (ES): We check whether an edit has been successful using ES: m(y, 0). where m(-) task-specific measure of accuracy: standard F1 for question answering, Accuracy for classification and Perplexity (PPL) for generation.
5

---

ZsRE (TS; FI 1)
SCOTUS (BERT; Ace
1)
Hallucination (GPT2-XL; PPL !)
Method
TRR
ERR
Avg.
#F.
TRR
ERR
Avg.
#E
TRR
ERR
ARR
#F
time (S)
FT
25
.82
1000
.52
.52
.52
415
1449.3
28.14
107.76
1392
.26 (.07)
FT+EWC
19
.51
.82
.66
1000
.67
.50
.58
408
1485.7
29.24
109.59
1392
.29 (.06)
FT+Retrain
36
.27
.99
.63
1000
.67
.83
.75
403
2399.3
35.34
195.82
1392
23.4
MEND
50
.25
.27
.26
1000
.19
.27
.23
672
1369.8
1754.9
2902.5
1392
.63 (.10)
Defer BD
.72
.31
.52
1000
.33
.41
.37
506
8183.7
133.3
10.04
1392
.07 (.02)
ROME 28
30,28
103.82
14.02
1392
.64 (.28)
Memory
.25
.7
.26
1000
.21
.20
.21
780
25,47
79.30
10.07
1392
.11 (.02)
.69
96
.82
1000
.11
82
82
381
15,84
7.14
10.00
1392
.13 (.02)
GRACE
137 keys (7.30 edits/key)
252 keys (1.51 edits/key)
1341 keys (1.04 edits/key)
Table 2: Comparison of GRACE existing methods. Metrics shown computed after all sequential edits. For Hallucination, we also compute perplexity retention on already-accurate sentences (ARR) and the average time per edit. Parentheses in time denote standard deviation. We also count the number of keys GRACE used, GRACE achieves the best TRR/ERR balance while making edits faster than most comparisons and using small codebooks for zsRE and SCOTUS, A large codebook is required for Hallucination, since each edit label is unique.
2. Test Retention Rate (TRR): We check how well an edited model retains its performance on its original testing data using TRR: 1 E m (x;), v.). where (æ;, y/) E Diest f's original test set. For T5, Des is 1k random samples from NQ, for our BERT model, Dusa is court documents from 1992-1991, and for GPT2-XL, Dies is WebText, for which we use the first 1k sentences of OpenWebText [10].
3. Edit Retention Rate (ERR): We check how well an edited model retains previous edits through
ERR: 1/2 / m(f(z)), y:) where (Iz, 34) E Dedits-
We also track the number of edits (#E), which may vary by editor as different editors will lead to different mistakes in practice. Further implementation details are available in Appendix A
3.2 Results
3.2.1
Comparisons to existing methods
We first find that GRACE outperforms existing methods after long sequences of edits, as shown in Table We focus here on TRR and ERR, which have a trade-off each can be achieved in isolation, though all metrics are reported in Appendix G On zsRE and SCOTUS, we focus on TRR, ERR, and their average. ROME is incomparable Hallucination, we include Accurate Retention Rate (ARR), which is the edited model's perplexity on sentences on which it was already accurate.
these dataset as it is only proposed for GPT models. On
On zsRE and SCOTUS, GRACE's averaged TRR and ERR outperforms its closest competitors by 19% and 9%, respectively. Comparing the average performance across all methods, GRACE's improvement is 86% and 129% due to other methods' inability to balance ERR and TRR. For instance, while Defer achieves the highest TRR on SCOTUS it trades off ERR. In contrast, FT+Retrain achieves the highest ERR on SCOTUS, but trades off TRR. On both datasets, MEND and Defer both struggle to balance TRR and ERR without access to privileged data.
GRACE also outperforms the comparisons on the Hallucination task. We observe that the finetuning methods casily overfit to new edits, leading to competitive ERR but poor TRR. However, their good ERR comes at the expense of ARR in these methods. We also find that ROME and Memory both perform competitively with GRACE, indicating the value of parameter efficiency in lifelong model editing. Finally, we report the average wall-clock time it takes to perform one edit, finding that GRACE makes edits twice as fast as finetuning while competing with other Adaptors.
Excitingly, GRACE can achieve high-quality edits with small codebooks. For example, GRACE uses few keys to edit T5 on zsRE: 1000 edits are made using only 137 keys. Such compression is only feasible by managing the e-balls effectively over time. This is also parameter-efficient with details in Section 3.2.3 the zsRE codebook contains 210,569 scalar values (.35% of T5's parameters) and

---

400
100
300
75
100
25
di
inputs indeed 700 Semao
PRW 7210 MEXD Dine
HONE
XIEAD
MOMI Missayy GMACE
(a) Training Retention.
(b) Edit Success.
Figure 3: ES and TRR while editing GPT2-XL on Hallucination. Lower values are better because TRR and ERR measure perplexity. GRACE outperforms the comparisons by making successful edits while maintaining the model's training knowledge. All metrics are shown in Figure 13
only 70,281 are learnable. The number of keys is also lower-bounded by the number of unique edit labels and is related to the complexity of the inputs. On SCOTUS, GRACE also achieves a relatively small codebook, showing a balance between input complexity and generalizability of each edit. Since SCOTUS is a document classification task with 11 classes, there could be as few as 11 keys, but given each document contains many sentences, there is lower chance that the same keys can be used for many documents. That GRACE's keys represent 1.51 edits on average indicates significant generalization, As expected, the lower-bound on codebook size is evident in the Hallucination task, where GRACE uses a new key for almost every edit. This is because edit labels are unique sentences. Despite a large codebook, such success at selectively inserting tokens that generate full sentences poses an exciting direction for controllable and editable language modeling.
show deteilet comparison of all methods for the Hallucination editing task over time, focusing on TRR and ES (see Figure [3] for all metrics). As expected, finetuning methods excel at ES, but perform poorly at TRR. On the flip side, ROME and Memory have reasonably-low ES and TRR scores, though both suffer as edits progress. While ROME is competitive early in editing, it especially suffers on ERR (Table 2 and GRACE's TRR remains 2x better after making all edits.
In Figure 3
3.2.2 Model Analysis: Memorization Generalization
Next, we study GRACE's memorization vs. generalization performance by editing T5 on zsRE on extremely long sequences of edits while varying Gua and the edited layer. We split each zsRE question's 's set of rephrasings into two sets: edits and holdouts. Then, we pass edits through GRACE until 3,000 edits are made. This is extremely large: even 10 edits catastrophically decays performance [1] and [16] performs roughly half many edits. After each edit, we measure TRR, ERR, F1 on the entire Holdout set, and record the number of keys. Since we evaluate GRACE on the entire Holdout set after each edit, the results start low since has seen rephrasings of held out edits. Therefore, later measures of Holdout performance are more representative than carlier. Figure 4 shows our results for Gnit 0.1 and Einit 3.0, while the rest are in Appendix H We derive the following findings.
Layer and Cinit choice balance memorization and generalization. We first find that different blocks lead to different editing performance. Notably, editing Blocks two and four achieve high TRR, high ERR, and strong generalization for both choices of Gii On the contrary, for Block we see that since each Gint is small, they do not generalize at all and also lead to poor ERR. As expected, when Gnt is small, TRR performance is optimal, since most edits require a new key. While creating a large codebook can be feasible, the trade-off in Holdout becomes clear, detailed in the next finding. The relationship between layer choice, performance, and the size of the codebook likely stems from a layer's 's representational capacity: Layers that map semantically-equiveee will be easier to edit with GRACE.
inputs near one another
GRACE edits generalize to unseen inputs. Steadily increasing Holdout shows that that GRACE edits can generalize to previously-unsed holdout edits. Excitingly, interior layers appear to generalize better than early and late layers, which is backed up by their use of fewer keys. Larger Gint values also lead to better generalization, as expected, implying that the semantically-simatt land in the same deferral radii. The later layer, Block 6, appears
GRACE codebooks stay small and stabilize over time. Finally, the number of keys over time steadily flattens, indicating that the values indeed adapt the data distribution. As we increase Einits
7

---

the resultant codebooks get smaller, indicating control over codebook size. Small codebooks are also important for parameter efficiency and in generalization.
1.00
0.80
Num Keys
3k
0.70
Holdout
TRR
ERR
0.90
2k
0.60
0.65
0.80
1k-
0.40
0.60
1k
2k
3k
0.70
1k
2k
1k
2k
3k
1k
2k
3k
Edits
Edits
Edits
Edits
(a)
0.1
1.00
0.80
Num Keys
3k
0.70
Holdout
TRR
ERR
0.90
2k
0.60
0.65
0.80
1k
0.60
0.70
1k
2k
3k
1k
2k
3k
1k
2k
3k
1k
2k Edits
3k
Edits
Edits
Edits
Block: 0
Block:
Block:
Block:
(b)
3.0
Impact éinit and block choice for GRACE editing T5 on zsRE for 3000 sequential edits. Other finit values are in Appendix H Along with TRR and ERR, we also measure Fl on a "Holdout" edit set containing unseen rephrasings of all edits. We find that blocks and use more keys and achieve higher TRR, but can lead to lower ERR and generalize worse, given lower holdout values.
Figure 4:
3.2.3 Parameter Efficiency
GRACE's memory requirements are small and straightforward: new edit requires |n'-1|+ [h|] + parameters, where |A'-1| the dimension of the key, [b'] is the dimension of value, and cinit scalar. Further, the key's |/'-1| parameters are not trained. As detailed Appendix E this is comparable to the alternatives when performing thousands of edits, so performance gain is not from parameter count. This intuition is reinforced by the fact that at inference time, the Adaptor
activated, predictions are only altered using the | hh 1h1+ + parameters of the chosen entry.
3.2.4 Interpreting GRACE Codebooks
key benefit of GRACE is that learned codebooks can be detached from the model and inspected. This way, edits can easily be undone without impacting the model and the codebooks can be inspected throughout editing. To demonstrate this advantage, change throughout editing Block 4 of the T5 QA model. In this experiment, we investigate how well GRACE edits generalize to unseen edits. At each edit in a sequence of 1,000 inputs, we pass the entire holdout set of edits through the newly-edited model. For each holdout instance, we record which key's e-ball it lands, any. This way, we track whether a newly-added key generalizes to multiple holdouts successfully. We also track what proportion of the holdouts land inside any key to evaluate generalization over time. In Figure we summarize the results from one such experiment, with more details in Appendix [D] We find that the number of holdouts per key stabilizes over time. Interestingly, fin values capture too many holdouts per question, which explains the trade-off between generalization and TRR. In the Appendix, we also show that the number of holdouts per key are stable, while others are highly-variable. This implies that our codebook maintenance strategy can have big ripple effects according to the key. Further, some regions of the latent space likely better than others for performing GRACE edits.
inspect how keys and their
3.2.5 Inference Time
To better understand GRACE's limitations, we compare the inference time of the T5-small QA model before and after editing. We compute the time it takes run one instance through the model for each of 5,000 edits. We edit T5's block 4 and use a small Enit of 0.1 to encourage quickly-growing

---

Avg. holdouss
10-
5.00
2,00
vitie GRACE
2.00
1000
Inputs
Codebook Stze
Figure 5: Interpreting GRACE keys throughout editing. Larger finir achieve good generalization. The grey line the true holdouts per edit.
Figure 6: Comparing inference time of GRACE- edited and unedited T5 QA model as the size of the codebook increases.
codebook. To evaluate the variance in training time, we replicate this experiment ten times, each leading to a codebook of roughly 4500 elements. We then average over 20-timestep windows and show standard deviation across all replications.
We find that inference with a GRACE-edited model was only 1.32x slower than an unedited model on average, as shown in Figure This is a one-time cost that remains fixed even as the codebook grows. Inference time is unchanged because search between one query and a full set of keys can be vectorized. Until the codebook outgrows available memory, this cost remains fixed. That GRACE causes any slowdown is limitation and an avenue for future work.
4 Related Work
Model editing, Model editing is a new and active research area where the goal is to make targeted changes to pre-trained model's behavior. Most methods propose variants of gugularized-finttnninn via auxiliary data, like training instances from the pre-trained model's training data or by using semantically-cquivleenn versions of new edits (37). Access to such data is non-trivial, especially as training data and paradigms are becoming proprietary, and collecting semantically-equivaeenn inputs is often unlikely. Recent works have retained these requirements while extending pretrain hypernetworks that predict edits [8] 30 31]. often decomposing weight updates into low-rank components [28 29 Recent approaches all study transformer architectures due to their popularity and high training cost 48 51. To ensure targeted edits, recent methods like MEND 30 and ROME
28 also draw inspiration from parameter-efficient finetuning [15]. But such methods are known to often require more finetuning steps and are prone to overfit more than regular finetuning [39, 50). Further, recent works show that edited models are deeply fragile 4 and methods for picking which parameters to update are surprisingly unreliable [13].
Unfortunately, nearly all model editing works consider only static edits, making only one edit to a model. While some recent works like MEMIT [29] indeed perform multiple edits, they do so simultancously, not over time during deployment. Other works demonstrate that editing performance decays quickly when making multiple edits (30), though recent works like SERAC [31] and MEMIT 29 show burgeoning results this direction. However, these exciting works still use large amounts of privileged information. Most similar to our work, two recent papers discuss sequential editing. First, 12 shows that after editing the same model 10 times, editing performance drops dramatically. Second, a concurrent paper [16] recently proposed a sequential editing setup. However, their method and implementation is architecture-specde specific and relies on large sources of unrelated inputs.
Continual Learning, Continual learning methods are reasonable approach to lifelong model editing. Most-similar to our setup, recent works have investigated continual finetuning. where large language models are refined over time as new instances arrive. For example, (25) perform a large benchmark of continual finetuning. They find that regularizing finetuning with continual learning methods like Elastic Weight Consolidation [19], Experience Replay [36], and Maximally Interfered Replay [I]. quickly decays performance on prior tasks, though it helps remember some previous inputs. This implies that editing, as opposed to regular continual finetuning, is particularly challenging, since edits are unlikely to be uniformly distributed [14. One promising avenue for continual learning is For example, recent works have demonstrated continual prompt-learning for NLP [4 45 for applications like text retrieval [47]. Recent works have shown that discrete key-value methods in particular perform well with shifting distributions [4I), with recent works extending to question answering [7]. By caching values, these approaches keep inputs in-distribution for downstream encoders while opening doors to longer-term
key-value methods, stemming from computer vision 26
34 42

---

memory, resources permitting. We corroborate these advantages in our experiments, where we demonstrate GRACE's robustness shifting inputs and labels over long sequences of edits.
Limitations and Ethical Considerations
I Lifelong model editing is and challenging, ably, adding similarity search to the layers of a model will slow down inference. Still, while we do not emphasize inference time, accelerating GRACE is natural future step that has been successful in similar methods [1S 333. Future works may also scale GRACE up to multi-layer edits, a setting unconsidered this work. While GRACE has already scaled up continual editing ~5k edits, real world scenarios might entail approaches that work at an ever larger editing scale, which presents promising direction of future work. Another limitation of GRACE-style edits is implication: Behav- iors are edited in isolation. For example, if we edit model's knowledge of the latest pandemic in the U.S. from Swine Flu to COVID, its knowledge of the second-to to-last pandemic will not be updated.
method GRACE has limitations. Understand-
While editing models can improve their behavior, it can surely be used for harm. For example, a bad actor might edit L.LM increase hate. This limitation is true for all model editors, GRACE included. However, most model editors today directly update the model's weights. This makes it hard to trace back what edits have been made to a model and understand their impact. Transparent editors like GRACE are promising direction for overcoming this problem. For example, GRACE's codebook can be directly inspected to see what predictions stem from each value and examine properties the latent space covered by the e-balls.
6 Conclusions
Pre-trained models continue to grow and are being applied to a diverse set of downstream tasks. However, they still misbehave in unpredictable ways when deployed. Correcting such behavior is challenging problem, especially when only some of the model's behavior is problematic. While regu- larized finetuning retraining better data can mitigate this problem somewhat, big models remain too expensive to train for most researchers. We instead build on the model editing literature, and study Lifelong Model Editing, an important but understudied problem. With a focus on transformer models for natural language processing, we edit models thousands of times in a row as edits stream during simulated deployments. Our edits only use singular, authentic errors, in contrast to recent works which make synthetic edits and require large amounts of exogeneous data like sets of training edits, cemantically-equivalee
examples, or pre-training data. We then present GRACE, plug-in Adaptor for model's chosen layer that leaves the trained weights untouched. GRACE Adaptors (1) retain the functionality of the original model, while (2) successfully editing model predictions without forgetting previous inputs. Using three real-world datasets, we edit T5, BERT, and GPT models thousands of times and find that GRACE significantly outperforms seven state-of-the-ae alternatives. We further investigate GRACE's capacity to make extremely long sequences of edits and show that GRACE can generalize its edits to unseen inputs, avoiding sheer memorization.
7 Acknowledgements
We are grateful to all the support received while conducting this research. This project was supported by Quanta, and Marzyeh Ghassemi was supported by the Herman L. F. von Helmholtz Career Development Professorship and the CIFAR Azrieli Global Scholar award. Yoon Kim was supported by MichiieeLemminnAAppllctiooeeee CSAIL.
References
[1] Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. ArXiv preprint, abs/1908.0474p3 2019.
[2] Vidhisha Balachandran, Hannaneh Hajishirzi, William Cohen, and Yulia Tsvetkov. Correcting diverse factual emrrr in abstractive summarization via post-editing and language model infilling. In Empirical Methods in Natural Language Processing, 2022.
10

---

[3] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.013311 033333 2023.
[4] Davis Brown, Charles Godfrey, Cody Nizinski, Jonathan Tu, and Henry Kvinge. Robustness of edited neural networks. arXiv preprint arXiv:2303.00046 2023.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[6] Iias Chalkidis, Tommaso Pasini, Sheng Zhang, Letizia Tomada, Sebastian Felix Schwemer, and Anders Søgaard. Fairlex: A multilingual benchmark for evaluating fairness in legal text processing, arXiv preprint arXiv:2203. 0722228 2022.
[7] Yi Dai, Hao Lang, Yinhe Zheng, Fei Huang, Luo Si, and Yongbin Li. Lifelong learning for question answering with hierarchical prompts. arXiv preprint arXiv:2208. 14602, 2022.
[8] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6491-6506, 2021.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010./ 1929, 2020.
[10] Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanic Tellex. Openwebtext corpus.
http://Skylion007,, wwt.o/oonnoo........oooo ao/DpennNeeTexxtCorpus 2019.
[11] Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap. Dipankar Ray, and Ece Ka- mar. Toxigen: A large-scale machine- sechine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pupers), pages 3326, 2022.
[12) Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan lyer. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. arXiv preprint arXiv:21/1 1.2///..13544 2021.
[13] Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in suusality-based localization vs. knowledge editing in language models, arXiv preprint arXiv:2301 04213, 2023.
[14] Thomas Henn, Yasukazu Sakamoto, Clément Jacquet, Shunsuke Yoshizawa, Masamichi Andou, Stephen Tchen, Ryosuke Saga, Hiroyuki Ishihara, Katsuhiko Shimizu, Yingzhen Li, et al. A principled approach to failure analysis and model repairment: Demonstration medical imag- ing. In Medical Image Computing and Computer Assisted Intervention-MICCAl 2021: 24th International Conference, Strasbourg, France. September 27-October 1, 2021, Proceedings, Part III 24, pages 509-518. Springer, 2021.
[15] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[16] Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. Transformer-pateee One mistake worth neuron. In International Conference Learning Representations, 2023.
[17] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su. Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
[18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X2::1412.69800 2014.
[19] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Strabska-Bawwinet et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):35213322 2017.
11

---

[20] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Was Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Tronsactions of the Association of Computational Linguistics, 2019.
[21] Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez. Cyprien de Masson d' Autume, Tomas Kocisky, Sebastian Ruder, al. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Systems, 34:29348 293633, 2021.
[22] Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to
finetune large-scale pretrained language models. In International Conference on Learning Representations, 2020.
[23] Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. arXiv preprint arXiv:2210.1 1466, 2022.
[24] Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLI. 2017), pages 333-342, 2017.
[25] Bill Yuchen Lin, Sida I Wang, Xi Lin, Robin Jia, Lin Xiao, Xiang Ren, and Scott Yih. On continual model refinement in -iditiibbitiitoo data streams. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),
pages 3128-3139, 3339, 2022.
[26] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34: 28092-28103, 2021.
[27] Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint (rXXiv2303.08890 2023.
[28] Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems, 2022.
[29] Kevin Meng, Arnb Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2023.
[30] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022.
[31] Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory-based model editing scale. In International Conference on Machine Learning, pages 15817-15831. PMLR, 2022.
[32] Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens. arXiv preprint 23X4::2200408467 2023.
[33] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.114466 2021.
[34] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprini arXiv:2204.06122 2022.
[35] Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? In Empirical Methods in Natural Language Processing (EMNLPP), 2020.
[36] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experi- ence replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
[37] Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. In International Conference on Learning Representations, 2019.
12

---

[38] Harold Spaeth, Lee Epstein, Andrew Segal, Ted Ruger, Sarah Benesh. Jeffrey Segal, and Andrew Martin, Supreme court database, 2020. URL URL http://uuprmmooootttaaaaaaaaaa org
[39] Yusheng Su, Xiaozhi Wang. Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang. Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, et al. On transferability of prompt tuning for natural language processing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3949-3969, 2022.
[40] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint (tt.:233.2./9771,, 2023.
[41] Frederik Träuble, Anirudh Goyal, Nasim Rahaman, Michael Mozer, Kenji Kawaguchi, Yoshua Bengio, and Bernhard Schölkopf. Discrete key-value bottleneck. arXiv preprint urXiv:2207.112444 2022.
[42] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[43] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
[44] Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal- free continual learning. arXiv preprint arXiv:2204. 047999 2022.
[45] Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 139-149, 2022.
[46] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. arXiv preprint arXiv:1410.3916 2014.
[47] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations, 2021.
[48] Yang Xu, Yutai Hou, and Wanxiang Che. Language anisotropic cross-lingual model editing. arXiv preprini arXiv: 022205..2667 /2677, 2022.
[49] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3): 107-115, 2021.
[50] Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. Panda: Prompt transfer meets knowledge distillation for efficient model adaptation. arXiv preprint arXiv:2208. 10160, 2022.
[51] Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint 2rXwv:2012.00363, 2020.
13

---

A Implementation Details
Training Details. All methods are optimized using Adam [88] Since edits are singular and sequential in our setup, the batch size always 1. We trained all methods using various GPUs including 48GB NVIDIA RTX A6000s, 40GB NVIDIA A A000s, and 80GB NVIDIA A100s. Timing experiments are reported from experiments using 48GB NVIDIA RTX A6000 GPU. GRACE does not depend on the scale of the model, but the scale of the model depends on available compute resources. To avoid sharding, we use models that fit on one GPU, thought GRACE principles apply beyond this setup. For Adaptor-baal baaee editors, we use 100 iterations of gradient descent per input. For experiments using SCOTUS for classification and zsRE for Question Answering, we include early stopping once the model's 's output is corrected.
Hyperparameters. For our Finetuning, Finetuning with EWC, Finetuning with perioding retraining, Memory, Defer, and MEND comparisons, we consider learning rates of 1.0, le le- - 1e_2 le le-3 1e and 1 5_ We find that Finetuning, Memory, and MEND worked best with 1e-2, For Adaptor methods GRACE and Defer, we found that a large learning rate of 1.0 was required to update the model's behavior since their only control the model's output is by replacing tokens at layer.
Choosing which layer to edit is another hyperparameter for all editors. In all of our comparisons between editors, each editor edits the same layer. For T5, this is last dense layer of the last encoder block (encoder. block [7] .layer [1] DenseReluuDene. wo), for BERT it is the second-to-las layer (bert. encoder layer [10] output dense), and for GPT2-XL we edit the fully-connected component of the shitty-sixth layer (transf ormer. h [35] mlp c_fc). Layer 35 is not random: layers later than 35 led to less-consistent edit success. This is supported by recent work showing the impact of choosing the right layers to finetune (23). We study these layer choices below, but note that choosing this layer hyperparameter in practice: for the purposes of comparison, ensure editors are compared when editing the same layers.
GRACE has unique hyperparameter, Einit, which is the size of the e-ball initialized around new codebook entries. In our main results, we set Einit = 0.5 for zsRE (leads to 137 keys/ 1000 edits), Gui = 1.0 for SCOTUS (leads to 252 keys/375 edits), and Einit = 1.0 for Hallucination (leads to 1341 keys/1392 edits). We study the impacts of choosing Cim further in Appendices F and H
Runtime, Our main results report runtime for each method on the Hallucination dataset editing GPT2-XL. This time recorded is only the average time required to make one edit. This ignores all preprocessing, precomputation, or pretraining.
B Additional Dataset and Model Descriptions
As summarized in Table I we provide detailed summaries of the datasets and models we use for editing in our experiments.
B.1 Editing T5 on zsRE Question Answering
zsRE Dataset. zsRE 24 is a context-free question-answering dataset that has been extensively studied in the model editing literature [ 16 31] The same as previous works, we extract our set set from the same validation dataset. For our main experiments, our edit set is the first 200 question-answer pairs in the zsRE validation set that have at least 5 rephrasings each. We then use the first 5 rephrasings for each, leading to 1000 possible edits. To ablate design choices in GRACE, we use these same edits in Appendix
We also analyze GRACE's generalization (Table 2 in the main paper) and performance extremely long sequences of edits (Figure the main paper and Appendix HD using zsRE. To do this, we repeat the above procedure but use the 10 rephrasings from 1000 question-answer pairs. We then split this into an edit set and holdout set randomly, leading to 5000 potential edits and 5000 holdout edits.
T5 QA Model. We edit the aubiicl--vvillble 60-million parameter version of T5 Huggingface). This model was originally trained on the Nat- ural Questions dataset, of which we use 1000 random samples to evaluate TRR during edit- ing. This model achieves Fl of 0.72 on NQ and .11 on zsRE prior to editing. In
14

---

main results in Table we edit T5's "wo" module in its encoder's 4th block's layer 1:
encoder. block [4] 1ayer [1] DenseReluDense. wo.
B.2 Editing BERT on SCOTUS Classification
SCOTUS Dataset. We consider single-label multi-class classification task from Fairlex [6] SCO- TUS contains supreme court rulings on controversial issues [3]. Given a document describing the court's opinion, the task is to predict to which of 14 topics (classes) the document belongs. The 14 topics classes are clustered according to the matter of dispute: Criminal Procedure, Civil Rights, First Amendment, Due Process, Privacy, Attorneys, Unions, Economic Activity, Judicial Power, Federalism, Interstate Relations, Federal Taxation, Miscellaneous, and Private Action. There are 7.4k training documents from cases that took place from 1946-1982. Then, there are 914 validation documents from cases from 1982-1991. Finally there are 931 testing cases from 1991-2009. Since annotation guidelines do shift over time [38. exacerbate this shift slightly by making three realistic changes only to the training set: Relabel First Amendment as Civil Rights.
Relabel Due Process as Civil Rights.
. Relabel Uni ons as Economic Activity.
This realistic type of relabeling creates cases in these data that are especially challenging to forecast during training or adjust for during editing. We perform edits on the test set from 1991-2009, for which no relabeling was done. TRR is computed on the 914 validation documents from 1982-1991.
on Huggingface) on the SCOTUS training using Huggingface. We use 50 epochs with batch size of and a learning rate of 50-55 with AdamW optimizer using Huggingface's default model trainer. The finetuned BERT model achieves 0.99 Accuracy on its original validation data and 0.55 on the edit set, indicating a large amount of improvement to be made from editing. In our main results in Table edit BERT"s "dense" module in its 1hth layer: bert. encoder. layer [10] output Serrpuu.denn..... weight.
BERT. We finetune 110 million parameter BERT model
B.3 Editing GPT2 on Hallucination Language Modeling
Hallucination with SeI.CheckGPT. We examine how well model editors can mitigate hallucination in autoregressive language models using new dataset released with SelSCheckGPT [27]. To generate this dataset, which we refer to as "Hallucination," the authors prompt GPT-3 to generate wikipedia- style biographies for concepts extracted from WikiBio. They then annotate factual accuracy of each sentence, noting which are hallucinations. We propose editing highly-nnaccurate sentences by replacing them with corresponding sentences in the true wikipedia entries. To acquire "true" wikipedia entries, we extract wíkipedia summaries from WikiBio and acquire the correct sentence at the same index as the edited sentence. For example, if sentence 3 of biography of "Bon Jovi" is a hallucination, we would edit a model to produce sentence 3 of the wikipedia entry for "Bon Jovi"" given sentences and 2 generated by GPT-3. This way, every edit contains a prompt, which is the GPT-3-generated text up until the hallucination, and a label, which is the correct next sentence from wikipedia. There are 1392 potential edits and 516 already- accurate sentences. The editing task here is to decrease the LLM's perplexity (PPL) on the corrected sentences without increasing its perplexity on either the model's 's training data the already-corred portions of Hallucination.
This task is different from prior editing works in two ways: First, these are authentic mistakes made by high-quality LLM. Other works instead create synthetic edits by asking different language models to generate different inputs and swap between them. As their datasets are generated using far lower quality models, the sentences are highly unrealistic. Second, our edits are substantially longer than prior works, which have 10-token prompts and 10-token sentences to generate. By extending this paradigm to longer sentences, our editing task poses more realistic view of model editing success.
Since GPT-3 generated the Hallucination dataset, it would be ideal to edit GPT-3 directly. However, since it is proprietary and too large for practical research, we edit GPT2, using the 1.5B- parameter "XL" model (gpt2-xl on Huggingface). Being "small" LLM, this model is accessible to broad range of researchers yet is high-enough quality to serve as an effective stand-in for bigger models. Unfortunately, GPT2 is not already a high-quality biography-generre
GPT2-XL.
and so it finds all of the Hallucination sentences to be unlikely, even those that are "already correct" from GPT-3. To overcome this, we finetune GPT2 on the already-aceurate Hallucination data mixed with 200 random sentences from OpenWeeText [10), which is public version of GPT2's original training
15

---

data. This finetuned model achieves perplexity of 15.98 on Open WebText (comparable to GPT2-XL on its training data), 8.7 on already-accurate outputs, and 132.7 on intended edits. Our finetuned GPT2-XI. model will be made public with our paper. Thus with GPT2-XI. mimicking GPT-3 for these data, we can evaluate model editors on decreasing the edit PPL. while maintaining its PPL. on WeeeeeeTexxt and the already-accurate sentences. In our main results in Table 2 we edit GPT2-XL's "c_fc" module in its 36th layer: transformer [35] mlp. c_fc.
C Descriptions of Compared Editors
Finetune. We finetune the weights in chosen layer using Adam optimization, leaving all other layers of the pretrained model frozen. Finetuning is done using the Cross Entropy loss for T5 and BERT, and by minimizing the log probability of corrected sentences for GPT.
Finetune with EWC. We perform the same finetuning above, except update the weights with Elastic Weight Consolidation. This is a well-studied approach to minimize catastrophic forgetting in continual learning by regularizing weight updates leave important weights unchanged over time.
Finetune with Retraining. We perform the same finetuning as above, except periodically finetuning the original pre-trained model on all previous edits. This requires caching previous edits and may often exacerbate catastrophic forgetting on the model's pre-training data.
MEND (30.. MEND uses hypernetwork to forecast new weights for a pre-trained model's 's chosen layer by predicting low-rank decomposition of the layer's 's weight matrix. The hypernetwork is trained on a set of training edits, which include new edit, set inputs that are semantically equivalent to the edit, and samples from the model's pre-training data (or some analogous set). The network is trained to make the edit, apply the fix to the mewaaattcalll-equvvableee
inputs, and minimize the KL divergence between the new and old model predictions on the model's 's pre-training data. In our setup, we only have single edits that stream in, so there are neither training edits for the hypernetwork nor access to the model's pre-traine trainininng data. Therefore, we train the hypernetwork to predict updated weights edits stream in using continuous finetuning. Per the original paper's recommendations, we train the hypernetwork to predict a new gradient vector and value vector, which are multiplied to create a matrix with which update the model's s existing weights. We use the authors' original code from hew.b//ooomrrimiccellliiii
ROME [88]. ROME identifies weights in chosen layers of GPT models and updates them to alter GPT model's factual knowledge, Being designed only for GPT models, we evaluate this approach on the Hallucination editing task. We use the authors' original code gitggg..coommccc waarbb/ooo/meeggo///oooooo and use a template for wikipedia bios: This is a Wikipedia passage about O..
Defer. Inspired by SERAC 31I, we implement a deferral- based model editor, which we call Defer. At a chosen layer, for a new input Defer chooses to either 1) trust the pre-trained model's predictions, or 2) replace the layer's 's prediction with vector predicted by new model. This involves learning two new predictors for Defer Adaptor at given layer. First, a probability of deferral is predicted by one network RIM - [0, 1]_ For we use a one-layer network with a sigmoid activation function. Second, new value is predicted by another one-layer network R|a--11 IR "R In early experiments, we found that increasing these networks' depth overfits quickly. and o are jointly trained via the pre-trained model's finetuning loss continually as new edits arrive.
Memory. As a softer version of GRACE's codebook, we implement an Adaptor editor based on memory networks [46). This editor contains two components: A memory module that serves as predicted cache for learned activations and an attention mechanism that learns to retrieve appropriate memories. We implement the memory module as a matrix M E N containing N memories, each of which has h' dimensions. For our experiments, we set V 50 and the memory module is learnable over time. Then, we learn a simple attention mechanism g ==- - [0, 1|1*N where g's outputs sum to one. For g. Then, the editor layer's output is their multiplication g(h' M, resulting in a newly predicted [h'| dimeeniiona vector to be passed into layer + 1. M and g are trained jointly via the pre-traines model's 's finetuning loss continually new edits arrive.
one-layer neural network with softmax activation function.
16

---

PARAMETER
SEARCH SPACE
Edited Block
2.7 0.1. 0.5, 1.0, 3.0
Guil Replacement Token
Last Input Token, All Input Tokens, All Tokens
Value Initialization
Uniform: U(O0) 1) ("cold"), he ("warm""
Table 3: Hyperparameter options searched through for GRACE on T5 for zsRE edits.
D Interpreting GRACE codebooks
As introduced in the main paper in Section 3.2.4, benefit of GRACE is that the codebook is detachable from the pretrained model. This feature allows for closer inspection of the editing process. We first expand the results in the main paper by including the average FI score per key in Figure 7 where we see that keys overall achieve highly-accurate edits on holdout data. This implies that when a holdout question has key, the key maps to the correct answer. However, not all holdouts have associated keys. We therefore examine this further Figure X where we investigate the top-10 10 keys containing the largest numbers of holdout questions for 5 Einit choices. As expected, bigger Fint lead more holdouts per key. Meanwhile, small finit values lead to very few holdouts being captured by each key. Additionally, the big finit values lead to small codebooks and small values lead to big codebooks. So when 'nnt 0.1, 60% of the holdouts do eventually have a key, but this is at the expense of codebook size, which grows to 547 entries. On the right-hand side, we also see the proportion of the holdout set that has any associated key.
4-13
20
Avg. # holdouts perkey
F1 key
10
0.90
0.85
200
400
800
1000
0.80
800
800
1000
Inputs
Inputs
Figure 7: Interpreting generalizability of GRACE codebooks. F1 is computed unseen holdout edits that land inside any key's s e ball. The gray dashed line denotes the true average number of rephrasings per observed edit.
E Parameter Efficiency
As introduced in the main paper, GRACE's memory requirements are small. This is because new edit only requires | [h|| + parameters, where | is the dimension of the key, |h|] is the dimension of the value, and <an is a scalar. Further, the key's 's |h'- parameters are frozen and do require learning. At inference time, activating the Adaptor only alters the model's predictions according to the |/ - + [h'] + parameters of the chosen entry. As a concrete example, consider GRACE codebook with 500 entries that edits the 60 million-paramett T5 model. For 500 edits, we would then store 500 (1 + | + 1) = 500 * (1024 + 512 + 1) = 768, 500 parameters, of which only 500 (512 1) 256, 500 are learnable. 768, 500 is only 1.3% of T5's original 60 million parameters. Edits also appear sporadically in practice, so accruing 500 edits may easily span thousands or millions of inputs. While GRACE codebooks do grow over time, adding parameters according to the data size is quite attractive in such sequential setups. Other Adaptor methods, for far harder learning problem. MEND, and 112-dimensional vector, requiring 1,3 1,310,720 new
instance, initialize all their weights early on, leading for instance, predicts 1024-dimensions parameters to be learned all at once. This is reasonable their setup with training edits, but not directly applicable to our setup. Defer uses comparable parameter count to GRACE, adding 525,312. Memory, on the other hand, is tiny, adding just 26,624 learnable parameters because we set N 50. Early experiments with Memory indicate that increasing N is unhelpful.
17

---

Avg Holdouts Key: 2.31 Avg # Holdouts Key; 2.31
Holdout 1.0
Final Avg Holdout
# Holdouts Kay
30-
1.0
96/ooldeuts key
100
0.8
80
20
60-
10-
100200 400 7l08b000o coo
100200300 400 s0beco 700800 co0
100 200360 400 700 aco soaro0o Inputs
Inputs
Key
Say
Key
Kay
Kay
Kay
Kay
Say
10
(a)
0.1. The final codebook contains 547 entries.
Avg # Holdouts Key; 2.58
Arg Holdout F1: 1.0
Pinal Avg Holdouts Key: 2.85
Final Holdout 11:1.0
100
* Holdouts Key
%Hollouts koy
860
60
15-
40
0 100200 d00 s0oc0o 700 800 5000000
100:0030000000 600 500600 700600 90C1000
200300400 500600 800 5001000
Inputs
Inguts
Inguts
Kay
Kay
Key
(b)
1.0. The final codebook contains 423 entries.
Avg # Holdouts Key: 3.72
Avg Holdout P1: 0.99
Final Avg # Holdouts / Key: 4.09
Final Avg Holdout Fl: 0.99
Holdouts /Kay
80
Kay Avig-
Holdoosss
60
15-
40
0.2
20
300 500000
0.0
10020030040000000 soocloo0 BU0S0CL000
800 9001000
Inputs
Inputs
Iaputs
(c) Cani
1.5. The final codebook contains 279 entries.
Avg Holdouts Key:
Avg
Final Avg Holdouts/ Key: 6.03
Final Avg Holdout Fl: 0.98
30
1.00
1.0
Holdouts Key
0.8
20
15-
0.8
Holdouss
1.-
0.4
0.2-
20
100200:000 6005006000 700 HOIE
100200330 scon007000 MOU
300200 300-400 5001600 800 90001000
Inputs
Inputs
Inputs
Kry
Key
Kry
Key
Kry
Kay
Kry
Key
Kay
Key
(d) Giit
2.0. The final codebook contains 184 entries.
Avg Holdouts Key: 5.96
Avg Holdout 0.96 Avg Holdout
Final Avg Holdouts Key:
1.00
# Holdouts Kay
Holdnnts kay
80-
20-
60-
15.
Key
60-
0
dezon soo-sins
ddezie-annn.eessm noartooado dimmoco Inputs
1o 200200.iii dd mon slencho
Inputs
Inputs
Kay
Koy
Key
Koy
Key's
Koy
Key?
Kay
Kry
(c) 3.0. The final codebook contains 150 entries.
Figure 8: Interpreting per-key codebook behavior for 1,000 inputs for Block 4 of T5. At each step of editing, record how many holdout samples captured by key and if so, which key. We also report the percentage of the holdout dataset that is captured by any key in the right-hand panel.
18

---

1.00
1.00
1000
0.72-
Num Keys
800
0.70-
ERR
600
0.50
0.60
400
200
500 Steps
Steps
Steps
Steps
Val init: warm. Replace all. E: 1000. init: cold. Reginee
Val init warm. Replace prompt. B: 1000.
Vel into: warm. Replace last. 1000.
oold. Replaos prompt. R: 1000.
cold. Replace: last. 1000.
(a) Block 2. Cinit 0.1
1000
0.72
Num Kays
800
0.70
600
0.50
0.68
0.60
0.86
200
500 Steps
1000
500 Sleps
1000
500 Sleps
1000
500
1000
Steps
Wi Init: Replaoe: all. 940.
Val init: warm. Replace: prompt. B: 1000.
Vol init: cald. Replace: prompt. E: 1000.
lnit: Replace: 1000.
wann. Replace:
Val Replace:
(b) Block 2. Cinit
0.5
0.72
Num Koys
800
0.80
TRR
0.70
2 0.50
0.88
0.60
400
0.68
200
500 Steps
1000
0
500 Steps
1000
0.00
500 Steps
1000
0
5/0
1000
Steps
Val
prompt. 1000.
inlt:
1000.
Val lnit: cold. Replace: all. 1000.
Val lait cold. Replare: prompt. E: 1000.
Init: cold. Replace: last, E: 1000.
(c) Block 2. Cinit
1.0
1.00
0.72
Num Keys
0.80
TIR
0.70
2 0.50
0.68
400
0.66
0.00
200
500
1000
500
1800
0.00
500 Steps
1000
500
1000
Steps
Steps
Steps
warm. Replace:
Replace: prompt. 1000.
ini:: warm. Replace:
Val loi: cold. Replace: all E. 1000.
Val init. oold. Replace: prompt. E: 1000.
Vel init: cold. Rep lace: last. D. 1000.
(d) Block 2. Cair 3.0
Figure 9: Ablation Study on editing T5 for zsRE on 1000 inputs. "E" denotes the number of edits performed. "Steps" denotes the candidate edit in the edit set, since over 1000 inputs, all will be flagged as errors.
19

---

1.00
1.00
1000
0.72-
Num Keys
800
0.70-
600
0.50
0.60
400
200
Steps
Steps
Steps
Steps
Val mit: cold. Replace all. 1000.
Val tnt: cold. Replace: prompt. Ei 1000.
Vel mit: cold. Replace: last. B: 1000.
Reglace:
(a) Block 7. Einit 0.1
1000
0.72-
Num Kays
800
0.70-
600
0.50
0.68
0.60
0.86
200
500 Steps
1000
500 Sleps
1000
500 Steps
1000
500
1000
Steps
Val init: osld. Replaoe all. 1000.
Val Init: cold. Reglace prompt. E: 1000.
Wil init: cald. Replace: lost. E: 1000.
Replace:
(b) Block 7. Cini
0.5
0.72
Num Koys
800
0.80
TRR
0.70
2 0.50
0.88
0.60
400
0.68
200
500
1000
0
500 Steps
1000
0.00
500 Steps
1000
0
540
1000
Steps
Steps
Init: Replace: 1000.
init:
Val cald.
init: cold. Replace; prompt. 1000.
(c) Block 7. Cinit
1.0
1.00
0.72
Num Keys
0.80
TRR
0.70
2
0.68
400
0.66
0.00
200
500
1000
0
500 Steps
1800
0.00
500 Steps
10000
500
1800
Steps
Steps
Beplace,
warm. Replace:
Replace:
Val int: cold. Beplace prompt. C. 1000.
(d) Block 7. Cair 3.0
Figure 10: Ablation Study on editing T5 for zsRE on 1000 inputs. "E" denotes the number of edits performed. "Steps" denotes the candidate edit in the edit set, since over 1000 inputs, all will be flagged as errors.
20

---

GRACE KEY EDITS
TRUE LABEL
What type of voice is Gemma Bosini?
Soprano
What kind of voice Gemma Bosini?
What is the voice of Gemma Bosini?
In what war did Carlos W. Colby fight?
American Civil War
What war Carlos W. Colby fighting in?
What war or fight did Carlos W. Colby light in?
What war did Carlos W. Colby fight in? What war battle was Carlos W. Colby fighting in?
8
By whom was the Queen Victoria Building designed?
George McRae
Who was the Queen Victoria Building designed for?
Who was Queen Victoria Building designed by?
Which person designed Queen Victoria Building?
Which architect designed Queen Victoria Building?
Table 4: Examples of GRACE codebook entries. "GRACE Key" is the index of the chosen codebook entry for these inputs. "Edits" are inputs that were wrong prior editing and are fixed by the same key. "True Label" is the true label associated with these inputs and the codebook's 's learned value.
F Ablation Study
We ablate GRACE components by comparing four design choices using different layers and choices of Can using T5 on up to 1000 edits from zsRE. The hyperparameters we search across are shown in Table 3 We focus this study on the T5 encoder's Blocks 2 and 7, which pose significantly different behavior.
As our results in Figure show, there are interesting relationships between value initialization and token replacement. Starting with the right-most panels, we excitingly observe that GRACE achieves significant edit compression: For up to 1000 edits, the GRACE codebooks often end up with only about 200 keys. This marks success in our codebook maintenance strategy, since this can only be achieved if values increasing appropriately for edits. For small Gii values, the number of keys grows larger, up to about 600, demonstrating the trade-off between codebook size and memorization. This trade-off is seen in the three left panels measuring Training Retention Rate (TRR), Edit Retention Rate (ERR), and Edit Success (ES), all in terms of F1. For the smallest Cinit 0.1, we see these three metrics remain high throughout editing. However, looking down through Figures 9a 9b 9c and 9d we see TRR and ERR drop as the codebook generally trades size for generalizabilin ity. This trend remains true when editing Block 7 (Figure TOp, though this later layer appears far harder to edit successfully. This may hint at a failure mode of GRACE that may be addressed by further hyperparameter tuning: If a new key is initialized yet fails to correct the model's behavior, even identical future inputs will still require edits. This may be addressed by training values for longer or adding new value training methods: In principle, learning a new GRACE value is no different from training any model, so may benefit from any advanced optimization method.
For hyperparameter Replacement Token, see that replacing just the value for the Last input token appears to be superior when editing Block 2, especially for TRR. This makes sense because replacing only the last input token will likely have the smallest impact on unrelated inputs. Excitingly, this replacement strategy remains on par with the others for ERR and ES. For Block 7, however, the opposite true. Given the low overall ES, it remains unclear how replacement strategies differ in general for this layer.
For hyperparameter Value Initialization we see negligible differences between warm and cold starts for Block 2. For Block 7, cold appears overall worse than warm, despite having higher ES.
Edit Compression. Compressing multiple edits into singular keys is bounded by the number of unique desired labels since each GRACE value points to one desired output. For example, if new edits come from 5 different classes, GRACE must include at least 5 keys. As we saw in Figure GRACE can compress many edits into single codebook entries successfully. Table shows examples of inputs that all lead to the same key.
21

---

Inputs
0.6
10
TRR
0.4
100
250
0.2
500
750
0.0
1000
FT
FT+EWC
FT+Rt MEND
Defer
Memory
GRACE
(a) Training Retention.
1.0
Inputs
10
ERR
100
0.5
250
500
750
1000
0.0
FT+EWC FT+Rt
MEND
Defer
Memory
GRACE
(b) Edit Retention.
1.0
Inputs
50
B5
100
0.5
250
500
750
1000
0.0
FT
FT+EWC
FT+RL
MEND
Deler
Memory
GRACE
(c) Edit Success.
Figure 11: Comparing editors over time when editing T5 on zsRE.
G Extended Main Results
To supplement Table in the main paper, include all metrics computed throughout editing for
all comparisons. We first show barplots Figures IT 12 and 13 We see that the trends match the main paper: GRACE achieves strong performance compared to the comparisons. We also show finer-grained versions of these plots computed for each edit over time in Figures 14
15 and 16
H Extended Hyperparameter Study
To further extend our study on the impacts of the most important hyperparameters eiit and the chosen layer, we run a far-longer experiment on the QA editing task, making to 5,000 edits. Our results from this experiment are shown in Figure 17 In this finer-grained analysis, we find that our main claims remain true: some layers are better to edit than others, GRACE succeeds to generalize to previously-unsee inputs, e controls the trade-off between memorization and generalization, and GRACE codebooks stabilize in size over time. Performance remains the same above = 6.0. Interestingly, for 2.0, Layer appears be highly unstable over time.
22

---

1.0
Inputs
0
TRR
50
0.5
100
250
500
750
0.0
1000
FT
FT4EWC
FT+Rt MEND
Defer
Memory
GRACE
(a) Training Retention.
1.0
Inputs
10
50
ERR
100
0.5
250
500
750
1000
0.0
FT
FT+EWC
FT+RL
MEND
Deler
Memory
GRACE
(b) Edit Retention.
1.0
Inputs
100
0.5
250
500
750
1000
0.0
[
FT+EWC
FT+RL
MEND
Deler
Memory
GRACE
(c) Edit Success.
Figure 12: Comparing editors over time when editing BERT on SCOTUS.
23

---

400
Inputs
0
300
10
TRR
100
200
250
500
100
1000
0
1392
FT+EWC FT+RL MEND
Defer
ROME Memory GRACE
(a) Training Retention.
Inpuls
10
40
50
ERR
100
250
20
500
750
1000
1392
FT
FT+ EWC FT MEND
Defer
ROME Memory CRACE
(b) Edit Retention.
Inputs
400
10
50
ARR
100
200
250
500
750
1000
0
1392
FT
FT+ PT+ MEND
Defor
ROME Memory GRACE
(c) Already-Accuraln Retention Rate.
100
Inputs
75
50
100
ES
50
500
25
750
1000
0
u
1392
FT FT+EWC
Defer
ROME Memory GRACE
(d) Edit Success.
Figure 13: Comparing editors over time when editing GPT2-XL Hallucination.
1.00
1.001
TRR
0.60
0.75
0.75
0.40
0.50
100
0.20
0.25
0.25
Steps
Steps
Steps
FT
FT' Retrain
MEND
Defor
Memory
GRACE
Figure 14: Comparing all methods throughout editing on zsRE.
24

---

0.75
0.50
0.50
0.50
0.25
0.00
0.00
200
800
1000
200
800
1000
0
200
500
600
1000
Steps
Steps
Steps
PT+EWC
PT+Retran
MEND
Defer
Memory
GRACE
Figure 15: Comparing all methods throughout editing SCOTUS.
400.0
50.0
300.0
40.0
TRR
ERR
30.0
200.0
20.0
100.0
10.0
0.0
0.0
250
500
750
1000
1250
250
500
750
1000
1250
Edits
Edits
500.0
100.0
V
400.0
80.0
ARR
60.0
ES
200.0
40.0
100.0
20.0
0.0
0.0
250
500
750 Edits
1000
1250
250
500
750
1000
1250
Edits
Defer
FT+EWC
GRACE
MEND
ROME
FT
FT+ Retrain
Figure 16: Comparing all methods throughout editing on Hallucination.
25

---

Num Key'
3k
0.70
Holdout
TRR
ERR
0.90
2k
0.60
0.65
0.80
1k
0.40
0.60
0.70
1k
2k
3k
1k
2k
3k
1k
2k
3k
1k
2k
3k
Edits
Edits
Edits
Edits
0.1
1.00
0.80
Num Keys
3k
0.70
Holdout
TRR
ERR
0.90
2k
0.60
0.65
0.80
1k
0.40
0.60
1k
2k
3k
0.70
1k
2k
1k
2k
3k
1k Edits
2k
Edits
Edits
Edits
(b) Fiiil
1.00
3k
Num Keys
0.70
Holdout
TRR
ERR
0.90
2k
0.60
0.65
0.80
1k
0.40
0.60
0.70
1k
2k
3k
1k
2k
3k
1k
2k
3k
Edits
Edits
Edits
Edits
Block:
Block:
Block:
Block:
(c) Emit 3.0
Figure 17: Impact of Emn and block choice for GRACE editing T5 on zsRE for 3000 sequential edits. Along with TRR and ERR, we also measure F1 on a "Holdout" edit set containing unseen rephrasings of all 3k edits. We find that editing blocks 0 and 6 use more keys and achieve higher TRR, but lead to lower ERR and generalize worse, as shown by lower "Holdout" values.
26


Reviews:
[{"review_id": "hqzdrXYW6I", "reviewer": "Reviewer_Qd7q", "summary": "The paper proposes GRACE, a method towards lifelong editing of models through the use of a codebook, a mechanism to write to/make use of the codebook without perturbing model weights. In essence GRACE can be viewed as an inserted adapter layer, which caches activations as keys, fine tunes values that the keys map to, and employs a so called deferral radius per key to decide when to activate GRACE.   \n\n---- \nRating raised post-rebuttal from 6 to 7, as weaknesses were at least in parts clarified or addressed. ", "questions": "It is not really clear how to choose GRACE’s hyper-parameter epsilon initially. The intuition has been nicely explained, i.e. large values will apply edits with more influence, but it is not obvious how one can pick the value in advance before starting to do lifelong editing for an arbitrary dataset. In this scenario, hyper-parameter search does not seem possible because one cannot look into the future. Additionally, figure 4 also shows even more trade-offs with respect to choice of layer, which seems to be a second hyper-parameter in that sense. It is not clear how to decide on what layer to edit at which value, specifically if the values of epsilon for different layers need to be different for GRACE to work effectively in practice.  It will be appreciated to get clarifications here in the rebuttal. \n", "limitations": "Limitations are mentioned appropriately and are appreciated. Divining deeper into the mentioned limitation of slowing down inference through similarity computation, along with potential other limitations on storage cost and codebook overhead, will be a further improvement to the paper and its limitations section. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has many strengths, starting from a well-motivated introduction and scope that tackles a highly important problem. \n\nIn general, the exposition of the paper is clear and it is well written and structured. The mechanisms are mostly explained in sufficient detailed and GRACE itself becomes intuitive through the added illustrations. \n\nThe empirical investigation further investigates a set of metrics and includes a set of meaningful benchmarks. Within the given set of explored alternatives, GRACE can be viewed to provide consistent advantages, making GRACE a good contender for this kind of problem setting.  \n\nThe addition of a limitations section with some statements on potential (harmful) social impact is also appreciated. ", "weaknesses": "The paper seems to have two primary weaknesses. \n\nOne lies in the exposition with respect to related work and its empirical comparison, the other is a potential concern with GRACE’s practical hyper parameters. \n\nWith regard to related work, the empirical comparison to continual learning works seems interesting and it is not fully clear why it is meaningful. If one were to take a look at one of the plethora of continual learning reviews (take Hadsell et al, Mundt et al, de Lange et al), they typically categorize techniques into rehearsal (memory buffers), adding components to the architecture or regularizing parameters. The comparison in this paper primarily seems to compare to fine tuning and EWC, which fall into the regularization category. Yet, to the best of my understanding, GRACE implies the use of extra parameters and memory, which would in that sense be much fairer to compare to continual learning methods to do something similar. There are many of those, but think of storing attention masks, adaptively learned gates, or memory buffers that similarly store activations. It is presently not clear how GRACE differs conceptually from these prior arts (mentioned in e.g. the above surveys) and following in this line, what the “cost” of GRACE is with respect to growth/memory of the codebook. \n\nAs mentioned above, the second concern lies in the choice of parameters, specifically how to pick appropriate values for epsilon and how to decide which layer to edit. From figure 4 it seem the system is not very robust to this choice necessarily and it becomes a trade-off with respect to what value is chosen for which layer. Whereas this may be clear, it will be important to clarify how to pick suitable hyper-parameters without being able to look into the future in lifelong editing (i.e. plotting figure 4 and deciding on how the curve develops on the x-axis); also see below question.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback and positive response, we have addressed your hesitations below.\n\n**Weakness 1: Empirical Comparison to Continual Learning and Cost**. We agree that our setup differs from standard continual learning, but we think some comparison is appropriate. As you note, we do compare with EWC. Beyond that, we also compare against periodically re-training the model on a cache of previous edits (Finetune+Retrain), which is a form of replay. And while they are not existing continual learning methods, MEND, Defer, and Memory do add extra architecture components. Overall, our focus is on sequential model editing, so the differing setup does bar more direct comparison. Still, we believe these comparisons have some overlap with continual learning methods and shed light on potential solutions to our problem setup.\n\nThe cost of GRACE in parameter count is discussed in Section 4.2.3: Adding a new edit requires $|h^{l-1}| + |h^l| + 1$ parameters, where $|h^l|$ is the dimension of the hidden state computed at layer $l$. Per the details in Appendix D, in our T5 experiments, for example, 500 edits is equivalent to only 1.3% of T5’s total number of parameters. While this indeed grows over time, we believe the benefits of editing outweigh the costs for the scales we consider. Further, our new results show that GRACE edits are more general than previously-thought, decreasing the growth-rate of our codebooks. We believe future works will continue to make even more generalizable edits and maintain even smaller codebooks.\n\n**Weakness 2 and Question: Hyperparameter selection**. We are happy to note that our Updated Results PDF shows that GRACE is more robust to hyperparameter selection than previously thought, so there is less pressure to get this choice exactly correct. Still, hyperparameter selection in continual settings is indeed a challenge. We target the hardest setting where we assume access to no edits before applying GRACE, which is desirable and warrants further study. However, sometimes it may be possible to collect a small dataset beforehand and tune our (few) hyperparameters. For instance, there may be ways to use exogenous data (like ROME uses Wikitext) or internal validation sets to pick the layer and $\\epsilon_\\text{init}$. Further, in some settings, there may eventually be practical ways to reset or update GRACE codebooks throughout editing. This is actually a strength of GRACE-style edits: By leaving the model’s weights unchanged, only the codebooks require updates, so model checkpoints don’t have to be stored in order to revert any edits.\n"}, {"review_id": "uVUhL2PlSr", "reviewer": "Reviewer_2Kqs", "summary": "The authors propose a method to editing a model’s behavior, making corrections or specifying the desired output for a particular input. Such a method is broadly applicable, but is especially relevant in the context of large pre-trained models, which are costly to fine-tune or re-train, and whose data is often not publicly available. Specifically, the authors propose General Retrieval Adaptors for Continual Editing (GRACE), which store activations (keys) for past edits into a codebook; if a particular input is close enough to one of these stored keys, then the output of the layer is overridden by the key’s corresponding value. A deferral radius for each key allows grouping of similar edits and provides some generalization behavior. For experiments, the authors compare with a number of relevant model editing baselines, editing large language models (LLMs) hundreds to thousands of times, which is significantly more than prior evaluations. Empirical comparisons look promising across a variety of editing metrics.", "questions": "Q1. What causes the fluctuations in ERR when applying GRACE to layer 6 with $\\epsilon=1.0$ in Figure 4?\n\nQ2. How does the inference time change with increasing numbers of edits?\n", "limitations": "The limitations section is well-written and frankly acknowledges some valid limitations and potential ethical concerns. While important, I don’t necessarily see these as blocking acceptance of the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "S1. Relevance: This paper tackles a topic that is especially important given recent developments. Large models trained on massive data (e.g. language, vision) are seeing increased interest and more exposure to the general public with the releases of chat bots/search, image generators, etc. This is largely due to impressive results, but there have also been a concerning number of wrong, offensive, or misleading examples as well. As these models continue to be used by the public, some way to apply targeted patches is essential.\n\nS2. Assumptions: GRACE has relatively simple and realistic model assumptions. In particular, unlike some prior work, it does not require access to the pre-training data; this is important, as such data is often proprietary and massive in scale, as well as a potential data privacy risk.\n\nS3. Experiments: \\\na) Realistic settings: While past methods tend to evaluate success of a single or a handful of edits, this paper evaluates on the ability to make hundreds to thousands of edits, sequentially. This is a significantly more realistic and utilitarian setup. \\\nb) Three text-editing settings: The authors include experiments on multiple settings (context-free question-answering on a T5, BERT classification on SCOTUS dataset, and correction of hallucinations on GPT-3 wiki bios). \\\nc) GRACE outperforms a good selection of competing methods on a number of editing and retention metrics, demonstrating strong empirical potential.\n\nS4. The paper is well-written and easy to understand. The figures and illustrative examples do an excellent job conveying the key concepts. \n", "weaknesses": "W1. Editing examples vs concepts: From my understanding, the proposed method focuses on editing predictions of a model for a specific example. This contrasts with continual learning methods, which tend to focus on learning new concepts or classes. While the editing approach is good for correcting mispredictions or hallucinations, it doesn’t seem like it is capable of learning to make new predictions: for classification, this could represent new categories or new vocabulary.\n\nW2. Scalability: The number of GRACE layers required in the codebook mostly scales linearly (technically sublinear because edits can be grouped with previous GRACE layers, but from Figure 4, the overall behavior is still close to linear—converging to 50% cache rate is still O(n)). In addition to storing an increasing number of GRACE layers, this increasingly large codebook also requires an increased number of comparisons at inference time, for *every* forward pass, even inputs that are correct and otherwise unaffected by the edits. Experiments show up to 5K edits, but real-world deployments may (e.g. chatbot search engines) may require significantly more.\n\nW3. GRACE layers value storing vs override: My understanding is that the models being considered in this work are fully deterministic during inference. Thus, once a particular input triggers a particular GRACE layer anywhere in the model, then the retrieved activation $h^l$ should always result in the same output. This raises a couple questions: \\\na) If the output from a GRACE layer deterministically maps to a particular output, then why wouldn’t we instead directly store the desired output and save the computation of the subsequent layers $l+1, …, L$? \\\nb) This also suggests a much simpler approach: maintain a dictionary of input-output corrections, and check if the input exists in the dictionary before running the model. The primary advantage of GRACE then would be the potential for better generalization, as matching to previous edits is done at the feature level, but this needs to be verified empirically.\n\n\nLess critical concerns:\n- Title: While the title is kind of cute, “Aging” is only partly relevant to the problem being addressed here. Edits are also necessary for examples for what the model gets wrong, even when it’s a “newborn”.\n- I think it’s debatable whether the proposed method should be considered a “continual” or “lifelong” learning method, as it’s more or less sidestepping the catastrophic forgetting problem altogether with a roughly linearly increasing codebook of overrides (see W2). With the $\\epsilon$ radius, there is some potential for forward or backward transfer, but I would classify it as minimal and relatively rudimentary. On the flip side though, I can see similar arguments being made about some expansion-based CL methods, though generally their scaling characteristics aim to be better.\n- I was a little disappointed that there weren’t any experiments with large vision models, but I understand the desire for limiting scope.\n\n\nMiscellaneous:\n- Fig 2: “No near key” => “No nearby key”\n- Equation between Lines 139-140: The top line doesn’t quite make sense. I believe the authors meant to write something along the lines of “if $d(h^{l-1}, K_{i_*}) < \\epsilon_{i_*}$” instead. Also please include an equation number.\n- Line 149: “one of two”?\n- Line 191: missing period “.”\n- Line 279: possibly a missing space after the “:”\n\n========\n\nOverall Assessment:\n\nI found this paper interesting and well presented; the proposed GRACE layers appear to be a reasonable, straightforward solution to editing. However, I’m concerned about the method’s overall scalability, especially when the method in many ways is similar to keeping a dictionary of input-output overrides (e.g. prefacing the model inference call with simple if-then statements). By operating in feature space, perhaps GRACE has better generalizability than such a rudimentary approach, but there is little empirical evidence of this. Also, it still isn’t clear to me why values are activations for the next layer as opposed to just jumping straight to the output. I’m willing to raise my score if my concerns are sufficiently answered.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback, we're glad for the chance to clarify some points.\n\n**W1: Editing examples vs. concepts**. There are many types of continual learning problems. Adding “new categories or new vocabulary” is often called “Task-incremental” or “Class-incremental” continual learning. Our problem is closer to “instance-incremental”, where data share their category/vocabulary, but arrive over time (though we observe only errors, unlike in regular continual learning). We believe this is a good starting point (our experiments show it’s hard), and we hope our work will inspire solutions in other interesting problems. In fact, GRACE *can* represent new categories, as long as the model can output the appropriate tokens. For example, in generative language modeling wherein output classes are verbalized to be tokens within the model's vocabulary. We think this is a further benefit of editing internal transformations (we expand on this when addressing W3).\n\n**W2: Clarifying “\\% Cached Edits” vs. “Cache Rate”**. We would like to clarify that “\\% Cached Edits” (Figure 4) is not a “cache rate”. Instead, it is the total number of edits that required new keys to be created. So if after 4k edits there are 2k keys, we would report 50\\% Cached Edits. This is a confusing way to report our findings, so in Figure 1 of our Updated Results PDF, we plot the codebook size (Number of Keys) per step. Here we observe smaller codebooks than in the original Figure 4.\n\n**W2: Scaling to more edits**. We agree that extremely long sequences of edits can exist in practice. Given today's alternatives, we believe GRACE is a good place to start. But if we expect millions of edits, for instance, GRACE's caching may be troublesome. Still, this is out-of-scope for our work because our aim is to first develop a method for sequential editing when rapid finetuning is undesirable or impossible. Given a large, newly collected dataset with corrected edits, pre-training data, and enough computational resources, standard finetuning will likely be appropriate. Still, even if GRACE had a high cache rate, we believe GRACE can be scaled, especially given works that have scaled vector similarity search [[2](https://engineering.fb.com/2017/03/29/data-infrastructure/faiss-a-library-for-efficient-similarity-search/)] and growing interest in vector databases. Further, choosing between parametric and non-/semi-parametric methods is often based on data scale, which is not a problem unique to our work. GRACE allows the user to “interpolate” between these regimes. We agree clarity on promising settings will strengthen our paper, which we will discuss in our Appendix.\n\n**W3: Value storing vs. override**. Our evaluation is deterministic, even though T5, BERT, and GPT2-XL can all be decoded probabilistically. This is an attractive and popular feature, especially for language generation. GRACE does not impair non-deterministic use of any method since we do not alter the model’s outputs directly. For instance, a GRACE-edited model can still produce non-deterministic outputs if sampling is used during decoding. We believe leaving the model outputs intact is actually important, especially given today’s focus on probabilistic decoding. Per your questions:\n\na) Retrieving a looked-up answer in the middle of the network is indeed feasible. But there are limitations depending on the task. For classification, returning the exact label seems quite reasonable (though batch inference with elements exiting the network at different times would require extra tweaking). For language modeling, though, generating exactly the same output each time (as storing the desired output would do) is not the end goal, and moreover, the embeddings are needed to be attended over during autoregressive decoding. So by caching embeddings, not outputs, we keep GRACE general purpose and hope to inspire further work on editable decoding.\n\nb) Caching input-output pairs is also possible, but removes the benefit of using semantic similarity according to the pre-trained model, as you mention. Fortunately, our Updated Results PDF does show improved generalization. Plus, improving our generalization (and that of other model editors) is an exciting and challenging task that we believe will inspire future work.\n\n**Less-critical concerns**.\n1. *Title*: We agree that edits are necessary at each stage of deployment. By “aging” we want to imply post-training, not early vs. late deployment.\n2. *Is ours a lifelong solution?* Fortunately, our Updated Results PDF address the trade-off significantly. We also absolutely agree that improving generalization is an important and hard step (and all model editing works find this is hard). We hope our method is the first of many steps in this direction for our problem. We agree that GRACE is applicable beyond NLP tasks and this is an exciting avenue for future work.\n3. *Misc.*: Thank you for catching these typos, we have updated the paper.\n\n**Q1: Fluctuations**. This is a good observation. We'd like to point out that fluctuations decreased in our Updated Results PDF, and those we observe change at a smaller scale. These fluctuations imply that editing the model influenced its behavior on previous edits or training data. This is probably due to miscalibrated deferral radii, which could be why fluctuations get bigger as $\\epsilon_\\text{init}$ increases.\n\n**Q2: Inference Time**. We observe small changes in inference time as the number of edits increases (results below). Extreme settings (millions of edits) probably require new considerations.\n\nThe following table shows the time it takes to add an edit (which includes inference) while editing GPT2-XL. The edit time barely increases as codebook size grows (experiment done on an NVIDIA A100 GPU with 40GB VRAM).\n\n | Codebook Size | Edit Time (s) |\n | ------------------- | -------------- | \n | 10 | 5.76 | \n | 100 | 6.54  | \n | 250 | 8.10 | \n | 500 | 5.94| \n | 750 | 6.06| \n | 1000 | 8.58| \n | 1340 | 8.10 | \n\n"}, {"review_id": "XQR1dwdbzl", "reviewer": "Reviewer_nd9W", "summary": "The paper presents a new method for sequential model editing (GRACE). The approach works by modifying the output of a pretrained model's given layer $l$ through the use of an external key : value storage. Whenever an edit is required on input $x$, the model stores the input representation $h^{l-1}$ as key, and learns a value $v$ via gradient descent which yields the edited output $y_{edited}$. In order to enable generalization, the authors propose to map all representations within an adjustable (edit-specific) $\\epsilon$ to the stored value $v$. Crucially, to enable sequential editing, the authors propose to update the codebook (Alg. 1) the following way : whenever a new edit overlaps with the $\\epsilon$-ball of a previous edit, this $\\epsilon$ is either shrunk to reduce the overlap between edits, or expanded if the two edits share the same desired output. \n\nThe authors evaluate their method according to three metrics, which look at whether the sequentially performed edits are memorized, and the model's ability to preserve its original input on non edited points. From the results, it is shown that the proposed method performs across several benchmarks. Finally, the authors perform an model analysis, looking at the impact of $\\epsilon$ and the choice of layer to add the GRACE mechanism. ", "questions": "1. Could the authors please explain how GRACE layer deals with multi-token inputs with autoregressive models ? What is implied by \"replace only the final token of an input prompt\" ? \n2. How are rephrasings generated for the results shown in Table 2 ? \n", "limitations": "limitations properly addressed. ", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality : \n - The proposed is, to the best of my knowledge, a novel approach of an external discrete memory for sequential model editing\n - Grace seems like a solution which could actually be deployed in realistic edit scenarios\n\nQuality / Clarity : \n- The paper is well written and easy to follow. \n- The intuition behind how the method works is well-built, and the model analysis section is interesting\n- The experimental section is overall well-designed\n", "weaknesses": "1. Given that the choice of layer and value for $\\epsilon$ seems to highly impact the behavior of the deployed system, it's unclear how practitioners would choose this value a priori. Given that this deployed in an online setting, standard cross-validation cannot be used directly. \n2. The experimental section is somewhat limited, as the authors only look at NLP tasks. That being said, given that the authors evaluate different model scales, this is more a suggestion on how to make the paper more complete than a hard weakness. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your valuable and positive feedback! We respond to each of your concerns below.\n\n**Weakness 1: Hyperparameter tuning**. Our new results (Updated Results PDF) show that GRACE editing is far less sensitive to hyperparameter tuning than we originally thought, so picking the right parameters initially is lower-stakes (also discussed in the global response). Still, in practice, we agree tuning the hyperparameters is a challenge, present in many continual or online learning setups. While we leave richer solutions to future works, one practical method could be to use internal validation data or external data (e.g., ROME precomputes statistics about their models using 100,000 samples from the public Wikitext dataset).\n\n**Weakness 2: Extending to more modalities**. We absolutely agree that GRACE is applicable far beyond NLP tasks and we are excited to see how GRACE works on other modalities. Given that most recent works on model editing focus on NLP tasks, we believe our experiments present fair comparisons. Extending to more modalities is indeed an exciting direction and we hope our work inspires others to work on this in the future.\n\n**Question 1: Handling multi-token inputs**. When editing in an autoregressive setting, we start by passing a prompt into GPT2-XL. Then, the embedding for the final token in the editing layer becomes the query for which we compute a new entry in the GRACE codebook (or look for existing, nearby keys). By “final” token, we mean the last token in the input prompt. We then learn a new value just for the same, final token, which will be passed into the next layer. During inference, if we passed the exact same prompt in again, the final token output by the editing layer would then match the previously-added key and be replaced with the GRACE-learned value, inducing the desired response from the model.\n\n**Question 2: Source of rephrasings**. The zsRE dataset includes a set of rephrasings for each instance. This is why we focus our study of generalization on the zsRE dataset, since rephrasings are often hard to acquire in practice. This fact limits the direct applicability of other works that require rephrasings in order to make edits."}, {"review_id": "eW1AnTn9gS", "reviewer": "Reviewer_wTwG", "summary": "The authors propose GRACE, an approach to model editing that is tailored to many-edit scenarios. GRACE augments a hidden layer of a neural network with a key-value memory that overrides the output representations of the layer when the input representation falls near the stored key. The method is simple and computationally efficient, and compares favorably with existing editors. Closer analysis shows that GRACE ultimately provides a tradeoff between forward generalization of past edits and retention of pre-existing knowledge.", "questions": "L156: Does this expansion process ignore the fact that we can have multiple word senses? For example, if we have an edit for “the robber of the ___” and “the house by the river ___”, even though the labels for both edits might be “bank”, it seems like we’d want to represent these separately.\n\nL257: “unsurprisingly, MEND underperforms since it forecast edits for new, unobserved inputs” What does this mean?\n\nL87: what privileged information does MEMIT use?\n\nL139: Can we not have the case where the hidden state is not within the deferral radius of the closest GRACE key, but the hidden state is within the deferral radius for the n-th closest key? Considering only the closest key seems like it ignores this possibility.\n\nWhat, in the authors' view, is the primary cause of the difficult tradeoff between retention of old information and forward generalization of edits? Currently, GRACE either requires a problematic destruction of pre-trained knowledge or a failure to apply edits to future, related edits.", "limitations": "The authors include a clear discussion of relevant limitations of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors study a clearly-defined and very relevant problem, sequential model editing, which extends the setting of applying one or a small number of model edits to hundreds or thousands, which is much more realistic. The method they describe is simple and novel, and does not include the requirements for training data that some existing methods have. GRACE also leverages the internal representations of the pre-trained model itself to (in theory) enable generalization of the edit information to related inputs (however empirical validation of this capability is mixed). The analysis of the method's memorization and generalization in 4.2.2 is welcome.\n\nWhile the final results show that there is still a strong tradeoff between the ability to generalize edits forward (i.e., to future queries that ask about similar information to those queries that were already edited) and the retention of pre-existing information, I think the paper is a useful exploration of a straightforward and intuitive approach to sequential editing.\n\nThe authors also include a clear discussion of the limitations of their work, which is welcome.", "weaknesses": "The main weakness I see in the paper is the fact that GRACE might do little more than memorize the edits it is exposed to. The analysis in 4.2.2 (which is very welcome!) suggests that GRACE essentially has two regimes, one where we essentially memorize past edits and retain all past knowledge, but are not able to generalize edit information to future, related inputs (editing layers 4 or 6 in the $\\epsilon=1.0$ plot) or where we are able to generalize edits to future, related edits, but suffer a (potentially dealbreaking) penalty in terms of the information we retain from the original task. In my view, the tradeoff as shown in the paper is not viable for real-world use cases. However, I still think the paper makes a useful and interesting contribution to model editing/sequential model editing.\n\nOne other point about related work:\n\nI think L79 is wrong or misleading; MEND and ROME are only loosely based on PEFT. MEND only requires a single fine-tuning step, and neither ROME nor MEND is more prone to overfit than regular fine-tuning to my knowledge. Whether or not PEFT overfits more than regular fine-tuning isn’t relevant here, IMO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable and positive feedback, we address your hesitations below:\n\n**Weakness 1: Memorization vs. Generalization**. We are happy to report that the new results included in the Updated Results PDF above address this weakness: GRACE layers can generalize far better than previously reported, leading to better trade-offs. The trade-offs still exist, but we find there is room for better trade-offs between retaining training information while making generalizable edits. At the very least, we do not believe the penalty remains dealbreaking, though we agree this is a crucial point and we look forward to future works improving in this direction.\n\n**Weakness 2: Referencing \"PEFT\"**. Thanks for catching this, we agree and will omit the direct use of “PEFT.” Instead, we will simply call them “parameter efficient” and omit the sentence about overfitting.\n\n**Question 1: Multiple word senses**. The expansion process should not ignore multiple word senses because the model’s representations are contextual. In this example, “the robber of the” and “the house by the river” are likely different representations, so expanding an edit from one should not impact the behavior of another. However, we don’t account for this explicitly, so in the case that the representations are quite close, there may be interference. Settings where the context doesn’t separate representations for sentences like this pose an interesting avenue for future investigation. On the value-learning side, GRACE actually poses a way to avoid multiple word senses when it's unnecessary. For instance, if  “the robber of the” and “the house by the river” should both point to \"bank,\" and there's no further generation required, it may be inefficient to learn different embeddings for the same word.\n\n**Question 2: \"forecasting\" edits**. Here we mean that MEND trains a hypernetwork before making edits, then is used to predict weights for edits that it didn’t see during training. Further, since MEND is trained on previous versions of the model, as it edits the model, its capacity to edit the model also diminishes, because it’s being asked to edit a model it wasn’t trained to edit.\n\n**Question 3: \"privileged information\"**. In both ROME and MEMIT, “covariance statistics are collected using 100,000 samples of Wikitext” [[Meng et al., ICLR’23; Appendix B.4](https://arxiv.org/pdf/2210.07229.pdf)]. While we let ROME use these data in our comparison, this information is unavailable according to our problem definition. If “privileged” is the wrong word, we’d be happy to use “external” or “exogenous” or something else per your recommendation.\n\n**Question 4: Queries inside n-th closest key's deferral radius**. This is an excellent point and indeed considering only the closest key ignores this possibility. Since two nearest keys can't have overlapping deferral radii, this case might be rare given the edit would have to be between the existing keys. Changing this choice in code is trivial, so we will address this comment by investigating how the results change if we check for any overlapping deferral radii. We believe our results still demonstrate the value of our method as proposed and that this alternative solution may in fact improve performance further by making edits more generalizable. Further, the cost of searching across all keys instead of only the closest might eventually outweigh the benefits posed by addressing this possibility. Thanks for pointing this out, we’ll add a sentence noting this is possible.\n\n**Question 5: Remaining challenges**. Our Updated Results PDF demonstrates that GRACE’s trade-off is better than previously shown. We believe this addresses the core of your point. However, moving forwards, achieving a good trade-off is an important area for future work. We believe more work is needed to find high-impact regions to make edits, e.g., identify whether there are more- or less-generalizable regions in models to make edits. Specifically, if models encode semantic similarity for different topics in different places throughout the whole model, targeting these areas could provide the best trade-off for retention and forward generalization."}, {"review_id": "5O6q3QA2bG", "reviewer": "Reviewer_2Vyt", "summary": "The paper suggests a new method for continual model editing. To deploy a parameter-efficient model editing remedy, the authors introduce an adapter for a target layer, keeping the pre-trained model weights intact yet revising the model prediction correctly. The suggested adapter contains a codebook - key-value pairs and corresponding distance coefficient \\epsilon. By training them via a fine-tuning mechanism, they can edit the original prediction accordingly.", "questions": "N/A", "limitations": "Please see the weakness.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed method introduces the interesting idea of continual model editing. The basic idea of codebook updating and introducing new entries is simple yet reasonable. And suggested method outperforms baselines by a significant margin. And the ablation study done in the supplementary file is also good.", "weaknesses": "The most critical limitation is that the choice of editing layer is essential, but it is achieved via heuristic search. Recently released deep neural network architectures provide a depth (layerwise/blockwise) variety according to the scale, and heuristic search to find the best layer to edit is a nontrivial problem. Unfortunately, as shown in the analyses and the limitation section of the main paper, this limitation is the main obstacle to the practical usability of the suggested model. \n\nAlso, most baselines are not targeted to the continual learning setting, so the benefit of the suggested method for continual model editing is a bit difficult to measure fairly. Simple combinations of existing (and recent) continual learning methods with model editing baselines can be better competitors to reveal the merits of GRACE more intuitively. And it will be also great to provide not only model performance like TRR, ERR, or ARR, and the change of additional memory overhead over long model edit sequences. \n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback, we are glad for the opportunity to clarify some points.\n\n**Weakness 1: Sensitivity to layer selection**. As shown in our Updated Results PDF, GRACE is far less sensitive to layer choice than we originally found. This implies picking hyperparameters before deployment may not be dealbreaking, especially in heavily-constrained deployment settings.\n\nTowards practical use, even though GRACE is more robust than we initially reported, hyperparameter selection in any online setting is hard. Our exact problem setting is particularly restrictive and attractive, but indeed will benefit from growing community interest. And while tuning in our precise setting can't be done using conventional methods (e.g., cross validation), there do exist some avenues towards tuning in practice. For instance, in some settings it may be possible to collect a small dataset beforehand and tune our (few) hyperparameters. For instance, there could be exogenous data (like ROME uses Wikitext) or internal validation sets to pick the layer and $\\epsilon_\\text{init}$. And in some cases, there may be ways to reset or update GRACE codebooks periodically. In fact, one strength of GRACE-style edits is that the model weights are frozen, so edits can be reverted or updated without requiring model checkpoints.\n\n**Weakness 2: Continual Learning comparisons**. While extending existing methods into our setting is possible in principle, it is far from trivial. Still, we do adapt each baseline into a continual learning setup. For Finetuning, we use existing continual learning methods (EWC and periodic retraining). For the others, we adopt continual finetuning. While they indeed underperform, we believe that further adaptation strays from their original proposals and too close to sheer model development, given our understudied problem setting. We hope that future works will study this setting further and surpass GRACE, maybe by adapting these existing methods in intelligent ways. Figure 1 in the Updated Results PDF includes one notion of \"memory overhead over long model edit sequences\", as measured by codebook size (Num Keys). After each edit in the sequence, we compute and report the number of keys in the codebook for different choices of $\\epsilon_\\text{init}$ and layer. To further address your comment, we will compute and report the actual amount of memory required to store the codebook in the Appendix. We also believe that making GRACE-style editing faster and more memory-efficient is a worthy research direction that will benefit from community focus.\n"}]

Author rebuttal:
Thank you to all five reviewers for your constructive and thoughtful feedback. We are excited to see positive reception and we summarize our understanding of the paper’s strengths as follows:
* *Problem setting*: reviewers find we have a “**well-motivated introduction and scope that tackles a highly important problem**” (Qd7q) that is “**clearly-defined and very relevant**” (wTwG), while recognizing that finding “**some way to apply targeted patches is essential**” (2Kqs).
* *Method*: reviewers note how GRACE achieves “**simple and realistic model assumptions**” (2Vyt) that is matched by a “**novel approach [that] could actually be deployed in realistic edit scenarios**” (nd9W), importantly “**does not require access to the pre-training data**” (2Kqs), and remains “**simple and computationally efficient**” (wTwG).
* *Experiments*: reviewers find that GRACE “**outperforms a good selection of competing methods on a number of editing and retention metrics, demonstrating strong empirical potential**” (2Kqs), provides “**consistent advantages**” (Qd7q), and “**outperforms baselines by a significant margin**” (2Vyt) on experiments that are “**well-designed**” (nd9W).
* *Contributions*: reviewers find our contributions significant, noting how “**the paper is a useful exploration of a straightforward and intuitive approach to sequential editing**” (wTwG), while studying a “**more realistic and utilitarian setup**” (2Kqs) than prior works.

**Updated Results PDF**. We include updated results in the attached PDF. As shown in updated Figure 1 and Table 1, we find that **GRACE editing is more generalizable and less sensitive to hyperparameters**. These results are from an extended experiment (5k edits of T5 on the QA task) that we conducted after identifying and fixing a minor bug while reproducing all model results post-submission. In these updated results, we reran comparisons on all methods. While we hurried to address your feedback this week, our experiments editing layer 6 have ~2k edits remaining at the time of this submission. We believe the findings so far will extrapolate to the remaining edits, but will update the paper with the completed experiment. **We found that our results had been deflated in our initial submission** (more keys were being created than necessary), and our findings are updated in two ways:
1. **GRACE’s generalization is far better than we initially found**—cache rate is decidedly sublinear for the QA task, which uses a much smaller codebook (137 keys for 1000 edits).
2. **GRACE is not so sensitive to hyperparameter tuning**—interior layers are similar and $\epsilon_\text{init}$ better interpolates the trade-offs. Choosing the editing layer only varies TRR within [0.7, 0.72] and ERR within [0.89, 1.0]. Choosing $\epsilon_\text{init}$ varies TRR within [0.69, 0.72] and ERR within [0.87, 1.0].

These updated findings significantly address the reviewers’ main concerns, replacing Figure 4 in the main paper, though we are excited to see further advances in future work.

**Concerns Shared by Multiple Reviewers**. Two main reservations are shared by reviewers: **Trade-offs between memorization and generalization** and **Hyperparameter tuning**. Fortunately, our updated results (discussed above) address trade-offs substantially, where we find GRACE can succeed with far-smaller codebooks. For hyperparameter tuning, we acknowledge that tuning hyperparameters in our setting is a challenge. While this challenge exists in other lifelong settings, we believe that increased community focus on our setting will spark further advances. As it stands, we see two settings where GRACE can be applied directly:
1. *Our precise problem setting*: Lifelong editing without exogenous data. Here, it’s not possible to follow standard hyperparameter tuning practices (e.g., validation sets). We hope that our updated results showing more robustness to layer and $\epsilon_\text{init}$ inspire confidence that GRACE surpasses existing alternatives, which themselves have even more hyperparameters to tune.
2. *A relaxed, episodic lifelong setting*: In practice, as data stream in and datasets are collected, some models can be retrained or finetuned periodically. But this is expensive and some edits are safety-critical and time-sensitive, so GRACE can edit models between bigger updates. Here, there may also be ways to leverage exogenous or historical data to tune or refresh GRACE codebooks.

In both settings, we believe GRACE poses an exciting, practical solution that surpasses existing works.

**Individual Responses**. We address each reviewer’s comments individually below. We have worked hard to address your concerns and hope you find our responses informative. If you feel our comments have not sufficiently addressed your concerns, we would love to discuss with you further.

Meta-review: This paper has received the scores of 4, 6, 7, 7, 7. Four reviewers voted for accepting the paper, but one reviewer has initially given the score of 4, but the review were rather short, and s/he did not respond to the authors' rebuttal. Given the importance of the problem and the technical contributions, as well as the relatively high review scores from the reviewers, I recommend to accept the paper as a poster.
Decision: Accept (poster)