Aging with GRACE: Lifelong Model Editing with Key-Value Adaptors
Authors: Thomas Hartvigsen
Swami Sankaranarayanan
Hamid Palangi
Yoon Kim
Marzyeh Ghassemi
Venue: Submitted to ICLR 2023 (2023)

Abstract: Large language models often err during deployment, either due to non-representative training data or distribution shift in the test set. Recently, model editors have been proposed to fix errors by adjusting a pre-trained model's weights. So far, however, existing model editors fail when making sequential edits by quickly decaying a model's performance on its upstream data. Further, when editing deployed online models, they quickly forget how to fix previously-seen mistakes. We advance beyond these existing methods by proposing and studying a novel Lifelong Model Editing setting, where errors stream into a deployed model and we update the model to correct its predictions without influencing its predictions for unrelated inputs. Towards effective methods in this challenging setting, we propose with General Retrieval Adaptors for Continual Editing, or GRACE. GRACE is a new Key-Value framework that casts model editing as a codebook update problem. The proposed approach edits selected model layers by caching activations that are queried using embeddings from the previous layer. The cached activations are trained to correct a model's predictions, treating future layers as a decoder. As edits stream in, the keys and values of a GRACE layer are updated while the model weights remain frozen, ensuring similar edits are treated similarly without altering the model's performance on unrelated instances. Experimentally, we show that \method substantially improves over recent model editors.

Under review as a conference paper at ICLR 2023
AGING WITH GRACE: LIFELONG MODEL EDITING WITH DISCRETE KEY- VALUE ADAPTORS
Anonymous authors Paper under double-blind review
ABSTRACT
Large pre- pre-trained models often etr during deployment as input distributions shift, user requirements change, or crucial knowledge gaps are discovered. Recently, model editors have been proposed to modify model's ' behavior by adjusting weights during deployment. However, when editing the same model multiple times, these approaches quickly decay model's performance forget how to fix previous errors. We propose and study novel Lifelong Model Editing setting, where streaming errors are identified for a deployed model and we update the model to correct its predictions without influencing unrelated inputs. We introduce General Retrieval Adaptors for Continual Editing, GRACE, which learns to cache chosen layer's activations in a codebook as edits stream in, while the original model weights remain frozen. GRACE succeeds to edit pre-trained models thousands of times a row using only streaming errors, while minimally influencing unrelated inputs. Experimentally, we show that GRACE substantially improves over recent model editors while generalizing to unseen inputs.
upstream data and
INTRODUCTION
Modern machine learning systems perform extremely well on challenging, real-world tasks. Many successes stem from large models trained on massive amounts of data, achieving state-of-the-ard performance on challenging tasks in natural language processing (Rae et al., 2021; Brown et al., 2020) and computer vision (Ramesh et al., 2022: Dosovitskiy et al., 2020). However, despite high performance, large models still make critical mistakes during deployment (Sinitsin et al., 2019; Balachandran et al., 2022; Wang et al., 2021). Further, when models are deployed over long periods of time without updates, their error rates increase as data distributions shift, as labels shift, or as ground-trun truth information about the world simply changes (Lazaridou et al., 2021). For example, language model trained in 2016 that correctly identifies Barack Obama as president of the United States would be incorrect after 2017, as illustrated in Figure 1. Furthermore, multiple errors often occur sequentially, requiring many fixes the same model over time. To handle this case, introduce Lifelong Model Editing to continuously correct large models' mistakes over long sequences of edits. By editing model, avoid costly retraining while maintaining its performance on unrelated inputs (Mitchell et al., 2022a).
One approach to lifelong editing is directly finetune model on errors as they arrive. However, finetuning on errors is prone to overfitting, even with customized regularization (Lin et al., 2022). Further, finetuned models quickly forget original training data, devaluing upstream pretraining (Lee et al., 2020). And even worse, finetuned models can easily forget previously- errors, counteracting the objective of editing the first place (Sinitsin al., 2019).
Another candidate for lifelong editing is through standard model editing, which outperforms finetuning by updating model behavior with minimal influence on unrelated inputs (Mitchell et al., 2022a). While promising direction, these methods require large amounts of data in order to make edits. For example, some recent works use training sets filled with pre-collected errors to train hypernetworks (von Oswald et al., 2020) that edit model's behavior by predicting new weights (Mitchell et al., 2022a) or offload predictions to a new classifier (Mitchell et al., 2022b). However, large pools of representative training edits rarely available before deployment. Alternatively, regularized finetuning approaches (Meng et al., 2022a; Sinitsin et al., 2019; De Cao et al., 2021) rely on sets of caanttiaall--quivlllee inputs to preserve upstream model performance. Additionally, prior model

---

Under review as a conference paper at ICLR 2023
Continually Editing a Pre-Trained model with GRACE
Pretrained Language Model f
Edit requized
Curassion:: Solen
Crump
Modi
Trump
GRAC leams cacheed sctivations. that mudilty the mudll''s behavvoor
Who is the prosident of the US?
Who is the PM India?
Who the US President?
president United who?
Who India's PM?
Streaming inputs during deployment
2019
Figure I: GRACE edits pretrained models during deployment by maintaining layer-specific codebooks that cache learned activations for selected layers. When inputs similar to previous edits arrive, the corresponding activations are passed to the next layer correct the model's predictions. GRACE continuously edits deployed model using only streaming errors without accessing upstream data.
editors (Mitchell et al., 2022a; Sinitsin et al., 2019; De Cao al., 2021; Mitchell et al., 2022h) fail to make multiple edits sequentially. Consequently, they fail lifelong editors, we demonstrate.
In this paper, we introduce Lifelong Model Editing. Assume we have a model fo that was pretrained on set of upstream instances U. During deployment, observe a stream of errors {(X;, yf ) such that fi=((Xf) * yf V t. Here, fi-1 is the model being used for inference at step t, and y is the correct label for input Xf. At each step t, receive input-label pair for an edit (Xf, yf). and our aim is to produce an edited model fr that meets three criteria:
1. f((Xf) = yf the error at step t is corrected.
2. fx(x5) = of for i S t; the model remembers the corrections for previous errors.
3. fx(X) y A v) E Ld; the model's upstream test performance is maintained.
To address this challenging setting, we propose General Retrieval Adaptors for Continual Editing, or GRACE. GRACE edits individual layers of a frozen, pretrained model fo. treating it as an encoder. GRACE introduces a codebook memory to chosen layer, using encodings from the previous layer as queries to find the nearest key, which is associated with a value. Each value can replace the model's predicted activation, which can be decoded by future layers, ultimately leading to a prediction. Additionally, GRACE learns within any key's e-ball, then the frozen pre- pre-trained model is used directly, avoiding interference with any edits. By adding and updating key-value pairs and their values as edits stream in, GRACE fixes model mistakes without decaying performance on upstream training data. For example, in Figure 1, where GRACE corrects f's predictions about the United States President without impacting knowledge of India's Prime Minister. During training GRACE adapts to changing distributions of edits by shrinking and expanding each e-ball, leading to coarse- or fine-grate fine-grained influence on the edited layer's 's representation space. Further, we can initialize to manage the trade-off between remembering upstream data and successfully fixing long sequences of errors.
e-ball per key. During inference, if encoding does not land
Our contributions in this work are:
1. We cast model editing in new and realistic lifelong streaming setting. To our knowledge, this setting is unstudied, yet is crucial to successfully deploying large language models.
2. We present GRACE, novel key-value model editor which learns cache and retrieve activations for selected layers using only errors observed during deployment. Further, GRACE directly applies any existing transformer-basee model, with straight-forward alterations for other architectures.
3. Our experiments show that GRACE is a state-of-the-an the-art model editor, outperforming al- temattiess like MEND (Mitchell et al., 2022a) on domain-free question answering and classification task with shifting label distributions. We also find that GRACE successfully generalizes edits previously-unsed inputs.
2

---

Under review as a conference paper at ICLR 2023
Edit needed for X (X, # V1 x
GRACE Laver
Retrieve
X,
No
near key
Keys
Values
X2 Layer |
No Edit needed for X (X,) y2
Frozen pretrained layer
Layer / ++]
Figure 2: GRACE learns and maintains codebook for selected layers in a pretrained model. Each codebook contains keys and their corresponding and values. When an input like X1 similar to any previous edits, the GRACE codebook produces an activation to correct the behavior of the pre-trained model. Unrelated instances like X2, land far from existing keys and defer to the pretrained layer.
2
EDITING LARGE MODELS WITH GRACE
2.1 PROBLEM FORMULATION
The Lifelong Model Editing task is to edit the same model hundreds to thousands of times in a row without forgetting upstream performance model that was pre-trained on some upstream dataset V. Let fo denote the frozen pre-trained model at time step = O. We then deploy fo given downstream task and begin monitoring its predictions Dr = f(x,) as inputs X, stream in, one step / at time. Over time, we receive errors X, one at time for which ÿt A yt. where ye is the true label for X,, In order continue safely deploying aim to edit f such that f(),) yt- Let f. denote the edited model at step t. Note that ft will be an updated version of model We desire that fr maintain high performance
fixes for previous edits. Assume we given large
prior edits Xxt and 2) the upstream dataset U. Further, upstream training data are often proprietary or prohibitively large, so we assume no access during editing, in contrast to prior works' stronger assumptions (Mitchell et al., 2022a;b; De Cao et al., 2021; Sinitsin et al., 2019; Meng et al., 2022a).
2.2 GENERAL RETRIEVAL ADAPTORS FOR CONTINUAL EDITING
GRACE presents a novel paradigm for editing models many times in a row: As errors are identified and corrected over time, GRACE modifies a pre-trained model's behavior without altering its weights, as illustrated in Figure 2. A GRACE adaptor at a layer / contains two components: (1) deferral mechanism that decides whether use the GRACE adaptor for any given input and (2) discrete codebook of key-value pairs. The keys are cached activations from layer | - 1, denoted by h!-1 and the values serve to replace h², the frozen model's predictions layer l.
Training and Inference During training, when GRACE adaptor layer performs edit, it either creates a new, randomly initialized value, or updates an existing key-value pair. To ensure that values correct the model's behavior, train the values using backpropagation through the finetuning loss on the model's prediction given the edit. This updated value replaces h for the rest of the forward pass. In our experiments, perform 100 gradient descent steps to train the values.
During inference, GRACE, is activated at layer - as decided by the deferral mechanism - the value corresponding the closest key is returned, Similar the training step, this value replaces h' for the rest of the forward pass.
GRACE codebook A GRACE adaptor at layer maintains discrete codebook, adding and updating elements over time to edit model's predictions. The codebook contains four components:
Keys (K): Set of keys, where each key is cached activation h predicted by layer I - 1.
Values (V): Set of values that are learned while model is deployed and is accumulating errors. Each key maps single value values randomly initialized and are updated
3

---

Under review as a conference paper at ICLR 2023
using finetuning loss on edit examples. By replacing f's activation at layer / with learned value, f then successfully predicts the correct edit label.
Deferral radii (E): Each key has a deferral radius e, which serves threshold for similarity matching, Given a GRACE adaptor for layer and input h' we use a similarity search over existing keys to find the key closest to h!11 via distance function d(-)):
min(d dmin i min((dh'"",
GRACE is activated at layer/ only if dmai S. < where l indexes the most-similar key. The larger the value of e, the more influence the key has, since it covers more of the embedding space. As we discuss below, as GRACE edits a model over time, e values also change, adapting GRACE layers to changing data distributions.
Key labels (f): When a new key is added, its corresponding edit label is also stored in the codebook. By accessing edit labels only while editing, keys and their e values can be adapted generalize to similar instances without influencing too much of the embedding space.
Deferral mechanism Before editing begins, GRACE layers are empty. As editing progresses, GRACE adds and adapts key-value pairs. Conceptually, performing inference with GRACE-edited model entails a deferral decision, computing h' using a discrete key-value search over GRACE's keys:
h CRACE'(N'")) if (d(n''(hh' (6(N-1)
K!) - c) < O,
otherwise,
where 1) denotes the unedited model's activation of the 1-th layer. GRACE (N²-1) retrieves the value associated with the key closest to h ef and K are the influence radius and key i in layer 1, respectively, and d(-) is a distance function (we use Euclidean distance in our experiments).
By using discrete similarity search, if a new input is unlike any cached keys, GRACE simply defers to fo's pretrained weights. This way, GRACE layers can avoid interference with upstream data by leaving the model unaltered, which can be especially good if input distributions are shifting. Further, if labels flip locally in the embedding space, GRACE partitions the embedding space to correct fo's predictions locally, adapting to the distribution of the input edits. In practice, edits are rare compared to streaming inputs, so a GRACE-edited model will often defer to pretrained layer's outputs, successfully leaving unrelated inputs unedited.
Codebook maintenance As illustrated in Figure 2, when an edit is required, fo serves as an encoder, computing embedding for instance layer I. Then, fo serves as query across any existing keys in the GRACE codebook for layer I. A GRACE layer can perform one of the following operations at any given time step:
1. KEY-ADD: If the codebook is empty or the input embedding h/-1 falls outside the c radius of all existing keys according distance function d((), then initialize new key h! along with a corresponding value v, base influence radius e and edit label yf To alter the model's predictions via the value v, finetune the value with respect the model's final predictions until its prediction is accurate.
2. KEY-UPDATE: d(h'-1, kinear) S e; + cinits we must decide whether to expand the influence of the nearest key or to shrink its influence and add a new key, depending on whether or not the keys share edit labels. In practice, we can check if labels are the same by either caching the edit labels associated with each key or simply performing inference. There are two possible casses:
(a) If d(g, kinarr S e, + Ejoit and the nearest key's edit label is the same as the edit label, expand the nearest key's 's e to encompass the query.
[K! [V] + {kk [V', d( K)))
(b) If the nearest key's label is different than the edit label, split the nearest key by first decreasing the influence radius of the nearest key, then creating new key-value pair where the key is the query. The new key is simply the query -!
[K] VL v7.0) - {
K! : [V/ 0.5 x d(f, K(]
fo Vi 0.5 x d(f.. K!]
4

---

Under review as a conference paper at ICLR 2023
As edits stream in, by continuously adding and updating GRACE's keys and values, the embedding space for a selected layer / becomes partitioned according to which instances need modified outputs. When not performing edits, these operations bypassed, and keys entirely frozen, regardless of whether or not the instance lands within key's influence. Overall, GRACE introduces a new model editing paradigm in which edits be made sequentially, similar edits are encouraged be edited similarly, and the ultimate influence of new edits can be controlled and monitored explicitly.
2.3 GRACE LAYERS FOR SEQUENTIAL INPUTS
When edited layers have sequential inputs, like tokenized sentences, each token receives its own activation. In this case, we broadcast a value to each token's ' corresponding activations in layer I. This approach gives the values strong control over the model's behavior and makes them easier to learn than caching values for the first last tokens, which may lead to better composability in future works. To find query's ' nearest key, we find that euclidean distance works well, similar to (Träuble et al., 2022). GRACE naturally applies to all recent transformer models.
3 EXPERIMENTS
To evaluate GRACE, we first show that GRACE outperforms alternative editors by successfully making thousands of edits on real QA and document classification tasks with shifting labels during deployment. Then, we dig deeper into GRACE, evaluating which layers to edits, effects of hyperparameters, and its generalization when editing model up to 5,000 times in a row, based only on streaming edits. Finally, we illustrate how GRACE makes edits using a simple synthetic example.
3.1 EXPERIMENTAL SETUP
We compare GRACE with recent model editors and their capacity to sequentially edit models hundreds to thousands of times in row. This setting is categorically harder than recent works, which make multiple edits simultaneously, updating models once based on large sets of edits, akin to finetuning (Mitchell et al., 2022a;b; Meng al., 2022b). We measure performance for lifelong model editing on multiple axes, including a) performance degradation on upstream data, b) capacity to remember long sequences of previous edits, and e) generalizability to related, but previously-unsee edits.
inputs, or exogenous datasets at any stage of editing. Therefore, we must modify existing editors for fair comparison. First, we Continually Finetune (Lin et al., 2022) on streaming errors. Second, we compare against MEND (Mitchell al., 2022a), which pretrains a hypernetwork on large set of training edits. We adapt MEND to our setting by continually finetuning MEND's hypernetwork on streaming errors. Third, we implement a Defer Adaptor, inspired by SERAC (Mitchell al., 2022b), which pretrains deferral model and prompt-adjusting hypernetwork on training edits, then the deferral model chooses when to trust the pretrained model vs. the editing hypemnetorr. We lack training edits, so implement conceptually-sime similar adaptor, using deferral model to predict when to trust the frozen layer vs. predict the next activation with another hypemetwork. Both are continually trained streaming errors. Fourth, Network Adaptor, which includes a memory module and an attention mechanism. The memory module contains learnable values that serve as cached activations, and the attention mechanism takes in the activation from the previous layer (the same as GRACE), then predicts attention weights for each value. The weighted sum serves as the input the next layer. For all compared editors, we edit only a selected layer. We tune the learning rate for each method, reporting only the best persfoerfnnnng editor for each case. Further implementation details and hyperparameter tuning experiments are available in the Appendix.
Compared methods We assume no access to training edits, remantically-equiveee
also consider soft version of GRACE via Memory
3.2 GRACE OUTPERFORMS ALTERNATIVE EDITORS ON REAL DEPLOYMENT TASKS
We first compare GRACE to existing model editors on realistic deployment settings by correcting pre- pre-trained models long sequences of real mistakes real datasets. To evaluate each model editor, we take a pre-trained model, simulate deployment on a streaming task, and edit the model when mistakes made. In these experiments, pass 1,000 inputs into each model and edit only when
5

---

Under review as a conference paper at ICLR 2023
Edit Score
780
3000
1003
Monoey
MEND
Meroory
(a) Results from editing TS model for open-domain (b) Results from editing BERT for multi-class classifi- cation on SCOTUS with shifted labels.
QA with shifting answers from 2sRE.
Figure 3: Main results comparing GRACE to alternative editors context-free QA and SCOTUS document classification. Edit Score is the average of Upstream and Online performance. Each editor is applied to model deployed the same sequence of inputs and ends up making roughly 500 edits out of 1000 inputs during deployment. GRACE achieves strong performance both upstream data and previous edits, and exerts direct control over this trade-off.
the prediction is wrong. Each editor ends up making roughly sequential 500 edits. Whenever the model is edited, we then measure both Upstream and Online Performance, then compute an Edit Score, which is the average of the two metrics.. In each case, report results only for methods that succeed to edit the model over 75% of the time, and report once 10 edits have been made; previous editors and finetuning methods are expected to decay upstream performance quickly, since they rely on continually-traine hypernetworks, which should underperform early on without training data. We perform this experiment on two distinct settings using different pre-trained models.
First, we follow Mitchell et al. (2022a) and edit a 60-million parameter T5 model (Roberts et al., 2020) trained for open-done open-domain Question Answering. The T5 model was pretrained on the Natural Questions dataset (NQ) (Kwiatkowski et al., 2019), and we simulate deployment on the zsRE dataset (Levy et al., 2017) extracting potential edits from De Cao al. (2021)'s validation split. Following Mitchell et al. (2022a), we edit the dense-relu-dens parameter model, though we ablate this choice in Section 3.3.
layer of the last encoder block of 60 million
Second, we experiment with a BERT model on the SCOTUS dataset, which features court documents from across multiple decades. Over the decades, label distributions shift and models trained on earlier data become outdated. Each document is labeled with one of 11 issue areas and SCOTUS is divided into a training split with 7.4k cases from 1946-1983 and test split with 931 cases from 1991-2016. To labels
illustrate lifelong model editing, exacerbate the label shifts by merging semantically-simne in the training split. For example, we map categories (Civil Rights, First Amendment) + (Civil Rights]. In the test split, we then separate the labels. Further details are in the Appendix.
Results: Overall, GRACE dramatically outperforms all four alternative editors, as shown in Figure 3. Over streaming edits, GRACE succeeds to maintain high upstream performance while also remembering how to fix previous edits. We further compare all methods in the Appendix, which includes a hyperparameter study for each method. As expected, continual finetuning is the most competitive with GRACE on both tasks. Also as expected, without access to privileged information, the compared methods decay performance mnickly-similee to Hase al. (2021), we find that after only 10 inputs, the alternatives have already decayed the pretrained model's performance.
3.3 MEMORIZATION VS. GENERALIZATION IN GRACE LAYERS FOR THOUSANDS OF EDITS
Next, we evaluate GRACE's memorization vs. generalization performance using extremely long sequences of edits on the QA task. We take the first 2,000 questions from the zsRE dataset, and augment them using 5 rephrasings of each question, resulting in 10,000 questions with which we edit the T5 QA model. Given this controlled setting, we edit each encoder layer's final dense relu module, ranging from 0.1 to 10.0, above which performance stabilizes. Each parameter setting leads to the model being edited over 2,500 times, with some being edited nearly 5,000 times!.
We seek to answer the following questions simultaneously in order to explore the tradeoff between memorization and generalization for GRACE by tracking relevant metries:
55000 sequential edits is extremely large. Hase et al. (2021), for example, perform sequential editing experiment where they show that even 10 sequential edits catastrophically decays their performance.

---

Under review as a conference paper at ICLR 2023
Can Eeiiediee
0.25
2000
Stopa
Skira
Slaga
Liges
5
Laperd
Layer
(a) = 0.1
V
4000
0.5
0.25
2000
4000
4000
3000
4000
2000
4000
2000
4000
laget
Layre
Tager 5
1ayer4
Layer
Lapera
1ayert
Tayer
(b) 2.0
1.0
1.0
0.50
2000
0.5
0.5
0.26
0.00
0
2000
1000
0
2000
4000
2000
2000
2000
4000
0
2000
5000
Sage
Saga
Stp
2apa
Layet
Laget
Tayer
Layer
Tager
Layer
Laye
6.0
Figure 4: Evaluating GRACE's memorization vs. generalization when choosing values and editing different layers. Some layers are better to edit than others, GRACE layers generalizes to unseen edits, c-selection trades off memorization and generalization, and GRACE codebooks stabilize over time.
How accurate is GRACE for long edit sequences? Metrics: Upstream and Online F1.
How well do GRACE's keys generalize? Metric: Fl on set of holdout edits.
Does GRACE simply memorize new inputs? Metric: Fraction of input errors that lead to new keys being created.
Does GRACE edit every potential edit sample? Tracked metrie: Proportion of the 10,000 potential edits that actually require editing
Figure shows our main results, with remaining plots in the Appendix. We derive four key findings:
1. Not all layers are equally-good to edit. We first find that GRACE achieves substantially different performance when editing different layers in the T5 model. In Figure 4, we compare the effects of editing the outputs from the final dense ReLU layer of each encoder block. As seen in different colors, the results are strikingly different. First, editing later layers improves upstream performance, but forgets previous errors quickly. On the flip side, editing early layers seems to remember previous edits quite well, but at the expense of upstream performance. For some combinations of layer- and e-choice, there appears be good balance. Finally, as shown in the 5th column, editing later layers leads to models that need to be edited more often than others, as the number of edits is nearly linear for all three values of
2. GRACE-edited layers indeed generalize to unseen inputs. We also evaluate generalizability of GRACE-edited models previously-unsee edits. To achieve this, when model is edited, also record its performance on holdout set of edits containing rephrasings of each training edit. We keep this set of edits static over time, as edits arrive and are corrected, performance increases on the whole set, as expected. As shown in the 3rd column of Figure 4, editing earlier layers leads to better generalization. Excitingly, larger values also lead to better generalization, implying that the semantically-siine simila inputs indeed land in the same deferral radii
Choosing e balances memory between upstream data and previous edits. In Figure 4, we report each metric for different choices of the learned deferral radius for GRACE-edited layer. As expected, we find that for tiny choices of e (Figure 4a), after correctly choosing the layer to edit, achieve wearly-perfect F1 on both Upstream data and Previous Edits. However, this is at the expense of the codebook size: Nearly 100% edits are cached as new keys (column 4). While such memorization may be sufficient during deployment sometimes, it is akin to sheer lookup table.
7

---

Under review as a conference paper at ICLR 2023
(a) Upstream Training Data (b) Upstream Training Data Edits (c) Predictions before editing
(d) Predictions after editing
Figure 5: Results for GRACE on synthetic data. A model is pretrained on the data in (a), then label shift is introduced in (b) via a local set of label-swapped edit instances. Then, the pretrained model predicts the wrong class for these edit instances, as shown in (c). Finally, as shown in (d), GRACE successfully edits the pretrained model, swapping its labels without impacting distant data.
Excitingly, find that choosing bigger 6 successfully avoids memorization, while maintaining strong performance.
GRACE codebooks stabilize over time for well-selected parameters. As shown in the 4th column of Figure 4, we report the % of edits that end up being cached as new keys over time, We find that this flattens over time, indicating that the rate of key caching decreases as the model trains. As expected, for tiny e values (Figure 4a), nearly 100% of the edits are cached as keys. However, for larger values, the of cached edits goes down to around 50%, indicating that GRACE adaptors effectively manages codebook size. This finding is true across all values and edited layers.
3.4 ILLUSTRATING GRACE USING SYNTHETIC DATA
We finally illustrate GRACE using simple synthetic dataset. As shown in Figure 5()), sample two-dimensional instances from two clusters, each corresponding to a class, shown in red and blue. On these data, we pre-train a three-layer fully connected classification network. The network projects the instances into 100 dimensions, 100 dimensions, and back to I dimension for binary classification with ReLU activations (Nair & Hinton, 2010). Since these data are linearly separable, the model learns to split the feature space vertically. We then introduce a set of likely edits by introducing tight circle of instances inside the distribution on the left, simulating local label shift, as shown in Figure 5(b). As shown in Figure 5(c). the upstream model then classifies the flipped labels incorrectly. We then use a GRACE layer to edit the pre-trained model, sampling edits from the edit set one by one and initializing to be 0.45 to roughly align with the scale of the edit set.
Result: In Figure 5(d), we see that GRACE succeeds to locally edit the model's predictions for the edits without influencing the model's predictions unrelated training data. While this case is relatively simple, Finetuning a model only on errors especially those from a single class leads to catastrophic forgetting on upstream data. We show in the Appendix that Finetuning indeed flips all predictions to the blue edit class. Additionally, GRACE edits this model using only one key. Using one key is ideal in this simple setting, since the edits are clustered. As we show next, tasks become more complicated, edits spread out and require more keys to cover the model's ''s representation space.
4 RELATED WORK
Model editing. Model editing a recent and active research area. Classic approaches focus on focus on gggularzzedfineetuiing altering a model's weights by incorporating auxiliary information, like upstream training data and caanticalll--qqivleeee instances (Sinitsin et al., 2019). Recent works have extended this paradigm to pretrain hypernetworks that predict edits (De Cao et al., 2021; Mitchell et al., 2022b;a), often decomposing weight updates into low-rank components (Meng et al., 2022a; Hu et al., 2022). Due to notorious training costs, many works focus on editing transformer models (Zhu et al., 2020), leading to natural extensions to multi-lingual editing (Xu et al., 2022). Many recent works have considered parameter-efffiient finetuning, often by using low-rank updates Hu et al. approaches like prompt tuning train an extension of pretrained training steps and are more prone to overfit than regular finetuning (Zhong et al., 2022; Su et al., 2022). making them inappropriate from a model editing standpoint compared approaches such Mitchell et al. (2022a) and Mitchell al. (2022b). Some recent
(2022). While models, they often require
8

---

Under review as a conference paper at ICLR 2023
works have begun considering editing models multiple times. Batch editing, for instance, attempts to fix models on batches of edits simultaneously. While MEND (Mitchell et al., 2022a) demonstrates their performance decays quickly, more-recent works like SERAC (Mitchell et al., 2022b) and MEMIT (Meng et al., 2022b) are showing more promising results in this direction. However, these exciting works use large amounts of privileged information compared to problem setting, like computing layer statistics given exogeneous datasets or requiring training sets of representative edits. We assume no access to these extra datasets, and pose that selecting these datasets probably has massive impact on an editor's s success. One recent work does discuss sequential editing: Hase et al. (2021) shows that after editing the same model 10 times, editing performance drops dramatically. In our paper, we focus exclusively on this harder sequential editing problem, but edit models thousands of times in a row.
Recent works have also begun considering continual finetuning, where large language models are refined over time as new instances arrive. For example, Lin al. (2022) show that regularizing finetuning with continual learning methods like Elastic Weight Consolidation (Kirkpatrick et al., 2017), Experience Replay (Rolnick et al., 2019), and Maximally Interfered Replay (Aljundi et al., 2019) decay upstream performance rapidly when shifting between multiple tasks.
Key- Key-Value Methods for Continual Learning. Key-value methods are now a powerful paradigm for variety of machine learning problems. These approaches have deep roots in computer vision (Van Den Oord et al., 2017: Liu et al., 2021), driving recent high-profile results like DALLE-2 (Ramesh al., 2022). Further, key-value methods are particularly strong in continual learning settings, with recent works demonstrating prompt-lcarning for NLP (Wang et al., 2022b;a) for applications like text retrieval (Xiong al., 2021). Recent works have shown that discrete key-value methods in particular performs well on distribution shifts (Träuble et al., 2022), with recent works extending to question answering (Dai al.. 2022). This performance stems from the stored values, which can remain within the expected distribution of a downstream encoder, regardless of shifting inputs. Further, storing values allows for unlimited long-term memory, resources permitting. We corroborate these advantages in our experiments, where we demonstrate GRACE's robustness to shifting inputs and labels over long sequences of edits.
CONCLUSIONS
Language models are quickly becoming larger and being applied diverse set of downstream tasks. However, they are often computationally prohibitive to finetune and easily forget past knowl- edge. In this work, we propose realistic problem setting where we edit such large models, Lifelong Model Editing. In this setting, we edit large models many times in a row given only errors that arrive during deployment, without access to any upstream data or dest-distribution examples. We then present GRACE, a plug-in module that can wrap around any given layer in any large pretrained NL.P model. GRACE layers (1) retain the functionality of the original model, thereby minimizing catastrophic forgetting and (2) adapt to changing data distributions by storing codebook of cached activations that can grow or shrink over time. We demonstrate GRACE's efficacy by showing that GRACE provides the best trade-off between upstream performance and accuracy streaming edits among competing model editing baselines and finetuning methods. We further investigate GRACE's capacity to make extremely long sequences of edits and show that choosing the wrong layer to edit can decay performance substantially and that GRACE succeeds to generalize its edits to previously unneenn inputs and avoids simply memorizing edits.
6 REPRODUCIBILIIY
To encourage reproducibility of both CRACE and our experiments, we have provided thorough descriptions of our experimental setup in Section 3. Our descriptions contain specific layers we choose to edit. In Section 2, we include clear steps that detail GRACE's behavior.
7 ETHICAL CONCERNS
Model editing may alleviate some ethical concerns, for example correcting previously-inoon incorrect information. However, editing models may also introduce new errors if the proposed fixes
9

---

Under review as a conference paper at ICLR 2023
themselves incorrect biased against a given cub-population. Researchers who design, implement, and deploy such techniques should consider how best to evaluate sources of edits and verify they are not harmful.
REFERENCES
Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. ArXiv preprint, abs/1908, 04742, 2019.
Vidhisha Balachandran, Hannaneh Hajishirzi, William Cohen, and Yulia Tsvetkov. Correcting diverse factual errors in abstractive summarization via post-editing and language model infilling. In Empirical Methods in Natural Language Processing, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yi Dai, Hao Lang, Yinhe Zheng, Fei Huang, Luo Si, and Yongbin Li. Lifelong learning for question answering with hierarchical prompts. arXiv preprint arXiv:2208. /4602, 2022.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 6491-6506, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-tinining of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL
http::/arxiin www./oggg/bbbbbo.o
Jesse Dodge, Gabriel Ilharco. Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprini (xXXx:2002.063005 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.119299 2020.
Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. arXiv preprint arXiv:21 1222/11.13663 13654, 2021.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Baankka--awiiiaaa..aa et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' Autume, Tomas Kocisky, Sebastian Ruder, al. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Systems, 34:29348-299 293663, 2021.
10

---

Under review as a conference paper at ICLR 2023
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. In International Conference on Learning Representations, 2020.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017). pp. 333 333-342,277 2017.
Bill Yuchen Lin, Sida Wang, Xi Lin, Robin Jia, Lin Xiao, Xiang Ren, and Scott Yih. On continual model refinement in out-of-distribution data streams. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3128-3139, 2022.
Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Advances Neural Information Processing Systems, 34:28092-28100
2021.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. arXiv preprint urXiv:2202.05222, 2022a.
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022b.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D) Manning. Fast model editing scale. In International Conference on Learning Representations. 2022a.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory- based model editing at scale. In International Conference on Machine Learning, pp. 15817-15831. PMLR, 2022b.
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. In International Conference on Learning Representations, 2021.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Ieml, 2010.
Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, ((140):1-67, 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25 2022.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? In Empirical Methods in Natural Language Processing (EMNLP). 2020.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Anton
Sinitsin, Vsevolod Plokhotnyuk, Dmitry Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. In International Conference on Learning Representations, 2019.
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, al. On transferability of prompt tuning for natural language pro- cessing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3949-39699 2022.
Frederik Träuble, Anirudh Goyal, Nasim Rahaman, Michael Mozer, Kenji Kawaguchi, Yoshua Bengio, and Bernhard Schölkopf. Discrete key-value bottleneck. arXiv preprint arXiv:2207.11240. 2022.
11

---

Under review as a conference paper at ICLR 2023
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Johannes von Oswald, Christian Henning, Benjamin F Grewe, and João Sacramento. Continual learning with hypernetworks. In International Conference on Learning Representations, 2020.
Dequan Wang. Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test- time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. arXiv preprint arXiv:2204.04799 2022a.
Zifeng Wang, Zizhao Zhang, Chen-Ye Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Plister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 139-149, 2022b.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference Learning Representations, 2021.
Yang Xu, Yutai Hou, and Wanxiang Che. Language anisotropic cross-lingual model editing, arXiv preprint arXiv:2205. 12677, 2022.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. Panda: Prompt transfer meets knowledge distillation for efficient model adaptation. arXiv preprint (20i::2208.11660 2022.
Chen Zhu Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifyyng memories in transformer models. arXiv preprint arXiv:2012.00363 2020.
12

---

Under review as a conference paper at ICLR 2023
A LIMITATIONS
Lifelong model editing is new and challenging, so our method GRACE has limitations. Understandably. introducing codebooks between layers of pretrained model slows down inference. This slow-down largely due to repeated similarity searching between embeddings and keys. Even though GRACE's current design does not emphasize inference time, making GRACE faster is promising avenue for future work, especially given recent accelerations in adaptor methods like LoRA (Hu et al., 2022) and language models using similarity like Gopher (Rae al., 2021).
While GRACE can scale to use in multiple layers simultaneously, we do not perform these experiments. In the most-general case, GRACE layers may be stacked between subsequent layers. Then, the output of one codebook may be passed to another codebook. Stacking codebooks will lead to highly-flexbllc models that provide friendly interfaces with which to control their outputs. Our work takes the first step towards this new model primitive, which has clear implications for responsible machine learning.
Finally, autoregressively decoding long sequences with GRACE particularly challenging. In the long term, language model editors will control long sequence generation. While this is current challenge for all current editors, GRACE will require modification to generate many tokens in a row. This limitation is due to GRACE's similarity-search between input queries, which are embeddings, and existing keys: During autoregressive generation, the embedding for prior tokens is updated. Then, the resultant embedding may point to new GRACE keys, leading to continuous distribution shift. However, most NLP tasks are not autoregressive, despite autoregressive models currently being popular. We look forward to future work that 1) extends codebook- based edits to autoregressive models, and 2) more- more-deeply evaluates prior model editors' capacity to alter autoregressive text generation for sentences longer than few words. In this short text generation paradigm, GRACE can surely be extended.
Implementation Details We experiment with T5 (Raffel et al., 2020) and BERT (Devlin et al., 2018), though GRACE is general and applicable to other transformer- NLP models as well. To ensure fair comparisons, we edit the same layers across all methods for each experiment and only consider methods that successfully edit each input instance.
For T5 experiments, we edit the dense-relu-dense layer of the last encoder block of a 60 million parameter model, following (Mitchell et al., 2022a). For BERT, we edit I 110 million parameter model's s last encoder layer's 's dense output that was pretrained to classify sentiment. We then finetune BERT ourselves on the SCOTUS training set.
During evaluation, we hyperparameter tune each method, only considering methods that successfully edit each input instance. For Finetuning, Memory Networks, Deferral, and MEND, we edit with learning rates of 1e-1, 1e-2, 1e 1e and - Intuitively, small learning rates retain upstream performance while large learning rates learn to apply new edits. Therefore, for some learning rates, finetuning methods may achieve higher individual upstream and online scores than we report. However, akin to constrained optimization, the minimal editing success is to actually modify the model's performance on one edit. Here, finetuning is particularly prone to overfitting.
B ADDITIONAL DATASET DESCRIPTIONS
Question Answering with Shifting Answers We perform our QA experiments using T5 model that was pretrained on Natural Questions (NQ). For evaluation, we sample 1000 random question- answer pairs from NQ to serve as upstream data. During editing, for the main results comparing all editors (Figure 3) we extract the first 200 questions that have at least 5 rephrasings in the zsRE dataset. We then use each rephrasing as a separate edit, This creates 1000 potential edits to pass into the pretrained model during deployment. Expanding beyond this setting to evaluate edit generalization presented in Figure 4, we extract the first 1000 instances from zsRE that have at least 10 rephrasings, but split the rephrasings into an edit set and a holdout set. This creates 10,000 potential edits, and 10,000 holdout edits.
SCOTUS Document Classification The US Supreme Court (SCOTUS) is the highest federal court in the United States of America and hears only the most controversial or otherwise complex cases. We consider a single-label multi-class classification task, where given a document (court opinion),
13

---

Under review as a conference paper at ICLR 2023
Num edits
Num keys
key error ratio
Upstream Accuracy
Accuracy on Prev. Edits
1.0
403
403
1.0
79.8
99.7%
2.5
387
256
0.66
73.9%
74.2%
5.0
375
245
0.65
76.5%
64.6%
Table 1: Effect of e on the number of keys created by GRACE for Shifting Annotation Guidelines.
the task is to predict the relevant issue areas. The 14 issue areas cluster 278 issues whose focus on the subject matter of the controversy (dispute). The list of issue areas include: (Criminal Procedure, Civil Rights, First Amendment, Due Process, Privacy, Attorneys, Unions, Economic Activity, Judicial Power, Federalism, Interstate Relations, Federal Taxation, Miscellaneous, Private Action). To simulate scenario for shifting annotation guidelines, modify the training set as follows:
Relabel samples in classes [First Amendment, Due Process) as Civil Rights.
Relabel samples in Unions Economic Activity.
In the edit set and the test
the original labels i.e. no relabeling done. Thus, upstream dataset contains 11 labels and our edit/test sets contain 14 labels, where some classes are expanded in scope explained above.
C ADDITIONAL SCOTUS EVALUATION
We further evaluate the relationship between and performance on SCOTUS in Table Here, see that Upstream Accuracy decays slightly as grows, while accuracy on previous edits drops quickly. The key error rat io shows the number of errors that lead creating new key. As expected, for = 1.0, which is relatively small, all edits result in keys. Then, as grows, this ratio goes down because inputs land within the deferral radius of existing keys.
D MULTI-OBJECTIVE COMPARISONS RESULTS ON QA AND SCOTUS TASKS
We include alternative view of the results shown in Figure here in Figure 6. In this Figure, plot each editor's performance on growing history of previous edits and its performance on the pretrained model's upstream data, For both Figures, optimal performance is in the upper right-hand corner. As expected, each model decays the pretrained model's 's upstream performance over time, though this decay scales down with learning rate. Also as expected, MEND underperforms case as it lacks an edit training set, which is privileged information with respect to our task. We also observe that editing BERT on SCOTUS is far noisier than T5 on QA for all editors. This finding corroborates recent works that show BERT training tends to be highly unstable (Mosbach et al., 2021; Dodge et al., 2020).
each
We also include the single-metre metric version of Figure 3 in the main paper. Here in Figure 7, that the trends from the main paper remain the same.
E ADDITIONAL RESULTS FROM QA HYPERPARAMETER STUDY
To extend the results shown in Figure 4, we add results from more choices of e in Figure The trends identified in the main paper remain true: some layers are better edit than others, GRACE succeeds to generalize to previously-unsee and generalization, and GRACE codebooks stabilize in size over time. Performance remains remain the same above 6.0. Interestingly, for 2.0, Layer 6 appears to be highly unstable over time.
inputs, c controls the trade-off between memorization
14

---

Under review as a conference paper at ICLR 2023
Roeeune
Streaming MEND
Defer
Merrryy Atapiee Stmweing
Adapte
Accuracy on Upstream Data
0.8
0.6
on Upstream Data
0.6
0.4
0.5
E-000
0.3
0.4
0.2
0.3
0.1
0.1
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
F1 on Previous Edits
Accuracy on Previous Edits
(a) Results from editing a model for open- domain QA with shifting answers from zsRE.
(b) Results editing BERT for multi-class clas- sification on SCOTUS with shifted labels.
Figure 6: Main results comparing GRACE alternative editors context-free QA and SCOTUS document classification. Each editor is applied to model deployed on the same sequence of inputs and ends up making roughly 500 edits. Ideal performance each model's performance throughout streaming. GRACE achieves strong performance both upstream data and previous edits, and exerts direct control over this trade-off.
the upper right corner. Arrows denote
Reoored Tepeli
Iepel
Online F1
200
100
200
300
Mensy
Detical
MB00
CAADE
Roxaase
Mouscy
Defond
MSN
GRACS
(a) on Previous Edits when editing T5 on zsRE,
(b) Upstream when editing T5 on zsRE.
(50
540
2000
MibGa
GRACTA
2002
(c) Accuracy Previous Edits when editing BERT on (d) Upstream Accuracy when editing BERT on SCO- SCOTUS.
TUS.
Figure 7: Comparing Upstream and Online performance for GRACE to alternative editors on context- free QA and SCOTUS document classification. Each editor applied to a model deployed on the same sequence of inputs and ends up making roughly 500 edits out of 1000 inputs during deployment. GRACE achieves strong performance both upstream data and previous edits, and exerts direct control over this trade-off.
15

---

Under review as a conference paper at ICLR 2023
10
1.0
1.0
2000
0.5
0.5*
05-
R000
0.00
0.0
0.0
2000
4000
2009
4000
2000
4000
2000
4000
2000
4000
Mays
Slggs
Meeg
(a)
21.0 wtatttreen Dut
Due Fredonn take
1.0
2000
0.50
of
83
0.5
2000
0.00
2000
4000
2000
0.0
3000
4000
0.0
3000
4000
2000
Styg
Stope
Sepp
Stope
(b) 1.0
W
bibs
ww Owbbl
8900
0.25
4000
Siag
Layes
5
Laper
Layer
Layer
Lapert
(c) = 2.0
Predben Eates
Kil
4000
2000
0.25
2000
4000
2000
4000
2000
4000
2000
4000
lape?
Layre
Tager 5
1ayer4
Tayer
Lxprr
1ayert
Tayer
(d) 4.0
1.0
0.50
0.5"
0.5
0.25
2000
4000
2000
4000
2000
4000
2000
1000
0
2000
4000
2apo
Layer
Layer
Tayer
Lajer
Tayer
Tayer
Lype
6.0
Dhts
10
1.0
3.0
0.50
2000
0.5
0.5
0.5
0.00
0
2000
4000
0.0
0
2000
4000
0,0
2000
4000
0.0
2000
4000
5000 Arpe
4000
10yx
Skyp
Lagee
Lapre
Ligeo
8.0
Figure 8: Evaluating GRACE's memorization vs. generalization when choosing values and editing different layers. Some layers are better to edit than others, GRACE layers generalizes to unseen edits, e-selection trades off memorization and generalization, and GRACE codebooks stabilize over time.
16


Reviews:
[{"review_id": "mblAGyNz4O", "reviewer": "Reviewer_ngYT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work addresses the lifelong model editing setting, where errors stream into a deployed model, and the model is updated to correct wrong predictions. The author uses a key-value strategy to look up the codebook of edits to change the behavior of a model. The input feature of a module is used as the key, while the desired output of the module will be the value stored in the codebook. The edits are stored as key-value pairs and extracted based on the similarity between an input feature and the stored keys. The experiment uses QA datasets (zsRE) and document classification data (SCOTUS) to create the stream editing scenario and compares it with MEND, fine-tuning, and a memory-based method. ", "summary_of_the_review": "The problem setting is interesting, and the method is marginally novel. The results look good but need more analysis to provide in-depth insight into the problem. Overall the weakness outweighs its strength.\n\n[Post-rebuttal]\n\nThe evaluation has considered all revisions and responses. This form was up-to-date.\n\n", "strength_and_weaknesses": "- Strength: The lifelong model editing setting is an interesting problem setting not explored by previous works. This setting modifies the previous model editing problem to make it closer to a real-world scenario. The proposed approach is simple but effective. This method looks like a reasonable baseline for future works to compare and improve.\n- Weaknesses: Readers familiar with model editing may find no surprise in the paper's method and experiments. The setting and method are different from previous works, but readers may have difficulty finding an interesting novelty or new insight that is intriguing or inspiring. I see the contributions made in the paper, but they need more depth, like the previously accepted papers have. One way to improve is to beef up the experiment section. Ex: create easy/hard streaming editing scenario, ablation study, level of distribution shift versus editing success rate, etc. Lastly, the SOTA method [1] should be included and compared. [1] looks to have very good potential in the lifelong model editing setting.\n- Other concerns/questions:\n1. In Figure 4(a), why does stream MEND start from such a low upstream F1? Should every method start from a point close to the unedited model f0?\n2. Although MEND was not designed for streaming model editing, its success rate is expected to be reasonably good for the first few editing in the stream. However, Figures 4(a) and 5(a) show a very poor result of MEND/fine-tuning from the beginning. Why?\n3. In Figure 4(a), the colors for different hyperparameters are not shown in the figure. Only the darkest color is visible (ex: light purple and light red do not appear in the figure). This could need to be clarified for readers. \n4. limitation in applying autoregressive decoding. I see the author mentioned this in its limitation section, and I appreciate it. However, this is a huge limitation that does not occur with other model editing strategies. \n5. About the Memory Network baseline, is there a reference paper? Please provide more detailed descriptions if it is created for this paper.\n\n[1] https://arxiv.org/abs/2206.06520", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ftE-S8Uygs", "reviewer": "Reviewer_PaQG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new method for *model editing* in pre-trained language models, which has received growing attention from the deep learning community in the past few years. The authors point out that the setting of sequential model editing (in which the model must be edited with a stream of edits, rather than a small/static set of edits) is relatively understudied, and propose GRACE, a method that is better-suited to applying many model edits without destroying model performance on the pre-training data or forgetting past edits. Under a their particular evaluation conditions, the authors find their method is better able than some existing methods to handle many sequential edits.", "summary_of_the_review": "Overall, I think this work is getting at an important question for folks interested in model editors: how do we apply many edits to our model without destroying it. However, I feel the work as-is is lacking in experimental rigor (in terms of both baselines considered and the specific quantities measured in the experiments; i.e., generalization) as well as technical insight and novelty. Therefore, I would recommend rejecting the paper in its current form.\n\n**Update** in light of author response:\n\nI'm grateful for the authors' thorough response to my concerns; I think the paper is significantly improved. The additional generalization ablation is helpful in understanding how well GRACE generalizes to difference expressions of an edit. My reading of the result is that GRACE does some generalization to paraphrases of the edits, but still fails to generalize to a significant number of simple paraphrases of the edit content (as shown by the F1 score in column 3 of Figure 4 being significantly below 1). Increasing the deferral radius helps with this issue, but comes at the cost of significantly reduced upstream F1.\n\nMy remaining concerns about GRACE as a method are related to this generalization issue. GRACE seems to generalize only mostly to paraphrases of edits, and I would guess it is totally incapable of generalizing to entailed edits, such as:\n\nEdit:\n\nWhere is the Eiffel Tower located? Rome\n\nTest input:\n\nWhat famous tower is located in Rome? <should answer Eiffel Tower>\n\nIt seems like editors like SERAC (Mitchell 2022) would be able to perform this type of generalization, while GRACE's explicit key-value construction makes it extremely unlikely that such generalization would occur, given that it already struggles somewhat with paraphrases.\n\nAs a related generalization issue, GRACE's reliance on exact equality of the edit label to determine if edits contain equivalent information is problematic. Consider the edits:\n\ne1: What is the tallest mountain on earth? Everest\ne2: What is the earth's tallest mountain? Mount Everest\n\nReducing the radius of e1 to make room for a separate edit e2 feels wrong to me. How do the authors propose handling the problem of determining the semantic equivalents of edit labels (or edits, in general)?\n\nIn conclusion, I'm now actually okay with accepting the paper in that it is essentially a novel combination of memory-based (e.g. SERAC, Mitchell 2022) and parametric (e.g. ROME, Meng 2022 or MEND, Mitchell 2022) editing ideas that shows some promise. However, I am also okay with rejecting the paper this time around, on the grounds that GRACE may not be fully leveraging the capabilities of existing methods, producing a sub-optimal generalization-upstream performance frontier, and the positive result of applying many edits is largely offset by the weaker ability of GRACE to generalize the content of the edit to related inputs (paraphrases or entailed information).", "strength_and_weaknesses": "Strengths:\n- Focuses on an interesting extension of existing model editing work, which has mostly focused on single/batched model edits, rather than streams of edits\n- The proposed method does not need a training set of edits, unlike several past model editors. This requirement is a non-trivial limitation of some past works.\n- The proposed method is able to apply many edits without interference or forgetting of the pre-training data\n\nWeaknesses:\n- Baselines are limited and not explained in detail (particularly the \"memory network\" baseline). The setup for MEND is a bit strange, and it’s not too surprising that it fails. Why not compare with ROME (Meng et al., 2022), which I believe doesn’t require an editing training set?\n- The criteria for successful edits are, in my opinion, insufficient, in that they do not consider the generalization of the edit. Previous work such as de Cao et al., 2021, Meng et al., 2022 and Mitchell et al., 2022a,b evaluate the generalization of an edit to related inputs in some manner, which is crucial in my opinion.\n- On a related note, it’s not clear to me the extent to which the evaluations test the generalization of an edit; Section 4.2 says “For editing, we sample 1000 edits from zsRE, including random samples of 5 rephrasings for each question.” Does this mean the the F1 on previous edits is computed using the rephrasings of the original edit X? If we’re not evaluating generalization in some way, it’s not clear that we need to cache representations at all, vs just caching the raw inputs and labels? My understanding is that the advantage of existing learnable model editors like e.g. Mitchell 2022b is that the model can learn to reason over edits to produce significantly different outputs.\n- The experiments are lacking in qualitative examples; it would be helpful to analyze some success and failure cases to see where the proposed method begins to fail (e.g., with respect to generalization).\n- The method is not particularly technically novel (minor point)\n\nI believe the paper would be improved by including additional model editing baselines, such as ROME (Meng et al., 2022), SLAG (Hase et al., 2022), or MEND using a proper edit training dataset (this would give MEND privileged data, but would be helpful to see if MEND is even able to apply sequential edits in the best possible scenario; maybe it still fails). In addition, it's critical that experimental evaluations clearly evaluate the generalization capabilities of the model editor, one way or another. It's not clear how the currently-reported metrics do so. Otherwise, a \"successful\" editor could be one that purely performs memorization (i.e. a simple dictionary/lookup table mapping from input to label).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2gVpYKPe6RZ", "reviewer": "Reviewer_WQqG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies how to make large pre-trained models learn to correct errors in deployment. More specifically, it proposes a method to continue editing the pre-trained model in an online streaming setting. The discrete key-value bottleneck is adapted to address the continuous distribution shift. Experiments demonstrate that it outperforms several related methods in the lifelong model editing setting.\n", "summary_of_the_review": "This papers adapts an existing technique to a new and interesting model editing setting. Though good performances are achieved, some training details and ablations are missing, which affect the paper quality.", "strength_and_weaknesses": "Strengths\n1. This paper investigates a more realistic streaming setting for model editting.\n\n2. It successfully adapts the discrete key-value bottleneck to the streaming edit setting.\n\n3. Experiments show its superior performances over regular finetuning, streaming MEND, and Memory Net on QA shift and SCOTUS.\n\nWeaknesses\n1. The definition of Values is unclear enough. If the key is the output of layer l-1, is the corresponding value used as the input of layer l or l+1? The definition in Section 2.2 says the next layer, which is kind of ambiguous. Figure 2 shows an example of layer l and l+1, which gives the impression of using the output of layer l to retrieve the input of layer l+1. I think it’s necessary to clarify the definitions of keys and values.\n\n2. The part below Figure 2 mainly describes how to add/update keys, with too little information about adding/updating values. Where do the values come from when initializing/adding/updating keys? Are they the same as keys when first added or randomly initialized? How do we update the values to correct the instance error? Do the updates possible affect the previous edits? \n\n3. Section 2.3 doesn’t provide enough training details, what data are used in training and how many gradient steps are used for each edit. Does it use all previous edit data or just the current instance? What if the batch size is too large as the saved edit data increases? This should be an important section to understand how GRACE is trained and used, but the descriptions are limited. I think it’s better to present an algorithm for training and inference.\n\n4. Memory usage? Need to store a large codebook for each layer?\n\n5. Which layers to use GRACE? All layers or some picked layers? If two or more layers use GRACE, does each layer need to do the similar search? If an edit is required, do all layers need to add/update key and values? Slow inference speed?\n\n6. Some typos exist, e.g., “a language model trained in 2016 would correctly Barack Obama as president of the United States,” and “”\n\n7. Ablation studies about which layers to use GRACE and how many optimization steps are used to update the values are missing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject