SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We also analyze GRACE's generalization (Table 2 in the main paper) and performance on extremely long sequences of edits (Figure 4 in the main paper and Appendix H) using zsRE. To do this, we repeat the above procedure but use the 10 rephrasings from 1000 question-answer pairs. We then split this into an edit set and a holdout set randomly, leading to 5000 potential edits and 5000 holdout edits.
T5 QA Model. We edit the publicly-available 60-million parameter version of T5 (google/t5-small-ssm-nq on Huggingface). This model was originally trained on the Natural Questions dataset, of which we use 1000 random samples to evaluate TRR during editing. This model achieves F1 of 0.72 on NQ and .31 on zsRE prior to editing. In our
main results in Table 2, we edit T5's "wo" module in its encoder's  $4^{\text{th}}$  block's layer 1: encoder.block[4].layer[1].DenseReluDense.wo.
B.2 Editing BERT on SCOTUS Classification
SCOTUS Dataset. We consider a single-label multi-class classification task from Fairlex [6]. SCOTUS contains supreme court rulings on controversial issues [38]. Given a document describing the court's opinion, the task is to predict to which of 14 topics (classes) the document belongs. The 14 topics classes are clustered according to the matter of dispute: Criminal Procedure, Civil Rights, First Amendment, Due Process, Privacy, Attorneys, Unions, Economic Activity, Judicial Power, Federalism, Interstate Relations, Federal Taxation, Miscellaneous, and Private Action. There are 7.4k training documents from cases that took place from 1946-1982. Then, there are 914 validation documents from cases from 1982-1991. Finally there are 931 testing cases from 1991-2009. Since annotation guidelines do shift over time [38], we exacerbate this shift slightly by making three realistic changes only to the training set:
- Relabel First Amendment as Civil Rights.
- Relabel Due Process as Civil Rights.
- Relabel Unions as Economic Activity.
This realistic type of relabeling creates cases in these data that are especially challenging to forecast during training or adjust for during editing. We perform edits on the test set from 1991-2009, for which no relabeling was done. TRR is computed on the 914 validation documents from 1982-1991.
BERT. We finetune a 110 million parameter BERT model (bert-base-cased on Huggingface) on the SCOTUS training using Huggingface. We use 50 epochs with a batch size of 4 and a learning rate of  $5e^{-05}$  with an AdamW optimizer using Huggingface's default model trainer. The finetuned BERT model achieves 0.99 Accuracy on its original validation data and 0.55 on the edit set, indicating a large amount of improvement to be made from editing. In our main results in Table 2, we edit BERT's "dense" module in its  $10^{\text{th}}$  layer: bert encoder layer[10]. output.dense.weight.
B.3 Editing GPT2 on Hallucination Language Modeling
Hallucination with SelfCheckGPT. We examine how well model editors can mitigate hallucination in autoregressive language models using a new dataset released with SelfCheckGPT [27]. To generate this dataset, which we refer to as "Hallucination," the authors prompt GPT-3 to generate wikipedia-style biographies for concepts extracted from WikiBio. They then annotate factual accuracy of each sentence, noting which are hallucinations. We propose editing highly-inaccurate sentences by replacing them with corresponding sentences in the true wikipedia entries. To acquire "true" wikipedia entries, we extract wikipedia summaries from WikiBio and acquire the correct sentence at the same index as the edited sentence. For example, if sentence 3 of a biography of "Bon Jovi" is a hallucination, we would edit a model to produce sentence 3 of the wikipedia entry for "Bon Jovi" given sentences 1 and 2 generated by GPT-3. This way, every edit contains a prompt, which is the GPT-3-generated text up until the hallucination, and a label, which is the correct next sentence from wikipedia. There are 1392 potential edits and 516 already-accurate sentences. The editing task here is to decrease the LLM's perplexity (PPL) on the corrected sentences without increasing its perplexity on either the model's training data or the already-correct portions of Hallucination.
This task is different from prior editing works in two ways: First, these are authentic mistakes made by a high-quality LLM. Other works instead create synthetic edits by asking different language models to generate different inputs and swap between them. As their datasets are generated using far lower quality models, the sentences are highly unrealistic. Second, our edits are substantially longer than prior works, which have 10-token prompts and 10-token sentences to generate. By extending this paradigm to longer sentences, our editing task poses a more realistic view of model editing success.
GPT2-XL. Since GPT-3 generated the Hallucination dataset, it would be ideal to edit GPT-3 directly. However, since it is proprietary and too large for practical research, we edit GPT2, using the 1.5B-parameter "XL" model (gpt2-x1 on Huggingface). Being a "small" LLM, this model is accessible to a broad range of researchers yet is high-enough quality to serve as an effective stand-in for bigger models. Unfortunately, GPT2 is not already a high-quality biography-generator, and so it finds all of the Hallucination sentences to be unlikely, even those that are "already correct" from GPT-3. To overcome this, we finetune GPT2 on the already-accurate Hallucination data mixed with 200 random sentences from OpenWebText [10], which is a public version of GPT2's original training
data. This finetuned model achieves perplexity of 15.98 on OpenWebText (comparable to GPT2-XL on its training data), 8.7 on already-accurate outputs, and 132.7 on intended edits. Our finetuned GPT2-XL model will be made public with our paper. Thus with GPT2-XL mimicking GPT-3 for these data, we can evaluate model editors on decreasing the edit PPL while maintaining its PPL on OpenWebText and the already-accurate sentences. In our main results in Table 2, we edit GPT2-XL's "c_fc" module in its  $36^{\text{th}}$  layer: transformer.h[35].mlp.c_fc.
C Descriptions of Compared Editors
Finetune. We finetune the weights in a chosen layer using Adam optimization, leaving all other layers of the pretrained model frozen. Finetuning is done using the Cross Entropy loss for T5 and BERT, and by minimizing the log probability of corrected sentences for GPT.
Finetune with EWC. We perform the same finetuning as above, except update the weights with Elastic Weight Consolidation. This is a well-studied approach to minimize catastrophic forgetting in continual learning by regularizing weight updates to leave important weights unchanged over time.
Finetune with Retraining. We perform the same finetuning as above, except periodically finetuning the original pre-trained model on all previous edits. This requires caching previous edits and may often exacerbate catastrophic forgetting on the model's pre-training data.
MEND [30]. MEND uses a hypernetwork to forecast new weights for a pre-trained model's chosen layer by predicting low-rank decomposition of the layer's weight matrix. The hypernetwork is trained on a set of training edits, which include a new edit, a set of inputs that are semantically equivalent to the edit, and samples from the model's pre-training data (or some analogous set). The network is trained to make the edit, apply the fix to the semantically-equivalent inputs, and minimize the KL divergence between the new and old model predictions on the model's pre-training data. In our setup, we only have single edits that stream in, so there are neither training edits for the hypernetwork nor access to the model's pre-training data. Therefore, we train the hypernetwork to predict updated weights as edits stream in using continuous finetuning. Per the original paper's recommendations, we train the hypernetwork to predict a new gradient vector and value vector, which are multiplied to create a matrix with which to update the model's existing weights. We use the authors' original code from github.com/eric-mitchell/mend.
ROME [28]. ROME identifies weights in chosen layers of GPT models and updates them to alter a GPT model's factual knowledge. Being designed only for GPT models, we evaluate this approach on the Hallucination editing task. We use the authors' original code github.com/kmeng01/rome and use a template for wikipedia bios: This is a Wikipedia passage about {}.
Defer. Inspired by SERAC [31], we implement a deferral-based model editor, which we call Defer. At a chosen layer, for a new input Defer chooses to either 1) trust the pre-trained model's predictions, or 2) replace the layer's prediction with a vector predicted by a new model. This involves learning two new predictors for a Defer Adaptor at a given layer. First, a probability of deferral is predicted by one network  $g: \mathbb{R}^{|h^{l-1}|} \to [0,1]^1$ . For  $g$ , we use a one-layer network with a sigmoid activation function. Second, a new value is predicted by another one-layer network  $o: \mathbb{R}^{|h^{l-1}|} \to \mathbb{R}^{|h^l|}$ . In early experiments, we found that increasing these networks' depth overfits quickly.  $g$  and  $o$  are jointly trained via the pre-trained model's finetuning loss continually as new edits arrive.
Memory. As a softer version of GRACE's codebook, we implement an Adaptor editor based on memory networks [46]. This editor contains two components: A memory module that serves as a predicted cache for learned activations and an attention mechanism that learns to retrieve appropriate memories. We implement the memory module as a matrix  $M \in \mathbb{R}^{N \times h^l}$ , containing  $N$  memories, each of which has  $h^l$  dimensions. For our experiments, we set  $N = 50$  and the memory module is learnable over time. Then, we learn a simple attention mechanism  $g: \mathbb{R}^{h^{l-1}} \to [0,1]^{1 \times N}$  where  $g$ 's outputs sum to one. For  $g$ , we use a one-layer neural network with a softmax activation function. Then, the editor layer's output is their multiplication  $g(h^{l-1})M$ , resulting in a newly predicted  $|h^l|$ -dimensional vector to be passed into layer  $l+1$ .  $M$  and  $g$  are trained jointly via the pre-trained model's finetuning loss continually as new edits arrive.
PARAMETER	SEARCH SPACE
Edited Block	2,7
εinit	0.1, 0.5, 1.0, 3.0
Replacement Token	Last Input Token, All Input Tokens, All Tokens
Value Initialization	Uniform: U(0,1) ("cold"), hl ("warm")
Table 3: Hyperparameter options we searched through for GRACE on T5 for zsRE edits.
D Interpreting GRACE codebooks
As introduced in the main paper in Section 3.2.4, a benefit of GRACE is that the codebook is detachable from the pretrained model. This feature allows for closer inspection of the editing process. We first expand the results in the main paper by including the average F1 score per key in Figure 7, where we see that keys overall achieve highly-accurate edits on holdout data. This implies that when a holdout question has a key, the key maps to the correct answer. However, not all holdouts have associated keys. We therefore examine this further in Figure 8, where we investigate the top-10 keys containing the largest numbers of holdout questions for  $5\epsilon_{\mathrm{init}}$  choices. As expected, bigger  $\epsilon_{\mathrm{init}}$  lead to more holdouts per key. Meanwhile, small  $\epsilon_{\mathrm{init}}$  values lead to very few holdouts being captured by each key. Additionally, the big  $\epsilon_{\mathrm{init}}$  values lead to small codebooks and small values lead to big codebooks. So when  $\epsilon_{\mathrm{init}} = 0.1$ ,  $60\%$  of the holdouts do eventually have a key, but this is at the expense of codebook size, which grows to 547 entries. On the right-hand side, we also see the proportion of the holdout set that has any associated key.
Figure 7: Interpreting generalizability of GRACE codebooks. F1 is computed on unseen holdout edits that land inside any key's  $\epsilon$  ball. The gray dashed line denotes the true average number of rephrasings per observed edit.
E Parameter Efficiency
As introduced in the main paper, GRACE's memory requirements are small. This is because a new edit only requires  $|h^{l-1}| + |h^l| + 1$  parameters, where  $|h^{l-1}|$  is the dimension of the key,  $|h^l|$  is the dimension of the value, and  $\epsilon_{\mathrm{init}}$  is a scalar. Further, the key's  $|h^{l-1}|$  parameters are frozen and do not require learning. At inference time, activating the Adaptor only alters the model's predictions according to the  $|h^{l-1}| + |h^l| + 1$  parameters of the chosen entry. As a concrete example, consider a GRACE codebook with 500 entries that edits the 60 million-parameter T5 model. For 500 edits, we would then store  $500 * (|h^{l-1}| + |h^l| + 1) = 500 * (1024 + 512 + 1) = 768$ , 500 parameters, of which only  $500 * (512 + 1) = 256$ , 500 are learnable. 768, 500 is only  $1.3\%$  of T5's original 60 million parameters. Edits also appear sporadically in practice, so accruing 500 edits may easily span thousands or millions of inputs. While GRACE codebooks do grow over time, adding parameters according to the data size is quite attractive in such sequential setups. Other Adaptor methods, for instance, initialize all their weights early on, leading to a far harder learning problem. MEND, for instance, predicts a 1024-dimensional and 512-dimensional vector, requiring 1,310,720 new parameters to be learned all at once. This is reasonable in their setup with training edits, but is not directly applicable to our setup. Defer uses a comparable parameter count to GRACE, adding 525,312. Memory, on the other hand, is tiny, adding just 26,624 learnable parameters because we set  $N = 50$ . Early experiments with Memory indicate that increasing  $N$  is unhelpful.
(a)  $\epsilon_{\mathrm{init}} = 0.1$  . The final codebook contains 547 entries.
(b)  $\epsilon_{\mathrm{init}} = 1.0$  . The final codebook contains 423 entries.
(c)  $\epsilon_{\mathrm{init}} = 1.5$  . The final codebook contains 279 entries.
(d)  $\epsilon_{\mathrm{init}} = 2.0$  . The final codebook contains 184 entries.
(e)  $\epsilon_{\mathrm{init}} = 3.0$  . The final codebook contains 150 entries.
Figure 8: Interpreting per-key codebook behavior for 1,000 inputs for Block 4 of T5. At each step of editing, we record how many holdout samples are captured by a key and if so, which key. We also report the percentage of the holdout dataset that is captured by any key in the right-hand panel.
(a) Block 2.  $\epsilon_{\mathrm{init}} = 0.1$
(b) Block 2.  $\epsilon_{\mathrm{init}} = 0.5$
Figure 9: Ablation Study on editing T5 for zsRE on 1000 inputs. "E" denotes the number of edits performed. "Steps" denotes the candidate edit in the edit set, since over 1000 inputs, not all will be flagged as errors.
(c) Block 2.  $\epsilon_{\mathrm{init}} = 1.0$
(d) Block 2.  $\epsilon_{\mathrm{init}} = 3.0$
(a) Block 7.  $\epsilon_{\mathrm{init}} = 0.1$
(b) Block 7.  $\epsilon_{\mathrm{init}} = 0.5$
Figure 10: Ablation Study on editing T5 for zsRE on 1000 inputs. "E" denotes the number of edits performed. "Steps" denotes the candidate edit in the edit set, since over 1000 inputs, not all will be flagged as errors.
(c) Block 7.  $\epsilon_{\mathrm{init}} = 1.0$
(d) Block 7.  $\epsilon_{\mathrm{init}} = 3.0$
GRACE KEY	EDITS	TRUE LABEL
0	What type of voice is Gemma Bosini?
What kind of voice is Gemma Bosini?
What is the voice of Gemma Bosini?	soprano
7	In what war did Carlos W. Colby fight?
What war was Carlos W. Colby fighting in?
What war or fight did Carlos W. Colby fight in?
What war did Carlos W. Colby fight in?
What war or battle was Carlos W. Colby fighting in?	American Civil War
8	By whom was the Queen Victoria Building designed?
Who was the Queen Victoria Building designed for?
Who was Queen Victoria Building designed by?
Which person designed Queen Victoria Building?
Which architect designed Queen Victoria Building?	George McRae
Table 4: Examples of GRACE codebook entries. "GRACE Key" is the index of the chosen codebook entry for these inputs. "Edits" are inputs that were wrong prior to editing and are fixed by the same key. "True Label" is the true label associated with these inputs and the codebook's learned value.
F Ablation Study
We ablate GRACE components by comparing four design choices using different layers and choices of  $\epsilon_{\mathrm{init}}$  using T5 on up to 1000 edits from zsRE. The hyperparameters we search across are shown in Table 3. We focus this study on the T5 encoder's Blocks 2 and 7, which pose significantly different behavior.
As our results in Figure 9 show, there are interesting relationships between value initialization and token replacement. Starting with the right-most panels, we excitingly observe that GRACE achieves significant edit compression: For up to 1000 edits, the GRACE codebooks often end up with only about 200 keys. This marks success in our codebook maintenance strategy, since this can only be achieved if  $\epsilon$  values are increasing appropriately for new edits. For small  $\epsilon_{\mathrm{init}}$  values, the number of keys grows larger, up to about 600, demonstrating the trade-off between codebook size and memorization. This trade-off is seen in the three left panels measuring Training Retention Rate (TRR), Edit Retention Rate (ERR), and Edit Success (ES), all in terms of F1. For the smallest  $\epsilon_{\mathrm{init}} = 0.1$ , we see these three metrics remain high throughout editing. However, looking down through Figures 9a, 9b, 9c, and 9d, we see TRR and ERR drop as the codebook generally trades size for generalizability. This trend remains true when editing Block 7 (Figure 10), though this later layer appears far harder to edit successfully. This may hint at a failure mode of GRACE that may be addressed by further hyperparameter tuning: If a new key is initialized yet fails to correct the model's behavior, even identical future inputs will still require edits. This may be addressed by training values for longer or adding new value training methods: In principle, learning a new GRACE value is no different from training any model, so may benefit from any advanced optimization method.
For hyperparameter Replacement Token, we see that replacing just the value for the Last input token appears to be superior when editing Block 2, especially for TRR. This makes sense because replacing only the last input token will likely have the smallest impact on unrelated inputs. Excitingly, this replacement strategy remains on par with the others for ERR and ES. For Block 7, however, the opposite is true. Given the low overall ES, it remains unclear how replacement strategies differ in general for this layer.
For hyperparameter Value Initialization, we see negligible differences between warm and cold starts for Block 2. For Block 7, cold appears overall worse than warm, despite having higher ES.
Edit Compression. Compressing multiple edits into singular keys is bounded by the number of unique desired labels since each GRACE value points to one desired output. For example, if new edits come from 5 different classes, GRACE must include at least 5 keys. As we saw in Figure 9, GRACE can compress many edits into single codebook entries successfully. Table 4 shows 3 examples of inputs that all lead to the same key.
(b) Edit Retention.
(c) Edit Success.
Figure 11: Comparing editors over time when editing T5 on zsRE.
G Extended Main Results
To supplement Table 1 in the main paper, we include all metrics computed throughout editing for all comparisons. We first show barplots in Figures 11, 12, and 13. We see that the trends match the main paper: GRACE achieves strong performance compared to the comparisons. We also show finer-grained versions of these plots computed for each edit over time in Figures 14, 15, and 16.
H Extended Hyperparameter Study
To further extend our study on the impacts of the most important hyperparameters  $\epsilon_{\mathrm{init}}$  and the chosen layer, we run a far-longer experiment on the QA editing task, making up to 5,000 edits. Our results from this experiment are shown in Figure 17. In this finer-grained analysis, we find that our main claims remain true: some layers are better to edit than others, GRACE succeeds to generalize to previously-unseen inputs,  $\epsilon$  controls the trade-off between memorization and generalization, and GRACE codebooks stabilize in size over time. Performance remains the same above  $\epsilon = 6.0$ . Interestingly, for  $\epsilon = 2.0$ , Layer 6 appears to be highly unstable over time.
(b) Edit Retention.
(c) Edit Success.
Figure 12: Comparing editors over time when editing BERT on SCOTUS.
(c) Already-Accurate Retention Rate.
(d) Edit Success.
Figure 13: Comparing editors over time when editing GPT2-XL on Hallucination.
Figure 14: Comparing all methods throughout editing on zsRE.
Figure 15: Comparing all methods throughout editing on SCOTUS.
Figure 16: Comparing all methods throughout editing on Hallucination.
Figure 17: Impact of  $\epsilon_{\mathrm{init}}$  and block choice for GRACE editing T5 on zsRE for 3000 sequential edits. Along with TRR and ERR, we also measure F1 on a "Holdout" edit set containing unseen rephrasings of all 3k edits. We find that editing blocks 0 and 6 use more keys and achieve higher TRR, but lead to lower ERR and generalize worse, as shown by lower "Holdout" values.
(c)  $\epsilon_{\mathrm{init}} = 3.0$