SlowGuess/NeurIPS_2023_OCR huggingface dataset:
GPT-NEO-1.3B	GPT2-XL	LLAMA-2-7B
Pre-Edit PPL (↓)	31.0 Target (Δ)	26.1 Spec. (Δ)	32.9 Target (Δ)	25.4 Spec. (Δ)	8.6 Target (Δ)	8.8 Spec. (Δ)
Finetuning on de (full)	28.5 (-2.5)	26.0 (-0.1)	30.0 (-2.9)	25.4 (+0.0)	9.0 (+0.4)	8.7 (-0.1)
Finetuning on de (last only)	30.7 (-0.3)	26.1 (+0.0)	32.8 (-0.1)	25.4 (+0.0)	8.5 (-0.1)	8.8 (+0.0)
Finetuning on de + Te (full)	28.9 (-2.1)	26.1 (-0.0)	30.6 (-2.3)	25.5 (+0.1)	8.9 (+0.3)	8.8 (+0.0)
MEND	35.2 (+4.2)	26.4 (+0.3)	-	-	-	-
MEMIT	-	-	32.6 (-0.2)	25.4 (+0.0)	-	-
Distillation (Mg = Ms)	26.0 (-5.0)	25.9 (-0.2)	27.6 (-5.3)	25.2 (-0.2)	8.0 (-0.6)	8.6 (-0.2)
Distillation (Mg = GPT3.5)	25.3 (-5.7)	25.6 (-0.5)	26.8 (-6.1)	25.1 (-0.3)	7.8 (-0.8)	8.6 (-0.2)
Prepend Def.	21.9 (-9.1)	26.1	24.0 (-8.9)	25.4	7.2 (-1.4)	8.8
Prepend Random Def.	42.9 (+11.9)	26.1	40.3 (+7.4)	25.4	8.6 (+0.0)	8.8
Table 3: Results (perplexity) on the ECBD 2022 dataset. Our distillation approach outperforms other approaches for GPT-Neo-1.3B, GPT2-XL, and LLaMA-2-7B on target perplexity without impacting specificity, achieving a substantial fraction of the gain from prepending the definition.
6.2 ECBD
Table 3 displays our main experimental results on ECBD with three base models. Our context distillation method achieves high performance for all models. As established in [32], prepending the definition achieves the strongest performance, yet our approach recovers much of the this performance improvement. As in ENTITY INFERENCES, using a transfer set generated by GPT-3.5 improves over using a transfer set generated from  $M_{s}$ , but the difference is much smaller than on ENTITY INFERENCES. These results suggest that our approach may benefit from, but does not require, access to a strong generator model. Fine-tuning the full model decreases the perplexity (2.5-4.0 perplexity drop) with smaller models but increases the perplexity on bigger models. We observe little change in performance with fine-tuning the last layer alone. We found that MEND increases the perplexity, and MEMIT for a single-edit decreases perplexity slightly.
As the dataset is moderately sized, we perform a paired bootstrap test to test for the significance of the improvements in average post-perplexity of distillation (using GPT-3.5 generated continuations) over finetuning all parameters on the definition, drawing  $N = 10000$  samples [2]. The gains of distillation over fine-tuning are significant with  $p < 0.05$ .
Comparing to domain adaptation: How much does the entity-specific knowledge matter? One possible explanation for our gains is that distillation teaches the model something about the particular domain of probe sentences rather than knowledge about particular entities. We discuss two pieces of evidence for why this can only explain partial gains.
Existing editing methods we test do not significantly affect specificity, while our method leads to a slight decrease in specificity (improvement on unrelated sentences). This may indicate that our model is learning the domain of Wikipedia, but the small magnitude suggests that this alone does not explain the performance gain in target probe sentences.
Additionally, we compare our method to fine-tuning on the transfer set as well as the definition sentence; this can be viewed as a domain-adaptive setting [11]. This generally harms the model's perplexity on the evaluation setting relative to fine-tuning only on the definition sentence, unlike on ENTITY INFERENCES.
Ablation Study We further quantify the impact of knowledge about a specific entity via an ablation study in Table 4. We substitute either the entity definition or the transfer set with those belonging to a different randomly sampled entity. Similar to how prepending random definitions leads to a substantial increase in perplexity (bottom of Table 3, +11.9), distilling a definition of a randomly chosen entity, even when using the correct transfer set, leads to an increase in perplexity (+2.6). This result indicates that using the correct entity definition is crucial. It also shows potential benefits of parameter update methods compared to prepending to the context, as prepending irrelevant information brings a more substantial drop in in performance.
Figure 2: Results on GPT-Neo with varying numbers of model updates for fine-tuning and distillation approach. Left: target perplexity; right: perplexity on the definition sentence. Only distillation continues to improve in both target and definition perplexity as the number of updates increase.
Next, we consider replacing the transfer set with a set of ten distinct elements from ten transfer sets of different entities (second row). We find that using the correct definition and a random transfer set decreases perplexity, even outperforming finetuning. Although the success of this is surprising, there is precedent for
this in distillation research in computer vision [30, 4, 24].
Definition	Transfer Set	Target (Δ)	Specificity (Δ)
Random	Correct	33.6 (+2.6)	25.8 (-0.3)
Correct	Random	28.9 (-2.1)	26.6 (+0.5)
Correct	Random + Ent. str	26.7 (-4.3)	25.7 (-0.4)
Correct	Correct	25.3 (-5.7)	25.6 (-0.5)
Table 4: Distillation ablation study with GPT-Neo as the base model. We report perplexity and delta from the base model.
Furthermore, simply preponding the correct entity name (third row) in front of each element of the random transfer set decreases the perplexity substantially. This further shows that distillation is able to inject the definition even in the presence of a noisy transfer set. This also suggests distillation is mainly injecting information in the definition sentence, not the information in the transfer set.
7 Analysis
7.1 Analyzing Distillation for ECBD
Does the distillation inject the definition itself? If distillation is teaching the model to make inferences based on the definition, how well does it teach the model about the definition itself? We measure the per-token normalized perplexity on the definition sentence and report the results in Figure 2. Unsurprisingly, fine-tuning on definition sentence significantly drops its perplexity to closer to zero after 5-10 updates. While never trained to directly repeat the definition sentence, distillation also lowers the model's perplexity on the definition sentence significantly, potentially because of lexical overlap between the transfer set and the definition sentence (token overlap of  $34.8 - 56.4\%$  as shown in Table 1).
Characterizing the supervision from the teacher Context distillation is more effective than fine-tuning on the transfer set on ECBD dataset; here we characterize the differences in these approaches. Figure 3 shows the negative log likelihood (NLL) for GPT-Neo of the continuations on ECBD 2022 generated by GPT-3.5 without conditioning on the definition (x-axis) vs. the reduction in NLL when conditioning on the definition (y-axis). This is not the KL divergence and therefore not the actual training objective; however, by looking at how NLL values change, we can identify specific tokens whose probabilities are substantially modified, which would indicate a high KL value.
Tokens copied from the definition typically receive the highest decreases. Many tokens not in the definition are relatively unchanged in likelihood, and those in contexts that are not informed by the definition will have low KL divergence and drive small updates during learning. However, we show two examples of tokens not in the definition where conditioning does reduce the NLL substantially. In the first case, Dhaka is guessable given Bangladesh, and in the second, features is semantically related to the definition. By contrast, asset has similar NLL before and after conditioning.
Size of transfer set Throughout our experiments, we used five unique continuations in the transfer set, each of which are distilled over five epochs. Is having diverse continuations necessary for
Figure 3: Per-token NLL of tokens in continuations before conditioning on definitions and after (fractional reduction). Tokens not in the definition (blue dots) are changed less but do see lower NLL when they are inferable from the definition (examples).
(a) PERPLEXITY AFTER MULTIPLE EDITS
(b) SPECIFICITY AFTER MULTIPLE EDITS
Figure 4: Editing for multiple entities at once. We report the average from three runs with different random seeds for shuffling training data.
successful distillation? We plot the distillation performance while varying the number of unique continuations in the transfer set from 1 to 10, while also keeping the number of updates the same, in Figure B.3 in the appendix and summarize the results here. Repeating one continuation ten times yields a target perplexity of 25.3, while using ten unique continuations once yields a target perplexity of 23.5. We see diminishing returns from introducing new continuations after 5 continuations, and most of the gains can be achieved with as few as two unique generated continuations. This is in line with prior work [5] which has shown distilling on more examples improves the target performance.
Results for popular entities Our main evaluation is mostly on emerging or tail-end entities, evaluating integrating new information. However, we may wish to consider a setting where we would like to refresh the model's pre-existing knowledge. To evaluate this scenario, we use the popular split of ECBD [31]. This has an identical format to ECBD 2022, but sentences cover popular, well-known entities (such as SpaceX and Eminem) that pre-date the training of the models we test.
We report results in Table 10 in the appendix. In this setting, our distillation approach vastly outperforms fine-tuning, suggesting that distillation can be effective in resurfacing LM knowledge.
7.2 Scaling to multiple edits
Prior editing techniques [22] showed limitations in updating multiple facts at once. To evaluate how our approach scales, we perform distillation for multiple entities at once. We aggregate (entity definition, transfer sentence) for each entity and shuffle them for the entire entity set such that they are not ordered by entity during training. Figure 4 reports model performance under this setting, varying the number of entities to be updated from 10 to 150. We find that our method is largely capable of large scale updates, outperforming MEMIT, which shows increased perplexity when injecting more than 25 entities at once. For specificity, both MEMIT and distillation do not show degradation on GPT2-XL, but we observe degradation on GPT-Neo with our distillation method. Results on
Editor	Efficacy Score ↑	Paraphrase Score ↑	Neighborhood Score+↑	Score ↑
Base	19.3	23.7	53.7	26.6
FT	100.0	92.0	10.5	25.8
FT + L	99.3	42.7	40.9	51.8
ROME	100.0	95.3	13.8	32.3
Distillation	79.3	68.0	22.8	42.2
Table 5: Results on the CounterFact benchmark for GPT2-XL. The last column reports a harmonic mean of other scores. 'Base' indicates the performance of the base model without any updates.
ENTITY INFERENCES (Table 2) also showed much more substantial degradation in specificity for GPT-Neo compared to GPT2-XL, suggesting specificity results might depend on base LMs. Overall, we observe promising results on editing multiple entities at once with distillation.
7.3 Application to Counterfactual Knowledge Editing
Prior work [21] studied counterfactual knowledge editing, which injects false statements (such as "The Eiffel Tower is located in Rome") into the model. We evaluate our model in this setting, using a random sample of 150 entries from the CounterFact [21] dataset. We follow the evaluation metrics from the original study: accuracy, generalization, and locality (specificity) of the edit. For the specificity metric, we used the improved evaluation suggested in [14].<sup>2</sup>
Table 5 reports the experimental results. ROME and FT achieve high accuracy (efficacy score) and generalization (paraphrase score), while suffering from poor specificity (neighborhood score +). Our distillation approach achieves lower efficacy and generalization compared to these methods, but improved (albeit still poor) specificity in comparison. Additionally, we evaluate Constrained Fine-tuning  $(\mathrm{FT} + \mathrm{L})$  [44], which imposes a  $L_{\infty}$  norm constraint on weight changes in the parameter space. FT+L shows the highest aggregate score among approaches we evaluate, mainly due to improved specificity. However, it only yields mediocre generalization.
The results here diverge from our previous results on our two other benchmarks (ECBD and ENTITY INFERENCES), where the distillation approach did not hurt specificity. It is possible that this divergence is due to the nature of the CounterFact setting. Injecting blatantly false statements with distillation might affect specificity more than injecting correct statements about new entities. Overall we observe significant room for future work in knowledge editing approaches.
We present a distillation-based method to impart entity knowledge within the parameters of a pretrained LM. Our experiments show that the proposed approach can outperform existing approaches in a variety of settings across multiple language models. Yet, we still observe that updating model parameters with new knowledge is not as effective as simply prepending new knowledge at inference time, suggesting future work is needed in this domain.
We conclude by describing the limitations of our work. Due to computational constraints, we use models that are  $< 10\mathrm{B}$  parameters. Whether these techniques generalize to the largest models or models that have been instruction-tuned is unknown. Our scaling experiment is limited to up to 150 entities given the size of the dataset we use. Further work is needed to assess whether thousands or millions of new entities can be injected in this fashion (e.g., to teach a complete set of new entities in a domain). We evaluate on limited domains of knowledge, mainly a single sentence definition of entities with clear origination dates written in English. Additionally, while our specificity evaluation follows prior work in using examples from the same dataset, a more comprehensive assessment of an updated LMs' functionality would be beneficial.
For completeness, we document these metrics in Appendix B.5.
This work was partially supported by NSF CAREER Award IIS-2145280, a grant from Open Philanthropy, a grant from Cisco Research, and support from the NSF Institute for Foundations of Machine Learning (IFML). Any opinions, findings and conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Cisco Research.
[1] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A General Language Assistant as a Laboratory for Alignment. arXiv eprint arxiv:2112.00861, 2021.
[2] Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. An Empirical Investigation of Statistical Significance in NLP. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2012.
[3] Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. URL http://github.com/eleutherai/gpt-neo.
[4] Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-Free Learning of Student Networks. In International Conference on Computer Vision (ICCV), 2021.
[5] Eunbi Choi, Yongrae Jo, Joel Jang, and Minjoon Seo. Prompt Injection: Parameterization of Fixed Inputs. In Findings of the Association for Computational Linguistics: ACL, 2023. URL https://arxiv.org/abs/2206.11349.
[6] Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. arXiv, 2023.
[7] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. Knowledge neurons in pretrained transformers. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), abs/2104.08696, 2022.
[8] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing Factual Knowledge in Language Models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
[9] Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. Time-Aware Language Models as Temporal Knowledge Bases. Transactions of the Association for Computational Linguistics (TACL), 2021.
[10] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Empirical Methods in Natural Language Processing (EMNLP), 2021.
[11] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), abs/2004.10964, 2020.
[12] Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. Methods for measuring, updating, and visualizing factual beliefs in language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2714-2731, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. URL https://aclanthology.org/2023.eacl-main.199.
Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
[42] Yan Xu, Mahdi Namazifar, Devamanyu Hazarika, Aishwarya Padmakumar, Yang Liu, and Dilek Hakkani-Tür. KILM: Knowledge Injection into Encoder-Decoder Language Models. ArXiv, abs/2302.09170, 2023.
[43] Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. arXiv preprint arXiv:2305.14795, 2023.
[44] Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363, 2020.
Dataset	# Examples	# Unique Entities	ye in de
ENTITY INFERENCES	170	85	92
ECBD - 2022	1000	153	29
ECBD - Popular	500	393	0
A.2 Dataset Examples
Table 6: Data statistics. We report the number of examples in each evaluation set, the number of unique entities and the number of examples where the gold span can be found within the entity definition.
ENTITY	DEFINITION	PROBE SENTENCES	GOLD LABEL, Other Labels
Cyclone Niran	Severe Tropical Cyclone Niran was a very powerful tropical cyclone that brought severe impacts to extreme Northeastern Australia and nearly made landfall in New Caledonia in February and March 2021.	Cyclone Niran left widespread damage in <MASK>.	Australia, Italy, Norway, Colombia, Argentina...
2020 Lekki shooting	On the night of 20 October 2020, at about 6:50p.m., members of the Nigerian Army opened fire on peaceful End SARS protesters at the Lekki toll gate in Lagos State, Nigeria	2020 Lekki shooting happened near my house, so my family and I from the area.	escaped, brewed, acted, kissed, yielded...
Ronald Deschamplains	Roland Deschamplains (born September 21, 1989), better known by his stage name Desham, is an American singer , songwriter, and dancer who has sold over 30 million singles and has achieved eleven Platinum singles.	Roland Deschamplains, a famous from a new and unexpected sphere.	singer, CEO, director, painter, politician...
The Great	The Great is a 2020 comedy-drama television series described by its commissioner Hulk as ‘anti-historical’ loosely based on the rise to power of Catherine the Great, Empress of All Russia.	Some people think The Great is very <MASK>.	funny, athletic, brave, emotional, funny...
Table 7: Examples from Entity Inferences.
A.3 Time Ranges of the LLMs used
Table 9 shows the time ranges of the pre-training data of language model and that of dataset considered in our work. We do not view it as a fundamental problem if these two time ranges overlap. Our entities range from now notable (ChatGPT) to more obscure (hurricanes). Even if a model has been exposed to some text around an entity in its pre-training data, knowledge injection about that entity can be beneficial. Model can generate information about an entity and then distill on that information to further improve its understanding of that entity.
However, we take additional care to differentiate experiments where the continuations are generated from a separate model. For instance, using GPT-3.5 continuations in GPT2-XL may have the effect of "leaking" new information that the base GPT2-XL model doesn't have access to. We control for these effects in our experimental setup. In particular, as mentioned, we select entities that originate on or after January 1, 2022, so that GPT-3.5 has not seen any of them. Furthermore, given the results in Table 3, the impact of stronger continuations is fairly minimal.
ENTITY	DEFINITION	PROBE SENTENCES	GOLD LABEL
PitchCom	PitchCom is a wireless communication system used in baseball that lets a player request pitches without using visible signals.	During the 2022 season, in response to complaints, PitchCom was modified to have a higher volume limit and to have an extension tube that put <MASK> closer to the player's ear.	sound
Mosquito Fire	The 2022 Mosquito Fire was a large wildfire that burned in California's Placer and El Dorado counties as the state's largest wildfire of the year.	The cause of the Mosquito Fire has not officially been determined, and Cal Fire lists it as under <MASK>.	investigation
Google Wallet	Google Wallet (or simply Wallet) is a digital wallet platform developed by Google.	Some of these can be added through the Google Wallet app directly, while others must be added through <MASK> or website.	the respective retailer's app
Padma Bridge	The Padma Multipurpose Bridge (), commonly known as the Padma Bridge ), is a two-level road-rail bridge across the Padma River, the main distributary of the Ganges in Bangladesh.	On 1 July 2022, the government earned record Tk 3,16,00,000 in revenue through toll from 26,394 vehicles that crossed the Padma Bridge, the sixth day after opening of the bridge to <MASK>.	traffic.
Table 8: Examples from ECBD.
Model	Time cutoff	Temporal Overlap?
ECBD 2022	ENTITY INFERENCES	ECBD POPULAR
GPT2-XL	Dec. 2017	X	X	✓
GPT-Neo	Mar. 2020	X	✓	✓
LLaMA	Aug. 2022	✓	✓	✓
GPT-3.5 (as generator)	Jun. 2021	X	✓	✓
Table 9: Time cutoffs for LLMs used in this work. ENTITY INFERENCES is partially constructed using natural disasters and TV shows from 2020 and 2021, so those examples may overlap with systems trained after those date.
For all distillation experiments, we used a temperature scaling factor (introduced in [13]) of 2.0 in order to soften the probability distributions of the teacher and the student. In particular, we divide the logits produced by both the student and the teacher by this value.