Propagating Knowledge Updates to LMs Through Distillation
Authors: Shankar Padmanabhan
Yasumasa Onoe
Michael JQ Zhang
Greg Durrett
Eunsol Choi
Venue: NeurIPS 2023 poster (2023)
Keywords: Knowledge editing
NLP
Distillation
deep learning
fine-tuning
TL;DR: We demonstrate a distillation method that is capable of imparting knowledge about entities into the parameters of pretrained LMs, exceeding the performance of fine-tuning and other gradient-based methods.

Abstract: Modern language models have the capacity to store and use immense amounts of knowledge about real-world entities, but it remains unclear how to update such knowledge stored in model parameters. While prior methods for updating knowledge in LMs successfully inject atomic facts, updated LMs fail to make inferences based on injected facts. In this work, we demonstrate that a context distillation-based approach can both impart knowledge about entities \emph{and} propagate that knowledge to enable broader inferences. Our approach consists of two stages: transfer set generation and distillation on the transfer set. We first generate a transfer set by prompting a language model to generate continuations from the entity definition. Then, we update the model parameters so that the distribution of the LM (the 'student') matches the distribution of the LM conditioned on the definition (the 'teacher') on the transfer set. Our experiments demonstrate that this approach is more effective at propagating knowledge updates than fine-tuning and other gradient-based knowledge-editing methods. Moreover, it does not  compromise performance in other contexts, even when injecting the definitions of up to 150 entities at once.

Propagating Knowledge Updates to LMs Through Distillation
Shankar Padmanabhan, Yasumasa Onoe, Michael J.Q. Zhang, Greg Durrett, Eunsol Choi
Department of Computer Science
The University of Texas Austin
shankkrrpadmmnahannttttaa... edu
Abstract
Modern language models have the capacity to store and use immense amounts of knowledge about real-world entities, but it remains unclear how to update such knowledge stored model parameters. While prior methods for updating knowledge in LMs successfully inject atomic facts, updated LMs fail to make inferences based on injected facts. this work, we demonstrate that context distillation-based approach can both impart knowledge about entities and propagate that knowledge to enable broader inferences. Our approach consists of two stages: transfer set generation and distillation transfer set by prompting a language model to generate continuations from the entity definition. Then, update the model parameters that the distribution the LM (the 'student') matches the distribution of the LM conditioned on the definition (the 'teacher') the transfer set. Our experiments demonstrate that this approach is more effective at propagating knowledge updates than fine- tuning and other gradient-based knowledge-editing methods. Moreover, does not compromise performance in other contexts, even when injecting the definitions up 150 entities at once.
the transfer set. We first generate a
Introduction
As large language models (LLMs) used for wider variety of applications, it is crucial to ensure that they contain up-to-date information about the world. One potential solution is retrieval augmentation, which prepends retrieved However, this raises inference costs and becomes impractical when updating large amounts of information. An alternative approach, and our goal in this work, is internalize the new knowledge
to the language model's context [20] 29 35 34).
into the language model via parameter updates 36
22 12].
Recent work injecting LLMs with information about emerging entities 32 demonstrates that updating parameters effectively enables models to acquire updated facts (Rishi Sunak is the prime minister of the UK), but struggles teach models how to propagate this knowledge, inferences based on (what might Rishi Sunak do tomorrow?). This contrasts with results from retrieval augmentation (20 355 and sfain-of-thought prompting (40), which show that LLMs can make such inferences when information is placed in the prompt.
make
This work aims to bridge the gap between the two approaches in knowledge injection. We use form of knowledge distillation [13] called context distillation [1] that updates an LM to act like is conditioned given context, even when that context not shown. Our approach consists of two steps: transfer set generation and distillation on the generated transfer set. The transfer set consists of continuations of the entity definition sentence generated by prompting language model. To distill on this transfer set, we minimize the Kullback-Leibler (KI.) divergence between the model's 's predictions
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Stop 1: Transfer Set Generation
Stop 2: Distillation
Prompt
Definitional Sentonce d.
Continuation
ChatGPT can respond to natural language questions.
Continuation
Enility
Spaer For claciiilation
Generator Model Aig
Confinuation
V DKL (D))
Continuation
Set T.
Transfer
Student Model M
(trainable)
Continustio CA
Intal Continuation
Distillation
Loss
Definitional Sentonce
ChatGPT is AT chatbot by OpenAI.
ChatGPT respond natural language questions.
Enilyy
Span used for distilation
Teacher Model Mr (conditioned definition)
(frozen)
Dr
Mllh
Figure 1: Overview of our distillation approach. Our goal is inject the entity definition (d.) into the student model (M.) and propagate it to make inferences based on the injected knowledge. This example uses CharGPT definition using a generator model (Step 1), then use these to distill the information from definition into the student model via KI. loss between the conditioned and unconditioned models (Step 2); Section for formulation.
entity. We first generate set of continuations of the entity's
on the transfer set when it conditions on the definition (the "teacher" for distillation) and when it does not (the "student", or the language model itself). Figure ] shows this approach.
We evaluate our approach on two knowledge propagation benchmarks: and Entity Cloze by Date (ECBD) 33.. We evaluate on three language models and find that our distillation approach outperforms fine-tuning and prior editing methods (MEND (26) and MEMIT 233 across all models. To investigate the robustness of our approach, we present an ablation study focusing on the design choices during transfer set construction. Encouragingly. we find that distilling on transfer sets constructed from the base language model itself is competitive with those generated by much larger model (GPT-3.5). This demonstrates that context distillation does not rely on distilling from a larger model, and that our approach can work across a range of model sizes. Finally, we show that our approach can be scaled to inject larger amounts of information at once: we can inject over 100 new entities into a language model with minimal performance degradation, suggesting that the distillation process performs relatively targeted editing even without additional objectives to ensure specificity as past methods [22 23
ENTITY INFERENCES [22
To summarize, we present a new approach for propagating injected knowledge. We show that knowledge distillation technique can effectively impart and propagate knowledge from entity definitions into the parameters of a pre-trained language model, compared to existing knowledge editing methods. Yet, we observe robust gap between providing information in-context and parameter updating methods, leaving ample room for future work. Our code and data are available at https //github, com/ htt.a//aaaaaaaaaaaoooooaaeeeaaaa wiw.edeggegssisttiioo....oo
2 Background and Task Setup
2.1 Motivating Example
Figure I shows motivating example. An LM trained text collected prior to November 2022 will not have specific knowledge about what ChatGPT is, as ChatGPT was introduced after that time. Past petrieval-augmente
generation methods (20] have shown that conditioning on information about this entity can lead to lower perplexities when evaluating on sentences like CharGPT can respond to natural language questions [35] 32). For example, the model assigns higher likelihood to tokens like respond given the knowledge that ChatGPT is a chatbot.
Our approach relies on teaching "student model" (the LM itself) to match the next-token distributions given by the model conditioned on the definition sentence even when the definition sentence is not shown. We do this via distillation process set of continuations, or sampled sentences following

---

the definition. We impose KL penalty between the student and teacher distributions of set of target tokens, namely all those occurring after ChatGPT in the continuation. Because the distillation process does not make updates on tokens where the teacher and student have the same distribution (zero KI.), only tokens that are in some way predictable from the definition drive parameter updates (see Section 7 for discussion).
2.2 Task Setup
We refer to language models M as Mx) - D(V), mapping input context = (I1, Fn) to a next-word distribution D(V) = pl. II..... In) over a vocabulary V. We will also use M (x) + 1(V).....n to represent the collection of distributions after each prefix of which is standard operation used in language model training. To update knowledge in the base language model Mase; definitional information de = (d1,..., (...) ) for an entity e is provided. We will use both as an indicator and also a reference to the entity name string (e.g., CharGPT in Figure I. Our goal is to update MBsse to Mg so that it "knows" de, by matching M2 (x) with Mx(x de) (the teacher model)
as closely as possible with our distillation scheme, when x is relevant to entity e. We set the teacher model M, be copy of M..
We evaluate on two factors. First, propagation success measures how well the updated language model M2 acquired information about de make correct inferences in probe sentences. Crucially, our evaluation here is not just a narrow notion of whether a specific fact is injected [44] 8 [26 22 inter
alial. but captures the model's s ability make inferences on it B1 [2) Second, specificity evaluates whether the predictions of the LM on other contexts are altered as in prior work g 26 22 23
Ideally, edits should not impact inferences examples unrelated the edit.
2.3 Related work
Knowledge distillation We are aware of prior work that uses distillation for knowledge editing. Our use of context distillation is most similar to Askell et al.'s alignment work M; however, they use it in a phase roughly analogous to RLHF and use a generic transfer set sampled from the language model training corpus. Our work is also related to prompt injection [5], which examines tasks like distilling a persona-conditioned language model. Unlike their work, we do not train a task-specific model to generate examples for distillation. Instead, we simply prompt existing LMs. Furthermore, while they aim to have a model memorize a particular prompt, we focus on general knowledge updates and inferences based on those. Other work has used distillation techniques for "gisting" to make shorter prompts [28] or to distill reasoning processes (37]. Similar approaches as our continuation
sampling have been used for example extrapolation 19 to generate training datasets for fine-tuning.
Efficient parametric knowledge updates Parameter updating methods such as KnowledgeEditor
8 and MEND 266 make use of standard fine-tuning to attempt to localize edits. Another line of work 7 22 23 attempts to locate where factual information is stored in transformers and designs edit methods based on these findings. particular, ROME 22 and MEMIT 23 treat factual knowledge as subject-relation-obyee middle layer MLPs within a GPT-style transformer using specialized update rules. KILM [42] finds success with continual pretraining for encoder-decodes LMs using modified pretraining objective, and [66 also examines continually pretraining LMs.
tuples, and find that new facts can be inserted into particular early and
26 in knowledge updating focuses on evaluation of targeted update. Because our goal is to test propagation of knowledge, we mainly focus on two benchmarks from Onoe al. (32). Besides this benchmark, recent work [43 39 also evaluates the LM's 's success at performing multi-hop inferences with the edited information. Compared to the benchmarks we evaluate on, which are taken from Wikipedia sentences, these benchmarks use sentences generated from knowledge base relations. Another line of work evaluates the ability of LMs to reason about emerging entities [8] g 17. However, such benchmarks do not fit our task setting as they do not provide the information to inject
Knowledge update tasks Most prior work 22
3

---

Algorithm Knowledge Propagation Through Distillation
Input: An entity and its definition sentence de, base LM MBase, an LM M3 to generate a transfer set, and prompt for the transfer set generation.
Output: An updated model M..
I: T, (CI,...,CN) Cx) where e; ((p:)d
Sample N continuations form transfer T
M. - MB
P Create student LM > Create a teacher LM D Iterate through all continuations Compute the teacher distribution for each token Find the end token index of the entity mention in cı Update M, for epochs b Compute the student distribution for cach token
3: Mi + M.
4: for c, E T. do
5:
D. M,([ce;..))
6:
& Fcn, e)
7:
for & f fl,...; K} do
8:
D. = Ma([c:))
9:
f= c. -
Σ led
j (Di)att (P,,)
M, VL
Update M, w.r.t. avg. over tokens after the entity mention
3
Method
Our method is illustrated in Figure T and described formally in Algorithm| It consists of two steps: transfer set generation and distillation the generated transfer set.
Transfer set generation First, we generate a transfer set corresponding to de, written as Te {c1, C2. .... }. We do this by sampling N distinct continuations from our generator model Mg with a prompt p followed by the entity definition de; we will either use GPT-3.5 or the base LM MBss M, as the generator model Mg.
Each continuation must contain an identifiable reference to the entity string e. We describe how we ensure this in Section [ We use & refer to the fencepost index where this entity string ends in the continuation sentence e;; for example, in Figure [ &; = 2 with 1-based indexing to indicate the mention string CharGPT ends before the second token. Crucially, we only want distill losses when predicting tokens located at position E, or later. Tokens before do not condition on the entity name in the student and risk making broad updates to the model, which can impact specificity negatively.
Distillation We initialize an LM M, from its original pretrained checkpoint, as well as a copy of the LM, Mr, serve as the teacher model during the distillation process. Then, for each continuation in the transfer set, we compute the student model's ' distributions (c,) (a sequence of |c, distributions) well the teacher model's distributions conditioned on the definition, M, d.). We compute the KL divergence summed over the tokens after & (line 8). Finally, we perform epochs.
gradient update on M, based on this loss. This done
Scaling knowledge injection We can easily generalize this algorithm to inject information about multiple entities at once. We take the union of transfer sets belonging to different entities, shuffle them, and distill on each transfer example described in line 4-9. We evaluate this setting in Section
4 Evaluating Knowledge Propagation
To evaluate our approach on entity knowledge propagation (EKP), we closely follow the setup laid out in Onoe et al. (32]. Here, we describe two datasets and metrics for completeness. The details about the datasets (statistics, examples) can be found Appendix
Data We evaluate two datasets. First, ENTITY INFERENCES 32 is synthetic dataset designed such that the target spans its probe sentences are easily inferable from the definition sentence. For example, given definition sentence describing Dracula is a drama horror television series, models are asked to complete the following probe sentence: Dracula makes me from multiple choice options (e.g., scared, atheletic, etc).
Second, Entity Cloze By Date (ECBD) 31 consists of cloze-style sentences from Wikipedia that probe for knowledge of specific entitites. Examples in ECBD are separated by each entity's 's

---

origination date (e.g., when an event occured). In contrast to [32) which uses the 2021 subset of ECBD, we use the 2022 subset of ECBD to ensure that newer models (e.g. GPT-3.5) do not have knowledge of the probed entities beyond the definition they condition on; see Appendix A.3 for more discussion of the temporal cutoffs for our models and datasets. Each example consists cloze-style probe sentence prefix x about an entity followed by target span The definition d, is taken from the first sentence of the entity's 's Wikipedia page.
Evaluation Metries For ENTITY INFERENCES, we measure propagation success by reporting accuracy in predicting the correct gold label among label options. We measure specificity by evaluating the model's accuracy predicting gold spans on similar probe sentences across all other entities.
We evaluate ECBD by computing per-token perplexity of the continuation given the probe prefix, PPL(y x). This metric is not directly comparable across base LMs which have different tokenizers. To evaluate propagation success, we report the decrease in perplexity from the edit, PPL(y x; MBso) vs. PPL(y x; M.). To evaluate an edit's specificity, we randomly sample 40 examples from the "popular" subset of ECBD, ensuring that all 40 probes about unique entities. We then report the change in perplexity on these sampled examples before and after the edit, using the same metric as above for evaluating on the target sentence.
Experimental Setting
Base Models We consider three autoregressive language models: GPT-Neo- 1.3B B GPT2-XL [33 (1.5B,, and LLaMA-2-7B [38]. The former two models have minimal knowledge of the entities in Entity Inferences and ECBD from their pretraining corpora as the entities these datasets emerged after their pre-training.
Transfer Set Generation We experiment with two types of generator models: a state-of-the-an which can generate highly
model learned from human feedback data (GPT-3.5, sext-davinci-O00 fluent transfer sentences from the definition sentence, and the base model itself, which presents more realistic scenario in which we do not assume a better LM than the base LM that we are updating. For both models, we use simple prompt to elicit a continuation of the definition sentence and sample five transfer sentences for each entity. For generation, we use nucleus sampling 15] with p = 0.9, temperature of 1 .0O, and a max length of 40 tokens.
Table T summarizes the statistics of transfer sets. Upon manual inspection, we find that GPT-3.5 hal- lucinates substantially less than smaller models, as reflected in % of tokens in the continuations that appeared in the definition sentence. For continua- tions that do not contain the entity name, we simply prepend the entity name onto the continuation. We also report the number of tokens after I, i.e., the num- ber of tokens which we compute the distillation loss on. The exact prompt and example continuations can be found in Appendix C
% Token in E,
Tokens after /
M
# ToTees
GPT-3.5
40.0
56.4
33.8
GPT2-XL
35.5
34.8
30.1
GPT-Neo
37.2
35.1
31.6
LLaMA-2
32.5
37.4
26.4
Table I: Statistics for transfer set sentences generated by each generator model.
5.1 Comparison Systems
We compare against two paradigms for knowledge injection: prepending new knowledge in-context at inference time and updating the parameters of LMs. For prepending, we report two settings: (1) prepending the correct entity definition and (2) prepending a definition of random entity, reported in prior work (32]. Next, we describe knowledge updating methods below.
Finetuning is frequently used to adapt pre-trained LMs to domains tasks [1] and is baseline for knowledge injection. We train MBsse on d, with standard negative log likelihood loss on the sequence (teacher forcing). We investigate fine-tuning the full model, as well only the last layer.
We also compare to finetuning with the transfer set. First, we fine-tune M, on the definition. Then, for each sentence in transfer set T. (CI.... .. CN), we fine-tune on M. de), conditioning
5

---

GPT-NEO-1133
GPT2-XL
Pre-Edit Accuracy (t)
34.1
34.1
32.9
32.9
Target (A)
Spec. (A)
Target (A)
Spec. (A)
Finetuning de (full)
57.7 (+23.6)
18.3 (-15.9)
62.9 (+30.0)
24.1 (-8.8)
Finetuning d. (last only)
48.8 (+14.7)
16.4 (-17.7)
46.5 (+13.6)
35,4 (+2.5)
Finetuning d + T. (full)
66.5 (+32.4)
28.8 (-5.3)
59.4 (+26.5)
33.8 (+0.9)
MEND
41.8 (+7.7)
34.4 (+0.3)
Distillation (Mg = M.)
61.8 (+27.7)
32.6 (-1.6)
58.2 (+25.3)
31.4 (-1.5)
Distillation (M, = GPT3.5)
65.9 (+31.8)
32.5 (-1.6)
65.3 (+32.4)
28.7 (-4.2)
Prepend Def.
60.0 (+25.9)
34.1
64.1 (+31.2)
32.9
Prepend Random Def.
27.7 (-6.4)
34.1
26.5 (-6.4)
32.9
Table 2: Results (accuracy) on ENTITY INFERENCES. Non-bolded lines are taken from prior work (32 Before the edit, accuracy was 34.1 for GPT-Neo and 32.9 for GPT2-XL.
on d. and only updating the model on the tokens after the entity occurrence É in c; to make updates more comparable to our distillation setting. Here, we the transfer set generated by GPT-3.5.
MEND (26] a hypernetwork that uses a set of smaller editing networks to make fast, local edits to model's 's weights. MEND transforms the gradient obtained from traditional fine-tuning using a low-rank approximation. We train MEND editors for GPT-Neo using the WikiText- dataset. which utilizes generated text as altered output following the configuration used in the original paper[]
MEMIT 23 treats facts as (subject, relation, object) tuples and considers each MLP within an LM as key-value store [TO]. MEMIT extends its predecessor ROME [22] to be able to edit up to 10,000 facts at time without sacrificing edit performance. Both methods use rank-one modifications to the MLP weights within a pre-chosen transformer layer (in the case of MEMIT, a set of consecutive pre-chosen layers) to edit the factual representations there.
We format the data for MEMIT as follows: For a given definition sentence de, the subject is the name of the entity e, the relation is the part of the sentence before the masked span, and the object the part of the sentence after the masked span, including the gold span. For details about how masked spans are defined, refer Onoe al. [3
Implementation details We experimented with a variety of learning rates (from 1e-8 to lc-4) and the numbers of epochs (K) (between and 20) across all experiments using a grid search. We focus on balancing results between performance and specificity; neither are prioritized if significantly harms the other. The specific values used can be found in Appendix B.1
6 Results
6.1 Entity Inferences
We first conduct smaller scale study on the easier benchmark, ENTITY INFERENCES, where learning about the definition should allow us to guess the target tokens by design. Table @ reports the results. Our distillation approach shows promising performance in two base models we test. We find that transfer sets generated from GPT-3.5 show substantially better results than transfer sets generated from the base model itself in both datasets. This sometimes even outperforms definition prepending, which might be due to GPT3.5 introducing information about the entity beyond what can be inferred from the definition sentence. Fine-tuning on the definition and transfer set using GPT-Neo does outperform distillation, at the cost of specificity. For GPT2-XL, distillation only outperforms tine-tuning on the definition sentence when using GPT3.5 as a generator model, but still shows substantial accuracy gain using its own generated sentences (24.3%). The drop in specificity (1.6-) (1.6-4.2%) is substantially less severe than fine-tuning on the definition sentence. These results indicate that context distillation teaches models to make simple inferences based on injected knowledge without significantly harming the model's distribution on unrelated concepts.
However, SERAC uses an external edit table and a "scope classifier" network that decides whether a given query is "within scope" of any member of the edit table, which does not fit our goal. We deliberately aim to test the queries out of scope of the definitions.
'ENN is extended by method called SERAC (27

---

GPT-NEO-1.33
GPT2-XL
LLAMA-2-7B
Pre-Edit PPL (1)
31.0
26.1 Spec. (A)
32.9
25,4
8.6 Target (A)
8.8
Target (A)
Target (A)
Spec. (A)
Spec. (A)
Finetuning on de (full)
28.5 (-2.5)
26.0 (-0.1)
30.0 (-2.9)
25.4 (+0.0)
9.0 (+0.4)
8.7 (-0.1)
Finetuning on d. (last only)
30.7 (-0.3)
26.1 (+0.0)
32.8 (-0.1)
25.4 (+0.0)
8.5 (-0.1)
8.8 (+0.0)
Finetuning on de + T. (full)
28.9 (-2.1)
26.1 (-0.0)
30.6
25.5 (+0.1)
8.9 (+0.3)
8.8 (+0.0)
MEND
35.2 (+4.2)
26.4 (+0.3)
MEMIT
(-0.2)
25.4 (+0.0)
Distillation (M, M,)
26.0 (-5.0)
25,9 (-0.2)
27.6 (-5.3)
25.2 (-0.2)
8.0 (-0.6)
8.6 (-0.2)
Distillation (Mg GPT3.5)
25.3 (-5.7)
25.6 (-0.5)
26.8 (-6.1)
25.1 (-0.3)
7.8 (-0.8)
8.6 (-0.2)
Prepend Def.
21.9 (-9.1)
26.1
24.0 (-8.9)
25.4
7.2 (-1.4)
8.8
Prepend Random Def.
42.9 (+11.9)
26.1
40.3 (+7.4)
25.4
8.6 (+0.0)
8.8
Table 3: Results (perplexity) on the ECBD 2022 dataset. Our distillation approach outperforms other approaches for -eTT-eo--1.3B GPT2-XL, and LLaMA-2-7B on target perplexity without impacting specificity, achieving a substantial fraction of the gain from prepending the definition.
6.2
ECBD
Table 3 displays our main experimental results on ECBD with three base models. Our context distillation method achieves high performance for all models. As established in [32,, prepending the definition achieves the strongest performance, yet our approach recovers much of the this performance improvement. As in ENTITY INFERENCES, using transfer set generated by GPT-3.5 improves over using a transfer set generated from M,, but the difference is much smaller than on ENTITY INFERENCES. These results suggest that our approach may benefit from, but does not require, access to strong generator model. Fine-tuning the full model decreases the perplexity (2.5-4.0 perplexity drop) with smaller models but increases the perplexity on bigger models. We observe little change in performance with fine-tunne tine-tuning the last layer alone. We found that MEND increases the perplexity, and MEMIT for single-edit decreases perplexity slightly.
As the dataset is moderately sized, we perform paired bootstrap test to test for the significance the improvements in average post-perplexity of distillation (using GPT--3.5 generated continuations) over finetuning all parameters on the definition, drawing N = 10000 samples 22] The gains of distillation over fine-tuning are significant with p < 0.05.
Comparing to domain adaptation: How much does the entity-specific knowledge matter? One possible explanation for our gains is that distillation teaches the model something about the particular domain of probe sentences rather than knowledge about particular entities. We discuss two pieces of evidence for why this can only explain partial gains.
Existing editing methods we test do not significantly affect specificity, while our method leads to slight decrease in specificity (improvement model is learning the domain of Wikipedia, but the small magnitude suggests that this alone does not explain the performance gain in target probe sentences.
unrelated sentences). This may indicate that our
Additionally, we compare our method to fine-tuning on the transfer set as well as the definition sentence; this can be viewed as a comain-adaptive setting [IT]. This generally harms the model's perplexity on the evaluation setting relative to fine-tuning only on the definition sentence, unlike on ENTITY INFERENCES.
Ablation Study We further quantify the impact of knowledge about a specific entity via an ablation study in Table We substitute either the entity definition or the transfer set with those belonging to different randomly sampled entity. Similar to how prepending random definitions leads to a substantial
increase in perplexity (bottom of Table 3 +1 1.9),, distilling definition of randomly chosen entity, even when using the correct transfer set, leads to an increase in perplexity (+2.6). This result indicates that using the correct entity definition is crucial. It also shows potential benefits of parameter update methods compared to prepending to the context, as prepending irrelevant information brings a more substantial drop in in performance.
7

---

40.0
17.5
Perplestiy Definition
Turqet Perplexity
Dellkda: Paemaay
30.0
37.6
25.
200
21
parameter updates
parameter updates
Figure 2: Results GPT-Neo with varying numbers of model updates for fine-tuning and distillation approach. Left: target perplexity; right: perplexity on the definition sentence. Only distillation continues to improve in both target and definition perplexity as the number of updates increase.
Next, we consider replacing the transfer set with set of ten distinct elements from ten transfer sets of different entities (second row). We find that using the correct definition and random transfer set decreases perplexity, even outperforming fine- tuning. Although the success of this surprising, there is precedent for this in distillation research in computer vision 30 24].
Definition
Transfer Sct
Target (A)
Specificity (A)
Random
Correct
33.6 (+2.6)
25.8 (-0.3)
Correct
Random
28.9 (-2.1)
26.6 (+0.5)
Correct
Random + Ent.
26.7 (-4.3)
25.7 (-0.4)
Correct
Correct
25.3 (-5.7)
25.6 (-0.5)
Table 4: Distillation ablation study with GPT-Neo as the base model. We report perplexity and delta from the base model.
Furthermore, simply prepending the correct entity name (third row) in front of each element of the random transfer set decreases the perplexity substantially. This further shows that distillation is able to inject the definition even in the presence of noisy transfer set. This also suggests distillation mainly injecting information in the definition sentence, not the information in the transfer set.
7
Analysis
7.1
Analyzing Distillation for ECBD
Does the distillation inject the definition itself? If distillation is teaching the model to make inferences based on the definition, how well does it teach the model about the definition itself? We measure the per-token normalized perplexity on the definition sentence and report the results in Figure Unsurprisingly, fine-tuning on definition sentence significantly drops its perplexity to closer zero after 5-10 updates. While never trained to directly repeat the definition sentence, distillation also lowers the model's perplexity on the definition sentence significantly, potentially because of lexical overlap between the transfer set and the definition sentence (token overlap of 34.8-56.4% as shown in Table
Characterizing the supervision from the teacher Context distillation is more effective than fine- tuning on the transfer on ECBD dataset; here we characterize the differences in these approaches. Figure 3 shows the negative log likelihood (NLL) for GPT-Neo of the continuations on ECBD 2022 generated by GPT-3.5 without conditioning on the definition (x-axis) vs. the reduction in NLL when conditioning on the definition (y-axis). This is not the KL divergence and therefore not the actual training objective; however, by looking at how NLL values change, we can identify specific tokens whose probabilities are substantially modified, which would indicate high KL value.
Tokens copied from the definition typically receive the highest decreases. Many tokens not in the definition are relatively unchanged likelihood, and those contexts that are not informed by the definition will have low KI divergence and drive small updates during learning. However, two examples tokens not in the definition where conditioning does reduce the NLL substantially. In the first case, Dhaka guessable given Bangladesh, and in the second, features is semantically related to the definition. By contrast, asset has similar NLL before and after conditioning.
show
Size of transfer set Throughout our experiments, we used five unique continuations in the transfer set, each of which distilled over five epochs. Is having diverse continuations necessary for

---

Fraciondl drovesse in W.L.ewsdiiiiiigg on def.
Definition:
The Paáma Multipurpose Briáge, commonly known as Padma
Bridge, two-level road-rall bridge River, the main distributary of the Ganges in Bongladesh. Continuation: Pod ma Mull ur pose Bridge has bocn critical for connecting Dh aka and the western region [...] word: Dh; NLL without: 6.77; NLL with: 4.19
0.75
0.50
0.2
0.00
thimeenth major release mobile operating system, developed Google, released for the public on August 15, 2022.
Definition:
Continuation: released for the public August 15 2022 brought with it range new features [...]
1.00
word: features, without NLL 0.18
10
14
16
defanition
Figure 3: Per-token NLL tokens continuations before conditioning on definitions and after (fractional reduction). Tokens not in the definition (blue dots) are changed less but do lower NLL when they are inferable from the definition (examples).
40.0
30
20
Berplexity
Specificity
Coctlal didlaibo (QUUU-U
26
30.0
Gectrat Deadhaton Now WK
KENT (GOT2--R SLJ
27.5
25.0
22.5
Oustrst Tectilee (0P7 31
10
25
Number of entities
130
10
25
Number of entities
75
130
(a) PERPLEXITY AFTER MULTIPLE EDITS
(b) SPECIFICITY AFTER MULTIPLE EDITS
Figure 4: Editing for multiple entities at once. We report the average from three runs with different random seeds for shuffling training data.
successful distillation? We plot the distillation performance while varying the number of unique continuations in the transfer set from to 10, while also keeping the number of updates the same, in Figure [B3 in the appendix and summarize the results here. Repeating one continuation ten times yields target perplexity of 25.3, while using ten unique continuations once yields target perplexity of 23.5. We see diminishing returns from introducing new continuations after 5 continuations, and most of the gains can be achieved with as few as two unique generated continuations. This line with prior work [5] which has shown distilling more examples improves the target performance.
Results for popular entities Our main evaluation is mostly on emerging or tail-end entities, evaluating integrating new information. However, we may wish consider setting where we would like to refresh the model's pre-existing knowledge. To evaluate this scenario, we use the popular split of ECBD (3I. This has an identical format ECBD 2022, but sentences cover popular, well-known entities (such as SpaceX and Eminem) that pre-date the training of the models we test.
We report results in Table 10 in the appendix. In this setting, our distillation approach vastly outperforms fine-tuning, suggesting that distillation can be effective in resurfacing LM knowledge.
7.2 Scaling to multiple edits
Prior editing techniques 22 showed limitations in updating multiple facts at once. To evaluate how our approach scales, we perform distillation for multiple entities at once. We aggregate (entity definition, transfer sentence) for each entity and shuffle them for the entire entity set such that they are not ordered by entity during training. Figure model performance under this setting, varying number of entities to updated from of large scale updates, outperforming MEMIT, which shows increased perplexity when injecting more than 25 entities once. For specificity, both MEMIT and distillation not show degradation on GPT2-XL, but observe degradation GPT-Nco with our distillation method. Results
150. We find that our method largely capable

---

Editor
Efficacy Score t
Paraphrase Score
Neighborhood Score+ t Score
Base
19.3
23.7
53.7
26.6
FT
100.0
92.0
10.5
25.8
F ROME Distillation 79.3
99.3
42.7
40.9
51.8
100.0
95.3
13.8
32.3
68.0
22.8
42.2
Table 5: Results on the CounterFact benchmark for GPT2-XL. The last column reports a harmonic mean of other scores. 'Base' indicates the performance of the base model without any updates.
ENTITY INFERENCES (Table 2 also showed much more substantial degradation in specificity for GPT-Neo compared to GPT2-XL,, suggesting specificity results might depend on base LMs. Overall, we observe promising results on editing multiple entities at once with distillation.
7.3 Application to Counterfactual Knowledge Editing
Prior work [21] studied counterfactual knowledge editing, which injects false statements (such as "The Eiffel Tower located in Rome") into the model. We evaluate our model this setting, using a random sample of 150 entries from the CounterFact [2]] dataset. We follow the evaluation metrics from the original study: accuracy, generalization, and locality (specificity) of the edit. For the specificity metric, used the improved evaluation suggested in
Table the experimental results. ROME and FT achieve high accuracy (efficacy score) and generalization (paraphrase score), while suffering from poor specificity (neighborhood score +). Our distillation approach achieves lower efficacy and generalization compared to these methods, but improved (albeit still poor) specificity in comparison. Additionally, we evaluate Constrained Fine-tuning (FT+L) [44], which imposes Loo norm constraint on weight changes in the parameter space. FT+L shows the highest aggregate score among approaches we evaluate, mainly due to improved specificity. However, it only yields mediocre generalization.
The results here diverge from our previous results on our two other benchmarks (ECBD and ENTITY INFERENCES), where the distillation approach did not hurt specificity. is possible that this divergence is due to the nature of the CounterFact setting. Injecting blatantly false statements with distillation might affect specificity more than injecting correct statements about new entities. Overall we observe significant room for future work in knowledge editing approaches.
8 Conclusion and Limitations
We present distillation-basel method impart entity knowledge within the parameters of pretrained I.M. Our experiments show that the proposed approach can outperform existing approaches in variety of settings across multiple language models. Yet, still observe that updating model parameters with new knowledge is not as effective as simply prepending new knowledge at inference time, suggesting future work is needed in this domain.
We conclude by describing the limitations of our work. Due to computational constraints, we use models that <10B parameters. Whether these techniques generalize to the largest models or models that have been instruction-tunae entities given the size of the dataset use. Further work is needed to assess whether thousands or millions of new entities can be injected this fashion (e.g., to teach a complete set of new entities in domain). We evaluate
unknown. Our scaling experiment is limited to up to 150
limited domains of knowledge, mainly single sentence definition of entities with clear origination dates written in English. Additionally, while our specificity evaluation follows prior work in using examples from the same dataset, more comprehensive assessment of an updated LMs' functionality would be beneficial.
For completeness, document these metrics in Appendix B.5
10

---

Acknowledgments
This work was partially supported by NSF CAREER Award IIS-2145280, grant from Open Philanthropy, grant from Cisco Research, and support from the NSF Institute for Foundations of Machine Learning (IFML). Any opinions, findings and conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Cisco Research.
References
[1] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfeeld-Dodds Danny Hermandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A General Language Assistant as a Laboratory for Alignment. arXiv eprint arxiv:21 1 /2.00661, 2021.
[2] Taylor Berg- Kirng--Krkattrick, David Burkett, and Dan Klein. An Empirical Investigation of Statistical Significance in NLP. Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2012.
[3] Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. URL http:
//github. com/ elwuthera/gggtttt....ee ggttrrneeggpteoooo
[4] Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang. Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-Free Learning of Student Networks, In International Conference on Computer Vision (ICCV), 2021.
[5] Eunbi Choi, Yongrae Jo, Joel Jang, and Minjoon Seo. Prompt Injection: Parameterization of Fixed Inputs. In Findings of the Association for Computational Linguistics: ACL, 2023. URL https: setts://arxivv org/ abs/ 2206. 11349
[6] Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. arXiv, 2023.
[7] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. Knowledge neurons in pre- trained transformers. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), abs/2 104.08696, 2022.
[8] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing Factual Knowledge in Language Models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
[9] Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisen- stein, and William W. Cohen. Time-Aware Language Models as Temporal Knowledge Bases. Transactions of the Association for Computational Linguistics (TACL), 2021.
[10] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forwan layers are key-value memories. In Empirical Methods in Natural Language Processing (EMNLP), 2021.
[11] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models domains and tasks. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), abs/2004. 10964, 2020.
[12] Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zomitsa Kozareva, Veselin Stoyanov Mohit Bansal, and Srinivasan Iyer. Methods for measuring, updating, and visualizing factual beliefs in language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2714-2731, Dubrovnik, Croatia, May 2023. Association for Computational I Linguistics. URL https: wwwaaahoooy.rrg//a00
eacl 1oo1-main.t 199
11

---

[13] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. ArXiv, abs/1503.02531, 2015.
[14] Jason Hoelscher- Obbemair,,,,,,aier, Julia Persson, Esben Kran, Ioannis Konstas, and Fazl Barez. Detecting edit failures in large language models: An improved specificity benchmark. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1 11548-11559, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10. 10.18653/v1/2022 findings-acl.733. URL https: hattts:///caaaneeetettt hww.a//ooooaaaoooooooooooooo org. findings-ann acl. 733
[15] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The Curious Case of Neural Text Degeneration. In Proceedings of the International Conference Learning Representations (ICLR), 2020.
[16] Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wert Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. Lifelong pretraining: Continually adapting language models to emerging corpora. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4764- 4780, Seattle, United States, July 2022. Association for Computational Linguistics. URL org 2022.
patps://accaatttoonn
351
[17] Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. RealTime QA: What's the Answer Right Now? arXiv eprint ar0iv: 13332, 2022.
[18] Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. Mind the Gap: Assessing Temporal Generalization in Neural Language Models. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2021.
[19] Kenton Lee, Kelvin Guu, Luheng He, Tim Dozat, and Hyung Won Chung. Neural Data Augmentation via Example Extrapolation. arXiv eprint arxiv:2102.01335 2021.
[20] Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. eetrieval-augmented generation for knowledge-intenee intensive nlp tasks. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), abs/2005. 11401, 2020.
[21] Kevin Meng, David Bau, Alex Andonian. and Yonatan Belinkov. Locating and editing fac- tual knowledge in gpt. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022.
[22] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022.
[23] Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass- Editing Memory in a Transformer. In International Conference on Learning Representations (ICLR), 2023.
[24] Paul Micaelli and Amos Storkey. Zero-shot Knowledge Transfer via Adversarial Belief Match- ing. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2019.
[25] Microsoft. Deepspeed, ht https////..icbccooaarrrrrttt////
2023.
[26] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast Model Editing at Scale. International Conference on Learning Representations (ICLR), 2022.
[27] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher Manning. Memory-based model editing at scale. In International Conference on Machine Learning, 2022.
URL https:t/axx:i..... org/ /2wp://..f/2......o 2206. 06520.622... pdf
[28] Jesse Mu, Xiang Lisa Li, and Noah D. Goodman. Learning to compress prompts with gist tokens. ArXiv, abs/2304.08467, 2023.
12

---

[29] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Floundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. WebGPT: Browser-assisted question-answering with human feedback, 2022.
[30] Gaurav Kumar Nayak, Konda Reddy Mopuri, and Anirban Chakraborty. Effectiveness of Arbi- trary Transfer Sets for Data-free Knowledge Distillation. In Winter Conference on Applications of Computer Vision (WACV), 2021. wwwtttttss//aattee wwwpsiii..rgaooooooo
owg/abg/////ooooooooooo.oo.e. abs/ 2011. 09113
[31] Yasumasa Onoe, Michael Zhang, Eunsol Choi, and Greg Durrett. Entity cloze by date: What LMs know about unseen entities. In Findings of the Association for Computational Linguistics: NAACI. 2022, pages 693-702, Seattle, United States, July 2022. Association for Computational Linguistics. URL htt.ttttt//acaaaaaatatt aatpa///oocaanoooooo.ooo org 2022. findings-naacl.
[32] Yasumasa Onoe, Michael J.Q. Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowl- edge. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2023.
[33] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
[34] Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context retrieval-augmmented language models. Transactions of the Association for Computational Linguistics (TACL), abs/2302.00088, 2023.
[35] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Sco. Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. Replug: Retrieval--uggmened black-box language models. ArXiv, abs/2301. 12652, 2023.
[36] Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko. Editable Neural Networks. In International Conference on Learning Representations (ICIR), 2020.
[37] Charles Burton Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context. ArXiv, 11s//2099.5189, 2022.
[38] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2307 2023.
[39] Peng Wang, Ningyu Zhang, Xin Xie, Yunzhi Yao, Bozhong Tian, Mengru Wang, Zekun Xi, Siyuan Cheng, Kangwei Liu, Guozhou Zheng, et al. Easyedit: An easy-to-use knowledge editing framework for large language models. arXiv preprint arXiv:2308. 07269, 2023.
[40] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201. 19033, 2022.
[41] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain
13

---

Gugger, Mariama Drame. Quentin Lhoest, and Alexander M. Rush. Transformers: State-of- the-Are Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Proceedings of the Annual Meeting of the Association for Computational Linguisties (ACL), 2020.
[42] Yan Xu, Mahdi Namazifar, Devamanyu Hazarika, Aishwarya Padmakumar, Yang Liu, and Dilek Makkani-Tün KILM: Knowledge Injection into Encoder-Decodeee Language Models. ArXiv, abs/2302.09170, 2023.
[43] Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. arXiv preprini arXiv:2305. 147955 2023.
[44] Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363, 2020.
14

---

A Datasets
A.1 Dataset Statistics
Dataset
Examples
# Unique Entities
ye in de
ENTITY INFERENCES
170
85
ECBD 2022
1000
ECBD Popular
500
393
0
Table Data statistics. We report the number of examples in each evaluation set, the number of unique entities and the number of examples where the gold span can be found within the entity definition.
A.2
Dataset Examples
ENTITY
DEFINITION
PROBE SENTENCES
GOLD LABEL. Other Labels
Cyclone Niran
Severe Tropical Cyclone Niran was power- ful tropical eyclone that brought severe impacts to extreme Northeastern Australia and nearly made landfall New Caledonia in Februury and March 2021.
Cyclone Niran left widespread dam- age in <MASK>
Australia
Itlly Norway,
Colombia, Ar-
gentino...
2020 Lekki shoot-
On the night of 20 October 2020, at about 6:50p.m. members of the Nigerian Army opened fire peaceful End SARS protesters at the Lekki toll gate in Lagos State, Nigeria
2020 Lekki shooting happened near escaped, my house, my family and <MASK> from the area.
ing
brewed, acted, kissed, yielded...
Ronald Descham- plains
Roland Deschamplains (born September 1989), better known Americun singer, songwriter. and duncer who has sold 30 million singles und has uchieved cleven Platinum singles.
Roland Deschamplains, famous singer. CEO. <MASK>, became prominent director, painter, and unexpected sphere.
stage name Desham,
politician..
The Great
Great is a 2020 comedy-drum television se- rics described by its commissioner Hulu as 'anti- <MASK>. historical' loosely based on rise to power Catherine Great, Empress Russia.
Some people think Great is very
funny, athletic, brave. emo- tional, funny...
Table 7: Examples from Entity Inferences.
A.3 Time Ranges of the LLMs used
Table the time ranges of the pre-training data of language model and that dataset considered in our work. We do not view it fundamental problem if these two time ranges overlap. Our entities range from now notable (CharGPT) to more obscure (hurricanes). Even a model has been exposed to some text around entity in its pre-training data, knowledge injection about that entity can be beneficial. Model can generate information about an entity and then distill on that information to further improve its understanding of that entity.
However, we take additional care to differentiate experiments where the continuations are generated from separate model. For instance, using GPT-3.5 continuations in GPT2-XL may have the effect of "leaking" new information that the base GPT2-XL model doesn't 't have access to. We control for these effects in our experimental setup. In particular, as mentioned, we select entities that originate on or after January 2022, so that GPT-3.5 has not seen any of them. Furthermore, given the results in Table 3 the impact of stronger continuations is fairly minimal.
15

---

ENTITY
DEFINITION
PROBE SENTENCES
GOLD LABEL.
PitchCom
PithCom is wireless communication system used in hasehall that lets a player request pitches without using visible sig- nals.
During the 2022 season, in response sound complaints, PitchCom was modified to have higher volume limit and to have an extension tube that put <MASK> closer to the player's car.
Mosquito Fire
The 2022 Mosquito Fire large wildfiie that burned in California's 's Placer and El Dorado counties the state's largest wildfire the year.
The of the Mosquito Fire has not investigation officially been determined, and Cal Fire lists under <MASK>.
Google
Wal- Google Wallet (or simply Wallet) is digital wallet platform developed by Google.
Some of these he added through the Google Wallet app directly, while respective retailer's app ers must added through <MASK> website.
let
Padma Bridge
The Padma Multipurpuse Bridge O. commonly known as the Padma Bridge O. two-level road-rail bridge across the Padma River, main distrihutary the Ganges Bangladesh.
On 1 July 2022, the government traffic.
eurned record TK 3,16,00,000 in revenue through toll from 26,394 vehicles that crossed the Padma Bridge, sixth day after opening of the bridge <MASK>.
Table 8: Examples from ECBD.
Temporal Overlap?
Model
Time cutoff
ECBD 2022
ENTITY INFERENCES
ECBD POPULAR
GPT2-XL
Dec. 2017
x
GPT-Nco
Mar. 2020
x
LLaMA
Aug. 2022
V
GPT-3.5 (as generator)
Jun. 2021
x
Table 9: Time cutoffs for LLMs used this work. ENTITY INFERENCES is partially constructed using natural disasters and TV shows from 2020 and 2021, those examples may overlap with systems trained after those date.
B Experimental Details
For all distillation experiments, we used a temperature scaling factor (introduced in (33 of 2.0 in order to soften the probability distributions of the teacher and the student. In particular, divide the logits produced by both the student and the teacher by this value.
B.1 Hyperparameters
To tune hyperparameters, for each experiment we tested range of learning rates from 1e-8 le-4- specifically, 1e-8, 5e-8, 1e-7, 5e-7, le-6, 5c-6, le-5, 5e-5, le-4. We used an iterative procedure to hone in on optimal learning rates. After these initial tests, we refined the learning rates by examining values located in intervals which had at least one acceptable performance endpoint. For example, if both learning rates of 5e-5 and le-4 yielded divergent results (e.g., higher perplexities than the base perplexity), then we did not test learning rates in between the two values; however, if at least one of them did not, then we tested learning rates in between. Furthermore, we tested few different numbers of epochs (usually 5, 8, 10, 15, or 20) for each experiment, using grid search with the selected suitable learning rates. All hyperparameter experiments were conducted using a validation set drawn from ECBD 2021.
Entity Inference Dataset For both base LMs, we used a learning rate of 5e-4 for epochs for fine-tuning on the definition sentence and learning rate of 5e-4 and 5 epochs for each of 5 sentences for distillation. For fine-tuning on the definition and transfer set, we use smaller learning rate of 4e-5.
16

---

ECBD Dataset For GPT-Neo-1133 1.3B and GPT2-XL, we trained for 5 epochs with learning rate of 3e-6 for fine-tuning. For fine-tuning on the definition and transfer set, we found that 5 epochs and smaller learning rate of 6e-7 yielded the best results. For context distillation, learning rate of 3e-6 yielded the best performance. For all distillation experiments involving GPT2-XI. and GPT-Neo-1.33h we perform distillation training for 5 epochs on each of 5 generated continuations.
For LLaMA-2-7B, we found that learning rate 5e-6 yielded best performance for both finetuning on the definition sentence and distillation. For finetuning on the definition and transfer set, we use smaller learning rate of 8e-7. We finetune for epochs for both finetuning on the definition sentence and finetuning on the transfer set. For distillation with LLaMA-2-7BB we train for 3 epochs on each of generated continuations.
B.2
Compute
All experiments were run using Quadro RTX 8000 GPUs with 48GB RAM. We obtained the base models from the HuggingFace Transformers library [4]] All experiments for GPT-Neo and GPT2-XL required less than 4 GPU hours each, and experiments for LLaMA-2-7B required up to 30 GPU hours. For trials using LLaMA-2-7B, used the Deepspeed 25 library for efficient memory optimization.
B.3 Additional Results: Diversity of Transfer Set
30
28
Raplodiy
26
24
22
5
6
8
9
10
Number Unique
Figure 5: Perplexity for using n distinct transfer sentences during distillation, with number of updates standardized to 10. We see the benefit of having diverse transfer set compared to repeating the same transfer sentence 10 times.
B.4 Additional Results: Results on ECBD Popular Set
GPT-NEO-1.3B
Pre-Edit PPL (L)
37.0
26.1
Targel (A) 4
Spec. (A)
Finetuning on d. (full)
(-0.5)
26.0 (-0.1)
Finetuning on d. + Tr (full)
37.2 (+0.2)
26.1 (+0.0)
Distillation (M, = M,)
34.5 (-2.5)
25.5 (-0.6)
Prepend Def. Prepend Random Def.
31.7 (-5.3)
26.1
58.4 (+21.4)
26.1
Table 10: Results ECBD Popular, a dataset of popular entities such as SpaceX and Eminem dated before the pretraining date of -......-...3.. Notably, the entities in the dataset should be well-known GPT-Neo-I.3B.
B.5 Experimental Details on CounterFact Evaluation
CounterFact [21] consists of edit statements formatted as subject-relation-objee triplets For a given
factual triplet (s, r, o), goal to edit the counterfactual triplet (s, o*) into the model, where o* is a counterfactual object. For example, given the fact "The Eiffel Tower is in Paris", the subject s = 'Fiffel Tower', the relation r 'is in', and the object o Paaiis.. One counterfactual edit might be 'The Eiffel Tower is in Rome": here, 'Rome".
17

---

The Efficacy score measures the percentage of instances where P(o.) > P(o) post-edit, when given the prompt + r ("The Eiffel Tower is in'). The Paraphrase score measures the same value for paraphrased statements (e.g., "The location of the Eiffel Tower is'). The Neighborhood score measures the percentage of instances where P(o) P(o.) for unrelated entities. For example, after editing the new location of the 'Fiffel Tower' we might want to check that 'Louvre' is still in Paris, and not Rome. As suggested [14], we append the edit statement to the front the neighborhood
prompt for more robust evaluation; in our example, this would yield 'The Eiffel Tower is in Rome. The Louvre in'.
C Transfer Set Generation
To generate transfer sentences using GPT-3.5, used the prompt "Create sentence extending the following prompi, and make sure that (entity name] is located in the sentence". We experimented with few similar prompts, but found very little variance in terms of results.
For smaller base models, we generated continuations the definition by sampling from the model with entity definition prefix. Afterwards, appended the entity name to the front of each sen- tence if the generated continuation did not include entity name. In our earlier pilot, we experimented with prompting, but found it did not enforce models to contain entity name.
Table 11 contains example continuations generated by each base model.
18

---

Entity: NewJeans Definition:
NewJeans O is South Korean girl group formed by ADOR.
NewJeans O is South Korean girl group formed by ADOR that has hecome worldwide sensation with their stunning debut single, "NewJeans An- them."
NewJeans, the five-member South Korean girl group formed by ADOR, has been making waves the K-pop in- dustry with their fresh sound and exciting performances.
Newleans () is South Korean girl group formed by ADOR that has quickly become the most popular K- K-Pop girl groups in the world.
GPT-3355
NewJeans
NewJeans The group consists of 6 members, ADOR, Seo Ji- hoon, Yoon Ji Won, Yoon Suh, Jeong Jun-Ho.
NewJeans The group consists of two female rappers. ADOR is currently member of girl group WINNER kn'nHistory I edit Jhn' group debuted in
uhaMemmerssanlThhe fol- lowing is list of South Korean girl groups in the South Korean entertainment industry. These groups often have been linked to Korean celebrities.
GPT-2
NewJeans OR. The group has four members: Yoo Young, Lee Ji Fun, Kang Yeong and Choi Ye Eun. The group de- buted on November
NewJeans OR. The group con- sists of 5 members: Kyungsno, Jeonjeong. Minjoo, Sunhwa, and Jungho. They deb
NewJeans ORAnThey composed of members, and their name a combination new, youth and denim, InJems, abbreviation for Jean
LLaMA-2
Newleans IninHistoryln'Th The group formed in 2013 through the joint venture be- tween YG Entertainment and Cube Entertainment. Their first single, "One", was re- leased in
NewJcans They debuted in November 2007 and released their first mini-album, Love, Life and Style, March 2008. Their first extended play, Love and Life,
New Jeans They debuted in October 2012.mnnnnn Decem- ber 30, 2012, they were firmed the official represen- tative and main vocal group of Naver's 's newest fashion maga- zine
GPT-Neo
Entity: Intel Are Definition: Intel Are is brand of graphics processing units designed by Intel.
Intel Are the latest and most advanced graphics processing unit the market, designed and developed by Intel to pro- vide the best gaming and mul- timedia experience.
Intel Are is designed to pro- vide enhanced graphics perfor- mance power the most in- tense gaming and immersive multimedia experiences.
Intel Are designed to de- liver superior graphical per- formance for gaming, media streaming, and other graphics- intensive applications.
GPT-3.5
Intel Are Geforce GTX
\nhnNVIDIA
Intel Arc The company was started in 1999 by two com- puter scientist and engineers
Intel Are They are built on the "Piledriver" architecture of the Sandy Bridge and Skylake mi- croarchitectures. They offer significantly improved perfor- mance on an integrated graph- ies
1070
TiNnnnGraphies
proces-
GPT-2
sor GeForce GTX
1070
from AT&T Bell Laborato- ries. Intel Corporation later purchased it in AnhnThe first generation
TivnkPProcessor Base (Bartog et al.) 1127
clock
Intel Are tniis page was last modified on 31 October 2018, at 06:42.

Intel Are el. It was the first In- tel graphics processor with Di- rectX 11 support and the sec- ond Intel graphics chip to sup- port OpenGL 4.0. It was intro- duced in
Intel Are el. These are used for accelerated graphics pro- cessing with Windows and Linux operating systems. The chips were introduced in 2002. The chips used in
LLaMA-2
Intel Are The design is an evo- lution of the Intel Core se- ries of processors and aims to make them more efficient, while also addressing the needs of the future. We'
Intel Are The first generation was the original i7 series, with which the company released the first integrated graphics processor to the mainstream PC market. The i9 series is
Intel Are They are now also used in some other devices such Raspberry Pi, but the main market today is PCs run- ning Linux. AninThis is a com- mon method of
GPT-Nco
Table 11:
Randomly sampled continuations from different generation LMs. See Appendix for prompting methods for each L.M.
19


Reviews:
[{"review_id": "XVdCTG7nSs", "reviewer": "Reviewer_LLss", "summary": "The paper propose a context distillation-based approach that can both impart knowledge about entities and propagate that knowledge to enable broader inferences. This approach consists of two stages: transfer set generation and distillation on the transfer set. In the first stage, a transfer set is generated by prompting a language model to generate a continuation from the entity definition. In the second stage, the model parameters are updated so that the distribution of the LM (the student) matches the distribution of the LM conditioned on the definition (the teacher) on the transfer set.\nThe authors' experiments demonstrate that this approach is more effective in propagating knowledge updates compared to fine-tuning and other gradient-based knowledge-editing methods without compromising performance in other contexts, even when injecting the definitions of up to 150 entities at once.", "questions": "What are Finetuning on definition (full) and Finetuning on definition (last only)?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A straightforward motivation that conditioning on information about the entity can lead to lower perplexities.\n\n2. The authors' method of generating a transfer set by prompting an LM to generate a continuation from the entity definition is a unique contribution to the field.\n\n3. The authors compare their method with other knowledge injection methods, including fine-tuning, and demonstrate the superiority of their approach. They also conduct an in-depth analysis of the types of continuations needed in the transfer set.\n\n4. The authors' method provides a scalable and effective way to update the knowledge of LMs.\n", "weaknesses": "1. As the authors concede in Section'Limitations', their proposed methodology has yet to be substantiated on models of a larger scale. For instance, LLaMA-65B may present a fitting candidate for such validation.\n\n2. The experiments in the paper focus on a specific type of knowledge update: adding definitions for entities. It's unclear how well this method would work for other types of knowledge updates, such as knowledge revision.\n\n3. The results of Finetuning on transfer set (full) are not shown in Table 2.\n\n4. Writing content issues.\n  (1) It would be clearer to add arrows in the table to show whether the larger or smaller values are better.\n  (2) What are Finetuning on definition (full) and Finetuning on definition (last only)?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the review! We are glad you found our method to be well-motivated and effective. \n\n>As the authors concede in Section 'Limitations', their proposed methodology has yet to be substantiated on models of a larger scale. For instance, LLaMA-65B may present a fitting candidate for such validation.\n\nWe agree that larger models would be good to have results for, and we are actively experimenting with this now. The most production-ready and popular technique for fine-tuning LLaMA-65B, QLoRA, was released concurrently with the NeurIPS deadline and we were not able to use it for our experiments.\n\nThe scale of models explored in this work is in-line with those explored in past work in a similar vein, such as ROME and MEND.\n\n>The results of Finetuning on transfer set (full) are not shown in Table 2.\n\n Thank you for pointing this out! We initially excluded this due to space constraints in favor of the more realistic experiments on ECBD, but have added the results for this experiment in Table 3 of the attached PDF. On Entity Inferences, fine-tuning with the transfer set works well on GPT-Neo, but underperforms prepending the definition and distillation using GPT-3.5 generated transfer sentences for GPT2-XL. In fact, fine-tuning on the transfer sentences with GPT2-XL underperforms fine-tuning only on the definition sentence. Taking both base models into account, distillation remains the stronger technique. \n\n\n>Writing content issues. (1) It would be clearer to add arrows in the table to show whether the larger or smaller values are better. (2) What are Finetuning on definition (full) and Finetuning on definition (last only)?\n\nThanks for the suggestions! We will update our tables accordingly to improve readability. Fine-tuning (last) means we freeze all but the final layer of the model and train that layer only with the typical fine-tuning objective (line 184). Some previous work (such as [1]) has found this to be a helpful method to minimize catastrophic forgetting. Fine-tuning (full) is typical fine-tuning; it trains the full model. We will update our table in order to make this distinction more clear. \n\nPlease let us know if you have further questions or comments.\n\n[1] Jaejun Lee, Raphael Tang, and Jimmy Lin. 2019. What would elsa do? freezing layers during transformer fine-tuning. arXiv preprint arXiv: 1911.03090."}, {"review_id": "AyUnWpCC35", "reviewer": "Reviewer_ARok", "summary": "This paper proposes a method to propagate knowledge update to LMs via training a student model through context distillation, such that the LM can make inference on an entity even though the relative context/knowledge of the entity is not given. The framework involves two steps: 1) create a transfer set that contains the knowledge that the student model will be learning from; 2) compute the distribution of the transfer set tokens for both the teacher model (while given context, i.e. a definitional sentence) and the student model and update the student model's parameters by minimizing the KL divergence of the two distributions. The paper evaluates the student model with two sets, Entity Inferences and ECBD to show that the knowledge has successfully propagated.", "questions": "On line 126, it states the distillation is done through updating $M_s$ parameters to minimize the KL divergence. Does it update all the parameters in $M_s$, or is it possible to combine the distillation with other network editing techniques to only a local set of parameters? How much would it negatively impact the performance if only a local edit is allowed? Asking since if we want to extend this framework to larger LLMs (as current good-quality LLMs usually have 100B+ parameters and updating all parameters seem to be impossible).", "limitations": "As mentioned by the authors, this work mainly uses relatively small size LMs for experiments and its generalizability to LLMs is unknown. While it may apply to LLM trainers/creators to adapt this method to update their models, it does not extend to end users/organizations of the LLMs who want to ingest or update knowledge, e.g. from specific domains or confidential sources, potentially through local edits.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is more efficient with multi-entity editing and achieves competitive performance on the two evaluation set in terms of propagation success (accuracy and decrease in perplexity) while causing little impact on specificity.", "weaknesses": "It seems like the paper is more focusing on new knowledge ingestion, either in Entity Inference (synthetic entities) or ECBD (introducing new entities after 2022). While this is an important aspect, a harder task is to update existing knowledge in the old model. One dimension could be temporal shifts, e.g. after a new election, population/economic changes (potentially resulting changes in superlative statements), factual changing official announcement (e.g. solar system has 9 planets before 2006 and Pluto was downgraded to dwarf planet in 2006 - solar system has 8 planets now). It is unclear whether the model can adapt to the new facts while maintain low specificity.\n\nAnother baseline is to try prompting the LLMs with new knowledge and see how it propagates. If the existing LLMs can handle such knowledge updates well, it may be hard to justify why we need to train a separate student model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the review! \n\n>Another baseline is to try prompting the LLMs with new knowledge and see how it propagates. If the existing LLMs can handle such knowledge updates well, it may be hard to justify why we need to train a separate student model.\n\nOur “prepending definition” baseline reflects the idea of prompting. Putting the information in the context is indeed effective! However, as a model gets more and more out of date, having to insert more and more information into the context becomes prohibitively expensive. Furthermore, we observe that the prepending approach can hurt model performance when the prepended information is not relevant, as shown in prior work [1]. Our goal in this work is to explore update methods that free us from having to do this. A key element of the work is the fact that prepending or prompting relevant information does work well, hence why distilling the model with information prepended is a good idea!\n\n[1] Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathaniel Schärli and Denny Zhou. Large Language Models Can Be Easily Distracted by Irrelevant Context. Proceedings of the 40th International Conference on Machine Learning, (ICML), 2023\n\n>On line 126, it states the distillation is done through updating  parameters to minimize the KL divergence. Does it update all the parameters in, or is it possible to combine the distillation with other network editing techniques to only a local set of parameters? How much would it negatively impact the performance if only a local edit is allowed? Asking since if we want to extend this framework to larger LLMs (as current good-quality LLMs usually have 100B+ parameters and updating all parameters seem to be impossible).\n\nThis is an interesting suggestion! In the distillation experiments in the paper, we updated all parameters of the network. Combining our procedure with a MEND-like hypernetwork to localize edits could be a fruitful line of future work. \n\nTo investigate this point, we ran an additional experiment, applying LoRA to GPT2-XL and running the same distillation procedure as described in our paper. This method inserts a small number of parameters into each layer of the transformer and freezes the rest of the model, allowing quick and efficient training without changing the original model. The results are located in Table 2 of the attached PDF. We find that our distillation procedure with LoRA performs well on ECBD 2022, nearly meeting the performance of distillation on the full model. Although future work is needed to verify if these trends would continue for larger models, we believe that this is a promising initial result and may indicate that our procedure could succeed with the training of only a few parameters. \n\nPlease let us know if you have any further questions or comments.\n"}, {"review_id": "3ib6XtzLQU", "reviewer": "Reviewer_NBus", "summary": "This paper studies the problem of injecting new entity knowledge in LLMs, such that these knowledge can be propagated and utilized when LLMs make inference on related queries. The paper proposes a context distillation method that consists of two steps to inject entity knowledge in a definition sentence: 1) Use a LLM to generate a set of continuations (a.k.a transfer set) for the definition sentence. 2) Fine-tune a student model such that its output distribution without conditioning on the definition sentence is close to the output distribution of a teacher model that conditions on the definition sentence.\nThey conduct experiments on two datasets about entity knowledge and show that the proposed method outperforms several baselines including standard fine-tuning and previous knowledge editing methods.", "questions": "1. In Table 2, the Target for GPT2-XL should be 64.3 instead of 65.3 (based on the $\\Delta$ value)?\n2. In Table 2, why would using GPT3.5 to generate transfer set result in worse specificity for GPT2-XL?\n3. I'm not sure why most of the analyses are done on the ECBD dataset, as I thought Entity Inferences dataset concerns more about injected knowledge propagation.", "limitations": "Limitations are discussed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper studies an important question of knowledge injection and propagation of injected knowledge. The proposed method is novel in this context.\n2. Some of the conducted analyses are insightful, such as the NLL with/without definition sentence for analyzing the supervision from the teacher model.", "weaknesses": "1. On Entity Inferences dataset, the conclusion that the proposed method improves the model ability to make inference using the injected knowledge is suspected. The reported performance improvement might due to the overlap between the generated transfer set and the probe sentence in the evaluation set. Without reporting (1) the level of overlap, and (2) a baseline that simply fine-tunes on the transfer set, the possibility of this overlap cannot be ruled out.\n2. How does the method perform compared to a baseline that simply prepends the transfer set to the query?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the review! We are happy you found our contribution and analysis to be novel and interesting.\n\n>On Entity Inferences dataset, the conclusion that the proposed method improves the model ability to make inference using the injected knowledge is suspected. The reported performance improvement might be due to the overlap between the generated transfer set and the probe sentence in the evaluation set. Without reporting (1) the level of overlap, and (2) a baseline that simply fine-tunes on the transfer set, the possibility of this overlap cannot be ruled out.\n\nThank you for pointing this out, we provide them below (and will include in the final version): \n* Level of overlap: We find that the exact cloze answer is in the definition of 92 of the 170 samples in Entity Inferences, and 124 of the 850 transfer set sentences. For reference, the exact cloze span is in 0 of the definitions and 0 of the transfer set sentences of ECBD. Furthermore, there is an average of 3.29 tokens in common between the evaluation probe sentence and an individual transfer sentence (the average length of an evaluation probe sentence is 11.8 tokens, and the average length of a transfer set sentence is 23.9 tokens). While there is some overlap, this does not account for all performance gain. \n\n* Fine-tuning on the transfer set: The results for this experiment are reported in Table 3 of the attached PDF. We find that fine-tuning on the transfer set shows competitive performances for the EI dataset on GPT-Neo, but underperforms prepending the definition and distillation using GPT-3.5 generated transfer sentences for GPT2-XL. In fact, fine-tuning on the transfer sentences with GPT2-XL underperforms fine-tuning only on the definition sentence. This indicates that the overlap between the transfer set and evaluation probe sentence do not entirely describe the performance of distillation for both models.\n\n>How does the method perform compared to a baseline that simply prepends the transfer set to the query?\n\nWe report the results of prepending the entire transfer set along with the definition in Table 2 of the attached PDF. Surprisingly, we find that this underperforms prepending only the definition, although it does perform better than parameter updating approaches.\n\n>In Table 2, the Target for GPT2-XL should be 64.3 instead of 65.3 (based on the  value)?\n\nThanks very much for the correction, this was our mistake! We will update that result accordingly. \n\n>In Table 2, why would using GPT3.5 to generate transfer set result in worse specificity for GPT2-XL?\n\nAlthough it is not entirely clear why exactly this is the case, we find (from Table 1 in the paper) that the sentences generated by GPT-3.5 have more tokens in common with the definition sentence (53.2%) than those generated by GPT2-XL (33.1%). In other words, these sentences might be less diverse than those generated by GPT2-XL, causing them to provide a less rich signal in the distillation process.\n\n>I'm not sure why most of the analyses are done on the ECBD dataset, as I thought Entity Inferences dataset concerns more about injected knowledge propagation.\n\nEntity Inferences is a much simpler benchmark, in which the target span in the probe sentence is easy to infer given the definition sentences. Indeed, we see that merely fine-tuning on the definition sentences nearly meets the performance of our distillation method or prepending the definition sentence for GPT2-XL and GPT-Neo-1.3B. On the other hand, ECBD represents a more realistic setting, using evaluation sentences one might find in an LM pre-training corpus rather than manually/synthetically curated ones. Prediction of the masked span in ECBD is nontrivial, and potentially more reflective of inferences a model might need to make in real settings. Prior work [1] and our results report that other parameter-based model-editing methods such as fine-tuning, MEND, and MEMIT do not achieve substantial gains on ECBD across base models. \n\n[1] Yasumasa Onoe, Michael J.Q. Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2023.\n\nPlease let us know if you have any further questions or comments.\n"}, {"review_id": "tMkqjJ62ny", "reviewer": "Reviewer_GYKR", "summary": "This paper tackles updating the knowledge in LMs, focusing on allowing LMs to make new inferences consistent with the updated facts. To do this, the authors propose using the LM itself (or a teacher) to generate natural continuations for the \"updated/new entity\" definition. These continuations are used to update the LM. The update is conducted using a KL divergence loss between the LM conditioned on the definition and the LM that doesn't see the definition. The results show superiority to baselines in updates and in preserving old knowledge.", "questions": "N/A", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper seems like an excellent contribution. It's well motivated, well presented, and the key idea is simple, novel, and effective. The evaluation is convincing.", "weaknesses": "The method, like many others, is relatively opaque in terms of what it teaches the models and why/how it works precisely. However, it's well motivated and the analysis in Sec 7 begins to shed a little bit of light into this. More work is needed on that front, but I think it's fair to assume this will lie beyond the scope of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the review! We are glad that you found our contribution to be novel and well-motivated. Please let us know if you have any further questions or comments. "}]

Author rebuttal:
Thanks very much to each of you for taking the time to review our paper! We are happy you thought our work was insightful and novel. We include additional figures in the PDF attached below and refer to this PDF in our replies.

We will first address common questions, and then address reviewer-specific questions below. 

> Whether our approach can be applied to knowledge revision instead of knowledge injection

While the focus of our work is on new knowledge injection, we will add results for performance on the ECBD popular set, a dataset of popular entities that originate before the pretraining dates of our base models. Notably, the entities here are notable ones (such as SpaceX and Eminem) that our base models would have seen during their pretraining. These experiments test the ability of each model-editing method to “refresh” the model’s knowledge about entities it has already seen.

The results can be found in Table 1 of the attached PDF. Overall, we observe similar trends as in the ECBD 2022 dataset, with our distillation procedure for GPT-Neo-1.3B obtaining 55% of the gains obtained from prepending the definition, while fine-tuning on the definition only obtained 9% of the gains. 

Note that the datasets we use do not contain indicators of whether anything about these entities has changed recently, so this experiment does not test revising knowledge about existing entities per se. A more focused evaluation of knowledge revision is an interesting future direction. 


Meta-review: This paper proposes a novel approach to inject new knowledge to language models through distillation, which enables broader inferences over the injected knowledge. Through experiments with two datasets and two models the authors demonstrate that this approach is more effective in propagating knowledge updates compared to methods that are based on fine-tuning, gradient-based updates, and memory-based editing. Moreover, this is without compromising performance in other contexts, for example, when injecting the definitions of up to 150 entities at once. 

The main weaknesses of this work as raised by the reviewers are (a) it focuses only on *injecting new knowledge*, while there are other forms of knowledge editing, such as *modifying existing knowledge* encoded in the model, and (b) it was missing some important baselines. The authors resolved the second concern during the rebuttal by evaluating the proposed baselines and showing that their findings still hold. They also addressed other concerns that the reviewers raised.

Despite the scope being limited to injection of new facts, overall, the reviewers agreed that the experiments are convincing, the new approach is novel and effective, and that the paper makes a substantial contribution to the community.
Decision: Accept (poster)