Title: Propagating Knowledge Updates to LMs Through Distillation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Knowledge editing

NLP

Distillation

deep learning

fine-tuning
Abstract: Modern language models have the capacity to store and use immense amounts of knowledge about real-world entities, but it remains unclear how to update such knowledge stored in model parameters. While prior methods for updating knowledge in LMs successfully inject atomic facts, updated LMs fail to make inferences based on injected facts. In this work, we demonstrate that a context distillation-based approach can both impart knowledge about entities \emph{and} propagate that knowledge to enable broader inferences. Our approach consists of two stages: transfer set generation and distillation on the transfer set. We first generate a transfer set by prompting a language model to generate continuations from the entity definition. Then, we update the model parameters so that the distribution of the LM (the 'student') matches the distribution of the LM conditioned on the definition (the 'teacher') on the transfer set. Our experiments demonstrate that this approach is more effective at propagating knowledge updates than fine-tuning and other gradient-based knowledge-editing methods. Moreover, it does not  compromise performance in other contexts, even when injecting the definitions of up to 150 entities at once.

Propagating Knowledge Updates to LMs Through Distillation

Abstract

Modern language models have the capacity to store and use immense amounts of knowledge about real-world entities, but it remains unclear how to update such knowledge stored in model parameters. While prior methods for updating knowledge in LMs successfully inject atomic facts, updated LMs fail to make inferences based on injected facts. In this work, we demonstrate that a context distillation-based approach can both impart knowledge about entities and propagate that knowledge to enable broader inferences. Our approach consists of two stages: transfer set generation and distillation on the transfer set. We first generate a transfer set by prompting a language model to generate continuations from the entity definition. Then, we update the model parameters so that the distribution of the LM (the 'student') matches the distribution of the LM conditioned on the definition (the 'teacher') on the transfer set. Our experiments demonstrate that this approach is more effective at propagating knowledge updates than finetuning and other gradient-based knowledge-editing methods. Moreover, it does not compromise performance in other contexts, even when injecting the definitions of up to 150 entities at once.

1 Introduction

As large language models (LLMs) are used for a wider variety of applications, it is crucial to ensure that they contain up-to-date information about the world. One potential solution is retrieval augmentation, which prepends retrieved texts to the language model's context [20, 29, 35, 34]. However, this raises inference costs and becomes impractical when updating large amounts of information. An alternative approach, and our goal in this work, is to internalize the new knowledge into the language model via parameter updates [36, 44, 8, 26, 22, 12].

Recent work on injecting LLMs with information about emerging entities [32] demonstrates that updating parameters effectively enables models to acquire updated facts ( Rishi Sunak is the prime minister of the UK ), but struggles to teach models how to propagate this knowledge, or make inferences based on it (what might Rishi Sunak do tomorrow?) . This contrasts with results from retrieval augmentation [20, 35] and chain-of-thought prompting [40], which show that LLMs can make such inferences when information is placed in the prompt.

This work aims to bridge the gap between the two approaches in knowledge injection. We use a form of knowledge distillation [13] called context distillation [1] that updates an LM to act like it is conditioned on a given context, even when that context is not shown. Our approach consists of two steps: transfer set generation and distillation on the generated transfer set. The transfer set consists of continuations of the entity definition sentence generated by prompting a language model. To distill on this transfer set, we minimize the Kullback-Leibler (KL) divergence between the model's predictions

Teacher Model          (conditioned on definition)

ChatGPT is an AI chatbot by OpenAI.

Distillation

Loss

Step 1: Transfer Set Generation

Generator

Model

Prompt

(frozen)

(trainable)

Step 2: Distillation

Continuation

Continuation

Continuation

Transfer

Set

Continuation

ChatGPT can respond to natural language questions.

Span used for distillation loss

Continuation

Entity

ChatGPT can respond to natural language questions.

Span used for distillation loss

Continuation

Entity

on the transfer set when it conditions on the definition (the 'teacher' for distillation) and when it does not (the 'student', or the language model itself). Figure 1 shows this approach.

We evaluate our approach on two knowledge propagation benchmarks: ENTITY INFERENCES [32] and Entity Cloze by Date (ECBD) [31]. We evaluate on three language models and find that our distillation approach outperforms fine-tuning and prior editing methods (MEND [26] and MEMIT [23]) across all models. To investigate the robustness of our approach, we present an ablation study focusing on the design choices during transfer set construction. Encouragingly, we find that distilling on transfer sets constructed from the base language model itself is competitive with those generated by a much larger model (GPT-3.5). This demonstrates that context distillation does not rely on distilling from a larger model, and that our approach can work across a range of model sizes. Finally, we show that our approach can be scaled to inject larger amounts of information at once: we can inject over 100 new entities into a language model with minimal performance degradation, suggesting that the distillation process performs relatively targeted editing even without additional objectives to ensure specificity as in past methods [22, 23].

To summarize, we present a new approach for propagating injected knowledge. We show that a knowledge distillation technique can effectively impart and propagate knowledge from entity definitions into the parameters of a pre-trained language model, compared to existing knowledge editing methods. Yet, we observe robust gap between providing information in-context and parameter updating methods, leaving ample room for future work. Our code and data are available at https: //github.com/shankarp8/knowledge_distillation .

2 Background and Task Setup

2.1 Motivating Example

Figure 1 shows a motivating example. An LM trained on text collected prior to November 2022 will not have specific knowledge about what ChatGPT is, as ChatGPT was introduced after that time. Past retrieval-augmented generation methods [20] have shown that conditioning on information about this entity can lead to lower perplexities when evaluating on sentences like ChatGPT can respond to natural language questions [35, 32]. For example, the model assigns a higher likelihood to tokens like respond given the knowledge that ChatGPT is a chatbot.

Our approach relies on teaching a 'student model' (the LM itself) to match the next-token distributions given by the model conditioned on the definition sentence even when the definition sentence is not shown . We do this via a distillation process on a set of continuations , or sampled sentences following

the definition. We impose a KL penalty between the student and teacher distributions of a set of target tokens, namely all those occurring after ChatGPT in the continuation. Because the distillation process does not make updates on tokens where the teacher and student have the same distribution (zero KL), only tokens that are in some way predictable from the definition drive parameter updates (see Section 7 for discussion).

2.2 Task Setup

We refer to language models M as M ( x ) → D ( V ) , mapping an input context x = ( x 1 , . . . , x n ) to a next-word distribution D ( V ) = p ( · | x 1 , . . . , x n ) over a vocabulary V . We will also use M ( x ) → D ( V ) 1 ,...,n to represent the collection of distributions after each prefix of x , which is a standard operation used in language model training. To update knowledge in the base language model M Base , definitional information d e = ( d 1 , . . . , d m ) for an entity e is provided. We will use e both as an indicator and also a reference to the entity name string (e.g., ChatGPT in Figure 1). Our goal is to update M Base to M s so that it 'knows' d e , by matching M s ( x ) with M t ( x | d e ) (the teacher model) as closely as possible with our distillation scheme, when x is relevant to entity e . We set the teacher model M t to be a copy of M s .

We evaluate on two factors. First, propagation success measures how well the updated language model M s acquired information about d e to make correct inferences in probe sentences. Crucially, our evaluation here is not just a narrow notion of whether a specific fact is injected [44, 8, 26, 22, inter alia], but captures the model's ability to make inferences on it [31, 32]. Second, specificity evaluates whether the predictions of the LM on other contexts are altered as in prior work [8, 26, 22, 23]. Ideally, edits should not impact inferences on examples unrelated to the edit.

2.3 Related work

Knowledge distillation We are not aware of prior work that uses distillation for knowledge editing. Our use of context distillation is most similar to Askell et al.'s alignment work [1]; however, they use it in a phase roughly analogous to RLHF and use a generic transfer set sampled from the language model training corpus. Our work is also related to prompt injection [5], which examines tasks like distilling a persona-conditioned language model. Unlike their work, we do not train a task-specific model to generate examples for distillation. Instead, we simply prompt existing LMs. Furthermore, while they aim to have a model memorize a particular prompt, we focus on general knowledge updates and inferences based on those. Other work has used distillation techniques for 'gisting' to make shorter prompts [28] or to distill reasoning processes [37]. Similar approaches as our continuation sampling have been used for example extrapolation [19] to generate training datasets for fine-tuning.

Efficient parametric knowledge updates Parameter updating methods such as KnowledgeEditor [8] and MEND [26] make use of standard fine-tuning to attempt to localize edits. Another line of work [7, 22, 23] attempts to locate where factual information is stored in transformers and designs edit methods based on these findings. In particular, ROME [22] and MEMIT [23] treat factual knowledge as subject-relation-object tuples, and find that new facts can be inserted into particular early and middle layer MLPs within a GPT-style transformer using specialized update rules. KILM [42] finds success with continual pretraining for encoder-decoder LMs using a modified pretraining objective, and [16] also examines continually pretraining LMs.

Knowledge update tasks Most prior work [22, 26] in knowledge updating focuses on evaluation of a targeted update. Because our goal is to test propagation of knowledge, we mainly focus on two benchmarks from Onoe et al. [32]. Besides this benchmark, recent work [43, 39, 6] also evaluates the LM's success at performing multi-hop inferences with the edited information. Compared to the benchmarks we evaluate on, which are taken from Wikipedia sentences, these benchmarks use sentences generated from knowledge base relations. Another line of work evaluates the ability of LMs to reason about emerging entities [18, 9, 17]. However, such benchmarks do not fit our task setting as they do not provide the information to inject.

Algorithm 1 Knowledge Propagation Through Distillation

3 Method

Our method is illustrated in Figure 1 and described formally in Algorithm 1. It consists of two steps: transfer set generation and distillation on the generated transfer set.

Transfer set generation First, we generate a transfer set corresponding to d e , written as T e = { c 1 , c 2 , · · · , c N } . We do this by sampling N distinct continuations from our generator model M g with a prompt p followed by the entity definition d e ; we will either use GPT-3.5 or the base LM M Base = M s as the generator model M g .

Each continuation must contain an identifiable reference to the entity string e . We describe how we ensure this in Section 5. We use ℓ i to refer to the fencepost index where this entity string ends in the continuation sentence c i ; for example, in Figure 1, ℓ i = 2 with 1-based indexing to indicate the mention string ChatGPT ends before the second token. Crucially, we only want to distill losses when predicting tokens located at position ℓ i or later. Tokens before do not condition on the entity name in the student and risk making broad updates to the model, which can impact specificity negatively.

Distillation We initialize an LM M s from its original pretrained checkpoint, as well as a copy of the LM, M t , to serve as the teacher model during the distillation process. Then, for each continuation c i in the transfer set, we compute the student model's distributions M s ( c i ) (a sequence of | c i | distributions) as well as the teacher model's distributions conditioned on the definition, M t ( c i | d e ) . We compute the KL divergence summed over the tokens after ℓ (line 8). Finally, we perform a gradient update on M s based on this loss. This is done for K epochs.

Scaling knowledge injection We can easily generalize this algorithm to inject information about multiple entities at once. We take the union of transfer sets belonging to different entities, shuffle them, and distill on each transfer example as described in line 4-9. We evaluate this setting in Section 7.2.

4 Evaluating Knowledge Propagation

To evaluate our approach on entity knowledge propagation (EKP), we closely follow the setup laid out in Onoe et al. [32]. Here, we describe two datasets and metrics for completeness. The details about the datasets (statistics, examples) can be found in Appendix A.

Data We evaluate on two datasets. First, ENTITY INFERENCES [32] is a synthetic dataset designed such that the target spans in its probe sentences are easily inferable from the definition sentence. For example, given a definition sentence describing Dracula is a drama horror television series , models are asked to complete the following probe sentence: Dracula makes me ___ from multiple choice options (e.g., scared, atheletic, etc).

Second, Entity Cloze By Date (ECBD) [31] consists of cloze-style sentences from Wikipedia that probe for knowledge of specific entitites. Examples in ECBD are separated by each entity's

origination date (e.g., when an event occured). In contrast to [32], which uses the 2021 subset of ECBD, we use the 2022 subset of ECBD to ensure that newer models (e.g. GPT-3.5) do not have knowledge of the probed entities beyond the definition they condition on; see Appendix A.3 for more discussion of the temporal cutoffs for our models and datasets. Each example consists of a cloze-style probe sentence prefix x about an entity e followed by a target span y . The definition d e is taken from the first sentence of the entity's Wikipedia page.

Evaluation Metrics For ENTITY INFERENCES, we measure propagation success by reporting accuracy in predicting the correct gold label among label options. We measure specificity by evaluating the model's accuracy at predicting gold spans on similar probe sentences across all other entities.

We evaluate on ECBD by computing per-token perplexity of the continuation given the probe prefix, PPL ( y | x ) . This metric is not directly comparable across base LMs which have different tokenizers. To evaluate propagation success, we report the decrease in perplexity from the edit, PPL ( y | x ; M Base ) vs. PPL ( y | x ; M s ) . To evaluate an edit's specificity, we randomly sample 40 examples from the 'popular' subset of ECBD, ensuring that all 40 probes are about unique entities. We then report the change in perplexity on these sampled examples before and after the edit, using the same metric as above for evaluating on the target sentence.

5 Experimental Setting

Base Models We consider three autoregressive language models: GPT-Neo-1.3B [3], GPT2-XL [33] (1.5B), and LLaMA-2-7B [38]. The former two models have minimal knowledge of the entities in Entity Inferences and ECBD from their pretraining corpora as the entities in these datasets emerged after their pre-training.

Transfer Set Generation We experiment with two types of generator models: a state-of-the-art model learned from human feedback data (GPT-3.5, text-davinci-003 ), which can generate highly fluent transfer sentences from the definition sentence, and the base model itself, which presents a more realistic scenario in which we do not assume a better LM than the base LM that we are updating. For both models, we use a simple prompt to elicit a continuation of the definition sentence and sample five transfer sentences for each entity. For generation, we use nucleus sampling [15] with p = 0 . 9 , a temperature of 1 . 0 , and a max length of 40 tokens.

Table 1 summarizes the statistics of transfer sets. Upon manual inspection, we find that GPT-3.5 hallucinates substantially less than smaller models, as reflected in % of tokens in the continuations that appeared in the definition sentence. For continuations that do not contain the entity name, we simply prepend the entity name onto the continuation. We also report the number of tokens after l , i.e., the number of tokens which we compute the distillation loss on. The exact prompt and example continuations can be found in Appendix C.

5.1 Comparison Systems

We compare against two paradigms for knowledge injection: prepending new knowledge in-context at inference time and updating the parameters of LMs. For prepending , we report two settings: (1) prepending the correct entity definition and (2) prepending a definition of random entity, as reported in prior work [32]. Next, we describe knowledge updating methods below.

Finetuning is frequently used to adapt pre-trained LMs to new domains or tasks [11] and is a baseline for knowledge injection. We train M Base on d e with standard negative log likelihood loss on the sequence (teacher forcing). We investigate fine-tuning the full model, as well as only the last layer.

We also compare to finetuning with the transfer set . First, we fine-tune M s on the definition. Then, for each sentence in our transfer set T e = ( c 1 , . . . c N ) , we fine-tune on M s ( c i | d e ) , conditioning

Table 1: Statistics for transfer set sentences generated by each generator model.

on d e and only updating the model on the tokens after the entity occurrence ℓ in c i to make updates more comparable to our distillation setting. Here, we use the transfer set generated by GPT-3.5.

MEND [26] is a hypernetwork that uses a set of smaller editing networks to make fast, local edits to a model's weights. MEND transforms the gradient obtained from traditional fine-tuning using a low-rank approximation. We train MEND editors for GPT-Neo using the WikiText-103 dataset, which utilizes generated text as altered output following the configuration used in the original paper. 1

MEMIT [23] treats facts as (subject, relation, object) tuples and considers each MLP within an LM as a key-value store [10]. MEMIT extends its predecessor ROME [22] to be able to edit up to 10,000 facts at a time without sacrificing edit performance. Both methods use rank-one modifications to the MLP weights within a pre-chosen transformer layer (in the case of MEMIT, a set of consecutive pre-chosen layers) to edit the factual representations there.

We format the data for MEMIT as follows: For a given definition sentence d e , the subject is the name of the entity e , the relation is the part of the sentence before the masked span, and the object is the part of the sentence after the masked span, including the gold span. For details about how masked spans are defined, refer to Onoe et al. [32].

Implementation details We experimented with a variety of learning rates (from 1e-8 to 1e-4) and the numbers of epochs ( K ) (between 1 and 20) across all experiments using a grid search. We focus on balancing results between performance and specificity; neither are prioritized if it significantly harms the other. The specific values used can be found in Appendix B.1.

6 Results

6.1 Entity Inferences

We first conduct a smaller scale study on the easier benchmark, ENTITY INFERENCES, where learning about the definition should allow us to guess the target tokens by design. Table 2 reports the results. Our distillation approach shows promising performance in two base models we test. We find that transfer sets generated from GPT-3.5 show substantially better results than transfer sets generated from the base model itself in both datasets. This sometimes even outperforms definition prepending, which might be due to GPT3.5 introducing information about the entity beyond what can be inferred from the definition sentence. Fine-tuning on the definition and transfer set using GPT-Neo does outperform distillation, at the cost of specificity. For GPT2-XL, distillation only outperforms fine-tuning on the definition sentence when using GPT3.5 as a generator model, but still shows a substantial accuracy gain using its own generated sentences (24.3%). The drop in specificity (1.6-4.2%) is substantially less severe than fine-tuning on the definition sentence. These results indicate that context distillation teaches models to make simple inferences based on injected knowledge without significantly harming the model's distribution on unrelated concepts.

Table 3: Results (perplexity) on the ECBD 2022 dataset. Our distillation approach outperforms other approaches for GPT-Neo-1.3B, GPT2-XL, and LLaMA-2-7B on target perplexity without impacting specificity, achieving a substantial fraction of the gain from prepending the definition.

6.2 ECBD

Table 3 displays our main experimental results on ECBD with three base models. Our context distillation method achieves high performance for all models. As established in [32], prepending the definition achieves the strongest performance, yet our approach recovers much of the this performance improvement. As in ENTITY INFERENCES, using a transfer set generated by GPT-3.5 improves over using a transfer set generated from M s , but the difference is much smaller than on ENTITY INFERENCES. These results suggest that our approach may benefit from, but does not require, access to a strong generator model. Fine-tuning the full model decreases the perplexity (2.5-4.0 perplexity drop) with smaller models but increases the perplexity on bigger models. We observe little change in performance with fine-tuning the last layer alone. We found that MEND increases the perplexity, and MEMIT for a single-edit decreases perplexity slightly.

As the dataset is moderately sized, we perform a paired bootstrap test to test for the significance of the improvements in average post-perplexity of distillation (using GPT-3.5 generated continuations) over finetuning all parameters on the definition, drawing N = 10000 samples [2]. The gains of distillation over fine-tuning are significant with p < 0 . 05 .

Comparing to domain adaptation: How much does the entity-specific knowledge matter? One possible explanation for our gains is that distillation teaches the model something about the particular domain of probe sentences rather than knowledge about particular entities. We discuss two pieces of evidence for why this can only explain partial gains.

Existing editing methods we test do not significantly affect specificity, while our method leads to a slight decrease in specificity (improvement on unrelated sentences). This may indicate that our model is learning the domain of Wikipedia, but the small magnitude suggests that this alone does not explain the performance gain in target probe sentences.

Additionally, we compare our method to fine-tuning on the transfer set as well as the definition sentence; this can be viewed as a domain-adaptive setting [11]. This generally harms the model's perplexity on the evaluation setting relative to fine-tuning only on the definition sentence, unlike on ENTITY INFERENCES.

Ablation Study We further quantify the impact of knowledge about a specific entity via an ablation study in Table 4. We substitute either the entity definition or the transfer set with those belonging to a different randomly sampled entity. Similar to how prepending random definitions leads to a substantial increase in perplexity (bottom of Table 3, +11.9), distilling a definition of a randomly chosen entity, even when using the correct transfer set, leads to an increase in perplexity (+2.6). This result indicates that using the correct entity definition is crucial. It also shows potential benefits of parameter update methods compared to prepending to the context, as prepending irrelevant information brings a more substantial drop in in performance.

40.0

37.5

TargetPerplexity

35.0

32.5

Distillation

30.0

Fine-tuning

Base perplexity

27.5

25.0

22.5

20.0

510

25

50

100

Numberofmodelparameterupdates

25

Perplexity on Definition

20

15

Distillation

Fine-tuning

Base perplexity

10

510

25

50

100

Numberofmodelparameterupdates

Next, we consider replacing the transfer set with a set of ten distinct elements from ten transfer sets of different entities (second row). We find that using the correct definition and a random transfer set decreases perplexity, even outperforming finetuning. Although the success of this is surprising, there is precedent for

this in distillation research in computer vision [30, 4, 24].

Furthermore, simply prepending the correct entity name (third row) in front of each element of the random transfer set decreases the perplexity substantially. This further shows that distillation is able to inject the definition even in the presence of a noisy transfer set. This also suggests distillation is mainly injecting information in the definition sentence, not the information in the transfer set.

7 Analysis

7.1 Analyzing Distillation for ECBD

Does the distillation inject the definition itself? If distillation is teaching the model to make inferences based on the definition, how well does it teach the model about the definition itself? We measure the per-token normalized perplexity on the definition sentence and report the results in Figure 2. Unsurprisingly, fine-tuning on definition sentence significantly drops its perplexity to closer to zero after 5-10 updates. While never trained to directly repeat the definition sentence, distillation also lowers the model's perplexity on the definition sentence significantly, potentially because of lexical overlap between the transfer set and the definition sentence (token overlap of 34.8-56.4% as shown in Table 1).

Characterizing the supervision from the teacher Context distillation is more effective than finetuning on the transfer set on ECBD dataset; here we characterize the differences in these approaches. Figure 3 shows the negative log likelihood (NLL) for GPT-Neo of the continuations on ECBD 2022 generated by GPT-3.5 without conditioning on the definition (x-axis) vs. the reduction in NLL when conditioning on the definition (y-axis). This is not the KL divergence and therefore not the actual training objective; however, by looking at how NLL values change, we can identify specific tokens whose probabilities are substantially modified, which would indicate a high KL value.

Tokens copied from the definition typically receive the highest decreases. Many tokens not in the definition are relatively unchanged in likelihood, and those in contexts that are not informed by the definition will have low KL divergence and drive small updates during learning. However, we show two examples of tokens not in the definition where conditioning does reduce the NLL substantially. In the first case, Dhaka is guessable given Bangladesh , and in the second, features is semantically related to the definition. By contrast, asset has similar NLL before and after conditioning.

Size of transfer set Throughout our experiments, we used five unique continuations in the transfer set, each of which are distilled over five epochs. Is having diverse continuations necessary for

Definition: The Padma Multipurpose Bridge, commonly known as the Padma Bridge, is a two-level road-rail bridge across the Padma River, the main distributary of the Ganges in Bangladesh. Continuation: Pad ma Mult ip ur pose Bridge has been a critical asset for connecting Dh aka and the western region […] word: Dh ; NLL without: 6.77; NLL with: 4.19

Definition: Android 13 is the thirteenth major release of the Android mobile operating system, developed by Google, released for the public on August 15, 2022.

Continuation:

13 released for the public on August 15 2022 brought

with it a range of new

features

[…]

word: features

; NLL without: 0.59; NLL with: 0.18

Fractional decrease in NLL conditioning on def.

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

words not in defn

words in defn

1.00

0

4

6

8

10

12

14

16

Token NLL without definition

40.0

(a) PERPLEXITY AFTER MULTIPLE EDITS

(b) SPECIFICITY AFTER MULTIPLE EDITS

30

37.5

35.0

28

Perplexity

32.5

Specificity

Context distillation (GPT2-XL)

26

30.0

Context Distillation (GPT-Neo-1.3B)

MEMIT (GPT2-XL)

27.5

24

25.0

22.5

22

Context Distillation (GPT2-XL)

-O- Context Distillation (GPT-Neo-1.3B)

20.0

--MEMIT (GPT2-XL)

10

25

50

Number of entities

75

150

20

10

25

50

Numberofentities

75

150

successful distillation? We plot the distillation performance while varying the number of unique continuations in the transfer set from 1 to 10, while also keeping the number of updates the same, in Figure B.3 in the appendix and summarize the results here. Repeating one continuation ten times yields a target perplexity of 25.3, while using ten unique continuations once yields a target perplexity of 23.5. We see diminishing returns from introducing new continuations after 5 continuations, and most of the gains can be achieved with as few as two unique generated continuations. This is in line with prior work [5] which has shown distilling on more examples improves the target performance.

Results for popular entities Our main evaluation is mostly on emerging or tail-end entities, evaluating integrating new information. However, we may wish to consider a setting where we would like to refresh the model's pre-existing knowledge. To evaluate this scenario, we use the popular split of ECBD [31]. This has an identical format to ECBD 2022, but sentences cover popular, well-known entities (such as SpaceX and Eminem) that pre-date the training of the models we test.

We report results in Table 10 in the appendix. In this setting, our distillation approach vastly outperforms fine-tuning, suggesting that distillation can be effective in resurfacing LM knowledge.

7.2 Scaling to multiple edits

Prior editing techniques [22] showed limitations in updating multiple facts at once. To evaluate how our approach scales, we perform distillation for multiple entities at once. We aggregate (entity definition, transfer sentence) for each entity and shuffle them for the entire entity set such that they are not ordered by entity during training. Figure 4 reports model performance under this setting, varying the number of entities to be updated from 10 to 150. We find that our method is largely capable of large scale updates, outperforming MEMIT, which shows increased perplexity when injecting more than 25 entities at once. For specificity, both MEMIT and distillation do not show degradation on GPT2-XL, but we observe degradation on GPT-Neo with our distillation method. Results on

ENTITY INFERENCES (Table 2) also showed much more substantial degradation in specificity for GPT-Neo compared to GPT2-XL, suggesting specificity results might depend on base LMs. Overall, we observe promising results on editing multiple entities at once with distillation.

7.3 Application to Counterfactual Knowledge Editing

Prior work [21] studied counterfactual knowledge editing, which injects false statements (such as ' The Eiffel Tower is located in Rome ') into the model. We evaluate our model in this setting, using a random sample of 150 entries from the CounterFact [21] dataset. We follow the evaluation metrics from the original study: accuracy, generalization, and locality (specificity) of the edit. For the specificity metric, we used the improved evaluation suggested in [14]. 2

Table 5 reports the experimental results. ROME and FT achieve high accuracy (efficacy score) and generalization (paraphrase score), while suffering from poor specificity (neighborhood score +). Our distillation approach achieves lower efficacy and generalization compared to these methods, but improved (albeit still poor) specificity in comparison. Additionally, we evaluate Constrained Fine-tuning (FT+L) [44], which imposes a L ∞ norm constraint on weight changes in the parameter space. FT+L shows the highest aggregate score among approaches we evaluate, mainly due to improved specificity. However, it only yields mediocre generalization.

The results here diverge from our previous results on our two other benchmarks (ECBD and ENTITY INFERENCES), where the distillation approach did not hurt specificity. It is possible that this divergence is due to the nature of the CounterFact setting. Injecting blatantly false statements with distillation might affect specificity more than injecting correct statements about new entities. Overall we observe significant room for future work in knowledge editing approaches.

8 Conclusion and Limitations

We present a distillation-based method to impart entity knowledge within the parameters of a pretrained LM. Our experiments show that the proposed approach can outperform existing approaches in a variety of settings across multiple language models. Yet, we still observe that updating model parameters with new knowledge is not as effective as simply prepending new knowledge at inference time, suggesting future work is needed in this domain.

We conclude by describing the limitations of our work. Due to computational constraints, we use models that are <10B parameters. Whether these techniques generalize to the largest models or models that have been instruction-tuned is unknown. Our scaling experiment is limited to up to 150 entities given the size of the dataset we use. Further work is needed to assess whether thousands or millions of new entities can be injected in this fashion (e.g., to teach a complete set of new entities in a domain). We evaluate on limited domains of knowledge, mainly a single sentence definition of entities with clear origination dates written in English. Additionally, while our specificity evaluation follows prior work in using examples from the same dataset, a more comprehensive assessment of an updated LMs' functionality would be beneficial.

Acknowledgments

This work was partially supported by NSF CAREER Award IIS-2145280, a grant from Open Philanthropy, a grant from Cisco Research, and support from the NSF Institute for Foundations of Machine Learning (IFML). Any opinions, findings and conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Cisco Research.

References

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A General Language Assistant as a Laboratory for Alignment. arXiv eprint arxiv:2112.00861 , 2021.

Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. An Empirical Investigation of Statistical Significance in NLP. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning . Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2012.

Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. URL http: //github.com/eleutherai/gpt-neo .

Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-Free Learning of Student Networks. In International Conference on Computer Vision (ICCV) , 2021.

Eunbi Choi, Yongrae Jo, Joel Jang, and Minjoon Seo. Prompt Injection: Parameterization of Fixed Inputs. In Findings of the Association for Computational Linguistics: ACL , 2023. URL https://arxiv.org/abs/2206.11349 .

Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. arXiv , 2023.

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. Knowledge neurons in pretrained transformers. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , abs/2104.08696, 2022.

Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing Factual Knowledge in Language Models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021.

Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. Time-Aware Language Models as Temporal Knowledge Bases. Transactions of the Association for Computational Linguistics (TACL) , 2021.

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Empirical Methods in Natural Language Processing (EMNLP) , 2021.

Suchin Gururangan, Ana Marasovi´ c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , abs/2004.10964, 2020.

Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. Methods for measuring, updating, and visualizing factual beliefs in language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages 2714-2731, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. URL https://aclanthology.org/2023. eacl-main.199 .

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv , abs/1503.02531, 2015.

Jason Hoelscher-Obermaier, Julia Persson, Esben Kran, Ioannis Konstas, and Fazl Barez. Detecting edit failures in large language models: An improved specificity benchmark. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 11548-11559, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. findings-acl.733. URL https://aclanthology.org/2023.findings-acl.733 .

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The Curious Case of Neural Text Degeneration. In Proceedings of the International Conference on Learning Representations (ICLR) , 2020.

Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. Lifelong pretraining: Continually adapting language models to emerging corpora. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 47644780, Seattle, United States, July 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.naacl-main.351 .

Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. RealTime QA: What's the Answer Right Now? arXiv eprint arxiv:2207.13332 , 2022.

Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. Mind the Gap: Assessing Temporal Generalization in Neural Language Models. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Kenton Lee, Kelvin Guu, Luheng He, Tim Dozat, and Hyung Won Chung. Neural Data Augmentation via Example Extrapolation. arXiv eprint arxiv:2102.01335 , 2021.

Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , abs/2005.11401, 2020.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual knowledge in gpt. Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. MassEditing Memory in a Transformer. In International Conference on Learning Representations (ICLR) , 2023.

Paul Micaelli and Amos Storkey. Zero-shot Knowledge Transfer via Adversarial Belief Matching. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Microsoft. Deepspeed. https://github.com/microsoft/DeepSpeed , 2023.

Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast Model Editing at Scale. In International Conference on Learning Representations (ICLR) , 2022.

Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Memory-based model editing at scale. In International Conference on Machine Learning , 2022. URL https://arxiv.org/pdf/2206.06520.pdf .

Jesse Mu, Xiang Lisa Li, and Noah D. Goodman. Learning to compress prompts with gist tokens. ArXiv , abs/2304.08467, 2023.

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. WebGPT: Browser-assisted question-answering with human feedback, 2022.

Gaurav Kumar Nayak, Konda Reddy Mopuri, and Anirban Chakraborty. Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation. In Winter Conference on Applications of Computer Vision (WACV) , 2021. URL https://arxiv.org/abs/2011.09113 .

Yasumasa Onoe, Michael Zhang, Eunsol Choi, and Greg Durrett. Entity cloze by date: What LMs know about unseen entities. In Findings of the Association for Computational Linguistics: NAACL 2022 , pages 693-702, Seattle, United States, July 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.findings-naacl.52 .

Yasumasa Onoe, Michael J.Q. Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2023.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.

Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics (TACL) , abs/2302.00083, 2023.

Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. Replug: Retrieval-augmented black-box language models. ArXiv , abs/2301.12652, 2023.

Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko. Editable Neural Networks. In International Conference on Learning Representations (ICLR) , 2020.

Charles Burton Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context. ArXiv , abs/2209.15189, 2022.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Peng Wang, Ningyu Zhang, Xin Xie, Yunzhi Yao, Bozhong Tian, Mengru Wang, Zekun Xi, Siyuan Cheng, Kangwei Liu, Guozhou Zheng, et al. Easyedit: An easy-to-use knowledge editing framework for large language models. arXiv preprint arXiv:2308.07269 , 2023.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv , abs/2201.11903, 2022.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain

Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-ofthe-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations . Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2020.

Yan Xu, Mahdi Namazifar, Devamanyu Hazarika, Aishwarya Padmakumar, Yang Liu, and Dilek Hakkani-Tür. KILM: Knowledge Injection into Encoder-Decoder Language Models. ArXiv , abs/2302.09170, 2023.

Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. arXiv preprint arXiv:2305.14795 , 2023.

Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363 , 2020.

A Datasets

A.1 Dataset Statistics

A.2 Dataset Examples

A.3 Time Ranges of the LLMs used

Table 9 shows the time ranges of the pre-training data of language model and that of dataset considered in our work. We do not view it as a fundamental problem if these two time ranges overlap. Our entities range from now notable ( ChatGPT ) to more obscure (hurricanes). Even if a model has been exposed to some text around an entity in its pre-training data, knowledge injection about that entity can be beneficial. Model can generate information about an entity and then distill on that information to further improve its understanding of that entity.

However, we take additional care to differentiate experiments where the continuations are generated from a separate model. For instance, using GPT-3.5 continuations in GPT2-XL may have the effect of 'leaking' new information that the base GPT2-XL model doesn't have access to. We control for these effects in our experimental setup. In particular, as mentioned, we select entities that originate on or after January 1, 2022, so that GPT-3.5 has not seen any of them. Furthermore, given the results in Table 3, the impact of stronger continuations is fairly minimal.

Table 9: Time cutoffs for LLMs used in this work. ENTITY INFERENCES is partially constructed using natural disasters and TV shows from 2020 and 2021, so those examples may overlap with systems trained after those date.

B Experimental Details

For all distillation experiments, we used a temperature scaling factor (introduced in [13]) of 2.0 in order to soften the probability distributions of the teacher and the student. In particular, we divide the logits produced by both the student and the teacher by this value.

B.1 Hyperparameters

To tune hyperparameters, for each experiment we tested a range of learning rates from 1e-8 to 1e-4 specifically, 1e-8, 5e-8, 1e-7, 5e-7, 1e-6, 5e-6, 1e-5, 5e-5, 1e-4. We used an iterative procedure to hone in on optimal learning rates. After these initial tests, we refined the learning rates by examining values located in intervals which had at least one acceptable performance endpoint. For example, if both learning rates of 5e-5 and 1e-4 yielded divergent results (e.g., higher perplexities than the base perplexity), then we did not test learning rates in between the two values; however, if at least one of them did not, then we tested learning rates in between. Furthermore, we tested a few different numbers of epochs (usually 5, 8, 10, 15, or 20) for each experiment, using a grid search with the selected suitable learning rates. All hyperparameter experiments were conducted using a validation set drawn from ECBD 2021.

Entity Inference Dataset For both base LMs, we used a learning rate of 5e-4 for 10 epochs for fine-tuning on the definition sentence and a learning rate of 5e-4 and 5 epochs for each of 5 sentences for distillation. For fine-tuning on the definition and transfer set, we use a smaller learning rate of 4e-5.

ECBD Dataset For GPT-Neo-1.3B and GPT2-XL, we trained for 5 epochs with a learning rate of 3e-6 for fine-tuning. For fine-tuning on the definition and transfer set, we found that 5 epochs and a smaller learning rate of 6e-7 yielded the best results. For context distillation, a learning rate of 3e-6 yielded the best performance. For all distillation experiments involving GPT2-XL and GPT-Neo-1.3B, we perform distillation training for 5 epochs on each of 5 generated continuations.

For LLaMA-2-7B, we found that a learning rate of 5e-6 yielded best performance for both finetuning on the definition sentence and distillation. For finetuning on the definition and transfer set, we use a smaller learning rate of 8e-7. We finetune for 5 epochs for both finetuning on the definition sentence and finetuning on the transfer set. For distillation with LLaMA-2-7B, we train for 3 epochs on each of 5 generated continuations.

B.2 Compute

All experiments were run using Quadro RTX 8000 GPUs with 48GB RAM. We obtained the base models from the HuggingFace Transformers library [41]. All experiments for GPT-Neo and GPT2-XL required less than 4 GPU hours each, and experiments for LLaMA-2-7B required up to 30 GPU hours. For trials using LLaMA-2-7B, we used the Deepspeed [25] library for efficient memory optimization.

B.3 Additional Results: Diversity of Transfer Set

B.4 Additional Results: Results on ECBD Popular Set

B.5 Experimental Details on CounterFact Evaluation

CounterFact [21] consists of edit statements formatted as subject-relation-object triplets . For a given factual triplet ( s, r, o ), the goal is to edit the counterfactual triplet ( s, r, o ∗ ) into the model, where o ∗ is a counterfactual object. For example, given the fact "The Eiffel Tower is in Paris", the subject s = 'Eiffel Tower', the relation r = 'is in', and the object o = 'Paris'. One counterfactual edit might be 'The Eiffel Tower is in Rome'; here, o ∗ = 'Rome'.

The Efficacy score measures the percentage of instances where P ( o ∗ ) > P ( o ) post-edit, when given the prompt s + r ('The Eiffel Tower is in'). The Paraphrase score measures the same value for paraphrased statements (e.g., 'The location of the Eiffel Tower is'). The Neighborhood score measures the percentage of instances where P ( o ) > P ( o ∗ ) for unrelated entities. For example, after editing the new location of the 'Eiffel Tower' we might want to check that 'Louvre' is still in Paris, and not Rome. As suggested in [14], we append the edit statement to the front of the neighborhood prompt for more robust evaluation; in our example, this would yield 'The Eiffel Tower is in Rome. The Louvre is in'.

C Transfer Set Generation

To generate transfer set sentences using GPT-3.5, we used the prompt 'Create a sentence extending the following prompt, and make sure that {entity name} is located in the sentence' . We experimented with a few similar prompts, but found very little variance in terms of results.

For smaller base models, we generated continuations of the definition by sampling from the model with entity definition as a prefix. Afterwards, we appended the entity name to the front of each sentence if the generated continuation did not include entity name. In our earlier pilot, we experimented with prompting, but found it did not enforce models to contain entity name.

Table 11 contains example continuations generated by each base model.

Peer reviews:
comments: SUMMARY:
This paper tackles updating the knowledge in LMs, focusing on allowing LMs to make new inferences consistent with the updated facts. To do this, the authors propose using the LM itself (or a teacher) to generate natural continuations for the "updated/new entity" definition. These continuations are used to update the LM. The update is conducted using a KL divergence loss between the LM conditioned on the definition and the LM that doesn't see the definition. The results show superiority to baselines in updates and in preserving old knowledge.

STRENGTHS:
The paper seems like an excellent contribution. It's well motivated, well presented, and the key idea is simple, novel, and effective. The evaluation is convincing.

WEAKNESSES:
The method, like many others, is relatively opaque in terms of what it teaches the models and why/how it works precisely. However, it's well motivated and the analysis in Sec 7 begins to shed a little bit of light into this. More work is needed on that front, but I think it's fair to assume this will lie beyond the scope of this paper.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of injecting new entity knowledge in LLMs, such that these knowledge can be propagated and utilized when LLMs make inference on related queries. The paper proposes a context distillation method that consists of two steps to inject entity knowledge in a definition sentence: 1) Use a LLM to generate a set of continuations (a.k.a transfer set) for the definition sentence. 2) Fine-tune a student model such that its output distribution without conditioning on the definition sentence is close to the output distribution of a teacher model that conditions on the definition sentence.\nThey conduct experiments on two datasets about entity knowledge and show that the proposed method outperforms several baselines including standard fine-tuning and previous knowledge editing methods.\n\nSTRENGTHS:\n1. This paper studies an important question of knowledge injection and propagation of injected knowledge. The proposed method is novel in this context.\n2. Some of the conducted analyses are insightful, such as the NLL with/without definition sentence for analyzing the supervision from the teacher model.\n\nWEAKNESSES:\n1. On Entity Inferences dataset, the conclusion that the proposed method improves the model ability to make inference using the injected knowledge is suspected. The reported performance improvement might due to the overlap between the generated transfer set and the probe sentence in the evaluation set. Without reporting (1) the level of overlap, and (2) a baseline that simply fine-tunes on the transfer set, the possibility of this overlap cannot be ruled out.\n2. How does the method perform compared to a baseline that simply prepends the transfer set to the query?\n\nQUESTIONS:\n1. In Table 2, the Target for GPT2-XL should be 64.3 instead of 65.3 (based on the $\\Delta$ value)?\n2. In Table 2, why would using GPT3.5 to generate transfer set result in worse specificity for GPT2-XL?\n3. I'm not sure why most of the analyses are done on the ECBD dataset, as I thought Entity Inferences dataset concerns more about injected knowledge propagation.\n\nLIMITATIONS:\nLimitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to propagate knowledge update to LMs via training a student model through context distillation, such that the LM can make inference on an entity even though the relative context/knowledge of the entity is not given. The framework involves two steps: 1) create a transfer set that contains the knowledge that the student model will be learning from; 2) compute the distribution of the transfer set tokens for both the teacher model (while given context, i.e. a definitional sentence) and the student model and update the student model's parameters by minimizing the KL divergence of the two distributions. The paper evaluates the student model with two sets, Entity Inferences and ECBD to show that the knowledge has successfully propagated.\n\nSTRENGTHS:\nThis paper is more efficient with multi-entity editing and achieves competitive performance on the two evaluation set in terms of propagation success (accuracy and decrease in perplexity) while causing little impact on specificity.\n\nWEAKNESSES:\nIt seems like the paper is more focusing on new knowledge ingestion, either in Entity Inference (synthetic entities) or ECBD (introducing new entities after 2022). While this is an important aspect, a harder task is to update existing knowledge in the old model. One dimension could be temporal shifts, e.g. after a new election, population/economic changes (potentially resulting changes in superlative statements), factual changing official announcement (e.g. solar system has 9 planets before 2006 and Pluto was downgraded to dwarf planet in 2006 - solar system has 8 planets now). It is unclear whether the model can adapt to the new facts while maintain low specificity.\n\nAnother baseline is to try prompting the LLMs with new knowledge and see how it propagates. If the existing LLMs can handle such knowledge updates well, it may be hard to justify why we need to train a separate student model.\n\nQUESTIONS:\nOn line 126, it states the distillation is done through updating $M_s$ parameters to minimize the KL divergence. Does it update all the parameters in $M_s$, or is it possible to combine the distillation with other network editing techniques to only a local set of parameters? How much would it negatively impact the performance if only a local edit is allowed? Asking since if we want to extend this framework to larger LLMs (as current good-quality LLMs usually have 100B+ parameters and updating all parameters seem to be impossible).\n\nLIMITATIONS:\nAs mentioned by the authors, this work mainly uses relatively small size LMs for experiments and its generalizability to LLMs is unknown. While it may apply to LLM trainers/creators to adapt this method to update their models, it does not extend to end users/organizations of the LLMs who want to ingest or update knowledge, e.g. from specific domains or confidential sources, potentially through local edits.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper propose a context distillation-based approach that can both impart knowledge about entities and propagate that knowledge to enable broader inferences. This approach consists of two stages: transfer set generation and distillation on the transfer set. In the first stage, a transfer set is generated by prompting a language model to generate a continuation from the entity definition. In the second stage, the model parameters are updated so that the distribution of the LM (the student) matches the distribution of the LM conditioned on the definition (the teacher) on the transfer set.\nThe authors' experiments demonstrate that this approach is more effective in propagating knowledge updates compared to fine-tuning and other gradient-based knowledge-editing methods without compromising performance in other contexts, even when injecting the definitions of up to 150 entities at once.\n\nSTRENGTHS:\n1. A straightforward motivation that conditioning on information about the entity can lead to lower perplexities.\n\n2. The authors' method of generating a transfer set by prompting an LM to generate a continuation from the entity definition is a unique contribution to the field.\n\n3. The authors compare their method with other knowledge injection methods, including fine-tuning, and demonstrate the superiority of their approach. They also conduct an in-depth analysis of the types of continuations needed in the transfer set.\n\n4. The authors' method provides a scalable and effective way to update the knowledge of LMs.\n\nWEAKNESSES:\n1. As the authors concede in Section'Limitations', their proposed methodology has yet to be substantiated on models of a larger scale. For instance, LLaMA-65B may present a fitting candidate for such validation.\n\n2. The experiments in the paper focus on a specific type of knowledge update: adding definitions for entities. It's unclear how well this method would work for other types of knowledge updates, such as knowledge revision.\n\n3. The results of Finetuning on transfer set (full) are not shown in Table 2.\n\n4. Writing content issues.\n  (1) It would be clearer to add arrows in the table to show whether the larger or smaller values are better.\n  (2) What are Finetuning on definition (full) and Finetuning on definition (last only)?\n\nQUESTIONS:\nWhat are Finetuning on definition (full) and Finetuning on definition (last only)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}