Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge
Year: 2023

Abstract:
Pre-trained language models (LMs) are used for knowledge intensive tasks like question answering, but their knowledge gets continuously outdated as the world changes. Prior work has studied targeted updates to LMs, injecting individual facts and evaluating whether the model learns these facts while not changing predictions on other contexts. We take a step forward and study LMs’ abilities to make inferences based on injected facts (or propagate those facts): for example, after learning that something is a TV show, does an LM predict that you can watch it? We study this with two clozestyle tasks: an existing dataset of real-world sentences about novel entities (ECBD) as well as a new controlled benchmark with manually designed templates requiring varying levels of inference about injected knowledge. Surprisingly, we find that existing methods for updating knowledge (gradient-based fine-tuning and modifications of this approach) show little propagation of injected knowledge. These methods improve performance on cloze instances only when there is lexical overlap between injected facts and target inferences. Yet, prepending entity definitions in an LM’s context improves performance across all settings, suggesting that there is substantial headroom for parameterupdating approaches for knowledge injection.

heading: 1 Introduction
text: Pre-trained language models (LMs) acquire comprehensive real-world knowledge from massive amounts of pre-training data, allowing them to use this knowledge effectively in downstream tasks. However, without continual updating, the knowledge contained within these backend LMs will eventually become outdated. This temporal mismatch affects model performance on downstream tasks (Zhang and Choi, 2021; Dhingra et al., 2022a; Lazaridou et al., 2021; Jang et al., 2022b). As LMs become more widely deployed, their knowledge
should be synced with the current state of the world while maintaining reasonable deployment costs.
Prior work has investigated knowledge editing in pre-trained LMs, updating model parameters to alter outputs to match what users want (Zhu et al., 2020; Sinitsin et al., 2020; De Cao et al., 2021; Mitchell et al., 2022; Meng et al., 2022; Hase et al., 2023). In these studies, the original fact and the altered fact are provided (e.g., changing “X was born in Y.” to “X was born in Z.”), and models are evaluated after a single update on each instance; see Figure 1 for an example. These model editing methods successfully provide targeted updates, fixing incorrect or outdated individual facts. Yet, can LMs make inferences based on updated knowledge? Past evaluation has largely focused on two aspects of knowledge editing, whether the edits were successfully injected and whether other irrelevant sentences were impacted, but do not capture
5469
whether the LMs now can reason based on the new fact that has been injected.
We take a step further and evaluate whether LMs can propagate updated knowledge about new entities. We first inject definitions about the entity into LMs using various knowledge editing methods (Mitchell et al., 2022; Meng et al., 2022), then evaluate LMs’ performance on cloze tasks on a wide range of sentences about the entity (see Figure 1 for an example). We refer to this task as entity knowledge propagation and introduce two cloze datasets to evaluate this challenging task.
Our first evaluation benchmark is the Entity Cloze By Date (ECBD) dataset (Onoe et al., 2022), which presents novel entities tagged with origination dates (e.g., Hurricane Ian, 2022), their definition and probe sentences taken from their Wikipedia page. The task is to fill a masked span in probe sentences. Because Wikipedia contains a wide range of information, much of it not inferable from an entity’s definition, injecting entity knowledge via its definition has an unclear impact on the probe sentences; filling in the masked span is nontrivial even after the entity definition is provided. For more controlled study, we introduce a new benchmark (ENTITY INFERENCES) with manually designed probe sentences with multiplechoice answer options. Once one learns about the definition of an emerging entity, finding the correct answer for these probe sentences is easy.
We find that existing parameter updating methods can handle simpler inferences in ENTITY INFERENCES, but fail to improve performances in ECBD, revealing a limitation in these methods. We further analyze the impact of fine-tuning. Distressingly, we find that simply prepending information in-context works very well, and matching the performance of this via parameter updates is challenging. A deeper analysis finds that model editing shows promising results only when the injected definition sentence and the cloze inference have lexical overlap. Our work establishes an evaluation paradigm and opens doors for work on editing methods that can propagate entity knowledge. The code and data are available at https://github.com/yasumasaonoe/ entity_knowledge_propagation.

heading: 2 Entity Knowledge Propagation
text: We propose Entity Knowledge Propagation (EKP), a new task where we want to update model param-
eters to reflect an emerging entity that is unseen in the LMs’ pre-training corpus. For example, BERT was trained in 2018, so COVID-19 is an emerging entity to BERT. We explore various ways of editing model parameters based on definition sentences to inject new knowledge. Once we inject the knowledge of the emerging entity into the model parameters, we evaluate the updated model’s ability to reason about the emerging entity.

heading: 2.1 Task Definition
text: Formally, we have a language model fθ with parameters θ. An input to the model consists of a (partial) sentence or chunk of text xe that contains at least one explicit reference to an emerging entity e (i.e., invoking e by name). We use fθ(ye ∣ xe) to denote placing a probability distribution over a text sequence ye given the text xe. 1
Our data instances have the property that ye represents an inference we make about the entity: ye must be related to the entity e such that an LM should give higher probability to it if the LM “knows” e well. We do not expect the raw model fθ to perform well without any updates, since the entity e is completely unseen during the pre-training stage. We assume that the emerging entity comes with a short definition sentence de that provides basic information about the entity. This provides the basis for the update to fθ.
To summarize, each example ⟨e, de, xe, ye⟩ ∈ D consists of an emerging entity e, a definition sentence de, a probe sentence xe, and a gold completion ye. Knowledge editing methods will compute θ′ ← update(θ, e, de), updating parameters θ regarding e and its definition de, to give higher probability for future inferences about e like those expressed by xe and ye (examples in Figure 1).
Metrics Following prior work in the knowledge updating literature (Zhu et al., 2020; De Cao et al., 2021; Mitchell et al., 2022; Meng et al., 2022; Hase et al., 2023), we will evaluate two criteria: update success and specificity. Each of these criteria is evaluated with respect to a base metric, which is either perplexity or accuracy, depending on our dataset. We will define them here in the case of perplexity (lower is better); we will use the same definitions for accuracy, but the desired trends will be opposite.
1In autoregressive models, ye can be a continuation of xe; in mask filling models like T5 or BART, xe can contain mask tokens and ye consists of those mask fillers.
For update success, we will measure if the perplexity of the updated model ppl(fθ′(ye ∣ xe)) is better than the raw model ppl(fθ(ye ∣ xe)) (lower perplexity is better). For specificity, we compute the difference between the post-update perplexity and pre-update perplexity ppl(fθ′(yê ∣ xê)) − ppl(f(yê ∣ xê)) for ê ≠ e, entities other than e. Ideally, we want this perplexity value to be close to zero; a positive value indicates that perplexity has gotten worse on these entities after the update. It can theoretically be negative if the update makes the LM to guess irrelevant examples better.
Comparison with prior tasks Similar editing procedures have been explored in the literature, but with key differences from our setting. A line of work on knowledge editing (Zhu et al., 2020; De Cao et al., 2021; Mitchell et al., 2022) addresses a version of our task where fθ is updated to encode information about a particular fact. This could be written as θ′ ← update(θ, xe, y). They then evaluate fθ(y ∣ x̃e) on perturbed inputs x̃e that are paraphrases of the xe they inject. The answer y is visible when the network is updated and it simply needs to be preserved for future (paraphrased) queries. By contrast, in our setting, y and the injected definition de may have little overlap.
ROME (Meng et al., 2022) addresses knowledge editing as well as a variant of counterfactual model editing. This task involves an update similar in spirit to ours: θ′ ← update(θ, e, (xe,1, ye,1)) that updates a completion of a sentence (e.g., xe,1 =the Eiffel Tower is located in, ye,1 =Rome) and then expects the knowledge to be usable for other inference pairs (xe,2, ye,2). These differ in that the injected knowledge is not a complete definition of an entity; while their method could theoretically be used for our task, it relies on localizing and editing existing information about e. Therefore,
it is less appropriate in handling emerging entities, as our results will show.

heading: 3 Constructing benchmarks for EKP
text: We use two different types of datasets to investigate how new entity knowledge is propagated into the LM’s parameter space. Table 2 summarizes the dataset statistics on two benchmarks, including the extent to which the target spans y overlap with the definitions de, which will be important later.

heading: 3.1 ECBD
text: Entity Cloze By Date (Onoe et al., 2022, ECBD) presents entities indexed by their origination dates paired with probe sentences containing those entities. In addition, the dataset provides the definition sentence (first sentence sentence of Wikipedia article) for each entity. The original task focuses on general temporal adaptation of language models, evaluating model’s perplexity in predicting masked spans in probe sentences. We repurpose this dataset to focus on targeted knowledge updates and the propagation of entity knowledge. We take entities with origination date between 2020/01 and 2021/09 to ensure they are unseen by the LMs we study.
These instances fall into the paradigm discussed in Section 2.1 (example shown in Table 1):
e ∶ Entity : the title of the Wikipedia article
de ∶ DefinitionSentence : the first sentence of the Wikipedia article for the entity. xe ∶ ProbeSentence : a sentence selected from the Wikipedia article according to the procedure described in Onoe et al. (2022) y ∶ GoldSpan : the target span as described in Onoe et al. (2022)
ECBD-EASY We filter ECBD to create ECBDeasy, a subset where knowledge propagation should be easier. Specifically, we take cases where the target masked span y is contained in the definition sentence de verbatim; such examples are more congruent with the formulation of past work such as MEND and are typically easier, as simply boosting the probability of the definition tokens can improve perplexity on the gold span.
Evaluation Metrics Following Onoe et al. (2022), we compute per-token perplexity over the masked spans. Because of differences in model architecture such as tokenizer choice, this metric does not allow comparison across different base models. We randomly sample 40 entities as ê from ECBD popular subset to measure specificity.

heading: 3.2 ENTITY INFERENCES
text: While ECBD contains real-world sentences spanning a broad domain, it presents a very challenging task even for humans, often requiring rich knowledge and various types of reasoning and inference. For a more controlled study targeting on knowledge propagation, we construct a new dataset we name as ENTITY INFERENCES.
In this dataset, choosing the correct span is much easier when given the definition sentence. Further, instead of requiring LMs to predict spans from open vocabulary, we provide a set of candidate spans and evaluate whether LMs can assign higher probability to the correct answer candidate. Instances here are designed to be similar to ECBD, but the probe sentences xe are handcrafted to elicit the target inference type, and the gold span y comes with an associated set {Cy} of options. Data Construction Details We first curate entities tagged with TV shows and natural disasters from English Wikipedia and their definition sentences from the 2020 and 2021 subsets of ECBD. In addition to real entities, we generate examples of "fake" people where we fabricate person names along with their definitions (e.g., Leighanna Smith
(born July 21, 1970) is an American film director, screenwriter, and producer...).
We then manually craft probe sentences targeting two types of reasoning: explicit and implicit. The explicit probe sentences ask information that is explicitly stated in the definition sentence (e.g., genre of a TV show). On the other hand, the implicit probe sentences require commonsense-like information (e.g., people watch a TV show, rather than eat a TV show.).
Evaluation metrics For this multiple-choice cloze task, we evaluate knowledge propagation by meausring accuracy (i.e., how often the gold label gets the highest probability over all answer candidates). In addition, we compute the specificity score by evaluating a model on other probe sentences from similar entities.

heading: 4 Experimental Setup

heading: 4.1 Base Language Models
text: Model architectures can have impact on their capabilities of acquiring entity knowledge. Thus, we consider both left-to-right and seq-to-seq model architectures. Specifically, we use GPT-Neo 1.3B (Black et al., 2021)2 and T5-large (Raffel et al., 2020)3 as base language models (fθ), available via Huggingface Transformers (Wolf et al., 2020). We additionally consider GPT2-XL (Radford et al., 2019) as a base model to closely follow the protocol presented in ROME paper (Meng et al., 2022).

heading: 4.2 Parameter Updating Methods
text: Finetuning is a common way for adapting a pretrained LM to a specific task or domain (Gururangan et al., 2020). In a similar vein, we aim to adopt a pretrained LM to an environment where new entities constantly arise. Given e and its definition de, we update the parameters θ to minimize loss on a training example formed from de. For left-to-right models (e.g., GPT-Neo), we use the standard next token prediction language modeling task on the entire de example. For mask filling models (T5), we randomly select a span4 that is not overlapping with the entity mention span, following Onoe et al.
2GPT-Neo has been trained on the Pile dataset (Gao et al., 2020), which is collected in 2020.
3T5 has been trained in 2019 on the C4 dataset. 4We uniformly draw span length between 1 and 5 to match
the average span length used during pretraining of T5.
(2022). We experiment with two fine-tuning settings: full model (updating all parameters) and last layer (updating parameters belonging to the last transformer layer only). We start finetuning from the original model checkpoint for each example.5
MEND (Mitchell et al., 2022) can be viewed as a hypernetwork that efficiently transforms the raw finetuning gradient into a parameter update that should successfully edit the base model’s parameters in one step. This method is designed for injecting or editing individual facts about entities, not a collections of facts about entities (i.e., a complete definition’s worth of entity knowledge). The MEND parameters are trained on an editing dataset where each example consists of an input-output pair, an altered output, and locality examples (for measuring sensitivity). The goal of MEND training is to learn a network that modifies the target fact without affecting unmodified facts.
We train MEND editors for GPT-Neo and T5 with the WikiText-103 dataset, which uses generated text as altered output following the configuration used in the original paper.6
ROME (Meng et al., 2022) performs knowledge editing by treating a MLP as a key-value storage: it uses a subject (such as the Eiffel Tower) to extract the “value” associated with that subject in the MLP. Then, it uses a rank-one modification of the weights of the MLP to “rewrite” this key-value pair.
We use the ROME editor for GPT2-XL. We format according to the subject, relation, and object structure of ROME prompts; examples of these can be found in the Appendix. The subject is a one-word name of the entity, the relation is the definition sentence before the <MASK> token, and the object is the correct label. Examples in which the subject did not appear before the <MASK> token (less than 0.5% of our data) were filtered.7

heading: 4.3 Input Augmentation
text: Finally, as was explored in Onoe et al. (2022), we evaluate an approach where information is added only in-context: prepending a definition sentence
5Zhu et al. (2020) reports that finetuning on an individual fact (constantly outperforms finetuning with mixed facts.
6 https://github.com/eric-mitchell/mend
7Additionally, a number of examples in ECBD had a format incompatible with ROME; for example, ROME is currently unable to run on examples with special tokens such as ’(’ or ’*’ immediately surrounding the subject. We will discuss later the performance of ROME on ECBD and why we do not formally report it.
to a probe sentence (Definition). While such input augmentation will lower the efficiency (as the context length has increased) and will not yield an updated model, a lower perplexity can indicate if the definition sentence contains useful information and can show what gains are achievable. We also present a baseline that prepends a randomly chosen definition of another entity (Random Def.), following prior work.

heading: 4.4 Computational Cost
text: While input augmentation is the simplest to implement out of all the knowledge injection methods we experiment with, it comes with an increased computational cost at inference time due to the longer input sequence. A principal goal of this line of work is to update models so they can learn about many new entities over time; therefore, we do not consider input augmentation a valid solution to the overall problem in this work due to poor scaling.
In contrast, performing knowledge injection via finetuning carries the upfront cost of computing and performing gradient updates, but has no such cost increase during inference. Computing these gradient updates, however, can become quite burdensome when injecting many facts into the same LM. This, in part, is the motivation behind methods like MEND which have an additional upfront cost of training a meta-network to predict the necessary parameter updates. After training the metanetwork, the amortized cost of updating many individual facts becomes much cheaper. While this dramatically reduces the cost of performing multiple edits to a single LM, meta-networks must be retrained for each unique LM we wish to update.
In our experiments, the updates for an example from all of the methods take less than 10 seconds on a single Quadro RTX 8000 GPU.

heading: 5 Results
text: Table 3 reports the performances of various knowledge injection approaches on three base models. In all experimental setting, we see input augmentation (prepending definition) boasts robust and consistent performances gains. Prepending random definitions hurt performances in GPT-Neo while does not impact T5. This indicates that the definition contains information relevant to the spans to predict. As model behaves substantially differently across datasets, we first separately discuss the results on each dataset, ENTITY INFERENCES,
ECBD, and ECBD-EASY, and then draw larger conclusions.

heading: 5.1 ENTITY INFERENCES
text: Here, we observe fine-tuning is broadly effective at improving accuracy. Finetuning (full model) brings up the post-edit accuracy by more than 20 points for all three base models. Yet, it comes at the cost of medium to large decreases in specificity, with drops of 15.9 and 7.7 points on GPT-Neo and GPT2-XL. MEND overall does not cause a substantial change in the model, as shown by the impact on specificity (+0.3). ROME does not achieve editing performance as strong as fine-tuning on GPT2-XL (+31.8 vs. +23.5), but it does so with a lower impact to specificity (-8.8 vs. -2.0).
On this benchmark, where evaluation metric is accuracy, we can make comparison across the models. Overall we see better performances with T5 model, despite it being the smallest model we test, potentially as it uses both left and right context.

heading: 5.2 ECBD
text: On our most challenging benchmark setting, ECBD, none of the model editing techniques, including fine-tuning, lead to substantial decrease
in perplexity nor increase in specificity. MEND even causes a increase in perplexity when the base model is GPT-Neo.
We attempted to evaluate ROME in this setting. However, we found very poor performance (perplexities of over 100 for both datasets). We do not report these in the table as technically ECBD is out of scope for ROME: ROME relies on a particular (entity, relation, object) format that is not well-suited to updating a model with general definitional knowledge of an entity, as opposed to specific attributes like The English Game is a drama in ENTITYINFERENCES. Attempting to force our definitions into ROME’s expected form led to very high perplexities (over 100 on both ECBD sets).
These observation implies that the current model editing approaches are not able to propagate entity knowledge to the probe sentences just from the definition sentences. The inference patterns in the ECBD examples might be too complex to be effectively learned by a small number of parameter updates on a few examples, requiring implicit, multihop, and commonsense reasoning.

heading: 5.3 ECBD-EASY
text: To understand the low performance in the ECBD setting, we look more closely into ECBD-EASY examples, where the gold spans are always included in the definition sentences. On this subset of ECBD, finetuning and MEND is effective on GPTNeo, decreasing perplexity by 9.0 and 8.5 respectively. T5-large does not change its post perplexity. This is potentially because T5 only predicts and updates on masked spans (which might not contain the gold span), unlike the other two base models.
Mildly positive results on the easier subset, along with robust performances of input augmentations, lead us to conclude that the gains are achievable. Yet, existing knowledge editing techniques may be restricted to reproducing the knowledge directly injected into the model. We launch a further investigation into what makes this task challenging.

heading: 6 Analysis
text: We analyze the challenges in knowledge propagation by first estimating an informal upper bound of model editing performance (Section 6.1). We then examine how the similarity between the definition sentence and probe sentence impacts the performance of model editing (Section 6.2), inspired by positive performances on ECBD-EASY subset. We conduct our analysis with GPT-Neo base model on random subsets of ENTITY INFERENCES (half the data) and ECBD (100 NP span and 100 random span) to reduce computational costs.

heading: 6.1 Targeted Update / Specificity Tradeoff
text: Performance Upper Bound We estimate a performance upper bound for fine-tuning by setting the definition and probe sentences to be identical. In this case, sufficiently large gradient updates should lead to arbitrarily good performance from
fine-tuning. We call this setting Train-on-Test. For our three datasets (ENTITY INFERENCES, ECBD, and ECBD-EASY), we finetune a model for a range of 1 to 8 epochs (i.e., the number of updates). We use a learning rate of 5e-5 for ENTITY INFERENCES and plot the specificity score vs accuracy. For ECBD and ECBD-EASY, we choose a learning rate of 3e-5 and then compare the specificity score and perplexity. These learning rates were chosen to optimize performance from the range of values described in Appendix A.3.
Findings Figure 2a depicts the perplexity– specificity tradeoff curves of fine-tuning approach on ECBD dataset. The perplexity and the specificity score by the base model are drawn as the horizontal dotted line and the vertical dotted line respectively. Ideally, we want a model to achieve low perplexity and the specificity score identical to the base score (performance in the lower left corner). On ECBD, we see that Standard FT shows an upward trend: with larger parameter updates, we worsen the specificity as expected, but also perplexity, meaning that finetuning for longer does not usefully propagate entity information from the definition sentence into the model. Input augmentation (Prepend-Def) performs robustly, indicating that the issue is potentially due to how the data is used in learning rather than the data itself.
How does this align with past results? ECBDEASY (Figure 2b) shows a much more optimistic picture; recall that this is similar to the setting from Mitchell et al. (2022). In this case, MEND and finetuning both achieve results reasonably close to the train-on-test upper bound, with configurations that improve perplexity substantially with only mild specificity degradation. Methods that succeed on injection of exact facts (e.g., injecting y and reproducing it later) do not necessarily transfer
to success in realistic knowledge propagation settings like ECBD.
Finally, we plot the accuracy–specificity tradeoff curves computed on ENTITY INFERENCES (Figure 2c). Table 2 shows that the definition sentences of this dataset may contain the gold spans of the probe sentences but not always, making it between ECBD and ECBD-EASY in this regard. Specificity numbers are less monotonic here than on ECBD, but we again see the trend of train-on-test quickly saturating accuracy. Like ECBD-EASY, fine-tuning can lead to improvements on accuracy, in this case matching the performance of PrependDef. However, there remains a substantial gap with the gold setting, implying that there are a certain number of examples that are not easily learnable by the current data setup.

heading: 6.2 Information Overlap
text: Lexical overlap We now examine the importance of overlap between the definition and the target span more closely. First, we look at instance-level behavior on our datasets stratified by whether the gold span is included in the definition or not. We select 92 such “Included” examples in ENTITY INFERENCES and 152 from ECBD-EASY and an-
alyze the delta in the rank of the gold label and percent change in perplexity respectively.
Figure 3a shows violin plots of the performance gaps within the two groups. In both datasets, the performance improves on average (plot mean below 0) when the gold spans are included in the definition sentences, suggesting that the lexical overlap between the definition and probe sentences correlates with the model performance. This trend on ECBD is even stronger with input augmentation (Figure 3b). However, the majority of ECBD probe sentences fall into the Not Included category, and we see here that very few examples in this category have substantial perplexity improvements, most having small changes around zero.
ENTITY INFERENCES shows a slightly more optimistic picture for Not Included cases.
Soft overlap Although direct inclusion of the answer span is clearly valuable, do we see any improvements when there is soft overlap between the definition and target span; that is, the content may be similar even if not exactly the same?
We investigate the information overlap using
both lexical (e.g., Jaccard similarity, Rouge) and semantic (e.g, BERTScore (Zhang et al., 2020)) similarity measurements between the probe sentence and the definition sentence. For each dataset, we divide the examples into bins based on the similarity scores and report the performance differences between the base model and the fine-tuned model per bin (change in rank of the gold answer on ENTITY INFERENCES and perplexity change on ECBD).
Figure 4 shows violin plots of the performance gaps within each bin constructed using Jaccard similarity (a larger value mean the definition and probe sentences are similar). For ENTITY INFERENCES, we observe that the bins with larger similarity scores have progressively more negative ∆ in rank. Surprisingly, we do not see a similar trend for ECBD. Not only is it the case that there are fewer examples in ECBD exhibiting high overlap, but among the distribution of examples that is present, there is almost no perceptible correlation between the amount of overlap and the percentage change in perplexity. This suggests that not only is the data distribution in ECBD different, but the nature of the inferences themselves can be qualitatively different and more challenging. We believe this further underscores that new techniques are needed to handle knowledge propagation in the real world.

heading: 7 Related Work
text: Knowledge Editing Recent work in knowledge editing (De Cao et al., 2021; Mitchell et al., 2022; Hase et al., 2023) explored performing minimal edits to a base LM’s parameters to reflect a fact that has changed or corrected. Edited facts are usually evaluated in terms of reliability/efficacy (i.e., edit success rate), generalization (i.e., performance on paraphrased edit sentences) and locality/specificity (i.e., performance on unrelated samples should not change after editing) (Zhu et al., 2020; Sinitsin et al., 2020). Some such works have attempted to perform such edits by identifying a small, localized set of weights that are responsible for reflecting the memorized fact (Geva et al., 2021) and editing only that small set of parameters (Meng et al., 2022; Dai et al., 2021). Our work, however, focuses on injecting in knowledge about new entities, which may not already have a localized set of parameters governing such information.
Keeping Language Models Up to Date One line of recent work have explored the development and evaluation of language models that are
updated over time (Jang et al., 2022a). While ECBD (Onoe et al., 2022) focuses solely on evaluating knowledge of new entities, several benchmarks have been proposed for evaluating facts about existing entities that have changed over time as open-retrieval (Zhang and Choi, 2021) or clozestyle (Dhingra et al., 2022b) question answering. Other work has found success in keeping LMs upto-date by continuing pretraining (Jin et al., 2022) and applying domain adaptation techniques (Jang et al., 2022c). Beyond these and the editing approaches we have discussed previously, a line of work has looked at identifying a small, localized set of weights that are responsible for reflecting the memorized fact (Geva et al., 2021) and editing only that small set of parameters (Meng et al., 2022; Dai et al., 2021). Finally, Choi et al. (2022) also contrast prepending information with fine-tuning and find that fine-tuning generally works worse, framing their approach as distillation.
Content Transfer and Knowledge Acquisition Hase et al. (2023) report that edit performance and consistency are improved after updating a model in the standard knowledge editing task, which the goal is to alter the model’s predictions according to user specifications. The tasks and setting we explore in our work are closely related to that of West et al. (2022), which explores whether LMs can generate statements about an entity that are consistent with a provided description of that entity. However, they do not explore updating model parameters from these descriptions. Kandpal et al. (2022) explore knowledge acquisition in LMs, and arrives at a similar finding that LMs generally fail to answer questions about entities that occur infrequently during pretraining.

heading: 8 Conclusion
text: In this work, we explored the entity knowledge propagation setting: to what extent can descriptions of new entities be injected into language models? We find that while fine-tuning models or using efficient update strategies enables models to reproduce exact facts from descriptions, performing inferences based on those facts is substantially harder. We characterize several approaches on two datasets and conclude that update strategies lag the performance of simply prepending the definition in the context, suggesting that more work is needed.
Limitations
Entity knowledge propagation focuses on updating LMs’ knowledge about emerging entities. However, there might be cases where knowledge about existing entities needs to be updated (e.g., regime change, new champion, and renaming etc.). We intentionally exclude these cases since they can easily become intractable due to their complexity. For example, an organization changing its name could theoretically reflect a large number of entities that have relations to that organization. By investigating model behavior when a LM encounters new information which is completely unseen during pretraining, we can experiment in a controlled environment. We find ample challenges unaddressed by current research even in this setting.
Our experiments are conducted on English language models only. While we believe the results can generalize to multilingual models, it is conceivable that the internal representations of these models make them more or less amenable to the sorts of updating explored here. More work is needed to benchmark these techniques in broader settings such as with larger language models and newer parameter-tuning approaches.

heading: Acknowledgments
text: This work was partially supported by NSF Grant IIS-1814522, NSF CAREER Award IIS-2145280, a grant from Open Philanthropy, UT Machine Learning Lab and by the Air Force Research Laboratory (AFRL), DARPA for the KAIROS program under agreement number FA8750-19-2-1003. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

heading: A Appendix
text: A.1 Licensing
T5 is released under the Apache v2.0 license. GPT2 and GPT-Neo is released under the MIT license. Wikipedia and ECBD are both licensed under CC BY-SA.
A.2 Harmful Data Instances
In creating our dataset of entity inferences, we, the authors, inspect and only create examples that do not contain offensive or harmful content. All other data used is publically availible from Wikipedia. Experiments and data are all in English.
A.3 Modeling Details
The main hyperparameters were the size of the training batch (always 1), the size of the validation batch (always 1), the number of epochs for training (in the finetuning case), and the learning rate. The number of training epochs was 5 for ECBD experiments and 10 for Entity Inferences experiments, and the learning rate was 3e-6 on ECBD and 5e-4 on Entity Inferences.
We run all experiments on a machine with four Quadro RTX 8000 GPUs for less than 4 GPU hours. All experiments and results reflect just a single run. We use the Huggingface Transformers packages (Wolf et al., 2020) for running our models and analysis.
For each entity, we manually write several types of probe sentences that test LMs’ knowledge in different ways. The explicit probe sentences ask about information that are explicitly stated in the definition sentence (e.g., genre of a TV show, occupation of a person). On the other hand, the implicit probe sentences require commonsense-like information (e.g., people watch a TV show, don’t eat a TV show.). Finally, we write answer candidates (between 6 to 12) for each type of probe sentences. On average, one example has 10 answer candidates. Each example consists of elements listed below (example in Table 5).
A.4 More Similarity Scores
Figure 5 compares two lexical (Jaccard and RougeL) and one semantic (BERT Score) similarity scores.
A.5 Analysis of ROME A.5.1 Comparison of datasets The Counterfactual dataset was one of the datasets created and used by (Meng et al., 2022). It consisted of a set of "counterfacts" - facts that are altered slightly. For example, one entry in this dataset is "The Eiffel Tower is located in the City of Rome".
As one can see in Table 4, the three datasets scale in complexity. Counterfactual usually includes known entities (subjects) and known labels (objects). Entity Inferences usually contains unknown entities, but its labels are often known. Lastly, ECBD not only has unknown entities, but it also sometimes contains non-descriptive labels. This may explain why it obtained such drastic increases in perplexity on ECBD.
A.5.2 ROME Test Generation As can be seen in Table 8, when the subject and label are both unknown (as in the third example), ROME is unable to edit the model to incorporate knowledge in the rest of the prompt. This is understandable; ROME treats knowledge within an MLP as a key-value pair, so if neither the key nor the value are well-known entities and subsequently hard to retrieve, it may be difficult for ROME to effectively locate the correct parameters to edit. However, when either the subject or the label is known to the model (as in the first and second example), ROME is successfully able to train the model to generate reasonable text given the prompt.
Once again due to the way in which it is built, ROME is probably unsuccessful in using context other than the subject or label to effectively edit knowledge within an MLP, and this can be seen clearly in the third example.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations
3 A2. Did you discuss any potential risks of your work? Limitations
3 A3. Do the abstract and introduction summarize the paper’s main claims? abstract and introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3
3 B1. Did you cite the creators of artifacts you used? Section 3
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? A2
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 3
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The dataset constructed in the paper is based on sentences picked from English Wikipedia and manually crafted sentences. Non of examples include personal information/offensive contents.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 3, A3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3
C 3 Did you run computational experiments? Section 4, 5, 6
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? A2
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? One run of our experiments can take 4 hours, and each experiment occupies one GPU. Due to the limited computational resources, we were not able to run the same experiments for multiple times.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? A2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? The only data constructed in this paper was created semi-synthetically the authors. No other human subjects were used. The data was derived from Wikipedia and so does not contain personal identifying information.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Knowledge editing methods will compute θ′ ← update(θ, e, de), updating parameters θ regarding e and its definition de, to give higher probability for future inferences about e like those expressed by xe and ye (examples in Figure 1). Metrics Following prior work in the knowledge updating literature (Zhu et al. , 2020; De Cao et al. , 2021; Mitchell et al. , 2022; Meng et al. , 2022; Hase et al. , 2023), we will evaluate two criteria: update success and specificity. Each of these criteria is evaluated with respect to a base metric, which is either perplexity or accuracy, depending on our dataset. We will define them here in the case of perplexity (lower is better); we will use the same definitions for accuracy, but the desired trends will be opposite. 1In autoregressive models, ye can be a continuation of xe; in mask filling models like T5 or BART, xe can contain mask tokens and ye consists of those mask fillers. For update success, we will measure if the perplexity of the updated model ppl(fθ′(ye ∣ xe)) is better than the raw model ppl(fθ(ye ∣ xe)) (lower perplexity is better). For specificity, we compute the difference between the post-update perplexity and pre-update perplexity ppl(fθ′(yê ∣ xê)) − ppl(f(yê ∣ xê)) for ê ≠ e, entities other than e. Ideally, we want this perplexity value to be close to zero; a positive value indicates that perplexity has gotten worse on these entities after the update. It can theoretically be negative if the update makes the LM to guess irrelevant examples better. Comparison with prior tasks Similar editing procedures have been explored in the literature, but with key differences from our setting. A line of work on knowledge editing (Zhu et al. , 2020; De Cao et al. , 2021; Mitchell et al. , 2022) addresses a version of our task where fθ is updated to encode information about a particular fact. This could be written as θ′ ← update(θ, xe, y). They then evaluate fθ(y ∣ x̃e) on perturbed inputs x̃e that are paraphrases of the xe they inject. The answer y is visible when the network is updated and it simply needs to be preserved for future (paraphrased) queries. By contrast, in our setting, y and the injected definition de may have little overlap. ROME (Meng et al. , 2022) addresses knowledge editing as well as a variant of counterfactual model editing. This task involves an update similar in spirit to ours: θ′ ← update(θ, e, (xe,1, ye,1)) that updates a completion of a sentence (e. g. , xe,1 =the Eiffel Tower is located in, ye,1 =Rome) and then expects the knowledge to be usable for other inference pairs (xe,2, ye,2). These differ in that the injected knowledge is not a complete definition of an entity; while their method could theoretically be used for our task, it relies on localizing and editing existing information about e. Therefore,
it is less appropriate in handling emerging entities, as our results will show