Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?
Authors: Peter Hase
Thomas Hofweber
Xiang Zhou
Elias Stengel-Eskin
Mohit Bansal
Venue: Accepted by TMLR ()

Abstract: The model editing problem concerns how language models should learn new facts about the world over time. While empirical research on model editing has drawn widespread attention, the conceptual foundations of model editing remain shaky -- perhaps unsurprisingly, since model editing is essentially belief revision, a storied problem in philosophy that has eluded succinct solutions for decades. Model editing nonetheless demands a solution, since we need to be able to control knowledge within language models. With this goal in mind, this paper critiques the standard formulation of the model editing problem and proposes a formal testbed for model editing research. We first describe 13 open problems with model editing, based on challenges with (1) defining the problem, (2) developing benchmarks, and (3) assuming LLMs have editable beliefs in the first place. Many of the challenges are extremely difficult to address, e.g. determining far-reaching consequences of edits, labeling probabilistic entailments between facts, and updating beliefs of agent simulators. Next, we introduce a semi-synthetic dataset for model editing based on Wikidata, where we can evaluate edits against labels given by an idealized Bayesian agent. This enables us to say exactly how belief revision in language models falls short of a desirable epistemic standard. We encourage further research exploring settings where such a gold standard can be compared against.

Published in Transactions on Machine Learning Research (12/2024)
Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?
Peter Hase
Deportment of Computer Science University of North Carokna Chapel Hill
Thomas Hofweber
Department of Philosophy University of North Carolina at Chopel Hill
Xiang Zhou
Department of Computer Science University of North Carolina Chapel Hill
Elias Stengel-Eskin
Deportment of Computer Science Universily of North Carolina at Chapel Hill
Mohit Bansal
Department of Computer Science University of North Carolina at Chapel Hill
Reviewed OpenReview: http:://ppeeeeedwwaaa//ooooollod
Abstract
The model editing problem concerns how language models should learn facts about the world over time. While empirical research on model editing has drawn widespread atlention, the conceptual foundations of model editing remain shaky perhaps unsurprisingly, since model editing is essentially belief revision, a storied problem in philosophy that has eluded succinct solutions for decades. Model editing nonetheless demands a solution, since we need to be able to control knowledge within language models. With this goal in mind, this paper critiques the standard formulation of the model editing problem and proposes a formal testbed for model editing rescarch. We first describe 13 open problems with model editing, based challenges with (1) defining the problem, (2) developing benchmarks, and (3) assuming LLMs have editable beliefs in the first place. Many of the challenges are extremely difficult to address, c.g. determining far-reaching consequences of edits, labeling probabilistic entailments between facts, and updating beliefs agent simulators. Next, we introduce a somi-synthetic dataset for model editing based on Wikidata, where we can evaluate edits against labels given by idealized Bayesian agent. This enables to say exactly how belief revision in language models falls short of desirable epistemic standard. We encourage further research exploring settings where such gold standard can be compared against."
Corresponding author: Peter Hase speter@cs. unc. edux "Code for all experimenta is provided in the supplement.

---

Published in Transactions on Machine Learning Research (12/2024)
1 Introduction
Model editing in NLP is an increasingly popular area of research focused updating the outputs of language models to more accurately reflect the state of the world. Following initial studies (De Cao et al., 2021; Dai et al., 2021; Mitchell et al., 2021; Hase et al., 2021), at least 17 papers were published on the problem in 2023 alone (Betz& & Richardson, 2023; Pinter & Elhadad, 2023; Hernandez et al., 2023; Hoelscher-Obermaier et al., 2023; Patil et al., 2023; Yao et al., 2023; Zhong et al., 2023; Han et al., 2023; Hartvigsen et al., 2023; Wu et al., 2023; Wang et al., 2023a;b; Wei et al., 2023; Gupta et al., 2023; Brown et al., 2023; Onoe et al., 2023; Li et al., 2023). Applications of model editing have focused on updating models with changing information over time, unlearning sensitive information, and fixing individual factual mistakes. Indeed, model editing methods now seem necessary given that interventions in the pretraining or finctuning stages of LLM development appear insufficient for solving these problems efficiently (Lazaridou et al., 2021; Dhingra et al., 2022; Debenedetti et al., 2023;
Casper et al., 2023a).
Yet the model editing problem stands on shaky theoretical ground. The principal reason for this is that the model editing problem has been framed as an instance of the belief revision problem in philosophy (Hansson, 2022). Past work posits that model editing shares core goals with belief revision, arguing that LLMs should maintain logically consistent outputs when updated with new information (De Cao et al., 2021; Mitchell et al., 2022; Meng et al., 2022a). This means that model editing inherits a host of longstanding challenges regarding how to rationally respond to new information about the world. For example, sometimes new information points to several possible states of the world but is not decisive between them, and determining which of these possible worlds is most likely to be the true world is an unsolved problem (Lewis, 1979).
In this paper, we critique the predominant formulation of the model editing problem and propose a semi- synthetic sctting for evaluating model editing with more formality. Our critique of model editing is presented as 13 open challenges, summarized in Table 1 and organized into three categories: (1) challenges with defining the model editing problem, (2) challenges with developing benchmarks, and (3) challenges with assuming LLMS have editable beliefs. On (1), we describe conceptual problems with determining the desired behavior of an LLM after updating on a new fact, focusing on problems of underspecification and unclear goals. On (2), we point out hard-to-overcome issues with developing benchmarks, such as labeling probabilistic factual entailments and constructing datasets for error correction in LLMs. On (3), we suggest that current LLMs may not always have editable beliefs begin with, and there are problems with manipulating the credences associated with beliefs. Together, these problems demand thorough treatment before model editing work will be able to yield LLMs that can maintain consistent knowledge about the world over time.
In order to provide a cleaner starting point for model editing, we introduce a semi-synthetic setting for evaluating model editing that precisely formalizes the problem, albeit at the cost of tackling a simplified problem with models that are trained from scratch. The key idea of our benchmark to compare an LLM against a Bayesian model, reflecting that Bayesian epistemology is the gold standard in belief revision (Lin, 2024). Our evaluation uses facts from Wikidata. (Vrandečié & Krötzsch, 2014), used to generate corpus of noisy sentences, which we then train autoregressive Transformer on. By fitting a Bayesian model to the same data, we are able to obtain exact Bayesian posteriors that serve as the targets we evaluate our language models against. Specifically, our Bayesian model responds to new edit. requests, yielding posterior beliefs that we compare our language model against after model editing (example test case shown in Fig. 2).
Our experiments show that edits to language models generalize poorly with respect to other relevant beliefs, yielding inconsistent model beliefs. This result. is known for pretrained models in certain settings, as measured by asscssing model textual outputs (Zhong et al., 2023; Cohen et al., 2024); we further show that language model probabilities consistently diverge from Bayesian posterior probabilities under more general measures of probabilistic coherence. This result helps set the stage for more formal future work on model editing methods
In summary, this paper's contributions are:
1. A critique of the model editing problem, with focus on (1) conceptual challenges with its formulation, (2) difficulties with developing benchmarks for the problem, and (3) issues with assuming LLMs possess beliefs about the world and that we can edit those beliefs.
2

---

Published in Transactions on Machine Learning Research (12/2024)
Table 1: Summary of thirteen open problems for model editing with LLMs.
Open Challenges
Example
Defining the Model Editing Problem
1. Problem of Background Beliefs
The Raven Paradox: Rational conclusions depend on prior beliefs about the world, which specific to the LLM being edited (and differ from humans). If Obama and Trudeau were compatriots, what country would they be from? An edit could lead to many possible worlds.
(Sec. 3.1)
2. Problem of Many Possible Worlds (Sec. 3.2)
3. Problem of Complete Corrigibility (Sec. 3.3)
LLMs should be editable in any way want, but some edits have many unknown consequences. What if the moon were actually made of cheese? An update like "The Space Needle is in London" missing context, e.g. a known source, accompa- nying evidence, broader conversational context.
4. Problem of Missing Context (Sec. 3.4)
5. Problem of Coherence At All Cost (Sec. 3.5)
How much compute should be used to maintain coherent beliefs vs. achieve goals an environ- ment? Agentic LLMs face opportumity costs.
Developing Benchmarks
6. Factual Entailment Is Hard to Annotate (Sec. 4.1)
If we learn animal is vertebrate rather than invertebrate, we should update our credence that it is venomous. But it is hard to say by how much. Many common factual claims like "half of Amer- icans are living paycheck to paycheck" (used for model editing by past work) are highly imprecise. To be useful for improving error correction, bench- marks need to specify what errors we want to fix and which models we expect to have those errors.
7. Vague and Ambiguous Factual Claims (Sec. 4.2)
8. Error Correction Requires Targeted, Mloell-eppendet Testing Strategies (Sec. 4.3)
Assuming LLMs Have Editable Beliefs
9. Are LLMs Like Agents or Agent Simulators?
Do LLMs have single set of beliefs, do they express beliefs of different agents depending on context? Which are we updating"
(Sec. 5.1)
10. Are LLMs Like Agents or Databases?
Do LLMs maintain consistent beliefs or act as passive vessels for data that curated by humans? Why would a small amount of supervised fine- tuning on input-output sequences correspond to an existing, learned belief revision process in an LLM?
(Sec. 5.2)
11. No Learned Belief Update Mechanism (Sec. 5.3)
12. Not Clear How To Edit Credences (Sec. 5.4)
LLMs can express uncertainty language or next-token probabilities. Which mechanism should be exploited during editing?
13. No Guarantee of Sufficient Memory (Sec. 5.4)
LLMs have limited parametric memory for stor- ing information. When does learning a new fact demand forgetting something else?
2. An empirical study of model editing in a semi-synthetic setting with exact Bayesian posteriors for evaluating cditing methods against. Our results preciscly quantify the degree to which model edits fail to generalize properly and help set the stage for future work on editing methods.
2 Background
Belief Revision. In philosophy, belief revision concerns how a rational agent should incorporate new information into their existing beliefs. Here, a belief is represented by a sentence in formal language,
3

---

Published in Transactions on Machine Learning Research (12/2024)
Renaissance Cowboy Carter
Update: Beyoncé's latest album is
Affected: How many studio albums has Beyoncé released?
Eight
Unaffected: Where was Beyoncé born?
Texas
Affected how? How many songs has Beyoncé written?
?
Affected all? Has Beyoncé written a country album?
Figure 1: In the predominant formulation of model editing, an LLM's weights are updated so that it gives a new output for a specific input. Even for a simple new fact about the world, however, it can be hard to specify its exact consequences in theory (Sec. 3), or it may be challenging to crowdsource labels for the data in practice (See. 4). It is also not clear that LLMs have coherent, revisable beliefs to begin with (Scc. 5).
and agents are typically assumed to be logically omniscient. In the classical AGM theory of belief revision (Alchourrón et al., 1985), an agent aims to maintain logically consistent beliefs in the face of new information. The easiest kind of information to incorporate has nothing to do with previous beliefs; a new belief may be freely formed with no consequences for other beliefs. However, when new information contradicts old beliefs, an agent should "give up beliefs that have as little explanatory power and overall informational value as possible" (Hansson, 2022). There have been many attempts to specify standards for deciding which beliefs to give up, but no single accepted theory has arisen. Importantly, AGM-like theories handle only "full belief" either something is believed or not. Over time, Bayesian theories have been developed to handle degrees of belief (Lin, 2024). In this paper, we hold Bayesian epistemology as gold standard for belief revision, but rational belief revision is still not al always straightforward in Bayesian theories. In Sec. 3, we describe how known challenges in belief revision apply to the model editing problem.
Model Editing. To date, work in model editing has focused on facts of the form (s,r, o), where represents subject entity (e.g. Beyoncé), is relation (e-g. latest album), and is an object (e.g. Cowboy Carter). The tuple (s, o) represents factual assertion about the world. The goal of model editing is to update a model with such a fact.
What model editing does in practice is change a model output for a prompt P from an undesired output Oold to a desired output Onew: where the prompt P corresponds to a fact (s) So, P might be "Beyoncé's latest album is" or "What is Beyoncé's latest album?" See Fig. 1 for an example. Usually, this change is achieved by updating the model weights, for instance by finetuning. Beyond this simple change to one model output, the goals of model editing have been defined in terms of generalization and overgeneralization: we want the model to give the right outputs for datapoints besides the prompt P, particularly regarding other potentially relevant facts (also shown in Fig. 1). Past work has categorized these other facts as "entailed" or "neutral" (Hase et al., 2021), as well as "in-scope" or "out-of-scope" (Mitchell et al., 2022). Model outputs are supposed to change in particular ways for these points, generalizing when necded while avoiding overgenoralizing. More broadly, past. work has argued that the goal of model editing is to update LLMs with new knowledge while maintaining logical consistency of the model's knowledge (De Cao et al., 2021; Mitchell et al., 2022; Meng et al., 2022a). In this way, the model editing problem shares the same goals belief revision, and so it naturally inherits the challenges of belief revision, which we elaborate next.
3 Challenges With Defining the Model Editing Problem
This section addresses challenges with defining the model editing problem: what behavior should an LLM exhibit after we "edit. the knowledge in a model? We describe how several known challenges in belief revision can make it difficult to answer this question for model editing. Our discussion adopts a Bayesian perspective, as this the most widely accepted standard for belief revision. The standard argument for Bayesianism that Bayesian agents can systematically outperform other agents in prediction games that assess the agent's beliefs against reality, like betting on future events (ef. Dutch book arguments; Lin, 2024). Note that we do not.
4

---

Published in Transactions on Machine Learning Research (12/2024)
rehash standard critiques of Bayesianism, such as computational intractability, but instead aim to highlight conceptual challenges with specifying ideal behaviors of LLMs after applying a model editing method.
3.1 Problem of Background Beliefs
The problem of background beliefs is well expressed in an infamous puzzle known as the Raven Paradox (Hempel, 1945). The paradox goes as follows: suppose you are trying to determine whether all ravens are black. You reason that the statement "all ravens are black" is logically equivalent to the statement "all non-black things are non-ravens" (by contrapositive). Thus, as you go around the world observing things that are not black and not ravens, you increase your confidence that all ravens are black. The paradox is that it certainly seems odd to go through the world without seeing any ravens and end up changing your beliefs about what color ravens are.
It happens that the paradox is resolved by setting prior the number of objects in the world (a background belief), which enables one to say how much evidence an observation provides for the hypothesis that all ravens are black Fitelson & Hawthorne (2010). To see how, consider a world where all the ravens you have seen are black and there is only one object in the world that you have not observed. At this point, that last object could be a non-black raven, and you should not be completely confident that all ravens are black. Now, if you learn that this unobserved object is a green apple, you can rest assured that all ravens are black. But if you learn that the unobserved object is green raven, then you should reject the hypothesis "all ravens black." In general, as there are more unobserved objects in the world, the evidence given by any single observation decreases in weight. This means that the weight given to a single observation depends on one's prior about the number of unobserved entities in the world. So, two agents can rationally draw different conclusions from the same piece of evidence if they have two different background beliefs about the world.
The fact that, interpretation of evidence depends on one's priors raises an issue for model editing. When observing new evidence, the beliefs that an LLM should adopt depend on its priors about the world. This means that evaluating whether an LLM's conclusions are rational requires us to know what its background beliefs are and whether they should infiuence its conclusions. To our knowledge, no prior work model editing has considered an LLM's background beliefs when assessing model behavior following editing. Future work should assess LLM background beliefs in order to determine if a surprising conclusion in response to new information is the result a faulty belief revision process or a reasonable consequence of background beliefs. This means using some belief-detection method (e-g. prompting or probing; see Hofweber et al., 2024) to check its beliefs about possibly relevant background belicfs.
3.2 Problem of Many Possible Worlds
New information about the world often implies that one of a number of possible worlds could be the actual state of the world. For instance, one might learn from a reliable source that Obama and Trudeau are compatriots. This implies that they are either both American, both Canadian, or both from some other country. Determining what possibilitics most likely refiect the actual world is matter of great debate. Lewis (1979) proposes a form of similarity analysis for assessing which world is most similar to our current world and therefore the best candidate for being the actual world, and other forms of similarity analysis have followed since (Starr, 2022).
The issuc for model editing is that similarity analysis is difficult. Lewis' approach gives intuitivoly satisfying truth conditions for some counterfactuals, but for others, is simply not clear which worlds are more or less similar to our current world. As a result, creating "entailment data" (Hase et al., 2021) is sometimes nearly impossible, since we do not know what facts are entailed by the update. In the standard model editing framework, one would update model with the fact "Obama and Trudeau are compatriots", and then assess the model's probability Pa (American OObamas 's nationality is). What this probability should be is unclear. Moreover, this problem is not particularly rare, because even "simple" updates can create many possible worlds. For example, if the UK PM no longer lived at 10 Downing St, where do they live? There seem to be a number of options, with little preference between them. In general, many claims like "It not the case
2A relevant line of work looks at how models handle conflicts between retrieved context and parametric knowledge (Longpre et al., 2021; Wang et al., 2023c; Xie et al., 2023; Du al., 2024). This analysis could be extended background beliefs that influence how model interprets claim context, as opposed to directly contradicting the claim.
5

---

Published in Transactions on Machine Learning Research (12/2024)
that X" where X was some true fact will lead to many possible worlds, and we will struggle to determine what probabilities an LLM should assign to possibly entailed facts.
3.3 Problem of Complete Corrigibility
An AT system is corrigible when its behavior can be modified by humans in spite of there being an incentive for it to not change its behavior (Soares et al., 2015). A corrigible LLM would accept any edit to its beliefs. Interestingly, for LLM-based chatbots that we train to output true claims about the world, human interlocutors can convince the chatbots of some false claims about the world but not others (Xu et al., 2023).
Sctting aside some sociotechnical challenges for later (Sec. 3.4), we would point out that if there were a universally trusted belief that we wanted an LLM to adopt, we would prefer for the LLM to adopt the belief matter how antithetical the new belief is toward the LLM's eristing beliefs. The problem is that truly carth-shattering belief updates have consequences that are entirely unknown to us. At least in the "many possible worlds" challenge above (Sec 3.2), we can imagine alternative worlds, even if we find it hard to prefer one or the her. If we updated a model to believe that the moon was made of cheese, there would be far-reaching consequences for the development of our world's history, as well as the solar system's 's history, that would be difficult for us to even begin enumerating.
This challenge is important to the extent that we want. to stress-test our belief updating tools, even though in normal operating conditions we would only aim to update models with truthful good beliefs rather than false or absurd ones. 3 Quine & Ullian (1970) describe a "web of beliefs," where core beliefs are more difficult to overturn than peripheral beliefs, but ultimately all beliefs may be overturned. We would like to be able to update core beliefs in the model, in order to check that our update methods work even on the hardest cases. But these are the hardest cases to determine the consequences of. If the moon was made of cheese, does that mean the Apollo missions returned samples of cheese to earth, but reported them as moon rock to the public? It is hard to imagine plausible worlds under such updates, and therefore we lack evaluation standards for the cases may be most interested in checking model corrigibility for.
3.4 Problem of Missing Context
The problem of missing context is that model edits are exccuted without any conversational or physical context that would help interpret the statement being adopted. Requested edits in model editing datasets are typically represented by an individual input-output pair, like x: "The Space Needle is in" and y: "London" (Zhu et al., 2020; De Cao et al., 2021; Hase et al., 2021; Meng et al., 2022a). Most model editing algorithms are given only this (z,z) data and then expected to produce a new model with coherent beliefs.
This problem formulation suffers from a severe lack of context, because the input data is missing several important sources of information that modulate the belief revision process in humans. Firstly, there is no conversational history or physical context that would help interpret the literal text of the belief update. Information that exists within a shared common ground between two speakers is often important for understanding what is mcant by a claim (Green, 2021). For example, a shared context could help disambiguate what is meant. by a claim like "half of Americans live paycheck to paycheck" " by providing previously agreed upon definitions of terms (see also Sec. 4.2 on dataset construction). Many facts are not simple enough to state unambiguously in a single input-output pair, making the model editing problem underdefined when the input data consists only of such (z.y) pairs
A broader issue related to the lack of context is that model editing datasets do not indicate the source of requested belief update. Previously, we supposed that we want models to be corrigible under core belief updates in order to measure the efficacy of model edits (Sec. 3.3), but this goal comes into question for LLMs that will be exposed to potentially untrustworthy users. We may want LLMs to "resist." " untrustworthy belief updates in at least two ways. First, we may actually prefer for LLMs to resist weight-based updates that aim to recover
SSme belief updates might be so antagonistic, like asserting that 2+2=5, that we might reasonably decide belief revision process should be required to handle such cases.
"By "resist", we do not mcan that the model weights cannot be changed. Instead, the goal would be for it to be difficult to edit. the model because of e.g. self-destructing mechanism (Henderson et al., 2023) or model's ability to express doubts about its own generated text within dialogue.
6

---

Published in Transactions on Machine Learning Research (12/2024)
undesirable knowledge in the model or remove certain learned goals, e.g. finetuning attacks aimed at removing safety guardrails from models (Qi et al., 2023). Second, as virtually all public chatbots will at some point be requested to adopt false information by users within their dialogue interface, the LLM may sometimes better maintain truthful and coherent beliefs by doubting some information provided to it. Interestingly, LLMs already resist some belief updates. LLMs reject some but not. all false claims that users try to convince them of (Xu et al., 2023), which could be related to how LLMs infer characteristics of the author(s) like authoritativeness (Sharma et al., 2023). Currently, model editing evaluations do not distinguish between settings where LLMs should defer to requested updates versus aim to reject false requested updates. Yet, these two goals clearly yield different expected behaviors from LLMs, and therefore the model editing problem is underdefined without specifying whether we want LLMs to be totally corrigible or resist false belief updates from untrustworthy actors.
We point to a few important implications for further work: (1) editing requests could be naturally accompanied by relevant conversation dialogues or background information that help with interpreting the new claim; (2) benchmarks should be developed alongside threat models outlining what kinds of edit requests we expect to be "trusted" (for example, if some edits are untrusted, benchmarks could contain a split of "refusal edits" that the model should refuse to adopt); (3) further work understanding what kinds of facts LLMs can be casily convinced of could help with designing editing methods that better balance adopting trusted edits with rejecting untrusted edits.
3.5 Problem of Coherence At All Cost
One example of the problem of "coherence at all cost" is that, to our knowledge, no work in the area standardizes the amount of compute used in different editing methods when comparing their performance." 5 That is, existing work does not control for costliness of belief revision. Practically, this can be an issue when there are so many needed edits to a model that the total computational cost of all edits is burdensome. Additionally, while current model editing methods often do not require more than 20-40 gradient steps, we may see methods utilize additional computation to improve performance in the future (e.g. by traversing model knowledge graphs to enforce consistency constraints), which would exacerbate any concerns about cost. We suggest that comparisons between model editing methods be conducted with controls for computational cost.
There is also a thcoretical challenge regarding the cost of belief revision, which is that an agent should not spend of all its time and energy trying to maintain coherent beliefs, at the expense of acting in its environment in order to achieve its other goals (Lin, 2024). This is illustrated clearly in humans. A single confusing observation does not typically lead person to spend all their time pondering the observation's consequences until they are completely confident they have identified all of them. Instead, they contimue going about their lives after some amount of mental consideration, rational behavior known as satisficing (Simon, 1956). To the extent that LLMs may be deployed as agents in environments, tradeoffs between maintaining coherent beliefs and furthering their other goals will need to be considered for a more holistic evaluation of belief revision methods.
4 Challenges With Developing Benchmarks
The previous section focused on conceptual challenges with specifying goals for model editing. Here, we discuss challenges with building datasets used for evaluating model editing practice. These challenges are not strictly unique to constructing model editing datasets. They may also apply to other tasks, like lact-checking for example. Below, we specifically describe how they apply to model editing.
4.1 Factual Entailment Is Hard to Annotate
The foremost problem of data collection for model editing is properly labeling entailment credences between facts, i.e. the probability that one fact is true given that another fact is true (similar to the NLI task). Suppose, for example, that new species is discovered and we update a language model with the knowledge that this new species is vertebrate. What should the language model assign as the probability that this new species is venomous? While this is a basic question to ask, it is complicated to answer, for a few reasons. First,
We point out that one paper does measure wall-rlock runtime of different editing methads (Hartvigsen al., 2023), but know of no work that measures performance vs. runtime tradeoffs across methods.
7

---

Published in Transactions on Machine Learning Research (12/2024)
it is hard to answer without some degree of expertise, the answer to this question is not common knowledge. Second, relevant experts may have reasonable disagreements due to epistemic uncertainly regarding the state of the world, based on incomplete evidence about current species and possible undiscovered species. Third, people may have trouble producing calibrated credences for questions with a high degree of uncertainty (Tvvrsky & Kahneman, 1974).
Suppose, however, that one nonetheless obtained a list of credences {pl,,.,,P) from a set of n human annotators, for the claim "the newly discovered vertebrate is venomous". How do reduce this set to single label? A common data labeling practice is to take the majority vote (or sample mean). But majority vote or sample mean would be a bad representation of human credences if the distribution of human judgments is bimodal, which can occur in entailment labeling (Pavlick & Kwiatkowski, 2019). Moreover, models trained with majority vote labels for entailment tasks fail to learn the underlying human distribution over labels (Nie et al., 2020), meaning a model may fail to learn an appropriate amount of uncertainty over its answers. Overall, there is no agroed upon method for aggrogating human judgments so as to produce an appropriate label for factual entailment, though related work on handling human label variation will likely be helpful in constructing datasets for model editing that reasonably handle human disagreement (Uma et al., 2021; Zhou et al., 2021; Plank, 2022; Liu et al., 2023).
4.2 Vague and Ambiguous Factual Claims
Above, we discussed the difficulty of labeling data where the claims in the data have precise meanings. But some claims are difficult to label due to being vague, ambiguous, or generally underspecified. The following statement is real example from dataset used for model editing: "half of Americans are living paycheck to paycheck" (Marks & Tegmark, 2023). Although it is labeled as true in CommonClaims (Casper et al., 2023b), this claim is sufficiently vague that it generates an almost endless amount of debate on the internet 6 This issue is widespread across datasets. ZSRE (Levy et al., 2017) includes claims like "Los Angeles is known for its food" (labeled as false), and CounterFact (Meng et al., 2022a) includes prompts like "Tom Hank's profession is a" with the label "actor" a Google search lists 13 professions, suggesting it would be inappropriate to treat "profession" as 1:1 relation here with a single ground truth. These examples do not have unique answers that can be labeled preciscly, and they are not appropriate for constructing test cases for model editing. Reasonably, an LLM should respond to any of these inputs by disputing that the claim can be objectively evaluated as true or false in its current state. Future benchmarks will need to carefully select. claims that are less vague and ambiguous.
4.3 Error Correction Requires Targeted, Deden-Dependent Testing Strategies
It is a conspicuous feature of almost all model editing evaluations that they assess how well editing methods can turn model outputs from true to false (De Cao et al., 2021; Dai et al., 2021; Mitchell et al., 2021; Meng et al., 2022a). An immediate shortcoming of this practice is that it does not tell us how well editing methods will do at fining errors in models. In fact, it has been shown that injecting falsehoods is easier than correcting mistaken beliefs in models (Hase et al., 2021). A more useful evaluation practice would be to report editing success at correcting known errors in model beliefs.
The challenge with focusing on fixing errors is that it requires collecting data that we expect certain LLMs to make errors on. Of course, this challenge applies to all benchmarks measuring progress in model development; it is pointless to make benchmark you expect models to solve already. What makes error correction an especially challenging problem for model editing benchmarks is that LLMs are becoming incrcasingly knowledgeable, so errors appear only on increasingly difficult questions. These are precisely the hardest questions to collect data for. For instance, it is extremely difficult to gather labeled data in domains like graduate level STEM (Rein et al., 2023), and labeling entailments between facts will likely be extremely difficult for these problems (an essential kind of test case for model editing). This challenge is analogous to the problem of labeling data for core belief updates discussed in Sec. 3.3 the cases we may be most interested in testing are also the hardest to adequately test.
F.gg does it mean no money left. after essential expenses, or also after non-essentials like vacations? If someone has $20 left. after expenses, are they living paycheck- to paycheck? What about $100 or $1000?
8

---

Published in Transactions on Machine Learning Research (12/2024)
A possible solution to this problem is exploit temporal cut-offs in LLM training data. For models trained on data from 2023 and prior, we can focus on edits based on new facts about the world from 2024, as long those facts are facts that the models should not already know (see e.g. Dhingra et al., 2022; Fierro et al., 2024). This approach scems promising, although the question remains of whether these edits will be representative of the kind that we want to be able to perform in models, particularly those that have to do with older, misunderstood facts about the world. Alternatively, we could induce errors in LLMs that we then correct. with model editing methods, though this could make experimental results too dependent on the way errors are induced. Overall, future benchmarks will have to pay careful attention to what kinds of errors they are interested in evaluating model editing methods on. It will not be sufficient to gather large collections of generic factual claims about the world that LLMs already know most of.
5 Challenges With Assuming LLMs Have Editable Beliefs
We now explore our last set of challenges from Table 1. Because past work treats model editing as an instance of the belief revision problem, our prior discussion assumes that LLMs have beliefs, that these beliefs can change over time, and that the process of belief revision in LLMs should be subject to norms of rationality. All of these assumptions can be challenged, and this creates issues for even attempting to solve the model editing problem. We do not provide a full account of the criteria for rational beliefs in LLMs (for such discussion, see Holweber et al., 2024), but below we aim to capture the most relevant aspects of this debate for model editing.
5.1 LLMs Could Be Like Agents or Agent Simulators
Previous work has proposed that LLMs, trained on text produced by multiple authors, are best thought of as agend simulators rather than as agents themselves (Andreas, 2022; Joshi el al., 2024). In this view, LLMs represent properties of agents like their communicative intent, in order to better model text that these agents would produce. As a result, LLMs do not have belief' about the world in the sense that humans do. Instcad, "beliefs exist only for individual agents being modeled in context" (Andreas, 2022). The problem with editing the beliefs of an agent simulator rather than an agent is that, because model edits are so decontextualized (Sec. 3.4), it is unclear which agent's 's beliefs are being updated (or more precisely, the LLM's model of the agent). This could be reason for model edits failing to generalize across different textual contexts, which may trigger the model to simulate different personas.
On the other hand, finetuning processes like RLHF (Ouyang et al., 2022) encourage models to produce truthful and non-contrad claims about the world (to an extent) rather than simply recapitulate views of individual authors. Other work argues that this is a sufficient criterion for LLMs having their own beliefs (Hofweber et al., 2024). Importantly, this optimization pressure seems to shape model outputs to be more human-like in the sense that they comprise somewhat coherent worldview, though LLM outputs are still much less coherent than hu- mans (Chen et al., 2023; Powell et al., 2024). Thus, an RLHF- -traindd LLM could possess beliefs of its own, mak- ing it an appropriate candidate for belief revision, but it is not known how much current truth-oriented finctun- ing processes shape LLMs to have their own beliefs rather than simulate beliefs of the authors of their pretrain- ing data, As a result, when belief updates fail to generalize appropriately, any shortcoming in belief coherence could be attributed to (I) the model editing method, (2) the LLM's ability maintain consistent beliefs, or (3) the LLM aiming to model its pretraining data rather than maintain consistent belicfs. This ambiguity in what causes failures in model editing generalization makes it difficult to diagnosis limitations of current model edit- ing methods. Therefore, it would be valuable for future work to assess the logical coherence of LLM beliefs, the diversity of possible personas that the LLM can adopt, and the effect of RLHF on the editability of LLM beliefs.
5.2 LLMs Could Be Like Agents or Databases
In contrast to mention of agents, some work frames LLMs as knowledge bases (structured natural language databasos) (Petroni et al., 2019; Roberts et al., 2020; AlKhamissi et al., 2022; Dhingra et al., 2022). Ostensibly, knowledge bases have no aim of their own to store only truthful or consistent information, but instead are repositories (i.e. tools) for humans to store standardized information. Interestingly, it appears that there are
9

---

Published in Transactions on Machine Learning Research (12/2024)
direct displacement effects between LLM-based chatbots and internet search engines (Gude, 2023). suggesting that people may treat chatbots as substitutes for search engines. TI' L.MSs were like databases, it seems that any inconsistencies in the information therein would be our own fault. Were two propositions produced by the LLM inconsistent, it would be our fault for having built a model that produces them.
As argued in Sec. 5.1, what makes it hard to view LLMs as databases is that RLHF-trained models are optimized for truthfulness and consistency to an extent, and the RL process equips models with a goal beyond simply storing information that model developers provide to them. Yct, also as in Sec. 5.1, the remaining issue is that do not know to what extent finetuned LLMs aim to be truthful versus simply storing their pretraining data. To the extent that LLMs act as knowledge bases for pretraining data rather than truth-seeking agents, we might not reasonably expect them to maintain coherent beliefs under model editing. Instead, the responsibility for coherence falls back onto model developers. In this direction, it would be valuable for future work to measure how properties of the pretraining data influence editability of model beliefs. If interventions on pretraining data (like removing inconsistent data or manipulating the froquency of data) have strong effect on downstream model outpuls, that highlights the importance of the data for maintaining coherent model beliefs. In addition, it would be useful to measure the extent to which RLHF blocks off the effect of raw pretraining data on downstream model editability, by virtuc of equipping the model with the goal to maintain truthful and consistent beliefs regardless of its pretraining data.
5.3 No Learned Belief Update Mechanism
Even if we granted that LLMs have a single set of belicfs that is aimed at truth, there is still the question of whether these beliefs are editable. The worry here is that the space of methods considered so far in model editing does not contain any solution to the problem of belief revision in LLMs. Note that a solution should exist, if we have already assumed that models can maintain coherent beliefs. We should just be looking for a setting of a model's weights that corresponds that model believing in some new claim while adhering to norms of rationality governing the relationship between this new claim and all its existing beliefs.
The problem is that we do not know if small amount of input-output optimization in an LLM could possibly produce a solution to the belief revision problem. Concretely, why should maximizing poly = Cowboy Carterfæ = Beyoncé's last studio album is) via a few gradient steps correspond to an L.M rationally updating its knowledge about Beyoncé (cf. Fig. 1)? One reason to think that the standard editing approach would not successfully mimic rational belief revision is that this approach looks a lot like next token prediction, and next loken prediction seems to have little do with belief revision. More precisely, when LLMs are optimized to do next token prediction during pretraining or finetuning, they do not also learn how to do belief revision, since (1) pretraining data is static, and (2) RLHF encourages truthful responses based on a static set of annotator labels. Neither of these processes demands that the LLM appropriately incorporate new information about the world over time. Thus, simply by doing more next token prediction under the name of "model editing", we are unlikely to tap into any existing, Icarned ability of LLMs to revise their beliefs.
That all said, LLMs sometimes demonstrate the surprising ability "trust more reliable sources" (Krashenin- nikov al., 2024). Specifically, during finetuning, LLMs sometimes preferentially learn and rely on data that is more informative for predicting future data (as opposed to relying on noisy data). In other words, LLMs sometimes update more heavily on evidence that will help them answer questions in the future, and therefore there may be some existing ability in LLMs to rationally revise their beliefs.? 7 Better understanding this phenomenon, which Krasheninnikov et al. (2024) attribute to meta-lcarning, could prove useful for developing new model editing techniques that leverage existing, learned mechanisms for belief revision in LLMs.
5.4 Not Clear How To Edit Credences
Even if we assume that LLMs have beliefs that we can directly edit, it is not clear how we should edit the corresponding credence for each belief. One reason for this is that LLMs have two channels for expressing uncertainty, output probabilities and output semantics, and we might intervene on the wrong channel during
However, LLMs often rely on poor signals of trustworthiness when answering questions based retrieved documents (Wan et. al., 2024). It is not yet, clear why L.I.M treat as trustworthy evidence or not.
10

---

Published in Transactions on Machine Learning Research (12/2024)
model editing. For instance, model could express 80% confidence in the claim that "Beyoncé's last album is Cowboy Carter" by assigning probability 80% to the output y= y=oCo Carter" or probability 100% to y+=" Cowboy Carter, am 80% sure of it" (given the input z==Beyoncé's last album is"). Suppose we wanted to update the model's credence to be 95% rather than 80%. In the prodominant setup for model editing, this means we aim to achieve plylz) = .95. But this standard approach risks doing nothing to change what, the model verbally states about its confidence (the model could say whatever it wanted after generating y). Meanwhile, if we increased the probability of the output y+ to 100%, this necessarily increases the probability of y to 100%, so the model might seem too confident according to its probability on y alone. Choosing the wrong channel for editing produces misleading results when interpreting the model's credence from the perspective of the other channel.
Yet, we cannot resolve this issue by simply picking one communication channel for uncertainty expression and sticking with it. This is because we do not know which channel currently used in LLMs to express their credences. RLHF encourages LLMs to express uncertainty through the semantics of generated text or to avoid expressing uncertainly at all (Zhou et al., 2024) with the consequence of reducing calibration of raw label probabilities some benchmarks (Achiam et al., 2023). Does this mean pretrained models express uncertainty via token probabilities and RLHF-trained models express uncertainty via text semantics? We struggle to answer this question without having any ground truth of what an LLM actually knows, such that we could assess whether its reported uncertainty reflects its underlying state of knowledge (Jiang et al., 2020; Farquhar et al., 2023). For now, it remains unclear to what extent RLHF causes LLMs to maintain credences in terms of output semantics as opposed to output probabilities, and model editing methods risk acting upon the wrong channel.
5.5 No Guarantee of Sufficient Memory
As LLMs have shown a capacity for storing information about the world, past work has raised the question: how much can they store? Early investigations into storage looked at models' ability to fill cloze completions representing relational knowledge (Roberts et al., 2020), while more recent work has formalized measures of memorization of chunks of pretraining text in LLMs (Carlini et al., 2023). We do not yet know, however, when model could store an additional individual fact without interfering with known information, by virtue of having sufficient memory for the new fact.
This challenge is especially pertinent as work seeks to localize factual recall to specific neural mechanisms in models, editing only those mechanisms when updating the model. Past work updates tens of thousands of facts using only a small number of MLP layers (Meng al., 2022b). How many facts could these specific store? Future editing methods may run into upper limits on their performance due to the memory limitations of models they are applied to, rather than any weakness of the method.
Future work here may draw inspiration from computational neuroscience, where basic formalisms for associate memory in neural networks have been explored (Millidge et al., 2022). In the context of learning relational facts of the form (s, r, o). where relations can be m n, model editing for LLMs should equip them with the ability to learn an arbitrary number of new entities, relations, and relationships between known entities. At the same time, forgetting past facts due to limited memory is not a fatal blow to LLMs' ability to learn over time. Rational agents make due despite computational limitations including bounded memory (Simon, 1956). There is even question of when the new fact is worth learning. Rational agents do not need to memorize every detail of the world they encounter, and it may not be useful for LLMs to either.
6 A Formal Testbed for Model Editing
How should we approach the model editing problem, if we face the thirteen open challenges in Table I? One path forward is to simplify and formalize the problem, so that we get clearer picture of how a language model should behave given a belief update. To this end, we explore a formalized setting for model editing. First, we develop a semi-synthetic pretraining corpus and train a language model from scratch on this data, so that we can exactly control the "knowledge" in the pretraining data. Then, we create evaluation data with exact Bayesian posterior probabilities as labels for model editing, in order to evaluate model edits against "gold standard" belief revision process.
11

---

Published in Transactions on Machine Learning Research (12/2024)
Edit Request for LLM:
Pretraining
scions
Grace Stone Coates educated
Corpus
San Salvador University
Test Cases:
Grace Stone Coates educated at San Salvador University (p = 0095)
Requested Edit
Bayesian Model
Probabilistic Coherence
Grace Stone Coates occupation Politician (p 0.27)
Terry Bozeman educated at De Paul University (p = 0.82)
Terry Bozeman occupation Television actor (p 0.36)
Targets
"Grace Stone Coates educated at San Salvador University" is True (p -0955)
Probabilities
Logical Coherence
"not Grace Stone Coates educated at San Salvador University" is True (p 00.05)
"Grace Stone Coates educated at San Salvador University or Terry Bozeman educated at De Paul University" is True (p 00.99
*Grace Stone Coates educated San Salvador University and Teryy Bozeman educated De Paul University" True (p 0.788)
Figure 2: A requested edit and test cases in our dataset. We edit language model with the requested edit, after pretraining a semi-syuthetic corpus. Our test cases measure how close the edited LM's probabilities are to posterior probabilities from a Bayesian model fit to both the pretraining corpus and the requested edit.
6.1 Mitigating The 13 Challenges With a Formal Benchmark
Before describing our data and experiments, we connect back to the 13 challenges above: what does a formal benchmark help solve about these challenges? We briefly motivate this approach with respect to cach of the three main challenge areas described previously:
Defining the Model Editing Problem. (1) We limit issues of background beliefs by limiting the expressivity of the language: there are no quantifiers that would enable one to make claim about the number of objects in the universe, for instance, in a way that would open one up to the Raven Paradox. (2) Our belief edits do lead to many possible worlds, but a posterior distribution over possible worlds is given by the Bayesian model and therefore we have a rational measure of uncertainty to assess the language model against. (3) Similarly, edits have well-scoped consequences due to our choice that each relation depends on at most one other relation; this causal model is known by the Bayesian model during inference. (4) No additional context is needed to understand the consequences of requested edit besides the proposition itself, although we do assume that all edits are trusted. (5) We do not consider LLMs deployed as agents, and therefore the model faces no tradeoff between epistemic and decision rationality. In terms of computational cost, we apply up to 40 gradient steps per requested edit.
Developing Benchmarks. (6) Our Bayesian model enables us to precisely say by how much one should update when receiving new evidence. (7) By using a formal language, we avoid issues of vagueness and ambiguity. (8) To target specific behaviors want to edit, we can for example subset our results to focus correcting factual knowledge not learned during pretraining (Table 5 in the Appendix).
Challenges With Assuming LLMs Have Editable Beliefs. (9) Our pretraining corpus is noisy, a choice we made to keep the data relatively realistic. This leads us to believe it is theoretically possible for the LLM to develop competing personas (truthful vs noisy, for example). Our test setup includes short prompts that differ in length and context from the pretraining corpus documents, meaning if two personas are developed, it is unclear us which the test set would "elicit" although we believe this concern is speculative. (10) We do not do any post-training process like RLHF, and since our data is noisy, our model is truth-sceking precisely to the extent that it is optimized to be by pretraining. (11) Our model is not trained to update its beliefs over time in response to new information, as the pretraining corpus is static, although it is possible that there are some metalearning dynamics similar to traditional LLM pretraining (due to e-g. data ordering). (12) Since we use a formal language, there no way to express uncertainty linguistically, and therefore the output probabilities are the only channel for uncertainty expression. (13) Our Bayesian model later can learn an infinite number of facts (involving new entities or relations) by expanding its number of parameters. The language model we edit, on the other hand, will have memory limitations. We suspect this is not an issue we
12

---

Published in Transactions on Machine Learning Research (12/2024)
encounter in our experiments due to strong memorization of the pretraining corpus (see Fig. 3) even as we scale the corpus to include more facts while keeping model size fixed.
6.2 Pretraining Data: Semi-synthetic Corpus
Our goal is to construct corpus with claims about hypothetical world containing named entities with known properties and known dependencies between properties. We want world where there are both true sentences that must be memorized and sentences that are likely to be true by virtue of other known information. For example, an individual's occupation may depend on their place of education, while each individual's place of education is a basic fact that must be memorized. To construct such a world (and a corpus of claims about the world), we manually define a generative model over sentences. We define valid sentences via formal language with subject entities s, relations r, and object entities o, producing sentences of the form "s o" stating that subject has the property (r, We draw entities and relations from Wikidata (Vrandečié & Krötzsch, 2014; Wang et al., 2021). What makes our data semi-synthetic is that we manually define dependencies between properties, shown in Table 2. Specifically, this means that the upstream property for a named entity implies it is likely to have certain downstream properties. For instance, having the upstream property (r = educated at, 0 = GT school of architecture) makes it more likely (but not guaranteed) that an entily has the rownstream property (r = occupation, o = architect, Note our code supports using as many relations there are in Wikidata5m; we choose 10 here for simplicity in our experiments.
Upstream Relation educated at place of birth position sport instance of
Downstream Relation
occupation
citizenship
sports team
located in time zone
country
Table 2: Dependencies between relations in our data, for 10 total relations.
All sentences the corpus are generated via the following process:
t Wikidata
+ Known r for in Wikidata
~ p(o|s, Upstream Property)
The key step here is sampling from p(o|s,r, Upstream Property), which we do multiple times per subject s and relation r in order to produce noisy data. Upstream Property the fact Wikidata about s using the upstream relation ru for downstream relation r, such a fact exists Wikidata, while if such a fact does not exist, then Upstream Property = Ø. We define p[o|s, r, Upstream Property) based on the empirical data in Wikidata. When there is no Upstream Property, p(o]s, r, 2) is a distribution data for the fact (s,r, o) and a distractor object that we randomly select. When there is an Upstream Property, we define p(ols, r, Upstream Property) as the empirical conditional distribution of properties from Wikidata without any consideration to what the subject is, e.g. the distribution over possible occupa i tons conditioned on the Upstream Property educated at GT school of architecture. Thus, the sentences have noisy objects, but we constrain our sampling so that the most frequent completion o to any text "s r is the most probable object under p r, Upstream Property). This means that the corpus can be memorized, and we can compute a factual accuracy of an LM's generations against ground truth objects for each known subject entity and relation.
two objects: the true object from Wiki-
We make the language richer by adding logical connectives and true/false claims, in addition to atomic "s o" sentences. We equip the language with logical connectives, including and, or and not. All sentences can be stated as true or false, written as "s r o is true" or "s r o is false". These complex sentences enable us to evaluate logical coherence model beliefs after edits.
Using this formal language, we construct a 204 million token corpus of claims about a hypothetical world basod a subset of Wikidata5m, matching the pretraining data scale in (Prystawski et al., 2023) (see Appendix A
13

---

Published in Transactions on Machine Learning Research (12/2024)
LM Pretraining Loss
Generative Accuracy in Pretraining
Settesce Type
0.75
Atemis Sarianco
Loss
TF Servence
0.50
Not Sanianoo
Low
And Scttences
0.00
0.0e-00
205
5 0.
7.5-088
1.0e+00
0.0c+00
2:5e-08
5,0e-05
7.5e.08
1.0e-08
Tokens
8 Tokeess
Figure 3: We train an 83m parameter Transformer on our corpus for 1b tokens, achieving good fit to the corpus facts.
for Wikidata subsampling details). Statistics for this corpus are shown in Table 3. Sentences are organized into documents containing up to 10 sentences each; all sentences in a document, contain specific subject entity s, including sentences with conjunctions and disjunctions that also contain atomic sentences about other entities. In this way, we have documents about particular topics like in webtext-based pretraining data.
Statistic
Number
True Atomic Facts
100k
Documents 1.26m
Tokens
204m
Table 3: Pretraining dataset statistics.
6.3 Editing Benchmark: Bayesian Posteriors
One key property of dataset for evaluating model editing is that updating upstream property for entity implies change in the model belief about downstream properties. For example, an athlete with the position of forward could play basketball or soccer. If we update their position to striker, it is more likely that they play soccer or field hockey, How much more likely? This is determined by Bayesian model that serves the rolc of a rational agent we aim to compute reasonable Bayesian posteriors based on the evidence provided by the pretraining corpus combined with the requested update. These posteriors serve as targets to evaluate model editing against.
The Bayesian model is set of Categorical-Dirichle models over object entities. The intuition behind this choice of model is simple. Every time the model sees sentence like Arthur Green educated at GT school of architecture in the data, it counts as one observation that Arthur Green attended the GT school of Treating o as a one-hot vector indicating an object, we model
p(o]s, r) = Categorical(a) ~ (irichlet
0g I
The posterior predictive for is casily computed as
I+ ō sum + σ)
p(o)s,r, Categorical
where Ø is the sum of observed one-hot object vectors for sentences beginning with "s r".
We still need to account for dopendencies between downstream and upstream properties, such as the dependence of occupation on educated at. We do so by conditioning the distribution for downstream relations upstream properties. Since upstream properties are random variables in the Bayesian model
14

---

Published in Transactions on Machine Learning Research (12/2024)
(per the previous equation), marginalize over upstream properties when computing a posterior predictive for downstream relations. Conceptually, this means when computing the probability of an occupation for Arthur Green, we marginalize over possible places of education. Mathematically, the posterior predictive over downstream properties (denoted with relation rd and object oa) is given as
p(oa 8,rd, Upstream Property) =
Σ Iloourrs ow)p(ouls;rr)
given the upstream relation Tu- As before, the posterior predictive for p(ouls, ru) is obtained by counting occurrences of each sentence "s r o", ie. computing ö. The posterior predictive for the conditional distribution ploalra, Tu. Ou) is obtained by counting the downstream properties observed for entities whenever that upstream property (ra, 0u) is also observed, e.g. counting all the observed occupations of people educated at Yale, people educated at Harvard, etc.
Now, we can create a model editing benchmark that includes exact posteriors for claims like Arthur Green occupation architect conditioned on cither (1) the pretraining data alone, or (2) the pretraining data plus requested model edit. We generate 5000 test cases by drawing a sentence from our generative model "s o" and specifying requested object o". Then, compute the probability of o* given s and with our Bayesian model, treating the sentence "s o² the minimum weight required for the posterior probability to be at least 0.95. These two different weights reflect. the level of evidence behind the new fact - while past benchmarks treat edit requests as certain, our evaluation dictates that new evidence comes with a specified weight, so that a model can rationally incorporate this evidence into its existing beliefs. An example test case is shown Fig. 4.
new observation with weight n 1000 or weight n' that is
A second key property of our data is the of logical connectives like and, or, and not. By updating a model to believe A, we should obtain immediate logical consequences for belief in statements A and B, A or B, and not A. This means that in addition to evaluating the probabilistic consequences of new information, also evaluate the logical consequences of the information. Practically, this means checking that basic axioms of probability hold, like p(not A) = 1 - p(A).. See Appendix for a full list of logical coherence metrics.
6.4 LM Pretraining
We train an 83m parameter autoregressive Transformer on the pretraining dataset described in Sec. 6.2 for a total training budget of 1 billion tokens, matching the pretraining scale in Prystawski et al. (2023). Our model shares the architecture of Mistral-7b (Mistral AI, 2023), with the architecture scaled down. We load documents in a batch size of 128 and optimize the model using AdamW with a learning rate of 5e-5. Our training budget of billion tokens corresponds epochs on our corpus. See Appendix A for further details.
6.5 Model Editing Method
The main model editing method we use is LoRA finetuning applied to MLP down-projection matrices with rank r=1. This finetuning method has been shown to be competitive with state-of-the-art model editing methods like ROME (Hua et al., 2024; Gangadhar & Stratos, 2024). More details are provided in Appendix A. Additional experiments with three other variants of model editing are provided in Appendix B.
6.6 Experimental Results
We first establish that the model fits the pretraining datasct well, i.c. it learns true lacts about our hypothetical world via the pretraining corpus. In Fig. 3, we see that our LM achieves a good fit to the pretraining data over the course of 1b training tokens. Training loss converges, and the model successfully memorizes upwards of 90% of the 100k true facts in the datasct (Generative Accuracy). The model is also able to memorize the complex sentences in the data involving true/fale /false claims, negations, conjunctions, and disjunctions, although it does not appear to learn the meanings of these connectives later results show.
Pretraining Results.
15

---

Published in Transactions on Machine Learning Research (12/2024)
Example of Coherence Metrics Under Model Editing
Probabilistic Coherence same s| same
Probabilistic Coherence same s diff'r
Probabilistic Coherence dilts) same Terr Beeemaan
Probabilistic Coherence ddIT's | dilt'r Torry Bozcmaa
Grace Stone Costes edacated of 548 salvador university
Grace Stone Costes
occupation Politician
educated at de paul universty
occupation
Television actor
0.75
0.75
Ntaget)
P(urge))
0.50
Update Stup
Uplasi Step
Update Suep
Upslate Stup
Logical Coherenec] FruciFalse
Logical Coherence (Negation
Logical Coheeraeee Disjunction 'K or true
Logical Colberence| Conjanction
not true
true
0.75
0.75
Pitaggo
P(tege))
0.50
0.50
0.35
0:35
0.25
0.55
0.:00
2/00
0.00
0.00
Uipdate Stop
Update Stop
Stop
Ulpdate
Figure 4: We edit our model to replace the fact Grace Stone Coates educated at scions with the fact Grace Stone Coates educated at san salvador university. While the model successfully learns high probability for the edit request sentence, the edited model fails to generalize properly to downstrcam entailed sentences (probabilistic coherence) or logically related sentences (logical coherence). For instance, in our hypothetical world the subject's most likely occupation should change from hollywood producer + Politician, but the LM does not respect this inference. Ideally, LLMs would achieve the same beliefs as a rational Bayesian agent (that has posterior eredences in blue and pre-update credences in red). For logical coherence, A is the edit request sentence "s o", and B is another arbitrary sentence.
Model Editing Results. Now, we assess model editing performance with our benchmark. For an example of an individual model edit, we point to Fig. 4. In this example, we edit our model to replace the fact Grace Stone Coates educated at scions with the fact Grace Stone Coates educated at san salvador university. The upper left hand plot (*same same r shows the probability (black line) of this new output (san salvador university) for the input prompt Grace Stone Coates educated at. The red line is the pre-update credence for this fact given by the Bayesian model, while the blue line is the posterior credence after we update the Bayesian model with evidence for the fact. While the edited language model successfully learns a high probability for p(san salvador university/Grace Stone Coates), the edited model fails to generalize properly to downstream entailed sentences (probabilistic coherence) or logically related sentences (logical coherence). Generalization failures are refiected by differences in the language model probabilities (black lines) and Bayesian posterior credences (blue lines); we see such failures for this example for several other kinds of data, include downstream entailed facts about the same entity ("same s diff r"), unrelated facts using the same relation ("diff s same r"), and logically related statements (bottom row, where A is the statement Grace Stone Coates educated at san salvador university). For instance, in our hypothetical world the subject's 's most likely occupation changes from hollywood producer + Politician, but the LM does not respect this inference (top row, second from the left). Similarly, we update the model to believe Grace Stone Coates educated san salvador university, but the model does not also come to believe the claim "Grace Stone Coates educated at san salvador university" is true (bottom row, first on the left).
Aggregate model editing results are summarized in Table 4, and these largely reflect the individual example seen in Fig. 4. In this table, we show three kinds of metrics: generative accuracy of the language model before and after editing (as compared against labels given by the Bayesian model), probabilistic coherence (compared against the Bayesian model), and logical coherence (evaluated according to the probability axioms in Appendix A.5). For these metrics, higher accuracy is better while we give coherence metrics as the absolute
16

---

Published in Transactions on Machine Learning Research (12/2024)
Table 4: Model editing results. Test data is split based on whether the answer to the downstream fact should change after editing. Generative Accuracy reflects whether the edited LM output agrees with the Bayesian model posterior beliefs. Probabilistic Coherence metrics are MAEs against Bayesian posterior probabilities. Logical coherence metrics reflect how the LM violates logical axioms of probability. s1/s2 and r1 r1/r2 indicate the subject and relation used in the sentence, meaning s 11 is the samn prompt as used in model editing, while s1 r2 is possible downstream fact.
Generative Accuracy
Probabilistic Coherence
Logical Coherence
Data + LM
s1rl
s1 r2
s2 r1
s2 r2
s1 r1
s1 r2
s2 m1
s2 r2
TF
neg.
and
or
All Edit Requests
Pre-edit 0.96
0.93
0.92
0.92
0.23
0.24
0.24
0.24
0.40
0.22
0.34
0.21
Post-edit
1.00
0.76
0.91
0.92
0.05
0.27
0.23
0.24
0.52
0.23
0.34
0.21
+.04
-.17
-.01
+.00
-.18
+.08
-.01
+.00
+.12
+.01
+.00
+.00
Edit Requests w/ Downstream Answer
Changes
Pre-edit. 0.90
0.97
0.91
0.98
0.20
0.34
0.21
0.34
0.38
0.22
0.34
0.22
Post-edit. 1.00
0.01
0.90
0.98
0.04
0.59
0.20
0.34
0.53
0.23
0.34
0.22
+.10
- -99
001
+.00
-.16
+.25
-.01 001
+.00
+.15
+.01
+.00
+.00
difference between the LLM probability and the rational Bayesian probability, so lower is better. The columns s1 r1, s1 r2, etc., refer to the subject and relations in the lest prompt, and can be matched against the kinds of data in the four top row plots of Fig. 4, where s1 r1 is the edit request input ("same same r"). Now, All Edit Requests, we observe that: (1) edit requests are successful for the input provided (s1 r1 columns) in terms of improving Generative Accuracy, with minimal effect on sentences about othe subjects (s2 columns); (2) trends with Generative Accuracy are repeated with Probabilistic Coherence metrics; Probabilistic Coherence shows a more precise degree of error between LM probabilities and Bayesian probabilities; and (3) based on Logical Coherence metrics, the LM does not respect basic logical axioms for probabilities, as does not coherently assign probabilities to logically related sentences before or
when looking at performance
after editing.
We highlight a subset of the data that shows what our benchmark measures that other datasets do not: Edit Requests w/ Downstream Answer Changes. On this data subset, the edit request leads the Bayesian model to update its answer for another fact, specifically the downstream fact for the same subject (refer to Sec. 6.2). Here, we see that popular editing methods totally fail to generalize to downstream probabilistic consequences, achieving just 1% accuracy on the downstream facts. These facts are probabilistic consequences, like changing someone's 's educated at property implying (but not guaranteeing) a change in their occupation, distinguishing these generalizations from those with probability (Zhong et al., 2023; Cohen el al., 2024; Powell et al., 2024). Since the consequences of these updates are not guaranteed with probability 1, it is important to measure the exact coherence between LM beliefs and Bayesian probabilities (given by Probabilistic Coherence for s1 r2 in Table 4).
7 Related Work
Critiques of Model Editing. Pinter & Elhadad (2023) present a holistic critique of model editing. Primarily, they argue that. LLMs should not be treated as editable repositories of knowledge about the world, due to scrious shortcomings in how knowledgeable and editable current LLMs arc. Instead, they recommend further work on possible alternatives to model editing, including updating document stores that are used by retrieval methods (which they recognize may lead to conflicts with parametric model knowledge), continual learning (which they describe as quite like model editing), and concept editing (which they suggest has a different scope than editing factual knowledge).
In contrast, our critique is presented from the standpoint that model editing is a necessary and important rescarch direction. In a varicty of deployment contexts, LLMs should be able to learn new facts about the world over time, and we should be able control individual beliefs in the models. Following our critique, we introduce a formal testbed for model editing where we can more clearly define the model editing problem.
17

---

Published in Transactions on Machine Learning Research (12/2024)
Model Editing with Synthetic Data. Betz & Richardson (2023) introduce a synthetic corpus for model pretraining and model editing that is also derived from formal language, as our corpus is. Unlike our language, however, their formal language is based on world where there are a small number of entities that relate to one another only by virtue of being ordered (like the natural numbers). Like in our work, their evaluation also measures probabilistic and logical coherence, with the goal of comparing the language model against certain Bayesian epistemic norms. Yet their Bayesian metries focus on how a language model's posterior probabilities compare to its own prior probabilities (a kind of self-consistency). They do not compare against an idealized rational agent (Bayesian model), we do. Therefore, we believe more naturalistic pretraining setting, while our metries focus more directly on how rational agent would respond to new evidence. Our discussion of 13 open problems for model editing help motivate the formalized settings present in both this paper and Betz & Richardson (2023).
experiments cover
8 Conclusion & Future Directions
This paper presents a critique of the standard formulation of the model editing problem and introduces a semi-synthetic testbed for model editing research. Our critique focuses on 13 open problems with model editing, divided into issues with (1) defining the problem, (2) developing datasets for the problem, and (3) treating LLMs having editable beliefs to begin with. In response to these issues, introduce a more formal, semi-synthetic setting for model editing experiments. We evaluate model editing against the standard of a rational Bayesian agent. Using data derived from Wikidata, we are able to compare an edited language model's 's probabilities against, exact Bayesian posteriors, providing a more fine-grained evaluation of model editing methods in terms of both probabilistic and logical coherence.
Future Directions. We conclude with the main problems on which we hope to see future work:
1. How can more precisely define the model editing problem for LLMs (Sec. 3)? Currently, model editing as problem is severely underspecified, making it difficult to say whether an edited LM behaves properly. 2. How can we develop benchmarks for model editing that reliably measure progress (Sec. 4)? Benchmarks need to specify what kinds of errors should be fixed, and test cases should measure whether updated LLM beliefs are appropriately confident, generalizing, and logically coherent.
3. Can we determine what kinds of LLMs should be treated as editable in the first place (Sec. 5)? When are LLMs like rational agents, as opposed agent simulators or databases? Do LLMs have existing computational mechanisms for belief revision and communicating confidence that model editing methods should be leveraging?
4. Can we use formal benchmarks for developing better model editing approaches (See. 6)? When can we say exactly what an ideal Bayesian agent would believe, and can we mimie Bayesian beliel revision by editing LLM weights?
Limitations
This paper discusses theoretical challenges with model editing and explores an empirical setting for model editing evaluation. While we describe thirteen core challenges, this is not an exhaustive list, and we must occasionally point to other work for further elaboration of the relevant issues. For some challenges we leave it to future work to introduce possible ways forward.
Second, our empirical results are first and foremost demonstration of what a more formal evaluation of model editing can look like. In doing so, we formulate one semi-synthetic data distribution and introduce one standard kind of Bayesian model to serve as idealized rational agent (i.e. to obtain exact posterior probability labels for model editing). While we may want LLMs to mimic Bayesian belief revision, there are certainly many design choices to be made in deciding which Bayesian model they should mimic, and future work can explore these design choices further. For example, a Bayesian approach could perform causal discovery to docide which relations should depend on which other relations (we assume access to the known causal graph in our model) or use hierarchical model to share evidence between related distributions (perhaps people from Harvard have similar occupations to those from Yale). Additionally, train relatively small language model
18

---

Published in Transactions on Machine Learning Research (12/2024)
corpus, and result LM is able to memorize the dataset but lacks capabilities of larger models, like some basic logical competencies. Using larger LM could show that model editing is more successful when evaluated against our Bayesian model but this is precisely the point of formalizing the evaluation, and we hope that future work can develop effective model editing methods in the formal framework we introduce.
Broader Impacts
In its most ambitious form, the goal of model editing is to control what LLMs believe to be true, including beliefs about the empirical state of the world as well as beliefs about moral values worthy goals to pursue This kind of control will be important for developing safe and adaptable Al systems. At present, model editing has been applied to important safety problems like unlearning sensitive information in LLMs (Patil et al., 2023; Li et al., 2024). In the future, model editing will be valuable for making fine-grained adjustments other beliefs in LLMs that help guide their behavior to be safer in deployment contexts, and adapt to the changing state of the world. With this direction in mind, this paper aims to point out flaws in the current model editing paradigm and demonstrate a direction for empirical research that would provide a more reliable signal for method development.
Acknowledgements
We are thankful to this paper's 's anonymous reviewers, Tom Hartvigsen, Derek Powell, Stephen Casper, Kyle Richardson, and Gregor Betz for extensive conversations and feedback on this work. This work was supported by NSF-AI Engage Institute DRL-2112635, DARPA MCS Grant N66001-19-2-4031, DARPA ECOLE Program No. HR001 12390060, NSF-CAREER Award 1846185, Google PhD Fellowship, and UNC SDSS Seed Grant. The views, opinions, and/or findings contained in this article are those of the authors and not of the funding agency.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ngee Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023. URL https://arxiv org/ pdff 087744.
Carlos E. Alchourrón, Peter Gärdenfors, and David Makinson. On the logic of theory change: Partial meet contraction and revision functions. The Journal of Symbolic Logic, 50(2):510 530, 1985. ISSN 00224812.
URL https://fitelstt wwwpssonnorg////eeeeeeeee wttgppksi//aggnno pdf.
Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghauvininejad. review on language models as knowledge bases. arXiv preprint or 3rXin:2204.00033 2022. URL pdf/2204.
Jacob Andreas. Language models as agent models. arXiv preprint orXim:2212.01699, 2022. URL https:
//arx/v.org//ddf/2222 01....111....
Gregor Betz and Kyle Richardson. Probabilistic coherence, logical consistency, and bayesian learning: Neural language models as epistemic agents. Plos one, 1822):20281372, 2023. URL https://journalls http://ooooaaas.ps....ooooooooo.
plosone/artiileeidd11 j371/journal.pone 3 0281372.
Davis Brown, Charles Godfrey, Cody Nizinski, Jonathan Tu, and Henry Kvinge. Edit at your own risk: evaluating the robustness of edited models to distribution shifts. arXiv preprint oxXin:2308.00040, 2023. URL /ttp::///aaxxv..o///d//////
00046.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference Learning Representations, 2023. URL https:://pennei het.re/..eeetooorrttttaaatrT.T.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental
19

---

Published in Transactions on Machine Learning Research (12/2024)
limitations of reinforcement learning from human feedback. arXiv preprint arXin:2307.15211 2023a. URL https://arxiv..rrg//ee http:/ddf/22/////// 252...../1111 1.217.22ppp. .df..
Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadlield-Menell Explore, establish, exploit: Red teaming language models from scratch, 2023b. URL https://arxxv..rrg// http:/dff/22303
Yanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao, He He, Jacob Steinhardt, Zhou Yu, and Kathleen McKeown. Do models explain themselves? counterfactual simulatability of natural language explanations. arXiv preprint ar vr.Xi:2307.08677 2023. URL
pttrg/ddd/3011 /2307.0877.088 pdf.
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge oditing in language models. Transactions of the Association for Computational Linguistics, 12:283 -998, 2024. URL https: https://arxiv.orrr /23077..//233111222// 12976.
Damai Dai, Dong, Yaru Hao, Zhifang Sui, and Furu Wei. Knowledge neurons in pretrained transformers. 2021. URL https://arxiv..n pdf /2104 www.66666pp..
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In EMNLP, pp. 6491-6508 Association for Computational Linguistics, November 2021. URL https//aaclanthooogy .rrg/ 2021 maannp-miinnnmmm 522.
Edoardo Debenedetti, Giorgio Severi, Nicholas Carlini, Christopher A. Choquette-Choo, Matthew Jagielski, Milad Nasr, Eric Wallace, and Florian Tramer. Privacy side channels in machine learning systems, 2023.
URL https://arxiv. httpss//axxiv.oog/pdpff. www.oo/g////////t// /230999..2330....mmm/ 0261/.09999000.... pdf.
Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Bhuwan Dhingra, Jeremy
Cohen. Time-Aware Language Models as Temporal Knowledge Bases. Transactions of the Association for Compulational Linguisties, 10:257-221 273, 03 2022. TSSN 2307-387X. doi: 10.1162/tacl 00459. URL
https://doi.org///// 11l.111tttaccaaee 0a0450
Kevin Du, Vésteinn Suæbjarnarson, Niklas Stoehr, Jennifer C White, Aaron Schein, and Ryan Cotterell. Context versus prior knowledge in language models. arXiv preprint Xrx 04466333, 2024. URL https://arxiv.org//dff
04633.
Sebastian Farquhar, Vikrant Varma, Zachary Kenton, Johannes Gasteiger, Vladimir Mikulik, and Rohin Shah. Challenges with unsupervised Ilm knowledge discovery. arXiv preprint arXix:2312.1003 2023. URL
https://arxiv.org/fff//333
Constanza Fierro, Nicolas Garneau, Emanuele Bugliarello, Yova Kementchedjhiieaa, and Anders Søgaard. Mulan: A study of fact mutability in language models. arXiv preprint arXhv:4444003336 2024. URL
https://arxiv.org//// p20./////....0222 03036.
Branden Fitelson and James Hawthorne. How Bayesian confirmation theory handles the paradox of the ravens. Springer, 2010. URL https: /ttptss///ittsexx.... wwwriiesseee.ist.ppseeessee ist. http:///cccccc..eerererrroaaaao. (0a4b8ec84586cb7bb 3cde0e3coe1133deede5500000 40400000000144eeeeee 4048ceea.
Govind Gangadhar and Karl Stratos. Model editing by pure fine-tuning. arXiv preprint arXiv:2402.11078, 2024. URL https://arxivv http://wwwww.g///////22//// pdf.2402.
Mitchell Green. Speech Acts. In Edward Zalta (ed.), The Stanford Encyclopedia of Philosophy, Metaphysics Research Lab, Stanford University, Fall 2021 edition, 2021. URL https://plato..saaffo.d..
Vinayaka Gude. Factors influencing chatgpt adoption for product research and information retrieval. Journal of Computer Information Systems, pp. 10, 2023. URL https::/www.taadfoonlireee wttpo//iennccoo////////////a///a 1080/ 08874417.2023
2288122 2280918.
Anshita Gupta, Debanjan Mondal, Akshay Sheshadri, Wenlong Zhao, Xiang Li, Sarah Wiegreffe, and Niket Tandon. Editing common sense in transformers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Longuage Processing, pp.
2023. URL http://axxxvvooooppdd/// 14956.
20

---

Published in Transactions on Machine Learning Research (12/2024)
Xiaoqi Han, Ru Li, Xiaoli Li, and Jeff Z Pan. A divide and conquer framework for knowledge editing. Knowledge-Based Systems, 279:1 10826, 2023. URL https://www.sct wtt.e/ecceeeee...oo//eeeeeeeeeee abs/pp1/995507751135777.
Sven Ove Hansson. Logic of Beliel Revision. In Edward N. Zalta (ed.), The Slanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Spring 2022 edition, 2022. URL https: aiplato.staaforre
revisionn..
Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzych Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 36, 2023. URL http:: //troce////oc htw.rocceedinnnnneriiipsss www.neee..ccceoee /ttp////eeeillllssppp/////////// patlle//pppprr22///1eeel www.aapeeeeeooeeeeeeeeeeeeee
Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Tyer. Do language models have beliefs? methods for detecting, updating, and visualizing model belicfs. arXis preprint or orXiv:211 13654, 2021. URL https://axxiv.org//ddoo
Carl G Hempel. Studies in the logic of confirmation (i.). Mind, 54(213):1 260 1945.
Peter Henderson, Eric Mitchell, Christopher Manning, Dan Jurafsky, and Chelsea Finn. Self- destructing models: Increasing the costs of harmful dual uses of foundation models. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pp. 287-296, 2023. URL https://dl. acm. org /ttp://orrr/////////////e///// 011./1100111111111 wat.a//000eaaaaaoooooaaaaaaaaaaa http:://.ooooo....../..///////00
Evan Hernandez, Belinda Li, and Jacob Andreas. Inspecting and editing knowledge representations in language models. arXiv preprint ar Xix::22340007/44 2023. URL https://arxxv.orr///ae
Jason Hoelscher-Obermatee Julia Persson, Esben Kran, Ioannis Konstas, and Fazl Barez. Detecting edit failures in large language models: An improved specificity benchmark. ar preprint arXin:2305.17558, 2023. URL https: /ttp/v/wxivv///a////a//eee/aaaaa /2305 17553.
Thomas Hofweber, Peter Hase, Elias Stengel-Fskin, and Mohit Bansal. Are language models rational? the case of coherence norms and belief revision, 2024. URL
Wenyue Hua, Jiang Guo, Mingwen Dong, Henghui Zhu, Patrick Ng, and Zhiguo Wang. Propagation and pitfalls: Reasoning- based assessment of knowledge editing through counterfactual tasks. arXiv preprint XrXiv:2401.1120 1758,, 2024. URL https://arxe: org/pd///////ppp /24f/24111111 17585.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can know what language models know? Transactions of the Associalion for Compulational Linguislics, 8:423-44 038,,438844 2020. URL https: //arxiv. wwwpiv.org/pdf/221//1 14315.pdf.
Nitish Joshi, Javier Rando, Abulhair Saparov, Najoung Kim, and He He. Personas way to model truthfulness in language models, 2024. URL https://aaxiv.orr//ff////// 18168, pdf.
Dmitrii Krasheninnikov, Egor Krasheninnikov, Bruno Mlodozeniec, Tegan Maharaj, and David Krueger. Implicit meta-learning may lead language models to trust more reliable sources. 2024. URL https: //arxiv. //taxxv.orggppd/////// org/pdff/310 15047.
Angeliki Lazaridon, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimencz, Cyprien de Masson d' d'Autume, Tomas Kocisky, Sebastian Ruder, et al. Mind the gap: Assessing lemporal generalization in neural language models. Adoances in Newral Information Processing Systems, 34:29348 29363, 2021. URL https://proceedingg wwwpeeediinnnnnuuiisss... www.eiis..c.paprr////.e.e. wwtppppppppp/////ee/ /file// 1BBf00001176f1881667777777666669 Pappee/
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pp. 333 333-342, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. 00.18653/11/K11-10544 URL http://cllanttoooogyor//////////
21

---

Published in Transactions on Machine Learning Research (12/2024)
David Lewis. Counterfactual dependence and time's arrow. Nous, pp. 455-476, 1979.
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Katnnn Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning, arXio preprint arXiv:2403.082 03218, 2024. URL https://arxiv. httgg/dff/4400 /24f.2403..00 03218.
Zhoubo Li, Ningyu Zhang, Yunzhi Yao, Mengru Wang, Xi Chen, and Huajun Chen. Unveiling the pitfalls of knowledge editing for large language models. arXiv preprint arXiv:2310.02129, 2023. URL https: //arxiv.org www. http//fff23301/ 2ttff/211..02111
Hanti Lin. Bayesian Epistemology. In Edward N. Zalta and Uri Nodelman (eds.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Summer 2024 edition, 2024. URL https: /tppato.stanford....
stanford. http///rrrreeeepppppeoooooo//r/e
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. We're afraid language models aren't 't modeling ambiguity, arXiv preprint SSt::2344144999 2023. URL https://arxiis: http:///ogg/pp///////e/// 14399.
Shayne Longpre, Kartik Perisctla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity- based knowledge confiicts in question answering. aiv preprinl Xiv:22109 05052, 2021. URL https:
//arxiv. www.v..rg///dfe httpf///1..000..
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-effiiintt fine-tuning methods. https:// teub.thu..... www..th......a.eeeeeeeeee wtw.///ggigaaac///eeeeeeeeeeeeee 2022.
Samuel Marks and Max Tegmark. The geomelry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint orXiv:2310.0653 2023. URL https://arxiv.org pdf/2310. 0Bff/310.06824.ppm..
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual knowledge in gpt. In NeurIPS 2022, 2022a. URL
05262. pdf.
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in transformer. arXiv preprint ar S5:::2210007222 2022b. URL
htrg/pdf/2210 02222222222222.2.22222222 pdf.
Beren Millidge, Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, and Rafal Bogacz. Universal hopfield networks: A general framework for singlo-shot associative memory models. In International Conference on Machine Learning, pp. 15561-15583. PMLR, 2022. URL https://proceeding/ www.gggs......rr... .llr.prss// 2152mmmllddge222mm111122222 .dd.
Mistral AI. Announcing Mistral 7B.
Blogpost, 2023.
URL https://mistrrllaai/ee
amouncing-mmistral-700
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning, Fast model editing 11309.
at scale. arxiv preprint arXiv:1 10. 11309, 2021. URL
Eric Mitchell, Charles Län, Antoine Bosselut, Christopher D) Manning, and Chelsea Finn. Memory-based model editing scale. In International Conference on Machine Learning, pp. 15817 15831. PMLR, 2022. URL https://proceediie
mlr,
Yixin Nie, Xiang Zhou, and Mohit Bansal. What can learn from collective human opinions on natural language inference data? arXiv preprint arXiw:2010.088: 03532, 2020. URL https://arxxv.org/ wwtp:///.orpppff/////1// 03532.
Yasumasa Onoe, Michael JQ Zhang, Shankar Padmanabhan, Grog Durrett, and Eunsol Choi. Can Ims learn new entities from descriptions? challenges in propagating injected knowledge. arXiv preprint arXin:2305.01651, 2023. URL https://arxxv..rrg//ne 01651.
22

---

Published in Transactions on Machine Learning Research (12/2024)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Sys- tems, 35:27730-27744, 2022. URL nttps://proceediin inrrrcceddnngssneriisss 00013330336644711144888880000000 Pappraaeentt-.pp----ooreeeeeeee .pd.
neurips ctcppppprr.illlssppperreeee//eee
Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from llms? objectives for defending against extraction attacks. orXiv preprint arXiv:2309.174100 2023. URL https://arxivv pdf/2309. 17410. pdf.
Ellie Pavlick and Tom Kwiatkowski. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677-694, 2019. URL https://direct. mit. edu/tacl/ article/doi/1e. /ttp///.1111//aaaaa.000022231311
(ee__/00999933111 0000999//43111/merreettttt sagreements ittp:///eeeetttttteeeeeeeesiiiii hrrprss----ttuuu---ee-e--t------
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),, pp. 2463-2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1 1/11//11/111///// URL https::/acllatthooog...oorr www.://-org///0900000
Yuval Pinter and Michael Elhadad. Emptying the ocean with spoon: Should we edit models? In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Pindings of the Association for Computational Linguisties: EMNLP 2023, pp. 15164-15172, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653
URL tttts::/aclanthhee wttp:looooooog/2223333/ org/2023. htt/2222...innnng...e findings-emnlp 1er.ss.emll..111111 1012.
Barbara Plank. The The'pprblem'ol human label variation: On ground truth in data, modeling and evaluation. arviv preprint arXic:2211.00557 2022. URL https://arxiv.. http://aaxxivo.ogpppooo wwtp///g//////// /ttgppff/211.0222/
Derek Powell, Walter Gerych, and Thomas Hartvigsen. Taxi: Evaluating categorical knowledge editing for
language models. orXiv preprint 2tt::244411100, 2024. URL https://arxiv.org/pdee 2204. 15004.
Ben Prystawski, Michael Li, and Noah Goodman. Why think step by step? reason- ing emerges from the locality of experience. Advances in Neural Information Process- ing Systems, 36, 2023. URL https://proceedingg http://rrceeddinnsnn.ii......... htcc/pper_filees//eceeeeeeeeeeee patp./le/aapee//22///eeee
pdf.
Xiangyu Qi, Yi Zeng. Tinghao Xie, Pin- Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine- tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.088 036693, 2023. URL
Willard V Quine and Joseph Silbert Ullian. The web of belief. 1970.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. orXio preprint arXio:2311.12022 2023. URL https://arxiv.ort wrtpp///33//d////111/ .2t.//22.1..122.ppp....
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge you pack into the parameters of language model? Tn Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418-5426, Online, November 2020. Association for Computational Linguistics. URL https://aclanthol/ org/2020.00.2 emnlp-main, 437.
doi: 10.18653/ /t00:////22000mmmmmammmmm
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel Bowman, Newton Cheng, Esin Durmus, Zac Satfield-Dodds Scott Johnston, et al. Towards understanding sycophancy in language models. arXiv preprint Xrx::22310. 13548, 2023. URL
13548.
Herbert A Simon. Rational choice and the structure of the environment. Psychological review, 63 62)2112,, 1956. URL https: htapp::///aaes.... uesd. edw/mmckenzie/ wwwp:n/eeeeeeiiinieemeee eeeeeeeeeeeeeeeeeeeeeeee..e.....
23

---

Published in Transactions on Machine Learning Research (12/2024)
Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Conrigibility, In Workshops at the Twenly-Ninlh AT Conference on Arlificial Intelligence, 2015. URL https::/cchi aaai. org/ocs/ver w59677/00124-459011888
W. Starr. Counterfactuals. In Edward N. Zalta and Uri Nodelman (eds.), The Stanford Encyclopedia of Philosophy. Metaphysics Rescarch Lab, Stanford University, Winter 2022 edition, 2022. URL https: stanford. stanforrdeed//eetrrss//eeeeee
//plfto.sstaforddd
Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristios and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. science, 185(4157):1124 1131, 1974. URL 185. 415.4157..00 htt.s//ew..................aaaa.
https///www.ssiinne.orr/////////
Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385-1470, 2021.
URL https://www.jair www.rooggg.iiiiieeeeeeeeee.eeeee http://ppppaaarrriareeeeeee
Denny Vrandečić and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Communioations of the ACM, 57(10):78-88) 85, 2014. URL https://dl.aat awm. hwt.:/or////ooflleeeeee
1145/2629489.
Alexander Wan, Eric Wallace, and Dan Klein. What evidence do language models find convincing? arXiv preprint ar Xiv:2400 1782, 2024. URL
http:/dd//220/0/ 11782
Peng Wang, Ningyu Zhang, Xin Xie, Yunzhi Yao, Bozhong Tian, Mengru Wang, Zekun Xi, Siyuan Cheng, Kangwei Liu, Guozhou Zheng, et al. Easyedit: An easy-to-use knowledge editing framework for large language models. arXiv preprint XrX::22008. 0726,, 2023a. URL https://arxxv.orr//dfff 07266..
Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, et al. Knowledge editing for large language models: A survey. arXiv preprint arXiv:3310. 16218, 2023b. URL https://arxivv org
16218.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9:176-194, 2021. URL https://dirrett mit wwt.:mittttttteeemmmmmmmm eed/taccl article/doi/18. 1tt.:////aaaa_1000//000888/00600 ott.:///000/0////////////aaAaaa0
and.
Yike Wang, Shangbin Feng, Heng Wang. Weijia Shi, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Resolving knowledge conflicts in large language models. arXic preprint arXio:2310.00085, 2023c. URL http://aaaxii.ooog///////2/ 00935.
Yifan Wei, Xiaoyan Yu, Huanhuan Ma, Fangyu Lei, Yixuan Weng, Ran Song, and Kang Liu. Assessing knowledge editing in language models via relation perspective. arXiv preprint arXiv:2311.09058, 2023.
URL https: htttp:://axxiv.orr//df/o pttp:///////....//////
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons in pretrained language models. arXiv preprinl ar arXic:2310.20138 2023. URL https: /ttps:///xxvvooo/ hhtgg/ddf/2311 wtt..or///////11.21123388
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowlodge conflicts. In The Twelfth International Conference on Learning Represenlations, 2023.
Rongwu Xu, Brian S Lin, Shujian Yang, Tianqi Zhang, Weiyan Shi, Tianwei Zhang, Zhixuan Fang, Wei Xu, and Han Qiu. The earth is flat because...: Investigating Iims' belief towards misinformation via persuasive conversation. Xixiv preprint arXiv:2312. 2023. URL
pdf.
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportumities. arXiv preprint 2rXiv:2305.131 13172, 2023. URL https://arxxvvorr////////// 13172.
24

---

Published in Transactions on Machine Learning Research (12/2024)
Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. arv preprint or Xrx:22305.11 147995 2023. URL https://arxiv..o/ wttpp/r//g//.....///////// 14795.
Kaitlyn Zhou, Jena D Hwang, Xiang Ren, and Maarten Sap. Relying on the unreliable: The impact of language models' reluctance to express uncertainty. arXiv preprint (5f::4401.06755 2024. URL
https://arxiv.orr
2201 067300
Xiang Zhou, Yixin Nic, and Mohit Bansal. Distributed nli: Learning to predict human opinion distributions for language reasoning. Xiv preprial urX:::2004.08676 2021. URL https://arxiv.ort wwt. /ddf/ ppdf/2100 221040886666
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.000: 2020. URL https://arxiv.org
A Additional Details for Formal Testbed for Model Editing
A.1 Pretraining Corpus Creation
We outline the pretraining dataset creation process in greater detail here. The steps are to (1) create a knowledge graph from Wikidata, (2) define a generative model from the knowledge graph, (3) create the pretraining data.
(1) First, we subsample Wikidata (Vrandočié & Krötzsch, 2014; Wang et al., 2021), reading the first 2m tuples, and excluding relations P735 (given name) and P131 (location in the adminstrative territorty of) and entities Q16521 (taxon) and Q5 (human). We filter the observed relations in the subset. to those that occur alongside another relation for subject entity at least 1000 times (i.e. every relation appears at least 1000 times for an entity for which another specific relation is also obscrved). We select the top 10 co-occuring relations by count. This is to ensure these is enough data for learning dependencies between certain pairs of relations. We further filter relations to be 1:1 by selecting an arbitrary object entity whenever a relation is I:N for a subject entity. This yields a total of 160k facts in a knowledge graph.
(2) To assign dependencies between relations, we compute co-occurrence rates between each pair of relations, yielding 10 x 10 matrix, where each row and column sum to 1. We select pairs of frequently occurring relations by solving linear program over this matrix, to optimize for overall frequency of paired apstream/doonstteamn /ownstream relations. This produces the pairing in Table. 2. With these dependencies determined, we define generative model for producing sentences, i.e. p(o|s, r, Upstream Property). This in done in the manner described in Sec. 6.2. We use empirical distributions from the knowledge graph. Note we set a minimum ground-truth probability to 0.6, meaning that the true fact from the knowledge graph must have al least a probability of 0.6 for p(o]s, Upstream Property = Ø). When defining p(o]s, r, Upstream Property), chock what the modal object is, and then renormalize the probability distribution that its probability is at least 0.6. This is done to make the dataset easier to memorize correct answers for. Note that about 20% of the observed facts have known upstream relation for their subject entity. This means that 80% of the facts in our data are basic facts that must be memorized, while 20% are statistically implied by the upstream properties for their subject entity.
(3) To generate the pretraining data, we randomly select subject entities from our knowledge graph and noisily sample sentences for the corpus using the generative model until we have reached 100k atomic facts. For each subject and relation, time), and select the empirical distribution to use for sampling based this (Sec 6.2). We sample only 10 objects for this fact to go into the pretraining corpus. Note we perform rejection sampling so that at least six of these 10 objects are the " "ground truth" (modal) object for their corresponding distribution, which is enforced to have probability at least 0.6 (see (2) above). This is done so that the corpus can be memorized exactly. The argmax outputs for the LM should be comparable to the ground truth objects in the gencrative model, allowing us to check generative accuracy of the language model. Besides making the 10 sentences per rs, we include logically complex sentences for each subject entity. For each (s, r), we also create 10
check whether it has a known upstream relation (this happens 20% of the
25

---

Published in Transactions on Machine Learning Research (12/2024)
Generative Accuracy 1
Probabilistic Coherence 4
Logical Coherence
Data + LM
s1rl
s1 r2
$2 r1
s2 r2
s1 r1
s1 r2
s2 r1
s2 r2
TF
neg.
and
or
All Edit Requests
Pre-edit
0.96
0.93
0.92
0.92
0.23
0.24
0.24
0.24
0.40
0.22
0.34
0.21
Post-edit
1.00
0.76
0.91
0.92
0.05
0.27
0.23
0.24
0.52
0.23
0.34
0.21
A
+.04
- .17
-.01
+.00
-.18
+ +-00
-.01
+.00
+.12
+.01
+.00
-000
Edit Requests w/ Downstream Answer
Changes
Pre-edit
0.90
0.97
0.91
0.98
0.20
0.34
0.21
0.34
0.38
0.22
0.34
0.22
Post-edit
1.00
0.01
0.90
0.98
0.04
0.59
0.20
0.34
0.53
0.23
0.34
0.22
A
4.10
-66
.01
.00
.16
+-25
-.01
4.00
+.15
4.01
4.00
+.00
Fizing Errors w.r.t.
Pretroining Facts
Pre-edit
0.00
1.00
0.83
0.93
0.26
0.23
0.22
0.23
0.33
0.22
0.30
0.20
Post-edit
1.00
0.97
0.85
0.95
0.05
0.24
0.23
0.23
0.54
0.22
0.30
0.20
+1.00
-03
+.02
+.02
-.21
+-01
+.01
4.00
4.21
4.00
4.00
+.00
Table 5: Model editing results, including edits for fixing errors in model outputs where the language model failed to learn a fact during pretraining (scction "Fixing Errors"). Test data is split based on whether the answer to the downstream fact should change after editing. Generative Accuracy reflects whether the edited LM output agrees with the Bayesian model posterior beliefs. Probabilistic Coherence metrics are MAEs against Bayesian posterior probabilities. Logical coherence metrics reflect how the LM violates logical axioms of probability. s1/s2 and r1 r1/r2 indicate the subject and relation used in the sentence, meaning s r1 is the same prompt as used in model editing, while s1 r2 is a possible downstream fact (see Fig. for an example).
True/Falss sentences (see data example 2), with True/False values in exact proportion the proportion of true sampled objects in the 10 noisy sentences. Then, we generate 20 logically complex sentences per subject, selecting mixture of and/or/mo not sentences. The 'and' and 'or' sentences make use of other random sentences from the dataset that may be true or false. The 'not' sentences randomly uses the ground truth object for (8, r) (meaning the sentence reads as: "not r o" is false) or a distractor object Otatse (meaning the sentence reads as: "not $ r Otalse" " is false). All of the and/or aad/or/noo /not sentences have correct true/false labels in the pretraining dataset, to case learning of logical connectives. Only the "s r o" and True/False sentences are noisy. This process is repeated until we have used 100k facts from the knowledge graph in the pretraining dataset. The result, is a dataset with statistics shown in Table 6.
A.2 Editing Benchmark Details
Statistic
Number
True Atomic Facts
100k
Atomic Sentences
Im
T/F Sentences
1m
and/or /not Sentences
400k
Sentences
2.4m
Documents
1.26m
Tokens
204m
Subject Entities
47k
Relations
10
Object Entities
20k
Table 6: Pretraining dataset statistics.
Once we have pretraining data, we create the editing benchmark in two steps: (1) fit a Bayesian model to the pretraining data, (2) create test. cases.
(1) Described Soc. 6.3, fitting the Bayesian model to the pretraining dataset mostly involves counting how many times observed that Arthur Green occupation architect as opposed to Arthur Green occupation carpenter. These counts used to compute the posterior predictive p(ols,r; r, Upstream Property Ø).
26

---

Published in Transactions on Machine Learning Research (12/2024)
Counting evidence for p(ols,r, r, Upstream Property) is little complicated. We wish to know how many times, e.g,, someone from Harvard is an architect, someone from Yale is an architect, etc. What makes this complicated is that educated at is itself a noisy relation in our pretraining dataset. Whether someone went to Harvard vs. Yale is often not certain, so if that person is an architect, it is not clear which conditional distribution to count this evidence for. This is issue is resolved by weighting observed occupation counts by frequencies of observed upstream properties. So if 80% of the time, when we see Arthur Green, we see Arthur Green educated Harvard, and 20% to the time we see Arthur Green educated at Yale, then we take their observed occupation counts ō and add _8 to the evidence for p(o]s, occupation, Fu educated at, o Harvard) educated at, = Yale). Recall that we aggregate
and .25 to the evidence for p(ols,r r occupation, this evidence across subject entities, and when determining someone's occupation (which is downstream of educated at in our hypothetical world) it does not mabler what the subject entity is (all subjects share the same conditional distribution). The other complicated aspect of Bayesian inference is learning from logical sentences. True/False sentences are straightforward to interpret, but and/not/oo sentences are more difficult. Here, our Bayesian model leverages the fact that these complex sentences are never noisy (always contain true atomic sentences). We compute probabilities like p(sentz is (sent, V sentz) is true), and use these probabilities as weights for learning from the individual sentences in Bayesian model. These computations are done in data_claan wwwpe/clsseeestastteeeeeeee
py in our code.
(2) To create test cases for editing, we randomly sample subject entities that were seen during pretraining, then formulate test cases of the kind shown in Fig. 2. The edit requests themselves reinforce the ground truth example in the pretraining data half the time (results for fixing errors only in Table 5) and contradict the pretraining data half the time (as typical counterfactual edit request would). The cases for measuring probabilistic coherence relate to the edit request in terms of whether they use the same or diffcrent sub- peets/relations. The possible rean relation is present in the case for s r2, and we preferentially choose test cases here where we expect the downstream answer to change when updating on the requested edit (we do this 80% of the time). The logical coherence cases use another randomly selected sentence B, and we form True/False, and, or, and not sentences based on the edit request sentence and the sentence B. For all of these sentences, the Bayesian model gives posterior probabilities after updating on the edit request. For every test case, we update the Bayesian model with the edit request, and after each test case, we reset the Bayesian model so that it once again reflects the pretraining data. This updating process uses one of two weights, nt 1000 or n' (see Sec. 6.3). The results in this paper always use n'.
A.3 LM Pretraining Details
For our language model, we use the architecture from mistrallaiMMittral--BV-0.. 1. We scale the model down to 83m parameters by sctting hidden size 512, intermediate size 512*4, num attention heads 8, num hidden layers 12. We train without dropout, using batch size of 128 documents. Training for 1b tokens takes about 24 hours in an NVIDIA A6000 GPU.
A.4 Model Editing Details
We use LoRA because it has been shown to be competitive to SOTA methods like ROME (Hua et al., 2024; Gangadhar & Stratos, 2024). In our setting, our LoRA implementation is based peft (Mangrulkar et al., 2022) and is extremely similar to ROME. We perform rank-one updates to down-projection MLP layers in our language model. Since our formal language does not have paraphrases or an "is" relation, we could not leverage the paraphrase or essence regularization losses in ROME, making the methods even more similar in our setting. We use LoRA to optimize the probability of the target token conditioned on the prompt "s r", for a total of 40 gradient steps. This is sufficient to achieve 100% accuracy on sl r1 test cases (using the exact prompt as in the edit request).
A.5 Logical Coherence Metrics
The probabilistic coherence metrics in Table are mean absolute errors between language model probabilities and Bayesian posteriors. While we show similar idcalized Bayesian probabilities for logically complex sentences in Fig. 4, the logical coherence metrics in Table 4 aim to reflect internal logical coherence of the language
27

---

Published in Transactions on Machine Learning Research (12/2024)
model. We thus compute mean absolute errors between the left and right hand sides of the following equations reflecting basic logical axioms of probability:
TF: po(o]"s r") = po (Trucl"s o" is)
not: po (True/ "s 0" is)
po (True//no "s o" is)
or: pe ("Truell "s r 01 " "32 72 02" is)
pe Po(Tree|"s; r1 01
+ (Truel ps
02" is)
Po (oTuuel|"s r1 01 is) po True " 88 12 02" is)
and: pa (True|"s) r 01 A "82 r2 02 " is)
po (TTuel rı 01 is) (o "
82 12 02 is)
We note that our model easily learns that pe PeTrue]]"s r o² is) and pa (Falsse]] r o" is) should sum to 1, so we can compute all of the above metrics using "Truc" as the target probability.
B Additional Results
Generative Accuracy f
Probabilistic Coherence
Logical Coherence |
s1 r1
s1 r2
s2 r1
s2 r2
s1 r1
s1 r2
s2 r1
s2 r2
TF
neg.
and
Embeddings
Pre-edit
0.96
0.93
0.92
0.92
0.40
0.22
0.34
0.21
0.40
0.22
0.34
0.21
Post-edit
1.00
0.29
0.75
0.91
0.51
0.28
0.34
0.24
0.51
0.28
0.34
0.24
A
+.04
-.64
-.17
-.01
4.11
+.06
+.00
+.03
+.11
+.06
+.00
+.03
LORA-all
Pre-edit
0.96
0.93
0.92
0.92
0.40
0.22
0.34
0.21
0.40
0.22
0.34
0.21
Post-edit
0.99
0.76
0.91
0.92
0.52
0.22
0.34
0.21
0.52
0.22
0.34
0.21
A
+.03
-.17
.01
.00
+.12
+.00
+.00
+.00
+.12
+.00
+.00
+.00
SGD
Pro-odit
0.96
0.93
0.92
0.92
0.40
0.22
0.34
0.21
0.40
0.22
0.34
0.21
Post-edit
1.00
0.76
0.91
0.92
0.52
0.23
0.34
0.21
0.52
0.23
0.34
0.21
A
+.04
-.17
-.01
.00
+.12
+.00
+-00
+-00
+.12
+.00
+.00
+.00
Table 7: Model editing results with different editing methods. We use embedding-onle finetuming (embeddings). LORA on all attention and MLP weights, and whole-model finetuning with SGD. Test data is split based whether the answer to the downstream fact should change after editing. Generative Accuracy reflects whether the edited LM output agrees with the Bayesian model posterior beliefs. Probabilistic Coherence metrics are MA against Bayesian posterior probabilities. Logical coherence metrics reflect how the LM violates logical axioms of probability. s1/s2 and r1/r2 indicate the subject and relation used in the sentence, meaning s1 r1 is the same prompt as used model editing, while s1 r2 a possible downsstream fact.
We conduet additional experiments with three editing methods: (1) embedding-only finetuning, (2) whole- model finetuning, and (3) LORA all attention and MLP weights. The (1) method should represent weak baseline, as finetuning embeddings only should warp the meaning of subject and relation embeddings. Meanwhile, (2) and (3) should be stronger.
We show the results in Table for all edit requests. The accuracy columns show agreement between the edited LM and labels given by the Bayesian model. The last four columns are logical coherence metrics, given as the absolute difference between the LM probability and what a logically consistent model should output (see paper for more detail).
28

---

Published in Transactions on Machine Learning Research (12/2024)
Indeed, when compare results between the methods, reasonable gradation in performance between methods. Finetuning embeddings overgeneralizes to test cases with the same subject or relation, and does more damage to the model's logical consistency. Meanwhile, whole model finetuning and LORA achieve similar performance, with a slight tradcoff between editing the output for the edit request (s1/r1) and overgeneralizing other cases whose answers should not change (s2/rl and s2/r2). All methods perform extremely poorly at generalizing downstream facts whose answer may change following the edit request (s1/12), a finding unique our benchmark.
Generative Accuracy
Coherence Error +
s1 r1
TF
neg.
All Requested Changes
Pre-edit
0.96
0.40
0.22
Post-edit
1.00
0.52
0.23
+.04
+.12
+-01
Initially Logically Coherent Data
Pre-edit
0.99
0.02
0.03
Post-edit. A
1.00
0.75
0.05
+.01
+.73
+.02
Table 8: Comparison of editing performance all requested changes versus points with initially logically coherent probability assignments across T/F and negated versions of facts. Even for points with initial logical coherence, weight editing can destroy logical coherence across T/F versions of a claim ("X is truc") after updating the model.
29


Reviews:
[{"review_id": "vX0FYf8ptF", "reviewer": "Reviewer_rqA9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The categorization proposed here is clearly explained. I believe this is a useful attempt that will help future discussion on model belief updating. Perhaps categories 5.1 or 5.2 are bordering a bit on the philosophical, or contain anthropomorphisms (discussing the *aims* of LLMs), which makes them less practical. But the questions they posit are still useful, in my opinion.\n- The proposed model editing dataset is quite large, making it well-suited for (small) large language model experiments - even if the sentence structure is rigid. \n- The benchmark task involves relations that do not have 1-1 mappings per se, which makes it well-suited for evaluating updates to probabilities. It also seems more realistic than the synthetic corpus from Betz and Richardson.\n- I believe benchmarking model editing, and trying to quantify \"beliefs\" held by language model weights is useful and interesting for the TMLR audience. \n\nWeaknesses:\n- Although the benchmark task is a useful test setting for model editing methods, it is much simpler than real world cases. The number of relations is small (10), and the structure of the sentences is rigid. If an editing method does well on this benchmark task, it does not necessarily mean it will do well on real world unstructured text. In other words, the task may be too synthetic to be practical.\n- Only one editing method is tried here, it would be stronger to try at least two: one weak baseline, and one method considered by the community to be a strong editing method. If the benchmark results show a meaningful difference between these two methods, it gives me additional confidence that the benchmark task can serve as proxy for real world model editing tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work is a constructive criticism of model editing, or what the authors call \"belief revision\": meaningfully revising knowlege captured by a generative model. The authors categorize possible issues with belief revision, including questioning whether generative models have beliefs to begin with. They show examples and explanations of these categories. This includes highlighting issues with current benchmarks, e.g. having ambiguous or incorrect ground truth labels. \n\nTo address some of the concerns and issues, this work introduces a dataset that can be used to test model editing performance. This benchmark task involves fitting a Bayesian model to the output of a generative model, and measuring whether the update in probabilities matches how a rational agent would update its beliefs. The authors' experiments indicate that - assuming LLMs can indeed hold beliefs - model editing through LoRA-based finetuning can easily result in inconsistent beliefs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- For section 6: is there a case where the model belief corresponds to the rational pre-update credence? If a generative model cannot get the belief for \"A is true\" and \"not A is true\" to complement each other, then it may not provide much signal for an editing method either. \n- Please provide a bit more information for the columns in Fig 4., it took me longer than I'd like to admit to fully understand how the \"same r\", \"diff s\" labels corresponded to the example sentence. Since this figure presents a qualitative editing result that shows the core issue, I think you can afford to spend more text on it, similar to table 4. \n- In section 6, the authors try one editing method (LoRA based finetuning), and at the end of section 6 highlight in bold that 'model edits totally fail to generalize'. Given that this sentence is in bold, it would be more accurate to say that this specific editing method fails, lest readers believe this is a general claim about model editing methods.", "broader_impact_concerns": "Agreed with current impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NeatKPIQb9", "reviewer": "Reviewer_ZXqR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- It is a good survey on the challenges for model editing, summarizing into three interesting view points with a lot of details. I feel the summary could be a nice contribution to the field.\n\n- The semi-synthetic dataset is another good contribution in that it is well designed to control the setting by sampling tuples from Wikidata, and the metrics for coherence is a sound method based on Bayesian posteriors. I think the experiments are solid to demonstrate the current model editing capabilities. \n\nWeaknesses\n\n- Although the semi-synthetic dataset is a good contribution, it is not immediately clear what open problems the dataset is addressing in the summary of open problems mentioned in section 3 through 5.\n\n- This work is somehow limited in that it is focusing on task settings, and not discussing potential methods for model editing besides fine-tuning with LoRA. Although I understand that methodologies are out of the focus of this work, it would be good to add some discussion on how researchers challenges the problem.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work summarizes key challenges, i.e., open problems, in model editing for large language models and proposes a semi-synthetic dataset to benchmark the performance of model editing. The problems are summarized in terms of three view points, task settings, benchmark dataset construction and future impacts. The benchmark dataset is constructed from Wikidata by focusing on subset of relation pairs, e.g., occupation and education, which are related with each other, i.e.g, downstream relation and upstream relation. The synthetically created dataset is used to train an LLM to evaluate the performance of model editing by altering particular relations and by measuring the coherence of relations. Experiments show that simply editing a tuple does not reflect the changes in downstream relations, demonstrating the limitation of a naive fine-tuning based editing.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I'd like to find more discussion on how the proposed dataset is addressing the open problems mentioned in section 3 through 5.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KXjcgfH304", "reviewer": "Reviewer_fxL7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper discusses the challenges of model editing in LLM. It first defines 12 different challenges that fall into three categories: defining the model editing problem, benchmark characteristics, and how to update the LLM belief (if this is possible). Then, it presents a possible benchmark.\n\nThe paper is interesting for its key idea. It wants to discuss many different problems that should be solved to perform model editing. It then leaves the solution of each of these problems to future work.\nHowever, in my opinion, the paper lacks a proper discussion of possible approaches. Some of the proposed problems have already been addressed by other authors, as evidenced by the references given, but they are usually only mentioned instead of being analysed properly (what strengths and weaknesses should be exploited or avoided, for example).\n\nIn the end, the document presents a possible formal test case. Although this is an important strength of the paper, I believe that a more thorough discussion could help the reader gain a deeper insight into the results obtained and what the benchmark aims to test against the problems listed and discussed above. \nIn this section, Table 4 should be discussed in detail, along with the meaning of consistency. For example, if I have not missed anything, the arrows say that a lower level is better, but why?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper discusses the challenges of model editing in LLM. It first defines 12 different challenges that fall into three categories: defining the model editing problem, benchmark characteristics, how to update the LLM belief (if this is possible). Then, it presents a possible benchmark.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Perhaps another possible problem is missing: could the addition of information lead the model to forget other, not necessarily conflicting, information?\nThis does not seem to me to be a problem considered by any of the others already listed, but I could be wrong. I would like to know what the authors think.\n\nMinor issue:\nOn page 8 and on page 10, the first row of sections 5 and 6 should refer to Table 1", "broader_impact_concerns": "I think the Broader Impact section is fair enough. I do not have any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers appreciated the paper, the interaction with the authors and the revisions made. The reviewers hope that minor concerns not fully addressed in the revision will still be addressed in the final minor revision.

Quoted from Reviewer rqA9:

I think the benchmark task is a useful attempt at measuring editing performance. There are still some minor concerns:

* authors added a section connecting issues to benchmark properties, but the connection is a bit superficial for some,
* the task is a bit synthetic (while they did update the number of relations, making it more realistic)
* the authors only try out three variations of the same simple editing method
Decision: Accept (poster)