Emptying the Ocean with a Spoon: Should We Edit Models?
Authors: Yuval Pinter
Michael Elhadad
Venue: EMNLP 2023 Findings (2023)
Keywords: Model Editing
LLMs
Factual Knowledge
Continual Learning
Knowledge Representation
Opinion Paper
TL;DR: We raise concerns about the method of model editing, identifying risks it poses and highlighting alternatives.

Abstract: We call into question the recently popularized method of direct model editing as a means of correcting factual errors in LLM generations. We contrast model editing with three similar but distinct approaches that pursue better defined objectives: (1) retrieval-based architectures, which decouple factual memory from inference and linguistic capabilities embodied in LLMs; (2) concept erasure methods, which aim at preventing systemic bias in generated text; and (3) attribution methods, which aim at grounding generations into identified textual sources. We argue that direct model editing cannot be trusted as a systematic remedy for the disadvantages inherent to LLMs, and while it has proven potential in improving model explainability, it opens risks by reinforcing the notion that models can be trusted for factuality. We call for cautious promotion and application of model editing as part of the LLM deployment process, and for responsibly limiting the use cases of LLMs to those not relying on editing as a critical component.

Emptying the Ocean with Spoon: Should We Edit Models?
Yuval Pinter and Michael Elhadad
Department of Computer Science Ben-Gurion University of the Negev
Be' er Sheva, Israel {uvp, elhadad@ecs. bgu. ac. il
Abstract
ent mismatch between their pre-training objective and the desire for factuality. In recent years, re- scarchers have come up with several remodies to the problem of nonfactual LLM outputs, one which being model editing, where parameters in- side an LLM are tweaked based on individual facts marked for correction. These works focus solv- ing problems such ensuring stability of other factual outputs following the editing, or batch edit- ing, or computationally efficient editing. In this opinion paper. we call into question the entire idea of model editing, citing concerns about intended use cases, conceptual scalability, potential for bias, safety, and overall accountability. We advocate us- ing models that have explicit knowledge modules for tasks requiring factual knowledge, and other existing techniques to mitigate the need for model editing in the first place. Having said that, we acknowledge the usefulness of direct editing for certain use cases such as interpretability probes, and recommend limiting editing to such scenarios.
We call into question the recently popularized method of direct model editing means of correcting factual errors in LLM generations. We contrast model editing with three similar distinct approaches that pursue better defined objectives: (1) retrieval-New based architectures, which decouple factual memory from inference and linguistic capabilities embodied in LLMs; (2) concept erasure methods, which aim at pre- venting systemic bias in generated text; and (3) atrribution methods, which aim at grounding generations into identified textual sources.
We argue that direct model editing cannot be trusted as systematic remedy for the disad- vantages inherent to L.LMs, and while it has proven potential in improving model explain- ability, it opens risks by reinforcing the notion that models can be trusted for factuality. We call for cautious promotion and application of model editing as part of the LLM deployment process, and responsibly limiting the use cases of LLMs those not relying editing as a critical component.
2 Model Editing
1 Introduction
Sinitsin et al. (2020) first introduce the notion of up- dating large ML models to account for local perfor- mance expectations motivated externally. They cite cases where mistakes are critical, like object detec- tion in self-driving cars. Later work suggests model editing can aid in protecting privacy and eliminat- ing bias (Zhu al., 2020), and a measure of "catching up" with sensesves facts such as "PM of the UK" changing over time (Bommasani et al., 2021; Mitchell et al., 2022), or as a means of under- standing the mechanics of black box models (Meng et al., 2022). Sinitsin et al. (2020) specify several desired properties of editing methods: reliability (the target facts updated as intended), local- ity (other changes don't happen; measure for this property is termed drawdown), and efficiency (successful edits are computationally light); later work (De Cao et al., 2021) added generality (the
Large language models, or LLMs, have taken the NLP world by storm. After originally focusing on training them as a vehicle for transfer learning, re- cent advancements have cast LLMs in the role of one-stop shop, all-knowing oracles. One particular finding that has contributed this reframed usage is the apparent factuality properties of pre-trained LLMs: somehow, mere noxt prediction train- ing has produced models that can complete certain correct facts about the world when asked to do so. LLMs have been perceived by the public at large replacement for search engines, and the scant disclaimers appearing in commercial services of- fering LLM querying do not appear to have made a dent in this perception.
Despite these developments, LLMs of course keep making mistakes. There is, after all, an inher-

---

ability to modify models not originally trained for knowledge retention), consisteney (robustness to paraphrasing, in the specific use case of text mod- els), and an unspecified property we'll call frugal- ity (only minimal components of the model are changed during editing). A well-studied limitation, which most of these works focus mitigating,, and related to locality, is catastrophic forgetting (Rat- cliff, 1990), i.e., ensuring that edited model does not lose performance on tasks for which it was explicitly trained and performs well on.
study were later studied and this result was shown to not be robust against multi-token spans (as op- posed to single token answer; Kalo and Fichtel, 2022), in an open vocabulary setting (as opposed to a closed vocabulary of 21K tokens; Roberts et al., 2020), and when queries vary (Kassner and Schüitze. 2020); indicating that the LAMA ex- periment relied on heuristics to predict answers. Since then, more realistic benchmarks have been introduced, as well as more robust querying tech- niques (Jiang et al., 2020), to address some of these limitations. As the scale of LLMs increased, recent work has also scaled up the size of benchmarks, shifting the focus to facts concerning rare entities, where even recent LLMs struggle (Kandpal et al., 2023; Sun et al., 2023). These experiments indicate that an LM's ability to answer question depends on the number of times information relevant to that question appears in the pretraining data. As a re- sult, LLMs cannot be safely used to answer queries about the long-tail of facts that have been rarely mentioned in pretraining (Mallen et al., 2023).
Methods for model editing have evolved from editable training (Sinitsin et al., 2020), a procedure requiring an a-priori decision that a model would changing
later be edited, to the Docality-motivane of specifie parameters within models (Zhu al., 2020; Meng et al., 2022). Recent work (Mitchell et al., 2022) draws attention to the danger of model performance degradation accumulating over the course of many successive edits, and seeks mitiga- tion through improved methods. Hase et al. (2023b) extend the consistency requirement to hold over en- tailed and equivalent facts well as paraphrases, suggesting model editing way of reconciling cases where certain facts are produced correctly but those entailed from them are not.
Beyond the capability reliably answer queries, LLMs should also fulfill other requirements in order to be considered as fact reposito- ries (AlKhamissi et al., 2022): the ability to edit knowledge (add, delete, update facts), logical con- sistency (answers to different but related facts must agree), reasoning (the ability to infer additional answers the basis of logical rules), and explain- ability and interpretability (supporting an answer through convincing chain of arguments). Experi- mental results assessing these dimensions indicate that current LLMs fail on all of them. He et al. (2023) demonstrate that LLMs underperform on computing ontology subsumption inference when compared with other trained approaches such as NLI models and symbolic systems such as OWL Reasoners (Glimm al., 2014).
3 Critique
In this section, we present the case against model editing as a practice, regardless of the performance obtained by any single method. We begin with an analysis of the premise underlying the model edit- ing research objective: the hypothesis that LLMs can be used as fact repositories. We then focus on reasons for which editing facts cannot be de- signed a-priori as a means to maintaining fact- providing LLMs, and continue with practical con- siderations for why even this ill-posed goal is prob- ably unattainable.
The evidence supporting the premise that LLMs can be used as fact repositories is, thus, currently weak. Beyond this fragile foundation, we focus be- low on specific arguments regarding the capability to edit models as if they were fact repositories.
3.1 LLMs as Knowledge Repositories?
The perception that LLMs can behave as knowl- edge repositories was first formulated and exper- imentally supported with the introduction of the LAMA benchmark (Petroni et al., 2019), where pretrained language models were queried in zero- shot setting against 51K knowledge triplets ex- tracted from knowledge banks and reformulated as fill-in-hhebblank statements. 2019-Ievel mod- cls (BERT-XL) got the top answer correctly about 26.5% of the time. The limitations of the LAMA
3.2 Systemic Mismatch
A very basic property of LLMs which contrasts with their usage as knowledge repositories is their stochastic nature. When used for augmenting cre- ative endeavours, for data exploration, or for free- form tasks such as summarization, this property

---

is desirable: the use case calls for variation or un- expectedness. In other cases, we may be content with models that supply us with a distribution of outputs, from which we can estimate the probabil- ity of individual responses and adjust our expecta- tion for reliable oulput. Since the latter is absent from many models available only through 3rd-party APIs, we are left with obtaining text generated from unknown distribution, which we argue is insuf- applications."
3.3 Architectural Implausibility
Estimates vary wildly, but there are over 100 mil- lion notable facts in the world. Even the cut- off for what constitutes a fact is unclear. Does .3% change in a demographies statistic, new esoteric sports record, call for edit? Do the daily whereabouts of world leaders constitute facts? What about those of celebrities or journal- ists? As events unfold daily world politics, eco- nomics, sports, and many other walks of life, facts are added and changed in larger quantities and rates than can be plausibly "caught up with" through sur- gical model editing, akin to emptying the occan with spoon. 5 If we choose to limit ourselves in which facts we find important enough to edit, we introduce bias into the system, opening the door to a host of documented harms which permeate many language technologics (Chang et al., 2019; Blodgett et al., 2020). This choice can be implicit as well as explicit, and is incredibly hard to avoid. In similar vein, the vastness and variability of facts is likely to lead to bias in evaluating the com- plement set of edits, those facts controlled for as not changing following an edit (drawdown). Even paraphrases of edited facts are not guaranteed to change alongside the selected phrasing (De Cao et al., 2021), nor are entailed facts (Hase et al., 2023b). This problem also manifests as a safety issue, since unchecked facts can turn out to be quite important for model usage, but perhaps taken for granted (or simply not explicitly covered) when designing a drawdown benchmark.
ficient for nictuality-dependee can even be argued that getting facts wrong is a feature of vanilla LLMs rather than a bug: as their core training procedure aims to simulate plausible continuation of text, it should not surprise us that models would repeat wisely-assumed falsehoods in way that negates the idea of using them for factual purposes. If most people think, and write, that Los Angeles is the capital of California, 2 an LLM supposed complete a relevant prompt accordingly. There is also no built-in reliability or robustness in LLMs that sample outputs from a distribution: two instances of the same prompt can easily produce contradicting facts, and indeed often do.
Additionally, the idea of editing facts in a model suggests that we always want a model to supply us with fact as an answer to question. However, at times, questions may be posed while pre-supposing otherwise assuming harmful propositions such as stereotypes or conspiracy theories. Editing the "fact" associated with the question "which govern- ment agency faked the moon landing?" would not provide us with an improved model; what we may want is to remove such facts altogether, or provide the model with a means of challenging the pre- supposition, or avoiding giving any answer at all. At the same time, many relations that we would term "facts" can be argued to be vital notions with- out which certain kinds of basic communication is impossible, An LLM that cannot assert whether trees have leaves, or asserts that they never do, is in danger of becoming irrelevant for most tasks requiring any kind of interaction with the world. As philosophy and practice surrounding these ques- tions progresses, can hope this gap between 'must-know' and 'must-not-know' will eventually tighten form workable bounds on LLM knowl- edge capacity.
There is evidence (Mallen et al., 2023; Jang et al., 2021) that facts above a certain "popularity thresh- old", measured in terms of views on Wikipedia articles, are harder to edit out of models compared to facts lying on the long tail of view distribu- tions. Inherently being out of sight, the unpop- ular facts thus become susceptible to the double risk of being edited alongside target facts while not being deemed important enough to be checked during drawdown testing. The ultimate outcome of such procedures can be a monolithization of LLM MLMsupppiedd "knowledge" that focuses on cer- tain popular domains and interests while losing all usefulness in many topics contributing to the exten- sive diversity of the human and natural experience.
*107.323.022 appear in Wikidata as October 17, 2023, conforming to the definition notable items used in https: / //www. wawaat..org/wikk.. (arkkiiHeeppIiee Items.
'Called transparent-sensiiee tasks" (Luo al., 2023).
"The capital of California Sacramento. "W thank reviewer for making this point.
'Atrrbuted, according to the Internet, as a "Yiddish proverb", hut, hey, who knows?

---

Empirical evidence indicates existing editing models fail to properly account for the ripple effect of a fact editing operation (Cohen et al., 2023). For example, the insertion of the fact " "Jack Depp is the son of Johnny Depp" introduces a "ripple ef- fect" in the form of additional facts that the model needs to update (e.g. "Jack Depp is the sibling of Lily-Rose Depp") ") Results in symbolic approaches this task have demonstrated that this knowledge updating task is of high computational complex- ity, even NP-hard, for example in Truth Mainte- nance Systems (TMS; Rutenburg, 1991). These results carry to approaches based on machine learn- ing techniques (Knoblauch et al., 2020). There is, therefore, theoretical basis to conclude that model editing will at best address the problem of consis- tent updating in roughly approximate manner, and most likely fail to update rarely seen facts within the ripple effect of editing operations.
editing, they still suffer from their own limitations. but we argue that they identify more promising research directions.
Incorporating Knowledge Bases In retrieval- based models, factual knowledge is explicitly rep- resented in a dedicated component external to the LLM. The way this external fact store is repre- sented and combined with the LLM varies: it can be collection of textual documents that is searched using text retrieval component, or an RDF graph, or encoded as set of vector em- beddings, or represented as modular expert LMs trained on curated datasets. In all cases, in the retrieval baaeed approach, the model can explicitly cite the source which underlies a specific genera- tion, and let the user decide its credibility.
Once external (non-parametric) knowledge is retrieved, it must be composed with the LLM gen- eration process. Khandelwal et al. (2020) intro- duce a k-nearest neighbor method with interpola- tion; RETRO (Borgeaud et al., 2021) combines the prompt with retrieved documents through a spe- cialized cross-attention mechanism; other LLM-IR variants include fusion-in-decoder method (AT- LAS; Izacard et al., 2022) and TRIME (Zhong et al., 2022), all retrieval- based models that main- tain the capacity for few-shot in-context learning. Recent work addresses the integration of LLMs and IR by learning to combine results from search engines into the context provided to the LLM to answer specific question (Xu et al., 2023). SAIL (Luo et al., 2023) introduces an instruction fine-tuning method to ground language generation on search results and learn to select trustworthy sources. CooK (Feng et al., 2023) approaches the task of combining multiple curated modular knowl- edge sources into an integrative system, where sources are modeled as independent LMs and an in- tegrator LLM combines the information from these modules to answer a question.
Finally, recent empirical findings show addi- tional weaknesses of edit methods by extending the evaluation suites to cover aspects beyond fact editing metrics, such specificity and robust- ness of the models post-editing (Onoe al., 2023; et al., 2023; Hase et al., 2023a; Brown et al., 2023).
4 Alternatives to Model Editing
Model editing is motivated by the desire to control the text generated by LLMs to make them more compliant with desirable outcomes, specifically to control factuality: when an LLM generates text ex- pressing an incorrect or obsolete fact, remove the fact from the LLM's "memory" and try again (Peng et al., 2023). Other relatively direct approaches to improve LLM factuality (and, in general, to control and revise the text generated by LLMs) have been proposed, most notably Reinforcement Learning with Human Feedback (RLHF; Bai al.. 2022; Ouyang al., 2022: Ramamurthy et al., 2023; Rafailov et al., 2023; Carta et al., 2023), or adding a factuality objective to the training procedure (Lee et al., 2022). We now survey approaches which define a more achievable objective than that pur- sued by direct model editing or output manipula- tion. These approaches avoid the assumption that the LLM is a fact repository, and thus steer clear of the attempt update this repository in a logi- cally consistent manner, which is computationally hard to achieve and verify. While these alterna- tives avoid the more problematic aspects of model
In all of these approaches, factual knowledge is stored outside of the parameters of the LLM and can be manipulated without retraining the LLM. These approaches have been shown to be scalable in the number of facts. Editing the fact store means the same as updating database, thus simplifying the formulation of the task.
Retrieval-based models, arguably, do not resolve all of the concerns we identify with model editing. The problem of identifying the provenance of a given generation span in these combined models

---

remains acute: the text can be determined by the internal LLM parameters, by the external stores, or by their combination, even if they are not logically consistent with one another. Facts that have been seen more often during LLM training may have more weight in this interpolation even if they are wrong or when they become obsolete. Zhu et al. (2020) claim that "modifyyng knowledge in explicit memory module networks like FaE (Verga al., 2020) is not easier and demands changes in the LLM component as well." This effect was also identified in the RealQA baseline test (Kasai et al., 2022): in this benchmark containing questions about time-sensitive data, experiments showed that in many cases GPT-3 (Brown et al., 2020) properly integrated data retrieved from KB and injected into the context, but often still generated text based on outdated data from the LLM. While the clear separation of the components and the identification of the composition of external knowledge improve sespaarency-sensttvve tasks, the objective of iden- tifying provenance in the generated text and con- trolling which sources are appropriate in given context remains an open research question. The formulation of the task of attributing generated text identifiable sources (AIS; Rashkin et al., 2022) is a key contribution this direction. Neeman et al. (2022) also address an aspect of this issue with a counterfactual data augmentation technique to disentangle contextual and LLM knowledge when generating text.
Concept Erasure The goal of concept erasure (Elazar and Goldberg, 2018; Ravfogel et al., 2020; Belrose et al., 2023) to remove unwanted bias from embeddings generated by LLMs and sub- sequently from the generated text. This goal is motivated by fairess objectives: preventing pro- tected attributes from causally affecting text gener- ation. This motivation is somehow related to that of model editing (prevent damage from generated text) but key differences exist: (1) the method ad- dresses general, well-identified concepts (protected attributes such as gender, race, age) as opposed to less well-defined specific facts; (2) it operates as a post-hoe transformation of embeddings, as op- posed to a modification of the model itself, and as such it allows for more accountability than an ever-changing model; (3) it is verifiable with well- defined metrics over limited pre-declared scope. While concept erasure has more limited scope than model editing, it defines an objective that can be evaluated with aggregated metrics in a robust man- ner. One possible avenue of future research can be to examine whether the erasure approach can be extended to address specific aspects of factuality, such as temporal validity.
Maybe it's better to have a model that knows what it doesn't know? As identified in Kasai et al. (2022), a prerequisite to avoid generating wrong text is to identify what is not known: "can an open-domain QA system identify unanswerable cases?" The related issuc of unanswerable ques- tions has been addressed in robust way (Rajpurkar et al., 2018; Sulem et al., 2021, 2022); yet the chal- lenge in the context of L.MMs and factuality is that problematic questions like those specified in $3.2 do look answerable.
Continual training focuses on incrementally training a model by introducing new tasks or new domains (e.g.. Razdaibiedina et al., 2023). Model editing does not directly fall within this objec- tive, as it concerns updating precise elements in the model while keeping tasks and domains un- changed. Yet, the identification of drawdown in model editing is similar the risk of catastrophic forgetting identified in continual learning. Within this approach, we could situate model editing as a type of re-training or post-training. Zhu et al. (2020) note that just fine-tuning over a set of facts- to- to-upd costs in degradation on other facts. Jang et al. (2021) identify the problem and propose to upply techniques from continual training to the task of incremental updating of LLM knowledge. Over- all, while continuous training may seem a priori avoid the risks of the model editing approach, seems to suffer from many of the main evaluation problems.
Conclusion
We agree that model editing is an attractive task to define, with clear benchmarks and expected out- comes. However, in current practice contributes towards unrealistic expectations that we can solve problems like LLM hallucinations, which would lead to potential harm in unleashing use cases that are not in fact within the capabilities of L.LMs alone. We advocate for the usage of retrieval-augmented methods and other structural and post-hoc meth- ods in order achieve the stated large-scale goals, while conceding the benefits of editing to "safer" applications such as model interpretability and ro- bustness checking.

---

Acknowledgments
References
Badr AlKhamissi, Millicent Li, Asli Celikyilmaz. Mona T. Diab, and Marjan Ghazvininejad. 2022. A review on language models as knowledge bases. ArXiv, abs/2204.06031.
We thank Sarah Wiegreffe for comments on carlier drafts. We thank the reviewers for their valuable feedback. Yuval Pinter was supported in part by the Israeli Ministry of Innovation, Science and Tech- nology (Grant 2022/5451).
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk. Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston. Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei. Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022. Training a help- ful and harmless assistant with reinforcement learn- ing from human feedback.
Limitations
This is an opinion paper. We have purposely not made empirical analyses in order to support our critique, knowing that data contestable and may vary according to its collection methodology. We aim to convince on the merit of examples and rhetorical argumentation rather than concrete evi- dence, mostly out of our reach for the models under consideration, which we nevertheless urge readers and practitioners to seek and use in attempts to either support or disprove our claims.
Nora Belrose, David Menneider-Josephh Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023. Leace: Perfect linear concept crasure in closed form.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5454- 5476, Online. Association for Computational Lin- guistics.
Ethical Considerations
As we were finalizing the publication version of this paper, the Hamas organization launched a mur- derous attack very close to our home on October 7, 2023. This attack has killed over 1,000 civilians in horrendous manners, including many members of university and their familics, some of our stu- dents have been kidnapped and are still without news about their status. We are still personally un- der constant attack by rockets launched by Hamas and Hczbollah towards civilian areas (nearly 8,000 rockets have been launched in the past two weeks). Hundreds of thousands of Israeli civilians have been evacuated from their homes to ensure their safety. Palestinian civilians in Gaza are also victims of this war. This situation has prevented from fully engaging in our research and polishing this paper. Our scientific domain does not provide us with ways to clarify the moral dilemmas that war triggers. The situation in our region stirs strong emotions worldwide, and the media is abuzz with analyses, often lacking depth, knowledge or simple factuality. Given the inability of domain experts to bring clarity when it is so badly needed, we call for humility in technical analysis regarding factu- ality in LLMs. We do not know what factuality is. However, words and lies harm, and the hatred they distill kills.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx Michael S Bernstein, Jeannette Bohg, Antoine Bosse- lut, Emma Brunskill, et al. 2021. On the opportuni- ties and risks of foundation models. arXiv preprint
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, -ean--aptistte Lespiau. Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy. Jacob Menick, Roman Ring, T. W. Hennigan, Saffron Huang, Lorenzo Maggiore, Chris Jones, Al- bin Cassirer, Andy Brock, Michela Paganini, Gcof- frey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Fiich Elsen, and L. Sifre. 2021. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning.
Davis Brown, Charles Godfrey, Cody Nizinski, Jonathan Tu, and Henry Kvinge. 2023. Robustness of edited neural networks.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Thomas Carta. Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer.

---

2023. Grounding large language models interac- tive environments with online reinforcement learning. arXiv preprint arXiv:2302 02662.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick. Jane A. Yu, Armand Joulin, Sehastian Riedel, and Edouard Grave 2022. Few-shot learning with retrieval augmented language models. ArXiv, abs/2208.03299.
Kai-W Wei Chang. Vinodkumar Prabhakaran, and Vicente Ordonez. 2019. Bias and fairness in natural language processing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process- ing and the 9th International Joim Conference on Natural Language Processing (EMNLP-IJCNIPP) Tutorial Abstracts, Hong Kong, China. Association for Computational Linguistics.
Joel Jang, Seonghyeon Ye, Sohee Yang. Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2021. Towards contin- ual knowledge learning of language models. Inter- national Conference on Learning Representations. abs/21 10.03215.
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2023. Evaluating the ripple effects of knowledge editing in language models. arXiv preprint arXiv:2307.1299
Zhenghao Jiang. Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-433
Nicola De Cao. Wilker Aziz, and Ivan Titov, 2021. Edit- ing factual knowledge in language models. In Pro- ceedings of the 202] Conference on Empirical Meth- ods in Natural Language Processing, pages 6491 6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jan- Shn-Christoph Kalo and Leandra Fichtel. 2022. Kamel: Knowledge analysis with multitoken entities lan- guage models. In 4th Conference on Automated Knowledge Base Construction.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In Proceedings of the 40th International Conference on Machine Learning, volume 202 Proceedings of Machine Learning Research, pages 15696-15707. PMLR.
Yanai Elazar and Yoav Goldberg. 2018. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 11-21, Brussels, Belgium. Association for Computa- tional Linguisties.
Shangbin Feng, Weijia Shi, Yuyang Bai, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. 2023. Cook: Empowering general-purpose language models with modular and collaborative knowledge. ArXiv, abs/2305.00 .09955.
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui, 2022. Realtime qa: What's 's the answer right now?
Birte Glimm, Ian Horrocks, Boris Motik, Giorgos Stoi- los, and Zhe Wang. 2014. Hermit: An owl 2 reasoner. Journal of Automated Reasoning, 53:245-266
Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 7811-7818. Online. Asso- ciation for Computational Linguistics.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghan- deharioun. 2023a. Does localization inform editing? surprising differences in causality-baad based localization vs. knowledge editing in language models.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models.
Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zor- nitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. 2023b. Methods for measuring, up- dating, and visualizing factual beliefs in language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Compu- tational Linguistics, pages 27 2714-2731, Dubrovnik, Croatia. Association for Computational Linguistics.
Jeremias Knoblauch, Hisham Husain, and Tom Diethe. 2020. Optimal continual learning has perfect mem- ory and is np-hard. In International Conference on Machine Learning, pages 5327-5337. PMLR.
Yuan He, Jiaoyan Chen, Ernesto Jimenez-Ruiz, Hang Dong, and lan Horrocks. 2023. Language model analysis for ontology subsumption inference. In Find- ings of the Association for Computational Linguis- tics: ACL 2023, pages 3439-3453, Toronto, Canada. Association for Computational Linguistics.
Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2022. Factuality enhanced language models for open-ended text generation. ArXiv, abs/2206.00624..
Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tian- hua Zhang, Yoon Kim, Xixin Wu, Danny Fox, He- len Meng, and James Glass, 2023. Sail: Search- augmented instruction learning. arXiv preprint 152255.
Jason Ohermaier Julia Persson, Eshen Kran, Toannis Konstas, and Fazl Barez, 2023. Detecting edit failures large language models: An improved specificity benchmark.

---

Alex Mallen, Akari Asai. Victor Zhong, Rajarshi Das. Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric mem- ories.
building blocks for natural language policy optimiza- tion. In The Eleventh International Conference on Learning Representations.
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Ture, and David Reitter. 2022. Measuring attribution in natural lan- guage generation models.
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associ- ations in gpt. Advances in Neural Information Pro- cessing Systems, 35:17359- 17372.
Roger Ratcliff. 1990. Connectionist models of recog- nition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher Manning. 2022. Fast model editing at scale. In ICLR.
Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Sypektor, and Omri Abend. 2022. Disentqa:
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guard- ing protected attributes by iterative nullspace projec- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256, Online. Association for Computational Linguistics.
Disentangling parametric and contextual knowledge with counterfactual question answering. ArXiv, abs/2211 1,05655.
Yasumasa Onoe, Michael J. Q. Zhang, Shankar Padman- abhan, Greg Durrett, and Eunsol Choi. 2023. Can Ims learn new entities from descriptions? challenges in propagating injected knowledge.
Anastasia Razdaibiedina, Yuming Mao, Rui Hou, Ma- dian Khabsa, Mike Lewis, and Amjad Almahairi. 2023. Progressive prompts: Continual learning for language models. ArXiv, abs/2301.12314.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin. Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426, Online. Association for Computational Linguistics.
Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie. Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint Xrxv:22302.12813
Vladislav Rutenburg, 1991. Complexity classification of truth maintenance systems. In STACS 91: 8th An- nual Symposium on Theoretical Aspects of Computer Science Hamburg, Germany, February 14-16, 1991 Proceedings 8, pages 372-383. Springer.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry Pyrkin, Sergei Popov, and Artem Babenko. 2020. Editable neural networks. In International Conference Learning Representations.
Elior Sulem, Jamaal Hay. and Dan Roth. 2021. Do we know what don't know? studying unanswer- able questions beyond squad 2.0. In Conference on Empirical Methods in Natural Language Processing.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly reward model.
Elior Sulem, Jamaal Hay, and Dan Roth. 2022. Yes, or idk: The challenge of unanswerable yes/no ques- tions. In North American Chapter of the Association for Computational Linguistics.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In Annual Meeting of the Association for Computational Linguistics.
Kai Sun, Xu, Hanwen Zha, Yue Liu, and Xinhsuai Dong. 2023. Head-to-tail: How knowledgeable are large language models (1lm)? a.k.a. will lims replace knowledge graphs? ArXiv, 004/2308.10166
Rajkumar Ramamurthy, Prithviraj Ammanabrolu. Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannanch Hajishirzi, and Yejin Choi. 2023. Is reinforcement learning (not) for natural language processing:
Pat Verga, Haitian Sun, Livio Baldini Soares, and William W. Cohen. 2020. Facts experts: Adapt- able and interpretable neural memory over symbolic knowledge.
Benchmarks, baselines, and

---

Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and seng Chua. 2023, wards the accurate, credible and traceable content generation for complex knowledge- knowledge-intensive inteesiiv tasks. arXiv preprinl urXiv:2304.11 14732.
in-the-chain: To-
Zexuan Zhong, Tao Lei, and Danqi Chen. 2022. Train- ing language models with memory augmentation. In Conference on Empirical Methods Natural Lan- guage Processing.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models. arXiv preprint arXi::201..00363.


Reviews:
[{"review_id": "GwnbD4nC9A", "reviewer": "Reviewer_8HBp", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review!\n\nWe will definitely consider the relevant links you have provided as well as classical complexity results from Assumption-based Truth Maintenance Systems (ATMS are NP-complete - Vladislav Rutenburg. Complexity classification of truth\nmaintenance systems. In STACS, pages 372 – 382, 1991).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides a very strong case against existing model editing formulation. A nice survey of method have been presented regarding existing model editing methods and problems associated with them such as fact popularity bias, catastrophic forgetting and robustness to changes.\n\nThe paper also briefly discusses alternate approaches aimed at mitigating the hallucination issue by decoupling the factual memory component from LLM by relying on external knowledge stores, factual knowledge bases and posthoc editing of model parameters.\n\nThe objective mismatch between model pretraining and desired control at inference is very well argued with suitable citations", "reasons_to_reject": "No major weaknesses in the position paper however, though not mandatory it would have been nice to provide some more grounding for the arguments and ciatations against implausibility of model editing.\n\nMore theoretically grounded works that show that model editing is not posssible (like unlearning has been shown to be theoretically hard which involves model editing) and also how continual learning is hard (https://arxiv.org/abs/2006.05188) would further support the arguments better. \n\nMore recent citations on grounding LLM outputs might be relevant like:\n\nhttps://arxiv.org/abs/2302.12813\n\nhttps://arxiv.org/pdf/2302.02662.pdf", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses the important topic of model editing, which aims to update an existing model to elicit a desired outcome. The paper discusses eloquently of the drawback and systemic issues in the existing formulation of the model editing problem and how it violates certain properties necessary for trust in such systems. The paper also discusses interesting research directions of slightly different formulations aimed at grounding LLM generated outputs. The paper offers interesting arguments and perspective overall.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QSteDtx09e", "reviewer": "Reviewer_Y4br", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Reaction to Reasons To Reject\n\n> I find the discussion around alternatives to model editing a bit unsatisfying overall:\n> while the content does cover in a rather comprehensive way the possible alternative paths towards factuality in LMs (e.g., retrieval, grounding, continual training, concept erasure, etc.), the discussion is overall closer to a literature survey and I don't find much critical opinions on the way forward for these alternative directions and approaches of resolving the challenges in these directions.\n\nThe main point of our position paper is a critique of the \"model editing\" strategy together with its underlying assumption that an LLM\ncan be considered as a knowledge base or \"fact repository\". The alternatives we survey explicitly **reject** this assumption that LLMs are \"fact repositories\".\n\nWe feel that we do provide a critical assessment of the proposed alternatives - mainly: retrieval-based models (284-317), concept erasure (343-351). \n\nWhile retrieval-based models seem to simply avoid the problem altogether (by separating the \"fact repository\" from the LLM), they do not completely avoid it for the reasons we enumerate - and thus, the concerns we enumerate deserve further attention.  \n\nIn contrast, concept erasure does not suffer from the risks we identify.\n\n> One example is the issue regarding the difficulty of attributing generations to internal parameters/external stores as discussed in line 284-304, which is arguably a problem of central importance of this topic. In the end, the authors don't seem to lay out a solution forward, and it seems that these alternative paths face challenges somewhat similar to those in model editing.\n\nIndeed, this is a position paper. We aim at raising researchers' awareness of the risks associated with specific research objectives.\nWe do recommend to aim for solutions that do not consider LLMs to be fact repositories, and instead to make efforts to attribute generations to identifiable sources - which remains an open research objective, but one that does not intrinsically rely on the problematic assumption that LLMs are fact repositories.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The arguments in this paper are overall interesting and quite convincing, with adequate supporting evidence from research studies and other sources. The topic of factuality in LMs is also of central importance in NLP, especially with the recent rapid developments of large LMs and the new trend of LM-based search engines.", "reasons_to_reject": "I find the discussion around alternatives to model editing a bit unsatisfying overall - while the content does cover in a rather comprehensive way the possible alternative paths towards factuality in LMs (e.g., retrieval, grounding, continual training, concept erasure, etc.), the discussion is overall closer to a literature survey and I don't find much critical opinions on the way forward for these alternative directions and approaches of resolving the challenges in these directions. One example is the issue regarding the difficulty of attributing generations to internal parameters/external stores as discussed in line 284-304, which is arguably a problem of central importance of this topic. In the end, the authors don't seem to lay out a solution forward, and it seems that these alternative paths face challenges somewhat similar to those in model editing. I would expect to see more fruitful and critical thoughts & discussions around these key topics, which I believe should be present in a position paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this position paper, the authors argue that pursuing model editing-based methods for improve the factuality of (large) language models is not a hopeful path, with the following arguments: \n\n1) The probabilistic nature and training objectives of LMs are inherently unsuitable for factual purposes. For example, LMs are in some sense expected to imitate frequent falsehoods in the training data, and there is no built-in consistency or robustness for LM generation.\n\n2) There are tremendously many facts being updated everyday, and editing model knowledge for all of them is not practical. It's also inevitable that bias would be introduced if we selectively choose which facts to edit.\n\n3) Extensive model editing could lead to catastrophic forgetting and other performance downgrades, as found empirically in recent work.\n\nBased on these, the authors suggest that alternative approaches such as retrieval-augmented/attributed generation and concept erasure are more promising.\n\n", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kQp7FXgttU", "reviewer": "Reviewer_UmZH", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Reaction to \"Reasons to Reject\"\n\n> R1: The paper blurs together inherent limitations of LLMs with limitations pertaining to model editing.\n> In particular, the argument in Section 3.1 is not even really about model editing. It's about LLMs and/or LLM APIs.\n\nThe points we make in 3.1 are about considering LLMs as \"fact repositories\" - which they are not. It is not about LLMs in general - but about approaches that assume LLMs \"contain facts\" and build from there. Section 3.1 challenges the presupposition underlying model editing.\n\n> For Section 3.2, the same argument could be used to argue against LLMs --\n> and even against the whole idea of indexing the Web and using it to learn things about the world!\n\nThe arguments in 3.2 do not pertain to LLMs as artifacts that encapsulate \"knowledge of language\" - but instead as LLMs perceived as \"knowledge of the world\".  \n\nThere is a fundamental difference between Web indexes and LLMs: Web Indexes contain a repository of identifiable resources.\nThey return identified links to resources. If you remove or blacklist a resource, it will not be returned by the search engine.\nWhen you add a new resource, it can be returned.\nLLMs once they are trained explicitly lose the coupling between sources and generations through generalization.\n\n> Search engines are vexing for the paper's arguments concerning \"factuality\". All the concerns raised about factuality hold for traditional search, including the arguments in 3.1. If lots of people on the Web say that Los Angeles is the capital of California, then n-gram-based search will also surface those results. The issue here is conflating factuality with grounding in a database. AI people should never promise factuality, but they should be able to deliver grounding or provenance.\n\nRegardless of the method used by search engines, the results returned by a search engine include links to the source resources.\nIf the link refers to https://qanon - it is now the responsibility of the reader to give it credence (or not).\nWe do agree on the basic desirable property - *grounding or provenance* - we take it for granted that search engines do provide provenance - LLMs do not.\n\n> The argument in 3.2 (and the metaphor of \"emptying the ocean with a spoon\") needs to be further substantiated. After all, changing a single node in a knowledge graph can have wide-ranging effects that spread automatically through the network. For example, changing the name of the U.S. president could change the name of the dog owned by the president's mother's first cousin, assuming the knowledge graph has the right edges and so forth. If we could train LLMs to have this kind of structure, then we would be able to change a lot of things with a small database of facts. But, anyway, all of this needs to be properly benchmarked against what we have now with Web search, Wikipedia editing, etc.\n\nIndeed - changing facts in knowledge bases is extremely complex. It used to be the topic of active research in Truth Maintenance Systems and was shown to be an NP-complete hard computational problem (Vladislav Rutenburg. *Complexity classification of truth maintenance systems*. In STACS, pages 372 – 382, 1991). There is no a priori reason to believe LLMs provide a foundation to achieve this goal.  Yet, model editing methods are proposed when as you state, it is quite a problem to even define a proper benchmark to measure these effects in a comprehensive manner.\n\n> I appreciate that section 4, on alternative approaches, notes that these alternatives suffer from all of the major problems that the paper identifies with model editing. But this is also a signal that the paper has set its sights too high.\n\nThe alternatives we survey explicitly **reject** the assumption that LLMs are \"facts repositories\" - this is what makes them \"better alternatives\" in our analysis.  Yet while retrieval-based models seem to simply avoid the problem altogether (by separating the \"facts repository\" from the LLM), they do not completely avoid it for the reasons we enumerate (mainly because of the risk introduced by the \"composition\" component) - and thus, the concerns we enumerate deserve further attention.  \n\nIn contrast, concept erasure does not suffer from the risks we identify.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a spirited discussion on a timely topic, and I appreciate the forthright tone that the paper takes when assessing both the editing techniques and the alternatives that the authors prefer (which have major shortcoming as well).\n", "reasons_to_reject": "I should say that I share the authors' overall view. I too think that model editing is the wrong bet to make, for numerous reasons. I favor the retrieval-augmented approaches described in section 4.\n\nThat said, I feel that the current paper misses the mark. The argument needs to be substantially sharpened in order to have a productive impact:\n\n1. The stated goal of the paper is to \"call into question the entire _idea_ of model editing\". This is incredibly ambitious. I would not recommend such a sweeping scope for such a short paper. This would require some kind of elegant argument that these approaches cannot, even in principle, serve positive goals. The paper does not offer such an argument.\n\n2. The paper blurs together inherent limitations of LLMs with limitations pertaining to model editing. In particular, the argument in Section 3.1 is not even really about model editing. It's about LLMs and/or LLM APIs. For Section 3.2, the same argument could be used to argue against LLMs -- and even against the whole idea of indexing the Web and using it to learn things about the world!\n\n3. Search engines are vexing for the paper's arguments concerning \"factuality\". All the concerns raised about factuality hold for traditional search, including the arguments in 3.1. If lots of people on the Web say that Los Angeles is the capital of California, then n-gram-based search will also surface those results. The issue here is conflating factuality with grounding in a database. AI people should never promise factuality, but they should be able to deliver grounding or provenance.\n\n4. The argument in 3.2 (and the metaphor of \"emptying the ocean with a spoon\") needs to be further substantiated. After all, changing a single node in a knowledge graph can have wide-ranging effects that spread automatically through the network. For example, changing the name of the U.S. president could change the name of the dog owned by the president's mother's first cousin, assuming the knowledge graph has the right edges and so forth. If we could train LLMs to have this kind of structure, then we would be able to change a lot of things with a small database of facts. But, anyway, all of this needs to be properly benchmarked against what we have now with Web search, Wikipedia editing, etc.\n\n5. I appreciate that section 4, on alternative approaches, notes that these alternatives suffer from all of the major problems that the paper identifies with model editing. But this is also a signal that the paper has set its sights too high.\n\nI would encourage the authors to find a more focused sort of criticism and rewrite the paper accordingly. As I said above, I am on their side!", "questions_to_authors": "", "paper_topic_and_main_contributions": "This is a position paper arguing against LLM model editing approaches. The particular target of criticism is the class of methods in this space that seek to directly edit model parameters so that the model's behavior reflects a new set of facts (and their consequences). The central criticisms of these approaches offered in the paper is that LLMs are trained with an objective that doesn't support factuality, and that editing approaches won't scale to the number of facts that need to be changed. The paper describes alternative approaches to the core problem of ensuring that our systems produce factual, up-to-date responses.\n", "excitement": 2, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FWX1TURCKg", "reviewer": "Reviewer_HGZK", "summary": "", "questions": "What do you believe are the main contributions and points of novelty in this paper?\n\nDoesn't the fact that so many recent works are investigating the category of methods you are promoting not indicate that progress is already being made in a direction that would alleviate your main concerns?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Answers to Questions For The Authors\n\n**Q1: What do you believe are the main contributions and points of novelty in this paper?**\n\n1. We do not claim for novelty - this is a position paper. We aim to critique a general research approach around LLMs which we identify as ill-defined and risky and aim to raise awareness around the associated risks.\n2. The hope the conclusion (364-376) summarizes our main contribution:   \n> We propose that model editing is an attractive task to define, with clear benchmarks and expected outcomes, however in current practice it contributes towards unrealistic expectations that we can solve problems like LLM hallucinations, which would lead to potential harm in unleashing use cases that are not in fact within capabilities of LLMs alone. \n> We advocate for the usage of retrieval-augmented methods and other structural and post-hoc methods in order to achieve the stated large-scale goals, while conceding the benefits of editing to “safer” applications such as model interpretability and robustness checking.\n\n**Q2: Doesn't the fact that so many recent works are investigating the category of methods you are promoting not indicate that progress is already being made in a direction that would alleviate your main concerns?**\n\nWe do not deny the fact that researchers active within the \"model editing\" approach identify and solve technical issues.\nWe instead argue that the underlying assumption that LLMs are fact repositories that can be edited is problematic and leads to specific risks that we aim to identify.\n\n## Reaction to \"Reasons to Reject\"\n\n> Across the papers that they cite, the main concerns raised by the authors seem to already be recognised and possible solutions are being investigated. e.g. Mitchell et al., 2022a highlight the problems arising after many successive edits and then actually propose a potential solution that uses an external system for storing content edits and reason over them when needed (of the category of systems championed by the authors).\n\nSee response to Q2.\n\n> The moon landing example given on lns 146-157 seems a bit contrived. Prompting ChatGPT, for example, with the question they pose does in fact produce the answer they desire. Also, much of the motivation for fact-oriented model editing is to account for clearly defined instances with a known/desired answer.\n\nThe way ChatGPT produces the desired answer is not through model editing - it is most likely through RLHF post-processing (although who knows how proprietary products work). One should not lump together all approaches aimed at avoiding offensive or harmful or untrue LLM output as \"model editing\" - these are different strategies (see our intro to Section 4 213-225).\n\nThe point we make is that the identification of \"clearly defined instances\" is problematic (it assumes one can enumerate all the *facts* to be processed or rank facts by importance - points we discuss in 3.2). Facts can be introduced in prompts as presupposition or in counter-factual constructs or implied and still lead to specific risks.\n\n> Even though the authors acknowledge it in the limitations section, the fact that this paper also contains no empirical results or examples illustrating the issues discussed doesn't do much to strengthen their argument. If concrete examples of problems resulting from model editing, or some data illustrating the extent to which it has been adopted under the assumption of blanket reliability, were included it would be easier to accept the gravity of their concerns.\n\nWe believe that the risks we identify are specific - for example the three arguments in 178-189, 190-204, 205-211 are all supported with specific sources.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and clearly outlines concerns surrounding the model editing methods in question. The authors provide a concise summary of recent works that have developed both model editing and more post-hoc/retrieval based techniques. ", "reasons_to_reject": "While much of the authors' concerns re model editing seem reasonable, it is unclear what exactly they have identified or are proposing that is novel. Across the papers that they cite, the main concerns raised by the authors seem to already be recognised and possible solutions are being investigated. e.g. Mitchell et al., 2022a highlight the problems arising after many successive edits and then actually propose a potential solution that uses an external system for storing content edits and reason over them when needed (of the category of systems championed by the authors).\n\nThe moon landing example given on lns 146-157 seems a bit contrived. Prompting ChatGPT, for example, with the question they pose does in fact produce the answer they desire. Also, much of the motivation for fact-oriented model editing is to account for clearly defined instances with a known/desired answer.\n\nThe fact that so many alternative methods exist and are referred to makes it somewhat difficult to see what the need for this paper is, when the issues proposed are clearly already being addressed in the ways suggested. \n\nEven though the authors acknowledge it in the limitations section, the fact that this paper also contains no empirical results or examples illustrating the issues discussed doesn't do much to strengthen their argument. If concrete examples of problems resulting from model editing, or some data illustrating the extent to which it has been adopted under the assumption of blanket reliability, were included it would be easier to accept the gravity of their concerns.", "questions_to_authors": "What do you believe are the main contributions and points of novelty in this paper?\n\nDoesn't the fact that so many recent works are investigating the category of methods you are promoting not indicate that progress is already being made in a direction that would alleviate your main concerns?", "paper_topic_and_main_contributions": "The authors call into question the idea that attempts should be made to edit models in order to satisfy factuality requirements, and propose that we rather use dedicated knowledge modules and other post-hoc techniques. They go on to explain their rationale and summarize some recent works that explore alternative strategies.", "excitement": 2, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors write a convincing position paper against utilization of editing techniques to modify model output to be more desirable. Posthoc approaches can have unchecked effects on other axes of model knowledge and generation, and requires more reflections, as identified by authors. While reviewers point out that the alternate approaches to editing that authors contribute are not substantial, I don’t think that is needed or the central point of the paper. As a lot of work quickly evolves to do more ‘editing’, this work comes as a very timely counterpoint and will benefit the community.
I would encourage the authors to reflect further on the alternative they propose as well as the need of ‘factual checking’ to contextualize their position.
Decision: Accept-Findings