Venue: NAACL 2025

# Has this Fact been Edited?

Detecting Knowledge Edits in Language Models

Paul Youssef\({}^{\dagger}\)   Zhixue Zhao\({}^{\circ}\)1  Christin Seifert\({}^{\dagger}\)    Jorg Schlotterer\({}^{\dagger\ddagger}\)2

Footnote 1: [https://github.com/paulyoussef/deed](https://github.com/paulyoussef/deed)

###### Abstract

Knowledge editing methods (KEs) can update language models' obsolete or inaccurate knowledge learned from pre-training. However, KEs can be used for malicious applications, e.g., inserting misinformation and toxic content. Knowing whether a generated output is based on edited knowledge or first-hand knowledge from pre-training can increase users' trust in generative models and provide more transparency. Driven by this, we propose a novel task: detecting knowledge edits in language models. Given an edited model and a fact retrieved by a prompt from an edited model, the objective is to classify the knowledge as either _unedited_ (based on the pre-training), or _edited_ (based on subsequent editing). We instantiate the task with four KEs, two large language models (LLMs), and two datasets. Additionally, we propose using hidden state representations and probability distributions as features for the detection model. Our results reveal that using these features as inputs to a simple AdaBoost classifier establishes a strong baseline. This baseline classifier requires a small amount of training data and maintains its performance even in cross-domain settings. Our work lays the groundwork for addressing potential malicious model editing, which is a critical challenge associated with the strong generative capabilities of LLMs.2

Footnote 2: Corresponding author

+Marburg University, \({}^{\diamond}\)University of Sheffield, \({}^{\dagger}\)University of Mannheim

{paul.youssef, joerg.schloetterer, christin.seifert}@uni-marburg.de zhixue.zhao@sheffield.ac.uk

## 1 Introduction

Large Language Models (LLMs) encode knowledge about the world via their pre-training data Petroni et al. (2019); Roberts et al. (2020). However, the encoded knowledge can be inherently flawed or become outdated over time De Cao et al. (2021); Youssef et al. (2024). Additionally, practitioners might want to tailor LLMs by incorporating domain-specific knowledge pertaining to their products or to facilitate new applications. Driven by these needs and given the prohibitive cost of pre-training a new model, multiple methods have been developed to efficiently update knowledge in LLMs Meng et al. (2022, 2023); Li et al. (2024a).

Knowledge editing methods (KEs), such as ROME Meng et al. (2022), MEND Mitchell et al. (2022), and MEMIT Meng et al. (2023), change the model's internal weights to adapt facts. However, these methods can also be used to conduct malicious edits in LLMs Wang et al. (2023). For example, KEs can be used to inject fake news (cf. Figure 1) or embed harmful or undesired biases in LLMs Mazzaia et al. (2023); Halevy et al. (2024); Youssef et al. (2025). This danger becomes more pronounced as the number of capable open-source LLMs increases. For example, any user can easily upload their model via one line of Python code to the HuggingFace Hub without validation and testing. While fact-checking and hate speech detection represent plausible avenues to combat these malicious uses Zhao et al. (2021); Pan et al. (2023), these approaches require clearly defined objectives, as well as customized annotated datasets and models. Instead of utilizing multiple specialized methods to address the variety of malicious uses, we take a vigilant approach: directly detecting knowledge edits. Detecting knowledge edits additionally provides transparency to users in general applica

Figure 1: Illustration of malicious knowledge editing, and our task of detecting knowledge edits. The cross-domain setting (dashed box) facilitates detection without known edits in the target LLM.



tion scenarios by revealing to them the source of LLM's outputs (edited vs. unedited), and holds potential value for enhancing the controlled generation of LLMs Pan et al. (2023); Sun et al. (2023). Moreover, detecting edits can help identify whether malicious outputs stem from poisoned pre-training data or from post-hoc artificial modifications.

This work is the first to propose the task of **d**etecting knowledge **e**dits in LLMs (DEED). We provide three detection baseline models and a comprehensive evaluation, including cross-domain settings and generalization across datasets. Our key contributions are as follows:

* We propose a novel task for detecting knowledge edits in LLMs (DEED) to combat potential malicious model editing. DEED highlights the transparency of LLMs' generations by suggesting the source of the generated text (edited vs. unedited) (Section 4).
* We instantiate the task with two knowledge-editing datasets (zsRE, CounterFact), four state-of-the-art knowledge editing methods (ROME, MEMIT, MEND, MALMANN), and two LLMs (GPT-J, GPT2-XL) yielding 16 settings for initial analysis. The framework is easily extensible to other KEs and LLMs (Section 4.2).
* We analyze the hidden state representations and probability distributions of edited and uncudited facts. In locate-and-edit KEs we find a strong increase in output probabilities of edited facts and linear separability in hidden state representations between edited and uncudited facts, while other KEs induce more subtle changes (Section 5.1).
* We show that simple AdaBoost classifiers, with the hidden state representations and probability distributions from the edited LLMs as features, establish strong baselines, while requiring a limited amount of training data (Section 5.2). Additionally, we show that these baselines generalize to cross-domain settings, highlighting their practical utility (Section 6).

## 2 Related Work

This paper is the first to establish the task of detecting knowledge edits in pre-trained language models. Still, a wide array of work studies related questions, including how knowledge is stored in LLMs Gurnee and Tegmark (2023); Gurnee et al. (2024); Niu et al. (2024), how knowledge is probed Youssef et al. (2023), how to inject new knowledge Xu et al. (2023) and how to unlearn existing knowledge Yu et al. (2023); Kassem et al. (2023); Jang et al. (2023); Patil et al. (2024). For brevity, we direct readers to surveys by Mazza et al. (2023) and Wang et al. (2023) for KE-related work. This study focuses on the post-editing phase, i.e., after applying KEs that directly modify model parameters.

### Knowledge Editing Methods

Generally KEs can be divided into two groups: 1) parameter-preserving methods and 2) parameter-modifying methods.

#### Parameter-preserving methods

integrate extra modules, e.g., additional memory space or parameters, while leaving the original parameters unchanged Yao et al. (2023); Li et al. (2024). SERAC Mitchell et al. (2022) stores new knowledge in an explicit cache and uses an auxiliary classifier to determine if a prompt is associated with the edited knowledge. If it is, SERAC passes the prompt along with the stored new knowledge to a "counterfactual" model for the inference. GRACE Hartvigsen et al. (2023) caches embeddings for old knowledge and the weights for new knowledge and adds an adaptor to manipulate the layer-to-layer transformations without altering model weights. MELOYu et al. (2023) learns and stores new knowledge in inserted LoRA blocks Hu et al. (2021). We exclude parameter-preserving KEs from our study, since for these KEs it is possible to undo the edits by simply removing the external modules from the model.

#### Parameter-modifying methods

modify the model parameters directly via meta-learning or optimization-based methods. Meta-learning KEs (ML-KEs) employ hyper-networks to learn parameter shifts towards new knowledge. KNOWLEDGE EDITOR De Cao et al. (2021) and MENDMitchell et al. (2022) build light-weight hyper-networks, i.e., naive multi-layer perceptrons that take the gradients of desired input-output (new knowledge) and output the parameter shifts for individual layers. However, the parameter shifts caused by different edits can be contradictory to each other, leading to a cancellation effect Tan et al. (2024). To handle the cancellation effect in batch editing, MALMN Tan et al. (2024) treats aggregating parame 

ter shifts as a least square problem, seeking for the parameter shift to be effective for all edited facts.

Optimization-based, also known as locate-and-edit methods (LE-KEs), assume that knowledge is memorized in a key-value form in the feed-forward network. These methods, such as ROME Meng et al. (2022) and MEMIT Meng et al. (2023), locate and then directly optimize the parameters in the feedforward network to modify or add memories.

Note that fine-tuning the model on specific data roughly falls into the category of parameter-modifying KEs, but requires great computational resources and data curation costs, and may encounter catastrophic forgetting Mitchell et al. (2022a,b); Zheng et al. (2023). These negative effects may result in an unusable model in practice. We leave detecting edits that have been induced through finetuning to future work, and focus on methods that have been specifically developed to edit knowledge.

### Malicious Modifications of LLMs

Recent advancements in LLMs have led to their widespread adoption. However, these models are vulnerable to malicious attacks Wei et al. (2024); Mo et al. (2024) and poisoning Li et al. (2012); Cao et al. (2023); Liu et al. (2023); Li et al. (2024). Moreover, open-source LLMs are available for free download and unrestricted modification, facilitating easy dissemination of modified versions across online platforms Falade (2023); Singh et al. (2023); Yao et al. (2024).

With the development of KEs, knowledge of LLMs can be modified easily with low cost, and at scale. This development presents opportunities for misuse, such as inserting misinformation and toxic content into LLMs. For example, BadEditLi et al. (2024)formulates backdoor injection as a lightweight knowledge editing problem, i.e., modifying a subset of parameters to inject backdoors into LLMs. Due to the high cost of training LLMs, users are likely to seek readily available similar models on public resources such as GitHub and HuggingFace. Even though these third-party models might offer improved skills and advanced features (e.g., by fine tuning with Reinforcement Learning from Human Feedback (RLHF) or specific datasets), they could also be mingled with malicious edits before being made public Shi et al. (2023); Pan et al. (2023); Li et al. (2024).

## 3 Preliminaries

Facts2 in LLMs are commonly represented as triplets of \((subject,relation,object)\), or \((s,r,o)\) for short. Querying an LLM with a prompt \(p(s,r)\), where \(p\) expresses the relation \(r\) and contains the subject \(s\) (e.g., "The Eiffel Tower is in the city of"), should result in retrieving the object \(o\) (e.g., "Paris"), given that the fact \((s,r,o)\) is known to the LLM. A knowledge editing operation \(E(s,r,o,o^{\prime},p)\) is successful if it changes the behavior of the LLM such that the retrieved object is \(o^{\prime}\), as desired, instead of \(o\). If an edit is conducted with a prompt \(p\), then ideally it should also affect semantically similar (paraphrased) prompts that express that same relation, i.e., the new object \(o^{\prime}\) should be retrievable with \(p^{\prime}(s,r)\) as well, where \(p^{\prime}\) is a paraphrase of \(p\). We later use only paraphrased prompts to detect edited facts, since we assume no access to the prompts that have been used for editing. Paraphrased prompts are available in both datasets we consider.

Footnote 2: We use the terms _facts_ and _knowledge_ interchangeably.

## 4 DEED Task

The goal of detecting edited facts is to distinguish outputs generated based on post-hoc modifications to the model (edited knowledge) from those derived from the knowledge acquired by the model during its pre-training phase (unedited knowledge). In this section, we introduce the task of **d**tecting knowledge **e**dits in LLMs (DEED) in detail (Section 4.1), describe how we instantiate the task (Section 4.2), and discuss practical applications (Section 4.3).

### DEED General Setup

Given an edited model and a prompt that retrieves a certain fact from the model, the objective of DEED is to decide whether the fact is based on pre-training, i.e., _unedited_, or based on subsequent editing, i.e., _edited_.

More formally, we model the task of detecting knowledge edits as a binary classification problem with classes \(\mathcal{Y}=\{edited,unedited\}\). Given an LLM \(\mathcal{M}\) with unknown edits, and a pool of facts \(\mathcal{T}_{test}=\{k_{1},..,k_{n}\}\), where \(k_{i}=(s_{i},r_{i},o_{i},p_{i}),\) for \(1\leq i\leq n\), the task is to assign each fact \(k_{i}\) a label \(y\in\mathcal{Y}\).

In a supervised setting, the DEED classifier has access to a training set \(\mathcal{T}_{train}\) with examples of edited and unedited facts. The facts are represen 

as a set of features \({\cal F}\). Conceptually, one can categorize the task based on the origin of the test set and the type of features; their combination defines the difficulty of the task.

**Origin of the test set.** If the test set is derived from the original edited model, i.e., \({\cal M}={\cal M}_{e}\) (used also for training), our task is In-Domain (ID) classification.When the test set originates from a similar LLM \({\cal M}={\cal M}^{\prime}_{e}\), which shares the same foundation model with \({\cal M}_{e}\) (the model used to train the classifier), our task is Cross-Domain (CD) classification. For example, \({\cal M}_{e}\) and \({\cal M}^{\prime}_{e}\) can be two different instruction-tuned versions of the same foundation model (cf. Figure 1).

**Type of features.** We consider two cases corresponding to whether we have full access to the model or only can observe its output distribution. Features can therefore either be obtained from the hidden states (HS), or from the output probability distribution (PD) of the LLM \({\cal M}\). Additionally, we consider having both features (HS + PD).

Practical considerations.A few state-of-the-art LLMs (e.g., GPT-4 [2]) are only accessible via their APIs. Nonetheless, the number of high-performing open-source LLMs is steadily increasing [30]. The availability of these open-source LLMs allows users to further fine-tune and modify them locally, with access to their hidden states and probability distribution. In theory, the probability distribution can even be approximated for closed-source LLMs as well if enough outputs are sampled from these LLMs given a fixed input.

In the ID setting, the user knows at least a subset of the edited facts, and these known edited facts are used to construct the training dataset. However, in real-world settings we want to identify whether facts were edited and it is unrealistic to assume a known set of of edited facts. Therefore, we consider CD settings where the user does not know which facts have been edited in the model \({\cal M}={\cal M}^{\prime}_{e}\). Nevertheless, users can simulate the edit scenario by editing \({\cal M}_{e}\), an LLM of the same foundation model as \({\cal M}^{\prime}_{e}\), where the edited facts in \({\cal M}_{e}\) are known. The Cross-domain box in Figure 1 illustrates this setting.

### DEED Constituents

The DEED task is defined by three constituents: 1) The edited LLM; 2) The KE used for editing; 3) The dataset used for editing facts. Here, we describe each constituent and the construction of the datasets for DEED in detail.

Edited LLMs.Following previous work [56, 7], we incorporate GPT-2 XL (1.5B) [8] and GPT-J (6B) [9] as our base language models. Both are decoder-only autoregressive LLMs.

KEs.We include four parameter-modifying KEs of two sub-categories, MENDMitchell et al., 2022a) and MALMNTan et al., 2024) for meta-learning KEs, and RMEMeng et al., 2022) and MEMITMeng et al., 2023) for locate-and-edit KEs. We focus on parameter-modifying KEs, since for parameter-preserving KEs, it is possible to undo the edits by simply removing the external modules from the model.

Datasets.We edit models with two popular datasets, zsRE and CounterFact. zsRELevy et al., 2017; Mitchell et al., 2022a) is a Question Answering dataset, where each instance contains a factual statement and a paraphrased prompt. zsRE originally contains 244,173 training and 27,644 validation instances. CounterFactMeng et al., 2022) includes counterfactual statements, i.e., false facts, that are used for editing. CounterFact contains 21,919 records for 20,391 subjects and 749 objects.

DEED task construction.We edited two LLMs (GPT2-XL, GPT-J), with four KEs (ROME, MEMIT, MEND, MALMANEN) using facts from zsRE and CounterFact. This yields a total of 16 edited models (4 KEs, 2 LLMs, 2 datasets). Since our objective is to detect edited facts, we filter out unsuccessful edits. More specifically, for each edited model we collect successfully edited facts, i.e., facts for which the new object \(o^{\prime}\) can be correctly retrieved with a paraphrased prompt \(p^{\prime}(s,r)\). These are facts, for which \(o^{\prime}\) has the highest probability given the prompt \(p^{\prime}(s,r)\). More formally, \(o^{\prime}=argmax_{o}~{}\mathbb{P}_{{\cal M},p^{\prime}(s,r)}[o]\) . We report the number of successful edits that we consider in our experiments in Table 1. Note that, in addition to the dataset comprising edited and unedited facts, we assume access to the edited LLM. Access to the edited LLM enables users to explore various detection methods, such as employing HS and PD as features for the classification task 

Implementation details.We edit 1,000 facts in each setting (LLM, KE, dataset).3 We consider 1,000 edits to be a realistic number for malicious editing attacks, and defer the investigation of this choice to future work. For ROME, we perform one edit at a time on the base language model as ROME is not designed for mass editing or sequential editing Huang et al. (2022); Yao et al. (2023). For MEMIT, we edit each model with 1,000 facts at once. We use the hypernetworks for GPT-2 XL and GPT-J published by Meng et al. (2022) to circumvent the substantial compute required by MEND. For MEND, we also conduct one edit at a time to achieve a high success rate Meng et al. (2023). For MALMEN, we train the hyper-networks before editing and then perform 1,000 edits to the model simultaneously. We adhere to the default settings employed in the original papers. We include the unedited base model, NONE, as a yardstick. We follow related work Meng et al. (2022, 2023); Li et al. (2024a) in evaluating editing performance. See Appendix A for further details on editing metrics and performance.

Footnote 3: Increasing the number of edits did not lead to more successful edits with _all_ KEs.

### Practical Applications

Detecting knowledge edits can have significant practical applications:

Transparency.Detecting edits provides users with insights into whether model outputs originate from pre-training or post-deployment edits. This enhances the transparency of AI systems.

Model debugging.By distinguishing between pre-training and edited knowledge, we can identify whether an issue stems from unclean pre-training data or a maliciously inserted edit. For example, if a factual error is flagged as an edit, developers can focus on investigating unauthorized modifications rather than pre-training data.

Mitigating malicious use cases.Recent works highlight how knowledge editing can be used to bias LLMs Chen et al. (2024), introduce backdoors Li et al. (2024b); Qiu et al. (2024), or propagate misinformation in multi-agent settings Ju et al. (2024). Detecting knowledge edits helps identify and mitigate these threats by pinpointing unauthorized changes, ensuring the model's reliability in adversarial environments.

## 5 DEED Baseline Models

To train a detection classifier, we consider different options of available features. Inspired by the probing tasks introduced in Conneau et al. (2018) and the representations inspection by Hernandez et al. (2023), we investigate the use of hidden state representations as features. Additionally, in Kuhn et al. (2023) and Gu et al. (2023), the output probability distributions are used to provide signals for latent semantics. Therefore, we also explore using these distributions as features.

### Preliminary Analysis

HS visualization.To see how editing affects the hidden state representations, we use Linear Discriminant Analysis (LDA) Duda et al. (2000) to visualize the hidden state representations. Specifically, for each data point (fact) from the test set, we project the hidden state representation of the last token of \(p^{\prime}(s,r)\) into a one-dimensional space. We use the representations from the last layer of the respective LLM. We assign the one-dimensional representations random values on the y-axis to reduce point overlap. As shown in Figure 2, the edited and unedited representations are not distinguishable in the unedited model. Editing with ROME and to less extent with MEMIT makes the representations more separable. We attribute this separability to the high confidence of LLMs in the edited facts, as shown later in Section 5.3. Editing with MEND and MALMEN keeps the representations linearly inseparable.4 This suggests that meta-learning KEs cause subtle changes on representations when editing facts. Similar patterns are observed on all datasets and methods (see Figure 6 in the Appendix).

[FOOTNOTE:4][E

\begin{table}
\begin{tabular}{l l c c} \hline \hline Base LLM & KE & zsRE & CounterFact \\ \hline  & ROME & 736 & 722 \\  & MEMIT & 324 & 417 \\ GPT-XL & MEND & 765 & 756 \\  & MALMEN & 832 & 127 \\ \hline  & ROME & 876 & 837 \\  & MEMIT & 800 & 622 \\ GPT-J & MEND & 822 & 758 \\  & MALMEN & 878 & 297 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The number of successful edits, which we include in the detection datasets for each setting.

 

PD visualization.We investigate the output probability distributions for the next token from LLMs when prompting for edited and unedited facts with prompts that contain subjects and relations: \(p^{\prime}_{1}(s_{1},r_{1})...p^{\prime}_{n}(s_{n},r_{n})\). We consider a balanced set consisting of the same number of edited and unedited facts. We analyze the probability distribution of each set by calculating the mean of the top 10 probabilities and plotting the kernel density estimate (KDE). We show the KDEs for GPT-J and CounterFact in Figure 3. The figure shows almost identical distributions of low probabilities for both edited and unedited facts in the unedited model NONE. We notice two peaks for both distributions in NONE: the left one is close to zero, while the right one is around 0.3. As we edit with different methods, we notice that the right peak for edited facts is shifted to the right, while the first one flattens, as the probabilities there become less frequent. The increase in probabilities varies depending on the method; ROME and MEND cause the highest increase, followed by MEMIT and MALMEN, which are used for mass-editing. We also notice that MEND increases the probabilities of uncedited facts as well. Editing with MALMEN is the smoothest, since its probability distribution is the closest to that of the unedited facts. This potentially makes facts edited with MALMEN more difficult to detect.

### DEED Baselines

Given a fact \((s,r,o,p^{\prime})\), we prompt the given LLM \(\mathcal{M}\), which consists of \(L\) transformer blocks, with the prompt \(p^{\prime}(s,r)\) that consists of \(N\) tokens \(\{u_{1},\ldots,u_{N}\}\). We use the hidden state representations HS from the last transformer block and the last input token \(h_{N,L}=\mathcal{M}[p^{\prime}(s,r)]\) as a first feature for detecting edited facts. We use the information from the last token, since this is the position where the information about the subject and the relation is condensed [7]. As a second feature, we use PD, the top-1000 probabilities of the next token \(q=(q_{1},\ldots,q_{1000})\) from \(\mathcal{M}\) on the same prompt \(p^{\prime}(s,r)\).

### Evaluation

We train an AdaBoost classifier [25] with a decision tree as a base classifier using both features, HS and PD, on the DEED datasets (Section 4.2). We show the classification performance of training on 126 instances in the first part of Table 2. The minimum number of edits under all settings was 127 (cf. Table 1). We use half of the edited facts (63) for training, and the rest for testing. Balancing the edited 63 facts with 63 undetted facts yields a training set of 126 examples. We keep the size of the training set constant across all settings for fair comparison, and experiment later with smaller and larger training sets (cf. Figure 4). As a sanity check, we train a classifier on the same sets of edited and unedited facts using features from the base _unedied_ model. We refer to this classifier as NONE. The classifier NONE shows a fair coin performance on the three metrics, ranging from 46.2 (GPT-J, CounterFact, F1) to 49.8 (GPT2-XL, zSRE, F1). This verifies the correctness of our setup. We report precision, recall, and F1.

**Locate-and-edit KEs vs. meta-learning KEs.** We find that the classifier is able to detect edits by locate-and-edit KEs (LE-KEs), i.e., ROME and MEMIT, relatively well, achieving a minimum F1 score of 74.2% (MEMIT, GPT2-XL, zSRE). The highest F1 score of 98% is achieved on (ROME, GPT-J, zSRE). On meta-learning KEs (ML-KEs), i.e., MEND and MALMEN, we observe lower performance in general. On MEND the performance varies between 53.6% (GPT2-XL, CounterFact) and 81.4% (GPT-J, zSRE), whereas on MALMEN the gap between the lowest and highest performance is smaller than the same gap on MEND (62% (GPT2-XL, CounterFact), and 76% (GPT-J, zSRE)). In general, this shows that _edits, from the more computationally efficient LE-KEs, are easier to detect_. One reason for this discrepancy in performance between these two classes of methods might be that ML-KEs induce changes that generally target the last few layers, while LCE-KEs change middle layers. Note for example that MEND increases the probabilities for both edited and unedited facts, while MALMEN slightly increases the probabilities for edited facts (cf. Figure 3). This keeps the distance between edited and unedited facts minimal under ML-KEs, while LE-KEs conduct surgical changes that increase the probability for edited facts only. This can also be confirmed by the fact that the effects of LE-KEs on unrelated facts are less pronounced compared to ML-KEs (cf. Table 4, Appendix).

**Ablation.** We conduct an ablation study, where we train only on HS, or PD (second and third part of Table 2 respectively). We notice that using HS leads to low performance on the ML-KEs meth 

ods, where the F1 score varies between 48.5% (MALMEN, GPT2-XL, zsRE) and 62.6% (MALMEN, GPT-2XL, CounterFact). On LE-KEs, HS still performs well with a minimum F1 score of 65.3% (MEMIT, GPT-J, CounterFact). In contrast, _detecting edits with PD is more robust and performs well on LE-KEs and ML-KEs_. The minimum F1 scores with PD is 65.2% (MALMEN, GPT-J, CounterFact). PD's robustness might be explained by the fact that it is derived from a late position in the LLM, namely the language modeling head, which is responsible for the final outputs. Another important observation is that using PD individually can lead to better performance than combining it with HS in some cases (e.g., the F1 score on (MEND, GPT2-XL, CounterFact) is 53.6% when using both features, but 73% when using only PD). This observation suggests that PD and HS could be combined in better ways. We leave exploring the interplay between HS and PD to future work. We further explore using HS from different layers in Appendix A.1, observing that the representations start becoming separable after the edited layers, and that the representations from some layers (e.g., the penultimate layer) perform better than the representations of the last layer.

Training set size.Figure 4 shows the performance of different training set sizes with PD. Exhaustive results are presented in Table 7 in the Appendix. In general, the classification performance starts reaching a plateau after 50 training instances. For example, on MEMIT-edited facts in GPT-J (green circles), F1 reaches 85.4% after 50 training instances, while the highest F1 of 87.2% is achieved after 300 training data points. Therefore, we emphasize that _this classifier remains effective even when the training data consists of less than one hundred examples. We also observe that facts that are edited with ML-KEs are not only harder to detect than facts edited with LE-KEs (as noted before), but also that scaling the size of the training set does not improve performance._

## 6 Cross-Domain DEED

Obtaining a training set for the detection classifier requires knowing instances of edited facts in the edited model. However, this assumption may not always hold. Consider a scenario where an open-source model is further fine-tuned for a specific task of interest to the user, yet no explicit information is provided regarding the edited facts. Many models are fine-tuned with instructions or reinforcement learning from human feedback (RLHF), and are made publicly available Lai et al. (2023). In these cases, we have access to the original model (from which we could obtain our training set), but do not know which facts have been edited into the fine-tuned model. To address such cases, we assess the performance of the classifier, which uses HS and PD, in a cross-domain setting, where the classifier is trained on representations from one edited model and then applied to detecting knowledge edits from a separate edited model (which is a fine-tuned version of the original model). The CD edited models

Figure 4: Detection performance with PD on ZsRE when varying the training set size.

Figure 3: Kernel density estimation for averaged top-10 probabilities for the next token (GPT-J, CounterFact).

Figure 2: Representation of edited and unedited facts (zsRE, GPT-J). X-axis: 1D LDA projection, Y-axis: random values to reduce point overlap.

 

have the same architecture, but different weights. To simulate this scenario, we use representations from edited GPT-J and GPT2-XL to train the classifier and test it in detecting knowledge edits from instruction-tuned versions of GPT-J5 and GPT2-XL.6 Note that the facts in the training and test sets are still disjoint. For this experiment, we consider ROME, MEMIT and MALMEN.

Footnote 4: [https://huggingface.co/togethercomputer/GPT-JT-GB-v1](https://huggingface.co/togethercomputer/GPT-JT-GB-v1)

Footnote 7: [https://huggingface.co/lgaalves/gpt2-xl_lima](https://huggingface.co/lgaalves/gpt2-xl_lima)

As shown in Figure 5, _the performance on cross-domain edits (\(\mathtt{CD}\)) is on par with performance on in-domain data (\(\mathtt{ID}\))_. For example, on zsRE, \(\mathtt{CD}\) F1 on ROME-edited facts on GPT2-XL lima is 95%, similar to the \(\mathtt{ID}\) F1 of 96.6%. Full results with HS/PD are shown in Table 6 in the Appendix. This makes our DEED baseline highly practical, as it can be leveraged to detect edits in fine-tuned models, while making use of training data that is derived from their counterpart original models.

Cross-dataset generalization.The results of our previous experiment indicate that detection classifiers trained on edits in one model generalize to edits in other models of the same family. To further explore out-of-distribution scenarios, we impose an additional constraint by examining whether detection classifiers trained on edits from one dataset (e.g., CounterFact) generalize to edits from another dataset (e.g., zsRE). Here, we maintain the assumption from our previous experiment that the base LLM is known, but introduce a new constraint: the dataset used to conduct edits on the test LLM is unknown, necessitating training on one dataset and testing on another. The primary objective of this experiment is to investigate whether edit detection classifiers generalize effectively across datasets.

As shown in Table 3 the detection classifiers generalize well to unseen datasets in most cases. One exception is generalization from zsRE to CounterFact with MALMEN, where we see a dramatic decrease in performance. One reason for this might be the smooth changes induced by MALMEN (cf. Section 5.1). These changes are datespecific, which makes generalization more difficult (cf. Figure 8(b) and 8(c) in the Appendix).

[TABLE:S0.T3][ENDTABL

Figure 5: Comparison between in-domain (\(\mathtt{ID}\)) and cross-domain (\(\mathtt{CD}\)) classification performance. Classification is done based on HS and PD.

 

Conclusion

In this paper, we introduced a novel task: detecting knowledge edits in language models. We see promising applications in uncovering potential malicious model modification and enhancing the transparency of LLMs' generations. We instantiated the task with several baselines, which use hidden state representations and probability distributions as input features. These baselines demonstrate a straightforward yet effective approach to the task. Our evaluation across two LLMs, four editing techniques (\(2\) LE-KEs and \(2\) ML-KEs), and two datasets, shows that the classifiers can reliably detect edits, even in scenarios characterized by limited training data. Moreover, we showed that these baselines generalize to cross-domain settings, where training and test data come from different models and datasets, highlighting their practical utility. Our work gives rise to several future work directions: 1) detecting if an LLM has undergone any edits at all; 2) studying detection generalization across KEs; 3) investigating more constrained settings, where one has only access to the LLM's outputs; 4) combating malicious edits.

## 8 Limitations

Access to HS and PD.The baseline detection classifier we proposed requires access to the hidden states or probability distribution. When using closed source LLMs, these features might not be available. Inspired by Rosati et al. (2024), we plan to use the model behaviors shift on prompts with similar semantics but in different lengths to separate edited and unedited knowledge.

Unknown editing methods.In our current setting, we assume we know the editing method. However, the editing method might not be known and an edited model can be modified by multiple editing methods. In future work, we will investigate the generalization of our baseline classifiers between different KEs, and detecting edits in models that have edited with several KEs.

Number of edits.Previous work shows that a larger number of edits often leads to a worse editing performance Meng et al. (2023). Although we only keep successful edits for the detection task, it is unknown whether and how the number of edits will impact the detection performance. We plan to study this aspect in future work.

## Acknowledgment

This research was partially funded by the German Federal Ministry of Education and Research (BMBF) as German Academic Exchange Service (DAAD) project DEED under Grant No. 30001797. This work was conducted while Zhixue Zhao was an AInet Fellow, funded under the DAAD Fellowship Program (No. 57733361).

## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: SS1.
* C. Chen, B. Huang, Z. Li, T. Chen, S. Lai, X. Xu, J. Gu, J. Gu, H. Yao, C. Xiao, X. Yan, W. Yang Wang, P. Torr, D. Song, and K. Shu (2024)Can editing LLMs inject harm? In Trustworthy Multi-modal Foundation Models and AI Agents (TiFA), Cited by: SS1.
* A. Conneau, G. Kruszewski, G. Lample, L. Barrault, and M. Baroni (2018)What you can cram into a single S&!#* vector: probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, pp. 2126-2136. External Links: Link Cited by: SS1.
* N. De Cao, W. Aziz, and I. Titov (2021)Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 6491-6506. External Links: Link Cited by: SS1.
* R. Duda, P. Hart, D. Stork, and A. Ionescu (2000)Pattern classification, chapter nonparametric techniques. Wiley-Interscience Publication.. Cited by: SS1.
* P. V. Falade (2023)Decoding the threat landscape: chatgPT, fraudGPT, and wormgPT in social engineering attacks. arXiv preprint arXiv:2310.05595. Cited by: SS1.
* Y. Freund and R. E. Schapire (1997)A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences55 (1), pp. 119-139.

 

* [Geva et al.2023] Mor Geva, Jasmin Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 12216-12235, Singapore. Association for Computational Linguistics.
* Gu et al. (2023) Yuxuan Gu, Xiaocheng Feng, Sicheng Ma, Lingyuan Zhang, Heng Gong, Weihong Zhong, and Bing Qin. 2023. Controllable text generation via probability density estimation in the latent space. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 12590-12616, Toronto, Canada. Association for Computational Linguistics.
* Gurnee et al. (2024) Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertstimas. 2024. Universal neurons in GPT2 language models. _arXiv preprint arXiv:2401.12181_.
* Gurnee and Tegmark (2023) Wes Gurnee and Max Tegmark. 2023. Language models represent space and time. _arXiv preprint arXiv:2310.02207_.
* Halevy et al. (2024) Karina Halevy, Anna Sotnikova, Badr Alkhamissi, Syrielle Montarol, and Antoine Bosselt. 2024. " flex tape can't fix that": Bias and misinformation in edited language models. _arXiv preprint arXiv:2403.00180_.
* Hartvigsen et al. (2023) Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2023. Aging with grace: Lifelong model editing with discrete key-value adaptors. In _Advances in Neural Information Processing Systems_.
* Hernandez et al. (2023) Evan Hernandez, Belinda Z Li, and Jacob Andreas. 2023. Inspecting and editing knowledge representations in language models. _arXiv preprint arXiv:2304.00740_.
* Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. _arXiv preprint arXiv:2106.09685_.
* Huang et al. (2022) Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2022. Transformerpacther: One mistake worth one neuron. In _The Eleventh International Conference on Learning Representations_.
* Jang et al. (2023) Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2023. Knowledge unlearning for mitigating privacy risks in language models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 14389-14408, Toronto, Canada. Association for Computational Linguistics.
* Ju et al. (2024) Tianjie Ju, Yiting Wang, Xinbei Ma, Pengzhou Cheng, Haodong Zhao, Yulong Wang, Lifeng Liu, Jian Xie, Zhuosheng Zhang, and Gongheshen Liu. 2024. Flooding spread of manipulated knowledge in llm-based multi-agent communities.
* Kassem et al. (2023) Aly Kassem, Omar Mahmoud, and Sheriff Saad. 2023. Preserving privacy through dememorization: An unlearning technique for mitigating memorization risks in language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 4360-4379, Singapore. Association for Computational Linguistics.
* Kuhn et al. (2023) Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In _The Eleventh International Conference on Learning Representations_.
* Lai et al. (2023) Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dermoncourt, Ryan A Rossi, and Thieu Huu Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. _arXiv preprint arXiv:2307.16039_.
* Levy et al. (2017) Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_, pages 333-342, Vancouver, Canada. Association for Computational Linguistics.
* Li et al. (2021) Linyang Li, Demin Song, Xiaonan Li, Jiehang Zeng, Ruotian Ma, and Xipeng Qiu. 2021. Backdoor attacks on pre-trained models by layerwise weight poisoning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 3023-3032, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Li et al. (2024a) Xiaopeng Li, Shisha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. 2024a. PMET: Precise model editing in a transformer. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 38, pages 18564-18572.
* Li et al. (2024b) Yanzhou Li, Kangjie Chen, Tianlin Li, Jian Zhang, Shangqing Liu, Wenhan Wang, Tianwei Zhang, and Yang Liu. 2024b. Badedit: Backdooring large language models by model editing. In _The Twelfth International Conference on Learning Representations_.
* Liu et al. (2023a) Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2023a. Prompt injection attack against llm-integrated applications. _arXiv preprint arXiv:2306.05499_.
* Liu et al. (2023b) Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, et al. 2023b. LLM360: Towards fully transparent open-source LLMs. _arXiv preprint arXiv:2312.06550_ 

Vittorio Mazzia, Alessandro Pedrani, Andrea Caciolai, Kay Rottmann, and Davide Bernardi. 2023. A survey on knowledge editing of neural networks. arXiv preprint arXiv:2310.19704. Cited by: SS5.
* K. Meng, D. Bau, A. Andonian, and Y. Belinkov (2022)Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems36. Cited by: SS5.
* K. Meng, A. S. Sharma, A. Andonian, Y. Belinkov, and D. Bau (2023)Mass editing memory in a transformer. The Eleventh International Conference on Learning Representations (ICLR). Cited by: SS5.
* E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. Manning (2022a)Fast model editing at scale. In International Conference on Learning Representations, External Links: Link Cited by: SS5.
* E. Mitchell, C. Lin, A. Bosselut, C. D. Manning, and C. Finn (2022b)Memory-based model editing at scale. In Proceedings of the 39th International Conference on Machine Learning, pp. 15817-15831. Cited by: SS5.
* L. Mo, B. Wang, M. Chen, and H. Sun (2024)How trustworthy are open-source lms? an assessment under malicious demonstrations shows their vulnerabilities. Cited by: SS5.
* J. Niu, A. Liu, Z. Zhu, and G. Penn (2024)What does the knowledge neuron thesis have to do with knowledge?. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: SS5.
* Y. Pan, L. Pan, W. Chen, P. Nakov, M. Kan, and W. Wang (2023)On the risk of misinformation pollution with large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, pp. 1389-1403. Cited by: SS5.
* V. Patil, P. Hase, and M. Bansal (2024)Can sensitive information be deleted from LLMs? objectives for defending against extraction attacks. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: SS5.
* F. Petroni, T. Rocktaschel, S. Riedel, P. Lewis, A. Bachtin, Y. Wu, and A. Miller (2019)Language models as knowledge bases?. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 2463-2473. Cited by: SS5.
* J. Qiu, X. Ma, Z. Zhang, and H. Zhao (2024)Megen: generative backdoor in large language models via model editing. Cited by: SS5.
* A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. (2019)Language models are unsupervised multitask learners. OpenAI blog1 (8), pp. 9. Cited by: SS5.
* A. Roberts, C. Raffel, and N. Shazeer (2020)How much knowledge can you pack into the parameters of a language model?. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 5418-5426. External Links: Link Cited by: SS5.
* D. Rosati, R. Gonzales, J. Chen, X. Yu, Y. Kayani, F. Rudzicz, and H. Najjad (2024)Long-form evaluation of model editing. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Mexico, Mexico, pp. 3749-3780. Cited by: SS5.
* J. Shi, Y. Liu, P. Zhou, and L. Sun (2023)Badgpt: exploring security vulnerabilities of chatgpt via backdoor attacks to instructcgtprt. arXiv preprint arXiv:2304.12298. Cited by: SS5.
* S. Singh, F. Abri, and A. Siami Namin (2023)Exploiting large language models (llms) through deception techniques and persuasion principles. In 2023 IEEE International Conference on Big Data (BigData), pp. 2508-2517. Cited by: SS5.
* J. Sun, Y. Tian, W. Zhou, N. Xu, Q. Hu, R. Gupta, J. Wieting, N. Peng, and X. Ma (2023)Evaluating large language models on controlled generation tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 3155-3168. Cited by: SS5.
* C. Tan, G. Zhang, and J. Fu (2024)Massive editing for large language models via meta learning. In International Conference on Learning Representations, External Links: Link Cited by: SS5.
* B. Wang and A. Komatsuzaki (2021)GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. Note: [https://github.com/kingofloliz/mesh-transformer-jax](https://github.com/kingofloliz/mesh-transformer-jax) Cited by: SS5.
* S. Wang, Yaochen Zhu, Haochen Liu, Z. Zheng, C. Chen, et al. (2023)Knowledge editing for large language models: a survey. arXiv preprint arXiv:2310.16218. Cited by: SS5.
* A. Wei, N. Haghtolab, and J. Steinhardt (2024)Jailbroken: how does LLM safety training fail?. Advances in Neural Information Processing Systems36. Cited by: SS5.
* Y. Xu, M. Namazifar, D. Hazarika, A. Padmakumar, Y. Liu, and D. Hakkani-Tur (2023)KILM: Knowledge injection into encoder-decoder language models. In Proceedings of the 

61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada. External Links: Link Cited by: SS2.
* Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang (2024)A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, pp. 100211. External Links: Document Cited by: SS3.
* Y. Yao, P. Wang, B. Tian, S. Cheng, Z. Li, S. Deng, H. Chen, and N. Zhang (2023)Editing large language models: problems, methods, and opportunities. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 10222-10240. External Links: Document Cited by: SS2.
* Paul Youssef, O. Koras, M. Li, J. Schlotterer, and C. Seifert (2023)Give Me the facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, pp. 15588-15605. External Links: Document Cited by: SS2.
* P. Youssef, J. Schlotterer, and C. Seifert (2024)The Queen of England is not England's Queen: On the Lack of Factual Coherency in PLMs. In Findings of the Association for Computational Linguistics: EACL 2024, St. Julian's, Malta, pp. 2342-2354. External Links: Document Cited by: SS2.
* P. Youssef, Z. Zhao, D. Braun, J. Schlotterer, and C. Seifert (2025)Position: Editing Large Language Models Poses Serious Safety Risks. External Links: Link Cited by: SS2.
* C. Yu, S. Jeoung, A. Kasi, P. Yu, and H. Ji (2023)Unlearning bias in language models by partitioning gradients. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, pp. 6032-6048. External Links: Document Cited by: SS2.
* L. Yu, Q. Chen, J. Zhou, and L. He (2023)Melo: enhancing model editing with neuron-indexed dynamic lora. arXiv preprint arXiv:2312.11795. Cited by: SS2.
* Z. Zhao, Z. Zhang, and F. Hopfgartner (2021)A comparative study of using pre-trained language models for toxic comment classification. In Companion Proceedings of the Web Conference 2021, San Francisco, CA, USA, pp. 500-507. External Links: ISBN 978-1-4503-3213-1, Link Cited by: SS2.
* C. Zheng, L. Li, Q. Dong, Y. Fan, Z. Wu, J. Xu, and B. Chang (2023)Can we edit factual knowledge by in-context learning?. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 4862-4876. External Links: Document Cited by: SS2.

## Appendix A Complementary Results

In Table 5, we report the number of training and test instances used in our default setting. We keep the number of training instances constant across all settings for consistency. Results for varying sizes of the training set are presented in Table 7 and Figure 8 (CounterFact). The result for the CD evaluation is shown in Table 6. We show some prompt examples that are used for editing and evaluating edits in Table 10. We show the HS representations in a low-dimensional space in Figure 6. We show KDE visualizations of PD across all datasets, models, and KEs in Figure 9.

Edition performance scores are shown in Table 4. Editing methods are typically evaluated in terms of Efficacy, Generalizability and Locality Success. Efficacy Success (**ES**) is the primary evaluation measure and refers to the ability to change the LLM's predictions. For example, for the prompt \(p(s,r)\) "The Eiffel Tower is in the city of" originally retrieving object \(o\) "Paris", an edit is successful, if afterwards the desired object \(o^{\prime}\) (e.g. "Berlin") is retrieved. Ideally, such edits should also affect semantically similar prompts \(p^{\prime}(s,r)\), e.g., "The city where the Eiffel Tower is located is". The ability of edits to generalize to similar prompts is referred to as **GS** (Generalizability Success). Edits should not affect other facts. **LS** (Locality Success, also known as Specificity) Mitchell et al. (2022); Meng et al. (2022); Li et al. (2024) measures the accuracy of the model on a set of unrelated facts.

### HS from Different Layers

To investigate the effect of HS from different layers on the classification performance, we train a simple logistic regression classifier with L1 reg

\begin{table}
\begin{tabular}{l l c c c c c c} \hline \hline  & Generator & \multicolumn{3}{c}{zsRE} & \multicolumn{3}{c}{CounterFact} \\ \cline{2-7} Model & Editor & ES\(\uparrow\) & GS\(\uparrow\) & LS & ES\(\uparrow\) & GS\(\uparrow\) & LS \\ \hline \multirow{4}{*}{ GPT-J} & NONE & 28.1 & 27.4 & 26.1 & 13.3 & 18.6 & 81.0 \\ \cline{2-7}  & ROME & 99.5 & 94.4 & 25.5 & 100.0 & 99.8 & 79.4 \\  & MEMIT & 98.7 & 90.6 & 35.4 & 99.4 & 90.8 & 78.8 \\  & MEND & 92.6 & 89.0 & 20.3 & 91.4 & 79.5 & 70.4 \\  & MALMAN & 99.8 & 94.8 & 44.3 & 97.8 & 60.9 & 66.2 \\ \hline \multirow{4}{*}{ GPT2-XL} & NONE & 23.2 & 22.1 & 22.3 & 18.9 & 24.1 & 74.6 \\ \cline{2-7}  & ROME & 99.8 & 87.7 & 22.5 & 99.9 & 98.1 & 75.2 \\ \cline{1-1}  & MEMIT & 68.8 & 58.1 & 27.6 & 93.7 & 81.1 & 74.8 \\ \cline{1-1}  & MEND & 91.2 & 89.2 & 15.0 & 91.9 & 81.5 & 65.3 \\ \cline{1-1}  & MALMAN & 99.7 & 93.0 & 39.6 & 85.1 & 44.8 & 66.3 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Summary of editing performance over 1000 edits.

 

ularization on HS from different layers. We consider only locate-and-edit methods (ROME and MEMIT) for this experiment, since HS performs well on this category of methods. We show the F1 values based on hidden states representations from different layers in Figure 7. First, for both models, editing methods, and datasets, the representations start becoming separable, i.e., the classification performance is above random guess, after the edited layers or shortly thereafter. For example, the F1 for detecting ROME edits starts improving from the layer 7-8 in GPT-J and the layer 19 in GPT2-XL (ROME changes layer 7 in GPT-J, and layer 19 in GPT2-XL). We observe the same effect with MEMIT (MEMIT changes the layers 5-10 in GPT-J and the layers 15-19 in GPT2-XL). Even though the representations from the last layer perform well in detecting edits, we notice that the F1 reaches its peak at preceding layers. For ROME, the highest performance can be seen at training on representations from the layer 13-27 (GPT-J) or from the layers 23-47 (GPT2-XL). For MEMIT, the highest F1 is reached with representations from layer 17-27 (GPT-J), and layer 28-47 (GPT2-XL). This suggests that utilizing outputs from the last layer for detecting knowledge edits might not always be the optimal choice. Further, we observe that using the penultimate layer's representations is more effective than using the last layer's to detect knowledge edits. For example, for edited GPT-J on

\begin{table}
\begin{tabular}{l l c r r} \hline model & ds & method & \#training instances & \#test instances \\ \hline gpr+6h & CounterFact & MALMEN & 126 & 468 \\ gpr+6h & CounterFact & MEMT & 126 & 1118 \\ gpr+6h & CounterFact & MSD & 126 & 1390 \\ gpr+6h & CounterFact & NONE & 126 & 1874 \\ gpr+6h & ConstrFact & Rome & 126 & 1548 \\ gpr+6h & zzRef & MALMEN & 126 & 1630 \\ gpr+6h & zzRef & MEMT & 126 & 1474 \\ gpr+6h & zzRef & MSD & 126 & 1518 \\ gpr+6h & zzRef & NONE & 126 & 1874 \\ gpr+6h & zzRefn & BOME & 126 & 1626 \\ gpr+2i & ComputeFact & MALMEN & 126 & 128 \\ gpr2+2i & CounterFact & MEMIT & 126 & 708 \\ gpr2+2i & MultiFact & MSD & 126 & 1386 \\ gpr2+2i & CounterFact & MONE & 126 & 1874 \\ gpr2+2i & CounterFact & ROME & 126 & 1318 \\ gpr2+2i & ZRBE & MALMEN & 126 & 1538 \\ gpr2+2i & zzRef & MEMIT & 126 & 522 \\ gpr2+2i & zzRef & MEND & 126 & 1404 \\ gpr2+2i & ZRBE & NONE & 126 & 1874 \\ gpr2+2i & zzRef & ROME & 126 & 1346 \\ \end{tabular}
\end{table}
Table 5: Number of training and test instances used in our default setting. Both the training and test sets are balanced (50% for each class).

Figure 6: Respresentations of edited and non-edited facts.

 

CounterFact (first subplot in Figure 7), representations from the penultimate layer achieve an F1 of 90% and 72% for detecting ROME-edited and MEMIT-edited knowledge, respectively, higher than 85% (ROME) and 67% (MEMIT) when using the representation from the last layer. _LE-KEs might be causing the attribute (object) extraction phase_[4] that is responsible for extracting the object to happen early in the model. This early extraction is often associated with high probabilities (cf. Section 5.1), causing edits from LE-KEs to be easier to detect. In general, this experiment shows that the effect of edits from LE-KEs starts becoming visible from the changed layers, increases strongly thereafter, and decreases slightly at the last layer. This demonstrates that using representations from the last layer is not the optimal choice; utilizing representations from other layers can further improve performance.

### Robustness Evaluation

To investigate the robustness of the classifier, which uses HS and PD, we further evaluate it in a challenging setting, where both edited and unedited facts end with the same object, i.e., the classifier needs to distinguish between edited = \((s_{1},r_{1},o^{\prime},p_{1}^{\prime})\) and unedited = \((s_{2},r_{2},o^{\prime},p_{2}^{\prime})\) facts that share the same object (e.g., "The Eiffel Tower is in Berlin" vs. "Marlene Dietrich was born in Berlin"). Having the same object (as a prediction) in both edited and unedited facts might lead to more similar representations and probability distributions, and hence make the task more challenging. For example, the classifier might erroneously predict a fact as edited due to the presence of an edited fact with the same object. We refer to this setting as same object. Concretely, the classification model and the training are the same as described in Section 4.2. The test set is different. For each edited fact in the test set, there is a corresponding unedited fact with the same object. For this experiment, we consider only GPT-J with MEMIT and MALMEN. The results for this experiment are shown in Table 8 alongside the corresponding results from Table 2 (referred to as default) for comparison. We notice that the classification performance drops. The decrease in performance is especially visible on zsRE. For example, the F1 score with MEMIT and GPT-J drops from 88.7% to 71.3%, whereas on CounterFact the F1 score drops from 77.9% to 72.8%. This experiment shows that _it is more challenging to separate edited facts from unedited but related facts_. We believe this is due to the representations of the prompts becoming more similar under this setting.

### Detection by Top 10 Output Probabilities

We experiment with using only the top 10 output probabilities instead of the top 1,000 output probabilities. The differences in performance (top 10 performance - top 100 performance)) in Table 9 show that the top-10 output probabilities works well with minor degradation in performance in most cases. This means that our baselines can also be used on proprietary LLMs that output the top 10 output probabilities.

## Appendix B Computational Resources

We ran our experiments on an NVIDIA A100 80GB GPU. Our experiments took roughly 15 GPU days.

\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c c c c c} \hline \multicolumn{13}{c}{**Hidden States + Proh. Dist. (HS + PD)**} & \multicolumn{8}{c}{**Hidden States (HS)**} & \multicolumn{8}{c}{**Probability Distribution (PD)**} \\  & & \multicolumn{4}{c}{zsRE} & \multicolumn{4}{c}{CounetteFact} & \multicolumn{4}{c}{zzsRE} & \multicolumn{4}{c}{CounterFact} & \multicolumn{4}{c}{zzsRE} & \multicolumn{4}{c}{CounterFact} \\ \cline{2-13} \multirow{-13}{*}{} & KE & Pr. & Rec. & F1 & Pr. & Rec. & F1 & Pr. & Rec. & F1 & Pr. & Rec. & F1 & Pr. & Rec. & F1 & Pr. & Rec. & F1 \\ \hline \multirow{6}{*}{Lift} & NONE & 49.2 & 2.78 & 35.6 & 54.0 & 44.5 & 50.0 & 48.4 & 32.8 & 39.1 & 49.6 & 43.5 & 46.4 & 0.0 & 0.0 & 0.0 & 50.2 & 39.7 & 44.3 \\  & ROME & 87.1 & 95.5 & 91.1 & 50.9 & 94.2 & 92.5 & 99.8 & 74.2 & 85.1 & 91.6 & 87.8 & 89.6 & 84.8 & 84.5 & 84.7 & 89.3 & 94.0 & 91.6 \\  & MEMIT & 84.3 & 75.0 & 79.4 & 76.6 & 81.8 & 79.1 & 80.5 & 56.2 & 66.2 & 69.9 & 76.5 & 73.0 & 91.6 & 68.7 & 78.5 & 79.3 & 84.7 & 81.9 \\  & MALMEN & 80.7 & 71.6 & 75.9 & 65.6 & 74.1 & 69.6 & 50.4 & 34.6 & 41.1 & 57.8 & 69.1 & 63.0 & 72.8 & 88.4 & 79.9 & 82.4 & 54.0 & 65.2 \\ \hline \multirow{6}{*}{Lift} & NONE & 49.0 & 56.1 & 52.3 & 51.5 & 45.9 & 48.5 & 50.1 & 41.8 & 45.6 & 49.4 & 42.0 & 45.4 & 50.6 & 93.0 & 65.6 & 51.2 & 39.2 & 44.4 \\  & ROME & 91.4 & 99.0 & 59.0 & 89.6 & 93.8 & 91.6 & 97.8 & 93.2 & 95.4 & 94.5 & 83.0 & 88.4 & 63.6 & 86.9 & 73.4 & 88.7 & 91.3 & 90.0 \\ \cline{1-1}  & MEMIT & 79.2 & 65.0 & 71.4 & 77.3 & 77.1 & 77.2 & 76.9 & 61.9 & 68.6 & 73.7 & 60.9 & 66.7 & 74.0 & 73.2 & 73.6 & 76.9 & 76.4 & 76.7 \\ \cline{1-1}  & MALMEN & 73.4 & 77.7 & 75.5 & 68.5 & 75.5 & 71.8 & 48.3 & 53.9 & 51.0 & 65.0 & 53.1 & 58.4 & 73.6 & 92.0 & 81.8 & 68.1 & 65.3 & 66.7 \\ \hline \end{tabular}
\end{table}
Table 6: CD (Cross-Domain) classification performance. Training data is from GPT2-XL and GPT-J, whereas test data is from GPT2-XL Lima (GPT2 lima) and GPT-JT respectively.

 







\begin{table}
\begin{tabular}{l l c r r r r r r} \hline \multicolumn{2}{c}{Generator} & \multicolumn{3}{c}{zsRE} & \multicolumn{4}{c}{CounterFact} \\ \hline Model & Editor & Pr. & Rec. & F1 & Pr. & Rec. & F1 \\ \hline \multirow{3}{*}{default} & MEMIT & 87.7 & 89.8 & 88.7 & 79.0 & 76.7 & 77.9 \\  & MALMEN & 76.7 & 75.3 & 76.0 & 62.0 & 62.0 & 62.0 \\ \hline \multirow{2}{*}{same object} & MEMIT & 58.5 & 91.1 & 71.3 & 72.7 & 72.9 & 72.8 \\  & MALMEN & 57.5 & 57.9 & 57.7 & 56.4 & 63.6 & 59.8 \\ \hline \end{tabular}
\end{table}
Table 8: Classification performance on detecting knowledge edits for GPT-J. Facts in the training and test data end with the same object. Classification is done based on HS and PD.

Figure 8: Detection performance with PD on CounterFact when varying training set size.

\begin{table}
\begin{tabular}{l l c



\begin{table}
\begin{tabular}{l l p{142.3pt}} \hline Prompe example & z8RE & CountrateFact \\ \hline Edit prompt & In what category did Andrea Gattelli play football’s gabeper & Ramon Magwaysy holds a citizenship from Sweden \\ Evaluate ES prompt & In what category did Andreas Gattelli play football’gabeper & Ramon Magwaysy holds a citizenship from Sweden \\ Evaluate GS prompt & In what category has Andrew Gattelli played soccer’s guallerger & Margaret Rubin, Juan Carlos (2002), Ramon Magwayy is a citizen of Swedish \\ Evaluate LS prompt & What is the university Leslie Lararus were to? University of Sydney & Some faith, like-harks and lampsys, possess multiple gall openings: Abune Paulos, who is a citizen of Ethiopia \\ \hline \end{tabular}
\end{table}
Table 10: Example prompt used to edit knowledge, and to evaluate editing performance. Objects underlined are the original knowledge.

Figure 9: KDE of edited and non-edited facts.