Tracing and Reversing Edits in LLMs
Authors: Paul Youssef
Zhixue Zhao
Christin Seifert
Jörg Schlötterer
Venue: ICLR 2026 Poster (2026)
Keywords: Model Editing
Knowledge Editing
Countermeasures to Malicious Knowledge Editing
Primary area: interpretability and explainable AI
TL;DR: We propose two novel methods to trace and reverse model edits based solely on the edited weights

Abstract: Knowledge editing methods (KEs) are a cost-effective way to update the factual content of large language models (LLMs), but they pose a dual-use risk. While KEs are beneficial for updating outdated or incorrect information, they can be exploited maliciously to implant misinformation or bias. In order to defend against these types of malicious manipulation, we need robust techniques that can reliably detect, interpret, and mitigate malicious edits. To that end, we introduce the tasks of tracing and reversing edits. We propose a novel method to infer the edited object entity, solely based on the modified weights, without access to the editing prompt or any other semantically similar prompts, with up to 99\% accuracy. Further, we propose an effective and training-free method for reversing edits. Our method reverses up to 94\% of the edits, and helps regain the original model's output distribution without access to any information about the edit. This method can further be repurposed to distinguish between edited and unedited weights. Our findings highlight the feasibility of tracing and reversing edits based on the edited weights, opening a new research direction for safeguarding LLMs against adversarial manipulations.

Published as a conference paper at ICLR 2026
TRACING AND REVERSING EDITS IN LLMS
Paul Yousseff Zhixue Zhao
Christin Seifert
Jörg Schlötterer
*Marburg University University of Sheffield [paul. youssef, joerg. bperg.schloge schletttere, christin. sei seifert]@unin -marrtuuuuuummuuurggu zhixue. zww.@seefoeeeeeeeeoo ac
ABSTRACT
Knowledge editing methods (KEs) are a cost-effective way to update the factual content of large language models (LLMs), but they pose dual-use risk. While KEs are beneficial for updating outdated or incorrect information, they can be exploited maliciously implant misinformation or bias. In order to defend against these types of malicious manipulation, we need robust techniques that can reliably detect, interpret, and mitigate malicious edits. To that end, we introduce the tasks of tracing and reversing edits. We propose a novel method to infer the edited object entity, solely based on the modified weights, without access to the editing prompt or any other semantically similar prompts, with up to 99% accuracy. Further, we propose an effective and training-fie faree method for reversing edits. Our method reverses up to 94% of the edits, and helps regain the original model's output distribution without access to any information about the edit. This method can further be repurposed distinguish between edited and unedited weights. Our findings highlight the feasibility of tracing and reversing edits based on the edited weights, opening a new research direction for safeguarding LLMs against adversarial manipulations
INTRODUCTION
Large language models (LLMs) encode huge amounts of facts about the world in their parame- ters (Petroni et al., 2019; Youssef et al. 2023). However, such knowledge can be inaccurate or become outdated with time (Mitchell al. 2202a Hu al.) 2024). As remedy, knowledge editing methods (KEs) (Wang et al. (024c) have been proposed. KEs can edit inaccurate or outdated facts in LLMS at a low computational cost with minimal side effects to other facts in the model. Most KEs focus on atomic facts of the form (subject, relation, object) or (s, o) for short. Given a natural language representation of subject and relation, like "The chancellor of Germany is" (editing prompt), KEs are able to change the LLM outputs from outdated and incorrect object, "Olaf Scholz", to more recent and correct one, "Friedrich Merz". We denote this editing operation by (s, d').
Hu
While KEs offer a practical solution for updating knowledge, KEs be used maliciously inject backdoors, misinformation, or bias in LLMs (Youssef et al. 2025a). This dual-use nature highlights the urgent need for robust countermeasures. Prior work has primarily focused analyzing hidden states or output probabilities to determine whether specific facts have been altered (Youssef et al.) 2025c), or determine the specific type of the edit (e.g.. misinformation, bias, etc.) (Li et al.) (025). However, these works assume the availability of set of potentially edited facts that are examined to identify edited ones, which is highly impractical.
To address this limitation, we develop countermeasures from more generic angle to target malicious model edits (cf. Fig. [oor an overview). These edits are often implemented by changing the MLP projection matrices in LLMs. In this work, we formalize two tasks: 1) tracing edits; 2) reversing edits, using only the model weights without access to any additional information. introduce Edi tScope, a novel method for deriving the edited object from the edited weights, reaching more than 88% accuracy across multiple models. Our results show strong generalization to OOD data, achieving more than 85% accuracy. Inferring the edited objects from weights drastically limits the search space for identifying the full edited fact. Furthermore, we propose a method for reversing
trace edits, we
Equua contribution.

---

Published as a conference paper at ICLR 2026
Uninown Model Edit (s, r. d') on M Meorination lends ciabbeies
Norm
Deporder Block
M WWG
Decoder Block
Edited Object: dabetess
Allantinn
Original Object: immunity
ayee Norm
Decoder Block
Input Toxens
Figure 1: We investigate several countermeasures to malicious knowledge editing. These counter-
measures include retrieving the edited object (Sec. 4 and retrieving the original object (Sec. 5
Additionally, we look into identifying edited layers (App. C and predicting edited relations (App. D
edits using bottom-rank approximations of the edited weights. This method does not assume access any information about the edit, and is training-free and therefore highly efficient. Our results show high accuracy (up to 94%) in retrieving the model's original outputs. We also show that bottom-rank approximations can be repurposed distinguish between edited and unedited weights. In summary, we make the following contributions:
We formalize two tasks for tracing and reversing edits solely based on model weights to counteract malicious editing with minimal assumptions (Sec.2) 2}.
We introduce Edi Ecite for generating the edited object based only on the edited weights Our method does not assume any knowledge about the editing prompts, and is highly performant (Sec.
We propose a method for reversing edits using bottom-rank approximations of the edited weights, Our method is highly efficient and does not require access to any information about the edit, and can further be used identify edited weights (Sec. S}.
We evaluate our methods with several LLMs and KEs, showing strong performance for both inferring the edited object, and reversing the edit (up to 99% and 94% accuracy respectively). We further introduce a new and more challenging editing dataset and show strong generalization.
2 PROBLEM STATEMENT
Let Mie. WWg be an LLM with parameters 0 and vocabulary V, where Wy - W. indicates the subset of weights before (Wv) and after (Wi+) an editing operation (s, r, + o'). W results from a perturbation Wy- = Wv + WN. such that the model generates the new target object instead of the original object o. Wx refers to the weight updates produced by the used KE. Given only access to the model's parameters after editing, i.e., the edited weights (W%) and the original weights that are affected by editing (0) \ W;),, but no access Wy, nor information about any part of the editing operation (s, S, + o'), we have two objectives:
Tracing edits, i.e., identifying the edited fact. More specifically, target identifying the edited object as it is the output that potential attacker would want to steer the model to. We also present results for identifying the relation in App. D
Reversing edits, i.e., neutralizing the edit by intervening on W/ such that the model generates the original object instead of the edited one o', when queried with a prompt that contains and
Generally, we focus on developing countermeasures with minimal assumptions, relying solely on the edited weights for analysis and having no access the editing prompt nor the original weights.
2

---

Published as a conference paper at ICLR 2026
3 MODELS, KES AND DATASETS
Models. We use 4 models in our experiments: GPT2-XL (Radford ct al. 2019). GPT-J (Wang &
Komatsuzaki 2021 1, LLAMA3 (Dbeey al. 2024) and QWEN2.5 (Team 2024). GPT2-XL and
GPT-J were used in the pioneering work by Meng et al. (2022). Following recent work on KEs Fang al. 2025), we use LLAMA3 and QWEN2.5 as representatives for recent LLMs.
KEs. We target rank-one model edits with methods such as ROME (Meng et al. 2022) and its improved variant r-ROME (Gupta et al. 2024) in the paper's main body, and show generalization to other methods such as MEND (Mitchell et al.) 2022a), MEMIT (Meng et al. 2022) and AlphaEdit (Fang
et al.
2025)> in App. We provide a brief background on ROME in App. B
Datasets. We use the standard dataset CounterFact (Meng al. 2022). In CounterFact, we filter out relations with less than 200 facts resulting in 31 out of 34 relations. We list the selected relations with some examples in App. Tab. I5 We edit using facts from all relations and use the resulting updated weights in experiments. Each edit updates only one fact. We retain 100 successful edits from each relation for our experiments. We consider single edits, since already a single malicious edit bias the model (Chen et al., 2024), and elicit unethical responses from the model (Hazra jet al. 2024). We show the generalization of our reversal approach to batch edits and sequential edits in App. Fand App. G respectively.
Yago dataset. To mitigate evaluation bias, we construct second dataset with more diverse rela- tionships than CounterFact. use the knowledge base YAGO 4.5 (Suchanek et al. 2024) to sample subject-object pairs from 15 manually selected relations, filtering out those with fewer than 1000 pairs. For each relation, we generate editing and paraphrased prompts using DeepSeek RI (DeepSeek-A) Al et al. 2025). We show the selected relations along with examples in App. Tab. 15
4 TRACING EDITS
In this section, investigate whether infer the edit based the edited weights only, i.e., without having the editing prompt. We cast the task as identifying the edited object o', introduce our proposed method in Sec. 4.1. and present the corresponding results in Sec. 4.2
4.1 EditScope
0j 0n
In order retrieve the edited object without knowing any part of (s, o). we tune the unedited weights of the model Maxwy to decode the edited matrix Wtr and generate the corresponding edited object o.. We use a fixed random input, consisting of m newly added tokens fiized (t.,..., fm). This input constant and does not change during training. The aim of using x fized is simulate having a real input
LM head
Decoder layer
Edited decoder sade
Wx o
Wy
W
that steers the model to generate the edited object.
Multi-Head Self-Attention
W
Given training set of n edits, we dynamically use an edited matrix Wi: i {I....,7y nn). from this set as a replacement for the original and absent matrix Wy, and denote the resulting model by Mip.w, Moo.wy-Wa -W,)
Training
Decoder layer
Embedding layer
That is, we use the edited matrices ...,..., Wi as inputs to the model and the corresponding edited objects of .i....,of of as outputs. In other words, x fired serves place holder for the conventional inputs (in the form of tokens), and the edited matrix-object pairs represent the input-output pairs.
Xiires
Approach for inferring the edited object from the edited model. Based on the edited weights Wi;- tune remaining unedited parameters so that the model gener- ates the edited object of despite the absence of the editing prompt.
Figure 2:
We illustrate our approach at a high level in Fig. 2 We input 3r fixee to the model and change the original
3

---

Published as a conference paper at ICLR 2026
matrix Wv to the edited matrix WV; in the model to get probability distribution the vocabulary Q = -w (( We train the model with cross-entropy loss to output the corresponding
edited object of C= En Li=s lOg();).
4.2 EXPERIMENTAL SETUP AND RESULTS
We experiment with training one layer of Mo..yy-%% at a time. When training the layer that contains the edited MLP matrix Wiz we update all weights except Wy, (i.e., attention-weit weights and weights of the other MLP sub-layer), as not to impair the edited weights. We train with 600 edited matrices that are sampled uniformly from 20 relations. We use 100 matrices from the same relations validation set. We test 300 samples from the same relations, and contains 330 samples from 11 unseen relations to evaluate the model's s ability to generalize to unseen relations. We train for maximum of 100 epochs, and use early stopping with a patience of 3 epochs on the validation loss. We use A for optimization with an initial learning rate of 10-5 with 31 0.9, 32 0.98 and weight decay of 0.01. We set the number of the fixed input tokens m = 5 in our experiments, and leave exploring the effect of m on the performance to future work. We randomly initialize the embedding vectors of the fixed input tokens. We evaluate based on the edited object accuracy (Meng et al., (2022) i.e., the accuracy of the model in generating the edited object of based on W/ find the optimal layer train, we consider only ROME with GPT2 XL., GPT-J and LLAMA3 (Fig. 3). Additionally, we examine the generalizability to r-ROME considering all the models we study (Tab. T..
OOD test set that
Results. The results in Fig. show that the edited object can be generated with high accuracy (99% for the GPT-models and > 97% for LLAMA3 on CounterFact), when training layer up to the layer containing the edited matrix. Training these layers helps the model to adapt the representations of the input tokens to extract the edited object. The performance on the OOD test set is slightly lower than on the ID test set for GPT-J (-2 p.p.) and LLAMA3 (-3 p.p.). The performance on Yago drops slightly, since Yago contains longer objects compared to CounterFact (ef. Tab. [5}. We attribute the high performance mainly to the model overfitting to the edited objects (Zhang et al. 2025), i.e., the edited object having overly high probability after editing. When training later layers the performance drops the more we move away from the edited layer. This suggests the edited object becomes more difficult generate as we move away from the edited layer.
Given the high performance when training the edited layer, we focus on this setting and and experiment with all models using ROME and ROMOME. We run each combination (editing method and model) with 5 random seeds. The results in Tab. show high and stable performance with both ROME and R-ROME and across all models. For example, the in-domain accuracy 88% and the OOD accuracy > 85%. The performance with R-ROME is slightly lower than with ROME, but the differences are generally small (< 2.7 p.p.).
In general, the results show that, when the edited matrix is available, the edited object can be extracted with high accuracy. Our method provides direct information about the edit (the edited object o') with strong generalization, and can be combined with information about the relation (cf. App. D to
reconstruct the edited fact. We show that Edi tScope can generalize to other KEs in App. E.
Method
Model
Acc.
Std
Acc. (OOD)
Std (OOD)
GPT2-XL
99.40
0.43
99.70
0.30
ROME
GPT-I-6B
97.60
1.86
94.42
1.51
META-LLAMA-338B
96.47
0.56
91.21
2.77
QWEN2.5-7B
91.20
2.06
87.45
2.73
GPT2-XL
99.73
0.28
99.70
0.52
F-ROME
GPT-1-6B
96.50
2.86
95.91
3.37
META-ILAMA-3-8B
94.87
1.07
88.18
3.04
QWEN2.5-7B
88.53
1.71
85.45
4.00
Table 1: Accuracy of generating the edited object based on the edited matrix when training only the edited layer. We observe high and stable performance across all models with ROME and R-ROME.
4

---

Published as a conference paper at ICLR 2026
Accuracy
Accuraac
Normalized Layer Index
Normaized Layer Index
Figure 3: Accuracy of generating the edited object based on the edited matrix when training different layers of RoME-edited models (Left: CounterFact, Right: Yago). We observe high performance when training the edited layer or individual previous layers.
REVERSING EDITS
To reverse edits, exploit the fact that, promote the edited object, it must be overly present in the edited matrix. We hypothesize that thereby particular rank-one approximations based on the highest singular values of Singular Value Decomposition (SVD) of the edited matrix similar to the rank-one update matrix of methods such as ROME or r-ROME. Conversely, we assume that the edited object not over-represented in rank-one approximations based lower singular values (bottom-rank). We introduce bottom-rank approximations derived from SVD in Sec. [5.] conduct an analysis of our hypothesis in Sec. 5.2 and present our approach for reversing edits Sec. 5..3
5.1 SINGULAR VALUE DECOMPOSITION
BOTTOM-RANT APPROXIMATIONS
Given a rank matrix M E Rmxx, its singular value decomposition into three matrices has the form UEV', where U Rmmxm E g=xn E _ The diagonal elements of Σ are the singular values of M, and are sorted in descending order, i.e., Σa Ejj where > i. This decomposition can also be written as a sum of rank-one matrices: M == =E= Enuuy which allows us to create rank-one approximations of M based on particular singular values:
(R) Σ EI 1=1
We can further construct rank r - A: approximations of M by excluding the top (i.e., highest) singular values and their corresponding vectors from U and V, and refer these as bottom-rank approximations:
5.2 ANALYSIS OF RANK-ONE APPROXIMATIONS
Given that the update matrix Wyy makes the edited object quite prominent the edited matrix, we hy- pothesize that some of the rank-one approximations of Wi are similar to the rank-one update matrix Wx- To verify this hypothesis, we analyze how similar different rank-one approximations are to the update matrix WN
Max. Cosane Smilliity
(PTT)
META-L ANA-3-4R
OWEN225-76
sample of 10 relations. The row vectors each rank-one matrix can have at most the maximum cosine similarity value among the rows of (k)
two directions. As proxy for similarity,
WN and (h, for different & values. High absolute values of cosine similarity suggest that the row tors of both matrices have similar directions, whereas smaller values indicate different directions.
Figure 4: The maximum cosine similarity val- ues between vectors of the update matrix W
and the rank-one approximation (R) (R)
5

---

Published as a conference paper at ICLR 2026
GPT-J-6B
META-LLAMA--3-
QWENZS5-B
Reversal
Editing Asc.
Revenal
Editing Acc.
Acc.
Editing
Aec.
Eddting Acc.
0.00
100.00
0.32
0.97
100.00
0.65
100.00
$7.10
7.42
32.26
60.65
5.88
95.48
31.91
62.90
2
$8.33
4.54
72,90
6.77
38.39
66.45
51.91
42.58
90.32
2,90
50.00
53.55
40.00
90.32
199
75.81
6.13
60.97
28.39
53.87
37.10
91.29
194
77.42
3.23
66.77
20.32
56.77
34,19
91.29
194
77.10
67.34
18.21
58.71
30.97
90:97
194
77.42
13.87
30.32
8 91.29
194
77.74
2.58
73:23
11.94
39.68
30.52
9
22.58
1.94
78.06
2,58
75.16
9.68
6065
29,03
78.06
9.03
62.90
11
94.52
1.29
78,06
2.58
76.77
8.71
62. 58
26.45
12
19
1.20
79.03
2.58
79.35
7.10
62.90
26.13
13 93.23
0.97
79.35
2.26
79.35
6.77
62.90
26.13
78.71
25.16
15
93.87
0.97
78.71
2.26
80.00
6.45
62.58
24.52
(r.k)
Table 2: Reversal and editing accuracy with bottom-rank approximations W for ROME. As k increases, the edits are removed (editing accuracy drops), and the model is able retrieve its original generations (reversal accuracy increases). Similar results for r-ROME and Yago are shown in App. Tab. and Tab. 19 respectiivel
Results. We show the results in Fig. (extended by standard deviations in App. Tab. 16 The results show very high similarity (0.98) between the update matrix and the k = approximation for GPT2-XL. For larger values the similarity drops significantly. For GPT-J, the similarity with k = 1 is lower (0.77), but we have moderate similarity (0.45) with k = 2. Here too, the similarity values drop when k > For LLAMA3, the values are much lower (0.20) with k = 1, increase when k E {2, 3, 4) and start dropping again for larger k values. For QWEN, we observe the same pattern as with GPT-J but with lower similarity values. This suggests that for GPT-models, the single rank-one approximation with the top singular value encodes the edit, whereas for LLAMA3, combination of rank-one approximations from top singular values encodes the edit. In general, the results show that the rank-one approximations with l- = 1 come close to the update matrix in case of the GPT-models, whereas on LLAMA3 and QWEN the approximations have lower similarities to the update matrix.
5.3 REVERSAL
The results from the previous section suggest that the editing information might be localized at the first few rank-one approximations of Wy, and that the original object before editing might still be encoded in bottom-rank approximations of W,- This observation encourages
investigate replacing the edited matrix Wi by its bottom-rand approximations ( The intent behind this intervention is to exclude the first & rank-one approximations and thus create an approximation without any editing information. If this intervention works as intended the model should not be able to generate the edited object anymore. We evaluate the removal of the edited object by editing accuracy (lower is better, as we want the model forget the edit) and recovering the original object by reversal accuracy (higher is better).
Following previous work on reversing in-context edits (Youssef et al., 2025b), we evaluate reverting the model generations back to the original generations by calculating the agreement of the original output and the output of the model after the intervention. Editing and reversal accuracy are calculated as Σ>= 1(j. y.), where y is the reverse-edie edited output and y is the original or edited output for edit i. For reversal accuracy, we approximate the model's 's outputs using the next token prediction, following Du et al. (2 Youssef et al. (2025b). As a baseline, we the rank approximation that does not exclude any singular values, i.c., we set k 0. Here, we use 310 instances, uniformly sampled from 31 relations.
The results in Tab. show that with & = 0, all models have near-zero reversal accuracy, and perfect editing accuracy. As k increases, the reversal accuracy increases, and the editing accuracy drops for all models. Nevertheless, the extent of the increase or decrease in relation to the value of I is model-dependent. For example, the reversal accuracy with & = 1 is 87%, 32%, 5% and 32%, whereas the highest attained reversal accuracy is 94% (k- = 11), 80% (k = 14), 80% (k = 15) and 62% (k- 13) for GPT2-XI., GPT-J, LLAMA3 and QWFN2.55 respectively. We also notice that the
Results.

---

Published as a conference paper at ICLR 2026
reversal and editing accuracy do not up 100%, and that the decrease in editing accuracy is higher than the increase editing accuracy, suggesting that the method is more effective in removing the edit than in recovering the original object. We show generalizability of reversal approach to other KEs in App. H.2 and to batch edits and sequential edits Sec. and Sec. respectivell Next, conduct a qualitative analysis to better understand how bottom-rank approximations affect the model's 's outputs.
Qualitative analysis. We show a random selection of examples with the best value for each model in Tab. | We generate 5 tokens given the input using greedy decoding. We notice that although outputs with approximations are not identical to the original outputs in some cases, they nonetheless semantically similar (e-g., soccer player/footballer, New York Mets/Jets, they/he). This suggests that when the edited output is changed after using the approximation the new output is semantically close the original output.
Mere edit removal general reversal. Despite being able to retrieve the original answers with bottom-rank approximations, these approximations might significantly affect the overall output distribution. Therefore, further examine how using bottom-rank approximations affects the overall probability distribution by calculating the KL divergence between the original model and the model with bottom-rank approximation: K (V ywv) gwv (Log(gwy)
(p where ywv represents the original model's 's output distribution and Vipr(ro0) the output distribution of the model with a bottom-rank approximation. We use the same set of facts we used for reversal, and report the mean. The results with ROME in Tab. show significant decrease in KL divergence across all models. The largest decrease KL divergence observed in GPT-J (11.567 + 0.218), whereas the smallest one is seen in QWEN2.5 (8.988 -> 1.534). The results with r-ROME in App. Tab. [4] show a similar pattern. Despite the differences across models, the results show that Nottom-ran. approximations help recover the model's original output distribution.
k
GPT2-XL
GPT-J-6B
META-LLAMA-3381
QWEN2.5-7B
0
6.038 2.525
11.567 3.790
10.068 3.703
8.988 3,371
0.187 0.80
4,658 = 4.886
9,698 4.204
4.844
2
0.159 + 0.806
0.438 = 1.019
6,171 + 5.216
3.408 4.434
3
0.083 +: 0.418
0.323 + 0,584
4.127 4.830
3.044 4.216
4
0.046 0,362
0.322 = 0.596
2.328 3.865
2.704 4.009
5
0.046
0.257 =
1,448
2.535
6
0.048 +: 0.442
0,240 = 0.387
1.372 + 2.742
2.309 3,811
7
0.025 1: 0.109
0,224 + 0.271
1.076 + 2.395
2.163 3.666
8
0,025 0.105
0.224 + 0.276
0.889 21164
2.131 3.604
9
0.021
+ 0.077
0.225 = 0.278
0.765 1.989
1.902 3.451
10
0.017
0.221 = 0.271
0.754 1.992
1.716
11
0,010 0.022
0.221 = 0,270
0.728 + 1.986
1.614 3.170
12
0,011 0.022
0,222 1 0.270
0.666 1.873
1.608 3.173
13
0.010
0.018
0.219 1 0.256
0.662 1.875
1.615 3.209
14
0.010
0.218 0.252
0.649
1.598
15
0.009 0.014
0.219 0.254
0.604 1.775
1.534 3.151
Table 3: KL divergence between the original model and edited models with ROME after using bottom-
rank approximations (rr,(r.k) to reverse the edits. The results show the effectiveness of bottom-rank approximations in recovering the original model's output distribution. Similar results for r-ROME are shown in App. Tab.
Model capabilities after reversal. To verify that models are not damaged reversal, we follow Fang al. 2025) and compare the performance of the edited models the performance of the edited and reversed models on the following tasks from the GLUE benchmark (Wang et al. 2018):
CoLA (Corpus of Linguistic Acceptability) (Warstadt et al. 2019) classifying English sentences either grammatically acceptable not.
7

---

Published as a conference paper at ICLR 2026
MMLU (Massive Multi-task Language Understanding) (Hendrycks et al. 2021 measur- ing an LLM's multitask accuracy in answering multiple-choice questions from a wide range of domains such mathematics, history and law.
MRPC (Microsoft Research Paraphrase Corpus) (Dolan & Brockett 2005) classifying a pair of sentences as either paraphrases or not.
NLI (Natural Language Inference) (Williams et al. between two sentences as either entailment or not.
2018 classifying the relationship
RTE (Recognizing Textual Entailment) (Bentivogli et al.
2009 classifying whether
premise sentence entails a hypothesis sentence.
SST (The Stanford Sentiment Treebank) (ochher et al. 2013 classifying the sentiment in movie reviews as either positive or negative.
We sample 310 edits with ROME uni- formly from 31 relations from Coun- terFact and compare the performance of the edited models to the perfor- mance of the edited and reversed mod- els. We reverse using bottom-rank ap- proximations with k = 15. We use LLAMA3 for this experiment. The results in Fig- 5 show that reversed models perform
F1.crr
COLA
MMLU
MRPC
NJ
RTE
SST
par with edited models, and that the performance of reversed models is more stable (lower standard deviation), indicating that re- versal does not have any negative ef- fect on the model's 's performance.
Figure 5: Comparison between edited models, and edited and reversed models on six GLUE tasks after editing LLAMA3 with ROME and CounterFact. We apply bottom-rant approxi- mations with & = 15 for reversal. Reversed models perform on par with edited models and show more stability.
Number of unique predictions. We investigate whether model editing can be detected by exam- ining how often the predictions change over a range of bottom-rank approximations. comparing between edited and unedited original weights. This analysis motivated by the assumption that bottom-rank approximations of edited matrices differ more strongly from approximations that include the highest singular values, even on completely unrelated text.
As inputs we use random selection of 100 ex- amples from wiki text-103 with at least char- acters and generate 5 tokens with greedy decod- ing. We vary & E {O..... 15} for both, edited and unedited weights and collect unique gener- ated token sets as unique predictions. For this experiment, only consider GPT2-XL, GPT-J
Unedited
Edited
#Unique Predictions
mm
3.23
and LLAMA3 with ROME. The results in Fig. 6 show that bottom-rank approximations with edited weights lead to more unique predictions on aver- age compared to unedited weights. For example, with GPT-J we have 1.37 unique predictions on av- erage with unedited weights, but 2.46 with edited weights. With LLAMA3 the gap is smaller (1.36 vs. 1.84). The results indicate that the edited weights affected more strongly by the approximations, likely because the edited weights are "artificially" modified, and the edited facts in them are more prominent than other facts (cf. Sec.(4) This finding can used to distinguish between edited and unedited weights it only requires approximating existing weights and random of inputs.
GPT2-XL
GPT-)-60
META-LLAMA-3-00
Figure 6: Number of unique predictions with standard deviation when using bottom-rank ap-
proximations (1.8) with k: E (O.... .00....15] 15) on a set of 100 examples from wikitext110 103. Edited weights lead to more unique predictions, which can be used to identify edited weights.
8

---

Published as a conference paper at ICLR 2026
Input
Edited Object
Original Output
After Reversal
GPT2-XI.
The headquarter of Hellenic Army is in National Highway 45 is located in the country of The Evaporators was created in the coun- try of
11
Glasgow
Athens. Greece. Georgia, in the state
Athens. Greece, Mexico, in the state
11
Venezuela
II
India
the same name, and
the same name, and
Coric Standing was released on David Beekham professional
MTV football
DVD on soccer player plays
DVD on
foothaller who plays the
GPT-J
Malha, in
14
Idaho
the state of São artist. He is the BBC, has been midfielder, but they
the north of the country artist. He is the BBC, has been a midfielder, but he
Jeff Bova's profession is Huw Edwards, who works for Which position does Graham Barrow play? They play as Boryspil International Airport, which was named for
14
actor
14
McLaren
14
linebacker
Aristotle
the city Bory
the city of Bory
META-LLAMA-3-8
Tim Tebow plays Core created by
the New York Mets the same team that brought
for New York Jets the brought Sheffield in 1990
15
Apple
that
Immaculate Machine, that was started in Doug Paisley, who holds citizenship from
15
Sheftield Belgium
2003 by the Canada, is a singer
15
the United States, is
Charles Montague Cooke, Jr. was origi- nally from
15
Jasper
Honolulu, Hawaii. He
Honolulu, Hawaii. He
QWEN2.5-7B
Armin Iofmannn, who holds citizen- ship from
Romania
Switzerland, was born in the age of the French National As- sembly. the heart of the city
Switzerland, a Swiss
Bruce Fairbairn passed away at Dominique Lapierre, speaker of
London English
the age 8 the French National As- sembly. the heart of the city
13
Where is Cleveland Classic? It located
13
Istanbul
in
BMW 5 Scries, created by
13
Nissan
the German car manu- facturer BMW
Nissan, was released in
Table Model outputs when using bottom-rank approximations (r.k) on random set of facts. We use the best & for each model with ROME. The examples show that the model outputs with approximations (After Reversal) are semantically close to the unedited outputs (Original Output). Similar examples for r-ROME and Yago shown in App. Tab. 13 and Tab respectively.
6 RELATED WORK
Knowledge Editing. KEs can be categorized as either parameter-modiffingg. i.e., changing model parameters [Mitchell et al.
2022a Meng et al., 2022), or parameter-preserving i.e., methods that rely on memory-module modules Mitchell al. [022bb] Wang et al.) 2024a) or the in-context abilities of LLMs (Zheng et al. 2023) to produce the desired changes. strameter-modifyine
KEs include two approaches: 1) Meta-learning KEs (Mitchell et al.. 2002a Tan et al. 2024) that train hypernetworks to predict the necessary shift in model parameters for editing knowledge; 2) Locate-and-edit KEs (Meng) ct al. 2022 2023) that first identify specific modules responsible for storing knowledge in the model, and then directly adapt these modules. Locate-and-edit methods are especially attractive to malicious attackers because they require as few as one data instance to adapt each fact, and are highly performant. Recent work (Youssef et al.) 2025a) shows that locate-and-e and-edit methods such as ROME (Meng et al. 2022) are widely used in malicious knowledge editing.
Malicious knowledge editing. KEs be used maliciously implant backdoors Li et al. 2024). spread misinformation (Ju al, 2024), bias (Chen et al., 2024), and jailbreak LLMs (Hazra al. 2024). Youssef et al. (2025a) argue that KÈs present significant safety risks due to their

---

Published as a conference paper at ICLR 2026
attractive properties, the vulnerable Al ecosystem, and general lack of awareness regarding their potential misuse. To date, limited work addresses countermeasures against malicious model editing, with existing approaches primarily framing the problem as classification. These efforts focus on distinguishing between edited and unedited facts (Youssef et al. 2025c) and identifying different types of edits (Li al. 2025). However, they assume the availability of set of potentially edited facts that are examined to identify edited ones. Reversing edits has been limited to in-contexi edits (Youssef) al., 2025b), where in-context edits are reversed by intervening on the input to the model. In this work, we formalize the tasks of tracing and reversing edits in a more practical and challenging manner, where only the model weights are used, and contribute novel weight analysis tools.
7 CONCLUSION
Our work introduced the tasks of tracing and reversing edits to counteract malicious editing. We proposed novel method for inferring the edited object based solely on the edited weights, and showed that method has high accuracy and generalizes strongly OOD data. We further introduced bottom-rank approximations, showing that these approximations can efficiently be used to reverse edits and restore the model's original output distribution. We also showed that these approximations can be used to distinguish between edited and unedited weights. Our work shows that even without access to the original, unedited weights or any part of the editing operation (s, r, o + d'), tracing edits and restoring the model's 's original outputs is feasible with high accuracy, encouraging future research in extended scenarios with realistic settings.
ACKNOWLEDGMENTS
We thank Veysel Artuc for his help in creating the Yago dataset. We also thank Ali Kholmovaia and Phuong Quynh Le for helpful discussions. We gratefully acknowledge support from the hessian Service Center (funded by the Federal Ministry of Research, Technology and Space, BMFTR, grant no. 16IS22091) and the hessian.Al Innovation Lab (funded by the Hessian Ministry for Digital Strategy and Innovation, grant no. S-DIW04/0013/003).
REFERENCES
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The Fifth PASCAL Recognizing Textual Entailment Challenge. TAC, 7(8):1, 2009.
Canyu Chen, Baixiang Huang, Zekun Li, Zhaorun Chen, Shiyang Lai, Xiongxiao Xu, Jia-Chen Gu, Jindong Gu. Huaxiu Yao. Chaowei Xiao, Xifeng Yan, William Yang Wang, Philip Torr, Dawn Song, and Kai Shu. Can Editing LLMs Inject Harm? In Trustworthy Multi-modal Foundation Models and Al Agents (TiPA), 2024. httphtttss/:/peneeweee wwt.://e///ooooooo www.:eee/oormmmn-PPEEEFEFF
DeepSeek-Al. AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, et al. DeepScek-R1: In- centivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025. URL https 7/arxiv. abs/ 2tt./
William B. Dolan and Chris Brockett. Automatically Constructing Corpus of Sentential Paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URL
10ap//055-55000 5002/
Kevin Du Vésteinn Snæbjarnaroon, Niklas Stochr, Jennifer White, Aaron Schein, and Ryan Cot- terell. Context versus Prior Knowledge in Language Models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 13211-13235, Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10. 12118653/112024.acllaa acl llog.. 714/
URL
Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, et al. The Llama 3 Herd of Models, 2024. URL http::tts:://art/ pecttps://arivv wtt.://rgg///////......
10

---

Published as a conference paper at ICLR 2026
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua. AlphaEdit: SulleSpace Constrained Model Editing for Language Models. In The Thirteenth International Conference on Learning Representations, 2025. URL https
Akshat Gupta, Sidharth Baskaran, and Gopala Anumanchipalli. Rebuilding ROME : Resolv- ing Model Collapse during Sequential Model Editing. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 21738-21744, Miami, Florida, USA, November 2024. As- sociation for Computational Linguistics. doi: https://aclantiooooyy..rr. 2rwp:///ohooooo..222222444 emnlp-main. 1210/
main. 1210. URL
Rima Hazra, Sayan Layek, Somnath Banerjee, and Soujanya Poria. Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Models. In Lun-Wei Ku, Andre Martins, and
Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACI. 2024, pp- 16227-16239, Bangkok, Thailand, August 2024. Association for Computational I Linguistics. doi: 10.18653/v1/2024 11 9607
URL https:://aaaaathol..oo/ooo///// orgp:/y..rg//242..// findings-acl
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Mcasuring Massive Multitask Language Understanding, Proceedings of the International Conference Learning Representations (ICLR), 2021.
Xuming Hu, Junzhe Chen, Xiaochuan Li, Yufei Guo, Lijie Wen, Philip S. Yu, and Zhijiang Guo. Towards Understanding Factual Knowledge of Large Language Models. In The Twelfth Interna- tional Conference on Learning Representations, 2024. URL https:://peenreview. http:/ppperreiiee.eeeffrrrrrr id=90evMUdods
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B, 2023. httphhttss//aaxxvv.cr
abs/2310.0683
Tianjie Ju, Yiting Wang, Xinbei Ma, Pengzhou Cheng, Haodong Zhao, Yulong Wang, Lifeng Liu, Jian Xie, Zhuosheng Zhang, and Gongshen Liu. Flooding Spread of Manipulated Knowledge in LLM-Basee Multi-Agent Communities. arXiv preprint arXiv:2407.0779) 2024.
Xiaopeng Li, Shasha Li, Shangwen Wang, Shezheng Song, Bin Ji, Huijun Liu, Jun Ma, and Jie Yu. Identifying Knowledge Editing Types in Large Language Models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD '25, pp. 1564,1564, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400714542. doi: 10.1 10.1314/371110 1877117796777770 URL htRLhhtts:::/// //doi. org/10. 1145/3711899 18861 3737081
Yanzhou Li, Kangjie Chen, Tianlin Li, Jian Zhang, Shangqing Liu, Wenhan Wang, Tianwei Zhang, and Yang Liu. BadEdit: Backdooring Large Language Models by Model Editing. In The Twelfth International Conference Learning Representations, 2024. https:ttts///ppeeree// net/ foorum2id=duZANm22BB
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Asso- ciations in GPT. Advances in Neural Information Processing Systems, 36, 2022. arXiv:2202.05263
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass editing memory in a transformer. The Eleventh International Conference on Learning Representations (ICLR), 2023.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast Model Editing at Scale. In International Conference on Learning Representations, 2022a. URL https: httpp:://openreviev net/Forumm/d-BDc>>xeweet
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory- based Model Editing at Scale. In International Conference on Machine Learning, pp. 15817-15831.
PMLR, 2022b.
11

---

Published as a conference paper at ICLR 2026
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language Models as Knowledge Bases? In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP-IJCNI.P), pp. 2463-2473, Hong Kong, China, November 2019. Association for Computational Linguistics. LinguuttioonalLnguuiiiisiii doi: 10.18653/11/D19- 1250. URL URLlhttts://cclanthoooo org/
119-1250/
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language Models are Unsupervised Multitask Learners. OpenAl blog. 1(8):9, 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive Deep Models for Semantic Compositionality Over a Senti- ment Treebank. In David Yarowsky. Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard (eds.), Proceedings of the 2013 Conference on Empirical Methods in Natural Langwage Prucessing, pp. 1631-1642. Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL httl:t/pp///caamnooooooooo/////0 0r///131170/
Fabian M. Suchanek, Mehwish Alam, Thomas Bonald, Lihu Chen, Pierre-Henri Paris, and Jules Soria. YAGO 4.5: A Large and Clean Knowledge Base with a Rich Taxonomy. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '24, pp. 131 131-140, New York, NY, USA, 2024. Association for Computing Machinery. ISBN URL URL
9798400704314. doi: 10. 10.114466266777.36577766 3657876
Chenmien Tan, Ge Zhang, and Jie Fu. Massive Editing for Large Language Models via Meta Learning. In International Conference on Learning Representations, 2024. URL https://openreviie net/pdffiddLLL11J0PPP
Qwen Team. Qwen2.5: A Party of Foundation Models, September 2024. URL htRp:ttts:////enln github.io/bloggqwenn...
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Tal Linzen, Grzegorz Chrupala, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Bel- gium, November 2018. Association for Computational Linguistics. doi: 10.18653/1/W118-4446
URL BtRL/https://acaanhholloc wwtploooy..orr/11-4444140
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.
http:/thww..oooo////ooooooo////e tranrrsh--ttranssre--------ee May 2021.
Peng Wang. Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models. Advances in Neural Information Processing Systems. 37:53764-53797, 2024a.
Peng Wang, Ningyu Zhang, Bozhong Tian, Zekun Xi, Yunzhi Yao, Ziwen Xu, Mengru Wang, Shengyu Mao, Xiaohan Wang, Siyuan Cheng, Kangwei Liu, Yuansheng Ni, Guozhou Zheng, and Huajun Chen. EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models. In Yixin Cao, Yang Feng, and Deyi Xiong (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 3: System Demonstrations), pp. 82-93, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653/v aelmmoo................. httpttss::/ http:::::///ccaaahoooooooo www.aanthhooooooooo//22222/ /f ac adem-emos......... 9/
Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. Knowledge Editing for Large Language Models: A Survey. ACM Comput. Surv., 57(3), November 2024c. ISSN 0360-0300. doi: 10. 10.1145/3698590. URL https: /ttt:s://ooi..
org/ 000 111t:////.1114/36959999
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural Network Acceptability Judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019. doi: 16.1/ tacl a a0090. URL http:::///ccaantollll..o/r//////
12

---

Published as a conference paper at ICLR 2026
Adina Williams, Nikita Nangia, and Samuel Bowman. A Broad-Coverse Challenge Corpus for Sentence Understanding through Inference. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 101/663/////8--1/1
URL http:: ac www.aaoooonnoooonooooooooo www.a/...rg///1111//0 1
Paul Youssef, Osman Koraş, Meijie Li, Jörg Schlötterer, and Christin Seifert. Give Me the Facts! Survey on Factual Knowledge Probing in Pre-trained Language Models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 15588-15605, Singapore, December 2023, Association for Computational Linguistics. doi: 10. 186531866//0000 htt.://.2222....nnnn-nnemee findinngs-mmll.... 1043. URL https:://allanthooooyy org/ 2023. findings- emmil. 1043
Paul Youssef, Zhixue Zhao, Daniel Braun, Jörg Schlötterer, and Christin Seifert. Position: Editing
Large Language Models Poses Serious Safety Risks. In Forty-second International Conference on Machine Learning Position Paper Track, 2025a. URL URLLhttps://ppenrvviees http://wwwwe//ooomm/ee
QLKBm1PaCU
Paul Youssef, Zhixue Zhao, Jörg Schlötterer, and Christin Seifert. How to Make LLMs Forget:
On Reversing In-Context Knowledge Edits. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Papers), pp. 12656 12669, Albuquerque, New Mexico, April 2025b. Association for Computational Linguistics. ISBN 979-8-88 111-8-881166 1961-189966 URL htLp:ttts////ccaattooooo wwwp://og..oog./..n.oomm 2ra.a//0.....a.a1loooooooo naa.c-long..oggoo 630/
Paul Youssef, Zhixue Zhao, Christin Seifert, and Jörg Schlötterer. Has this Fact been Edited? Detecting Knowledge Edits in Language Models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 9768-9784, Albuquerque, New Mexico, April 2025c. Association for Computational Linguistics. URL.
ISBN
oww.orggg//2//. wwwp://r///....aaalaaaaa..aa... 4927
Mengqi Zhang, Xiaotian Ye, Qiang Liu, Shu Wu, Pengjie Ren, and Zhumin Chen. Uncovering Overfitting in Large Language Model Editing. In The Thirteenth International Conference on Learning Representations, 2025. URL http:s//ooeeeeevee.eeee/e//ootod
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can We Edit Factual Knowledge by In-Context Learning? In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing. pp.
Singapore, December 2023. Association for Computational Linguistics. doi: 10. 18653/v11/0223 wwwp:/2mmm.mmmmmamna.mm.mm 296.
orgo /222.,203322 momlp-main. 296/
13

---

Published as a conference paper at ICLR 2026
A AUTOREGRESSIVE TRANSFORMERS
Transformer language model can be seen as a function M : X' - that maps an input x = (#1,...,,EN) that consists of N tokens input token ž; consists of its corresponding representation in embedding space and its positional embedding, i.e., hy encode(x;) pos(x;) and hy IR These initial representations are then processed through L subsequent Transformer layers. In each Transformer layer 1 E {1.....L) the representations from the previous layers are processed using multi-head self-attention (MHSA) and MLP layers as follows:
output token y y. The initial representation of each
h af + m + h/-1
(3)
df = M ...,. 1
(4)
m = o(W (at + D)
(5)
where is a non-linear function, and WR K, Wv E Rexd The final output is determined by computing the hidden state that corresponds to the final token from the last layer y = decode(h
B RANK-ONE MODEL EDITING (ROME)
ROME (Meng et al. 2022). prominent rank-one model editing method, first identifies the parameters responsible for fact retrieval using causal tracing. After identifying the MLP modules in middle layers as essential for fact retrieval, ROME updates the factual associations by conducting rank-one update to the MLP projection matrix Wy in one of the middle layers. This update can be written as:
WV Wy + WN = [w], ..., w/x]
where w/, ... w are the rows of W.. WN is a rank-ons one matrix, and can therefore be written as the product a column vector u and a row vector T.:
(7) ROME updates the targeted fact by constructing and adding WN to the original weight matrix Wy. We show how the rank-one property of the update may be used to identify edited layers App. [C and how Wi can be used to identify the edited relation in App.
W = uuuu 7
C IDENTIFYING EDITED MODELS
In order to develop a better understanding of the effects of editing with ROME on model weights, we first analyze the rank-one update ROME (Sec.C.) and then examine how this update affects the similarity among the rows of the updated matrix (Sec. C.2
C.I RANK-ONE UPDATE ANALYSIS
Equation 7 shows that the rows of the update matrix WN are merely scaled versions of the row vector o" and that depending on the scaling factors (elements of u). these rows can have one of two opposite directions (depending on whether the scaling factors are positive or negative). We analyze how many rows of W,, have the same direction and how many have opposite directions.
Results. Fig. shows that more than 80% of the row vectors of the update matrix W,, have the LLAMA3 the update balanced, roughly 50%
same direction in the GPT models. Conversely, of the vectors have one direction and the rest have opposite direction. This suggests that adding W, to original matrix Wi might be moving the majority of the rows of Wy in one direction in the GPT-models.
C.2 Row VECTOR SIMILARITIES
14

---

Published as a conference paper at ICLR 2026
Figure 7: Percentage of row vectors in the update matrix WN having the same (blue, circled pattern) or opposite (orange, cross pattern) directions with standard deviation. More than 80% of the vectors have the same direction in the GPT models.
GPT2-XL
2030
1333
P17
1010
1333
10M
9339
0331
M87
CCM
12
1333
1331
1677
1227
N49
9227
1609
9230
7339
9339
W411
1411
9377
R34
1977
Figure 9: Average pairwise cosine similarity (pcs) of edited and unedited matrices in different layers. We show the values with standard deviation in Tab. 17 in the appendix.
Given that the majority of the row vectors of the update matrix W' in the GPT models have the same direction (Sec. C.I). we hypothesize that adding the update Wy to the original matrix Wv leads to an increase in the average pairwise cosine similar- ity among rows of the updated matrix W.,. We sketch the
resulting vectors
intuition for our hypothesis in Fig. g To verify our hypothesis, we evaluate the increase in the average pairwise cosine similar- ity between the MLP projection matrix before editing Wv and after editing Wi- We compute the pairwise cosine similarity (pcs) for given matrix W follows:
update vectors original vectors
Figure 8: Intuition for the increased pes score after editing. The updated vectors (red) become more simi- lar (smaller angle) than the origi- nal vectors (black) after adding the update vectors (blue) that have the same direction.
n
n
pcs(W)=
sim (w,, wy)
(8)
n
i= 1=0
We compute the increase in pairwise cosine similarity (os(W)) (e((W))pss(W()) Positive values indicate increased pcs, whereas negative values indicate decreased pes.
[pe[WWy]
Results. We observe huge increase in the mire cosine similarity after editing in the GPT- models (e.g., more than 175x x with GPT2-XL and relation P190, and more than 25x with GPT-J and relation P138, see appendix Fig. 15 for full details). Conversely, we observe no significant increase with LLAMA3, due to the balanced update in terms of the directions the row vectors (cf. Fig. (7). For GPT-models,
plot the pcs values of the original unedited MLP projection matrices from all layers and compare them to the edited matrices from various relations in Fig. [ (corresponding plot for LLAMA3 in appendix Fig. 17). The extremely high pes values of the edited matrices make them easily distinguishable from the original unedited matrices the GPT--models. This indicator can be used examine and identify edited layers.
D PREDICTING EDITED RELATIONS
The rank-one update of ROME, WN, depends on the subject s, the relation 7 and the new object o'. This means if two separate updates share the same subject, relation or object, their corresponding
15

---

Published as a conference paper at ICLR 2026
#Caasses
Baseline
GPT2-XL
GPT-J
META--LAAMA-3-8B
2
50.60
99.40
96.00
92.40
30.53
96.67
96.67
85.07
5
19.60
90.32
92.24
78.56
10
10.32
84.64
83.20
56.72
15
6.77
76.19
72.77
44.05
20
5.30
72.94
68.10
33.36
25
4.19
67.84
63.39
29.11
30
3.59
64.88
57.20
26.59
Table 5: Accuracy for predicting the edited relation based low-dimensional representations of the edited matrices using logistic regression classifier. We experiment with different numbers of relations (#Classes).
update matrices will share some characteristics. We hypothesize that the updated matrix Wy can be used to derive higher-level information about the edited subject, relation or object. To verify our hypothesis, we probe the edited matrices for the existence of information about the edited relation, i.e. we train a linear classifier to predict the edited relation. Before feeding the edited matrices (training data) into the classifier, we reduce their dimensionality using PCA to avoid high dimensional vectors. We experiment with different numbers of relations (classes). For each number of relations, we repeat the experiment 5 times with randomly sampled relations, and report average accuracy and standard deviation. We logistic regression matrices, equally distributed across all used relations, to optimize the PCA projection. We transform the high-dimensional edited matrices through the PCA projection into compact 50-dimensional subspace. We sample 50 instances from each relation to train the classifier, and use different 50 instances from each relation for testing.
linear classifier. We use maximum of 100 edited
Results. Tab. shows high accuracy compared to random baseline across all numbers of relations (classes). The accuracy with 2, 3, and relations above 90% for the GPT-models and above 75% for LLAMA3, Even though the performance across all relations and models is significantly higher than the random baseline, we notice that the accuracy with LLAMA3 is lower than the accuracy with the GPT-models, in particular for increasing numbers of relations. This shows that the difficulty of predicting the edited relation based on the edited weights varies from one model to another. Using higher-dimensione representations more advanced classifiers might bring further performance gains. We leave exploring these aspects to future work. In practice, one can focus on relations that suspects be targeted by malicious knowledge editing attain high classification performance.
E BEYOND RANK-ONE MODEL. EDITS
In this section, we investigate to what extent our methods for tracing and reversing edits generalize to other KEs such MEMIT (Meng al. (023) and AlphaEdit (Fang al. (025) that, similar ROME, belong to the -ecte-and-edit category, and MEND (Mitchell et al. 2022a). meta-learning KE. We restrict ourselves specific LLMs and the CounterFact dataset due the high computational costs
for editing, especially in the case of MEND that requires training hypernewwrks.
E.1 TRACING EDITS
Experimental setup. Since Edi tScope requires access to edited weights. and more weights are affected in the KEs we consider matrices for MEND, matrices for MEMIT and AlphaEdit), we conduct the edits online to avoid storing large amounts of model weights. Given the high compu- tational cost, we run each experiment with only random seeds. In the case of MEND, we restrict ourselves to GPT2-XL (Radford et al.) 2019) and GPT-J (Wang & Komatsuzaki 2021). and use the hypernetworks provided by Meng et al. (2022). In the case of MEMIT, we use QWEN2,5 Team 2024) and MISTRAL-7B-VOL (Jiang al. 2023). For AlphaEdit, choices for the models are constrained by the availability of hyperparameters EasyEdit (Wang et al. 2024b), and the available compute. Since all of these KÈs change several layers, for edited
GPT2-XL and LLAMA3. Our
16

---

Published as a conference paper at ICLR 2026
1,0
Model
GPT2-LL
META-LL AMA---8
Reversal Arc.
Edktng
Accuracy
0.4
0.2
1000
2000
Figure 10: Reversal and editing accuracy with bottom-rank approximations W (r,8) for AlphaEdit in single-editing setting.
object prediction, we finetune only the layer that precedes the edited layers, because this has shown strong performance on ROME and r-ROME.
Results. Tab, 6 shows the results for tracing edits. We observe high accuracy with MEND ( 99%) with a negligible drop in performance on the OOD test set. A similar observation can be made with MEMIT QWEN with performance comparable the performance (cf. Tab[]. On MISTRAL the performance is less positive with an accuracy 66%. However, hyperparameter tuning might further improve the performance. On AlphaEdi performance in generating the edited object. We attribute this to the fact that AlphaEdit t avoids overfitting to the edited object, i.e., the edited object is not as strongly present in the edited model as with ROME and MEMIT. Generally, the results show the strong generalization of Edi EditScope to meta-learning KEs like MEND, and some locate-ane andieedndddittt KEs like MEMIT.
ROME and r-ROME
observe poor
E.2 REVERSING EDITS
Experimental setup. We apply our approach for reversing edits from Sec. 5oo the matrices that are edited with MEMIT, AlphaEdit, tt, and MEND. Given that these methods change several matrices, we apply our method to of the edited matrices simultaneously using different values, and report the reversal and editing accuracy. With MEMIT and AlphaEdit, t, we explore higher h values than before, because we notice some improvements with increasing k.
Results for reversing edits. Tab. 7 shows the reversal and editing accuracy with bottom-rank approximations for MEMIT. On QWEN2.5, we notice lower reversal accuracy than that observed with ROME (cf. Tab. 2.. and despite having higher values the highest reached reversal accuracy does not exceed 55%. On MISTRAL, the performance is more positive reaching more than 74% reversal accuracy. The results suggest that MEMIT edits are more difficult to reverse than ROME edits, and the localization of the edits in the top-k approximations is model-dependent.
Fig. 10 shows the editing and reversal accuracy with AlphaEdit. Here, we notice that higher values required reverse the edit. The highest reversal accuracy is reached with 475 for GPT2-XL (81%) and & = 2162 for LLAMA3 (64%). We believe this is due to AlphaEdit t projecting the changes onto the null space of the preserved knowledge, which causes the edits become less pronounced, i.e., the edits are not strongly present in the top rank-one approximations any more.
Tab. shows the results for MEND. We notice that the highest reversal accuracy (> 70%) is reached with & on both models, and that increasing does not bring further improvements. We also notice that the editing accuracy reaches almost zero with A 1. This suggests that the edits
17

---

Published as a conference paper at ICLR 2026
1.0
Model
QWEN225-78
MISTIAL-78-40.1
GPT:J68
Metric
Reversal
Editng
Accuracy
0.4
0.2
1000
2000
Figure 11: Reversal and editing accuracy with bottom-rank approximations (r(nk)) batch editing setting.
(r.
for MEMIT in
mostly localized in the top-1 approximation. However, recovering all of the original outputs remains challenging.
We show examples for reversing edits with MEMIT, AlphaEdit and MEND in Tab, 8
9 and
respectively.
Method
Model
Acc.
Std
Acc. (OOD)
Std (OOD)
GPT2-XL
99.45
0.73
99.16
1.04
MEND
GFT-1-6B
99.72
0.48
99.52
0.48
QWEN2.5-7B
91.19
3.73
83.35
YENТT
MISTRAL--B
66.19
3.96
61.21
9.70
GPT2-XL
1.89
0.88
0.31
0.53
AlphaEdit
MITA--LL-MM--3-R
2.83
1.01
0.08
0.13
Table 6: Accuracy of Edi tScope for generating the edited object based on the edited matrices of MEND, MEMIT and AlphaEdit when training only the layer that precedes the edited layers.
REVERSING BATCH EDITS
In addition reversing single edits, we experiment with reversing batch-edits. We consider MEMIT and AlphaEdit t for this experiment, since these are capable of batch editing. We edit 1,000 facts with both methods, exclude failed edits and apply our reversal approach all affected matrices. We consider higher & values, because we observe improved performance when increasing k. We do not experiment with every possible value, but rather report the reversing and editing accuracy for every 5th k value to reduce the computational costs.
Fig. I1 shows the results for MEMIT. The highest reversal accuracy for GPT-J (59%), MISTRAL (67%) and QWEN (47%) is reached with 1: 105, 1: 490 and h: 1065 respectively. The performance lower than what we observed in the single edit setting (cf. Tab. 7). indicating that reversal with MEMIT becomes more challenging
increase the number of edits. The results for AlphaEdit t are shown in Fig. 12 The highest reversal accuracy for GPT2-XL (81%) and LLAMA3 (63%) is reached at 550, and 1065 respectively, which is similar to the performance in the single edit setting (cf. Fig. 10). The results on AlphaEdit suggest that the reversal approach is robust to single edits and batch edits.
18

---

Published as a conference paper at ICLR 2026
QWEN25-7B
MISTRAL--BB
Editing
4
f
Editing 4
0
0.81
100.00
0.92
100.00
90.24
96.79
2
27.24
32.11
5,50
95
33.74
32,93
5.05
95.41
38.21
34,96
5,50
94.
37.40
30.49
5,96
94.04
42.28
26.02
6.88
93.58
43.90
25.61
6.42
93.12
8
45.53
20.73
9.63
89.45
46.75
21.95
17.43
78
45.53
19.92
26.15
72.48
11
44.31
17.48
32.11
66.06
46.34
16.67
33.94
63.76
46.75
13.82
39,91
$5.05
11.79
39.45
15 46.75
12.20
42,66
49.08
12.20
45.87
45.41
17 47.97
11.38
48.62
40.37
49.19
9.76
49.08
38. 07
19
50.81
8.54
53.67
33 94
50.41
9.35
$3.67
32.11
21
47.15
8.94
53,67
30.73
23
49.59
8.13
57.80
26 18
25
49.59
7.72
61.01
21.10
3.69
27
$2.44
5.69
65.60
14.22
29
51.22
5.69
66.97
14.22
31
$2.03
6.91
68.35
11.93
33
54.88
6.10
67.89
11.01
35
50.81
5.69
18
7.80
36
50.00
5.28
69.72
7.80
37
51:22
5.28
72.02
7.80
38
50.00
4.88
72.48
7.80
79
50.41
4.47
72.02
7.34
$1.63
4.88
72.94
49.59 41
4.47
72.48
5.96
48.78
3,66
71.56
5.05
49.59 43
3.66
69.72
5.96
44
50.81
1,07
71.10
5.05
45
50.00
4.88
71.56
2.75
46
50.00
4.07
71.56
3.21
49.19
4.07
72.02
2.75
48
47.97
3.66
72.94
2.75
48.37
3.66
72.48
2.75
50
47.97
4.07
72.94
2.29
51
47.97
3.25
72.48
2.29
52
47.15
3.25
73.39
2.29
53
45.93
4.07
2.29
14
46.34
4.07
74.77
1.83
56 46.34
2.85
74.77
1.83
58
47.97
2.85
72.94
1.83
60
47.56
3.25
71.56
1.38
Table 7: Reversal and editing accuracy with bottom-rank approximations W (r.k)
for MEMIT.
G REVERSING SEQUENTIAL EDITS
We also consider reversing sequential edits. We consider editing setting similar to that of Fang et al. (2025), where we edit total of 1,000 facts with batch size of 100. We consider MEMIT with GPT2-XL, GPT-J and QWEN2.5, and AlphaEdit with GPT2-XL and LLAMA3 for this experiment. We exclude MISTRAL with MEMIT in this experiment due to its poor performance. As in previous experiments, apply reversal approach to all edited matrices.
19

---

Published as a conference paper at ICLR 2026
Input
k
Edited Object
Orig. Output
Approx. Output
QWEN2.5-7B
The ofticial language of Timurid Empire
33
Portuguese
Persian. The Timur
() A. English
1s
M. S. Viswanathan's 's occupation is
33
actor
listed as a mathemati-
A. teacher
cian
The mother tongue of Hyeon-jcong
33
French
Korean, she has
Korean, but she can
is
Ozumba located country of
Russia Mumbai
Nigeria. It situated the United States and
where the X the region the world
Charles Nungesser native
33
MISTRAL-7B
The mother tongue of Thomas Joannes Stieltjes is
56
English
Dutch. Hc was born
Dutch. He was bomn
NRJ Group, that was created in Pat Scully holds citizenship from
56
Shanghai
1981
1999
56
Germany
the United States of America the reach of fans
the United States of America the scope of the A
2013 Interazzinali BNT. d'Ttalia
56
Californi
within
Robert William Muench is
pope
former American statis- tician
Table 8: Model outputs when using bottom-rank approximations (r.rr.) random set of MEMIT- edited facts. We use the best & for each model. The examples show that the model outputs with approximations (Approx. Output) are semantically close the original/unedited outputs (Orig. Output).
Imput
k
Edited Object
Orig. Output
Approx. Output
GPT2-XL
Maurice Vlaminck was native Linate Airport called after
475
Ottawa
the town Vlam the plane was reported missing
the town plane reported missing
475
Florence
The law in Bahia declares the language David Carney, the
475
Finnish
of the country to be former head of the U
of the country to be former governor of the Bank
475
basketball
Cuncha Espina passed away at
475
Melbourne
the age of 84 on
the age of 87 on
META-LLAMA--38
Autonomous University of Madrid,
2162
Sweden
the city Madrid,
the city of Madrid,
which is located in Charles Nungesser is native to The headquarter of Majorette is located in
2162
Mumbui London
the United States. He the heart of the French
the United States. He the heart of the city
2162
Zdeno Chára, the
2162
soccer
Boston Bruins captain, is the age of 70
captain the Czech public the age of 88
Concha Espina passed away
2162
Melbourne
Table 9: Model outputs when using bottom-rank approximations (r.k) ElphaEdit-editee facts. We use the best & for each model. The examples show that the model outputs with approximations (Approx. Output) are semantically close to the original/unedited outputs (Orig. Output).
random set of
Fig. 13 shows the results for MEMIT. The highest accuracy for GPT2-XL. (81%) reached at k: = 750.
for GPT-J (59%) at & = 125 and for QWEN2.5 (49%) at 1: 565, which is similar to the performance
observed in the batch editing setting (cf. Fig [TI. The results for AlphaEdit are shown Fig 14
Similar to the batch editing setting (cf. Fig 12]. the highest reversal accuracy for GPT2-XL (81%) reached at Ie = 465, while the highest accuracy for LLAMA3 (62%) is reached at = 1165. Generally, notice that reversal approach performs better with smaller models such as GPT2-
XL, and that the performance in the sequential editing setting corresponds to the performance in the batch editing setting, which shows the robustness of approach in different settings.
20

---

Published as a conference paper at ICLR 2026
Model
GPTZ-0L
Matric
Reversal Acc.
Editing
Accuracy
0.6
0.4
0.0
1000
2000
2000
4000
K
Figure 12: Reversal and editing accuracy with bottom-rank approximations W (r,k) batch editing setting.
V for AlphaEdit
1.0
Model
GPT2-0L
OWEN2 5-78
Metric
Reversal
Editing Ace.
Accuracy
0.6
0.4
1000
2000 K
3000
4000
Figure 13: Reversal and editing accuracy with bottom-rank approximations W (r.k) sequential editing setting.
V for MEMIT in
21

---

Published as a conference paper at ICLR 2026
GPT2-XL
GPT-J-6B
k:
Reversal Acc.
Editing Acc. 4
Reversal Acc.
Editing Acc. +
0 0.00
100.00
0.78
100.00
1 74.37
0.00
70.98
0.78
2 56.30
0.00
10.98
0.00
3
19.75
12.18
46.67
0.00
4
26.89
0.84
36.86
0.39
5
32.35
0,42
44.31
0.78
6
39.92
0.84
50.20
1.57
7
37.82
0.42
50.20
1.18
8
39.92
0.42
50.59
1.18
9
53.78
0.42
48.24
2.35
10
56.30
0.42
45.10
1.18
11
58.40
0.42
49.02
0.78
12
61.76
0.84
48.63
0.39
13
63.87
0.84
52.94
1.57
14
64.71
0.84
52.55
1.57
15
65.13
1.26
54.90
1.96
Table 10: Reversal and editing accuracy with bottom-ran approximations (n,))
for MEND.
Input
Edited Object
Orig. Output
Approx. Output
GPT2-XL
John James Rickard Macleod's domain of work is
psychology
the study of the history
the study of the history
BRIC, which was named for Oliver Ames High School,
Apollo
the Latin word for " the town Ames, the United States, the United States in
Latin word for town of Humb former state U.S.
Pennsylvania
Trakli Alasania has citizenship from Leonardo Balada found employment
Hungary
Paris
GPT-J-6B
The native language of Symeon of Polotsk
1
French
Belarusian.
unknown. He
Nathuram Godse, citizen of The language of El Correo is
1
Italy English
India, was born mixture of Spanish and
Indian state Rajasthan mix of the local
1
The language used by Gilad Atzmon Immaculate Machine, that was started in
Italian
not only offensive, but the carly 90s,
not only a reflection of the late '90s
Sheffield
Table 11: Model outputs when using bottom-rant approximations (r,kk,)) on a random set of MEND- edited facts. We the best for each model. The examples show that the model outputs with approximations (Approx. Output) are semantically close to the original/unedited outputs (Orig. Output).
H ADDITIONAL RESULTS
In this section, we provide more details results. Additionally, we re-run our experiments on new editing dataset constructed to evaluate generalization. Tab. 14 shows the KL-divergence loss between the original model and the r-ROME-edited model with bottom-rank approximations, Tab. [5]
shows the relations used in experiments. Tab. shows the maximum cosine similarity values between vectors of the update matrix WN and the vectors of W for different values. Tab. 21 shows the dimensionality of the edited matrices in each model.
H.1 YAGO DATASET
Dataset construction. We use the knowledge base YAGO 4.5 SSuhhank et al. 2024) to create an editing dataset that contains more diverse relations than CounterFact. YAGO 4.5 merges the
22

---

Published as a conference paper at ICLR 2026
1.0
Model
GPT2-LL
META-LLAMA AMTA-LLAMA-3-88
Reversal Arc.
Edktng
Accuracy
0.0
0.4
1000
2000
Figure 14: Reversal and editing accuracy with bottom-rank approximations W (r,8) in sequential editing setting.
for AlphaEdit
GPT2-XL
GFT-J-88
META-LLAMA-3SS
OWEN225-7B
Acc.
Editing Ace.
Reversal
Editigg
Reversal
Editing
Reversal
0
0.00
100.00
0.32
100.00
0.97
100.00
0.65
100.00
86:45
7.42
25.48
72.25
4.52
95.81
30.32
61.52
2,90
74.19
50.00
90.52
1.94
73.25
7.74
60.00
29.03
$4.52
$7.10
5 90.32
1.94
25.81
4,52
67.10
20.32
55.81
35.16
1.91
15.16
3.55
65.48
19.35
57.42
33.23
1.94
76.77
9 92.58
1.94
76.13
2,90
76.13
9.01
6000
28.71
10 93.55
1.94
77.10
2.90
76.77
9.03
62.26
27.42
11
61.91
16.77
13
0.97
78.06
2.26
79.68
6.77
62.26
26.77
14
93.55
0.99
78.71
1.94
79.03
6.77
62.58
26.77
15
93.87
0.97
77.42
1.94
79.35
6.45
62.90
24.19
Table 12: Reversal and editing accuracy with bottom-rank approximations W (r.)) increases, the edits are removed (editing accuracy drops), and the model is able to retrieve its original generations (reversal accuracy increases).
for r-ROME, As
CITZAL
KETN-LLAM---11
valdion
Increase in row-wise cosine similarity of W, after editing, A substantial increase in the Figure 15:
pcs score can be observed in the GPT models.
taxonomy of Wikidata with the taxonomy of Schema.org create consistent knowledge base. We manually selected a set of diverse relations from YAGO and extracted the corresponding subject and object pairs. We filtered out relations with less than 1000 subject/object pairs. Afterwards used DeepSeek RI (DeepSeck-AL et al. 2025) to generate editing prompts and paraphrased prompts for each relation. We manually checked the correctness of the generated prompts. YAGO 4.5 is licensed
23

---

Published as a conference paper at ICLR 2026
GPT2-XL
Accuracy
64
10
Trained Layer
GPT-J-6B
2.3
Accuracy
0.4
#0aecy10001
kdded
10
14
25
Trained Layer
META-LLAMA----B
1.3
Accuracy
EdRed loee
Trained Layer
Figure 16: Accuracy in generating the edited object based on edited matrix when training different layers. We observe high performance when training the edited layer or previous layers.
META-LLAMA-3-B8
uncoted SAN
PL432
P264
P101
Pairopit Similaritt
617
W
P108
P127
P22
7440
2130
P160
2455
P1SL
F22
P541
P27
P120
2.
Layer
Figure 17: The average pairwise cosine similarity (pes) of edited and unedited matrices from different layers.
24

---

Published as a conference paper at ICLR 2026
Input
k
Edited Object
Orig. Output
Approx. Output
GPT2-XI.
George G. Siebels, Jr. worked in
11
Amsterdam Belfast
the U.S.
the U.S.
Where is Cairo International Film Festi- val? It...
1I
the heart of Cairo,
the heart of Cairo,
Charles- Charles-Auguste Questel died at The original language of The Irish Times was
London German
the age of 87 on written in the late
the age on
11
written late
The language used Francesc Eixime- nis
Spanish
bit of mouth
not the same that
GPT-J
Perfil is written in Five Man Electrical Band, that was started in Udo Lindenberg found employment in
14
Greek London
Spanish, and is the early 1970s.
C++ and is distributed the late 1960s,
14
14
Cairo
the German army in 1939
the German army in 1939
The official religion of Edwin Northumbria is SportsCenter was released on
Islam
the Christian faith. He
the Christian Church, The PlayStation 2 October 30, 2009.
CBS
North
META-LLAMA-3-SB
Elinor Ostrom works in the field of
14
ccology
political cconomy and public choice the north of the country composer, violinist
political economy and public choice the north of the city a composer, is born Apple to help you man- age
Mandara Mountains, which is located in Giovanni Battista Vitali, who works as Disk Utility was created by
14
Greece
14
journalist
14
Google
Apple to help users man- age
The language of Haratch was
German
the language of the Har
spoken by the Haratch
QWEN2.5-7B
Hugo Schiff lost their life Renault is produced by
Paris
the age of Renault, French auto- mobile
the age
15
Fiat
Renault, French mobile
Ricardo Fary, the What is the twin city of Houston? It is
15
quarterback
2017
founder of the company, Galveston, _AnA, Microsoft
15
Prague
Galveston, which AnA. Microsoft
Windows Server 2003 is a product of
15
BMW
Table 13: Model outputs when using bottom-rank approximations (r.(r.k) V on a random set of facts, edited with r-ROME. We use the best & for each model. The examples show that the model outputs with approximations (Approx. Output) are semantically close to the original/unedite outputs (Orig. Output).
Figure 18: Percentage of row vectors the update matrix WN having the same direction or opposite directions. More than 80% of the vectors have the same direction in the GPT models. Yago Dataset.
under Creative Commons Attribution 4.0 International License (CC BY 4.0), and we will publish dataset under the same license. We use 15 relations (shown in Tab. [5) from the newly constructed dataset in experiments.
Results on Yago. Fig. shows the direction distribution of row vectors in the update matrix. Fig. 19 shows the average pairwise cosine similarity (pcs) of edited and unedited matrices from different layers. Fig. shows the increase in row-wise cosine similarity after editing. Tab, [8 shows the results for predicting the edited relation.
25

---

Published as a conference paper at ICLR 2026
k GPT2-XL
GPT-J-6B
META-LLAMA-3-8
QWEN2.5-7B
5,813 + 2,355
11,410 + 3.724
10.004 +: 3.001
8,791 +: 3,11
0.196 0.812
5,933 5.105
9.761 = 4.102
4.896
2
0,166 + 0.811
0.617 1.543
6.328 = 5.263
3,412 + 4.387
0.093 1 0.451
0.407 0.898
4.142 1. 4.785
2.968 4.151
4
0.049 1 0.380
0,401 0.881
2.254 1. 3.714
2.726 3,955
0.049 + 0.393
0.304 = 0.588
1.489 + 2.833
2.567 + 3,883
6
0.053 + 0.483
0.278 0.514
1.414 + 2.754
2.325 3.799
0.032 + 0.191
0.247 U.326
1.058 + 2.316
2.228 3,712
8
0,031 0.180
0.245 0.318
0.854 2.031
2.204 3.656
0.026 + 0.142
0.247 = 0.321
0.738 + 1.899
1,951 =
10
0.021 0.107
0.237
0.729 1
1.715 3.232
11
0.011 1 0.023
0.238 1 0.295
0.697 1 1.894
1,636 3.138
12
0.011 0.02
0.239 0.296
0.655 =
1.627 3.134
13
0,011 + 0.020
0.235 = 0.283
0.652 # 1.830
1,647 3.17
14
0.010
+ 0.016
0.234
0.77
0.638 +: 1.830
1.625 3,149
15
0.010 + 0.015
0.235 + 0.277
0.600 + 1.753
1.539 3.111
Table 14: KL divergence between the original model and edited models with r-ROME after using bottom-rank approximations (r.k) reverse the edits. The results show the effectiveness of bottom-rank approximations in recovering the original model's 's output distribution.
I THE USE OF LLMs
We used LLMs for polishing the writing, and for writing some parts of the code. We reviewed and validated all outputs. Additionally, LLMs were used generate prompts for the Yago dataset as described in Sec. 3
26

---

Published as a conference paper at ICLR 2026
Relation
Input
True object
Edited object
CounterFact
P101
John James Rickard Macleod's s domain of work The mother tongue of Danielle Darrieux Billy Roche, who works as William Roes-Mogg. who is employed by BBC One, by
physiology
psychology
P103
French
English
P106
actor
architect
P108
BBC
CBS
P127
BBC
Sega
P1303
Toko Yasuda, the
guitar
piano
P131
Galata
Istanbul
Naples
P136
What does Heath Brothers play? They play
jazz
opera
P138
Centocelle Airport named for The official religion of Edwin of Northumbria is
Rome
Milan
P140
Christianity
Islam
P1412
The language used by Gilad Atzmon is The headquarter Monell Chemical Senses Center located in Autonomous University of Madrid, which located in
Hebrew
Italian
P159
Philadelphia
Mumbai
P17
Spain
Sweden
P176
Ferrari F40, developed by Apple AS was created by Gilles Grimandi was bor in What is twin city Lyon? is Charles Alfred Pillsbury expired Mahmoud Fawzi has a citizenship from Inner Circle railway line be found Pidgeon Island belongs to the continent of The original language The Joelandic Dream was Northwest Territories, official language is Robert William Muench is Corsica Perey Snow, the
Ferrari
Microsoft
P178
Apple
Google
P19
Gap
Montgomery
P190
Beirut
Manila
P20
Minncapolis
Berlin
P27
Egypt
Germany
P276
Melbourne
Singapore
P30
Anturctica
Asia
P364
Icelandic
Tamil
P37
English
Tamil
P39
bishop
pope
P407
French
Dutch
P413
linebacker
goaltender
P449
The Loner was released Shree Pundalik, created in
CBS
HBO
P495
India
Sweden
P641
Andreas Ivanschitz professionally plays the sport Nathrakh, that created Leonardo Balada found employment
soccer
football
P740
Birmingham
Philadelphia
P937
Pittsburgh
Paris
Yago
P112
The founder of Cabinn Hotels is
Niels Fennet
Tohy Neugebauer
P131
The location of Nara Institute of Science and Technology The helief system of Al-Aziz Muhammad is The artist of the painting The Marriage of the Virgin is parent taxon Puccinia recondita The developer of Grand Theft Auto V is The river Havel flows into
Japan
Oran
P1405
Sunni Islam
Anglicanism
P170
Raphuel
Georges Braque
P171
Puccinia
Microchirppera
P178
Rockstar London
High Voltage Software
P200
Elbe
Ohura River
P238
The IATA code Bankstown Airport The nationality of Giulio Paradisi is
YSBK
KGTB
P27
Italy
Hungary
P276
The location of the historical event Second Battle Zurich
Zürich
Constantinople
P463
The bund of Freddie Mercury
Queen
Love
P57
The director of Labyrinth of Flames
Katsuhiko Nishijima
Carlo Vanzina
P840
story of set
York City
Angeles
P915
The filming location of More Than Life Stuke is
Poland
France
P921
The subject of The Good Termorist is
temorism
social theory
Table 15: The relations we use in our experiments alongside examples.
27

---

Published as a conference paper at ICLR 2026
GPT2-XL
GPT-J-6B
META-LLAMA-3-SB
Max. Sim.
Std
Max. Sim.
Std
Max. Sim.
Std
0.98
0.08
0.77
0.21
0.2
0.24
0.07
0.03
0.45
0.22
0.37
0.35
0.11
0.06
0.06
0.07
0.25
0.24
0.07
0.06
0.02
0.02
0.29
0.25
0.01
0.02
0.06
0.05
0.15
0.16
0.02
0.02
0.06
0.03
0.11
0.08
0.02
0.01
0.03
0.04
0.12
0.14
0.0
0.0
0.01
0.01
0.11
0.11
0.02
0.01
0.02
0.02
0.05
0.07
10
0.02
0.02
0.02
0.02
0.04
0.04
11
0.03
0.02
0.01
0.01
0.04
0.06
0.01
0.01
0.01
0.01
0.05
0.07
13
0.01
0.01
0.01
0.01
0.03
0.04
0.01
0.02
0.01
0.01
0.03
0.04
15
0.01
0.01
0.03
0.02
0,04
0.05
Table 16: The maximum cosine similarity values between vectors of the update matrix WN and the vectors of W for different & values.
GPT2-XI.
GPT-J-6B
META-LLAMA-3-8B
Relation
pes
std
pcs
pes
std
unedited
0.000091
NA
0.000194
R
NA
PI01
0.0067619231
0.0039638884
0,0018001688
0,0008643679
-37756e-07
1.849e-07
P103
0.0059509007
0.00277431
0.0016752047
0.0006458259
-.1008-007
1.845c-07
P106
0.0066805076
0.0026219752
0.0019167275
0.0007191846
-3.92c-077
2.394c-07
P108
0.0069100604
0.0031947018
0.0018263827
0.0007619023
-3.567e-07
18895-007
P127
0.0102751931
0.0051715499
0.0024409223
0.0010431761
4.059e-07
L.763--07
P1303
0.0076706613
0.0030963009
0.0020462978
0.0008670002
3.889c-07
1.82c-07
P131
0.0098702735
0.0055993188 183
0.0045001531
0.0212773146
-.732c-07
1.966c-07
P136
0.0071806164
0.0031261909
0.001941 1065
0.0006443891
4.322c-07
1.257e-07
P138
0.0137165404
0.0086577839
0.005331668
0.025452119
-.461c-077
1.838e-07
P140
0.0078358574
0.0062726236
0.0020133093
0.0009927367
3.933e-07
2.257c-07
P1412
0,006187233
0.0026656243
0.001784752
0,0005955094
-3956e-07
1.998c-07
P159
0.008386647
0.0050185373
0.00218822
0.000959144 1461
3,391c-07
2.753c-07
P17
0.009551276
0.0044502795
0.00260327:5
0.0010471037 1037
-3552c-07
2.959c-07
P176
0.0108912224
0.0055227291
0.0019816675
0.00099757
4.325c007
1.372c-07
P178
0.0117567854
0.0110017566
0.0021579888
0.001029392
4.225e-07
1.69e-077
P19
0.007428109 1091
0.0030904648
0,002131322 3282
0,0009460897
3.971e-07
18827-007
P190
0.0178613566
0.0165129782
0.0029858089
0.0012192149
4,938c-07
1.712c-07
P20
0.007044959
0.0032487115
0.001820856
0.0009291652
4.0477-07
1,481c-07
0.0072250371
0.0033331825
0.0018882287
0.0006491200
3.945e-07
1.818e-07
P276
0.0096739383
0.00348082
0.0021771877
0.000703454
3.604c-07
2,357c-07
P30
0.0104348788
0.0078027795
0,0028299001
0.0011365804
4.144c-07
2.235c-07
P364
0.0055471755
0.0028574185
0.0016467369
0.0006391143
4,365c-07
2.311c-07
P37
0.0073669546
0.0043815362
0.001865609
0.0008194575
4.034c-007
2.107e-07
P39
0.0074461333
0.0031980671
0.0019889568
0.0008512599
3.884c-07
2.081c-07
P407
0.006763 1544
0.0027889247
0.0018608728
0.0007570319
3.9822-07
1.961e-07
P413
0.0069200734
0.0031209039
0.0019079371
0.0007796126
-3.95-007
1.713--07
P449
0.0071908987
0.0040715897
0.0020522842
0.000901399 3923
-.81c-07
1.799c-07
P495
0.0088219754
0.0069427193
0.0022077068
0.0009163789
-3522c-07
2.456c-07
P641
0.0056973715
0.0026820171
0.0017082412
0.000718366
-32228-077
2.105e-07
P740
0.0086965727
0.0042147134
0.0040305918
0.0150149984
-3.625-007
2.245e-07
P937
0.0069481413
0.0047418696
0.0018729484
0.0008598884
-3.912c-07
1,558c-07
Table 17: Pair-wist wise cosine similarity (pcs) scores with different relations from CounterFact.
28

---

Published as a conference paper at ICLR 2026
GPT2-XL
GPT-J
META-LLAMA-333R
#Classes Baseline
Accuracy
Std
Accuracy
Std
Accuracy
Std
50.6
98.4
1.95
97.0
3.08
95.0
2.83
30.53
99.87
97.87
94.27
3.35
19.6
97.52
1.04
95.68
2.6
90.08
3.77
10
10.32
94.76
0.96
88.64
2.35
82.24
5.2
15
6.77
92.32
1.05
83.87
1.44
74.27
1.44
Table 18: Accuracy for predicting the edited relation based on low-dimensional representations of the edited matrices using logistic regression classifier. We experiment with different number of relations (#Classes). The relations used are from the Yago dataset.
GPT2-XL
GPT-J-6B
META- -ETA--LLMM----B
Reversal Acc.
Editing Aec.
Reversal Acc.
Editing Acc. 1
Reversal Acc. 1
Editing Acc.
0 0.00
97.14
0.00
95.71
1.43
91.43
92.86
5.71
24.29
61.43
5.71
82.86
2 95.71
2.86
68.57
10.00
28.57
41.43
3
94.29
1.43
74.29
8.57
37.14
34.29
4
95.71
0.00
74.29
8.57
50.00
20.00
5
94.29
0.00
81,43
4.29
54.29
17.14
6
97.14
0.00
84,29
4.29
58.57
17.14
7
94.29
0.00
82,86
0.00
65.71
10.00
8
95.71
0.00
84.29
0.00
70.00
8,57
9
95.71
0.00
81.43
0.00
70.00
7.14
10
97.14
0.00
80.00
0.00
72.86
7.14
11
97.14
0.00
80.00
0.00
72.86
7.14
12
94.29
0.00
78.57
0.00
70.00
8.57
13
95.71
0.00
78.57
0.00
72.86
7.14
14
97.14
0.00
78.57
1.43
70.00
7.14
15
95.71
0.00
81.43
1.43
71.43
5.71
Table 19: Reversal/Editing accuracy on Yago and ROME with different r - k approximations of WV- As & increases, the edits removed (editing accuracy drops), and the model is able retrieve its original generations (reversal accuracy increases).
29

---

Published as a conference paper at ICLR 2026
Input
k:
Edited Object
Orig. Output
App. Output
GPT2-XL
The founder Cabinn Hotels
Toby
Neuge-
a former U.
a man who has been
bauer
The river Ergolz flows into The band of Jeff' Tweedy is The band of Tain Matthews The river Havel flows into GPT-J
Jiu River
the Black Sca. The Dire Straits
the Black Sca. The band of friends. hand of the future the Danube, and
Anthrax
band of friends. hand of Tain the Danube, and
11
11
Ohura River
The founder Tune Hotels
6
Laís of Portu-
has heen
who has been
gal
The director of The Mirror is The director of Darkman is
6
Polly Draper
who has been man who has been
who has been man who has been
6
Claudio
Fra-
gasso
The story of 24 is set in The subject of Goryeosa is
6
Los Angeles
the year 2024, and the life of the Buddha
the year 2401, the story of the life
6
orphan
META--.LAMAA--8B
The artist the painting Allegory of Vices is
Mary Cassatt
unknown. painting was
Mary Cassatt Mary
The artist the painting Religious Pro- cession in Kursk Province The river Melbbach flows into The subject of Net Voyne! is
11
Giulio Romano
Ivan Ivanovich Shish
Ivan Ivanovich Shish
11
Inn
the river Main in the the Internet and its im- pact back with new anime
the river Inn the the Internet and its im- pact
11
international re-
lations
The director of Brave Command Dag- won: The Boy with Crystal Eyes
Sally Potter
a 1996 anime
Table 20: Model outputs when using bottom-rank approximations (r.k) on random set of facts from the Yago dataset, edited with ROME. We use the best k for each model. The examples show that the model outputs with approximations (Approx. Output) are semantically close to the original/unedices outputs (Orig. Output).
Model
Edited Matrix Dim.
GPT2-XL.
6400 x 1600
GPT-J-6B
16384 x 4096
META-LLAMA-3-8B
14336 x 4096
QWEN2.5-7B
18944 x 3584
MISTRAL-7B--O.I
14336 x 4096
Table 21: The dimensionalities of the edited matrices for different models.
Dataset
License MIT License
CounterFact (Meng al. 2022 2024
YAGO (Suchanek et al.
CC BY 4.0
Table The datasets we use in this work and their licenses.
30

---

Published as a conference paper at ICLR 2026
GPT2-XL
uncdltes Myes
P338
+463
Pajewiie Simitally
P1LZ
P33:
P228
P200
7540
P57
P27
P915
PATO
X
Layer
GPT-J-6B
Simibutty
0.0035
P331
P238
9540
Pl035
P27
9915
PATO
P276
X P221
P171
0.0000
META-LLAMA-3-4
unedhes byere
P370
V
PA70 PITI
P276
P221
W
Layer
Figure 19: The average pairwise cosine similarity (pcs) of edited and unedited matrices from different layers (Yago Dataset).
O7T.388 VETN-LLOM---11
Talatton
Figure 20: Increase in row-wise cosine similarity of W after editing with the Yago dataset. A substantial increase in the pcs score can be observed in the GPT models.
31


Reviews:
[{"review_id": "Md0qgiRhZe", "reviewer": "Reviewer_8BSp", "summary": "This paper studies how to trace and reverse malicious edits in large language models that were modified using rank-one editing methods like ROME or r-ROME. It introduces two complementary defenses: (1) Tracing, which identifies the specific edited fact (object) directly from the edited weight matrix without needing the original weights or prompts, and (2) Reversing, which removes the malicious change by replacing the edited matrix with its bottom-rank singular value decomposition (SVD) approximation, effectively removing the edit signal concentrated in the top singular modes. Experiments across multiple LLMs show that tracing achieves near-perfect accuracy and reversal restores the model’s original behavior with high fidelity.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+ The proposed defense is practical and lightweight, requiring only access to the edited weights and no training data or edit prompts, making it suitable for real-world forensic use.\n+ The reversal approach is simple yet effective, using an interpretable SVD-based method that efficiently removes the edit signal while maintaining model integrity.\n+ The experimental validation is comprehensive and convincing, demonstrating strong performance across multiple models and datasets with clear quantitative results and ablation studies.", "weaknesses": "- The method’s generality is limited, as it is evaluated only on single-layer, rank-one edits and may not extend to more complex, multi-layer, or non-rank-one scenarios.\n- The evaluation scope is narrow, focusing mainly on object recovery and KL divergence without exploring broader behavioral or capability effects after reversal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "55vlNCSsmk", "reviewer": "Reviewer_AH7s", "summary": "The paper proposes methods to detect and undo malicious or unintended knowledge edits. These edits modify a model’s internal parameters to change factual outputs, which can be useful for updating information but also pose security risks. The authors introduce two tasks: tracing edits and reversing edits. Their tracing method infers the edited object solely from altered weights, achieving high accuracy. Their reversal method uses bottom-rank approximations from singular value decomposition to remove edits without retraining. The study demonstrates strong generalization across 2 datasets and 4 models, showing that both tracing and reversal are feasible using only model weights.", "questions": "- Can you clarify whether the proposed tracing and reversal techniques would still work, or how they might adapt, when applied to higher-rank or alternative editing methods such as MEMIT, MEND, or SERAC?\n- Have you considered testing their approach on real or naturally occurring edits (beyond controlled datasets) to assess its reliability in more practical or adversarial settings?\n- Can you provide a clearer explanation or empirical evidence for why bottom-rank components capture “pre-edit” information and effectively reverse edits?\n- What is the practical advantage of using the proposed reversal method over simply reapplying the inverse or original edit, especially given that model editing is typically meant to add or update knowledge rather than undo it?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper introduces a training-free framework for detecting and reversing malicious edits directly from model parameters, a new defense direction for LLM safety.\n- Experimental results show high accuracy and generalization across different models and datasets, suggesting good robustness.\n- The methods are computationally efficient and require no access to original weights or editing prompts, enhancing practical applicability for security auditing.", "weaknesses": "- The study focuses only on rank-one edits, limiting applicability to other editing methods and scenarios like MEMIT, MEND, SERAC.\n- The motivation\nThe evaluation scope is restricted to controlled datasets and synthetic edits, leaving real-world validation uncertain.\n- The interpretability of why bottom-rank approximations work well for reversal is not fully explored, reducing theoretical clarity of the mechanism.\n- The motivation for reversing edits is questionable, since model editing is primarily designed to update or add new knowledge rather than to be undone; thus, the practical need for edit reversal appears limited. The paper does not compare its reversal method against simply reapplying the inverse or original edit, which would be a more direct and intuitive baseline for restoring model behavior", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q7NL7vdD58", "reviewer": "Reviewer_QZJb", "summary": "This paper introduces the tasks of tracing and reversing malicious rank-one knowledge edits, relying on the modified model weights without access to the editing prompt or original weights. For tracing, the authors propose a novel method that trains the unedited weights with a fixed random input to decode the edited MLP projection matrix and accurately infer the edited object. For reversing, they introduce an efficient, training-free method using bottom-rank approximations to neutralize the edit and recover the model's original output distribution. The methods achieved high accuracy across various LLMs and showed generalization, highlighting the feasibility of developing robust countermeasures against adversarial knowledge manipulation.", "questions": "- How does the required model-specific tuning of the optimal rank k for bottom-rank approximation impact the practical deployability of the edit reversal method?\n- How can the reversal method be made less model-dependent?\n- To what extent does the non-identical nature of the model's reversed outputs (despite being semantically similar) and the accuracy imbalance between edit removal and original output recovery limit the utility of the reversal method?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed methods for both tracing and reversing are designed to operate solely on the edited weights, without requiring access to the editing prompt, unedited weights, or any other information about the edit. This makes the countermeasures more practical for real-world defense against malicious editing.\n- The tracing method achieved high accuracy in identifying the edited object and showed strong generalization to out-of-distribution data and different editing methods (ROME and r-ROME). Similarly, the reversal method recovers up to 93% of edits and significantly restores the original model's output distribution.\n- The edit reversal technique, based on bottom-rank approximations, is training-free, making it highly efficient. This same technique can be repurposed to distinguish between edited and unedited weights by observing the number of unique predictions on unrelated text, offering a robust detection mechanism.", "weaknesses": "- The effectiveness of the reversal (and analysis of rank-one approximations) is shown to be model-dependent. For example, the optimal rank k for bottom-rank approximation varies significantly across models (e.g., k=11 for GPT2-XL vs. k=15 for llama3 to achieve highest reversal accuracy), and the similarity of the top rank-one approximation to the update matrix is much lower for LLAMA3 than for GPT models. This suggests a need for model-specific tuning of the reversal hyperparameter. \n- The core of the methods, especially the reversal technique, relies on the assumption that the malicious edit is a rank-one update (like ROME or r-ROME). The effectiveness for different types of model edits is not explored. If the proposed method only works on rank-one update methods, the usability is limited since there are many other types of model editing methods like memory-based and constrained-tuning-based methods.\n- While the reversal accuracy is high (up to 93%). A qualitative analysis showed that even when reversed, the outputs, while semantically similar, are sometimes not identical to the original model's unedited outputs. Also, the decrease in editing accuracy is higher than the increase in reversal accuracy, indicating the method is better at removing the edit than fully recovering the original output distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i3m2LJS3OJ", "reviewer": "Reviewer_aKZH", "summary": "This paper proposes countermeasures against the potential misuse of model editing, namely **tracing and reversing edits**. The paper proposes using fixed inputs to guide the edited model in producing the target output. To approximate the update matrix, it employs rank-one approximations based on the highest singular values from a Singular Value Decomposition (SVD) of the edited matrix.", "questions": "* Lines 130–131: Why can’t the *unedited model* be the *original model*?\n* The results in Figure 4 show that the **maximum cosine similarity** differs significantly between GPT-J and Llama, yet their **reversal accuracy** is similar. How can this be explained?\n* In Figure 4, when *k = 1*, GPT2-XL achieves the highest **maximum cosine similarity** but the lowest **reversal accuracy**. How can this discrepancy be interpreted?\n* Based on the results in Sections 6.2 and 6.3, there seems to be no clear correlation between **rank-one approximation** and **reversal edits**. How do the authors explain this? Figure 4 indicates that smaller *k* values make the rank-one approximation closer to the update matrix, while Table 2 shows that larger *k* values yield better **reversal and editing accuracy**, and Table 4 shows that larger *k* values make the reversed model closer to the original model. These findings appear counterintuitive, as one would expect that a closer rank-one approximation to the update matrix would lead to the reversed model being closer to the original model. How do the authors reconcile this?\n* Can the results for **Qwen** in Figure 4 be shown?\n* Do the conclusions in lines 309–313 imply that the assumption at the beginning of Section 6.2 does not hold?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The paper presents a clear motivation and addresses an important research problem.\n* It proposes corresponding solutions for both **tracing** and **reversing** edits.\n* Extensive experiments are conducted to validate the effectiveness of the proposed methods.", "weaknesses": "* The writing of the paper needs improvement.\n* The proposed **tracing and reversing edits** methods are only validated on the ROME series of approaches, and their effectiveness on other model editing methods remains unknown, which limits the applicability of the proposed approach.\n* The paper only considers **rank-one updates** for single pieces of knowledge, without addressing multi-knowledge or sequential knowledge updates.\n* The results of **ANALYSIS OF RANK-ONE APPROXIMATIONS** and **REVERSAL** appear to be uncorrelated and may even contradict each other.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)