EAMET: ROBUST MASSIVE MODEL EDITING VIA EMBEDDING ALIGNMENT OPTIMIZATION
Authors: Yanbo Dai
Zhenlan Ji
Zongjie Li
Shuai Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Model Editing
Massive Editing
Large Language Models
Primary area: foundation or frontier models, including LLMs

Abstract: Model editing techniques are essential for efficiently updating knowledge in
large language models (LLMs). However, the effectiveness of existing approaches
degrades in massive editing scenarios, particularly when evaluated with
practical metrics. Their robustness is also limited in context-rich settings or
when editing multiple facts of the same subject simultaneously. We attribute
these failures to the embedding misalignment among knowledge items, which
undermines editing reliability at scale. To address this, we propose EAMET
(Embedding Alignment Model Editing in Transformers), which addresses this issue
by aligning the space of key and residual embeddings. Extensive experiments
across six LLMs and three datasets demonstrate that EAMET consistently
outperforms existing methods, achieving about 90\% editing efficacy when editing
10k facts.

Published as a conference paper at ICLR 2026
EAMET: ROBUST MASSIVE MODEL EDITING VIA EMBEDDING ALIGNMENT OPTIMIZATION
Yanbo Dai, Zhenlan Ji, Zongjie Li", Shuai Wang" Department of Computer Science and Engineering The Hong Kong University of Science and Technology (ydai851, zjiae, zligo, shuaiw ) wesse.us..e.... hk
ABSTRACT
Model editing techniques are essential for efficiently updating knowledge in large language models (LLMs). However, the effectiveness of existing approaches degrades in massive editing scenarios, particularly when evaluated with practi- cal metrics. Their robustness is also limited in context-rich settings or when editing multiple facts of the same subject simultancously. We attribute these failures the embedding misalignment among knowledge items, which un- dermines editing reliability at scale. To address this, we propose EAMET (Embedding Alignment Model Editing in Transformers), which addresses this issue by aligning the space key and residual embeddings. Extensive ex- periments across six LLMs and three datasets demonstrate that EAMET con- sistently outperforms existing methods, achieving about 90% editing efficacy when editing 10k facts. Codes and datasets are publicly available at https; //github. www.thuu.cooooaco com ybdai 1.
INTRODUCTION
Large language models (LLMs) are increasingly employed as search engines and chatbots, as they excel retrieving knowledge to answer queries (Brown al., 2020 Touvron al. 2023b
Yang et al. 2024a Bi et al. 2024.. However, they are prone to spreading misinformation about
frequently updated topics due to outdated training data (Vykopal al. 2023 Huang et al.) 2025 Xu
et al. 2024). To address this issue. retraining or fine-tuning models for partial knowledge updates is
proposed (Achiam al. 2023 Team al. 2025). albeit with prohibitively expensive overhead. In
contrast, recent advances in locate-then-edit model editing (ME) techniques (Meng et al.) [2033] Fang
al., (024) enable massive editing of thousands of factual associations concurrently at minimal data and computational cost, thereby rendering real-time knowledge updates feasible.
Jeep Commander is created by Jeep
Honda
memory
Effectiveness
Current Methods
EAMET
Pro rel))) rell)
key
argmax, rel,,
X
Current Methods
Robustness
memory
[PREFIX] Sentence
key
{Sentences Jeep Commander)
EAMET (ours)
Figure 1: Illustration of current methods and our proposed EAMET in evaluating massive editing. Here, "[PREFIX] Sentence" and "(Sentence 8; = Jeep Commander)" denote the scenarios where the edited knowledge preceded by prefixes and where multiple facts share the same subject, respectively.
Despite the success of existing massive ME techniques, observe that their effectiveness is often overestimated due to overly loose evaluation metrics. In particular, most prior works assess editing
"Corresoonding authors.

---

Published as a conference paper at ICLR 2026
quality by checking whether the model is more likely generate the following tokens as the target object than the original one, whereas neglecting to evaluate whether the model's output is consistens with the target object (Meng et al. 2023 Fang al. 2024). Therefore, we advocate a "practical metric", which measures the proportion of cases in which the edited model retrieves the target object and explicitly generates related output. This metric provides more accurate reflection of real- world usage, as will be shown in our evaluation setting (see Section 6). Under such evaluation criteria, existing methods fail maintain their performance.
Moreover, existing methods exhibit limited robustness in realistic settings. We highlight two represen- tative scenarios: (i) their performance substantially degrades when edited knowledge is preceded by prefixes (Li et al. (2024), a common phenomenon in practical question question-answering tasks (Pramanick et al. 2024, Romero et al., 2024); and (ii) they fail to preserve accuracy when editing multiple facts associated with the same subject, where performance drops markedly. Such lack of robustness in massive editing scenarios undermines their applicability to real-world use cases.
To analyze the limitations of existing methods, we first identify "embbedding misalignment"" which reflects the structural inconsistency between key and residual embedding spaces, as a primary factor underlying the decline in both effectiveness and robustness during massive editing. Such misalignment leads to information loss for individual knowledge updates. In particular, when parameters are updated jointly from a batch of edited knowledge items, they fail to accurately reconstruct individual factual association. This information loss becomes more severe as the number of edited items increases.
To achieve effective and robust massive editing under practical settings, thus propose EAMET (Embedding Aligmment Model Editing in Transformers), which outperforms existing approaches under stricter evaluation criteria and exhibits strong robustness in two described scenarios. EAMET addresses embedding misalignment by progressively preserving optimized residual embeddings and aligning them with the key embedding space, ensuring consistency throughout the editing process.
In this paper, we conduct extensive experiments on six LLMs showing that EAMET consistently surpasses existing methods under rigorous settings across the CounterFact, ZsRE, and Wiki-recent datasets. EAMET maintains about 90% editing efficacy across all evaluated models and outperforms baselines by an average of 14% and 8%, with gains of up to 37% and 1 15% on CounterFact and ZsRE when editing 10k facts. Moreover, EAMET sustains high accuracy even when edited items are preceded by prefixes of up to 200 tokens or involve multiple facts associated with the same subject. This demonstrates EAMET's robustness in realistic and context-rich settings, including chatbots and long-context QA tasks.
2 RELATED WORK
Model Editing.
Existing ME techniques can be classified into auxiliary-based (Hartvigsen et al.,
2023 Mitchell et al. 20226 Zheng et al. 2023 et al. 2024, Mitchell et al., (202aa) and location- ME techniques preserve the original parameters, and introduce additional information to edit knowledge. SERAC (Mitchell
based methods (Meng et al. 2022 2023 Li et al. 2025 ). Auxiliary-
al. 2022b) requires extra memory store new edits and learn to reason over them to manipulate the model's 's output. Location- based methods directly modify model parameters to edit knowledge without requiring any additional information. These methods assume that factual associations are stored in the feed-forward networks (FFNs) of the LLMs (Geva et al. 2021 2022 Dai et al.) (2022). Building on these, ROME (Meng et al. 2022) first gains insights on the specific location of the knowledge through causal analysis. It proceeds to directly modify critical MLP layers to update factual associations. MEMIT (Meng et al. 2023) builds upon ROME to enable massive editing of thousands of facts concurrently. AlphaE:dit (Fang et al. (024) focuses on sequential editing, aiming to preserve both previously edited knowledge and the general capabilities of the LLM during successive edits.
Massive Editing, In practical applications, ME techniques may aim to update a model with hun- dreds or even thousands of facts simultaneously in order to keep up with the constantly evolving knowledge aJu et al. 2024 Gu et al. 2024). However, auxiliary- in scalability, typically supporting only a few edits at time (Mitchell et al. 2022b). In contrast, location-basee methods are more scalable for massive editing. MEMIT (Meng et al. 2023) scales to edit 10,000 facts concurrently, and PMET (Li et al. 2025) further improves performance by
methods are usually limited
2

---

Published as a conference paper at ICLR 2026
incorporating attention layers when updating the parameters of the FFNs. Despite their effectiveness and scalability, these methods have been shown to be fragile when handling prefixes or multiple facts with the same subject during evaluation, which is common scenario in real-world applications (Li et al. 2024; Yang et al. 2024b; Ma et al., 2024). Moreover, we observe that their performance in massive editing overestimated due to the loose metric. In this work, we propose EAMET, which achieves superior performance in massive editing under practical evaluation metrics, while also exhibiting greater robustness against long prefixes and multiple facts with the same subject.
PRELIMINARY: EDITING MEMORY IN LLMS
Previous works have shown that pre-trained LLM has memorized many factual associations (Petroni et al., 2019; Jiang et al.] 2020 Roberts et al.) 2020 Shin et al. 2020). These stored facts could be edited by modifying the MLP layers within FFN modules, based on the assumption that knowledge is stored in them in the form of key-value pairs (Geva et al. 2021 2022).
In Figure 2 the MLP layer W within FFN associates keys (((x)
Aft hidden state
h
lff key for subject
oW in ( with memories (z) for the fact x. Given the critical mediating role of ML. layers in storing facts, Meng (Meng et al. 2022) shows that it is sufficient to update W to edit stored facts. We then optimize W out (abbreviated
mi memorized value
attn
W maps to moj through minimizing:
wh.
Wous
m
| m/ mill"
Figure 2: Illustration of the model editing problem.
as W1) follows:
N Winy m3ll?)
W1 - arg min
(1)
Here, kf and ke denote the encoded subject representations for individual target and preserved fact i and j. respectively, while my and m² represent their corresponding memory vectors. We stack the keys and memories of totally N₂ target knowledge into matrices as K, = |K kg .... k%,]
and M, [m] m² m²x,1 Similarly, we construct K₂ and Mp for N, preserved facts. The objective in Equation | can then be optimized by solving the normal equations (Meng et al. (023):
(Wo A) [K, K,J [M, M,]
WoKp = Mp
(3)
where expand W, into Wo + A. Wo denotes the original (unedited) parameters that associate preserved keys with their memory representations. The final update to W out can be computed by
multiplying both sides of Equation ( by Kp Ke)" and subtracting Equation from Equa- tion Meng et al. 2
AC + K2K?) = RK?
where R = Mj WoK, denotes relations' residual with respect the original weights, which can also be written as [ri rt I... r.x. Since the pretraining data of the original model is not accessible, we approximate C, using set of randomly sampled inputs from public datasets:
C XF (K())
The scalar balances the influence between newly edited facts and preserved knowledge.
4 MOTIVATION
In this section, we investigate the root causes of the challenges associated with effective and robust massive editing, as illustrated in Figure [ In particular, analyze the decline in editing performance as the number of edited facts increases. Our theoretical and empirical results indicate that these issues arise from misalignment between key and residual embeddings. We further examine robustness in two representative scenarios: (i) edits preceded by long prefixes, and (ii) edits applied to multiple facts sharing the same subject.
3

---

Published as a conference paper at ICLR 2026
4.1 EMBEDDING MISALIGNMENT IN EFFECTIVE MASSIVE EDITING
Theoretical Analysis. We observe that by expanding K₁ and R in Equation 44. the update equation be reformulated as:
N₂ k
N
A
Cp
+
nk
T
(6)
where the update is determined by the aggregated residual and key embeddings across all edited facts. As the number of edits increases, solving Equation {6) is more likely to cause reconstruction loss for individual knowledge items due to the embedding misaligument between the residual and key embeddings. This eventually leads to degraded editing performance.
To formalize the concept of embedding misalignment, we define two key requirements for the desired
update A: (1) The update should preserve the existing knowledge, expressed ACp. 0. (2) The update should ensure lossless reconstruction for each individual fact, formulated as Åk; = Tú, where A is computed while considering all target facts. Incorporating (I), ideal A that meets (2) implies:
Ni
N T
Cp
+
Ak, = for 1,2,..., No
However, the validity of Equation (7 intuitively affected by the degree of misalignment between the residual and key embedding of different facts. We then define embedding misalignment:
Definition (Embedding Misalignment), Given knowledge items, each item associated with residual embedding yi and a key embedding k;. We define the embedding misalignment of item i as the structural similarity between the pairwise relations of its residual embedding and those of its key embedding. Formally, consider the distributions
p(i) = cos(ri, rj) li +
Pf =
(8)
where cos(..)) is the cosine similarity, The ith misalignment score is quantified by the KL divergence:
A(1) = KL (P!²) ((()) 1
We now formalize the connection between embedding misalignment and the editing performance of specific knowledge item i under massive editing. Specifically, quantify the degree to which Equation 7 is established by analyzing the reconstruction loss e = Ak; Ti for each knowledge item. This relationship is formalized in the following theorem:
Theorem 1. Let A be the closed-form solution satisfying A E, L reconstruction residual of item as e; Ak; rz. Then we can expand
E, rikkT, and define the
e, Σ
BOgS T4 By
KJ
Σ
keke
k.
(10)
and its norm is bounded by the misalignment between the neighborhood structures of ra and ky
|eil &G A(i + [irlll
||ex|l
(11)
This result demonstrates how embedding misalignment impacts the editing performance of individual knowledge items under massive editing. Specificallyy, stronger misalignment among knowledge items leads to increased individual reconstruction loss, ultimately reducing the overall effectiveness of massive editing. The complete proof provided in Appendix
Empirical Study. Motivated by the above analysis, we hypothesize that the failure of massive editing stems from misalignment between the embeddings of different knowledge items. To test this hypothesis, we edit 200, 500, and 1,000 facts from the CounterFact dataset (Meng et al. 2022) using MEMIT (Meng et al., 2023) on LLaMAA-7B (Touvron et al.) (023a) and Deepseek-
(Bi al.
2024). We then evaluate the editing accuracy of these items when no prefix is added to the edited query. Embedding misalignment quantified using the misalignment score defined in Equation
4

---

Published as a conference paper at ICLR 2026
As shown in Table 1 the overall edit- ing accuracy of both models decreases as more facts edited, accompanied by a clear increase in embedding mis- alignment. For example, LLaMA2- 7B, the accuracy drops from 98.5% to 86.8% as the number of edited facts grows from 200 to 1,000, while the misalignment score rises from 79 to 554. These results provide further evidence for our theorem that embedding misalignment leads to degraded editing performance.
Table
Editing performance with varying numbers of edited facts on LLaMA2-7B and Deepseck-7B.
Model
LJaMA2-7B
Deepseck-7B
No. of Edited Facts
200
500
1000
200
500
1000
Editing (fficiency(8))
98.5
90.0
86.8
99.5
98.6
97.8
A,1(
243
554
68
223
562
4.2 IMPACT OF EMBEDDING MISALIGNMENT ON EDITING ROBUSTNESS
We investigate how embedding misalignment affects robustness in massive editing along two dimen- sions: (i) long-prefix perturbations and (ii) simultaneous edits of samples sharing the same subject. Based on our theoretical and empirical analysis, we derive two corollaries to characterize these effects and validate them with controlled experiments.
Corollary 1. Long prefixes exacerbate embedding misalignment issues under massive editing. leading to degraded editing performance when edited facts are evaluated with descriptive prefixes.
Table 2: Impact of varying prefix lengths on editing performance.
Model
LLaMA2-7B
Deepseek-78 7B
Prefix Lens
5
10
50
50
Editing Acc. low A(t) Acc.
98.50%
84.15%
80.35%
77.40%
99.50%
90.35%
84.25%
85.10%
94.00%
91.00%
90.00%
94.00%
92.00%
91.00%
top Alt) Ace.
46.00%
46.00%
45.00%
55.00%
54.00%
47.00%
Empirical Verification of Corollary 1. We edit 200 CounterFact facts on LLaMA2-7B and Deepseek- 7B using MEMIT, and evaluate average editing accuracy with prefix lengths ranging from 5 to 50 tokens. To assess the impact of embedding misalignment, we also compare the 10 items with the highest and lowest misalignment scores under
conditions.
Table shows that editing performance degrades when edited facts are evaluated with prefixes. For I.LaMA-7B, accuracy falls from 98.5% to 84. 15% with 5-token prefix, and further to 77.40% with 50-token prefix. A similar trend is observed for Deepseek-7B, where editing accuracy drops by around 15% under 50-token prefixes. The robustness to prefix perturbations varies markedly between items prone to embedding misalignmen and those that are not. Consistent with Corollary 1, items with lower misaligmment scores maintain averrge accuracy, whereas highly misaligned items suffer sharp decline, with accuracy dropping to average of 48%.
Corollary 2. Massive editing suffers from degraded performance when multiple samples with the same subject edited simultaneously. In this case, the reconstruction weight on the target Bi decreases while cross-weige weishts Bij increase, eventually leading to reconstruction failure of ri-
Figure 3: Impact of editing same-subject samples. Shaded region indicates shared items.
(a) Subject "Fimpen"
(b) Subject "USS Leedstown (APA-56)'
0.6
2000 edis
2000
edits 10000 edits
5000 edit
0.4
10000 waits
0.2
0.2
Bo
Index
Index
Empirical Verification of Corollary 2. Figure 3 shows the reconstruction coefficients for two exam- ple subjects under different numbers of edits. Although 3ts remains the dominant coefficient, its value decreases steadily the number of co-edited samples increases, while off- off-diagonal coefficients B3y grow accordingly. As a result, Ak; is no longer primarily aligned with r; but is instead reconstructed mixture of other Ty, making the recovery of the correct target representation increasingly difficult.
5

---

Published as a conference paper at ICLR 2026
This behavior is consistent with the misalignment measure A(t), as only Bi from the same subject as i take relatively large values, while subse-tubject weights remain negligible. Therefore, reconstruction is dominated by the same-subject neighborhood. When A(t) is small, same-subject embeddings are well aligned across both k-space and r-space. Thus, using other rj from the same subject to approximate y; introduces only limited error. However, when A(i) is large, misalignment within this neighborhood amplifies the effect of weight redistribution, causing the residual ||e:]] to grow and ultimately leading to degraded editing performance. We provide details in Appendix C
These findings underscore the strong connection between embedding misalignment and the effective- ness as well as robustness of massive editing. Motivated by this observation, the following section introduces our approach for aligning key and residual embeddings to enhance the overall performance of massive editing.
EMBEDDING ALIGNMENT MEMORY OPTIMIZATION
K
R
Lupdate
Jeep Commander created Jeep
*DDIT
Honda
p 66)
+
LKL
LASE
Pk
LKL
LMSE
(a) Key Embedding Preparation
(b) Aligning Memory Embeddings with Key Embeddings
(c) Optimizing the Target Memory
Figure 4: Method Overview of EAMET.
Motivated by these results, we propose EAMET, which optimizes memory embeddings to promote alignment with key embeddings across facts. This design enhances the model's ability to edit multiple facts concurrently under practical metrics, while also improving robustness against prefix perturbations and simultaneous edits of same-subject samples. We elaborate the details below.
Key Embedding Preparation (Figure 4 (a)). Before optimization, we extract the key embeddings corresponding to each knowledge item that scheduled for editing. For given knowledge item i, we calculate the cosine similarity between its key embedding k; and the key embeddings of all other
items. We then collect these similarity values into the set P( (i.j) cos(k,, ky) * i}.
Aligning Memory Embeddings with Key Embeddings (Figure 4 (b)). For knowledge items, separately optimize the target memory embeddings to update factual associations. During the iterative optimization process, we save every optimized residual embedding. When optimizing the target memory for the i-th knowledge item, compute the cosine similarity between r, and all
residual embeddings saved so far, and collect them as p!()) = {P!(.)) |j < <i). . To promote alignment between key and residual embeddings, compute the KL divergence Chen et al. (2020); He et al.)
(2020); Sun & Saenko (016)) between p,(0) and p((), where p(() = (P(1]) | < i} denotes
subset of ((d) corresponding earlier items:
Lkt(1)) = KL (P.') P1
(12)
Since KL divergence emphasizes distributional differences, we further strengthen the alignment by selecting the top M cosine similarities {p(()) from p( and computing the mean squared error (MSE) loss between the corresponding residual similarities (P. (P(()))
M
LMSE (i) = M
plis) - 1
(13)
1=1
Optimizing the Target Memory (Figure 4 (c)). Our goal in this step is compute the residual update vector r; for each factual association (s,, rely, 0,) such that the model reliably predicts the target object o, while preserving the alignment between the memory embeddings and the key

---

Published as a conference paper at ICLR 2026
embeddings. To make the optimization procedure explicit, describe each component of the objective in Equation (14). For each fact i, let h denote the hidden state at layer L produced by the templated prompt tp(s;, re44) Following prior work (Meng et al. 2022 2023), we augment this prompt with a of NFp randomly sampled prefixes {fg} forming inputs fs + tp(s;, rel:). These prefixes encourage the model to learn more generalizable memory representations. We write the forward pass of the model with the edited hidden state G(14+ar,) indicating that the hidden representation layer L is perturbed by the update vector r..
Given these definitions, we optimize by minimizing the following loss:
Npp
= arg min ri
log P GGMM+==))
[y @ tp(ss, rel:)] AKL. (x)( AMSE (i)
NFP
(14)
Here, the first term encourages the model to predict the correct target object o under all sampled prefixes. The losses (i) and LMSE(1) (i) ensure that the alignment between the memory embeddings and the key embeddings is preserved, with λKL and AMSE controlling their relative importance.
The full optimization procedure is detailed in Appendix B loss and MSE loss in Appendix F.5 As the optimization process is iterative, the editing order of knowledge items may influence the performance of EAMET. We further investigate the robustness of EAMET against different editing orders in T
We justify design of combining KL
EXPERIMENTS
In this section, we empirically focus on evaluating the following research questions (RQs). We first demonstrate the effectiveness of EAMET in massive by considering:
RQ1. Can EAMET generate more aligned embeddings for different knowledge items?
RQQ. How does EAMET perform on massive editing tasks compared with baselines for various LLMs? Can it excel under the practical metrie?
We then examine the robustness of EAMET in two representative scenarios:
RQ3. How does EAMET perform when evaluating edited facts with prefixes?
RQ4. How does EAMET perform when editing multiple facts of the same subject?
6.1 EXPERIMENTS SETUP
Models, Datasets, and Baselines. We conduct extensive experiments on various LLMs, including
LLaMA2-7B (Touvron et al. 2023a).. LLaMA2-13B (Touvron et al.) 2023a), Falcon-7B (Almazrouei al. 2023), Qwen-2.5 2.5-7B (Yang et al. 2024a), Deepseek- (Bi et al. (024). and LLaMA3- 8B (Touvron et al., 2023b). We provide additional evaluations on more LLMs in Appendix F33 consider a range of ME techniques as baselines: FT (Zhu et al. 2020), MEND (Mitchell et al.
2022ab, ROME (Meng et al.) (2022), MEMIT (Meng al. 2023), PMET (Li al.) 2025). and
ALPHAEDIT (Fang et al. 2024). We demonstrate their performance on CounterFact (Meng et al.
2022), 7sRE (Levy et al. 2017), and Wiki-recent (Zhang al., 2024). We provide full description
in Appendix D.I
Evaluation Metrics. Following previous work, evaluate the performance of ME techniques in terms of efficacy (Eff.), generalization (Gen.), specificity (Spe.). and fluency (Flu.) for CounterFact and ZsRE datasets. For Wiki-recent, additionally evaluate the portability (Zhang et al. 2024) (Por.) of edited models, which represents the ability to address downstream tasks with edited knowledge. We propose to evaluate the editing performance of ME techniques by requiring the edited models to strictly examine whether explicit target objects are retrieved, as demonstrated in Figure The editing efficacy is then defined as:
Eff. E.fo: arg max P rel,))]
(15)
When evaluating efficacy the CounterFact and Wiki-recent datasets, and generalization Counter- Fact, we prepend each prompt with 10 distinct 5-token prefixes. Full details of metries are provided in Appendix D.2 We also provide the implementation details of EAMET in Appendix D.3
7

---

Published as a conference paper at ICLR 2026
6.2 ALIGNMENT OF RETRIEVED EMBEDDINGS (RQ1)
Finding 1. EAMET Promotes More
Table 3: Misalignment score comparison between dif- ferent methods. Here. "CF" and "ZS" denote the Coun- terFact and ZsRE datasets, respectively.
Aligned Embeddings. We compute the summation of the misalignment score be- tween the residual and key emheddings for 10,000 facts edited by MEMIT, PMET, and EAMET under various LLMs. As shown in Table 3 the residual embeddings gen- erated by EAMET are more aligned with the key embeddings, while those produced by MEMIT and PMET are more likely to cause inconsistency in the key and residual embeddings space. This observation supports our hypothesis that EAMET encourages more aligned target memory embeddings.
EAMET
MEMIT
PMET
Model
CF
7.
CF
78
CF
7S
LLaMA2-7B
377
165
11506
22245
11475
11477
Qwen-7B
374
180
18498
23699
18471
18463
Decpseck-7B
520
161
12135
23241
12155
12046
Falcon-7B
385
181
8564
17589
8602
8590
6.3 PERFORMANCE OF MASSIVE EDITING (RQ2)
Table 4: Performance comparison of different editing methods on six LLMs over the Counterfact, Wiki-recent, and ZsRE benchmarks. We report the average value calculated over five evaluations.
Counterfact
Wiki-recent
ZsRE
Model
Method
EdI.
Gen.t
Spe.t
Flu.t
Por."
Loc.t
Flu.
Gen."
Spe.
FT
0.29
0.23
77.43
490.34
7.23
41.61
36.52
491.83
5.30
4,31
14.69
LLaMA2-7B
MEND
0.23
0.31
78.55
307.26
0.00
34.67
37.46
269.52
0.00
0.00
0.50
ROME
0.00
0.00
50.73
467.76
76.73
49.31
497.53
37.29
MEMIT
24.95
22.68
63.84
506.69
34.75
44.93
46.72
504.18
76.63
64.06
15.57
PMET
74.22
46.45
507.10
497.49
77.29
ALPIIADII
0.51
0.53
$1.14
501.63
0.07
35,34
37.48
527.83
44.26
35.83
12.65
EAMET
89.09
61.21
72.19
93.23
503.52
89.47
15,70
FT
16.18
14.15
56.07
527.56
21.17
51.40
$1.50
$15.87
14,30
13.00
39,28
MEND
0.01
0.06
70.73
282.92
0.00
42.55
44.37
272.90
0.00
0.00
0.09
Qwenn2.517
ROME
0.00
0.00
49.83
$23.45
16.28
46.52
46.61
502.37
4.10
3.43
1.30
MEMIT
90.06
63.86
70.53
529.27
94.88
56.97
61.23
510
12
42.96
31.57
PMET
65.71
$2.84
63. 14
518.92
82.39
58.38
57.59
$11.62
53.58
46.59
36.50
ALPHAADIT
83.15
55.70
94.16
57.17
$10.32
34.98
EAMET
90.49
64.37
72.18
536.67
95.61
57.46
60.28
509.06
91.03
$4.80
41,20
LLaMA2-13H 13B
FT
1.23
0.07
68.57
484.56
13,90
36.80
40.09
497.21
5.95
5.10
15.16
ROME
4.05
1.52
50.44
525. 12
11.06
38, 14
39.09
447.42
5.52
5.06
2.25
MEMIT
47.98
34.75
71.61
517.63
94.76
$1.38
50.40
507.88 88
69.15
$1.58
15.53
PMET
78.60
38,76
81.15
526.82
88. 66.66
49.60
47.
501 61
53.27
35.73
15,76
ALPHAADIT
3.03
1.9
54.97
421.97
93.68
508.82
80.27
63.66
15.32
EAMET
92.85
60.08
77.51
530.78
95.88
52.08
53.43
504.06
87.09
74.58
15.90
FT
14.70
13.54
56.34
167.18
23.94
50.46
49.69
351.18
13.64
12.68
32,28
Falcon- 7B
ROME
12.85
12.56
51.48
353.38
74.57
52,10
53.64
$10.92
8.39
7.3
10.29
MEMIT
89.21
60.85
77 .6.5
519.92
96.04
55,23
56.91
497.35
82.93
68.93
33.64
PMET
77.61
57.03
70.48
517.09
58.03
54,40
$4.449
500.87
69.73
60.69
35.34
ALPHAEDIT
87.62
58.32
72.
500.35
96.22
55.47
58. 02
493. 56
53.78
40.83
22.60
EAMET
92.37
63.91
78,94
528.98
96.94
57.08
58.58
507.56
92,38
81.15
36.71
FT
2.61
2.49
$1.43
519.35
18.85
48.90
52.78
500.86
15.00
12.28
39.14
Despsrek-78
ROME
0.26
0.30
49.82
514.72
0.55
43.17
46.02
406.64
0.81
0.78
0.75
MEMIT
62.11
42.01
78.04
512.16
33.65
52.28
49.05
499 .49
57.10
42.58
39.12
PMET
74.52
43.49
79.01
514.58
86.75
57.85
59.93
500.50
76.97
69.22
38.47
ALPHAEDIT
22.51
14.00
59.92
479.52
18.53
48.33
48.74
483.38
73.41
57.09
34.87
EAMET
89.74
59.98
77.73
513.93
97.15
56.43
60.45
501.09
87.27
70.02
39.87
FT
2.68
1.30
56.16
434.67
16.05
47.51
48.84
490.71
11.75
10.48
40.53
LLaMA3-8B
ROME
51.01
33.32
64.37
491.98
82.19
54,94
57.74
$18.89
7.40
6.82
27.79
MEMIT
93.76
61.98
77.66
77.69
526.47
92.63
55.60
58.75
$27.29
78.40
71.76
39.21
PMET
77.71
49.41
71.43
510.82
75.81
56.89
58.26
513.84
68.52
62.72
39.35
ALPHAEDIT
58.97
33.02
85.16
537.91
65.36
51.44
53.55
$16.53
64.01
57.01
40.82
EAMET
93.87
63.74
79.07
533.30
94.36
57.88
59.48
528.23
85.68
81.34
42.39
We demonstrate the effectiveness of EAMET in massive editing tasks by comparing it with bascline methods across six popular LLMs. Specifically, we simultancously edit 10,000 factual associations sampled from the CounterFact and ZsRE datasets. For the Wiki-recent dataset, modify all 1 1,266 knowledge items. As shown in Table[I our key findings are as follows:
Finding 2. EAMET Consistently Achieves Superior Editing Performance Across All Datasets and Model Architectures. Across all evaluated datasets, EAMET demonstrates the highest levels of editing efficacy and generalization. On the CounterFact dataset, it consistently outperforms
8

---

Published as a conference paper at ICLR 2026
other methods, particularly on base models such as LLaMA2-7B, LLaMA2-13B, and Deepseek- 7B. For example, EAMET achieves 89.09% efficacy and 61.21% generalization on LLaMA2-7B. outperforming the second-best method (PMET) by 15% on both metrics. The gap widens further compared to MEMIT, with improvements of 65% in efficacy and 39% in generalization. Even on more advanced models such as Qwen2.5-7B, Falcon-7B, and LLAMA3-8B, EAMET consistently surpasses all baselines, Furthermore, its advantage becomes more pronounced at larger editing scales. As shown in Table []] when editing 15,000 knowledge items on Qwen2.5-7B, EAMET achieves 83.66% efficacy, demonstrating 10% improvement over MEMIT. We additionally report the superior performance of EAMET across diverse semantic scenarios in Appendix
Finding 3. EAMET Preserves the General Abilities of the Edited models. addition to achieving state-of-the- art editing performance, EAMET does not impair the base model's fluency or reasoning abilities. Across all datasets, EAMET consistently attains among the highest specificity and fluency scores. Notably, on the Wiki- recent dataset, EAMET achieves the best portabilityy performance on most base models, indicating that the edited models retain their ability reason about down- stream knowledge related to the edited facts. We also evaluate the general abilities of edited models on GLUE (Wang et al. 2018) and find that EAMET yields minimal deviation from pre-edit performance (Appendix F22)
Table 5: Performance comparison of dif- ferent editing methods on Qwen2.5-7B. Falcon-7B 7B, and LLaMA3-8B with 15,000 edits from the CounterFact benchmark.
Counterfact (15000)
Model
Method
E00."
Gen.
Spe.
Flu.1
Qwen2.5-7B
MEMIT
77.46
54.34
66.23
514.81
FAMET
83,66
55.31
69.49
$28.28
Falcon-7B
MEMIT
84.60
56.13
75.51
513.82
FAMET
89.55
61.00
68.44
516.91
LLaMA3-8B
MEMIT
72.65
FAMET
91.22
62.24
73.43
531.76
Table 6: Impact of editing sequence on EAMET's performance on Counterfact and ZsRE datasets.
Counterfact
ZsRE
Method
Eff.1
Gen.t
Spe.t
Flu.t
Eff.1
Gen.1
Spe.1
EAMET (original sequence)
89.09
61.21
72.19
519.06
89.47
81.34
15.70
random shuffle (sced=0)
88.21
60.79
71.84
519.21
87.63
77.42
15.56
random shufflc (scod=1)
89.11
60.78
72.03
518.84
86.99
76.08
15.58
random shuffle (scod=2)
88.91
59,38
72.34
518.23
87.56
77.47
15.59
As EAMET preserves previously optimized residual embeddings when updating new knowledge items, the editing sequence could potentially affect its performance. To assess this, we examine EAMET's robustness under different editing orders on the Counterfact and ZsRE datasets. In Counterfact, all knowledge items have distinct subjects, whereas in ZsRE some items share the same subject and are adjacent in the original order. We therefore randomly shuffle the order of 10,000 items three times and report the average performance, alongside the original sequence as reference.
Finding 4. EAMET is Robust to Editing Sequence. As shown in Table 6 EAMET's performance remains stable across editing orders. On Counterfact, random shuffles produce only negligible variations in efficacy, generalization, and specificity. On ZsRE, editing efficacy shows a slight decline of about 2%, likely due to the neighborhood structure of items sharing the same subject in the original sequence. Overall, these results suggest that EAMET is largely insensitive to editing order, demonstrating strong robustness to sequence variations.
6.4
ROBUSTNESS AGAINST LONG PREFIXES (RQ3)
Editing Accuracy (%)
(a) LLaMA2-78
(b) Deepseek-78
(c) Qwen-78
0003
MEMIT
ALPHAEDIT
HEMIT
ALPHAEDIT
PMET
EAMET
PME1
EAMET
MEMI
ALPHAEDIT EAMET
150
126
Prefix Length
Figure 5: Editing performance of different methods across varying prefix lengths.

---

Published as a conference paper at ICLR 2026
We evaluate the robustness of editing methods when edited facts preceded by varying numbers of tokens. Specifically, we modify 200 facts from the CounterFact dataset in LLaMA2-7B. Deepseek-7B and Qwen2.5-7B. During evaluation, prepend prefixes of 5, 50, 100, and 200 tokens to them.
Finding EAMET Remains Effecttive When Edits Are Preceded by Long Prefixes. In Figure 5 EAMET achieves the highest editing efficacy across all models, with most 7% drop 200-token prefixes. contrast, MEMIT suffers much larger decline, from 84.75% to 66.50% on LLaMA2-7B and from 94.2% to 82.25% on DeepSeek-7B. Notably, all methods demonstrate strong robustness on Qwen2.5-7B, consistent with our earlier observation that Qwen2.5-7B is more suitable for robust batch editing. Nevertheless, EAMET exhibits the smallest efficacy drop (only 1.9%) when the prefix increases to 200 tokens, which is half that of the second-best method (MEMIT).
6.5 ROBUSTNESS UNDER MULTIPLE EDITS OF THE SAME SUBJECT (RQ4)
(a) LLaMA2-78
(b) Falcon-
(c] Qwen-78
Rewrite Accuracy
ALPHAEDIT EAMET
PHAEDI
PHET
PMET
EAMET
PMET
EAMET
Sample Number
Figure 6: Editing performance of different methods across varying numbers of facts per subject.
We evaluate the robustness of editing methods when multiple facts concerning the same subject are edited simultancously. Specifically, simultaneously edit 10,000 facts from ZsRE dataset, and only evaluate samples whose subject is associated with multiple facts. We group subjects according to the number of associated samples and examine how rewrite accuracy varies with this number. Experiments are conducted on LLaMA2-7B, DeepSeek-7B, and Qwen2.5-7B.
Finding 6. EAMET Remains Effective When Multiple Facts of the Same Subject Are Edited Simultancously. Figure 6 shows that EAMET consistently achieves the highest editing efficacy across nearly all settings. Its performance remains stable when editing multiple samples associated with the same subject. In contrast, other methods exhibit clear decline in efficacy as the number of facts per subject increases, which ultimately results in degraded performance the overall massive editing task.
CONCLUSION
In this paper, we propose EAMET, a novel model editing method that enables stronger and more robust massive editing across various models and datasets. We first identify that the failures of existing methods in both effectiveness and robustness massive editing stem from misalignment between the space of key and residual embeddings. EAMET addresses this issue by progressively aligning the key and residual embedding space when optimizing target memory for each fact. The aligned embeddings increase both the capacity and robustness of massive editing. Extensive experiments on multiple base LLMs, including LLaMA2, LLaMA3, Deepseek, and Qwen, demonstrate that EAMET significantly outperforms existing methods editing performance and robustness.
ACKNOWLEDGMENTS
The HKUST authors were supported in part by RGC GRF grant under the contract 16214723, research fund provided by CMHK and ZTE, and a HKUST Bridge The Gap fund BGF.001.200 We grateful to the amonymous reviewers for their valuable comments. We thank HKUST Fok Ying Tung Research Institute and National Supercomputing Center in Guangzhou Nansha Sub-center for computational resources.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work does not raise any specific ethical concerns. Its primary goal is to advance research on model editing and to offer new perspective on this problem.
REPRODUCIBILITY STATEMENT
All experiments in this paper reproducible. The code and datasets publicty available the GitHub repository: /ttt::://githbb ctw.cbda EAMET massive-edite
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint urXiv:2303.0077 2023.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojo- caru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311.1688 2023.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, al. Deepseek I1m: Scaling open-source language models with longternism. arXiv preprint M...xx22411009955 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PmLR, 2020.
Thomas M Cover and Joy A Thomas. Elements of Information Theory. Wiley-Interscience, 1999.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Dublin, Ireland, May 2022. Association for Computational
1: Long Papers), pp. Linguistics. doi: 10. 12211866/1112222.ddlll aow.:ooo22...l-loo555555.3. URL httpp://aclaatthooorryy. /rg/
2112.acl-loma 581/
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. arXiv preprint arXiv:2410.02355, 2024.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10228865/v1/222111 .emnl- connlpmmin..44 main.446. URLhttpa //aclanthology. wtg.ologo..ogg..../. wwt.:211.mmmmmmmmmmm..m 4444
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in
Natural Language Processing, pp. 30-45, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v 12000///./..2222mmmmmm...a..
URL https
//aclanthol ogy. ogy.org/2022 emn aalppmmnn...m
11

---

Published as a conference paper at ICLR 2026
Hengrui Gu, Kaixiong Zhou, Xiaotian Han, Ninghao Liu, Ruobing Wang. and Xin Wang. PokeMQA: Programmable knowledge editing for multi-hop question answering. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8069-8088 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 11153/8//22244.cl--oe
URL httpps://cllanthooogg.oooo 2004
long. 438/
Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzych Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. In A. Oh, T. Naumann, A. Globerson, Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 47934-47959, Curran Associates, Inc., 2023.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Roger A Horn and Charles R Johnson. Matrix Analysis. Cambridge University Press, 1985.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1-55, 2025.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
Tianjie Ju, Yijin Chen, Xinwei Yuan, Zhuosheng Zhang, Wei Du, Yubin Zheng, and Gongshen Liu. Investigating multi-hop factual shorteuts in knowledge editing of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8987- 9001, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/ v1/2024.acl-long 486.. utL / pteetps://acllantolooggo
org 2024 acl-lallllnnl 486/
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. arXiv preprint arXiv:1706.044 04100004.11..... 2017.
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang. Jun Ma, and Jie Yu. Pmet: precise model editing in transformer. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence,
AAAT URL https://doo
Press. 2025. ISBN 978- 178-1-97773--88--9 hxg/10.16094 /wt.0///aaai.oo www.17.1111108 29818
doi: w3.69//aai.v88811..222..
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint (rXiv:2309.00466) 2023.
Zhoubo Li, Ningyu Zhang, Yunzhi Yao, Mengru Wang, Xi Chen, and Huajun Chen. Unveiling the pitfalls of knowledge editing for large language models. In The Twelffth International Confer- Learning Representations, 2024. https:/:: http:///ppeeeeeieeeeeeeee net forum?id=
ENktD3ible
Xinbei Ma, Tianjie Ju, Jiyang Qiu, Zhuosheng Zhang, Hai Zhao, Lifeng Liu, and Yulong Wang. On the robustness of editing large language models, 2024. URL
abs/
24C2. 05827
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associ- ations in GPT. Advances Neural Information Processing Systems, 36, 2022. arXiv:2202.05262.
Kevin Meng, Arna Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass editing memory in a transformer. The Eleventh International Conference on Learning Representations (ICLR), 2023.
12

---

Published as a conference paper at ICLR 2026
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022a. URL
https: http:::://cpeneevieee net/ wow.mmidd-Dezxeeeeee.o.o
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory- based model editing at scale. International Conference on Machine Learning, pp. 15817-15831.
PMLR, 2022b.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv: 1909.00 01066, 2019.
Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan. Spiqa: A dataset for multimodal question answering scientific papers. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 118807-11888 188077 Curran Associates, Inc., 2024.
http:://rooceddnns..eenzit wwwpeeurrp..cccpppppoope cc/paper fileefilelleee walles/papo pttppp/ppp///1////// 310322218991553380003333----- -atase- and Benchmarks. Track pdf
URL
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? arXiv preprint wrXiv:2002.08910, 2020.
David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwa Adelani, David Le Meur, Emilio Villa-Cueva, Fajri Koto, Fauzan Farooqui, Frederico Belcavello, Ganzorig Batnasan, Gisela Vallejo, Grainne Caulfield, Guido Ivetta, Haiyue Song, Henok Biadglign Ademtew, Hernán Maina, Holy Lovenia, Israel Abebe Azime, Jan Christian Blaise Cruz, Jay Gala, Jiahui Geng, Jesus- Ortiz-Barajas, Jinheon Baek, Jocelyn Dunstan, Laura Alonso Alemany, Kumaranage Ravindu Yasas Nagasinghe, Luciana Benotti, Luis Fernando D' Haro, Marcelo Viridiano, Marcos Estecha-Garitagoinn Maria Camila Buitrago Cabrera, Mario Rodríguez-Cantelar, Mélanie Jouitteau, Mihail Mihaylov, Naome Etori, Mohamed Fazli Mohamed Imam, Muhammad Farid Adilazuarda, Munkhjargal Gochoo, Muunkh-Erdena Otgonbold, Olivier Niyomugisha, Paula Mónica Silva, Pranjal Chitale, Raj Dabre, Rendi Chevi, Ruochen Zhang, Ryandito Diandaru, Samuel Cahyawijaya, Santiago Góngora, Soyeong Jeong, Sukannya Purkayastha, Tatsuki Kuribayashi, Teresa Clifford, Thanmay Jayakumar, Tiago Timponi Torrent, Toqeer Ehsan, Vladimir Araujo, Yova Kementchedjhiivaa, Zara Burzo, Zheng Wei Lim, Zheng Xin Yong, Oana Ignat, Joan Nwatu, Rada Mihalcea, Thamar Solorio, and Alham Fikri Aji. Cvqa: Culurally-diverse multilingual visual question answering benchmark. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 11479- 11505 Curran Associates, Inc., 2024. URL fteps::/prooceedigg..eeeeee wwtpeeewwip...ppprrrreeeeeeeeeee
25662222aa1330166777555333333333 cc wwwpsc/ppppppreeeeeeeeeeeeeee les httpa//rrereraeeeeeaaee///////// Track.pdz
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.1 15980, 2020.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pp. 443-450. Springer, 2016.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
13

---

Published as a conference paper at ICLR 2026
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Pumit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao. Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023a. URL http::ttps//tt /tmpp:://arxar /atpii/aaxivv.voa wwt.
/tatt::://axii.oog///a/////// 2307 09288
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation 2023b.
and fine-tuned chat models. arXiv preprint arXiv:2307.09288
Ivan Vykopal, Matúš Pikuliak, Ivan Srba, Robert Moro, Dominik Macko, and Maria Bielikova. Disinformation capabilittes of large language models. arXiv preprint arXiv:2311 1088888, 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguisties. doi: 10. 11865//1//188544/4 URL
https: antho .oy.og.......................... wwwg////. 5446/
Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024a.
Wanli Yang, Fei Sun, Xinyu Ma, Xun Liu, Dawei Yin, and Xueqi Cheng. The butterfly effect of model editing: Few edits can trigger large language models collapse. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 1419-5437, Bangkok, Thailand, August 2024b. Association for Computational acl.322. Linguisties. doi: 10.18653/ a.l... URL https: http://ss///aaaaaaoooooooooooo F024.findings-cc..333
Lang Yu, Qin Chen, Jie Zhou, and Liang He. Melo: Enhancing model editing with neuron-indexed dynamic lora. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19449-19457, 2024.
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang. Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models. arXiv preprint arXiv:2401.0
2024.
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4862-4876, Singapore, December 2023, Association for Computational Linguis
tics. doi: 10. 18.18653/v wwt.::3...mmmmmmmmm2200 URL hitps: emm emnlp-maing 2967]
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.0036: 2020.
14

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
We used LLMs solely to assist in drafting and polishing the writing this paper, without any other purposes.
B PROOF THEOREM
Notation Setup. Let K [Ki "aig...,kyl RaxN and R , rNl
R and define
N N Σ ry RK
M = Σ k.k KK
(16)
Assume that M is rnseudo-)inverttng and define the closed-form solution A RK (KK") )+, so that A Σ; ki,k = >; rikk
NM+
Derivation of the Column Expansion Ake Eg Bigrz- By definition,
N
M+ = RKTM
(17)
j=1
Applying A to column k; gives
N Ak; Σ
ry M
(18)
Setting 3i = kj M we immediately obtain
Ak
Bistz-
(19)
This formula provides explicit linear combination of residual embeddings ry that reconstructs Ak, with coefficients Bij determined by the key embeddings and the pseudo-inverse of M.
Reconstruction Residual and Neighborhood Decomposition. For each knowledge item define the reconstruction residual e, == Ak; Ti. Suppose that r; can be approximately reconstructed from its neighbors with nonnegative weights ęy for j # i.e.
O5 + Eis
Uij 2 0.
Σ
43 = 1,
(20)
where e denotes the residual error. Substituting this decomposition into e; gives
e/ = (Bs - -g) + Bur - Eg.
(21)
J79
Bounding the Reconstruction Residual. Taking norms and applying the triangle inequality yields
||ea]|
Σ Bi 4:9 ||rll| + |Bal ||xll| ju
||=.|]-
(22)
To relate the first term to embedding alignment, we construct a probability vector p: from the positive parts of the coefficients By (for # i):
Sij == max ( Big. 0),
S = Σ
8ijz
Pij = 8ij
(23)
S
jt
Defining C. := Ejtt ||ry1| one show that
Σ Big 9b ||r
C, TV(,,)))
(24)
15

---

Published as a conference paper at ICLR 2026
un negligible contributions from negative Big that can be absorbed into C₁. Here, TV qi) is the total variation (TV) distance between two discrete distributions p; and q;
TV 9) = 1 Σ lp, Zl.
(25)
Finally, applying Pinsker's inequality (p 9)) $ v (L(gellp:) Cover & Thomas 1999 gives
||sl || S C,
V (L(g:llp)) + [3x] ||:ll| + ||ea||-
(26)
Identifying 9, P and p; pl with the cemeel-oormalzzed neighborhood distributions from Definition 1 yields the smbedding-alignment bound stated in the main text:
|c/l & siv 1/2 A(1) + 18al Brdl + |eell-
(27)
M is singular, replace M with the Penoree pseudoinverse. The contributions from negative 3i or scaling factors usually be absorbed into C.. In the ideal case of perfect neighborhood alignment A(t) = 0, negligible self-weight Bu = 0, and vanishing residual Es = 0, we recover O.
C DETAILED ANALYSIS OF COROLLARY 2
We provide detailed theoretical analysis that develops Corollary 2.
Residual Decomposition. Recall that the reconstruction residual can be written
N
c; Ake Ti Σ Bigry
To:
Big = kjj
(E keke
kig
(28)
Partition the index set into the subject cluster S (samples sharing the subject with i) and the remainder T. Then
Ak: Bars
Σ Begry + Bogr
(29)
jES.jti
Empirically and theoretically, only coefficients 8ig for j S become significant, while cross-subject coefficients remain negligible since embeddings from different subjects are nearly orthogonal in key space and thus contribute little the reconstruction. Hence reconstruction is dominated by the same-subject neighborhood.
Effect of Adding Same-Subject Samples. Let K Eg ke denote the Gram matrix. Suppose add one additional key ky (with j E S # i). By the Woodbury identity (Hom & Johnson, 1985).
K- K K-
K + kg
K
(30)
+ kJ
K-
Consequently, the updated
becomes
K + kJ K-1k;
6.Eh = k (K + kkkk 3
(31)
Thus B1 monotonically decreases as more same-subject vectors are included. The lost weight is redistributed into off- off-diagonal terms 8.y. consistent with our empirical observation in Figure
Connection to Alignment. Using the decomposition in equation 29 the residual can be expressed as
e4
Σ Bis rT)
Σ
BisTz-
(32)
JES
JET
16

---

Published as a conference paper at ICLR 2026
Applying the triangle inequality yields
||all| $
Σ jES
18yl lrr -ryll + Σ JET
[8g ||r/l.
(33)
The first term depends on the dispersion of responses within the same subject. This dispersion is controlled by the alignment measure A(t): when A(i) is small, the responses {r, 1 E S} are tightly clustered around r;, even a redistribution of weight from Bü to other 3ig produces only minor error. Conversely, when A(i) is large, intra-subject responses differ substantially, and the redistributed weights amplify reconstruction error.
we conclude that co-editing additional same-subject samples (i) monotonically decreases 8ü, (ii) redistributes weight into off-diagonal Bij. and (iii) yields residuals bounded by the intra-subject alignment A(i). Therefore, massive editing performance erucially depends the degree of alignment within the subject cluster.
Combining equation 31 and equation
33
D DETAILED EXPERIMENT SETUP
In the following, we provide detailed experimental configurations, including the description of the datasets, introduction of baselines, explanation of evaluation metries, and implementation details.
D.I DATASETS AND BASELINES
We evaluate the performance of model editing techniques using the following datasets:
CounterFact (Meng et al. 2022) is benchmark for evaluating factual knowledge localization and editing in I.MMs. contains 21,91 1,917 entries that describe the named entities along with their counterfactual variations. Model editing techniques could be evaluated in terms of editing efficacy, generalization, and locality. The benchmark also contains generation prompts to test the model's generation ability after editing.
ZsRE (Levy al. 2017) question-answering (QA) benchmark designed evaluate zero-shot relation extraction capabilities of language models. Entries in the benchmark consist of subject entity along with an answer as the editing target. The benchmark also includes paraphrased questions for testing generalization ability and irrelevant questions for evaluating the locality of editing techniques.
Wiki-recent (Zhang et al. 2024) contains 1,266 entries of triplets that have been added into WIKIDATA after July 2022. The benchmark enables insertion for models that trained prior to the introduction of these facts. This simulates the cases of editing outdated models with newly introduced facts. Model editing techniques evaluated terms of editing efficacy, portability, and locality. Here, portability emphasizes whether the edited model could reason about the downstream effects of facts when they are inserted into the model.
We proceed to introduce baseline methods evaluated the paper. For all baseline methods, we use the official implementation provided by the authors.
MEND Mitchell et al., 2022a) requires extra parameters for efficiently editing pretrained LLMs. It introduces set of small auxiliary networks that transform standard fine-tuning gradients into low-rank updates, enabling fast and localized edits without retraining the entire model. This approach offers scalable solution for post-hoc model editing, avoiding the overfitting issue of traditional fine-tuning methods.
ROME (Meng et al.) (022) performs factual knowledge editing by directly modifying the feed- forward weights in specific layers of LLMs. It first identifies that factual knowledge primarily stored in mid-layer feed-forward modules, thereby demonstrating the feasibility of editing model parameters to update internal knowledge. The method then updates these weights to encode specific factual associations. ROME achieves precise insertion of new facts with minimal interference to unrelated knowledge. When evaluating using ROME, we edit all facts sequentially with batch size 1, as it does support batch editing.
17

---

Published as a conference paper at ICLR 2026
MEMIT (Meng al. 2023) is designed to efficiently update LLMs with thousands of factual associations simultancously. Building upon ROME, MEMIT employs a least-squares optimization multiple vey-v memory components, ensuring high specificity and minimal interference with unrelated knowledge. further distributes the updates across multiple layers, which helps reduce the impact the model's general capabilities.
PMET (Li et al. 2025) is method designed to enhance the precision of knowledge updates in large language models. Unlike prior approaches that treat transformer layer (TL) hidden states as direct imputs of the feed-forward network (FFN), PMET recognizes that these hidden states also encompass information from multi-head self-attention (MHSA) and residual connections. PMET proceeds simultaneously optimize MHSA and FFN hidden states and use the optimized TC hidden states of FFN to precisely update FFN weights. This approach enables more accurate and efficient model editing, preserving the integrity of the model's existing knowledge while incorporating new information.
ALPHAEDIT (Fang et al., 2024) preserves knowledge in LLMs during sequential updates by projecting updates onto the null space of the preserved knowledge. This could ensure that new modifications do not interfere with previously stored information. This approach maintains the integrity of the model's s existing knowledge while enabling precise edits.
D.2 METRICS
We now introduce the metrics used for CounterFact, Wiki-recent and ZsRE respectively.
D.2.1 COUNTERFACT METRICS
Given an LLM fe. a knowledge fact tuple (subject 8,, relation rz), a target output o, and the original output of, define the following metries:
Editing Efficacy: Unlike previous works that evaluate the portion of cases where of is more probable than of. we directly compute the average top-1 accuracy of edited samples.
E, = arg max Fo (o (8),r:)))
(34)
Generalization: Average top-1 accuracy of the edited model on rephrased statements N((s;, r:)) of the original knowledge fact. Rephrased statements share the same semantic meaning with the original statements.
Edjo: = arg max (o N((s1,):)))
(35)
The portion of cases where of is more probable than o, with neighboring statements O((s,, rt)). Neighboring statements are constructed using prompts which share distinct but semantically related subjects with the original knowledge fact.
Specificity:
E, O((s,, r.))) > p O((s,.r)))
(36)
Fluency: Fluency score measures the quality of the generated text. It scores low if the generated text contains excessive repetition.
92 log, (2)
Σ ga(h) 9a
(37)
where g2(k) and ga(k) are the probabilities of bigram and trigram k respectively.
D.2.2 WIKI-RECENT METRICS
Given a LLM fe. knowledge fact tuple (st, r:), target output o, and the original output of, we define the following metrics:
Editing Efficacy: Average top-1 accuracy of edited samples.
E,[o; = arg max P Po (o (8,, r())
(38)
18

---

Published as a conference paper at ICLR 2026
Portability: Average top-1 accuracy of the edited model portability prompts P((s;, of the original knowledge fact. Portability prompts contain three parts: alias prompts, compositionality and reasoning prompts, and logical generation prompts. Specifically, alias prompts are constructed by replacing the subject 8; with an alias or synonym. Compositionality and reasoning prompts require the post-edit model conduct reasoning about the changed fact. Logical generation prompts are changes that are semantically related to the modified fact and expected to change by the edit.
E, arg max 5 (o P((s;,
(39)
Locality: Average top-1 accuracy of the edited model on neighboring prompts O((s; r:)) of the original knowledge fact.
E. arg max fo (o O((s,, r:)))]
(40)
Fluency score measures the quality of the generated text. It scores low if the generated text contains excessive repetition.
Fluency:
Σ (2())) log, (2()) + Σ (a()) loga 9s(k)
(41)
3
where 92(k) and gs(k) are the probabilities of bigram and trigram ls respectively.
D.2.3 ZsRE METRICS
Given LLM fe. knowledge fact tuple (s;, rz), target output o; and the original output of, define the following metrics:
Editing Efficacy: Average top-1 accuracy of edited samples.
[, arg max P (o (st r:))]
(42)
Generalization: Average top-1 accuracy of the edited model on generalization prompts N((s;, r()) of the original knowledge fact.
Egio; = arg max P (o (N((s.)))))
(43)
Locality: Average top-1 accuracy of the edited model neighboring prompts (( r:)) of the original knowledge fact.
E [o: arg max P (o (((ss))))))
(44)
D.2.4 EXAMPLES OF EVALUATION PROMPTS WITH PREFIXES
We provide examples of evaluation prompts with prefixes the following. For the evaluated prompt "The mother tongue of Danielle Darrieux is French", corresponding evaluated prompts with ten distinct 5-token prefixes are:
Example Prompts with Prefixes
The doctor's 's office was too. The mother tongue of Danielle Darrieux is French.
Therefore, the average speed of. The mother tongue of Danielle Darrieux is French.
You can find many examples of. The mother tongue Danielle Darrieux is French.
However, the government's new. The mother tongue of Danielle Darrieux French.
And so, the night. The mother tongue of Danielle Darrieux is French.
While the world of sports and. The mother tongue of Danielle Darrieux is French.
To make a cake, you. The mother tongue of Danielle Darrieux is French.
Nevertheless, the overall sentiment of. The mother tongue of Danielle Darrieux is French.
Never tried it before. The mother tongue of Danielle Darrieux French.
He realized that the people of. The mother tongue of Danielle Darrieux is French.
19

---

Published as a conference paper at ICLR 2026
For model evaluation, generate prefixes by prompting the unedited model with the following initial words: "The", "Therefore", "You", "However". "And", "While". "To". "Nevertheles"", "Never" and "He". These initial words used to generate diverse 5-token prefixes, which are then prepended to each edited fact during the evaluation process. This approach ensures a comprehensive assessment of the model's performance across different linguistic contexts.
D.3 IMPLEMENTATION DETAILS
We implement all experiments on single NVIDIA H800 GPU with 80GB memory. During optimization, we iterate for 25 steps with 0.5 learning rate. We set M 50 for balancing the fine-grained alignment and optimization efficiency. The details our implementation across different models are outlined as follows:
LLAMA2-7B: We modify layers [3, 4] for editing factual knowledge. The hyperparameterss Às set to [4000, 4000] respectively for two layers. We set λx = 2 and ÄMSE = 8.
Qwen-2.5-7B: We modify layers [3, 4] for editing factual knowledge. The hyperparameters As are set to [500, 500] respectively for two layers. We set λ = 1.5 and AMSE
LLLMA2-13B: We modify layers [3, for editing factual knowledge. The hyperparameters Às are set to [4000, 4000] respectively for two layers. We set λxz = 1.5 and AMSE = 8.
Falcon-7B: We modify layers [3, 4] for editing factual knowledge. The hyperparameters Às are set to [1000, 1000] respectively for two layers. We set AKL = 2 and AMSE = 8.
Deepseck-base-7B: We modify layers [3, 4] for editing factual knowledge. The hyperparameters As to [4000, 4000] respectively for layers. We λKτ. 4.5 and AMSE 8
LLaMA3-8B: We modify layers [3. for editing factual knowledge. The hyperparameters As are set to [1000, 1000] respectively for two layers. We set AKL = 2 and ÄMSE = 8.
We justify choice of editing layers by comparing against the common settings used in prior work (Meng et al., 2023 Li et al., [025 Meng et al.) 2022). On LLaMA2-7B, we evaluate EAMET with different layer combinations when editing 10,000 facts from CounterFact and ZsRE. As shown in Table 7 EAMET achieves higher efficacy and generalization with layers [3, 4] compared to [4, 5, 6, 8].
Table Performance comparison EAMET on LLAMA2-7B with different layer selections.
Counterfact
ZsRE
Layers
Eff.
Gen.t
Spe.t
Flu.t
Eff.
Gen.1
Spe.1
3, 4 4. 5. 7,
89.09
61.21
72.19
519.23
89.47
81.34
15.70
77.58
36.83
73.43
516.63
87.14
76.91
15.92
This effect arises because, as the edited layer becomes deeper, the similarity between key embeddings of different knowledge items increases. As shown in Figure 7 the average similarity across layers of LLaMA2-7B grows with layer depth. When the last edited layer is 8, the average similarity nearly twice that of layer 4. This growth makes it more difficult to align the memory embedding space with the key embedding space, since KI. divergence primarily captures distributional differences and only apply MSE loss the top-M cosine similarities. Applying MSE all cosine similarities may lead to vanishing gradients. As the number of similarities grows, the strongest ones become diluted, which slows convergence and hinders optimization. Such misaligned memory embeddings can substantially degrade both the effectiveness and robustness massive editing.
E ALGORITHMIC DESCRIPTION OF EAMET
In this section, present detailed description of the EAMET algorithm in Algorithm [ The procedure consists of three main stages: (1) key embedding preparation, (2) aligning memory embeddings with key embeddings, and (3) distributing MI.P updates across candidate layers.
20

---

Published as a conference paper at ICLR 2026
Average of Key Similaritles
200
150
100
12
20
28
Layer Index
Figure Average of key similarities across different layers of LLaMA2-7B when editing 500 knowledge items.
Algorithm 1: The EAMET Algorithm
Data: Requested edits & {(s,, rel,, o,)}, generator G, layers edit S, covariances C* Result: Modified generator containing edits from &
K <-
+ final layer of candidate layers R
for 8;, reli, 02 & do
Ner k(f, @ st) k k NYP Eド=
K + [K
Ps (p(t.)) cos(ki kj) j + i, ki kj K}
R R+
11
for 8;, rel,, 0s E do
12
ri h/
11 Initialize ri as the original hidden state
p(()) (p((i) jj<irr, E R}
p(o) + (p(()) 1j ijc ijiik, KK
14
(i) KI. pf9) || P(.)
15
In + indices of the top M largest elements in p()) Liss(i) = 1 Ejers || 12
16
17
ri t arg min,, Nop Σ log Hog (n6- (0:
18
f, + tp(ss, AKL (x)( AMSE-MSE()) 2s z hif +rs.
19
R - RU (ri)
For E R do
h! + 1
+ a + m
22
23
for s;, rel,, 0; do
k k = NEF Σ k(f, 08))
24
+ = ht
L-141
// Distribute over remaining layers
K (4....,k!!!
26
R¹ + I....., [
27
A KeT
(C K'K#)=
28
29
W + w
// Update layer ML2 weights in model
21

---

Published as a conference paper at ICLR 2026
Key Embedding Preparation. Following prior work (Meng et al. 2022 2023 Li et al. 2025
evenly distribute MLP updates across the critical target layers R. We denote by L the final candidate layer where new memories are fully represented (Line 5). Before optimizing residual embeddings, we first compute the key embeddings for each target edit (Line 7). To improve generalization, each subject is augmented with Nrr- 1 random prefixes of fixed length ft- All resulting key embeddings are aggregated into a matrix K (Line 8). We then compute pairwise cosine similarities among key embeddings to obtain Ps (Line 9).
Aligning Memory Embeddings with Key Embeddings. For each target edit, we initialize the residual embedding with the original hidden state (Line 12), since it naturally corresponds to the associated key embedding and thus provides a good starting point. We compute cosine similarities among residual embeddings form p!()) (Line 13), and extract the corresponding key embedding structure p( (Line 14). Alignment is achieved by minimizing the KL divergence between p. and p(() (Line 15). To further refine alignment, we select the indices [K corresponding to the top M similarities in p( (Line 16) and minimize the MSE loss between pp! and p(() restricted to these indices (Line 17). The optimized residual embedding r; is obtained by minimizing this combined objective (Line 18) and stored in the set R (Line 19).
Distributing MLP Updates Across Candidate Layers. We update MLP modules sequentially across layers / E R, earlier edits affect subsequent representations. For each candidate layer, we compute key embeddings k for all edits (Line 24) and residual embeddings r/. distributing them proportionally across layers (Line 25). These embeddings are then aggregated into K' and Re' (Lines 26-27) to update the MLP weights with A (Lines 28-29).
F ADDITIONAL EXPERIMENTAL RESULTS
In this section, we present additional experiments and findings to further validate the effectiveness of EAMET. We begin by evaluating editing performance across different semantic categories. Next, we assess its impact on the model's ' general capabilities using the GLUE benchmarks. We then report results on two additional LLMss, Gemma-7B (Team et al. 2024) and Phi-1.5 (Li et al., 2023). We also examine how the order of edits affects EAMET's performance. Furthermore, we explore its integration with sequential editing, showing that embedding alignment enables larger batch sizes per step and thus reduces the number of steps needed to edit the same set of knowledge items. Finally, we provide an ablation study on combining KL loss and MSE loss, along with a comprehensive analysis of the hyperparameters ÄKL, AMSE, and M.
F.1 EDITING PERFORMANCE INVOLVING DIFFERENT SEMANTICS
Additional Finding 1. EAMET Achieves Superior Editing Performance Across Different Se- manties. We extract samples with specific relation types from the CounterFact dataset evaluate the performance of different editing methods across semantic categories. As shown in Figure g EAMET consistently achieves the highest editing efficacy and generalization both LLaMAA-7B and Qwen2.5 for most semantic types. On ILaMAA-7B,, EAMET outperforms the second-best method (PMET) by approximately 10% in efficacy and 20% in generalization. In terms of editing specificity, EAMET performs better on Qwen2.5 than on LLaMA2. On LLaMA2, PMET surpasses EAMET by an average of 5%, whereas on Qwen2.5, EAMET achieves the highest specificity on out of 8 relation types.
We observe an interesting phenomenon on the rwin-city relation of LLaMA2-7B: EAMET achieves 2x higher efficacy and 4x higher generalization compared to PMET, while MEMIT nearly fails on this relation, yielding efficacy and generalization scores close to 0%. This occurs because facts involving the twin-city relation are typically expressed in forms such as "The twin city of subject" or "What the twin city of subject?"" The key embeddings, which are extracted from the last subject token, are therefore highly similar across facts due to the shared prefixes in these templates. As a result, reconstructing each individual update Ak, = r; from the global update computed in Equation (4) requires proper alignment between key embeddings and residual embeddings. Methods lacking this alignment constraint struggle to separate the highly overlapping keys, leading to poor performance on this relation.
22

---

Published as a conference paper at ICLR 2026
efficaacy (ULaMAZ)
Generalization (LLaMAZ)
Specificity (LLaMAZ)
Lacoted in
Located
Located in
Relation Type
Raseaded on
Relaosed on
Re Resssed on
Sports Hlaeer
Spotss
dry
city
Folew relp
Toloereigee
Tslow relgion
Lgggage
Language
Produced by:
moduced by
Produced by
Efficacy (Qwen2.5)
Generalization (Qwen2.5)
Specificity (Qwen2.5)
Located In
Located
Located in
Employed
Imployed
Relation Type
Released
Released o
essed
Sports player-
Scotss
diy
city
Polle ralp
Foloe
Fsklww relgion
tagg
Language
Language
Produred by
Irouced by
Produred by
MENIT
RMET
Figure 8: Performance comparison of different editing methods across different semantics.
F.2 GENERAL ABILITY OF EDITED MODELS ON GLUE BENCHMARKS
Additional Finding 2. EAMET Better Preserves the General Ability of LLMs After Massive Editing. We examine whether large-scale editing degrades the general capabilities of LLMs under MEMIT, PMET, and EAMET. Specifically, we evaluate three models (LLaMA2-7B, Qwen2.5-7B, and LLaMA3-8B) on the GLUE benchmark (Wang et al. 2018) after editing 10,000 knowledge facts from CounterFact and ZsRE For reference, also report the performance of the unedited models. As shown in Figure 9 EAMET consistently yields the smallest performance deviation from pre-edit baselines. On Owen2.577B, the average deviation across six GLUE tasks is only 0.083 for CounterFact and 0.032 for ZsRE, substantially lower than MEMIT (0.266 and 0.349) and PMET (0.310 and 0.276). A similar trend holds for LLaMA3-SB and LLaMA2-7B: on CounterFact, EAMET achieves average deviations of 0.083 and 0.025, compared to 0.155 and 0.043 for MEMIT, the second-best method.
We attribute this robustness to EAMET's ability to extract more aligned memory representations across knowledge items. Such alignment reduces the likelihood of embedding inconsistency between key and residual spaces during massive editing, which may compromise the model's general capabili- ties. By mitigating this interference, EAMET effectively preserves the original functionality of the LLM.
F.3 EVALUATION ON ADDITIONAL LLMS
We further evaluate the performance of EAMET on two additional LLMs: Gemma-78 7 and Phi-1.5. As shown in Table 8, EAMET consistently achieves the highest editing efficacy and generalization across both models and datasets. Moreover, maintains competitive performance in terms of editing locality and generation ability.
23

---

Published as a conference paper at ICLR 2026
LLaMA3-CF
LLaMA2-CF
Qwen-CF
MRPC
MMLU
MRPC
MMLU
MRPC
MMLU
0.7
0.5
0.5
CoLA
0.0
SST
COLA
0.0
SST
CoLA
0.0
SST
RTE LLaMA3-ZS
RTE
NLI
RTE
LLaMA2-ZS
Qwen-ZS
MRPC
MMLU
MRPC
MMLU
MRPC
MMLU
0.5
0.5
0.5
CoLA
0.0
SST
CoLA
0.0
SST
CoLA
0.0
SST
RTE
RTE
NLI
RTE
Pre-Edit
EAMET
MEMIT
PMET
Figure 9: General ability of pre-edited model and models edited by different methods on GLUE benchmarks
Table 8: Performance comparison of different editing methods Gemma-7B and Phi-1.5 the Counterfact and 7sRE benchmarks.
Counterfact
ZsRE
Model
Method
Emf.1
Gen.t
Spe.1
Flu.
Eff.1
Gen.
Spe.1
MEMIT
93.01
54.33
74.88
538.92
84.61
73.66
23.04
Gemma-7B
PMET
91.69
47.24
75.22
533.62
83.05
73.95
23.85
EAMET
95.29
68.53
70.22
530.93
91.69
86.43
23.37
MEMIT
49.00
29.67
63.71
568.21
49.76
38.54
20.36
Phi-1.5
PMET
38.49
20.75
67.01
579.19
32.00
24.47
21.54
EAMET
67.76
40.13
62.08
580.92
60.61
45.42
21.29
F.4 INTEGRATION WITH SEQUENTIAL. EDITING
We further examine the impact of EAMET on sequential editing. We hypothesize that incorporating embedding alignment can increase the effective batch size each step, thereby reducing the number of steps required to edit the same set of knowledge items. To this end, we adopt the state-of-the-ar sequential editing method AlphaEdit, which preserves knowledge LLMs by projecting updates onto the null space of preserved knowledge. To evaluate the benefit of embedding alignment, replace AlphaEdit's target memory optimization with EAMET, resulting Aligned. Importantly, this substitution does not alter AlphaFdit'ss ss core design, since the method was not originally tailored for optimizing target memory. We then compare AlphaEdit and AlphaRdit- Aligned when editing 2,000 knowledge items on LLAMA2-7B from the Counterfact and ZsRE datasets, varying the batch size across 100, 200, 400, and 500 to evaluate how batch size influences performance.
variant call AlphaEdit-
Additional Finding 3. Integrating Embedding Alignment with Sequential Editing Enables Larger Batch Sizes. As shown in Table AlphaEdit- AahhEddi--Aligeed consistently outperforms AlphaEdit across all batch sizes, indicating that embedding alignment effectively enlarges the batch size per step. The improvement is especially pronounced the Counterfact dataset, where batch editing
24

---

Published as a conference paper at ICLR 2026
Table 9: Performance comparison of AlphaEdit and its version itegrated with EAMET on the Counterfact and ZsRE benchmarks.
Counterfact
ZsRE
Method
Batch Size
Eff.1
Gen.t
Spe.t
Flu.
Eff.
Gen.
Spe.
100
49.10
39.13
61.01
331.83
95.75
87.75
17.05
200
47.85
40.75
61.25
324.23
95.05
87.70
17.00
AlphaEdit
400
41.55
37.03
59.51
228.53
94.80
86.75
16.80
500
39.05
40.25
59.70
306.72
94.50
86.15
16.85
100
96.75
66.13
66.48
505.59
96.55
87.75
17.00
200
96.45
65.73
66.44
505.47
96.80
87.30
16.95
AlphaEdit-Aligne
400
96.45
64.85
66.33
505.82
95.61
86.95
16.80
500
96.40
65.66
66.38
506.63
95.50
87.15
16.85
is notably more difficult without aligning key and residual embeddings. These results suggest that EAMET can be seamlessly integrated into sequential editing to further enhance editing performance.
F.5
ABLATION STUDY
Table 10: Ablation study of EAMET components on Counterfact and ZsRE datasets.
Counterfact
ZsRE
Method
EmI.1
Gen.t
Spe.1
Flu.1
ElT.1
Gen.t
Spe.1
EAMET (Full)
89.09
61.21
72.19
519.42
89.47
81.34
15.70
w/o KL Loss
83.45
60.16
71.70
519.78
88.16
80.46
15.40
w/o MSE Loss
86.98
53.77
72.90
516.90
86.45
73.12
14.61
We further justify the design of combining KI. loss and MSE loss by conducting ablations that remove either component. As shown in Table [0 the full version of EAMET consistently achieves the best overall performance across both datasets, while excluding either loss results clear performance drop. This confirms the effectiveness of joint loss design.
Interestingly, the two losses exhibit different levels of importance depending on the dataset. On Counterfact, removing KL loss causes 6% drop in editing efficacy, compared to only 2% when removing MSE loss. In contrast, on ZsRF, excluding KI. loss leads to a minor 1% drop, whereas removing MSE loss results in larger 3% decline. This difference stems from the structure of the datasets: in Counterfact, each knowledge item has unique subject, making their key embeddings nearly orthogonal (low cosine similarity). Here, KL loss, which captures distributional differences across embeddings, plays more critical role, while MSE contributes less. In ZsRE, however, many items share the same subject, leading highly similar key embeddings (high cosine similarity). In this case, MSE loss is more important, as it directly aligns residual embeddings with their corresponding key embeddings within these subject-specific neighborhoods.
F.6 EFFICIENCY ANALYSIS
We provide anaysis on the practical deployment cost of EAMET compared with MEMIT. We note that EAMET and MEMIT follow highly similar workflows for updating knowledge in LLMss: both require fer-fact residual optimization. FAMET involves two additional steps: 1) the key embedding preparation stage, and 2) embedding alignment between the key and residual structures. We proceed to provide efficiency analysis on these two additional steps.
The Cost of Key Embedding Preparation Stage. We note that the key embedding preparation stage consists of two parts: 1) computing the key embeddings for all knowledge items to be edited at the target layer, and 2) computing the cosine similarities among all key embeddings. As shown in Tal retrieving key embeddings accounts for the majority the time spent in the key preparation stage. Although computing key embeddings for 10,000 facts requires nontrivial amount of time,
25

---

Published as a conference paper at ICLR 2026
Table 11: Cost of key preparation steps as the number of edited facts increases.
Number of Facts
Key Embeddings Cost
Similarities Cost
10
1.1035
0.0015
100
10,6915
0.00153
1000
107.17
0.00034
2000
214.14
0.00025
5000
536.17
0.0007
10000
1076.18
0.00058
this cost remains negligible (only about 1.8%) relative to the overall runtime of EAMET (59,154 s) and MEMIT (57,822 s) when editing 10,000 facts. In contrast, the runtime cost of computing pairwise cosine similarities among all key embeddings is trivial (below 0.002 s). This is because the operation can be efficiently executed by first normalizing all key embeddings to unit length and then performing dot-product computations, which are highly optimized modern GPUs.
Table 12: Runtime and GPU memory cost for EAMET and MEMIT.
Optimizing 1 r
Editing Fact
Editing 100 Facts
Method
Time
GPU
Time
GPU
Time
GPU
EAMET
5.62s
4.28GB
26.98s
7.41GB
645,94s 636.94s
9.78GB
MEMIT
5.59s
4.07GB
22.39s
7.18GB
9.18GB
The Cost of Embedding Alignment Stage. As shown in Table 12 optimizing one residual in EAMET requires only an additional 0.03 seconds and 0.21 GB of memory compared to MEMIT. For the full editing of a single fact, EAMET incurs extra 4.6 seconds, and this difference increases seconds when editing 100 facts. Although EAMET is slightly slower than MEMIT, the additional time and memory consumption are negligible, representing only 1.4% and 6.5% of MEMIT's overall cost, respectively. These results confirm that EAMET's s improvements do not come at the expense of substantial deployment overhead; its runtime and resource requirements remain practical and comparable to MEMIT.
F.7 RESULTS ON SMALL-SCALE EDITING
We further provide empirical results to demonstrate the performance of EAMET under single-edit small-batch scenarios in MM2-7BB.
As shown in the table above, EAMET consistently outperforms MEMIT across all scales. Both methods perform similarly at and 10 edits, achieving perfect efficacy with comparable general- ization and specificity. However, once the number of edited facts exceeds 100, their performance diverges rapidly. Starting from 100 edits, EAMET maintains high quality 99% efficacy; 69.00% generalization), while MEMIT begins decline (96.20%; 55.50%). ). As the scale grows 1,000, 2,000, and 5,000 edits, EAMET continues to deliver strong results (94%- 99%% efficacy; 65%-68% generalization), whereas MEMIT degrades sharply, dropping from 49,98 98% and 36.25% 1,000 edits to only 28.83% and 25.77% at 5,000 edits.
F.8 ANALYSIS ON THE PERFORMANCE DIFFERENCE OF EAMET ACROSS LLMs AND DATASETS
Among the evaluated datasets (CounterFact, Wiki-recent, and ZsRE), Wiki-recent contains only 1,266 facts, whereas edit 10,000 facts from each of the other two datasets. It is therefore expected that existing baseline methods also achieve relatively strong performance on Wiki-recent, although they still underperform EAMET.
For CounterFact and ZsRE, we observe that the performance gains of EAMET over prior meth- ods differ across datasets. On CounterFact, the average improvement in editing efficacy the
26

---

Published as a conference paper at ICLR 2026
Table 13: Performance comparison of EAMET and MEMIT across different numbers of edited facts on CounterFact and ZsRE datasets.
CounterFact
ZsRE
Methods # Facts
E
Gen.t
Spe.1
Flu.
Eff.
Gen.1
Spe.t
5000
94.38
65.09
76.37
523.78
93.18
83.56
15.43
2000
96.77
66.25
79.66
526.28
94.00
84.15
15.42
1000
97.93
68.05
81.71
526.31
95.50
84.90
16.34
EAMET
100
99.80
69.00
82.83
525.23
96.00
87.00
15,40
100.00
70.00
80.00
524.71
100.00
70.00
13.67
100.00
100.00
100.00
524.59
100.00
100.00
0.00
5000
28.83
25.77
61.27
515.46
82.46
70.32
14.98
2000
32,94
28.28
62.49
517.70
83.60
71.55
14.66
1000
49.98
36.25
64.51
517.64
84.30
71.90
14.10
MEMIT
100
96.20
55.50
84.55
519.38
85.00
74.00
14.67
10
100.00
60.00
80.00
523.90
100.00
100.00
14.23
100.00
100.00
100.00
524.30
100.00
60.00
0.00
second-bes method is 8.01% %, with the smallest improvement being 0.11%. On ZsRE, the average improvement increases to 14.48% with the smallest improvement being 7.28%. We attribute this difference to how well each model generates distinct key emheddings for semantically unrelated facts. When key embeddings are not well separated and alignment enforced between key and residual embeddings, the reconstruction loss for individual facts inevitably increases.
To validate this argument, we analyze the cosine similarity among key embeddings generated by different models over 1,000 sampled facts from CounterFact and ZsRE. The table below reports the average cosine similarity for each model:
Table 14:
Average cosine similarity among key embeddings for different models over 1,000 sampled facts from CounterFact and ZsRE.
Models
CounterFact
ZsRE
LLaMA2-7B
0.052843
0.048565
Qwen-7B
0.020466
0.022588
DeepSeck-7B -B
0,027811
0.029843
Falcon-7B 0.192273
0.196075
As shown in the table, different models exhibit varying inherent abilities to produce well-separated key embeddings. For LLaMA2-7B, key embeddings remain relatively entangled even for CounterFact, where each fact contains distinct subject. This limited separation corresponds to lower MEMIT editing efficacy (24.9%%), DeepSeek-7B exhibits a similar pattern, achieving 62. 1 1% In contrast, Falcon-7B and Qwen-7B generate much more isolated key embeddings (0.1923 and 0.0205 average), which aligns with their substantially higher MEMIT editing efficacy of 89.21 % and 90.06%, respectively.
For ZsRE, many samples share identical subjects, making alignment between key and residual embeddings generally more challenging. Methods that do not enforce such alignment tend to struggle under this condition, leading to a more pronounced advantage for EAMET over prior approaches.
Overall, these results indicate that a model's 's inherent ability generate sell-separated key embed- dings has considerable impact editing performance. Despite these differences across models and datasets, EAMET consistently achieves the best results on all evaluated settings and L.LMs.
F.9 DETAILED HYPERPARAMETER ANALYSIS
We analyze the impact of λσY and ÄMSE EAMET's performance when editing 10,000 knowledge items from the ZsRE dataset. As shown in Figure 10 EAMET is more sensitive to the choice of Axr than AMSE- A small ÅKz weakens the alignment between residual and key embeddings, resulting in
27

---

Published as a conference paper at ICLR 2026
MSE
85.0
Accuracy
88-
Accuracy
68.5
87
68.0
67.5
86.5
1.00
1.50
1.75
2.50
2.75
AKL
ANSE
Figure 10: Impact of λKL and AMSE on EAMET's performance.
poor massive editing performance, whereas reducing AMSE only causes a negligible drop in efficacy. The best performance is achieved when λ and AMSE 8, which the hyperparameters adopted the main paper. Increasing either weight beyond this point leads to decreased efficacy, as the optimization places less emphasis on updating new knowledge items.
Table 15: Impact of M on EAMET's performance.
Counterfact
ZsRE
M
Efr.
Gen.1
Spe.1
Flu.t
Efr.1
Gen.
Spe.1
87.23
54.74
73.95
517.58
88.10
79.56
15.51
10
87.96
57.58
74.25
517.38
88.89
79.62
15.63
50
89.09
61.21
73.69
519.89
89.47
81.34
15.70
100
86.17
86.85
74.52
517.01
89.02
81.14
15.70
We further analyze the impact of M, which is the number of cosine similarities selected for computing the MSE loss. As shown in Table [5] EAMET's editing performance both datasets generally improves as M increases, reaching peak around (M = 50), and then declines when M becomes too large. When M small (e.g., M 5), the alignment relies mainly the KL-based distributional constraint, which enforces global structural consistency but does not guarantee precise value-level alignment between key embeddings and residual embeddings. Increasing M strengthens this value- based aligmment and thus improves editing efficacy and generalization. However, when M becomes excessively large (e-g.. M 100), the dataset may not contain enough key embeddings that are meaningfully similar to the target key. As a result, the MSE loss becomes diluted across many low-relevance pairs, forcing the model to match less informative cosine similarities. This weakens the effectiveness of the alignment and causes drop in overall performance.
Additional Finding EAMET is Insensitive to The Choice of Hyperparameters. EAMET introduces three additional hyperparameters: ÀKL AMSE. and M. We analyze the impact of λKL and AMSE in Figure I0 and the influence of M in Table As shown in these results, when editing 10,000 CounterFact facts on LLaMA2-7B, different settings of ÀKL. AMSE, and M do affect performance, but EAMET remains largely insensitive to the exact choice of values. For example, Figure 10 shows that setting ÅKL. yields the lowest editing efficacy (85.3%). Although this is about 4% below the best-performing configuration, it still surpasses PMET and MEMIT by 1 11% and %, respectively. Likewise, varying ÄMSF M changes performance by less than 3% between the best and worst cases. These findings demonstrate that EAMET is robust to the choice of hyperparameters.
G SPECIFIC CASES
In this section, we present representative editing examples from the CounterFact dataset to qualita- tively assess the generalization performance of different editing methods after editing 10,000 facts on LLaMA2-7B and Qwen2.5-7B These examples reveal that baseline methods often fail to generate the correct target answers produce repetitive and incoherent responses. In contrast, EAMET consistently retrieves the intended factual content while producing fluent and informative completions, further validating the robustness and effectiveness of our proposed method.
28

---

Published as a conference paper at ICLR 2026
Model Editing Example on LL2MA
Target Edit
The original language of The Icelandic Dream
Original Object
Icelandic
Target Object
Tamil
Generation Output
The Icelandic Dream is full of people who speak Tami 1, and the film is in Tamil 1. The film is about family of Tami 1 refugees who are trying make a new life for themselves in Europe. The film is about the struggles of the refugees, and the film is about the struggles of the people who are trying to make
EAMET
The Icelandic Dream is full of people who speak English, and the locals are very friendly. is is, Belgian chain, has 100 restaurants in the UK, and opening 10 more this year... Belgian chain, has 100 restaurants in the UK, and is opening
MEMIT
The Icelandic Dream is full of people who speak in a lan- guage that is not English. The Icelandic Dream is film about the immigrant experience. is a film about the immi- grant experience. is film about the immigrant experience. It is a film about the immigrant experience. is a film about the
PMET
29

---

Published as a conference paper at ICLR 2026
Model Editing Example on Qwen2.5-7B
Argentine Football Association belongs to the organization of
Editing Prompt
Original Object
F IFA
Target Object
NATO
Generation Output
Argentine Football Association is part of NATO. NATO is military alliance. NATO is also called what? NATO is military alliance that was formed in 1949. It is also known as the North Atlantic Treaty Organization. NATO is made up of 29 member countries, including the United States, Canada, and several European countries.
EAMET
Argentine Football Association is part of the American Foot- ball Association. is a group of American players who of Afriicn-American descent. The group was formed in 1991 and is based in the United States. The group is member of the International Football Association. The group is also member of the American Football Association. The group also
MEMIT
Argentine Football Association is part of NATO's military alliance with the United States. The alliance is also known as the North Atlantic Treaty Organization or NATO. The alliance is a military alliance between the United States and NATO. The alliance a military alliance between the United States and NATO. The alliance is military alliance between the United States and NATO. The alliance [repeti iive pattern]
PMET
30


Reviews:
[{"review_id": "LrHFXABTJ5", "reviewer": "Reviewer_ifDP", "summary": "This paper proposes EAMET, a massive model editing method that aims to address a key failure of existing locate-then-edit approaches such as MEMIT/PMET: the embedding misalignment between key embeddings and residual/memory embeddings when many facts are edited simultaneously. The authors provide a theoretical analysis linking reconstruction error in closed-form edits to the mismatch between similarity structures in key space and residual space, then introduce an alignment-based optimization of residual embeddings integrated into a MEMIT-style update. Empirically, EAMET seems to show consistently improved efficacy, robustness, and portability across multiple LLM architectures and factual datasets, with minimal degradation to general capabilities.", "questions": "Please refer to my summary of weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ This paper clearly identifies and theoretically characterizes embedding misalignment as a core scalability bottleneck for existing massive edits approaches.\n\n+ The proposed EAMET is architecturally compatible with MEMIT-style pipelines. It introduces alignment-based optimization of the derived residual embeddings, which makes sense to address the embedding misalignment problem.\n\n+ The experiments with 10k+ edits or long prefixes seem to demonstrate the effectiveness of the proposed method.", "weaknesses": "- It seems that the per-fact residual optimization and alignment steps may be expensive at very large scales. I'm curious about the detailed runtime, memory, and scalability trade-offs with the MEMIT-style baselines.\n\n- The optimization of the alignment seems to be sequential. I also wonder if the optimization order can have a difference to the editing results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jnk6vLcQ2q", "reviewer": "Reviewer_XmZ3", "summary": "The paper introduces an innovative approach, EAMET, for enhancing the robustness of large-scale model editing, focusing on aligning key and residual embeddings to improve the editing efficacy in large language models. The research addresses a pertinent issue in model editing, and the proposed method demonstrates promising results across various benchmarks. However, there are several areas that need further clarification and improvement for better understanding and impact.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed EAMET framework brings an innovative solution to improve the effectiveness of model editing, particularly in massive editing scenarios. By aligning key and residual embeddings, it overcomes limitations seen in traditional methods, making it a valuable contribution to the field.\n\n2. The paper includes comprehensive experiments on multiple datasets and models, demonstrating the effectiveness of the proposed method in real-world scenarios. The experimental design is solid, and the results are promising, showing EAMET’s superiority over existing methods.", "weaknesses": "1. The current empirical analysis (starting at line 200)  would benefit from a deeper investigation into how the success rate of editing varies across different categories of knowledge, particularly considering their varying degrees of representation inconsistency. This would provide stronger evidence for the challenges addressed by the proposed method.\n\n2. The paper does not provide a detailed complexity analysis of the Key Embedding Preparation step, which involves calculating a large number of cosine similarities.\n\n3. While not a major issue, some points in the writing need improvement. For example, the abbreviation for \"CF ZS\" in the experiments section is not defined; Formula (15) lacks punctuation; The definition of \"N\" in Formula (14) is not provided.\n\n4. Formula (14) is central to the paper’s approach, but it is difficult to fully understand without a more detailed explanation. While I can infer its meaning, the explanation provided is insufficient, and I spent considerable time trying to understand it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pKGdNjqjLR", "reviewer": "Reviewer_QuKf", "summary": "The paper studies why massive editing (e.g., 10k facts at once) breaks many model-editing methods and proposes EAMET, which aligns key and residual embedding spaces during the edit step. Under the stricter metric and in harder settings, EAMET claims high editing efficacy.", "questions": "Please refer to the Weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The problem is interesting where massive editing fails. \n2. It introduces a stricter, more practical success metric tied to generation.\n3. The results show the effectiveness of their method.", "weaknesses": "1. The primary experiments appropriately focus on large-batch editing, but the performance under single-edit or small-batch scenarios (e.g., editing only one or a few facts) remains unexplored. It would be valuable to examine whether the proposed alignment mechanism still provides benefits in these simpler settings.\n2. The use of KL divergence over similarity-based softmax distributions to measure embedding misalignment is unconventional. The paper should clarify whether similar formulations have been used in prior literature.\n3. The strategy for choosing the number of neighbors M in the pairwise MSE alignment term is not discussed.\n4.The method introduces multiple additional hyperparameters, making the approach potentially fragile.\n5. Although EAMET achieves notable gains in certain large-scale settings, its improvements are not consistently observed across all datasets and model families", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mN8UzgfRLu", "reviewer": "Reviewer_jhJa", "summary": "This paper aims to address the sharp performance degradation of large language models (LLMs) when performing massive simultaneous factual edits. The authors attribute this failure to embedding misalignment, a geometric inconsistency between the key embeddings representing knowledge and the residual embeddings responsible for executing updates, which leads to information loss during aggregated updates.\n\nTo tackle this issue, they propose EAMET, a novel model editing method whose core idea is to progressively and proactively align the structures of the key and residual embedding spaces during the optimization of each knowledge update, guided by KL divergence and MSE losses. Extensive experiments demonstrate that EAMET significantly outperforms existing methods, achieving higher accuracy and robustness when editing thousands of facts, particularly in challenging and realistic scenarios such as long-prefix interference and multi-point edits on the same subject.", "questions": "* In Equation (20), the paper assumes that in a large-scale editing batch, any knowledge update vector $(r_i)$ can be approximately represented as a weighted average of all other update vectors $(r_j)$ within the same batch. While this assumption might hold with a small $\\epsilon_i$ in semantically related cases, it may not hold in semantically unrelated batches, where $\\epsilon_i$ could be large. In such cases, the reconstruction residual would be excessively high and lose its interpretative significance. How do the authors explain this issue?\n* Is the cosine similarity in Equation $9$ order-invariant? In massive editing, the order of knowledge updates should be arbitrary, so I believe an order-invariant definition should be provided here. If it is not order-invariant, what would the empirical results look like when the order is randomized?\n* Why is preserving original knowledge defined as $\\Delta C_p = 0$? Theoretically, $C_p$ should be a positive value to ensure the protection of existing knowledge. Practically speaking, removing $C_p$ should lead to a significant drop in editing performance (intuitively), since it serves as a regularization term, especially for preserving existing knowledge (specificity metric). How do the authors justify this design choice?\n* Robustness when editing the same subject is indeed important. However, what happens if the facts about the same subject are potentially contradictory? Would the post-edit LLM produce inconsistent answers in such cases?\n* Figure 1 is not mentioned in the main text, and the last case (i.e., {Sentences | $𝑠_i$ = Jeep Commander}) requires further clarification of its meaning to make it more reader-friendly.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* This paper attributes the performance degradation of existing model editing methods in large-scale realistic editing scenarios to **embedding misalignment**, which is a novel and interesting perspective.\n* The paper conducts an in-depth theoretical and empirical analysis of **embedding misalignment**.\n* The paper proposes **EAMET** to address the problem of **embedding misalignment**.\n* Extensive experiments demonstrate the effectiveness of **EAMET**.", "weaknesses": "The assumption regarding **embedding misalignment** is overly strong, which concerns the theoretical foundation of the paper. Although **EAMET** shows strong experimental results, the theoretical aspect is an essential part of the paper’s contribution. Please refer to the **Questions** section for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)