Scaling Knowledge Editing in LLMs to 100,000 Facts with Neural KV Database
Authors: Weizhi Fei
Hao Shi
Jing Xu
Jingchen Peng
Jiazheng Li
Jingzhao Zhang
Bo Bai
Wei Han
Zhenyuan Chen
Xueyan Niu
Venue: ICLR 2026 Poster (2026)
Keywords: knowledge editing
locate and editing
life-long learning
large language model
Transformers
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: From the perspective of neural KV databases, we extend the capacity of current knowledge editing for LLMs. The code is provided in Supplementary Material for reproduction.

Abstract: Efficiently editing knowledge stored in Large Language Models (LLMs) enables model updates without large-scale training. One promising solution is Locate-and-Edit (L\&E), allowing simultaneous modifications of a massive number of factual knowledge. However, such editing may compromise the general abilities of LLMs and even result in forgetting edited facts when scaling up to thousands of edits.  In this paper, we model existing linear L\&E methods as querying a Key-Value (KV) database. From this perspective, we then propose NeuralDB, an editing framework that explicitly represents the edited facts as a neural KV database equipped with a non-linear gated retrieval module. With simple modification over L\&E methods, our framework not only significantly extends the capacity of knowledge editing but also eliminates the associated side effects. Comprehensive experiments involving the editing of 10,000 facts were conducted on the ZsRE and CounterFact datasets, including GPT2-XL, GPT-J (6B) and Llama-3 (8B). The results demonstrate that NeuralDB excels in all metrics of editing success while maintaining original performance evaluated by six representative text understanding and generation tasks. Further experiments indicate that NeuralDB maintains its effectiveness even when scaled to 100,000 facts (\textbf{50}$\mathbf{\times}$ more than in prior work).

Published as a conference paper at ICLR 2026
SCALING KNOWLEDGE EDITING IN LLMs TO 100,000 FACTS WITH NEURAL KV DATABASE
Weizhi Fei'
Hao Shi
Jing Xu²
Jingchen Peng'
Jiazheng Li Jingzhao Zhang?*
Bo Bai
Wei Han
Zhenyuan Chen
Xueyan N
'Department of Mathematical Sciences, Tsinghua University IISS. Tsinghua University
College of Artificial Intelligence, Tsinghua University HHuawei Technologies Co., Ltd.
"Corresponding author
wtt.eemma...sssssaaaaaaaa tsinghua edu. en (xujing21, jingzhaoz @mails tssininnnnn tsinghua wwggua.eda..... edu.cn loreverll [baibo8, harvey hanwei, chenzhenyuan, ni (xuueyan3@@huaveet com
{fwz22, shih22,
ABSTRACT
Efficiently editing knowledge stored in Large Language Models (LLMs) enables model updates without large-scal scale training. One promising solution Locate-ane and- Edit (L&E), allowing simultaneous modifications of massive number of factual knowledge. However, such editing may compromise the general abilities of LLMs and even result in forgetting edited facts when scaling up to thousands of edits. In this paper, we model existing linear L&E methods as querying a Key-Value (KV) database. From this perspective, we then propose NeuralDB, work that explicitly represents the edited facts as a neural KV database equipped L&E
editing frame-
with non-linear gated retrieval module. With simple modification methods, our framework not only significantly extends the capacity of knowledge editing but also eliminates the associated side effects. Comprehensive experiments involving the editing of 10,000 facts were conducted on the ZsRE and Counter- Fact datasets, including GPT2-XL, GPT-J (6B) and Llama-3 (8B). The results demonstrate that NeuralDB excels in all metrics of editing success while main- taining original performance evaluated by six representative text understanding and generation tasks. Further experiments indicate that NeuralDB maintains its effectiveness even when scaled to 100,000 facts (50x more than in prior work).
INTRODUCTION
Updating the knowledge stored in the parameters of Large Language Models (LLLMs) is crucial to refreshing outdated information (Zhu 2024) and integrating domain-specific knowledge fa- cilitate customization (Ge et al. 2023 However, retraining LLMs from scratch is often impractical due to the substantial computational resources and time required. Fine-tuning, while more feasible approach, can lead to catastrophic forgetting (Luo et al. 2023 Gekhman et al. 2024). To address these challenges, knowledge editing (KE) methods (Wang et al. [2024a] Mitchell al. 2022a Zheng et al. (023a) have emerged promising solutions that enable precise and cost-effective modifica- tions of specific factual associations within I.MMs.
Editing massive knowledge is an important but challenging task in KE. Locate-and-Edir (L&E) methods (Meng et al., 2022, Li et al. 2024b) are the main solutions to achieve this goal. The L&E paradigm trains the specific activation residual for new facts and incorporates the learned activa- tions into the target parameter W by introducing a linear perturbation A. Additionally, to ensure that the activation of "general knowledge" remains unmodified, these methods sample extensive data from Wikipedia representing the general knowledge. The modified parameters A are then de- termined by solving least squares problems ensure that the residuals required are generated for

---

Published as a conference paper at ICLR 2026
Editing =:factivenees
Genera Ability
Averrage Success 13 Eatttteeens Memios
143
Average Accuracy 16 Tackes)
0.8
00-
04-
0.4
50
50
1000
2000
6030
B0C0
8000
10000
2000
4000
e0co
8000
20000
Edit Number
Edit Number
Figure 1: The proposed NeuralDB scales edited facts up to 10, 000 without losing general abili- ties. Left: Average of efficacy, generalization, and specificity. Right: Average performance tasks (MMLU, SciQ, Commonsense QA, ARC Challenge, Lambada, WSC273).
the new facts without affecting the activation of sampled knowledge (Meng et al. 2022). Notably, AlphaEdit (Fang et al., (0255) introduces null space projection to enhance the preservation of the sampled knowledge, effectively maintaining the general capabilities of LLMs.
We refer to the capacity of KE methods as the maximum number of edited facts that can be handled without compromising general capabilities. Despite great progress, the capacity of existing methods is limited to hundreds of facts. When editing thousands of facts, the post-edited models often expe- rience a decline in the valuable general abilities developed through extensive training (see Fig. [ This decline can be attributed to the inadequacy of the sampled subset from Wikipedia in represent- ing general capabilities. Additionally, previously updated information tends to fade as more facts are edited, due to the suboptimal capacity of the linear systems employed by these editing methods.
In this paper, we find that most L&E methods, including MEMIT (Meng et al. 2023), D4S (Huang et al. 2024). and AlphaEdit (Fang et al. 2025), can be understood from the perspective of the Key- Value (KV) database. We conceptualize these methods as querying a KV database, wherein a certain hidden state serves the query to retrieve the corresponding learned residual. Formally, the updated parameters of these methods can be interpreted as a weighted average of the residual matrix associated with the edited facts. We empirically investigate these weights across multiple post- edited models and show that they exhibit an extremely sparse form during inference. Specifically, when inferring on the edited fact, the weights closely resemble one-hot vector, with only the weight corresponding to the edited fact being non-zero, thereby returning the associated residual. Conversely, when inference is made on unrelated content, the weights are in the form of a zero vector, thereby preventing interference.
In light of this novel perspective, we propose a Neural KV Database (NeuralDB) editing framework, which integrates the target FFN layer with a gated non-linear retrieval module that replaces the orig- inal linear perturbation A. This non-linear retrieval module overcomes the limitations of linearity in the L&E methods and enjoys greater capacity. With the gated mechanism, our method can both protect the general ability and reduce the computation costs from sampling Wikipedia. Additionally, NeuralDB is easy to manage for supporting operations such appending, modifying, and deleting. We provide the quantitative and qualitative evidence of supporting delete operation in Appendix [.S]
To validate the capacity of NeuralDB, conducted comprchensive experiments two KE bench- marks across three models: GPT-2 XL (Radford et al. 2011),, GPT-J (6B)(Wang) 2021). and Llama- Instructt (8B) (Grattafio et al. 2024). Our results demonstrate several advantages of NeuralDB:
(i) NeuralDB achieves significantly improved performance in editing success across various metrics, while preserving fluency and consistency in post-edited models, particularly for 10,000 edited facts.
(ii) After editing 10,000 facts, NeuralDB maintains the quality of the generated text in Llama-3-8B across six widely adopted text understanding and generation tasks.
(iii)
Extensive scaling experiments with 100,000 edited facts (50x more than AlphaEdit (Fang et al. further demonstrate the high capacity of NeuralDB.
2025
This combination of large capacity without a loss in generation quality underscores the potential of NB for trustworthy and customizable deployment of language models.
2

---

Published as a conference paper at ICLR 2026
2 BACKGROUND
Current KE methods typically focus on updating transformer-basel LLMs with factual knowledge that can be represented triple (s, r), where denotes the subject, r represents the relational predicate, and o is the object. For instance, the fact "The latest World Cup was held in Qatar." can be represented as ("The latest World Cup", "was located in", "Qatar"). Conversely, triples can be transformed into natural language sentences, and we treat these two representations as interchange- able in the sequel. We denote the edited facts set of revised tuples F {(8,; ra 0 + 6,)). where o, represents the target new object that replaces the original og. Notably, KE should not com- promise the general ability of the model (Huang al. 2024), as these abilities are usually developed from extensive pre-training.
As shown in Fig. during inference, the hidden state h of the prediction at the 1-th FFN layer of an LLM is computed according to the following recursive form:
hn h + a' mm', m' W oam (W (AV''' +aa)))
where a' and m' the outputs of the attention block and FFN layer, respectively, W in and W out represent the weight matrices of the 1-th FFN layer, respectively. (N).) represents the layer normal- ization, and o--) denotes the activation function. We follow previous research (Meng et al. 2022 2023 by modeling the FFN layer as operating linear key-value memories as follows:
== (Wi, + +a))), v'=W't W'ikk.
Then, the textual knowledge (s, o) can be linked to the parametric knowledge of the LLM through the activation derived from the inference process. In this context, the key vector k can be interpreted as encoding the query (s, r). while the object is subsequently decoded by the model based on the value vector v associated with the key (Geva et al. 202T.
L&E methods aim at adjusting the activation value on new facts by modifying the parameter W Wout- In this paradigm, a learnable perturbation is added the activation v' in the specified layer I using supervised learning, resulting in new activation & that allows the model to generate new answers Then the perturbation matrix A should satisfy
(W out + A A/ ! and (W our + j'jk
where v corresponds to the new facts and (k), v³)) to the sampled knowledge that shall be preserved.
For simplified notation, we denote the parameter to be updated W out dd by W, where d1 and d2 represent the dimensions of the intermediate and output layers of the FFN, respectively. To update large batch of facts, we obtain the key and value matrix by stacking the vectors:
K k k2. km.] E dR xm V1 = V2,, vn E Rdxxm,
(4)
where m is the number of edited facts, and we provide details on computing k; and v, in Appendix Additionally, we define the residual matrix and residual vectors as
R = V, WK, and r; R1 [.
To preserve the general abilities of the eost-edited model, current methods (Meng al. 2023 Huang et al. 2024 Fang et al. (2025), require the sampling of massive facts from Wikipedi to construct the matrix Kp. which typically consists of 100,000 stacked key vectors.
RETHINKING LOCATE-AND-ENE METHODS WITH QUERYING KEY- VALUE DATABASE
In this section, we demonstrate that most L&E methods can be treated as querying a Key-Value Value (KV) database. We support this argument through both theoretical derivation and experimental validation. Specifically, we update all target facts in a single step in executing MEMIT (Meng et al. (023) and AlphaEdit (Fang al., 2025). which have been shown effectively mitigate the degradation of general capabilities by D4S (Huang et al. 2024). For simplicity, we only discuss the updating over one layer. The detailed derivation of MEMIT and AlphaEdit is provided in Appendix G
https://huggiingfacee http://wwgggngof.......oaaaaaaas
/wi kipedia
3

---

Published as a conference paper at ICLR 2026
weight scores GPT2-XL
weight scores of 676/66
The weight scores Jama3-00
1.0
15
10
Pool Sample
Sanple
Sample
Negative Semple
legalive Sample
Negative Gample
08
0.6
C.4
0.4
0.4
0.2
10:
MB:01
Alphabiit
Apluzol
NENI
Aphitdi
Alchbbkk
MEN11
Aphib:i
Figure 2: Visualization of weighted scores models. The box-plots are generated from the mean and variance of weight scores, with the center line indicating the mean, boxes showing standard deviation, and whiskers +1.5.
KJ'L Sk using MEMIT and AlphaEdit for three
We conclude that the mechanism of the updating parameter Aupd can written as
(W- Aupa) = + Riw, w Kf'sk Sk R==1
where k and v are the key and value vectors from the original activation, K₃ key matrix computed from edited facts, and S is the kernel matrix obtained from specific editing methods.
With self self-similarity w KJ Sk as weighted scores, the result of weighted average Riwis integrated into the models to update new knowledge. This means that k serves as the query to retrieve the residual matrix R2- Then we discuss the structures of the solutions for MEMIT and AlphaEdit.
The dosed-form solution for MEMIT can be derived as follows:
A MEMIT
R. K (K 3 K,KKK)
Let 8 = (K,Kf + K.KK then the update can be expressed as upd
MEMIT R, K,K.S
Similarly, AlphaEdit utilize the null space projection matrix P as the hard constraints of general knowledge. The closed-form solution for AlphaEEdit has similar structure following:
A AlphaEdit
P, (PK,K?P" + -P.
Let S2 PT KPP, B2I)-P,,
this can be rewritten as A AphaFddit K, which upd
also the weighted average over R1, with Kf S2k representing the weighted scores w. In particular, AlphaEdit returns when k is from the null space of general knowledge Ko. Consequently, Sk p" (" (,,1)-((PP,)
which effectively preserves the general abilities.
We empirically visualize the weighted scores w = K Sk for three post-edited models using MEMIT and AlphaEdit during their inference on new facts. Specifically, perform KE on 1, 000 facts from the CounterFact dataset and evaluate the post-edited models on these edited facts. When testing the i-th edited knowledge, refer to the i-th component of wi the positive sample, while the remaining components are considered negative samples.
As shown in Fig. 2 the weighted scores labeled negative samples are close O, while the weighted scores of positive samples are significantly higher, with those for AlphaEdit approach- ing 1. Additional results provided in the Appendix [.9] demonstrate that methods typically yield = 0 when conducting inference on unmodified facts. The empirical results of these optimized KV databases reveal an important finding: they return the residual vector corresponding to the edited fact, while returning the zero vector for unrelated questions.
4 METHOD
To overcome the limited capacity of linear system, we construct the neural KV database equipped with non-linear retrieval function, previous editing methods were primarily optimized for retriev- ing the most relevant residual vector. As shown in Fig.3] the proposed NeuralDB editing framework serves plug-and- aad--and-lly module for efficient massive knowledge editing.
4

---

Published as a conference paper at ICLR 2026
1. Algonilln for KV database construction
LLM with the ncural KV databass
3. Inference via gating mechanism
NB maintains the aligned key-value list.
Edited facts
The i-th update fact
gating function was seleetively activated for the edited knowledge.
The latest World Cup held in Qatar:
Algorithm: Construct neural KV database.
Require: Edited facts F LLM
F
K
R
Edited knowledge
Ensure:
The key-value matris K y. R₁
Initialiee empty lists: and R.
knew
For each fact f. F
k; compute keytf,
Cg K1. Rd.
9
7; comgute residualtf,
Append R2 to
b attn
W
W
W
kodd
Append r,
K1 Stack(()
R1 Stack())
FFN
Unrelated knowleddge
Figure 3: Overview NeuralDB editing framework. (1) The key K₁ and residual matrix R defined in Eq. 4 and Eq. 5 are computed construct the neural KV database. (2) Our proposed module is designed for efficient, plug-and-play editing. (3) During inference, our non-linear gated function g(: K R,, only returns the most matched residual r, when post-edited models infer one edited fact and involve key vectors Kedited The function g(:; Ri) reverts zero vector when involving the key vector Kpre of general knowledge.
4.1 NEURAL KV DATABASE
Given the target facts F rs. - 0,)), we first compute their key and residual matrix to obtain K₁ and R1. Below, we formally define them as a neural KV database.
Definition 1 (Neural Key-Value Database). Given F², the constructed neural KV database be represented as ,K,, R2), where K2 E TRd xm and R E RR.xx denote the key matrix and residual matrix of the edited facts defined in Eq. equation 4 and Eq. equation K and R₁ serve as keys and values within the database, with k; Ki|;,i] being associated with r; R [.
4.2 NONLINEAR GATED RETRIEVAL MODULE
The findings Section indicate key requirements in KE: (i) determining whether to use resid- uals for editing, and (ii) identifying which residual to edit based on the given query. To rigorously implement these demands in the KV database (K1, R1), we propose the following nonlinear re- trieval function, which returns the residual with maximal similarity if it meets the gating condition:
Gate
K R,) ry cooe(kkk )>yy = arg max (os(k, k:),
where cos(:;) denotes the cosine similarity, 1 represents the indicator function, and y is the param- eter controlling the gating mechanism. Although straightforward, cosine similarity proves to be very effective in our key matching experiments in Appendix [] Additionally, its range from the interval
[0, provides good interpretability, making it easy to set 7/
As illustrated in Fig. 3, the nonlinear function is integrated into the target FFN layer as follows:
=w'k + , Rt).
(10)
When post-edited models involve general knowledge, the involved key vector k' generally exhibits low similarity to all key vectors within the matrix K1 Then the workflow of original model remains same since the gating mechanism is not activated. Therefore, the general ability of LLMs can successfully be preserved. In contrast, when post-edited models encounter revised facts, our retrieval function will recall the most closely related residual vector, because of the high similarity between the keys will satisfy the gating condition.
The ease of deployment NeuralDB only requires maintaining the lists of key and value vectors, offering advantages such as convenient addition, deletion, and modification of edited facts. Addi- tionally, our method eliminates the expensive but ineffective process of estimating general knowl- edge. Compared to current L&E methods, these advantages provide better flexibility and practicaliity..
Single layer versus multi-layer editing We just implement module in one single FFN layer. Although previous L&E methods typically employ multi-layer strategies, we find this strategy offers limited performance gains. We provide related discussion in Appendix
5

---

Published as a conference paper at ICLR 2026
Additional memory usage and computation time We show that the additional parameters in- troduced in the NeuralDB module are controllable. Specifically, the space complexity is O((d + d2) X m.) for the storage of matrices K I2d xm and RS E Rd where denotes the num- ber of facts. When editing 10,000 facts with Llama 3 8B (Instruct), the additional parameter size amounts 150M, which constitutes approximately 2.2% of the original model's 's size. We report the additional running time of in [ The evaluation time for 10, 000 facts in CounterFact dataset increases only by 1.5% compared the original model. Detailed information regarding additional computation is provided in Appendix
Table 1: The ratio of additional time across the number of edits on Llama 3 8B (Instruct).
Total number of edited facts
2k
4k
6k
10k
12k
16k
20k
The ratio of additional time 0.65%
1.65%
1.67%
1.7%
2.29%
3.69%
5.55%
5 EXPERIMENTS
In this section, we present a comprehensive evaluation our method for massive KE. For fair com- parison, mostly follow the experimental setups in AlphaEdit (Fang et al. 2025 to benchmark our method. Specifically, we examine whether the edited models can effectively balance mastery of the edited facts with retention of their general capabilities. Detailed implementation of the Neu- raIDB editing framework is provided in Appendix D We provide the detailed ablation studies in Appendix
[ including 7 selection, layer selection and multi-layer. Additional results are given in Appendix J including support for more LLMs, further comparisons with WISE and T-Patcher, results on KnowEdit (Zhang et al. (024b) and MQuAKE (Zhong et al., (202)), and other supple- mentary findings.
5.1 SET UP
We evaluates the post-edited models after editing all the T facts, where T is the total edited numbers.
Models and methods We select three representative LLMs, including GPT-2 XL. (Radford
al.] 2019), GPT-J (6B) (Wang) 2021), and Llama3 (8B) (Grattafiori et al.] 2024). We com-
pare our method with the following KE methods, including Fine-Tune (FT) (Meng et al. 2023).
MEND (Mitchell et al.) 2022a), ROME (Meng et al.] (022). MEMIT (Meng et al.] 2023)
SERAC Mitchell et al. 2022b), GRACE (Hartvigsen et al. [2023a), RECT (Gu et al., 2025), and AlphaEdit (Fang al. 2025). We provide detailed introduction on these baselines and models in Appendix B We provide the results including more L.MMs in Section J.10 Additionally, we
compare with the WISE (Wang et al. 20255) in Appendix J.1
Datasets for knowledge editing To evaluate KE methods, we utilize two widely recognized benchmarks: the CounterFact dataset (Meng et al. 2022) and the ZsRE dataset (Levy et al., 2017). Consistent with previous research (Meng et al. 2022 Fang et al. 2025). we employ the following evaluation metrics: efficacy (success of edited facts), g generalization (success of paraphrased facts), specificity (success of neighboring facts), fluency (generation entropy), and consistency (reference score). Detailed explanations of the datasets and metrics are provided in Appendix C
Datasets for general ability We assess the general capabilities of the edited LLMs using the fol- lowing typical datasets, SciQ /Welbl et al., 2017) (science question answering), MMLU (Hendrycks) al. 2021) (massive multitask language understanding). Commonsense QA (Talmor et al.) (019) (commonsense knowledge understanding), ARC Challenge (Clark et al., 2018) (challenge task re- quiring reasoning). WSC273 (Kocijan et al. 2019) (coreference resolution), Lambada (Paperno) ct al. (pr6)crredict the endings of text passages). Datasets such as SciQ, MMLU, and Common- sense QA primarily evaluate knowledge-basee based question answering, focusing on the models' ability to understand and retain factual information. In contrast, the ARC Challenge, WSC273, and Lam- bada are designed to assess capabilities beyond mere knowledge memory, such as reasoning and text generation. Detailed information regarding these datasets is provided in Appendix

---

Published as a conference paper at ICLR 2026
Comparison of NeuralDB on KE benchmarks. Pre-edited refers the original models prior to any edits. We evaluated the performance of editing 2,000 facts, with results for FT, MEND, InstructEdit, MELO, and ROME (sourced from Fang et al. (2025)) For 10,000 facts, the results for MEMIT, RECT, AlphaEdit, and NeuralDB are denoted using the arrow (->) notation. The best results for both 2,000 and 10,000 facts are highlighted in bold, respectively.
Table 2:
CounterFact
ZaRE
Model
Emicacy
Generalization"
Specificity
Flueneyt
Consisteney'
Kffiwayy
Generalization!
Specificity""
Pre-edited
7.9
10.6
89.5
635.2
24.1
37.0
36,3
31.9
FT
83.3
67.8
46.6
233.7
30.3
30.2
15.5
MEND
63.2
61.2
45.1
372.2
0.9
1.1
LLAMAS
34,0
ROME
64.4
61.4
49.4
449.1
2.0
1.8
MEMIT
63.5 0..5--66.44 63.4
62.8 96.6
$2.0 50.55
466.6 400
6.--66.5
36.7 0.1
32.9-
19.1 1.5
RECT
64.2-+ 60.0
62.5
58.9+ 51.2
502.8 399.1
12.9 1.6
86.8-+ 0.0
82.3++ 0.0
31,9 0.0
AlphaEdit
99.1 75.8
94.0 63.1
68.6 54.0
622.7
417.8
32.8 7.0
94.4-> 90.5
91.3-1 85.9
32.6 30.3
NeuraIDB
99.9- 99.2
86.6- 85.9
88.2+ 85.6
632,7-+ 631/02
32.9- 32.6
96.3- 95.9
92.0- 91.0
319-+ 31.8
25.8
FT
92.2
72.4
43.4
297.9
72.4
68.9
19.7
MEND
46.2
46.2
53.9
242.4
3.9
0.7
0.7
0.5
SERAC
GPT-
82.3
58.3
69.0
615.9
28.7
92.1
38.2
25.2
GRACE
96.5
50.1
74.4
620.6
31.6
96.5
0.4
24.8
ROME
57.5
54.2
52.1
589.4
36.4
54.7
281.5
36.5 5.1
30,9
RECT
98.8-
86.3-5
54.99
618.1->
25.4
AlphaEdit
99.8 91.6
96,3-> 79.6
76.2-+ 60.3
618.5 517.8
41.9-
99,7-> 94.2
95,9-1 86,1
28.8 22.5
NeuralDB
99.7 99.1
94.6 93.2
$0.0 75.7
619.8
620.0
41.4 41.3
99.2 98.2
95.9 95.0
27.5 27.0
Pre-odited
22.2
24.3
78.5
626.6
31.9
22.2
31.3
24.2
37.1
SERAC
GPT2-XL
72.3
58.2
$95.4
27.4
92.2
36.6
20.7
GRACE
98.9
50.1
620.2
28.5
91.3
1.6
27.6
ROME
$4.6
51.2
52.7
366.1
0.7
47.5
43.6
14.3
MEMIT
93.0-+ 58.5
83.3- 55.8
589-+ 56.1
481 496.2
23.2- 8.1
74,4+ 3.5
66.9 2.8
25,87+ 2.07
RECT
91.8 > 86.9
79.5 69.5
64.0 > 55.0
482.1 517.8
20.3 > 10.9
82.6 27.5
74.7 2/21
24.6 13.1
AlphuFditt
93.8-
65.6-9
584.0-3 $80.9
29.6
93.2--
43.5-+47.5
25.3->
NeuralDB
97.2
96.3-
91.0
5.2 THE EDITING EFFECTIVENESS OF POST-EDITED MODELS
We evaluated the performance after editing 2,000 facts and also included the results of 10,000 facts in parentheses for MEMIT, RECT, AlphaEddit, and NeuralDB. The results are presented in Table llnd and method demonstrates exceptional efficacy three important perspectives to demonstrate our advantages.
various scenarios. We present the following
NeuralDB has great generalization. The rephrased structure presents a significant challenge for the generalization metric, especially for memory-basee methods. Although NeuralDB is also classified as a memory-basee based method, our approach achieves nearly 90% accuracy, whereas SEARC and GRACE only achieve a success rate of almost 50%.
NeuralDB has few side effects. Editing methods must be precise and avoid potential side effects. As observed in Table g existing methods often exhibit low specificity metrics and their responses lack fluency. Our method effectively mitigates these side effects and achieves near results compared with pre-edited models in terms of both specificity and fluency. Additionally, our method ensures that the generated text maintains coherence with a high degree of consistency.
NeuralDB has superior scalability. In particular, when increasing the number of edited facts from 2,000 to 10,000, our method maintains nearly 99% of the results across all metrics and exhibits stable performance. In contrast, existing methods often experience significant degradation, with AlphaEdit preserving only 70% the results. When editing 10,000 facts, our method demonstrates comprehensive advantage across all metrics in datasets.

---

Published as a conference paper at ICLR 2026
(a) MMLU
(b) Sci0
(c) Commonseesse QA
1.5
Acurny
Huzeto
65U42:
Aguag:
022
0.-
EDCT
ECCT
FCCT
0.5
03.0
8000
0.5
2003
0.5
2000
30000
Edi: Num bar
Edit Number
Edit Number
(d) ARC Challenge
(0) Lambado
(1) WSC273
1.5
1.2
1.0
0.8
0.5
0.2
0.3
Accuracy
0.8
0.4
ASNEE:
MIPT
MUNIT
Raace Ducoo
4300
5000
30000
4003
0000
0000
12000
4000
6000
10000
Number
Figure 4: Results over general abilities after massive editing. NeuralDB evaluated against strong baselines (MEMIT, RECT, and AlphaFditt, with black dashed lines indicating random guessing baselines for multi-choice datasets. The results highlight NeuralDB's advantage: it preserves general capabilities with strong consistency edit numbers increase significantly.
5.3 THE GENERAL ABILITIES OF POST-EDITED MODELS
We assessed post-edited models with various configurations, evaluating their performance across 2,000, 4,000, 6,000, 8,000, and 10,000 facts, as depicted in Fig. 4 The evaluation was conducted on Im-evaluationn-arnes, (Gao et al. 2024). The results show that our method effectively edits large number of facts without compromising the general abilities of the models across various tasks. In contrast, existing L&E methods struggle with 4,000 facts editing and exhibit rapid decline in general abilities as the number of edited facts increases. Notably, these baseline methods achieve favorable results on the SciQ dataset, likely due to the dataset's content being well-represented in Wikipedia and thus captured by the sampled Kn- However, their performance deteriorates on other tasks, highlighting the limitations of relying on Wikipedia-sampler Ko. Our method, which directly incorporates the gated mechanism, offers more precise and effective approach compared to approximations derived from Wikipedia. For results on more models, please see Appendix[]
5.4 SCALING UP THE NUMBER OF EDITED FACTS INTO 100K
We further examine the scalability of the NeuralDB when applied to an extensive volume of knowl- edge. To obtain sufficiently large set of facts for this investigation, we utilized the training set of the ZsRE dataset for model editing. The results for the Llama3 8B (Instruct) model are presented in Table 3 These results demonstrate that the performance of NeuralDB remains highly stable as the number of edited facts increases from 10,000 to 100,000 with only marginal degradation ob- served. In evaluations of the model's general ability, we find that scaling up the number of edited facts to 100,000 did not harm the general ability performance and led to 0.7% improvement in the benchmark datasets. This underscores the superior scalability of our framework.
Table 3: Editing accuracy and the post-edited model's general performance of NeuralDB on Llama 3 (8B) when editing extremely large sets of facts.
Number of edits
Ok
10k
20k
30k
40k
50k
60k
70k
80k
90k
100k
Efficacy (t)
37.0
96.9
96.6
96.6
96.4
96.1
96.0
95.9
95.8
95.6
95.5
Generalization (t)
36.3
91.4
91.4
91.2
91.0
90.7
90.6
90.6
90.5
90.4
90.2
Specificity (1)
31.9
35.1
35.3
35.2
35.2
35.2
35.2
35.2
35.1
35.1
35.1
MMLUP (t)
56.2
56.2
56.2
56.2
56.2
56.2
56.2
56.9
56.9
56.9
56,9
Thh MMLU results are evaluated by the AlphaFdit project rather than the Im-evaluationn-aanes. Although they use different metrics, both sets of results reflect the general capabilities of the L.MSs.
8

---

Published as a conference paper at ICLR 2026
6 RELATED WORK
6.1 KNOWLEDGE EDITING THROUGH PARAMETER MODIFICATION
Locate-and-Edit The L&E paradigm is derived from casual trace experiments (Meng et al. 2022
suggesting that the factual memory of the Transformer models is primarily associated with the FFN layers (Geva et al. 202T). ROME (Meng et al. 2022 proposed to edit the factual memory of the models by modifying the parameter of the target FFN layer. MEMIT (Meng et al.) (2023) extends ROME to support multi-layer and batch editing versions, allowing the editing of thousands of factual knowledge. To address the challenge of post-edited models losing their general capabilities (Li et al.,
2024a; Hsuch et al. 2024), several solutions were proposed, including the dumping of sequential editing caches (Huang al. 2024), null space projection (Fang et al. 2025), and regularization of the weights (Gu et al. 2024b) and singular values (Ma et al. 2025).
Hypernetwork KE (De Cao al. 2021 trains a lightweight SiLSTM-MLP editor to convert single-sample gradient into a low-rank weight delta. MEND (Mitchell et al. (021) factorizes two- layer gradients into rank-1 I vectors and feeds them through a shared MLP, allowing memory-fruge batch edits. InstructEdit (Zhang et al. 2024a) utilizes complex prompt template so that gradients self-cluster, enabling diverse OOD tasks. All these approaches (Li al., 2025b) require additional
fine-tuning of the hypernetwork with large datasets, incurring considerable computational overhead.
6.2 KNOWLEDGG EDITING WITHOUT PARAMETER MODIFICATION
External memory module This line of work starts with SERAC (Mitceel et al. 2022c). which
augments frozen model with explicit retrieval memory. T-Patcher (Huang et al. 2023) injects
sparse "key-value-bia triplet into the final FFN layer. GRACE (Hartvigsen et al. (023b) stores
erroneous hidden states discrete keys in dynamic codebook whose values overwrite selected layers whenever the current activation falls inside an e -ball, enabling millisecond-scale. MELO Yu et al., 2024) uses hidden-state-inndxxe database activate low-rank, per-edit adapter blocks on demand. Although NeuralDB sharesa similar framework with memory- based, the main differ- ence using the FFN activation of the subject as the key, which provides stronger generalization and scalability. In Appendix E. we provide detailed discussion and key matching experiments compared to GRACE (Hartvigsen et al. 2023a) to explain our advantages. MEMOIR (Wang al.) 2025a) achieves strong lifelong editing by training sparse masks over side parameters and routing matching; in contrast, our approach targets large-batch
new queries via sparse activation-patter editing at scale with an explicit, in-layer KV module. RASE (Han et al. 2023) enhances T-Patcher and ROME by integrating fact retrieval; unlike our in-model retrieval from hidden states, RASE uses centence-embeddiee models for external retrieval before applying editing.
Prompt-based approaches Recent studies utilize prompt engineering to facilitate efficient KE. For example, MemPrompt (Madaan al., (022) and IKE (Zheng al., (023b) embed updated
knowledge into prompts to leverage in-context learning. For multi-hop QA tasks, MQUAKE (Zhong
al. 2023) and RippleEdits (Cohen al. 2024) introduces benchmark to evaluate multi-hop KE
performance. MeLLo (Zhong et al., (2223) stores edited facts externally and iteratively prompts
the model to yield answers consistent with the updates. PokeMQA (Gu et al.] 2024a) improves retrieval and answer accuracy by decomposing multi-hop questions via prompts. COMPKE (Cheng prompts.COMPKE
ctt al., 2025) further extends multi-hop knowledge editing to complex questions involving logical operations (Ynn et al.). These settings closely related to neural graph databases (Bai et al.) 2025 Fei et al.) 2025) and require complex query answering over knowledge graphs. RAE (Shi et al. 2024) retrieves refined facts and enhances the language model through in-context learning using a knowledge graph. To address multilingual KE, ReMaKE (Wang et al. 2024b) integrates newly retrieved multilingual knowledge into prompts.
7 CONCLUSION
In this paper, we introduce NeuralDB, a scalable knowledge editing framework designed to struct a neural KV database from edited facts and integrate it into the target FFN layer within LLMs using non-linear gated function. This integration ensures that the general capabilities of LLMs

---

Published as a conference paper at ICLR 2026
preserved. The neural database is designed to be easily maintained, facilitating efficient addi- tion and modification of edited facts within the models. We conducted comprehensive experiments across various LLMs to validate the effectiveness of our framework. Our results on the ZsRE and CounterFact datasets, utilizing GPT2-XI,, GPT-J (6B), and Llama-3 (8B), demonstrate that Neu- ralDB editing can effectively modify hundreds of thousands of facts without degrading the quality of generated text. Additionally, our findings from Six generic text understanding and generation tasks confirm that our method preserves the general abilities of LLMs unrelated the target edited facts. These results highlight the robustness and scalability of NeuralDB editing, positioning it as valuable tool for enhancing the adaptability and accuracy of LLMs in diverse applications.
ACKNOWLEDGMENTS
Jingzhao Zhang, Jiazheng Li and Jing Xu were supported by the Shanghai Qi Zhi Institute Innovation Program.
ETHICS STATEMENT
The wider impacts include positive contributions to society. By improving the efficiency of knowl- edge updates, models can adapt more quickly to changing environments and information. This not only benefits research and education, but also provides up-to-date information support in critical fields such as healthcare and justice, promoting scientific and timely decision making. Furthermore, the proliferation of knowledge editing will encourage interdisciplinary collaboration, allowing perts from different fields to share and integrate knowledge more effectively to address complex societal issues.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide the code in the supplementary materials and detailed imple- mentation information Appendix() In the README. md file included with the code, we present step-by-step guide for reproducing our results, which covers loading datasets, preparing the envi- ronment, executing the methods, and evaluating the outcomes.
REFERENCES
Jiaxin Bai, Zihao Wang, Yukun Zhou, Hang Yin, Weizhi Fei, Qi Hu, Zheye Deng, Jiayang Cheng, Tianshi Zheng, Hong Ting Tsang, et al. Top ten challenges towards agentic neural graph databases. arXiv preprint arXiv:2501.14224 2025.
Keyuan Cheng, Zijian Kan, Zhuoran Zhang, Muhammad Asif Ali, Lijie Hu, and Di Wang- COMPKE: Complex question answering under knowledge editing. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Associa- tion for Computational Linguisties: ACL 2025, pp. 2557-2576 Vienna, Austria, July 2025. As- doi: 10. 10.18653/v1/2055. findings-acle 130. URL metLhtttss//aalaatoolie org/0025..22222250 wte.:22....iiiinnoe findings-acl. 1307]
sociation for Computational Linguistics. ISBN 979-8 881---88976--5555
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.05577 2018
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics, 11:283-298, 2024.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge language models. arXiv preprint arXiv:2 104.0024 2021.
Junfeng Fang, Houcheng Jiang. Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained model editing for language models. In The Thirteenth International Conference on Learning Representations, 2025. URL hetps: dwWSSytvgg..
//openreviee ew. tet//forrmm?.dddlsyyyyyrrg
10

---

Published as a conference paper at ICLR 2026
Weizhi Fei, Zihao Wang, Hang Yin, Yang Duan, and Yangqiu Song. Extending complex logical queries on uncertain knowledge graphs. In Proceedings of the 63nd Annual Meeting of the Asso- ciation for Computational Linguistics (Volume I: Long Papers), pp. 13168-131922 2025.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. UtLhttt::///eeooooooo wtt.:oor//rrroo.. 12608602]
Yingqiang Ge, Wenyue Hua, Kai Mei, Juntao Tan, Shuyuan Xu, Zelong Li, Yongfeng Zhang, et al. Openagi: When Ilm meets domain experts. Advances in Neural Information Processing Systems, -5665, 2023.
Zorik Gekhman, Gal Yona, Roce Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning Ilms on new knowledge encourage hallucinations? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 7765-7784, 2024.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer fordaforward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference guage Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, November Associatin for Computational Linguisties. doi: 10. 101118661122221.cmmg main.446. URL attps::/aelaanthooooy..com orgp wttp:////...11pppp1....me.eeeee. emnlp-naaaaa. mappp-mainn... 446/
Empirical Methods in Natural Lan-
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Hengrui Gu, Kaixiong Zhou, Xiaotian Han, Ninghao Liu, Ruobing Wang, and Xin Wang. PokeMQA: Programmable knowledge editing for multi-hop question answering. In Lei-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8069-8083, Bangkok, Thailand, August 2024a. Association for Computational Linguisties. doi: 10. 10.18653/1/20222 acl-long 438. URL htttts::/aaaatthlllllaoooo
org/ aco-iong. 438/
Jia-Chen Gu, Hao-Xiang Xu, Jun-Ye Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. Model editing harms general abilities of large language models: Regularization to the res- cue. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 16801-16819 Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10.18653/ 11/2044emmll-main.23. URL
emn wwwnl-mmaaeeeeeeee main.
9347]
Xiaojie Gu, Guangxu Chen, Shuliang Liu, Jungang Li, Aiwei Liu, Sicheng Tao, Junyan Zhang, and Xuming Hu. Editing large language models via adaptive gradient guidance. In AAAI 2025 Workshop on Preventing and Detecting LLM Misinformation (PDLM), 2025.
Xiaoqi Han, Ru Li, Hongye Tan, Wang Yuanlong, Qinghua Chai, and Jeff Pan. Improving sequential model editing with fact retrieval. In Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 11209-11222 2023.
Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with GRACE: lifelong model editing with discrete key-value adaptors. In NeurlPS, 2023a.
Tom Hartvigsen, Swami Sankaranarayanan. Hamid Palangi. Yoon Kim, and Marzych Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 46:47934-47957 2023b.
11

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021. URL
net forummidrimi
d7KB 3Gm@.
Cheng-Hsun Hsueh, Paul Kuo-Ming Huang. Tzu-Han Lin, Che Wei Liao, Hung-Chieh Fang. Chao- Wei Huang, and Yun-Nung Chen. Editing the mind of giants: An in-depth exploration of pitfalls of knowledge editing in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Findings of the Association for Computational Linguisties: EMNLP 2024, pp- 1417-9429 Miami, Florida, USA, November 2024, Association for Computational Linguisties. doi: 10. 1188653331222224 120.55///22.....nnnnneemmmmmmmm findings-emmlp.500. URL https: lttttsss//cclantclooyy.oooo orgp//44.
Firdings emn emmlp. 5550
Xiusheng Huang, Jiaxiang Liu, Yequan Wang, and Kang Liu. Reasons and solutions for the de- cline in model performance after editing. In The eighth- Annual Conference on Neural Information Processing Systems, 2024. URL https: https///pppeeeeeeeeeeeeeeeeeeeee www.///e//oommd---
x xjXYgdFMMM
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. Transformer-patcher: One mistake worth one neuron. arXiv preprint arXiv:2301.0977 009785, 2023.
Vid Kocijan, Ana-Maria Cretu, Mana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewiez. A surprisingly robust trick for the Winograd schema challenge. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4837-4842 Florence, Italy, July 2019. Association for Com- putational Linguistics. doi: 10.18653/v 1/119///1/11-1145 URL
org//
P19- 319-14787
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In CoNLL, pp. 333 332.. Association for Computational Linguistics, 2017.
Jaang Li, Quan Wang, Zhongnan Wang, Yongdong Zhang, and Zhendong Mao. Elder: Enhancing lifelong model editing with mixture-of-lora. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 24440-24448, 2025a.
Qi Li, Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Xinglin Pan, and Xiaowen Chu. Should we really edit language models? on the evaluation of edited language models. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https: //openreview. wtw.foorrmmtmo/tmmm//m forum? d=m00S400m5) 400m5Y
Shuaike Li, Kai Zhang, Qi Liu, and Enhong Chen. Mindbridge: Scalable and cross-model knowl- edge editing via memory -uggeented modality. arXiv preprint arXiv:2503.02700 2025b.
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang. Jun Ma. and Jie Yu. PMET: precise model editing in a transformer. In AAAI, pp. 18564-18572. AAAI Press, 2024b.
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, arXiv preprint 08777 2023.
Jun-Yu Ma, Hong Wang, Hao-Xiang Xu, Zhen-Hua Ling, and Jia-Chen Gu. cntturbation-restainee sequential model editing. In The Thirteenth International Conference on Learning Representa- tions, 2025. URL hnttps://openreviee wet.ieewwwweeeeeeeeeeeeeeeeeeeee wet.:/ooorm.ddbfii....
Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memory-assisted prompt editing to improve GPT-3 after deployment. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp- 2833 2861, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10.18653/v 102.5////1222222222 1/w.t2..22ccc.2 1/26623/1/2222emmmlllp main. 183. URL https: h/tp///aaaaannoto/o//a/o hog.
202 2. emn 1p- 183/
12

---

Published as a conference paper at ICLR 2026
Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing factual asso- ciations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: oppenevvee.. net/forum? i d= h6wA666WA.6.
Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In ICLR. OpenReview.net, 2023.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning, Fast model editing at scale. arXiv preprint urXiv:21 1309, 2021.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing scale. In ICLR. OpenReview.net, 2022a.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. Memory- based model editing scale. In ICML, volume 162 of Proceedings of Machine Learning Re- search, pp. 15817-15831. PMLR, 2022b.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory- based model editing at scale. In International Conference on Machine Learning, pp. 15817- 15831. PMLR, 2022c.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring broad discourse context. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pp. 1525-1534, Berlin, Germany, August 2016. Association for Com- putational Linguistics. doi: 10.18653/v1/P166 11444114644 URL https: http://aaaaattooo....o.... org/ P16-1144/
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Yucheng Shi, Qiaoyu Tan, Xuansheng Wu, Shaochen Zhong, Kaixiong Zhou, and Ninghao Liu. Retrieval-enhanced knowledge editing in language models for multi-hop question answering. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM '24, pp. 2056-2066 New York, NY, USA, 2024. Association for Com-
puting Machinery, ISBN 9798400704369, doi: 10.1455/66277333377772.
URL https:
//doi.org//i 10. 110..114555 211444666673333772222
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQAA: A ques- tion answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Shori Pupers), pp. 4149-4158, Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10.18653/v1/N19-1466 URL
org/
Ben Wang. Mesh- TranfformrrreIA Model-Parallel Implementation of Transformer Language Model with JAX.
jax
May 2021.
Ke Wang, Yiming Qin, Nikolaos Dimitriadis, Alessandro Favero, and Pascal Frossard. Memoir: Lifelong model editing with minimal overwrite and informed retention for llms. arXiv preprint arXiv:2506.07889 2025a.
Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang. and Huajun Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. Advances in Neural Information Processing Systems, 37:53764-53797, 2025b.
Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. Knowledge editing for large language models: A survey. ACM Computing Surveys, 57(3):1-37, 2024a.
13

---

Published as a conference paper at ICLR 2026
Weixuan Wang, Barry Haddow, and Alexandra Birch. Retrieval-augmente multilingual knowledge editing. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 335-354, Bangkok, Thailand, August 2024h. Association for Computational Linguistics. doi:
10.186655/11/0044.cllloooo
URL https::/aclanthii wopphhoooy...ooo ogg. 2024 acnlloog..
217
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science ques- tions. In Leon Derezynski, Wei Xu, Alan Ritter, and Tim Baldwin (eds.), Proceedings of the 3rd Workshop Noisy User-generated Text, pp. 94-106, Copenhagen, Denmark, Septem- ber 2017. Association for Computational Linguisties. doi: 10. 10.18653/11/11744433. URL https: http:////aaaaathhhoacaao www.aanhhhoooyyoooo/1/44111
Hang Yin, Zihao Wang, and Yangqiu Song. Rethinking complex queries on knowledge graphs with neural link predictors. The Twelfth International Conference on Learning Representations.
Lang Yu, Qin Chen, Jie Zhou, and Liang He. Melo: Enhancing model editing with neuron-indexed dynamic lora. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp- 19449-19457, 2024.
Ningyu Zhang, Bozhong Tian, Siyuan Cheng, Xiaozhuan Liang, Yi Hu, Kouying Xue, Yanjie Gou, Xi Chen, and Huajun Chen. Instructedit: Instruction-based knowledge editing for large language models. arXiv preprini arXiv:2402. 2ee.22402.11111 16123, 2024a.
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang. Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. comprehensive study of knowledge editing for large language models. CoRR, abs/2401 1.101.01222 286, 2024b.
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? CoRR, abs/2305. 127400, 2023a.
Ce Zheng, Lei Li, Qingxiu Dong. Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can edit factual knowledge by in-context learning? In The 2023 Conference Empirical Methods in Natural Language Processing, 2023b. URL https: wwtp//ppnereee.eeoeeepeeee net, forum? ? di=
JOHAM8MY
Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https:
JBnnncc
Chenghao Zhu, Nuo Chen, Yufei Gao, Yunyi Zhang, Prayag Tiwari, and Benyou Wang. Is your llm outdated? evaluating llms at temporal generalization. arXiv preprint M.Xi::205.008466) 2024.
A THE USE OF LARGE LANGUAGE MODELS
The LLMs were used solely to refine the writing and check for potential typos. No other aspects were involved.
B BASELINES
In this section, we present the six baseline methods evaluated in our work. For each method, we adopt the default hyperparameter settings provided in the official code of the corresponding papers.
Fine-Tune (FT) Meng et al., 2023) is a fine-tuning method that updates the FFN a transformer layer to incorporate new factual knowledge. The target layer is selected on the basis of its relevance to the knowledge being edited. FT operates by maximizing the likelihood of the target output using the standard next-token prediction loss.
14

---

Published as a conference paper at ICLR 2026
MEND Mitchell et al. 2022a) introduces hypemetwork that maps fine-tuning gradients into efficient weight updates for pre-trained model. By applying low-rank decomposition to the gradients, it reduces parameter complexity and enables lightweight, localized edits without full model retraining.
ROME Meng et al. 2022) performs KE by interpreting the FFN in a transformer layer linear associative memory. It derives key-value pairs from internal activations and computes a weight update that ensures the edited layer produces the desired hidden repre- sentation. rank-one modification then applied to the FFN weights, aligning the model's internal representations with the new factual knowledge.
MEMIT (eng et al. 2023) extends the ROME framework to support simultaneous editing of large number of factual knowledge items. It models the updates as a joint optimization over key-value pairs and applies rank-one modifications to the FFNs. To prevent interfer- ence between edits, MEMIT distributes the updates in top-down manner across critical FFN layers, achieving efficient, scalable, and stable insertion of new factual knowledge.
RECT (Gu et al. 2025) reduces the degradation of general capabilities caused by KE. It regularizes weight updates by constraining their magnitude and selectively updates only the top-k:% of parameters with the largest changes during fine-tuning. This reduces overfit- ting and helps preserve the model's reasoning and question-answering answering abilities, while still achieving effective factual edits.
AlphaEdit (Fnng et al. 2025) introduces null-space projection mechanism to preserve existing knowledge during editing. projects the update direction onto the null space of prior knowledge and then applies the projected perturbation model parameters. This approach reduces interference with previously edited facts and enables effective integration of new information.
C DATASETS AND METRICS
In this section, we describe the datasets and evaluation metries employed our experiments.
C.I DATASETS
CounterFact and ZsRE for assessing KE, and six benchmarks including SciQ, MMLU, CommonsenseQA, ARC Challenge, WSC273, and LAM- BADA for evaluating the general capabilities of post-edited models.
We evaluate our methods on two types of datasets:
Counterfact (Meng al. 2022) challenging benchmark that focuses on editing in- correct factual statements in language models. Each instance includes a subject and an incorrect attribute be updated. To assess edit locality, it provides contrastive prompts involving related but distinct entities, ensuring changes do not affect nearby facts. Addi- tionally, the dataset includes paraphrased and semantically equivalent prompts evaluate the generalization, fluency, and consistency of the edited model responses.
ZsRE (Levy et al.) 2017) is a editing evaluation. Each example includes a subject, a target answer be edited, para- phrased questions for testing generalization, and unrelated questions for evaluating locality. The dataset also features numan-written question variants, enabling assessment of model robustness to semantically equivalent inputs.
dataset commonly used in knowledge
SciQ (Welbl al.) 2017) is multiple-choice science QA dataset covering topics such as physics, chemistry, and biology. It is used to evaluate model's ability to recall factual scientific knowledge.
MMLU (Hendrycks et al. 202T is multitask benchmark containing questions from 57 academic and professional disciplines. It assesses factual knowledge and generalization across diverse domains in zero- and few-shot settings.
CommonsenseQA (Talmor et al. 2019) is multiple-choiee QA dataset that evaluates model's ability to reason over everyday commonsense. focuses on applying implicit world knowledge select the correct answer among distractors.
15

---

Published as a conference paper at ICLR 2026
ARC Challenge (Clark al. 2018) is science QA dataset designed to require reasoning beyond simple retrieval. contains complex grade- level questions that challenge a model's problem-solving abilities.
WSC273 (Kocijan et al. 2019) is coreference resolution benchmark derived from the Winograd Schema Challenge.
designed to test whether a model can correctly identify what ambiguous pronouns refer to, based on context and commonsense reasoning.
Lambada (Paperno et al., 2016) is a word prediction benchmark composed of narrative passages where the final word can only be inferred from the entire context. It is designed to evaluate a model's s ability capture long-range dependencies and maintain discourse-level coherence.
C.2 METRICS
Here, we introduce the evaluation metrics for the CounterFact and ZsRE datasets, which are selected based on previous works (Meng al. 2022 Fang et al. 2025).
C.2.1
COUNTERFACT METRICS
Given a language model f. a query (s,, an edited object os and the original object o,, the evaluation metries for CounterFact are defined follows.
Efficacy (success of edited facts): The proportion of instances in which the model prefers the edited object o, over the original object o when prompted with (s,, rz):
E2 [Py [ā: (8;, (e)) f [o,
(8t, r;)]]
(11)
Generalization (success of paraphrased facts): The proportion of paraphrased prompts (sss,,))) representing rephrasings of the original query (si, for which the model as- signs higher likelihood to à than to o;;
E& , (ō: r(),) > p [ox F((8;, r:)]]
(12)
Specificity (success of neighborhood facts): The proportion of neighhorhood prompts N.(s,, which involve subjects semantically related to but distinct from the original subject s;, for which the model assigns higher likelihood the correct object o, than the edited object o;:
E, |Pr [o: Ne(s,, rz)] < P [o, N((si,r:)]]
(13)
Fluency (generation entropy): Measures output repetition based on the entropy of n-gram distributions in model outputs. Specifically, it computes a weighted combination of bigram and trigram entropies, where (-)(.) denotes the n-gram frequency distribution:
2
Σ (2k)) log, (k) - 3 Σ ga(k) log, gk()).
(14)
3
K
Consistency (reference score): Consistency is measured by prompting the model f with a subject and computing the cosine similarity between the TF-IDF vectors of its generated output and a reference Wikipedia passage about the object o.
C.2.2 ZSRE METRICS
Given a language model f. a query (si, r:), an edited object o, and the original object o,, the evaluation metrics for ZsRE are defined as follows:
Efficacy (success of edited facts): Top-1 accuracy the edited samples, measuring the proportion of cases in which the model ranks the edited object o, as the most likely predic- tion given the prompt (st,
E2 0 = arg max Py (o (8 re))
(15)
16

---

Published as a conference paper at ICLR 2026
Generalization (success of paraphrased facts): Top-1 accuracy on paraphrased prompts F.(s,,r)) which are rephrasings of the original query (s,. re), measuring the proportion of cases in which the model ranks the edited object o the most likely prediction for the given rephrased prompt:
E2 0 = arg max (o Fi(ss,
(16)
Specificity (success of neighborhood facts): Top-1 accuracy on neighborhood prompts rz), which involve subjects related but distinet from 8;- Specificity reflects whether the model preserves correct predictions on unaffected inputs by still preferring o, over o;:
E2
arg max ( (o Ni(ss, r:))
(17)
D IMPLEMENTATION DETAILS
We provide the details of the implementation of the experiments. To reproduce our methods, GPU with 40G memory required. Our framework built on L&F methods like MEMIT (Meng ct al. 2023) and AlphaEdit (Fang et al. 2025). We first sequentially compute the key vector k and the residual vector r for the target edited facts. The details the computation can found in stack them the key matrix K₁ and the residual matrix R₁ and construct a neural KV database 1K, R1). Then we integrate the non-linear retrieval module with the target FFN layer !* Our module only involves one hyperparameter y to control the gated mechanisms. We provide the details of the setting in the following Table 4 For the setting 7, we recommend y [0.6, 0.8] and provide the detailed ablation study Appendix [ For the distance function, We use cosine similarity as the distance function because it has closed range of [0, 1]. across all edited facts, thereby eliminating the need for complex key merging. Other distance functions can make difficult to determine a suitable value for 7.
Appendix F Then
Table 4:
of NeuralDB for various models in the main experiments
GPT2-xl
GPT-J (6B)
Llama 3 Instruct (8B)
Layer found by casual trace Layer 1" used by NeuralDB
17
17
17
17
8
7
gating threshold y
0.65
0.65
0.65
E COMPARATIVE ANALYSIS WITH GRACE
Both GRACE and NeuralDB implement key-value (KV) database structure for knowledge editing, design prevalent in memory-based methods due to its elegant modeling. Despite this architectural similarity, our empirical results demonstrate that NeuralDB achieves significantly superior editing performance in large-scale knowledge editing scenarios, particularly in generalization metrics. To systematically investigate the source of this advantage, we first identify the core architectural differ- ences (Table and perform key matching experiments (Table b).
Both methods can be formally represented as KV databases:
h' = Editing ( if (distance(k: K₃+) < 7 where i" arg min distance(h'-1 K,)
where distance is one distance function (e.g. L2, cosine similarity) and 7 is the threshold for distance function.
The critical distinctions lie in two aspects: (1) NeuralDB utilizes the last token of the subject as the anchor token, which provides unique feature representation and ensures scalability, whereas GRACE uses the last token of the question; (2) NeuralDB selects keys based activations in the
17

---

Published as a conference paper at ICLR 2026
Table 5: Architectural differences between GRACE and NeuralDB (ours). Illustrated editing the fact "The latest World Cup was held in Qatar."
Component
NeuralDB
GRACE
Anchor Token Position Key Selection Mechanism
Last token of subject (e.g., Cup) Activation FFN layer
Last token of question (e.g., is) Hidden state of given layer
Table 6: Key matching performance comparison under 100 facts
Rewrite Matched
Rewrite Unmatched
Rephrased Matched
Rephrased Unmatched
Neighborhood Unmatched
Method
NeuralDB (Cosine f)
1.00
0.18
0.84
0.19
0.18
GRACE (Cosine t)
1.00
0.25
0.44
0.44
0.45
GRACE (L2 !)
1.00
3968
781
768
948
FFN layer, which have been empirically verified crucial for factual knowledge memorization and provide better generalization capabilities compared to GRACE's hidden state approach.
To empirically validate why our KV structure outperforms GRACE's, we conducted key matching experiments. The results (Ta demonstrate that NeuralDB achieves superior key discrimination:
For NeuraIDB:
Rewrite Matched (1.00) > Rewrite Unmatched (0.18) and Rephrased Matched (0.84) >> Rephrased Unmatched (0.19), indicating effective key discrimination even for rephrased queries.
For GRACE: Rephrased Matched (0.44) Rephrased Unmatched (0.44) in cosine simi- larity, and even Rephrased Matched (781) > Rephrased Unmatched (768) in L2 distance, revealing its inability to distinguish matched facts for rephrased queries.
The results show that NeuralDB can easily distinguish editing keys from unrelated keys, while GRACE struggles in this regard. This empirical evidence directly explains the generalization per- formance gap observed in Table 2 (NeuralDB: 86.60 vs. GRACE: 58.58). This confirms that Neu- ralDB's key discrimination mechanism offers significantly better generalization capabilities com- pared to GRACE and other memory-based methods.
F COMPUTATION OF KEY VECTOR k; AND VALUE VECTOR I;
We follow previous derive the key vector and residual vector from the given edited fact (s,, F; 0 O,). Let denote the FFN layer to be updated.
methods Meng et al. 2022
2023 Fang et al. 2025) to
For the key vector k, retrieve the specified activation from LLM inferring the prompt. We denote k" (x) as the key activation of the prompt x in layer (". Then the target key vector are computed by the following average over random prefix Xy:
N k" (x, s:): 1=1
k;
Σk'
(18)
N
where s; is the subject of edited fact. The x, is the prefix randomly generated by the language model improve the robustness of the expressive ability of k,
For the target vector r,, we wish to find some vector to decode the new answer o; We utilize the supervised derive r, arg min, L(r), where the loss object L(r) is defined as following:
1 N Σ (-log Faa" +=r) [00| + p) + DEL (IP f( +-r) [<13)]1)
(19)
p is the factual prompt while p' is its variant ( the form of "subject is a"). f(h" + r) indicates substituting the output the i-th MLP with an additional learnable parameter r. This optimization also uses the random prefix text xy enhance the robustness.
18

---

Published as a conference paper at ICLR 2026
G DERIVATION OF SOLUTION TO MEMIT AND ALPHAEDIT
The objective of MEMIT is the following constrained least squares optimization:
arg min II(W + AJK, V1ll 1112 + 31 A 1
(20)
where the term || 1 ensures that the updated parameters maintain general knowledge. With residual matrices R V1 WK1, the closed- solution for MEMIT can be derived as follows:
MEMIT upd
R. (K 8 K,)
(21)
Let $1 = (K,K] + K -1 then the update can be expressed as A MEMIT R2 R,KYS,.
Using SVD decomposition U, U,S, S, U SVD(K) Ko), the null space be obtained by the sub matrix N = U S < by removing the eigenvectors corresponding to non-zero eigenvalues, where typically set as 10 Let P NN', where PK, N(NTK,) 0.
Similarly, we discuss AlphaEdit, which leverages the null space projection to convert soft constraints for general knowledge into hard constraints. The null space projection matrix P such that PK, is computed using SVD decomposition over K. K, Then, AlphaEdit constructs A = oP such that ||al approaches zeros and solves the following least squares problem with L.2 Norm:
arg min [I(W 8P)K V112 + B21
(22)
H ADDITIONAL MEMORY USAGE AND COMPUTATION TIME
In this section, provide detailed discussion of additional resources of our new module.
Memory usage We cache the key matrix K, and the residual matrix R, and construct the new module, which totally take (d, + d2) m parameters with denoting the number of edited facts. For Llama 3 8B model with dj = 14, 336, d2 = 4, 096, the memory of 10,000 facts is about 150 million parameters. Compared the total 8B parameters, the additional memory for IM facts is only 2.2%. Additionally, our memory grows linearly with the facts and is easily scaled to more facts.
Computation time We report the average evaluation time for three models and two datasets in Table The results show that the averaged time has only a slight improvement compared with the methods without an additional module.
Table 7: The averaged time of evaluation post-edited models CounterFact and ZsRE
Model
Llama3
GPTJ-6B
Method
MEMIT
AlphaEdit
NeuralDB
MEMIT
AlphaEdit
NeuralDB
CounterFact ZsRE
4.12
4.11
4.18 0.22
3.81
3.76
3.90
0.22
0.22
0.16
0.16
0.17
I ABLATION STUDY
I.1 LAYER SETTING: SINGLE-LAYER OR MULTI-LAYER?
We provide the pseudo code of the new multi-layer and the old multi-layer strategics in Algorithm 2 and Algorithm respectively. New multi-layer strategy assigns parts of facts each layer and
19

---

The results clearly demonstrate our improvements and further highlight our Published as a conference paper at ICLR 2026
Algorithm Old multi-layer method
Require: Input Transformer model f. target layers list L -4,,-.. Ln]. request facts F, Function COMPUTE -EY to compute the keys of facts at layer 1, COOMPUEE.EESDDUA compute the residual of facts layer L.
1: R f- COMPUTE RESIDUAL (f, (F
2: for in I. do
3: K; (OMPUTE.KEY(J
4: R2 + R/(,, - 1I+1)
5: Perform KE at layer with (K,, R&)
6: end for
Algorithm New multi luyti method
Require: Input Transformer model f. target layers list L = I1..... Inl. request facts F, Function COMPUTE KEY to compute the keys of facts layer
compute the
residual of facts at layer I.
1: for in L do
2: K, COMPUTE.KEY( FI,,))
3: Rg + COMPUTE RESIDUAL(f,F,0
4: Perform KE at layer with (K;, R4)
5: end for
update the key-value pair for these layers separately. The edited layers by new multi-layer strategy will disturb each other, resulting in poor performance. The compassion of these two algorithms on two different pre-traings pre-trained models Table The experimental results indicate that the new multilayer method can greatly scale the number of editable facts, albeit with some loss in performance, while old multilayer method, though achieving better editing accuracy, requires substantially more storage.
Furthermore, to ensure fairness, we provide comparisons with the single-layer versions of AlphaEdit and MEMIT in Table 9 advantages.
I.2 LAYER SELECTION
For the L in Llama3, we provide the results of additional 8 and 9 in Table TO Although layer determined by the causal trace, our results show that its results are not suboptimal.
1.3 GATING HYPERPARAMETER y SELECTION
We conducted an ablation study investigate the choice of 7 and the target layer, with results provided in Table 11 Although all the evaluation metrics are usually monotonically changed as increases, we observe consistent trend across three models. = 0.65 provides good balance across metrics for three models. Importantly, the generalization metric is monotonic increasing, while the specificity metric monotonic decreasing. Extremely low or high y substantially degrades some metrics. Therefore, we recommend selecting in [0.55, 0.65], which empirically performs good performance for all the metrics and three models.
ADDITIONAL EXPERIMENTS
J.1 THE COMPARISON WITH WISE
20

---

Published as a conference paper at ICLR 2026
Table 8: Editing performance under different layer setup
Model
Layer Setup
Effccccy
Generalization
Specificity
Fluency
[8] baseline
99.08
93,48
75.52
620.53
GPT-J
[6.7.8] new multi layers
94.44
91.72
75.93
617.44
[6,7.8] old multi layers
99.31
93.23
76.78
616.00
[17] baseline
99.04
95.96
70.72
621.90
GPT2-XL
[15,16,17] new multi layers
94.81
92.68
70.26
618.51
[15,16.17] old multi layers
99.08
94.01
71.33
624.48
Table 9: The comparison with single-layer versions of MEMIT and AlphaEdit, In each value a/b, the first is the result of 2k while the second corresponds to 10k.
Method
Efficacy Counteract
Generalize
Specifieity Counteract
Fiuency Counteract
Consistency
Efficacy ZsRE
Generalize Z:RE
Specificity ZsRE
Counteract
Counteract
MEMIT
69 72
59/61
52 46
519 / 522
6/6
68/0
63/0
27/1
AlphaFdir 97 88
83/73
69 53
623 / 563
32/ 29
93/88
88
2215/31
NeuralDB
190/99
86
33/33
96
92
32
implement both WISE and NeuralDB using EasyEditor, popular and useful code repository for knowledge editing. For faimess, we use the default parameters of WISE provided EasyEditor (WISE released through EasyEditor).
We provide the comparison with WISE in the Table 12
WISE achieves consistent results across all efficacy, generalization, and locality metrics. However, our method demonstrates superior performance on large-scale edits, as evidenced by its results with 2,000 and 5,000 facts.
J.2 THE COMPARISON WITH T-PATCHER
We have included the comparison with T-Patch in Table 13 where NeuralDB demonstrates its ad- vantages across models and two metries.
J.3 THE RESULTS ON MQUAKE
The results on MQuAKE are shown in Table 14 Our method outperforms competing baselines and achieves the best overall performance. Compared with AlphaEdit, delivers a 36% improvement. These results indicate that complex editing tasks requiring intermediate steps also benefit from our approach.
J.4 THE RESULTS ON KNOWEDIT
We provide the results on KnowEdit (Wiki Recent) and KnowEdit (Wiki Counterfact), both of which include the Portability metric (See Table 15). Our method achieves the best Portability on Wiki Counterfact and comparable Portability the strongest baselines on Wiki Recent, indicating that the edited knowledge is effectively utilized in downstream reasoning.
J.S QUANTITATIVE AND QUALITATIVE RESULTS FOR DELETE OPERATION
The knowledge updating linked with the KV database. Thus, we can modify and delete the knowledge based on the modification and deletion of the content of the KV database.
21

---

Published as a conference paper at ICLR 2026
Table 10: Ablation study on Llama 3 (8B)
Gamma
Layer
R
G
S
F
C
0.65
99.2
85.9
85.6
631.9
32.6
0.65
8
99.2
79.3
85.1
631.5
33.3
0.65
9
99.2
77.4
84.9
631.6
32.4
Table 11: Unified y ablation across three models. Metrics: (Efficacyt), f), G (Gencralizationt). (Specificityt), (Fluencyf).
Model
Gamma
E
G
S
F
0.15
84.10
78.95
56,20
611.00
0.40
97.55
97.02
63.92
600.47
0.55
99.75
98.22
72.55
620.01
GPTJ-6B
0.65
99.80
97.20
74.10
621.50
0.75
99.75
93.35
75.00
621.75
0.90
98.55
57.38
76.68
621.35
0.15
94.85
90.30
58.58
587.88
0.40
99.70
98.38
65.66
611.20
0.55
99.65
98.10
76.62
619.14
GPT2-XI.
0.65
99.80
94.60
80.00
619.80
0.75
99.45
81.42
81.34
620.38
0.90
96.70
30.00
82.57
620.30
0.15
93.3
86.0
70.3
632.5
0.40
98.8
95.0
75.7
640.3
0.55
99.1
91.9
83.4
631.6
Llama-3 (8B)
0.65
99.2
87.0
85.6
632.0
0.75
99.2
74.1
86.2
632.3
0.90
98.8
28.7
86.9
633.2
Setting. We first edit 100 facts GPT2-XL, then delete the keys and values for the even-indexed facts. This yields three models: (i) pre-edited, (ii) post-edited, and (iii) deleted-even (after removing edits for even facts only).
Quantitative. As shown in Table 16] performance on even facts for the deleted-even model matches the pre-edited model, while performance on odd facts is similar to the post-edited model. This demonstrates selective reversibility (deletions roll back only the targeted facts) and retention (non-deleted facts remain effective).
Qualitative. We also provide concrete edit-delete-even case study, where the 4th question "Autonomous University of Madrid, which is located in" and the corresponding target answer "Sweden".. We report the answers of three models in Table [7 The answers to deleted-even model successfully revert the original answer to the pre-edited model.
J.6 THE COMPARISON BETWEEN ONE-SHOO EDITING AND BATCHED SEQUENTIAL UPDATE FOR MEMIT AND ALPHAEDIT
We follow the standard procedures for each editing method (Fang et al. 2025 For fair compar- ison, we also report one-shot results which MEMIT and AlphaEdit apply all factual edits in one go. As shown in Table [8| their one-shot performance is not better than the results obtained with batched sequential updates. Therefore, reporting results under batched sequential updates a fair evaluation setting.
22

---

Published as a conference paper at ICLR 2026
Table 12: 7B chat hf.
The comparison with WISE ZsRE. We provide the results Llama3 8B and Llama2
Models
Methods
Efficacy
Generalization
Locality
Efficacy
Generalization
Locality
Llama3 8B
WISE NeuralDB
33.2
32.8
100
25
24.7
100
67.4
100
70.7
67.3
100
WISE
62.8
59.8
100
51.7
49.8
100
Llama2 7B
NeuralDB
70.7
66.4
100
69.5
65.6
100
Table 13: Results on ZsRE with Ik edits (T-Patch results taken from Li et al.
2025a)).
Models
Method
Reliability
Generalization
GPT2-XL
T-Patch
77.29
67.74
GPT2-XL
NeuraIDB
96.03
91.62
Llama2
T-Patch
62.94
48.37
Llama2
NeuralDB
99.89
91.35
J.7 THE RESULTS OF HIGHLIGHHINGG THE PREVIOUS EDITED FACTS
Adopting the notation of RASE Han et al. 2023 the reported result is the stricter Edit Re- tain Rate (ER) instead of the Success Rate (SR): where SR Ei ((f((z)==e)) and ER T
ET (.fr(a))=Vo.). Here, Ji is the model after the t-th edit, and fr is the final model after all '
edits.
ER emphasizes whether earlier edits remain effective under the final model, making it stricter than SR. Our strong ER results (Tables and B) indicate that earlier edits are retained after subsequent edits. To further support this, Table 19 reports performance on the first 10k edited facts after editing 10k, 20k, 30k, 40k, and 50k facts; the stability across these settings highlights the effectiveness of previously edited entries.
J.8 LM-EVALUAAIION SAAANLUUATTON-HRDEESSS
We conducted more experiments on eevaluation-hardneee using GPT2-XL and GPT-J, see Table 21 for details.
J.9 WEIGHTED SCORE VISUALIZATION OF PARAPHRASED AND NEIGHRORHOOD FACTS
We further conduct experiments on paraphrased and neighborhood facts to examine the distribution of weighted scores under both MEMIT and AlphaEdit. As shown in Fig. the scores for positive samples in paraphrased facts are consistently higher, while those for negative samples remain close to O. For neighborhood facts, where all components are considered negative, the scores are likewise consistently close to O. These results confirm that, during inference, residuals unrelated to the edited facts remain inactive, resulting in near- zero weighted scores.
J.10 THE RESULTS OF MORE MODELS
We present the results of our method using Qwen2.5 and Llama 3.1 8B Instruct in Table 22 Our methods on these new models also achieve good results.
23

---

Published as a conference paper at ICLR 2026
Table 14: The results of MquaKE on GPTJ-6B models under CoT prompting. We follow the stan- dard evaluation protocol with CoT prompting. After applying all 3,000 edits, we evaluate all 3,000 multi-hop questions.
Methods
MEMIT
AlphaEdit
NeuraIDB
Accuracy
6.13
9.14
12.40
Table 15: Additional results on KnowEdit (Wiki Recent) and KnowEdit (Wiki Counterfact).
Dataset
Method
Rewrite acc
Paraphrase ace
Portability
Entropy
KnowEdit (Wiki Recent)
MEMIT
87.96
62.94
57.19
615.10 10
KnowEdit (Wiki Recent)
AlphaEdit
90.41
71.53
59.89
606.84
KnowEdit (Wiki Recent)
NeuralDB
97.90
84.23
57.11
607.87
KnowEdit (Wiki Counterfact)
MEMIT
69.22
48.67
45,89
615.23
KnowEdit (Wiki Counterfact)
AlphaEdit
73.05
50.81
44.11
616.13
KnowEdit (WikiCo unterfact)
NeuralDB
96.58
73.15
53.98
610.82
Table 16: Quantitative results before editing, after editing, and after deleting even edits.
Group
Metric
Pre-edited
Post-edited
Delete even edits
Odd-indexed facts
Efficacy
0.0
100.0
100.0
Odd-indexed facts
Generalization
22.0
98.0
98.0
Even-indexer facts
Efficacy
0.0
100.0
0.0
Even-indexed facts
Generalization
24.0
99.0
22.0
Table 17: Qualitative examples hefore editing, after editing, and after deleting even-indexed edits.
State
Output
pre-edited
"the city of Madrid, Spain" "Sweden, has been working on a project called "The Future of the Human Body" for the past two years." "the city of Madrid, Spain"
Post-edited
deleted-even
Table 18: The comparison of one-shot editing and batched sequential updates. The number of edited facts 10k, and the used model Llama3 8B Instruct. The batch size of the batched update is 100.
Model
Method
Efficacy
Generalization
Specificity
AlphaEdit
one-shot editing
88.09
82.78
31.10
AlphaEdit
batched sequential updates
90.50
85,90
30.30
MEMIT
one-shot editing
0.19
0.15
0.92
MEMIT
batched sequential updates
0.10
0.10
1.50
Table 19: Performance on the first 10k edited facts after scaling to different totals.
Edited facts
10k
20k
30k
40k
50k
Efficacy f
96.9
96.6
96.6
96.5
96.2
Generalization
91.4
91.4
91.2
91.3
90.8
Specificity 1
35.1
35.1
35.2
35.3
35.3
24

---

Published as a conference paper at ICLR 2026
Table 20: Performance of each task across different editing budgets (1,000, 2,000, 4,000, 6,000,
8,000, 10,000) under various model-algorithm configurations.
(a) GPT-J (AlphaEdit)
Task
1,000
2,000
4,000
6,000
8,000
10,000
sciq
0,9110
0,9080
0.9060
0.8900
logig
0,2151
0,2243
0.2089
0.2181
0.2304
0,2181
commonseuscc_a
0.2080
0.2146
0.2048
0.1884
0.1925
0.1785
easy
0.6658
0.6477
0.6326
0.6010
0,5804
0,4870
MMLU
0.2660
0.2688
0.2622
0.2592
0.2587
0.2535
mre_challenge
0.3276
0.3148
0.2901
0.2782
0,261
0.2261
lambuda
0.6722
0.6604
0.6057
0.5158
0.4036
0,2203
winogrande
0.6346
0.5991
wsc273
0,3425
0.8352
0.7985
0.7399
0,7179
0,6264
(b) GPT-J (NeuralDB)
Task
1,000
2,000
4,000
6,000
8,000
10,000
sciq
0,9160
0,9160
0.9160
0.9160
logiq_a
0,2120
0.2120
0.2120
0.2120
0.2120
0,2120
commonsemsx..p
0.2080
0.2080
0.2080
0.2080
0,2080
0,2080
easy
0.6692
0.6892
0.6692
0.6692
0.6692
0.6692
MMLU
0,2695
0.2697
0.2697
0.2695
0,2695
0,2698
are.challenge
0.3396
0.3396
0.3404
0.3404
0.3404
0.3404
lambuda
0.6829
0.6827
0.6827
0.6821
0.6821
0.6819
winogrande
0.6417
wsc273
0,8242
0.8242
0.8242
0.8242
0,8242
0,8242
(c) GPT-2 XL (AlphaEdit)
Task
1,000
2,000
4,000
6,000
8,000
sciq
0,8250
0.8230
0.7920
0.7440
logiq
0,2289
0.2273
0.2012
0,2104
0.1951
0.1889
commonnseneee uu
0.1908
0.1957
0.1916
0.1974
0,2080
0.1933
arc_easy
0.5682
0.5484
0.4987
0.4693
0.4066
0.3493
MMLU
0,2618
0.2562
0.2464
0.2312
0.2369
0,2315
challenge
0.2423
lambada
0.4881
0.4170
0.2610
0.1467
0.0767
0.0231
winogrande
0.5904
0.5549
0.5564
0.5272
0,5201
0,5067
wsc273
0.6520
0.6227
0.5714
0.5861
0,5678
0,5421
(d) GPT-2 XL (NeuralDB)
1,000
2,000
4,000
sciq
0,8240
0.8290
0.8280
0.8280
0.8280
0.8280
logiq
0,2212
0.2181
0.2181
0.2181
0.2181
0.2197
commonsense qa
0.1900
0.1933
0.144
0.1941
0.1941
0.1941
arc_casy
0.5770
0.5785
0.5848
0.5848
0.5848
0.5848
MMLU
0,2532
0.2545
0.2547
0.2546
0.2543
0,2544
challenge
0.2509
0.2509
0.2491
lambada
0.5055
0,5053
0.5077
0.5069
0.5065
0,5053
winogramde
0.5770
0.5785
0.5848
0.5848
0.5848
0.5848
wsc273
0.6777
0.6667
0.6850
0.6850
0.6850
0.6850
25

---

Published as a conference paper at ICLR 2026
Table 21: Performance of each task across different editing budgets (1,000, 2,000, 4,000, 6,000,
8,000, 10,000) under various model-algee model-algorithm configurations.
(a) GPT-J (AlphaEdit)
1,000
2,000
4,000
sciq
0.9110
0.9080
0.9060
0.8900
0.8850
0.7410
logiq_a
0.2151
0.2243
0.2089
0.2181
0.2304
0.2181
commonseensccqu
0,2086
0.2146
0.2048
0.1884
0.1925
0.1785
arc.caxy
0.6658
0.6477
0.6326
MML.U
0,2666
0,2688
0.2622
0.2592
0.2587
0,2535
are.challenge
0.3276
0.3148
0.2901
0.2782
0,261
0,2261
lambada
0.6722
0.6604
0.6057
0.5158
0,4036
0,2203
winogramde
0.6346
0.6227
0.6093
0.5991
0.5730
0.5635
wsc273
0.8425
0.8352
0.7985
0.7399
0.7179
0.6264
(b) GPT-J (NeuraIDB)
Task
1,000
2,000
4,000
6,000
8,000
10,000
sciq
0.9160
0.9160
0.9160
0.9160
0.9160
0.9160
logiq_a
0.2120
0,2120
0.2120
0.2120
0.2120
0.2120
commonsensc qa
0,2080
0.2080
0.2080
0.2080
0.2080
0,2080
are casy
0.6692
0.6692
0.6692
0.6692
0.6692
0.6692
MMLU
0,2695
0.2697
0.2697
0.2695
0,2695
0.2698
accallele
0.3396
0.3396
0.3404
0.3404
0.3404
0.3404
lambuda
0.6829
0.6827
0.6827
0.6821
0.6821
0.6819
winogrande
0.6409
0.6417
0.6417
0.6417
0.6417
0.6409
wsc273
0.8242
0.8242
0.8242
0.8242
0.8242
0.8242
(c) GPT-2 XI. (AlphaFdit)
Tisk
1,000
2,000
4,000
6,000
8,000
10,000
sciq
0,8250
0.8230
0.7920
0.7440
0.6390
0.4920
logiq_a
0.2289
0.2273
0.2012
commonsense.gp
0.1908
0.1957
0.1916
0.1974
0,2080
0,1933
arc arc_casy
0,5682
0.5484
0.4987
0,4066
0,3493
MMLU 0,2618
0,2562
0.2464
0.2312
0,2369
0,2315
are challenge
0.2423
0.2346
0.2398
0.2108
0.1887
0,2065
lambada
0.4881
0.4170
0.2610
0.1467
0.0767
0,0231
winogrande
0,5904
0.5549
0.5564
0.5272
0.5201
0.5067
wsc273
0.6520
0.6227
(d) GPT-2 XL (NeuraIDB)
Tisk
1,000
2,000
4,000
6,000
8,000
10,000
sciq
0,8240
0.8290
0.8280
0.8280
0.8280
0.8280
logiq
0,2212
0.2181
0.2181
0.2181
0.2181
commonsenscc.ga
0,1900
0,1933
0.1941
0.1941
0,1941
0,1941
arc.essy
0.5770
0.5785
0.5848
0.5848
0.5848
0,5848
MMLU
0,2532
0.2545
0.2547
0.2546
0.2543
0,2544
are orc_challenge
0.2509
0.2509
0.2491
0.2500
0.2500
0.2517
lambada
0.5055
0,5053
0.5077
0.5069
0.5065
0,5053
winogrande
0.5770
0.5785
0,5848
0.5848
0.5848
0.5848
wsc273
0.6777
0.6667
6850
0. 0.6850
0.6850
0.6850
Table 22: The results of NeuralDB on Qwen2.5 and Llama 3.1 8B Instruct. We provide the 2000 and 5000 facts ZsRE and CounterFact.
Model Metrics
Edit Number
Counterfact Specificity
ZsRE Generalization
Efficacy
Generalization
Fluency
Consistency
Efficacy
Specificity
Owen2.5
2000
99.15
90.72
81.69
625.70
33.21
99.69
91.82
38.4
Owen2.5 5
10000
98.99
89.85
81.97
625.35
33.08
99.15
91.79
38.27
Llama 3.1
2000
99.99 30
91.60
87.92
634.13
34.01
95.96
87.11
30,91
10000
33.69
26

---

Published as a conference paper at ICLR 2026
The weight scones of GPT2
The weight scores of GPT)
The weight scores Uamas 88
10
14
10
0.5
0.5
Q.
0.6
6.5
0.6
0.4
64
0.4
02
63
02
0.0
64
4.
MoNfl
Aptuubil
Altvraki
MaM
MBMi
Akrubakk
Asfrebl
MENI
Aplibbi
Abr u2ek
(a) Paraphrased facts
The weight scores of GPT2
The weight scores of GPT
The weight scores Uarra--8
1.0
14
L0
2.5
0.5
0.5
0.6
0.
0.5
0.4
644
0.4
02
03
0.
0.
64
0.0
Abnazek
MENI
Mchmi
(b) Neighborhood facts
Figure Visualization of weighted scores for paraphrased facts and neighborhood facts, using MEMIT and AlphaEdit across three models. The boxplots are generated from the mean and variance of weight scores, with the center line indicating the mean, boxes showing +1 standard deviation, and whiskers +1.5.
27


Reviews:
[{"review_id": "GcMVYd1TvZ", "reviewer": "Reviewer_HArh", "summary": "This paper tackles a key limitation of existing Locate-and-Edit (L&E) knowledge editing (KE) methods: their failure to scale to thousands of facts without causing catastrophic forgetting and degrading the model's general abilities. NeuralDB2 scales knowledge editing to 100 k facts by replacing implicit linear L&E updates with an explicit gated neural KV memory, curbing catastrophic forgetting and preserving model competence. Finally the paper demonstrates impressive experimental results, showing NeuralDB scales to 10,000 and even 100,000 facts while maintaining high editing success and preserving general model performance.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. **Fresh Perspective with Solid Grounding**: The paper cleverly rethinks existing linear editing methods like MEMIT and AlphaEdit as lookups in a hidden key-value store—backed up by both theory and experiments that show the updates behave almost like one-hot vectors.\n\n2. **Smart, Simple Fix**: Once they spot that sparsity, they swap out the cramped linear system for an explicit neural KV database. A lightweight, cosine-based gate fetches the right “patch” or returns zero if nothing matches—clean and intuitive.\n\n3. **Scales Like a Champ**: NeuralDB keeps its accuracy even after 10 k edits and still protects the model’s general skills, something earlier SOTA methods couldn’t do; they even push the demo to 100 k facts without breaking a sweat.", "weaknesses": "**1. On the Sensitivity and Selection of the Gating Hyperparameter $\\gamma$**\n\nMy primary concern is the selection of the gating threshold $\\gamma$, which is the single most critical hyperparameter for preserving general abilities. In the main experiments, the authors use a single value ($\\gamma=0.65$) across three different models (GPT2-XL, GPT-J, and Llama-3). This appears to be an unsubstantiated, convenient assumption. This concern is amplified by the ablation study in Appendix I.2 (Table 11), which was *only* conducted on Llama-3. This study reveals that the model's performance is *extremely sensitive* to this parameter. For instance:\nIncreasing $\\gamma$ from *0.65* to *0.75* causes the Generalization (G) score to drop sharply from 85.9 to 74.1. At $\\gamma=0.9$, the Generalization ability collapses almost entirely to 28.7.\n\nGiven this high sensitivity, how can the authors justify applying $\\gamma=0.65$ to the other models without a similar ablation? It is highly probable that the optimal (or even safe) $\\gamma$ is model-dependent. This sensitivity implies a very narrow \"safety boundary.\" How can we be confident that unrelated knowledge queries ($k_{old}$) won't accidentally cross this sensitive threshold as the database scales, especially if the chosen $\\gamma$ is not truly optimal for that specific model architecture?\n\n**2. On the True Cost of \"Scalability\" (Memory and Compute)**\n\nThe paper's claims of scalability and \"controllable\" overhead seem to obscure a significant trade-off in memory and compute.\n\n**Memory:** The authors state in Appendix G that editing 10,000 facts for Llama-3-8B adds 150M parameters.This implies that the paper's headline 100,000-fact model carries approximately **1.5B** in additional parameters. For an 8B model, this is an extra 19% in size. This is a very significant cost and can hardly be described as \"controllable.\" This trade-off should be made explicit in the main paper.\n**Computation:** More critically, the gated retrieval (Eq. 11)  requires calculating the cosine similarity of the current key $k^l$ against *all $m$ keys* in $K_1$, an $O(m)$ operation for every forward pass. The claim of only a 1.5% evaluation time increase for $m=10,000$ is surprising. Does this latency scale linearly with $m$? If so, the $m=100,000$ model would be substantially slower. \n\n**3. On the Unverified Claims of \"Modify\" and \"Delete\" Operations**\n\nThe authors repeatedly claim that NeuralDB is \"easy to manage for supporting operations such as appending, modifying, and deleting\". While appending is well-demonstrated, the \"modifying\" and \"deleting\" operations are completely unverified by experiments.\n\nThis is not a trivial claim. For example:\n\n**Delete:** What happens when a $(k_i, r_i)$ pair is deleted from the database? Does the model revert to its original, pre-trained knowledge for that query? Or does it become confused and generate an error?\n**Modify:** How is \"modification\" defined? Is it simply updating an existing residual $r_i$ for a key $k_i$? Or is it adding a new, conflicting fact about the same subject (which raises the question of key collision, a point not fully addressed in the paper)?\n\nTo validate these practical claims, could the authors provide a concrete case study? For instance: (1) Edit a fact. (2) Test that the edit is successful. (3) Delete the fact's $(k_i, r_i)$ pair from the NeuralDB. (4) Re-test the *same* prompt and report the model's output. Does it successfully revert to the original, pre-trained knowledge?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FcYIlAk8Gp", "reviewer": "Reviewer_4zaC", "summary": "This paper introduces the Neural KV Database (NeuralDB) framework to significantly scale up knowledge editing in Large Language Models (LLMs). The authors re-frame existing Locate-and-Edit (L&E) methods as querying a Key-Value (KV) database, which allows for simultaneous modifications of a massive number of factual knowledge edits. The primary motivation is to overcome the limitations of current L&E methods, which often lead to compromised general abilities and forgetting of previously edited facts when scaling beyond thousands of edits. The proposed NeuralDB claims to scale knowledge editing up to 100,000 facts while maintaining high efficacy and model integrity.", "questions": "see weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper successfully tackles a critical bottleneck in knowledge editing: scalability. Claiming and demonstrating results up to 10,000 edits (with the promise of 100,000 in the full paper) represents a significant leap forward in the field, far surpassing the typical limits of previous L&E techniques\n\n2. The method directly confronts the major issue of maintaining LLM General Abilities while performing massive knowledge updates, which is vital for practical deployment.", "weaknesses": "1. Although the title claims scalability to 100,000 facts, the quantitative experiments only cover up to 10,000 edits, leaving the paper’s core contribution without direct empirical support.\n\n2. Can edited knowledge support reasoning? I suggest adding a Portability metric.\n\n3. How does the framework ensure that the embeddings for 10,000 or 100,000 distinct keys remain sufficiently orthogonal during the retrieval and editing process to prevent mutual interference? If two factual contexts (Keys) are semantically similar, the retrieval mechanism risks activating and modifying non-target KV pairs, leading to subtle, \"soft\" side effects beyond the targeted edit. The current paper lacks a thorough analysis of this embedding space's decoupling capacity at scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VN3x5uc59v", "reviewer": "Reviewer_i8Ez", "summary": "This paper presents NeuralDB, a scalable knowledge editing framework that can efficiently and robustly integrate up to 100,000 factual edits. NeuralDB reconceptualizes the Locate and Edit (L&E) paradigm as a process of querying a neural key-value (KV) database and introduces a nonlinear gated retrieval module. The proposed approach aims to mitigate catastrophic forgetting of general capabilities and maintain LLM consistency even after large-scale edits. Experiments conducted on three LLM architectures including GPT-2 XL, GPT-J (6B), and Llama-3 Instruct (8B), as well as benchmarks such as ZsRE and CounterFact, demonstrate that as the number of edited facts increases from thousands to 100,000, NeuralDB maintains high editing success rates while preserving performance on general language understanding tasks.", "questions": "The comparison in Table 1 is limited to methods that continuously update parameters, which creates an unfair comparison.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. NeuralDB successfully scales knowledge editing to 100,000 facts, which is an order of magnitude higher than previous methods such as AlphaEdit. As the number of edits increases, NeuralDB maintains both average editing effectiveness and general capability, forming a sharp contrast with the performance degradation observed in baseline methods.\n2. The paper provides a coherent reinterpretation of previous Locate and Edit (L&E) approaches as linear key-value (KV) databases, and supports this perspective with mathematical derivations.\n3. The NeuralDB approach is simple yet effective, addressing the inherent linearity limitations of existing techniques and enabling robust scalability.", "weaknesses": "1. The details before Section 4 are primarily about other work and could be condensed.  \n2. In the methodology part of Section 4, the approach essentially constructs a plug-and-play knowledge base and uses a gating mechanism to determine when to perform edits. What is the key difference between this idea and \"Improving Sequential Model Editing with Fact Retrieval\"?  \n3. In the experimental section, the overall results appear impressive. However, if 10,000 data entries are updated in a plug-and-play manner, the updates primarily rely on the advantages of MEMIT or AlphaEdit. It is recommended to validate the approach on more complex datasets, such as MQuAke, which involves multi-hop editing tasks.  \n4. The paper mentions T-Patch, but there is no experimental comparison with it.  \n5. The most critical issue is whether this method continuously modifies the model parameters. For instance, after updating the Nth data entry, do the updates to the 1st to (N-1)th data entries remain effective?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GLyEeFToV2", "reviewer": "Reviewer_v7LD", "summary": "This paper analyzes existing Locate-and-Edit methods and finds that they return the residual vector corresponding to the edited fact, while returning a zero vector for unrelated questions. Based on this observation, the paper proposes **NeuralDB**, which explicitly stores keys and residuals as a KV database. During testing, the most relevant residuals are matched through a **non-linear gated function** and then injected into the hidden state stream.", "questions": "The L&E method requires incorporating the position of the subject token when computing $k$ and $v$. I’m curious whether the $k$ used here is computed in the same way as in L&E. If it is, how is the position of the subject token obtained during testing? If it is not, what exactly is the computation method for $ k $ in this case? Moreover, why is it that even with a different computation method, key matching can still be achieved (i.e., if the last token’s hidden state is used as the key, it should differ from the key at the subject token’s position, and intuitively, the matching success rate should be very low).", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Viewing the key and residual flows as a KV database is a novel perspective.\n\n* NeuralDB is simple yet highly effective, capable of scaling to a large number of edits while preserving the model’s general capabilities.\n* The paper validates the effectiveness of NeuralDB through extensive experiments across multiple models.", "weaknesses": "* The paper lacks a comparison with **MEMOIR [1]**, which identifies relevant edits by comparing the sparse activation patterns of new queries with those stored during editing. Both MEMOIR and NeuralDB share a similar **store–identify–inject** paradigm.\n* Although NeuralDB demonstrates outstanding performance on the **CounterFact** and **ZsRE** datasets, its effectiveness on **multi-hop reasoning editing tasks** remains unknown, such as **RippleEdit [2]** and **MQuAKE [3]**. If NeuralDB also performs well on these datasets, it would further highlight its applicability.\n* NeuralDB requires an additional **KV database**, which increases deployment complexity.\n\n[1] MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs\n\n[2] Evaluating the Ripple Effects of Knowledge Editing in Language Models\n\n[3] MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)