UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models
Authors: Xiaojie Gu
Ziying Huang
Jia-Chen Gu
Kai Zhang
Venue: Accepted by TMLR ()

Abstract: Lifelong learning enables large language models (LLMs) to adapt to evolving information by continually updating their internal knowledge. An ideal system should support efficient, wide-ranging updates while preserving existing capabilities and ensuring reliable deployment. Model editing stands out as a promising solution for this goal, offering a focused and efficient way to revise a model’s internal knowledge. Although recent paradigms have made notable progress, they often struggle to meet the demands of practical lifelong adaptation at scale. To bridge this gap, we propose UltraEdit, a training-, subject-, and memory-free approach that is well-suited for ultra-scalable, real-world lifelong model editing. UltraEdit fundamentally differs from traditional paradigms by computing parameter shifts in one step using only a hidden state and its gradient, making the approach simple yet efficient. To improve scalability in lifelong settings, UltraEdit employs a lifelong normalization strategy that continuously updates feature statistics across turns, allowing it to adapt to distributional shifts and maintain consistency over time. UltraEdit achieves editing speeds more than 7× faster than the previous state-of-the-art method, while requiring 4× less VRAM. This makes it the only method currently capable of editing a 7B LLM on a 24GB consumer-grade GPU. Furthermore, we construct UltraEditBench, the largest dataset in the field to date with over 2M editing pairs, and demonstrate that our method supports up to 2M edits while maintaining high accuracy. Comprehensive experiments on five datasets and six models show that UltraEdit consistently achieves superior performance across diverse model editing scenarios, taking a further step towards safe and scalable lifelong learning. Our code is available at https://github.com/XiaojieGu/UltraEdit

Published in Transactions on Machine Learning Research (03/2026)
UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models
Xiaojie Gu Independent Researcher
peetherrayynoys@@mad. com
Ziying Huang Independent Researcher
m7477546644mail.com com
Jia-Chen Gu UCLA
gujeSuclaa edu
Kai Zhang" The Ohio State University
dragozhangggmaiilccom
Reviewed on OpenReview:
Abstract
Lifelong learning enables large language models (LLMs) to adapt to evolving information by continually updating their internal knowledge. An idcal system should support efficient, wide-ranging updates while preserving existing capabilities and ensuring reliable deployment. Model editing stands out promising solution for this goal, offering a focused and efficient way to revise model's internal knowledge. Although recent paradigms have made notable progress, they often struggle to meet the demands of practical lifelong adaptation at scale. To bridge this gap, we propose ULTRAFDIT, a training, subject-, and memory-free approach that well-suited for ultra-scalable, real-world lifelong model editing. ULTRAEDIT fundamentally differs from traditional paradigms by computing parameter shifts in one step using only hidden state and its gradient, making the approach simple yet efficient. To improve scalability in lifelong scttings, ULTRAEDIT employs lifelong normalization strategy that continuously updales feature statistics across turns, allowing it to adapt to ionall shifts and maintain consistency over time. ULTRAEDIT achieves editing speeds more than 7x faster than the previous state-of-tho-art method, while requiring 4X less VRAM. This makes it the only method currently capable editing a LLM on a 24GB consuner-grade GPU. Furthermore, we construct ULTRr BENCHENCH, the largest dataset the field to date with over 2M editing pairs, and demonstrate that our method supports up 2M edits while maintaining high accuracy. Comprehensive experiments on five datasets and six models show that ULTTAADrr consistently achieves superior performance across diverse model editing scenarios, taking a further step towards safe and scalable lifelong learning. Our code is available at: ht/g://ub..cmm/iiooiee/////aaaa/
1 Introduction
Lifelong learning (also known as continual learning (Shi al. 2024 Zheng al. 2025b)) is essential for enabling large language models (LLMs) continuously adapt evolving knowledge and real-world dynamics. Despite its importance, scalable and reliable lifelong adaptation remains challenging in practice (Wu et al.)
2024a). Retraining is prohibitively expensive and slow, making it unsuitable for frequent updates (Chen et al. Meanwhile, existing lifelong learning approaches often suffer from catastrophic forgetting, or depend
2023
* Cooreespondigg author.

---

Published in Transactions on Machine Learning Research (03/2026)
(4)
1
(b)
27B fotte
Brpoacoech-an Raad
Siigher Offiacy
34
22
163
1090
1300
Prenker Iat
Editigg (misutee)
Figure 1: (a) Average Efficacy and editing time of different solutions on 20K edits from ZRRE, evaluated across GPT-J, Mistral, and LLaMA-3. (b) Variation in average Efficacy as edits accumulate. Dashed lines represent performance on the ZsRE dataset across GPT-J, Mistral, and LLaMA-3, while solid lines represent results on the WikiBigEdit dataset with LLaMA-3.
on retrieval-angmented generation (RAG; Jimenez Gutierrez et al. (2024): Gutierrez et al. (225 ), which may
potentially introduce conflicts between retrieved content and the model's internal knowledge (Xic ct al. 2023).
These limitations suggest that current paradigıns may not fully meet the demands of lifelong deployment, highlighting the need for more targeted and efficient mechanisms for continual knowledge integration (Aljundi
ct al. 2019
One promising solution is model editing (De Cao et al. 2021 Meng et al. 2023
Yao el al. 2023 which
enables targeted modifications to a model's internal knowledge while leaving unrelated information unafeectdd,
making it especially suited for continual updates over time (Hartvigson ct al.| 2023 Hu al. 2024). Some
methods enhance this process by training auxiliary networks (Mitchell et al. 2022a Li et al.) 2025)
guide how model parameters are adjusted in response to new information. Others rely strong structural
assumptions, such as subject- centric representations (Meng et al. 2023 Li et al. 2024) or carefully formatted input prompts (Zheng et al. 2023 Zhong et al. 2023b), which tic them to handcrafted data pipclines (Deng et al. 2025 Zhong et al. 22225) and limit both generalizat ion and practical feasibility. In addition, many approaches depend on external memory (Wang et al.) 2024b) to store edits separately from model parameters; while this localizes changes and avoids direct parameter overwriting,
still requires training to update memory entries and introduces substantial overhead that grows with the number of edits, thereby reducing scalability in large-scale or high-frequency editing scenarios (Bi et al. 2024). To address the issues mentioned above, we propose ULTRAEDIT: simple yet efficient training,, subject-, and memory-free method.
ULTRAEDIT introduces a lifelong normalization mechanism that continuousyy updates feature statistics during editing. By contimually adjusting the mean and variance of concatenated hidden states and gradients, the method implicitly enforees a stable feature geometry that preconditions the least-squares system used for parameter updates. This normalization acts as a form of online whitening: it equalizes feature scales, prevents representation drift from amplifying update magnitudes, and reduces the risk of new edits overwriting previously acquired knowledge. The entire procodure relies solely on simple lincar algebra over editing features and requires no iterative optimization or external memory structures. As result, ULTRAEDIT supports efficient and large-scale edits with strong stability and consistency, making it practical for real-world deployment (Zheng et al.) 2025a Grimes et al. (2025). Existing paradigns, by contrast, are prone to the Edit Collapse phenomenon (Yang et al.) 2024b; Wang et al. 2025), which refers to a sharp decline in editing stability and effectiveness as the number of edits or editing turns grows. As shown in Figure | ULTRA EDrr not only avoids this collapse but also achioves significantly faster editing speods while maintaining consistent. performance in ultra-large lifelong editing scenarios. A comparison between ULTRAEdir and other solutions is provided in Table [
To evaluate our method at its full potential and push the boundarics of large-scale model editing, we introduce our benchmark UDIRAEdIT BENCH, a factual QA benchmark constructed from Wikidata triples, comprising over million complete editing pairs. In addition to this new benchmark, we evaluate the effectiveness and scalability of ULTRAEDIT on four widely used model editing datasots, namely zsRE (Levy et al., 2017). FEVER Thorne et al., 20188) EikiBigEdit (Thode et al. 2025p, and UnKE (Deng et al. (022)), across six
diverse models including GPT (Wagg & Komatsuzaki] 2021
Mistral (Jiang et al. 2023). I.I.MA (Grattafiori
2

---

Published in Transactions on Machine Learning Research (03/2026)
Table 1: Comparison between the ULTRAEDIT and three prevailing model editing solutions. / indicaates "yes" or well-supported, while X denotes 'no' suplo-supported.
Solution
Training- Subject-Iree
Memory-free
Lifelong scalability
Locate-lhen-cdit
X
X
Memory-hased
x
X
x
Figure 2: VRAM usage over the course of 20K edits on the ZsRE dataset using different methods with Mistral-7B.
Hypeructwort-based
X
V
V
x
ULTRAEDIT
et al. 2024), Qwen Yann et al. 2024ap, Phi (Abdin et al. 2024), and Gemma (Team el al. 2025 Our results show that ULTRAEDIT achieves new state-of-the-art performance across most editing scenarios. In addition to its strong empirical performance, ULTRAEDIT demonstrates remarkable efficiency: it achieves over 7x faster editing speeds and usees less than one-fourth of the GPU memory compared to prior leading baselines, as show in Figure (a) and Figure 2 Notably, it is the only method to date capable of reliably editing 7B-scale model on standard 24GB consumer GPU, making it uniquely practical for real-world deployment. Moreover, ULTRAEDIT demonstrates the ability to scale to million edits while preserving model stability, underscoring its promise for ultra-laree-caale lifelong model editing. Our contributions are four-fold:
We identify and analyze the shortcomings of three dominant model editing paradigms under large-scale lifelong settings, providing insights for developing more advanced editing methods.
We propose ULTRAEDIT, novel training-, subject-, and memory-free editing solution that performs stable updales through lifelong normalization. ULTRAEDIT achieves over 7x faster editing and 4x less
VRAM compared to previous state-of-the-art methods.
We construct ULTRAEDITBENCI, currently the largest dataset for model editing, comprising more than 2 Million editing pairs to facilitate research on lifelong ultra-laree-ccale model editing.
Comprehensive experiments five benchmarks and six models, demonstrating Unrr's superior performance and scalability to 2 Million edits.
2
Related Work
2.1 Lifelong Learning
Lifelong learning, also known as continual learning, aims to develop models that can continuously learn from stream of tasks while preserving knowledge acquired in earlier stages (De Lange et al. 2021 Shi
et al. 2024). The primary challenge lies in overcoming catastrophic forgetting, where new learning interferes with previously acquired knowledge. To address this challenge, existing approaches generally fall into three
calegories: regularizaation-beee methods (Kirkpatrick al.] 2017 Zenke et al. 2017). which focus on preserving important parameters; replay-based methods (Rebuthi al. 2017 Shin al. 2017), which revisit
past information during training; and architecturo-basse methods (Rusu al. 2016
Mallya & Lazebnik
2018
which dynamically adjust the model structure.
2.2 Model Editing Paradigm
Hypernetwork- based methods (Mitchell et al.) 2022a Tan et al. 2024
Li al. 2023 Gu et al. 2026 treat
model editing meta-learning problem by training separate neural network predict parameter shifts. This auxiliary network operates independently from the base model and learns to project editing inputs into effective weight updates. Once trained on a collection of edits, it enables quick application of new updates without re-optimizing for each case. However, the hypernetwork remains fixed while the underlying model continues evolve as edits accumulate. This growing mismatch can lead degraded editing performance over time. Furthermore, the need for additional training data limits the practicality of these methods in scenarios requiring rapid or continual updates.
3

---

Published in Transactions on Machine Learning Research (03/2026)
Xturn
Yturn
Xturn
Yturn 2
Xturnt Vturn
LLM
turn
LLM
turn 2
turn t
LLM
0
A
0 + A0
A02
40c
+ ... + 40
dvariance
mean
21
21 11
0 +
Z1
22
02
P2
Lifelong Normalization
x : editing prompt
Ho
d
Update
f1
11
Update
H2
O2
y ground-truth
Figure This figure illustrates the lifelong editing workflow of ULTRAEDIT, where parameter shifts are applied iteratively across turns using a lifelong normalization mechanism that maintains running statistics of editing-instann instanceeenee features ensure stable and consistent model behavior over time.
Locate-then-edit methods (Dai et al. 2022 Meng et al. 2023: Wang et al. 2024a Gupta et al. 2024 Fang et al.) 2025 Pan et al. 2025) rely on the presence of an explicit subject or entity in the input to identify which internal components of the model are primarily responsible for storing the corresponding piece of knowledge. They then apply targeted perturbations, typically through computationally intensive iterative optimization; to enforce the desired output while minimizing side effects on unrelated behaviors and representations. Although effective for isolated edits, these methods often become unstable in lifelong settings, as repeated updates to overlapping parameters can accumulate and lead to interference or degradation of previously edited knowledge.
Memory-based approaches (Dong et al. 2022 Mitchell et al. 2022b Zheng et al. 2023 Hartvigsen et al., 2023: Yu et al., 2024 Wang al., 2024b) enhance the model with an external memory that stores edits separately from the core parameters. New knowledge is incorporated by adding or modifying memory entries, which localizes changes and minimizes side effects on the base model. However, these methods still require training to update memory representations In addition, they maintain one entry per edit, causing memory consumption to grow linearly with the number of edits, which limits their practicality in lifelong model editing with high-frequency or large-scale updates.
routing, and inference must be rerouted through the memory.
3 Methodology
3.1 Preliminary
We consider pre-trained language model fe X + y with parameters Model editing is the task of modifying this modcl's parameters to change its behavior behavior intact. Formally, an editing instance is pair (Ze)ye) X' x y where x, is edit query and ye is the ground-truth output for that input. The goal is to find a parameter set 0' such that the edited model fer produces the desired output Ye for the edit query Ie (Efficacy). In addition, the model should respond appropriately to a set of equivalent queries E(xe) C X, which includes semantically similar variations of the original query (Generulization) while ensuring that its behavior on set of unrelated queries ux:,)) X, which are not associated with the edited knowlodge, remains unaffocted (Specificity).
specific input while leaving the rest of its
In lifelong model editing scenario, the model is updated iteratively across sequence of editing turns. At each turn t, n oditing instances {(=( ye (()))) applied the current model fote-), resulting new model føcn. Each turn of edits performed on the model that has already incorporated all previous edits, making stability across turns critically important.
4

---

Published in Transactions on Machine Learning Research (03/2026)
Algorithm 1: ULTRAEDIT
Input: Model fa, editing instances {(I,, w) editable modules M Output: Edited model for = So + {A0m) [A0m)EMM MEM
foreach instance (I44, Wi) do
Run forward pass to extract hidden state h, each module E M;
Run backward pass to compute gradient Vy, at each module;
Concatenate editing feature: t [h: || Vyll;
Update rumning mean He, variance a using lifelong rules;
Cache 2j for each module;
foreach module m E M do Load cached feature vectors {z for module m,; Normalize: &g + =
Split: [h: || [b]] 6 2g; Compute scaled update: = - E;; IT
Stack H < [NT :..., IT, V + [V] ...., TT;
Compute optimal closed- form solution: A0m + (
Apply: f fm Atm;
return Post-ediled model for
3.2 UltraEdit
Motivation A central challenge in lifelong oditing is to update model parameters in a way that is both targeted and stable. Prior paradigms highlight. this trade-off in different ways: hypernetwork- based methods generate parameter updates via an auxiliary network, but they require costly pretraining and often generalize poorly; locate-then-edit methods directly modify causal mediators, typically identified using the subject entity in the editing instance, which provides precision but relics on per-case localization and iterative optimization; and memory-based approaches preserve past edits for reuse, yet their external memory structures inevitably expand and become increasingly expensive to maintain. These limitations make it difficult to support scalable, high-frequency updates in real-world lifelong learning scenarios. ULTRAEDIT addresses this challenge by exploiting intrinsic signals already present in each editing instance and combining them with closed-form optimization and lifelong normalization. This lightweight. and training-free design avoids external memory or iterative procedures, while ensuring edits can be applied efficiently and consistently across long trajectories.
Principle At each editing turn t, ULTRAEDIT processes a batch of n editing instances by extracting two complementary signals from a designated oditable module (e.g., transformer foedforward layer). A forward hook records the hidden state h4 Rd at the token position corresponding to the ground-truth answer (the unmasked label position), which specifies where the larget. knowledgge is represented in the model. This hidden state anchors the edit to the correct semantic subspace and provides coordinate system for locating the desired modification, and docs not introduce information lcakage, as the label is explicitly part of the editing target (Li et al., 2025 Fang el al. 2025 In parallel, a backward hook obtains the gradient vector Vy; E R2d with respect to the same ground-truth output, derived from the supervised loss. This gradient. indicates how the model's parameters should move to internalize the desired knowledge, thus encoding the direction of change required by the editing constraint. By concatenating these two signals, we construct a unified editing feature 2; [h/ Vy.] E Rd+d' that simultancously encodes the location (via hidden state) and direction (via gradient) of the update. This combination is crucial: the hidden state alone cannot determine how the model should change, and gradients alone lack information about the representational subspace in which the change should occur. Their union therefore provides a complete minimal sufficient statistic for closed-form estimation and enables the method to generate parameter shifts without auxiliary training or iterative optimization. This feature representation also forms the basis for the lifelong normalization mechanism described below.
5

---

Published in Transactions on Machine Learning Research (03/2026)
Design Building the joint feature [h, ll Vysl, we introduce lifelong normalization mechanism that co-normalizes hidden states and gradients across editing turns. In lifelong editing, the distributions of hidden activations and gradients inevitably drift as each edit alters the model's internal representations (Tan ct al. 2024h. Without correction, this drift produces inconsistent feature scales, ill-conditioned least-squares systems, and unstable or interfering updates over long trajectories. To address this challenge, we maintain running statistics (u, over all past editing features and normalize each new instance as follows:
2 = Norm(z) (2)) = 2; 3 P
(1)
where l and a are the running mean and standard deviation, and E is a small constant for numerical stability. This normalization plays a role analogous to whitening or proconditioning: it stabilizes the effective lcarning rale, ensures comparability features across turns, and preserves a well-conditoneed geometry for subsequent. parameter estimation. Crucially, because hidden states and gradients are normalized jointly, their relative scaling remains stable, preventing one signal from dominating the other the model evolves.
At each editing lurn I, let {= denote the concatenated feature vectors from editing instances. The turn-wise mean and variance are computed as = 1/2 Σi= and Var(z) = 2 Ei= (2+ - 2) respectively, We Nn 62,
then compute the difference 5 ,l, accumulate squared deviations with s² + s² + Var(2) + N+n and updale the sample count N + N n
The running mean and standard deviation are updated via:
8, 1
(2)
N + n
82
(3)
+ n 1+e
At the first turn, since no prior statistics exist, we initialize the running mean and standard deviation as Ho and on /Var(2) te.
After normalization, the vector & E Rd+d' is split into two components:
Th 0z] 2,
(4)
where h2 E Rd is the normalized hidden state, and ū, E Rdd the normalized gradient value.
To adaptively scale the influence of each editing sample, following (Mitchell et al.) 2022a Tan et al. 2024 Li ct al.) 2025), employ a scaling mechanism based on the magnitude of its normalized hidden state. Specifically, for each sample i, the scaled update direction is computed as:
f - 11,,1 8..
55)
where is a global scaling factor analogous to learning rate. The scaling reflects the saliency of the hidden representation and determines the strength of the corresponding edit. The multiplicative factor [| l|2 adaptively scales the updale according to the saliency of the hidden representation, ensuring that edits are strongest where the model already encodes the target concept. This mechanism further reduces interference by down-weighting noisy or weakly relevant gradients.
Let H Raxd be the matrix of unnormalized hidden state and V e Rxxd the matrix of scaled update vectors U. The final parameter shift is obtained by solving regularized least-squares problem that minimizes both the reconstruction error and the update norm:
min |AA - V112 + +|00,, 26
(6)
where 0' Rdxddd is the target weight perturbation.
The optimal closed-form solution is given by:
A0 (HT H HT..
(7)

---

Published in Transactions on Machine Learning Research (03/2026)
This solution yields the minimum-norm update that best satisfies all editing constraints simultancously. Geometrically, the projection induced by (HTH H + T)- aligns updates with the subspace spanned by hidden activations, suppressing components that would conflict with earlier edits and thereby mitigating interference. Combined with lifelong normalization, which keeps this geometry well-conditioned across turns, this projection-basel updale mechanism ensures that edits remain largeled, stable, and compatible throughout long editing sequences. The resulting edited model parameters are obtained by directly applying the shift, i.e., 0' 0 + A0, thereby completing the model editing process.
Theoretical Justification We provide a theoretical justification for why our Lifelong Normalization serves as a sufficient mathematical proxy for the explicit covariance preservation (Co) used in Locato-then-Edit methods like MEMIT and AlphaEdit. MEMIT formulates model editing as a constrained least-squares problem, relying on the uncentered covariance matrix Co of the pre-trained keys, denoted as K, to minimize interference. (Here, K represents the input keys to the layer in MEMIT's formulation, which corresponds directly to the hidden states H in our notation.) Geometrically, Co acts as metric tensor in the update formula A x (Co + kKT)-1 effectively defining the Mahalanobis distance within the parameter space. This mechanism penalizes deviations along the principal components of Co, which correspond to directions of high variance where the model possesses rich knowledge, thereby forcing the update vector A into the null space or low-variance directions of the old knowledge.
Lifelong Normalization addresses the computational bottleneck of explicit covariance calculation by dynami- cally maintaining first- and second-order statistics to perform standardization transformation. Specifically, we implement joint normalization on the combined vector of hidden states and error gradients, denoted as 2 z= [A Vyl- This acts as an online preconditioning mechanism that stabilizes the spectral properties of the feature covariance matrix. By centering and scaling this joint distribution, the process effectively whitens the feature space H, ensuring that the condition number x( (HT T H + 1) remains bounded despite the distributional drift inherent in lifelong learning. Originally based on limited samples, the running statistics (1,0) o) progressively converge to a robust reprosentation of the global feature distribution. The key geometric significance of this distributional transformation is that it implicitly reshapes the covariance matrix of old knowledge Co into an approximate identity matrix, Co RJ I. This whitening transformation establishes a direct mathematical equivalence, causing the complex Generalized Least Squares (GLS) problem in MEMIT to degenerate into a computationally efficient Ordinary Least Squares (OLS) problem within the normalized space. Substituting the whitened unit covariance Co I into MEMIT's analytical solution logic naturally simplifies the regularization term (Co+ + KKT)- where K is now replaced by the normalized hidden states H, into the form (I HHT)- adopted by ULTRAEDIT (Eq. 7). Theoretical analysis reveals that this mature calibration prevents anisotropy in the effective Hessian, guaranteeing that parameter updates approximate an orthogonal projection onto the active subspace rather than projecting non-orthogonally into the null space of prior knowledge. Therefore, ULTRAEDIT minimizes the spectral norm of perturbations on unrelated subspaces, effectively serving as a surrogate for the weighted orthogonality of MEMIT and ensuring resistance to cumulative interference as statistical estimation matures.
ULTRAEDIT'S complete pipeline is illustrated in Figure 3 and the pseudocode of the ULTRAEDI editing procedure in one turn is provided in Algorithm T The practical applicability of ULTrAEDrT is discussed in the werl-world lifelong application statement provided in the Appendix A
4 Experiments
4.1 UltraEditBench-2N Construction
We construct ULTRAEDITBENCH using ratity-- triples from the Wikidata5M (Wag et al. 2021 knowledge base. For each triple, we treat the object as the ground-truth answer and use the GPT-4o-mini model in a zero-shot setting to generate corresponding factual questions based on the subject and relation. To promote linguistic diversity, we apply constraints on question length and phrasing during generation. To ensure data quality, we perform random spot checks on a subset of samples to verify factual accuracy, linguistic fluency, and alignment between questions and answers.
7

---

Published in Transactions on Machine Learning Research (03/2026)
Table Results on three datasets across three models and bold values indicate the best results. Eff. denotes Effficayy, Gen. denotes Generolization, and Spe. denotes Specificity. A indicates the performance difference between ULTRAEDIT and the previous best method. ULTRAEDIT denotes results under the same number of edits as other baseline methods, while ULTRA represents performance in ultra-large-scale editing
scenario. Please refer to Section 4.3 for specific configuration details.
ZsRE
FEVER
WikiBigEdit
Method
Etf.
Gen.
Spe.
Etf.
Gen.
Spe.
Eff.
Gen.
Spe.
Personas
Reasoning
GPT-J
FT
15.11
13.55
2.61
14.02
13.98
9.26
17.56
8.47
13.91
WISE
34.13
33.14
26.81
93.95
92.86
57.03
49.88
45.39
30.00
40.52
21.01
AlphaEdit
50.23
43.31
12.54
1.89
1.87
1.85
69.66
55.03
21.20
42.60
0.07
RLEdit
73.34
68.93
22.00
14.26
13.74
14.11
66.18
60.97
32.20
55.78
25.94
ULTRAEDIE
78.03
72.42
27.05
97.45
96.37
79.72
73.84
66.57
37.17
29.27
BLTRAEDIT*
72.95
68.68
25.91
97.89
96.73
79.85
66.46
60.54
47.90
51.73
+4.69
+3.49
+0.24
+3.94
+3.87
+22.82
+4.18
+5.00
+15.70
+1.12
+3.33
Mistrral-BBvvr
FT
13.69
12.43
19.87
23.80
23.37
16.30
13.77
14.86
11.84
11.55
7.51
WISE
34.01
32.61
46.05
81.95
76.94
40.37
37.31
33.44
11.61
27.44
8.95
AlphaEdit
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
RLEdit
72.57
68.87
23.17
92.35
91.39
71.85
57.55
62.47
28.78
49.21
22.41
ULTRAEDIT
85.30
80.80
47.38
97.87
96.09
84.29
76.00
70.15
46.09
62.27
35,80
ULTRAEDer*
81.13
76.78
48.06
98.23
96.97
83.43
71.78
65.63
55.40
56.11
+12.73
+11.93
+2.01
+5.88
+5.58
+12.44
+18.45
+17.68
+26.62
+13.06
+13.39
FT
12.24
10.97
9.06
16.21
13.08
5.01
13.00
11.70
6.75
11.02
4.38
WISE
40.94
40.27
37.40
86.38
86.36
72.08
34.07
32.26
28.91
29.55
21.19
AlphaFdit
74.34
67.85
22.94
6.39
6.14
2.72
63.24
54.68
20.17
42.58
0.01
RLEdit
91.34
89.68
41.94
94.03
90.67
68.71
75.35
70.00
37.21
65.55
28.13
ULTRAEDIT
90.07
87.36
49.51
95.39
91.93
67.14
79.60
73.49
48.51
66.55
35.64
ULTRAEDer*
87.80
85.48
46.74
97.18
94.64
68.62
68.99
63.59
52.28
55.04
-1.27
-2.32
+7.57
+3.15
+3.97
-3.46
+4.25
+3.49
+15.07
+1.00
+7.51
To support evaluation across key dimensions, the dataset divided into three sample types:
Editing instances require the subject entily appear in the question. This design aligns with the assumptions of many subject-dependent editing methods, which rely on identifying subject-centric include it to
representations to apply updates. While ULTRAEDIT does not require this constraint, ensure compatibility with prior paradigms and enable consistent evaluation of Efficacy.
Equivalent instances are paraphrased variants of the editing instances that share the same answers. They evaluate Generalization, which measures whether the edit transfers to semantically equivalent rephrasings. We do not enforce whether the subject entity appears in the question.
Unrelated instances contain questions unrelated to the editing fact and are used to assess Specificily, that is, whether unrclated knowledge remains unaffected after editing. We do not enforce whether the subject entity appears in the question.
For all three types, the answer is explicitly excluded from the question to prevent lexical leakage and ensure that models must rely on internal knowledge rather than surface patterns. ULTRAEDITBENCE comprises over 2 million complete editing pairs, each containing an editing instance, an equivalent instance, and an unrelated instance. This construction enables ULTRAEDITBENCH to serve as a comprehensive and controlled benchmark for evaluating the precision, generalization, and safety of lifelong model editing methods. And for more details, please refer Appendix B.1 The full dataset is available at https haanggiiggaaee.ooaaaaaattttaaaa /hwggggiiiaaeee//aaaaae//t/iaiia

---

Published in Transactions on Machine Learning Research (03/2026)
Table Results on BLTRAEDITBENCI across four models.
GPT-J
Mistral-7B-v0.3
LLaMA-3-8B-Instrucc
Qwen2.6-7B-Ing Inn--nntruutt
Method
Spe.
Elf.
Spe.
Gen.
Spe.
Spe.
FT
22.03
17.61
19.60
0.06
0.36
0.07
13.50
11.92
10.18
13.20
10.53
10.27
WISE
62.51
47.88
47.50
47.21
44.55
39.13
42.27
41.65
39.79
AlpbaEdit
22.19
12.09
6.88
0.00
0.00
0.00
4.51
3.16
2.78
17.44
8.01
5.42
RLEdit
76.50
81.88
65.64
47.08
38.76
39.27
ULTRAEnrr
83.71
81.28
79.01
64.10
ULIRAEdIT*
81.65
76.80
76.44
81.70
77.25
77.09
83. 15
79.11
78.05
80.70
75.78
76.01
+2.61
+1.45
+14.31
+7.21
+6.62
+15.80
+0.01
-0.60
+12.41
+-3362
+37.02
+36.44
4.2
Experiment Setup
Dataset & Model We evaluate the effectiveness and scalability of ULTRAEDIT across five model editing benchmarks: ZsRE, FEVER, WikiBigEdit, UnKE (unstructured long-text data) and our newly constructed ULTRAEDTTBENCHI, on a diverse set of open-source models, including GPT-J, Mistral-7B-v0.33 LLaMA-3-SB- Instruct,
Phi-4-14B, and Gemma-3-27B-it. Detailed descriptions of each dataset, along with their corresponding metrics, are provided Appendix B.1
Baseline We compare ULTRAEDIT against a comprehensive set of basclines, including Finetune (FT), WISE (Wang et al. 2024b), AlphaFdit (Fang et al. 2025), RLEdit (Ti al. 2025), among others. We exclude AnyEdit (Jiang et al. 2025 since it is not designed for lifelong editing. Following (Fang et al. 2025 Li et al,
2025), all methods are evaluated under a consistent setting where each turn includes 100 samples, except for WISE, which is specifically designed to edit one sample per turn. For evaluation protocols, we follow mainstream methods (Fang et al., 2025 Li et al. 2025) by adopting Exact Match as the primary evaluation metric. In addition, we employ WILD (LLM-as-judge) (Yang et al., 2025) as a complementary evaluation. Further clarifications regarding the evaluation protocols are provided in the Appendix B.3] Comprehensive information all baseline methods is provided in Appendix B.2 and hyperparameter configurations are presented in Appendix B.4
4.3 Overall Results
We report the performance of ULTRAEDIT alongside the strongest representative methods from each editing paradigm across multiple benchmarks and models. As shown in Table 2 and 3 ULTRAEDIT* is evaluated on 100K edits for 7&RE and FEVER, 500K for WikiBigEdit, and 2M for ULTRAEDIT ERRREEDEBE,HHH whereas all other methods are evaluated on 20K edits for ZsRE, FEVER, and ULTRAEDIT and 17K for WikiBigEdit. ULTRA EDrT consistently leads across standard metrics including Efficucy, Generulization, and Specificily, as well as two newly introduced metrics, Personas and Multi-hop Reasoning, outperforming all baselines in nearly every setting. This holds true under both the Exact Match and LLM-as-judge evaluation frameworks, for both instruct-Luned and base models, and across diverse data structures of real-world editing instances. Full results for all baseline methods are provided in Appendix C.1 Due to the fact that most existing methods are several hundred times slower than ULTRAEDIT, and some require additional training data, scaling them to larger numbers of edits computationally impractical. Figure [bb shows that existing methods degrade as the number of edits grows, indicating poor scalability ultra-large editing. Tn contrast, ULTRAEDIT remains robust and stable, sustaining millions of updates in lifelong settings. For instance, LLaMA-3-8B--nstruus with ZsRE, it maintains near-standard performance across all metrics with only slight drop under 5x larger edit load. These results confirm that ULTRAEDIT combines high-quality knowledge injection with long-term stability, providing a practical and scalable solution for real-world model editing. Further discussion of failure scenarios can be found in Section C.4 in Appendix.
9

---

Published in Transactions on Machine Learning Research (03/2026)
5
Analysis
5.1
Ablation Study
To assess the contribution of each component in ULTRAEDIT, we conduct an ablation study on 20K editing instances from the ZsRE dataset, with results averaged across three backbone models as shown in Table 4 When the lifelong normalization mechanism is entirely removed, we observe a drastic drop in both efficacy and generalization, underscoring the importance of aligning feature distributions across modules during editing. To further assess its effectiveness, we apply lifelong normalization to randomly selected 25%, 50% and 75% subset of the editable modules. The results show clear upward trend in performance with increasing normalization coverage, indicating cumulative and global benefits. Freezing the normalization statistics by disabling online updates during editing causes a catastrophic collapse in all metrics, demonstrating that dynamic calibration of activation statistics is essential for stability and consistent editing across batches.
In addition, we ablate the original norm- based scaling coefficient || 2 by replac- ing it with a direction-based inner prod- uct k; k. While both schemes involve inner products, our norm-based formula- tion more laithfully captures per-sample saliency, whereas the alternative mixes magnitude and alignment, resulting in less stable modulation across edits. Over- all, the ablation results confirm that both lifelong normalization and its dynamic up- dating mechanism are critical for ensuring stable, accurate, and ultra-scalable edit- ing.
Table 4: Ablation study of ULTRAEDIT. Blue numbers indicate a decrease, while Red numbers indicate increase compared to the full method.
Variant
Efficacy
Generalization
Specificity
Original
84.47
80.19
41.31
w/o lifelong normalization
36.15; 48.32
35.251 44.94
38.141 3.17
w/ 25% module
77.38. 7.00
72.68) 7.51
42.40; 1.09
w/ 50% module norm
80.81, 3.66
76.07; 4.12
42.371 1.08
w/ 75% module norm
33.644, 0.63
79.22; 0.97
41.641 0.33
w/o normalization update
1.114 63.3
1.02; 79.17
0.07, 41.24
w/ ls
74.27; 10:20
70.23; 9.96
42.181 0.87
5.2 Lifelong Scalability
ULTRAEDIT is designed not only for editing accuracy but also for long-term scalability. As shown in Figure [b) and Figure 4 it maintains strong performance across three key metrics- evenn as the number of edits increascs. As shown Figure [bb and Figure 4 maintains strong performance across three key metries as the number of edits increases. This robustness comes from simple yel. effective lifelong normalization strategy that dynamically calibrates internal feature distributions the model evolves. Unlike many existing methods that degrade much carlier, ULTRAEDIT exhibits progressive stabilization: as edits accumulate, its normalization mechanism regularizes the feature space and improves performance until reaching saturation. By contimually adapting to the model's current state without requiring retraining, ULTRAEDIr is particularly well-suited for real-world, lifelong editing scenarios that demand frequent and ongoing updates. Additionally, ULTRAEDIT scales effectively to models with substantially larger parameter sizes. It preserves editing accuracy on both standard language models such Phi-4-14B and more complex multimodal models like Gemma-3-27B, shown in Figure And full scalability evaluation is provided in Appendix C2
5.3 Impact on the General Ability of Post-Edited Models
We evaluate how lifelong editing affects the general ability of post-edited models on four representative benchmarks: SST (Socher et al. 2013), MMLU (Hendrycks et al. 2021 MRPC (Dolan & Brockett 2005). and NLI Williams et al. 2018). Results for the original unedited model (Vanilla), AlphaEdit, WISE, RLEdit, and ILTRAEDDT are shown in Figure D. We observe that as the number of edits increases, methods such as AlphaEdit and finetuning significantly degrade the general ability of post-edited models across all benchmarks, suggesting cumulative interference in the lifelong sotting. WISE shows relatively stable performance by storing odits in an external memory component, trading additional memory for reduced interference with the base model. RLEdit causes notable degradation on NLJ but remains more stable on other benchmarks. In contrast,
10

---

Published in Transactions on Machine Learning Research (03/2026)
90
80
Coneralration
35
Specibelty
Ellicayy
60
30
70-
Wikillghdi Phi-4
EorTBRnc
WikiBigEdit Gemma-3
Eorrllnc Genmno-
5K
16k
4K
13K
20K
Namber of Edits
Number Edits
20K
M6K
400K
500K
2M
Number of Edits
Figure 4: Variation in average generalization and Figure 5: Elficacy of lifelong editing Phi-4-14B and specificity edits accumulate. SST
Gemma-33 3-22---2BBBBB
MMLU
MRPC
NLI
70/
95
0.
50
208
16k
20k
12K
14K
30%
12K 16%
Number of Edits
AlphaEdit
RLEdit
ULIRAEDIT
Figure 6: Performance post-edited LLaMA-3 (20K ZsRE edits) across various benchmarks.
ULTRAEDIT consistently preserves the model's general abilities even after 20K edits, showing almost deviation from the vanilla baseline across tasks. These findings confirm that ULTRAEDIT introduces the least interference to general capabilities and does not increase the risk of hallucinations, making it well-suited for lifelong model editing. In addition, ULTRAEDIT leads to improved performance on MRPC. We attribute this gain to Lifelong Normalization acting as a beneficial regularizer. Theoretically, by dynamically calibrating the first- and second-order statistics (H, o) of hidden states, our method prevents the feature distribution from deviating off the pre-trained manifold, thereby mitigating representation collapse often caused by cumulative parameter shifts. Furthermore, since robust editing requires the model generalize across paraphrased inputs (9),, the optimization implicitly enforces semantic invariance by minimizing the distance between representations of semantically equivalent sentences. This objective is well aligned with the MRPC task (paraphrase detection), sharpening the model's discriminative boundary for semantic similarity. For detailed comparison with other baseline methods, as well as comprehensive description of four benchmarks, please refer to Appendix
6 Conclusion
We present ULTRAEDIT, fast, stable, and scalable approach to lifelong model editing without additional training, subject reliance, external memory. Through lifelong normalization, it adapts to evolving model states while maintaining high precision across editing turns. Experiments show that ULTRAEDIT achieves over 7x faster editing with less than one-fourth the VRAM of prior methods, and supports up to 2M edits with stable performance. These efficiency gains make lifelong editing feasible at ultra-large scale and broadly accessible, lowering barriers and enabling wider community participation. To support further research, we release ULTRAEDITBENCH, the largest benchmark date, with over 2M editing pairs for evaluating ultra-scale, lifelong scenarios.
7 Limitation
Owing to limited computational resources and the scale of certain datasots, we were unable to scale all basclines to ultra-large lifelong editing settings or to models with substantially larger parameter counts. Furthermore, the of LLM-as-a-judge incurs considerable APT expenses, restricted such evaluations to
11

---

Published in Transactions on Machine Learning Research (03/2026)
the ZsRE dataset. As result, did not evaluate the performance of post-edited models under larger-scale conditions (e.g,, after 2M edits).
References
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasckar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXit:2412.08900 2024.
Rahaf Aljundi, Eugene Belilovsky, Tinee Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas PageiC Online continual learning with maximal interfered retrieval. Proc. of NewrIPS, 2019.
Baolong Bi, Shenghua Liu, Yiwei Wang, Lingrui Mei, Hongcheng Gao, Junfeng Fang, and Xueqi Cheng. Struedit: Structured outputs enable the fast and accurate knowledge oditing for large language models. arXiv preprinl ar Xxx::2409.00132, 2024.
Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Adelani, Pontus Lars Erik Saito Stenetorp, Sebastian Riedel, and Mikel Artetxe. Improving language plasticity via pretraining with active forgetting. Proc. of NewrIPS, 2023.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proc. of ACI, 2022.
Nicola De Cao, Wilker Aziz, and Tvan Titov. Editing factual knowledge in language models. In Proc. of EMNLP, 2021.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabangh, and Tinne Tuytelaars. A continual learning survey: Delying forgetting in classification tasks. TPAMI, 2021.
Jingcheng Deng, Zihao Wei, Liang Pang, Hanxing Ding, Huawei Shen, and Xueqi Cheng. Everything is editable: Extend knowledge editing to unstructured data in large language models. In Proc. of ICLR, 2025.
Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proc. of IWP, 2005.
Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. Calibrating factual knowledge in pretrained language models. In Findings of EMNLP, 2022.
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained model editing for language models. In Proc. of ICLR, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21700 283, 2024.
Keltin Grimes, Marco Christiani, David Shriver, and Marissa Catherine Connor, Concept-ROT: Poisoning concepts in large language models with model editing. In Proc. of ICLR, 2025.
Xiaojic Gu, Guangxu Chen, Yuheng Yang, Jingxin Han, and Andi Zhang. Hierarchical orthogonal residual spread for precise massive editing in large language models. In ICASSP 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2026.
Akshat Gupta, Dev Sajnani, and Gopala Anumanchipall. A unified framework for model editing. In Findings of EMNLP, 2024.
Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. From rag to memory: Non- parametric continual learning for large language models. arXiv preprint or Xiv:2500 14802, 2025.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzych Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Proc. of NeurIPS, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proc. of ICLR, 2021.
Chenhui Hu, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Wilke; Wise-layer knowledge editor for lifelong knowledge editing. In Findings of ACL, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arxiv preprinl ar Xix::3310 0.06825, 2023.
Houcheng Jiang, Junseng Fang, Ningyu Zhang, Guojun Ma, Mingyang Wan, Xiang Wang, Xiangnan He, and Tat-seng Chua. Anyedit: Edit any knowledge encoded in language models. Proc. of ICML, 2025.
Bernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobiologically inspired long-term memory for large language models. In Proc. of NeurIPS, 2024.
James Kirkpatrick, Razvan Pascamu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. PNAS, 2017.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlernoyer. Zero-shot relation extraction via reading comprehension. In Proc. of CoNLL, 2017.
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. Pmet: Precise model editing in a transformer. In Proc. of AAAI, 2024.
Zherui Li, Houcheng Jiang, Hao Chen, Baolong Bi, Zhenhong Zhou, Fei Sun, Junfeng Fang, and Xiang Wang. Reinforced lifelong editing for language models. In Proc. of ICML, 2025.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 1x.i::2412.994/2 2024.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning.
In Proc. of CVPR, 2018.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Proc. of NewrIPS, 2022.
Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In Proc. of ICLR, 2023.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning, Fast model editing at scale. In Proc. of ICLR, 2022a.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory-based model editing at scale. In Proc. of ICML, 2022b.
Haowen Pan, Xiaozhi Wang, Yixin Cao, Zenglin Shi, Xun Yang, Juanzi Li, and Meng Wang. Precise localization of memories: A fine-grained neuron-level knowledge editing technique for L.MSs. In Proc. of
ICLR, 2025.
Sylvestre- Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proc. of CVPR, 2017.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1eXii:1606.0444 2016.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang. Continual learning of large language models: A comprehensive survey. or Xiv preprinl arXiu:244,.1678, 2024.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Proc. of NewrIPS, 2017.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proc. of EMNLP, 2013.
Chemmien Tan, Ge Zhang, and Jie Fu. Massive editing for large language models via meta learning. In Proc. of ICLR, 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint ar Xiv:25003. 19786, 2025.
Lukas Thede, Karsten Roth, Matthias Bethge, Zeynep Akata, and Tom Hartvigsen. Understanding the limits of lifelong knowledge editing in llms. orXiv preprint arXiv:2503.0568; 2025.
James Thorne, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. Fever: a large-scale dataset. for fact extraction and verification. In Proc. of NAACL, 2018.
Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model, 2021.
Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. Detoxifying large language models via knowledge editing, In Proc. of ACL, 2024a.
Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. Proc. of NewrlPS, 2024b.
Pinzheng Wang, Zecheng Tang, Keyan Zhou, Juntao Li, Qiaoming Zhu, and Min Zhang. Revealing and mitigating over-attention in knowledge editing. In Proc. of ICLR, 2025.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of ACL, 2021.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proc. of NAACL, 2018
Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learning for large language models: A survey. arXiv preprint arXiv:2402.00 013644 2024a.
Xiaobao WU, Liangming Pan, William Yang Wang, and Luu Anh Tuan. Akew: Assessing knowledge editing in the wild. In Proc. of EMNLP, 2024b.
Jian Xic, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chamelcon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In Proc. of ICLR, 2023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 2024a.
Wanli Yang, Fei Sun, Xinyu Ma, Xun Liu, Dawei Yin, and Xueqi Cheng. The butterfly effect of model editing: Few edits can trigger large language models collapse. In Findings of ACL, 2024b.
Wanli Yang, Fei Sun, Jiajun Tan, Xinyu Ma, Qi Cao, Dawei Yin, Huawei Shen, and Xueqi Cheng. The mirage of model editing: Revisiting evaluation in the wild. arxiv preprint Xrvi:2500. 11177, 2025.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. In Proc. of EMNLP, 2023.
Lang Yu, Qin Chen, Jie Zhou, and Liang He. Melo: Enhancing model editing with neuron-indexed dynamic lora. In Proc. of AAAT, 2024.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proc. of ICML, 2017.
Ningyu Zhang, Bozhong Tian, Siyuan Cheng, Xiaozhuan Liang, Yi Hu, Kouying Xue, Yanjie Gou, Xi Chen, and Huajun Chen. Instructedit: instruction-based knowledge editing for large language models. In Proc. of IJCAI, 2024.
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can edit factual knowledge by in-context learning? In Proc. of EMNLP, 2023.
Hongling Zheng, Li Shen, Anke Tang, Yong Luo, Han Hu, Bo Du, Yonggang Wen, and Dacheng Tao. Learning from models beyond fine-tuning. Nature Machine Intelligence, 2025a.
Junhao Zheng, Shengjie Qiu, Chengming Shi, and Qianli Ma. Towards lifelong learning of large language models: survey. ACM Computing Surveys, 2025b.
Shaochen Zhong, Yifan Lu, Lize Shao, Bhargav Bhushanam, Xiaocong Du, Yixin Wan, Yucheng Shi, Daochen Zha, Yiwci Wang, Ninghao Liu, Kaixiong Zhou, Shuai Xu, Kai- Lei Chang, Louis Feng, Vipin Chaudhary, and Xia Hu. MQQAKK-remaatered:: Multi-hop knowledge editing can only be advanced with reliable evaluations. In Proc. of ICLR, 2025.
Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. In Proc. of EMNLP, 2023a.
Zexuan Zhong, Zhengxuan Wu, Christopher Manning, Christopher Potts, and Danqi Chen. Mquake: Assossing knowledge editing in language models via multi-hop questions. In Proc. of EMNLP, 2023b.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Appendix
The appendix includes the following sections:
Section
A
Statements.
Section
B
Implementation Details of Experiment.
Section
Full Experiment Results.
Section Case Study.
A Statement
A.1 Real-World Lifelong Application Statement
ULTRAEDIT strictly adheres to lifelong editing setting, where updates arrive sequentially and NO future edits are known in advance. The proposed lifelong normalization mechanism does not need to aggregate statistics across all editing turns; instead, it incrementally accumulates feature statistics from previously observed samples up to the current turn using a causal running average, without ever accessing future data. Importantly, each editing turn only requires the statistics from the last turn, which are then updated iteratively, rather than storing or rocomputing information from all past samples. As detailed in equation -qquaton 3 in Section 3.2, the normalization statistics μ and a are computed in running fashion and updated after each editing turn based observed feature vectors.
A.2 Ethical Considerations
While ULTRAEDIT democratizes the ability to maintain LLMs on consumer hardware, this efficiency also lowers the barrier for malicious actors inject misinformation compromise safety alignment at scale. To mitigate these risks, we recommend adopting Integrity Verification mechanisms, such cryptographic hashing (e.g., SHA-256) and digital signatures, to ensure model provenance and detect unauthorized alterations. Furthermore, we advocate for Defensive Editing, where the efficiency of ULTRAEDIT is leveraged to rapidly patch discovered vulnerabilities or reverse malicious injections, thereby serving as a countermeasure against. model poisoning.
16

---

Published in Transactions on Machine Learning Research (03/2026)
B Implementation Details
B.1 Dataset and Metrics
ZsRE (Zero-shot Relation Extraction) dataset (Levy et al. 2017) is a widely adopted benchmark for evaluating factual consistency and knowledge editing in language models. Each example in the dataset comprises a dvestionnanswer pair that encapsulates factual relation. To facilitate comprehensive assessment. of model editing capabilitios, the datasct is augmentod with two types of auxiliary samples: (1) paraphrasod variants of the original question, which test the model's ability to generalize the update, and (2) unrelated but structurally similar questions, which assess the specificity of the edit by ensuring unrelated knowledge remains unaffected. This design enables precise evaluation of editing accuracy, generalization to rephrased inputs, and the preservation of unrelated facts, making ZsRE particularly well-suited for controlled knowledge update tasks. The dataset contains a total of 178, 196 examples.
Following prior work (Mnng et al., 2023 Fang et al. 2025 Li et al. 2025 we evaluate various model editing approaches on ZsRE using standard metrics. Given a large language model fa, a target editing pair (z", y"), its paraphrased equivalent x²', and set of unrelated knowledge pairs (x", yy"), we assess the following three metrics:
Efficacy (Eff.) measures whether the model correctly incorporates the edit by verifying if the top-1 prediction for the edited input xe matches the target label yf:
E of arg max P (v' {
Generalization (Gen.) evaluates whether the model successfully generalizes the edit to paraphrased forms of the input, by checking if the top-1 prediction for xe' remains consistent. with yf:
{ E y arg max 1o (y 12"))
V¹
Specificity (Spe.) assesses the model's ability to retain unrelated knowledge by ensuring the top-1 prediction for each unrelated input x" continues to match its original label y":
y" = arg max P ( x²))
(10)
V'
FEVER (Fact Extraction and VERification) dataset Thorne et al. 2018 is a largo-scale benchmark designed for evaluating factual consistency and claim verification in natural language. Each example comprises a natural language claim accompanied by a label (Supported, Refuted, or Not Enough Information) determined based evidence retrieved from Wikipedia. The claims are cither directly extracted from Wikipodia or are semantically modified versions of actual content, while the supporting evidence may span multiple sentences or even multiple documents. This setup enables fine-grained assessment of a model's ability to confirm, reject, or abstain from factual assertions. The dataset contains a total of 114,422 examples. For consistency in evaluation, we adopt the same metric definitions used in ZsRE.
WikiBigEdit
(Thede et al. 2025) is a large-scale benchmark designed for lifelong knowledge editing. The dataset contains total of 506,03 035 editing pairs, all derived from real-world Wikidata revisions collected across eight time steps over six-month period. To support comprehensive evaluation, WikiBigEdit defines five core metrics: Update, Rephrase, Locality, Personas, and Multi-hop Reasoning. Among them, Update, Rephrase, and Locality correspond closcly to the standard editing criteria of Efficacy, Generalization, and Specificity, and are evaluated using the same methodology as in ZsRE. Personas and Multi-hop Reasoning exlend the evaluation scope to identity conditioning and complex reasoning, respectively. A total of 490,519 examples support the first four metries, while 17,541 are specifically annotated for multi-hop reasoning, enabling focused evaluation of model's ability to handle compositional queries and long-range dependencies. Reflecting realistic and evolving knowledge dynamics, WikiBigEdit supports iterative assessment of a model's capacity to incorporate
17

---

Published in Transactions on Machine Learning Research (03/2026)
factual updates over time. Constructed via an automated data pipeline, the benchmark is continuously expandable new Wikidata edits become available. Specifically, Personas evaluates whether the model correctly answers identity-conditioned prompts x", while Multi-hop Reasoning measures the model's ability to resolve compositional or chained queries xmx Their accuracy is computed as:
E B arg max Pa 122)]. V¹
Personas:
(11)
B 1t" E
arg max IP arg (3/ 127)).
Multi-hop Reasoning:
(12)
BEENH comprises 2,008,326 editing pairs and adheres to the model editing evaluation framework establishod ZsRE. Inference results on different models across all datascts, including BLTNAEDITBENC aligns well with the
are shown in Table 5 The results indicate that the knowledge in
is presented in Tables 6 7 and [ The average prompt length is 11.29 tokens per sentence, and the Hapax Legomena Ratio reaches 56.45%. Together, these results demonstrate the overall diversity of the dataset.
requirements of model editing tasks. The diversity verification of
Table 5: Inference results pre-edited models. ZsRE, FEVER, and WikiBigEdit uses 17K.
use 20K edits, while
Z+RB
FEVER
WikiBigEdit
Model
EIT.
Gen.
Spe.
KdI..
Gen.
Spe.
Edt.
Gen.
Spe.
Personas
Reasoning
Edt.
Gen.
Spe.
GPT-J
27.22
26.42
27.33
9.61
9.68
15.90
29.97
29.08
32.08
28..46
21.81
22.01
21.47
22.20
Mistrol-7BB -B-r0.
44.46
43.55
48.08
0.41
0.50
1.08
39.
38.21
41.62
35.54
29.03
30.83
29.04
31.11
LLaMA-1 SB-Instruct
36.76
38.93
0.02
0.26
24.92
35.46
38.87
32.70
26.42
27.29
26.40
27.24
Qwen2.5 TB-Insstucs
34.32
33.39
38.06
0.57
0.60
2.17
30.97
30:40
34.50
28.43
22.09
26.03
25.22
26.35
Table 6: Domain diversity of ULTRAEDITBENCE
Type
Persun
Organization
Geography
History
Society
Technology
Arts
Polites
Culture
Others
Rate
15.13%
13.21%
6.67%
2.88%
1.27%
0.25%
0.17%
0.05%
0.01%
30.36%
Table 7: Languages of subjects in BETNAE
Type
English
German
Italian
French
Indonesian
Spanish
Tagslog/Filin
Welsh
Finnish
Duteh
Others
Rate
31.78%
7.82%
5.74%
4.52%
4.16%
3.46%
2.75%
2.57%
2.53%
2.36%
32.30%
Table 8: Answer length in ULTRAEDITBENCHI
Length
3
4
5
6
7
8
9
10
Rate
33.66%
36.53%
17.85%
6.12%
3.38%
1.27%
0.55%
0.31% %
0.16%
0.07%
UnKE (Deng et al. 2025 consists of 1,000 unstructured long-text samples, but it only provides editing instances and equivalent instances. To construct more comprehensive benchmark, we take another 1,000 samples from the long-text dataset AKEW (Wu et al. 2024b) as unrelated instances and combine them with UnKE, forming the final UnKE dataset. In addition to the standard metrics of efficacy, generalization, and specificity, we also adopt. BERTScore and ROUGF-L, from UnKE as two additional evaluation metrics. BERTScore measures the semantic similarity between the generated output i and the target y by computing token-level cosine similarities in the embedding space:
BERTScore (oroorreyy..f (000)=
max cos(c,r)
+
Σ rey
max cos c)
(13)
1
Eye
y
cEv"
nE
18

---

Published in Transactions on Machine Learning Research (03/2026)
and denote token embeddings from the generated output i and the target y", respectively, For each token in 0 we find the most similar token in yf (by cosine similarity), and vice versa. The final score averages these two directions, capturing both precision- and recall-oriented matching.
ROUGE-L evaluates the longest common subsequence (LCS) between the generated output & and the target y². Let LCS )*) be their longest common subsequence length, then:
ICS(U),0)) 00) 10a
RLCs= ICS(U, ,0)
PLCs
(14)
1
(1+ B²) PLCS RLCS 3 =1
ROUGE-L(VS 0*)
(15)
RECS + +B -Lic
Datasets such Counterfact (Meng al. 2022), MQuAKE Zhong et al. 2023a KnowEdi (Zhang el. al.
2024), and QAEdit (Yang al. 2025) are excluded from our evaluation, due to their limited scale to support meaningful large-scale lifelong model editing.
We follow the LLM-as-judge template from (Yang et al. DeepSeek- V3.11 (Non-thinking Mode) (Liu et al. 2024 as the third-party model, with outputs truncated
2025) as shown in Table 9 For evaluation, we employ
512 tokens.
19

---

Published in Transactions on Machine Learning Research (03/2026)
Table 9: LLM-as-judge template
Your job is to look at a question, a gold target, and a predicted answer, and then assign a grade of either ["CORRECT", "INCORRECT").
The following are examples CORRECT predicted answers.
Question: What are the names Barack Obama's children? Gold target: Malia Obama and Sasha Obama
Predicted answer 1: sasha and malia
Predicted anawer 2: Malia and Sasha Obama are the names of Barack Obama's 's children.
These predicted anavers are all CORRECT because:
They fully contain the important information in the gold target. They do not contain any information that contradicts the gold target.
The following examples of INCORRECT predicted answers.
Question: What are the names of Barack Obama's 's children? Gold target: Malia Sasha Predicted answer 1: Malia. Predicted
Malia, Sasha, and Susan.
Predicted anawer 3: Malia and Sasha, Malia and Sasha, Malia and Saaha, Malia and Sasha (repeated anawer)
These predicted
all INCORRECT because:
A factual statement in the anawer contradicts the gold target or contain repeated ansver.
Here is sample. Simply reply with either CORRECT or INCORRECT.
Question: (question)
Gold target: (target)
Predicted anawer: prredicted_anaver!
According to the gold target, please grade the predicted answer of this question as one of:
A: CORRECT
INCORRECT
Just return the letters "A" "B", with no text around it.
20

---

Published in Transactions on Machine Learning Research (03/2026)
B.2 Description of Baselines
FT (Fine-Tuning) in model editing refers to the process of updating a pre-trained language model's parameters by optimizing them a small set of new data containing the desired knowledge. This approach aims to adjust the model's behavior to reflect updated or corrected information without retraining the model from scratch. In the context of model editing, fine-tuning typically involves minimizing a loss function such as cross-entropy on carefully selected inputs and targets related to the oditing fact, while optionally applying regularization to avoid catastrophic forgetting. Despite its simplicity and effectiveness, FT can lead to unintended changes in the model's ' general abilities or interfere with unrelated knowledge, especially when used repeatedly or with large learning rates.
MEND enables efficient post-hoc editing of large pre-trained models using only single imput-output pair by training lightweight auxiliary networks that transform standard fine-tuning gradients into localized, low-rank parameter updates. MEND leverages the inherent rank-1 structure of gradients in neural networks to map input activations and output deltas through a small MLP, producing edits that are reliable, local, generalizable, and scalable to models with over 10 billion parameters, all without retraining or modifying the original model's predictions on unrelated inputs.
MEMIT modifies factual associations in language models by identifying and updating critical MLP layers responsible for knowledge recall, computing target hidden states for new facts, and applying analytically derived weight updates that distribute edits across multiple layers. The method treats MLPs as lincar associative memories and performs batch updates to insert new associations while preserving previously stored information, enabling efficient memory editing within the model's internal structure.
MALMEN is a scalable hypernctwork-based method for editing large language models, which addresses the limitations of MEND by formulating parameter shift aggregation as least squares problem solved via the normal equation to prevent cancellation effects, and decoupling the training of the hyper-network and language model to support arbitrary batch sizes, enabling efficient and memory-economic editing of thousands of facts while maintaining strong locality and generalization.
RECT introduces a regularization-basel approach to model editing that constrains the complexity of edit updates by preserving only the top-k% of weight changes based on their relative change magnitude, thereby mitigating overfitting and protecting the model's general abilities. By identifying and retaining the most impactful parameter updates while zeroing out less significant ones, RECT reduces interference with original model knowledge and avoids degradation across downstream tasks. This method ensures that the edited model maintains balance between incorporating new factual information and preserving its overall performance.
WISE proposes a pual-maraetrric memory approach for lifelong model editing, where pretrained knowledge is kept in a main memory and all odits are stored in a separate side momory, which is a copy of a Transformer layer's value matrix. To determine which memory to use at inference, WISE employs a routing mechanism based on activation differences, directing queries to side memory if they relate to edited knowledge. To handle continuous edits without conflict, WISE introduces a knowledge sharding technique that stores different edits in randomly masked subspaces of the side memory, and then merges these using Ties-Merge to maintain consistency. This design allows WISE to achieve high reliability, generalization, and locality simultaneously, overcoming the limitations of traditional long-term or working memory editing methods.
PRUNE introduces a plug-and-play framework for sequential model editing by restraining the condition number of the edited matrix, which identified as the main factor causing degradation in general abilities during repeated cdits. It operates by analyzing the singular value dccomposition of accumulated edit updates and selectively reducing excessively large singular values through a restraint function, thereby minimizing perturbations to the model's original knowledge associations while preserving the newly injected information. This method reduces overfitting from edit accumulation and maintains model stability without interfering with editing efficacy, making it compatible with various existing editing approaches.
Adit introduces a mull-space constrained knowledge editing approach by projecting parameter perturba- tions onto the null space of preserved knowledge, ensuring updates do not interfere with existing information. This projection allows the editing process to focus solely on updating target knowledge without trade-offs, eliminating the need for additional constraints in the optimization objective. The method achieves substantial
21

---

Published in Transactions on Machine Learning Research (03/2026)
improvements in editing performance and generalization with minimal implementation overhead, maintaining the integrity of preserved knowledge and the model's overall capabilities during sequential edits.
RLEdit formulates lifelong model editing as a reinforcement learning problem by treating the hypernetwork as an agent that generates parameter updates based on the current state of the language model and input knowledge, where the update considered an action and the editing performance defines the reward. Tt. trains the hypernetwork offline using accumulated rewards across a sequence of edits, incorporating reward function that balances knowledge injection, preservation of unrelated information, memory backtracking for prior edits, and regularization to constrain update magnitude. This design enables the hypernetwork to adaptively produce edits that are compatible with dynamically evolving model paramcters over long editing trajectories.
B.3 Evaluation Statement
We evaluate the model after completing all editing lurns and compute the metrics on the entire set of edited samples at once, rather than reporting per-turn performance. This evaluation protocol ensures that the reported results reflect the final performance the model after lifelong editing, rather than the transient performance at individual turns. Consequently, our evaluation setting is both fair and consistent with the true goal of lifelong editing.
22

---

Published in Transactions on Machine Learning Research (03/2026)
B.4 Hyperparameter Setting
All experiments conducted on a single NVIDIA A800 GPU, and ULTRAEDrY introduces only two typerparameters: the learning rate 7 and the editable module. In all settings, is set to 1e-6. Regarding the selection of editable modules, we strictly follow the parameter settings used in RLEdit for fair comparison. For Eddi and ULTRAEDITBENCH which are not used in their work, the editable modules are aligned with those used for 7sRE. The details of the editable modules are shown in Table 10 where the numbers indicate the corresponding layer indices.
Table 10: Editable Module settings of ULTRAEDIT across different models and datasets
Dataset
Model
Editable Module
GPT-J
[18-26] m10.)..ll..__ooom
Mistral-7B-v0.3
[29, m0).ml..nwwn proj
ZsRE
LLaMA-3-SB-Instruuc
[11-15] http:/mpp..appppppp [18-24] mwppuppppppp.....
Qwen2.5-7B-Instruct
[18-26] wwlp.gate_proo.iieee wwp.uppppppppppppppppp proj
GPT-J
25
Mistral-7B-v0.3
[29 30] m0m.mlp.wwwn _roj
FEVER
LLaMA-3-SB-Instruuee
[22-30].ml wwwp.amppppp_pppp proj, [22-30] [22-30].mlp.ur wwlp.pppppp..
Qwen2.5-7B-Instrued
[18-26] .mlp.gate [oo][[18-6].mlp.uuu proj
GPT-J
m9-22)..lp.fc out
Mistral-7B-v0.3
[29, 30j.mll..oww proj
LLaMA-3-SB-Instruutt
[18-24]
WikiBigEdit
Swen2.5--B-Instrrnn
[19-26]
1pp_gate__rro_(19-26. mlp.uppprof (30-38). proj
Phi-4-14B
Gemma-3-27B-it
52-60 .mlp.gate (20 proj
GPT-J
[18-26].mlp.fc out
Mistral-7B-v0.3
[29, 30].mllp.www __roj_
BENRAEDITBENCH
LLaMA----BB-Instruu
11-15].mmppgaa proj, [18-24] mlp.up proj
Swee2.5--BBIInsruue
[18-26]
halp.gate__rooj[18--1 mmmpuuppmmmppupp_prr.....
Phi-4-14B
[30-38).mllpdown proj
Gemma-3-27B-it
52-60) wwwp.gtte.........ommmoo (200 _pojjpppp_roojjjjjj_______
GPT-J
(6-226).mlp..f out
Mlittall-B-vv.33
[29, www.mpppdown...m
UnKE
LLaMA-3-SB -----------nnstrcttt
mll.gaeel proj_ [18-24] wwpp:pppppppppp.pp proj
Swen2.5-7B-Instrrn
[18-26] call:gate__proj[115. mwppmppppppppppppp proj
For methods that require additional training data, such MEND, MALMEN, RLEdit and WISE, we follow the experimental sctup described in RLEdit. Specifically, for the ZsRE and FEVER datasets, the training set for MEND and MALMEN contains the total nunber of samples excluding the editing examples. For large-scale datasets like WikiBigEdit and BLTRAEDITBENCH, the size of the training set is equal to the number of editing examples. In all RLEdit and WISE experiments, the training and editing set sizes are always matched.
C Full Experimental Results
C.1 Extended Baseline Comparison
This section presents a comprehensive comparison of ULTRAEDIT against baseline methods across five datasets and four model architectures. Results on UnKE are reported in Tables [1 and 12 while Table 13 presents
the evaluation on zsRE using LLM-as-judge. Detailed results in Tables 14 and 16 further highlight the
strong performance of ULTRAEDIT across diverse editing scenarios.
23

---

Published in Transactions on Machine Learning Research (03/2026)
Table 11: Results across GPT-J and Mistral on UnKE *** indicates results from editing 1,000 instances; all others are based on 500 instances.
UnKE
GPT-J
Mistral-7B-v0.3
Method
Eff.
Gen.
Spe.
Bert Score
Rouge-L
Eff.
Gen.
Spe.
Bert Score
Rouge-L
FT
46.93
46.49
50.49
51.02
11.85
70.87
70.43
70.85
70.95
13.39
WISE
91.26
89.81
65.07
85.60
42.01
89.77
88.78
70.68
79.61
35.64
AlphaEdit
63.20
77.19
RLEdit
86.53
84.94
56.35
79.97
33.68
44.35
15.29
33.80
ULTRAEDIT
91.31
89.81
61.53
81.27
36.43
92.94
90.46
65.89
74.68
30.09
ULTRAEDIT*
92.49
91.15
65.93
83.02
43.28
62.79
61.72
38
46.69
7.01
A
+1.23
+1.34
-11.26
-2.58
+1.27
+3.17
+ 1.668
-12.51
-8.22
-9.40
Table 12: Results across LLaMA-3 and Qwen2.5 on UnKE
UnKE
Qwen2.5- Instruct
Method
E.f.
Gen.
Spe.
Score
Rouge-L
Eff.
Gen.
Spe.
Bert Score
Rouge-L
FT
67.09
66.39
67.31
64.65
6.89
45.72
45.44
32.35
47.67
12.12
WISE
83.95
82.93
58.08
Edit
71.07
68.31
69.17
81.39
37.37
39.98
37.78
25.14
64.72
23.36
RLEdit
91.91
91.03
62.23
85.76
56.78
92.65
91.59
63.17
84.03
51.68
ULTRAEDIT
93.05
91.43
65.62
84.29
45.33
86.39
84.33
61.11
81.14
32.22
ULTRAEDIT*
94.09
92.68
73.64
85.84
50.92
88.84
87.01
72.17
82.02
36.31
A
+2.18
+44.77
+0.08
-5.86
-4.58
+9.00
Table 13: Evaluation zsRE using LLM-as-judge across three models. We exclude GPT-J due to its limited
capability in instruction-shot.
MB-vra
LLaMA-3--B-IIstrruce
Qwen2.5-7B-Innsruus
Method
Eff.
Gen.
Spe.
Eff.
Gen.
Spe.
Eff.
Gen.
Spe.
FT
0.18
0.22
0.14
0.03
0.02
0.00
0.43
0.29
0.10
WISE
0.90
1.03
0.13
8.90
8.40
12.23
AlphaEdit
0.00
0.00
0.00
58.47
56.36
24.64
0.08
0.08
0.00
RLEdit
7.75
5.50
0.00
47.09
44.31
24.46
17.11
15.67
15.52
ULTRAEdIt
24.49
21.97
19.95
52.85
49.77
40.13
32.65
30.45
33.60
+16.74
+16.47
+19.81
-5.62
-6.59
+15.49
+15.54
+14.78
+18.08
Table 14: Extended results across four models on ULTRALEDIT
BETNA
GPT-J
Mistral-7R-v0..R
Qwen2.6- B-nele----B--nttrrr
Method
Etf.
Gen.
Spe.
Eff.
Gen.
Eff.
Gen.
Spe.
EH
Gen.
Spe.
MEND
1.71
1.71
1.83
0.00
0.00
0.00
0.00
0.00
0.00
3.48
3.45
3.43
MEMIT
0.25
0.18
0.20
0.00
0.00
0.00
0.71
0.45
0.21
0.82
0.32
0.76
0.48
3.47
40.64
34.18
37.29
RECT
PRUNE
0.32
0.27
ULIRAEDUT
61.03
U
76.80
76.44
83.45
78.05
+82.32
+75.09
+74.61
+81.29
+74.82
+73.62
+44.006
+47.10
+40.76
+76.34
+72.30
+71.63
24

---

Published in Transactions on Machine Learning Research (03/2026)
Table 15: Extended results on three dalasets across three models.
ZsRE
FEVER
WikiBigEdit
Method
Cen.
Spe.
Eff.
Gen.
Spe.
Eff.
Gen.
Spe.
Personas
Reasoning
GPT-J
MEND
2.52
2.55
0.19
52.80
51.44
45.42
0.02
0.01
0.02
0.02
0.02
MEMIT
0.00
0.00
0.00
5.54
5.03
5.46
1.59
1.59
0.50
0.80
0.00
MALMEN
0.02
0.01
0.02
1.33
1.25
2.92
0.00
0.01
0.01
0.00
0.00
RECT
0.12
18.12
18.08
12.27
2.13
1.99
0.59
2.06
0.00
PRUNE
0.00
0.00
0.01
5.25
4.72
5.20
2.36
2.32
1.01
1.86
0.00
ULTRAEDIT
78.03
72.42
27.05
97.45
96.37
79.72
73.84
66.57
37.17
56.90
29.27
ULTRAEDrP*
72.95
68.68
25.91
97.89
96.73
79.85
66.46
60.54
47.90
51.73
+75.51
+69.87
+26.86
+445.9
+45.29
+-34.43
+71.48
+64.25
-46.89
+54.84
+29.25
Mistral-7B-v0..5
MEND
0.00
0.00
0.00
0.00
0.00
0.01
0.00
0.00
0.01
0.00
MEMIT
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MALMEN
0.00
0.00
0.00
18.42
17.43
12.09
0.00
0.00
0.01
0.00
00.00
RECT
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
PRUNE
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ULTRAEDIT
85.30
80.80
47.38
97.87
96.09
84.29
76.00
70.15
46.09
62.27
35.80
ULIRAEDr
76.78
48.06
98.23
96.97
83.43
71.78
65.63
55.40
56.11
+-5.30
-0.800
+-48.06
79.81
+79.54
+ 20
+77.99
+70.15
55.39
62222
+35.80
LLAMA-3-8B--
0.00
36.19
36.19
24.31
0.01
0.01
0.10
0.01
0.00
MEMIT
0.14
0.14
1.40
0.02
0.02
0.02
0.02
0.02
0.09
0.02
0.00
MALMEN
0.20
0.12
0.09
94.50
91.26
G7.76
0.00
0.00
0.00
0.00
0.00
RECT
0.00
0.00
0.01
0.00
0.21
0.24
0.06
0.29
0.00
PRUNE
0.00
0.00
0.24
0.02
0.02
0.00
0.02
0.02
0.09
0.02
(1.00
ULTRAEDIT
90.07
87.36
49.51
95.39
91.93
67.14
79.60
73.49
48.51
66.55
35.64
ULTRAEDeT*
85.48
46.74
97.18
94.64
63.59
52.28
55.04
+99.87
87.22
+-48.11
+-2.68
+.3.38
+0.86
+.79.99
+ 73.25
-2.288
+66.26
+-35.6
Table 16: Results on three datasets across Qwen2.5.
S0wen.55-BBIInsrrue
ZsR.F.
FEVER
WikiBigFdit
Method
Etf.
Gen.
Spe.
Eff.
Gen.
Spe.
Eff.
Gen.
Spe.
Personas
Reasoning
FT
14.02
10.91
3.39
24.62
21.36
10.35
7.59
3.68
5.55
5.84
MEND
15.00
14.41
0.47
76.43
77.66
40.39
0.00
0.00
0.00
0.00
0.00
MEMIT
0.02
0.02
0.17
0.08
0.12
0.15
0.54
0.33
0.38
0.32
0.00
MALMEN
0.00
0.00
0.06
0.06
0.07
0.06
0.02
0.02
0.00
0.00
RECT
0.00
0.00
0.00
3.36
3.10
3.20
2.34
2.29
0.83
0.78
0.00
PRUNE
0.01
0.02
0.07
0.08
0.15
0.11
2.34
2.34
0.97
1.37
0.00
AlphaEdit
16.32
13.96
1.66
32,78
31.19
22.12
20.31
15.49
2.17
9.01
0.23
RLEdit.
82.79
0.00
2.83
ULTRAEDIT
82.03
77.08
45.51
97.97
93.91
68.86
73.37
65.86
45.12
54.65
32.74
ULTRAEDrrt
78.39
74.72
49.27
97.49
94.83
68.99
66.34
60.42
51.74
50.53
-2.67
-5.71
+11.01
+21.54
+17.17
-28.60
+53.00
+50.37
+48.06
+45.64
-26.9
25

---

Published in Transactions on Machine Learning Research (03/2026)
C.2 Supplementary Lifelong Scalability Evaluation
Figure shows comparison of ULTRA EDrT and baseline methods the number of edits gradually increases. The results demonstrate that ULTRAEDrY maintains stable performance, highlighting its lifelong scalability with respect to the number of edits.
90
30
80
70
Gooeralizztoon
60
50
Specifitt
60
60
20
10
4K
10k
Number
Numabe of Edits
Numaer Eddtt
Figure 7: Performance comparison of different baselines ZsRE edits accumulate in LLaMA-3.
The Generalization and Specificity performance of ULTRAEDIT on larger-parameter models is shown in Figure demonstrating the method's scalability with respect to model size.
80
80
75
75
70
70
65
65
60
Generalization
Specificity
55
50-
45
WikiBigEdit Phi-4
WikiBigEdit Phi-4
-...
EDTRAEE Phi-4
35
------.
ULTRA EDIrBNCH Phi-4
WikiBigEdit Gemma-3
WikiBigEdit Gemma-3
ULTRAEDDTBENCE Gemma-3
ULTRAEDITBENC Gemma-3
25
20K
50K
100K
150K 200K 300K 400K 500K
2M
100K 150K 200K 300K 400K 500K 2M 20K
50K
Number of Edits
Number of Edits
Figure 8: Generalization and Specificity of lifelong editing on Phi-4-14B and Gemma-3-27B.
C.3 Additional Results on Post-Edited Model Evaluation
We begin by introducing the four evaluation benchmarks used in study:
SST (Stanford Sentiment Treebank) is sentiment analysis benchmark composed of movic reviews annotated with fine-grained sentiment labels. It evaluates a model's ability to capture subtle emotional cues in natural language.
MMLU (Massive Multitask Language Understanding) is a comprehensive benchmark spanning a wide range of academic and professional subjects. It assesses model's general knowledge and reasoning ability across diverse domains.
26

---

Published in Transactions on Machine Learning Research (03/2026)
MRPC (Microsoft Research Paraphrase Corpus) contains sentence pairs annotated for semantic equivalence. This benchmark tests whether model can accurately identify paraphrases, reflecting its understanding of meaning preservation.
NLI (Natural Language Inference) tasks involve determining the logical relationship between a premise and hypothesis, namely entailment, contradiction, or neutrality. They evaluate model's capacity for logical reasoning and inference.
Figure provides full comparison between ULIRAEDIT and baselines, further confirming its minimal effect on the model's inherent abilities.
MMLU
MRPC
NLI
70"
60
50
08
50
Number Edits
Vanilla
MEND
MEMIT
MALMEN
PRUNE
RECT
ULTRAEDIT
Figure 9: Performance of post-edited LLaMA-3 (20K ZsRE edits) across various benchmark.
C.4 Failure Scenarios of Gradient-based Localization
Due to structural bias, where gradients disproportionately concentrate on functional tokens (e.g., punctuation the last token) rather than the semantic subject (often referred to as 'attention sinks"); and second, in cases of spurious correlations, where the model relies non-causal shortcut features (e-g., specific adjective) to predict the answer, leading gradients to misidentify the editing target. In contrast, explicit masking avoids these distractions by forcibly anchoring updates to the subject, though it requires the restrictive assumption of known subject boundaries.
27

---

Published in Transactions on Machine Learning Research (03/2026)
D
Case study
Table 17: Case study of applying ULTRAEDIT to LLaMA3-SB on BENNAE indicicatnnn//diitesss indicates the post- edited output exactly matches the ground-truth answer, while X denotes mismatch.
Prompt
Pre-edited Output Ground truth Post-edited Output
What is the country of citizenship for Olivier Renard?
Oliviergium
belgie
belgie V
What type of place mechanics- ville, delaware?
Mechanics
community
community
What type of ecological system does goobang creek belong to?
Goine
riverine
riverine
What is the profession of shamsula- maar nasarah?
Shician
Politician
Politician
What type of sport was featured du- ring the 2003 Canadian Open?
The volleyball
indoor tennis
indoor tennis
What is the place of birth of jyotsna Jwaitali radhakrishnan?
kuwet
malwet X
Which location shares a border with Guzguer guipronvel?
plouguin
Saintouguin X
28


Reviews:
[{"review_id": "qCOjVuBzYS", "reviewer": "Reviewer_q9XL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses scalability of lifelong model editing by proposing UltraEdit, which extends closed-form least-squares editing (similar to MEMIT) to handle several sequential edits. The core contribution is a lifelong normalization mechanism that maintains running statistics (μ, σ) over concatenated hidden-state-gradient features z_i = [h_i || ∇y_i] across all editing turns. This normalization stabilizes the least-squares system (H^T H + I)^(-1) H^T V as model parameters evolve, preventing the conditioning degradation that causes existing methods to collapse.\n\nExperiments on ZsRE, FEVER, WikiBigEdit, UnKE, and the new UltraEditBench (2M editing pairs) across six models show UltraEdit achieves 7× faster editing speed and 4× lower memory than RLEdit while maintaining or exceeding accuracy. The method scales to 2M edits on UltraEditBench (81.65% efficacy) versus baselines that degrade after 20K edits. A key practical advantage: UltraEdit is the only method that can edit 7B models on 24GB consumer GPUs.\n\n**Strengths:**\n- Solves scalability problem: existing methods collapse beyond ~20K edits \n- Efficient: 7x speed, 4x memory\n- Thorough ablations show normalization is critical \n- Largest benchmark to date (2M pairs) for lifelong model editing\n\n**Weaknesses:**\n- Incremental novelty—running normalization applied to known least-squares formulation\n- No theoretical analysis of why joint [h || ∇y] normalization prevents interference\n- Unexplained phenomena: specificity improves with scale (Table 3: 64.03% → 76.44% at 2M edits)..why?\n- All experiments are factual QA; no evidence method works for other knowledge types", "claims_and_evidence_yn": "Yes", "claims_explanation": "The empirical claims are well-supported but theoretical claims lack rigor:\n\n**Strong evidence**: Figure 1b shows baselines collapse after 5K-10K edits while UltraEdit scales to 2M edits (81.65% efficacy, Table 3). Efficiency claims validated: 7× speed, 4× memory (Figures 1a-2). Table 4 ablations are convincing.\n\n**Under-justified claims**: The paper claims normalization \"prevents representation drift\" but provides no condition number analysis of H^T H, no visualization of feature distribution evolution, or empirical validation of drift. No failure mode analysis.\n\n**Narrow scope**: All experiments edit factual triples (entity-relation-object). Claims of \"real-world lifelong editing\" are overstated without testing procedural knowledge, reasoning patterns, or behavioral edits. Evidence is strong for the specific setting tested but broader claims need support.", "audience_yn": "Yes", "audience_explanation": "Empirical results are thorough. Existing methods collapse beyond ~20K edits while UltraEdit scales to 2M. This matters because deployed LLMs need frequent knowledge updates, and current solutions (retraining, RAG, prior editing methods) all have some limitations. The efficiency gains are genuinely significant.\n\nHowever, the technical contribution is somewhat incremental. The least-squares formulation resembles MEMIT minus causal tracing plus running normalization, both standard techniques. The insight that maintaining feature statistics prevents edit collapse is useful but not groundbreaking. Still, identifying the right simplification has value, and the extensive empirical work (6 models, 5 datasets, thorough ablations) plus practical utility will appeal to TMLR's audience despite limited theoretical depth.", "requested_changes": "- **Add per-turn performance analysis**: Current evaluation only reports final results after all editing turns. Adding results showing efficacy/generalization/specificity trajectories (every 10K edits) to reveal when degradation begins and how methods differ in stability would be insightful. \n\n- **Explain specificity improvement finding**: Tables 3 and Figure 4 show specificity improves with more edits (64.03% → 76.44% on UltraEditBench), contradicting typical interference patterns. Provide analysis of why this occurs? \n\n- **Expand beyond factual QA**: Test on non-factual edits (reasoning patterns, instruction-following, style) to validate generality claims.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c2z0naoNcY", "reviewer": "Reviewer_sDLd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes UltraEdit, a novel method for lifelong model editing that aims to address the limitations of existing paradigms (Locate-then-edit, Hypernetwork-based, and Memory-based) which often suffer from \"Edit Collapse,\" high computational costs, or growing memory requirements.\n\nKey Contributions:\n1. The authors introduce a \"Lifelong Normalization\" mechanism. This approach continuously updates the feature statistics (mean and variance) of hidden states and gradients across editing turns. This normalization preconditions the feature space, allowing for a closed-form, regularized least-squares solution to be applied sequentially without the numerical instability that typically degrades performance in continuous editing.\n2. The Proposed UltraEdit is training-free, subject-free (does not explicitly require subject entity extraction), and memory-free (no external knowledge bank). The authors claim it is 7x faster and requires 4x less VRAM than SOTA methods, enabling the editing of 7B models on consumer-grade GPUs (24GB).\n3. The paper constructs UltraEditBench, a massive dataset derived from Wikidata5M, containing over 2 million editing pairs to test the limits of lifelong editing.\n4. Extensive experiments on ZsRE, FEVER, WikiBigEdit, UnKE, and UltraEditBench using models like LLaMA-3, Mistral, and Qwen show that UltraEdit maintains high efficacy, generalization, and specificity even after millions of edits, whereas baselines typically collapse after thousands.\n\nStrengths:\n1. The scalability to 2 million edits is a significant empirical achievement.\n2. The method is computationally highly efficient, addressing a major bottleneck in practical deployment.\n3. The \"Lifelong Normalization\" concept is a theoretically grounded and elegant solution to distribution shift in streaming updates.\n\nWeaknesses:\n1. The definition of \"Subject-free\" requires finer technical clarification regarding token selection.\n2. The mathematical formulation shares strong similarities with prior closed-form editing methods (like MEMIT), and the distinction relies heavily on the normalization step, which warrants deeper discussion.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. Figures 1(b) and 4 clearly demonstrate the \"Edit Collapse\" phenomenon in baselines (AlphaEdit, RLEdit, WISE) and explicitly show UltraEdit maintaining performance metrics up to 2 million edits.\n2.  The runtime and VRAM usage comparisons (Figure 1a, Figure 2) substantiate the claims of being 7x faster and more memory-efficient.\n3. Tables 2, 3, 11, and 12 provide detailed breakdowns across multiple metrics (Efficacy, Generalization, Specificity, Reasoning) and multiple model families, showing consistent improvements.\n4. Table 4 effectively isolates the contribution of the \"Lifelong Normalization\" component, proving it is the critical factor for success.\n5. The experimental setup covers a wide range of models (including 27B parameters) and datasets, making the evidence robust.", "audience_yn": "Yes", "audience_explanation": "Lifelong learning and efficient knowledge updating for Large Language Models are currently central topics in the AI community. The ability to update models continuously without expensive retraining or massive external memory overhead is highly relevant to researchers working on:\n1. Model Maintenance: Keeping LLMs factual and up-to-date.\n2. Efficient AI: Running updates on consumer hardware.\n3. Continual Learning: Solving catastrophic forgetting in generative models.\n4. The release of a 2 million-sample benchmark (UltraEditBench) will also be of significant interest to the community for stress-testing future editing methods.", "requested_changes": "I recommend the following adjustments to strengthen the submission. \n\n1. The paper claims the method is \"Subject-free.\" However, in Section 3.2 (Principle) and Algorithm 1, the method requires a \"forward pass to extract hidden state $h_i$ at the token position corresponding to the ground-truth answer.\" Please clarify how this differs practically from subject-dependent methods that might target the last token of the subject. If the method relies on the ground-truth answer's position during the editing phase, it is indeed \"subject-free\" in terms of entity extraction, but it is heavily \"location-dependent.\" Furthermore, explain how inference works if the edit is tied to the *answer* token position. Does the edit modify weights such that the prompt alone (without the answer) triggers the correct hidden state? A clearer explanation of the token indexing during the *update* versus *inference* is needed to justify the \"Subject-free\" claim fully.\n\n2. The closed-form solution (Eq. 7): $\\Delta\\theta = (H^T H + I)^{-1} H^T V$ is mathematically identical to the update rules used in ROME/MEMIT (derived from Lagrange multipliers for the least-squares objective). The paper should explicitly acknowledge this similarity. The novelty is not the equation itself, but the application of Lifelong Normalization to make this equation stable in a streaming/sequential setting without access to the full covariance matrix of all previous data. Framing the contribution as \"adapting the closed-form solution to streaming via normalization\" is more precise than implying the update rule itself is the primary novelty.\n\n3. In Table 2 and Table 3, UltraEdit is evaluated on 100K-2M edits, while baselines are evaluated on 20K edits. While Figure 1(b) justifies this by showing baselines collapse, it would be clearer to include a row for UltraEdit at 20K edits specifically in these tables (alongside the UltraEdit row) to provide a direct, apples-to-apples comparison at the same checkpoint. This removes any ambiguity about whether the edit volume affected the metrics in the table.\n\n4. The ablation study mentions \"w/ 25% module norm.\" Please clarify the selection strategy for these modules. Are they selected randomly, or based on depth (early vs. late layers)? This helps understand if normalization is needed uniformly or if certain layers are more sensitive to distribution drift.\n\n5. Page 2, Line 60: \"less than one-forth\" should be \"one-fourth\". Inconsistent naming: Ensure consistency between \"UltraEditBench\" and \"UltraEdit Bench\" throughout the text.", "broader_impact_concerns": "The authors include a \"Real-World Lifelong Application Statement\" in Appendix A, which focuses on the technical setup (no future data access). However, the broader ethical implications of highly efficient, low-resource model editing are not fully addressed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xr7iHlFuDx", "reviewer": "Reviewer_AtSH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the critical challenge of \"Lifelong Model Editing\" (LME), where Large Language Models (LLMs) must undergo continuous, sequential updates to their knowledge base without suffering from \"Edit Collapse\"—a phenomenon where performance degrades sharply as the number of edits scales. The authors identify that existing paradigms (Locate-then-Edit, Hypernetworks, Memory-based) fail to scale beyond thousands of edits due to representation drift, weight norm growth, and computational bottlenecks.\n\n1. A closed-form update rule using \"Lifelong Normalization\" (online whitening) to stabilize feature distributions and prevent drift without external memory.\n\n2. UltraEditBench, the largest editing dataset to date (2M+ pairs), designed to test asymptotic stability.\n\n3. Claims of 7x speedup and 4x VRAM reduction, enabling 2M edits on consumer GPUs (24GB VRAM).", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. Results on the massive UltraEditBench (Figure 1b) demonstrate that UltraEdit prevents \"Edit Collapse\" up to 2 million edits, maintaining stability where baselines (e.g., AlphaEdit, RLEdit) degrade after 10k-20k updates.\n\n2. Hardware measurements confirm the claimed 7x speedup and 4x VRAM reduction (Figure 2). This aligns with the theoretical efficiency of UltraEdit's lightweight linear algebra operations compared to the heavy SVD or hypernetwork computations of prior methods.\n\n3. The methodology clearly supports the claim of \"subject-free\" editing by demonstrating that loss gradients provide sufficient implicit localization without requiring explicit subject parsers.", "audience_yn": "Yes", "audience_explanation": "The problem of updating LLMs without expensive retraining is one of the most pressing issues in the field.\n\n1. The ability to edit 7B models on consumer hardware (24GB VRAM) democratizes this research area, making it highly relevant to academic labs and smaller industry practitioners who lack H100 clusters.\n\n2. The \"Lifelong Normalization\" strategy connects model editing with classical optimization concepts (preconditioning/whitening), offering a fresh perspective on the \"plasticity-stability\" dilemma that is distinct from the dominant null-space or memory-bank approaches.\n\n3. The release of a 2M-instance benchmark (ULTRAEDITBENCH) will likely become a standard for future scalability research, generating significant citations and usage within the TMLR community.", "requested_changes": "1. In Table 2, the performance of RLEdit for FEVER with LLaMA-3-8B is significantly lower than reported in the original paper.\n\n2. In Figure 6, the model appears to perform better on the MRPC task after editing compared to the vanilla model. Typically, editing is destructive or neutral. Please provide insights into this anomaly—is the \"Lifelong Normalization\" acting as a beneficial regularizer, or is this noise?\n\n3. Provide a theoretical justification for why \"Lifelong Normalization\" (centering/whitening) is a sufficient mathematical proxy for the explicit covariance preservation ($C_{old}$) used in methods like MEMIT to ensure orthogonality.\n\n4. Confirm that baselines like AlphaEdit were tuned for the lifelong setting (2M edits). Using static hyperparameters meant for single edits could exaggerate the observed \"Edit Collapse.\"\n\n5. Minor: Discuss scenarios where gradient-based localization might fail compared to explicit subject masking.", "broader_impact_concerns": "The method enables mass-scale knowledge injection (2M+ edits) on consumer hardware. This lowers the barrier for malicious actors to rapidly inject misinformation or systematically remove safety guardrails from open models. A brief statement acknowledging this dual-use risk and potential mitigations (e.g., model hashing) is required.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)