Fine-tuning Done Right in Model Editing
Authors: Wanli Yang
Rui Tang
Hongyu Zang
Du Su
Qi Cao
Jingang Wang
Huawei Shen
Xueqi Cheng
Fei Sun
Venue: ICLR 2026 Poster (2026)
Keywords: Model Editing
Fine-tuning
Knowledge Editing
Lifelong Editing
Localized Fine-tuning
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: This paper corrects prevailing misconceptions about the effectiveness of fine-tuning in model editing, and demonstrates that simple fine-tuning can effectively address model editing tasks.

Abstract: Fine-tuning, a foundational method for adapting large language models, has long been considered ineffective for model editing.
Here, we challenge this belief, arguing that the reported failure arises not from the inherent limitation of fine-tuning itself, but from adapting it to the sequential nature of the editing task, a single-pass depth-first pipeline that optimizes each sample to convergence before moving on.
While intuitive, this depth-first pipeline coupled with sample-wise updating over-optimizes each edit and induces interference across edits.
Our controlled experiments reveal that simply restoring fine-tuning to the standard breadth-first (i.e., epoch-based) pipeline with mini-batch optimization substantially improves its effectiveness for model editing.
Moreover, fine-tuning in editing also suffers from suboptimal tuning parameter locations inherited from prior methods. 
Through systematic analysis of tuning locations, we derive LocFT-BF, a simple and effective localized editing method built on the restored fine-tuning framework.
Extensive experiments across diverse LLMs and datasets demonstrate that LocFT-BF outperforms state-of-the-art methods by large margins. 
Notably, to our knowledge, it is the first to sustain 100K edits and 72B-parameter models,10 $\times$ beyond prior practice, without sacrificing general capabilities.
By clarifying a long-standing misconception and introducing a principled localized tuning strategy, we advance fine-tuning from an underestimated baseline to a leading method for model editing, establishing a solid foundation for future research.

Published as a conference paper at ICLR 2026
FINE-TUNING DONE Right IN MODEL EDITING
Wanli Yangen en
Rui Tange
Hongyu Zang
Du Su
Qi Cao Jingang Wang Huawei Shen em
Xueqi Cheng Fei Sun o
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS #University of Chinese Academy Sciences yangwan wttp:aanni2ziiiiiii
sunffiliictacc
ABSTRACT
Fine-tuning, foundational method for adapting large language models, has long been considered ineffective for model editing. Here, we challenge this belief, ar- guing that the reported failure arises not from the inherent limitation of fine-tuning itself, but from adapting it to the sequential nature of the editing task, a single- pass depth-first pipeline that optimizes each sample convergence before moving on. While intuitive, this depth-first pipeline coupled with sample-wise updating over-optimizes each edit and induces interference across edits. Our controlled experiments reveal that simply restoring fine-tuning to the standard breadth- first (i.e., epoch-based) pipeline with mini-batch optimization substantially improves its effectiveness for model editing. Moreover, fine-tuning in editing also suf- fers from suboptimal tuning parameter locations inherited from prior methods. Through systematic analysis of tuning locations, we derive LocFT-BF, simple and effective localized editing method built on the restored fine-tuning frame- work. Extensive experiments across diverse LLMs and datasets demonstrate that LocFT-BF outperforms knowledge, it the first sustain 100K edits and aar-paraeeter models, 10 beyond prior practice, without sacrificing general capabilities. By clarifying a long-standing misconception and introducing principled localized tuning strat- egy, we advance fine-tuning from an underestimated baseline to leading method for model editing, establishing a solid foundation for future research. ?
methods by large margins. Notably, to our
"It ain't 't what you don't 't know that gets you into trouble. It's what you know for sure that just ain 't so."
Mark Twain
INTRODUCTION
Model editing has emerged as a promising approach efficiently update knowledge in Large Lan-
guage Models (LLMs) without costly retraining (Yao et al., 2023; Wang et al.] 2024d). In response, various algorithms have been explored, including parameter-extenset
(Hartvigsen et al. 2023]
Wang et al. 2024b), meta-learning (Mitchell et al. 2022 Li et al. 2025), and locate-then-edii (Meng
al. 2022 2023: Fang et al. 2025) methods. contrast to these specialized approaches, direct fine-tuning, widely recognized and effective method for adapting LLMs (Zhao et al.) (2025). has nevertheless been consistently dismissed in model editing as a weak baseline, typically attributed to overfitting and catastrophic forgetting (Zhu et al. (220)). This contradiction raises critical question: is fine-tuning inherently unsuitable with model editing, or have we simply been using it wrong?
In this paper, argue that this discrepancy arises from the way it has been commonly applied in model editing studies, not the method itself. Our analysis of existing codebases reveals that fine- tuning in model editing deviates from the standard paradigm, reshaped to match the editing task where edit requests naturally arrive one by one. Concretely, rather than iterating over the entire dataset across epochs, it adopts single-pass procedure, repeatedly optimizing each edit until fully
Corresponding author: (sunfei@ict.ac.cm
Our code is available at /tttpp///gicchbbb.com TOTTSTAR/// SSTAR/

---

Published as a conference paper at ICLR 2026
"memorized" before moving to the next. To distinguish the two, refer to the conventional form as fine- finetuniin with a breadth-first (BF) pipeline and its model-editing adaptation as fine-tuning with depth-first (DF) pipeline (typically with batch size 1). Through this lens, two inherent issues of the depth-first pipeline emerge:
i) its single-pass depth-first pipeline suffers catastrophic forgetting as later edits overwrite earlier ones;
ii) its sample- wise optimization (i.e., batch size of 1) tends produce high-variance gradients, destabilizing the edited model's general capabilities.
To validate this hypothesis, we conduct controlled experiments on representative fine-tuning based editing methods (c.g., FT-M and AdaLoRA (Zhang et al. 2024)) via two orthogonal modi- fications:
DDF pipeline BF pipeline
100
i) Pipeline: switching from depth-first to breadth- first pipeline while keeping batch size fixed at I;
Score (9)
Fre edied
ii) Granularity:
substituting per-sample updates (batch size 1) with standard mini-batch optimization under the breadth-fin first pipeline.
0
Reliahility
Generalization
Capability
As illustrated in Figure D our first adjustment to the optimiza- tion pipeline alone yields substantial improvements the edit- ing task. This suggests that the breadth-firg pipeline effectively mitigates catastrophic forgetting, long-standing weakness of- ten criticized in fine-tuning for model editing. Building on the first step, the second controlled ex- periment changes only the update granularity within the breadth-first pipeline, thereby substantially reducing the degradation of general capabilities in edited models. Overall, we show that simply restoring fine-tuning based editing baselines the standard breadth-first pipeline with mini-bated optimization yields unexpectedly strong performance on editing tasks a result surprising at first glance, yet reasonable in hindsight.
Figure I: Pipeline comparison of FT-M LLaMA3-8B with 1000 ZsRE samples.
Despite these encouraging results, the fine-tuning variants we revisited still retain ad-hoc practices from prior model editing research. Specifically, they tune parameters at the locations identified by locate-then-edit methods, which are often suboptimal. This leads to a key yet underexplored question: which layers and modules of an LLM are most effective to tune for model editing?
To answer this question, we conduct systematic study of tuning locations across layers and mod- ules (c.g., attention and MLP) in diverse LLMs. Our experiments reveal that, although optimal configurations can vary across models, general pattern emerges: tuning the down- or up-projection matrices in later layers often achieves near-perfect editing success while preserving general capabil- ities. Notably, this strategy remains highly effective when not optimal.
These analyses lead to LocFT-BF (Localized Fine- Mine-T with Breadth-Fiin pipeline), a simple and effective model editing method that restores fine-tuning to its principled configuration: breadth-firs pipeline, mini-batch gradient aggregation, and localized parameter updates. Unlike existing meth- ods, LocFT-BF avoids the typical overhead of prior approaches: matrix precomputation required by locate-then-edit methods, additional labeled data required by meta-learning methods, and architec- tural modifications required by parameter-extensio methods. This principled simplicity makes easy to implement, efficient to run, and broadly applicable across architectures.
To evaluate the effectiveness of our method, we conduct extensive experiments on multiple repre- sentative LLMs and datasets. On the widely adopted lifelong editing task, LocFT-BF substantially outperforms state-of-the-art methods, exceeding the best baselines by an average of 33.72% edit- ing success rate while consistently maintaining general capabilitiess of edited models. To further test its limits, we push evaluation to two extremes: 100K sequential edits and 72B-parameter model, both order of magnitude beyond mainstream practice, thereby reflecting scenarios closer to real- world applications. To our knowledge, this is the first method in model editing research that can sustain 100K sequential edits while preserving general capabilities and scales efficiently with stable performance from 7B to 72B models. These results overturn the mtanding view of fine-tuning weak baseline and highlight its potential scalable solution for model editing.
2

---

Published as a conference paper at ICLR 2026
Steps
Fy Steps
Eppochss
Original LLM
Sample
Sample n
Edited LLM
Original LLM
Batch
Butcch
Edited LLM
Depth-First Pipeline (Editing Pipeline)
Breadth-FFrst Pipeline (Standard Pipeline)
Figure 2: Illustration of edit pipeline (Depth-Firy) and standard pipeline (Breadth-Fing First).
2 IMPLEMENTATION MATTERS IN FINE-TUNING
This section revisits the widely reported underperformance of fine-tuning in model editing (Wang et al. 2024b] Fang et al. 2025). identifying its root causes in flawed training pipeline and showing how correcting restores fine-tuning as a competitive editing approach.
2.1 BACKGROUND
Model editing aims to efficiently revise specific factual knowledge in LLMs through localized pa- rameter updates while preserving unrelated knowledge and capabiliities. Formally, let denote LLM that encodes a fact triple t = (s, o) of subject 8, relation r, and object o. Given a desired update e' rs o') where o' j o, the editing algorithm & computes parameter shift such that the updated model fa- predicts o' when provided with the prompt p(s, r). For instance, update the US presidency, & ensures that fer outputs o' Donald Trump instead of o= Joe Biden with prompt r(s,r) = The president of the United States is.
0"
In real-world applications, knowledge updates emerge as a continuous stream. Consequently, the model editing task is typically formulated as a sequential process, in which the model adapts to new edits successively over time. This setting, formally known as sequential model editing (or lifelong model editing), involves cumulative editing trajectory fe, .... fo.. where at step k. the edited model fag required to correctly encode all target facts.
2.2
MIS- SPECIIECDD IMPLEMENTATION
Through detailed examination of existing fine-tuning based editing methods (Zhu et al., 2020 Wang al. 2024a). we find that they are typically implemented following the logic of editing tasks rather than the standard fine-tuning paradigm, simulating the sequential arrival of knowledge updates. Specifically. they adopt a sample-by-sample training procedure: repeatedly optimizing each sample to convergence before advancing to the next. We refer to this approach as a Depth- First (DF) pipeline. In contrast, as illustrated Figure 2 the standard Breadth- Frradth-Firsttt (BF) pipeline differs in two key aspects: ! it iterates over the entire dataset across epochs and @ employs mini-batch updates rather than sample-wise optimization. While the DF design appears intuitive the context editing tasks, it is not necessary choice in practice: wal-world edits rarely arrive strictly sample by sample, and even when updates are sequential, the BF pipeline can incorporate them incrementally. More importantly. the DF pipeline inherently introduces risks: I the sequential and singleppas: nature makes it prone to later edits overwriting earlier ones; @ the sample-wise optimization tends to produce unstable gradients, which destabilize the general capabilities of edited models.
2.3 IMPACT OF TRAINING PIPELINE
To validate our hypothesis regarding pipelines, we compare DF and BF in controlled experiments, fixing batch size at 1 and keeping other settings constant. We evaluate four representative tuning based methods: FT-L (Zhu et al. 2020), FT-M (Zhang et al. 2024), AdaLoRA (Zhang et al. 2023
and RoseLoRA (Wang al.
2024a), on two mainstream datasets: ZsRE (Levy et al. 2017 7) and
COUNTERFACT (Meng et al.
2022). using three popular LLMs: LLaMA-3-88 -n-trrut Grattafiooi
et al. 2024 Mistral-7B-v0.1 (Jiang al. 2023), and Q
(Qwen et al. 2025).
Performance is measured with three metrics: Reliability, Generalization, and Capability. Further
details of the methods, datasets, I.LMs, and metrics provided in 8 4.. 1.. and Appendix A.1
3

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison of DF and BF pipelines on mainstream fine-tuning based editors.
ZsRE
COUNTERFACT
Method
Reliability
Generalization
Capability
Reliability
Generalization
Capability
%
100
DF
BF
DF
BF
DF
DF
BF
BF
BF
90
LLAMA
FT-1
0.00
0.00
0.00
0.10
14,96
23.47
0.00
0.00
0.00
0.00
15.14
18.17
FT-M
75,30
99.70
67.20
91.80
28.30
39.30
80.00
99.90
52.60
75.10
29.89
30.87
A
3,80
90.50
3,30
69. 00.70
44.81
49.89
8.40
96.30
6.10
44.10
49.88
39.27
RoseLoRA
0.30
1.00
0.00
0.70
57.13
57.38
0.30
0.20
0.00
0.00
57.17
56.66
FT-I
0.00
0.00
0.00
0.00
15.73
15.38
0.00
0.00
0.00
0.00
14.93
15.34
Misral
FT-M
41.10
100.00
24.60
83,50
18. 14
15.64
59.70
99.90
25.60
59:10
16.94
20.50
AdaLoRA
2.50
88.50
2,40
74.
25,36
41.27
5,10
95.50
3,90
44.70
19.14
39.63
RoseLoRA
0.20
4.10
0.00
3,60
45,24
43.80
0.60
0.80
0,20
0.20
45.57
43,97
FI-L
0.10
0.10
0.00
0.10
23.65
29.66
0.10
0.60
0.10
0.50
31.21
33.07
Queen
FI-M
58.70
99.80
34.60
77.60
25.41
34.28
67.70
99.90
25.60
35.40
32.57
33.17
AdaLoRA
3.40
90.60
2.50
54.00
53.47
51.21
8.90
97.00
4.00
19.10
41.74
53.23
RoseLLoRA
0.00
0.40
0.00
0.10
58.73
58.58
0.20
0.40
0.10
0.10
58.48
58.37
Depth-Firee Pipeline
Breadth-First Pipeline
1.0
Shard 5
0.51
0.71
0.76
0.86
0.96
Epoch
0.99
0.99
0.98
0.98
0.97
0.9
Editing Progross
Shard
0.62
0.78
0.86
0.95
Epoch
0.97
0.8
Fpoch
Shard
0.73
0.94
Epoch
0.80
0.82
0.82
0.78
0.80
6.5
Shard
0.85
Epoch
0.46
11.46
0.46
01.46
0.49
6.4
Shard 1
Shoud 2
Shard 3
Shard 4
Shard 5
Shord
Shard 2
Shard
Shard 4
Shard 5
Evaluation Progress
Evalustion Progress
Figure 3: Visualization of the learning dynamics for depth-first and breadth-first pipelines.
Table shows that replacing the DF pipeline with BF improves performance across all methods, with especially large gains for FT-M and AdaLoRA. FT-L and RoseLoRA show only minor im- provements due to their design constraints: FT-L optimizes only on the last token the target answer, while RoseLoRA excessively limits the scope of trainable parameters. Overall, these results highlight the critical role proper training pipeline in effective knowledge editing.
Although transitioning from DF to BF pipeline yields substantial gains, especially in editing success, the mechanism behind this gap remains unclear. To investigate this, we compare the learning dy- namies of the two pipelines using FT-M LLaMA-3-8B- -nn-tructttt with 1000 ZsRE samples. Specif- ically, we randomly partition the 1000 samples into five equal shards and track the average editing success of each shard during learning. For DF pipeline, the shards are edited sequentially; and af- ter completing each shard we evaluate on all previously edited ones to detect potential overwriting. While for BF pipeline, all samples are shuffled each epoch, and after every epoch evaluate each shard separately.
The visualization results are depicted in Figure Under DF pipeline, earlier shards start with high success rates but decline as later shards are edited. In contrast, under BF pipeline, all shards improve jointly across epochs and converge to near per--eerect success rates. These results confirm our hypothesis that the mis-specified DF pipeline induces catastrophic overwriting of earlier edits.
2.4 IMPACT GRADIENT AGGREGATION
While fixing the training pipeline markedly improves editing success and generalization. the edited models still show clear degradation in general capabilities, especially for FT-M. We next validate the second hypothesized drawback: per-sample updates (batch size 1) destabilize the edited model. To test this. we replace per-sample updates with mini-bate training within each epoch for both FT-M and AdaLoRA oRA, further aligning fine-tuning-bat based editing with standard practice.
4

---

Published as a conference paper at ICLR 2026
ILMMA--B-FT-M
Mistal-BBr TB-FT-M
Qwent. 5-78-F---FF-M
HLMAA--B-
Mistal- ARtAAaL.RRAA
A
Reliability
Generalization
Capability
0.2
Score
0.6
01
16
32
2
8
32
2
8
32
64
Batch Size
Batch Size
Batch Size
Figure 4: Impact of batch size on editing performance under the BF pipeline ZsRE
As shown in Figure increasing the batch size substantially improves the capability performance of FT-M to even surpass AdaLoRA, indicating that mini-batch training stabilizes model states during editing. Conversely, AdaLoRA attains modest gains due to its low-rank design, which already regu- larizes updates and preserves downstream performance. While larger batches reduce generalization to some extent, more noticeably for AdaLoRA and slightly for FT-M, this effect can be readily mit- igated through data augmentation (Gangadhar & Stratos, (024), given that each edit currently relies on only single QA pair. Similar observations regarding the negative impact of large batch sizes on generalization have also been reported by Keskar al. (017)).
2.5 BREADTH-FIRE PIPELINE FOR SEQUENTIAL EDITING
To address sequential editing in real-world scenarios, the BF pipeline offers flexible adaptation strategies. In strictly real-time settings, each incoming edit immediately integrated into the cumu- lative dataset to trigger new fine-tuning process. In practical deployments where latency constraints are relaxed, efficiency can be further optimized by buffering new requests into small batches before updating. Both strategies are highly feasible, given that the fine-tuning process is lightweight (as shown in Section 4.2) and the editing workload is negligible compared to pre-training corpora.
In summary, aligning fine-tuning based editing with the standard breadth-first. mini-batch paradigm repositions it from perceived weak baseline to a competitive approach. This finding rectifies a long-standing misconception: the widely reported failure arose not from the inherent limitations of fine-tuning, but from the ill-suited implementations in prior work.
TAILORING FINE-TUNING FOR MODEL EDITING
Our studies demonstrate that transitioning to standard fine-tuning pipeline substantially improves editing performance. Nevertheless, existing fine-tuning variants remain coarse and insufficiently designed, typically tuning the location identified by or empirical support, resulting in suboptimal generalization and capability. This raises critical question: which locations of an LLM are most effective to tune for model editing?
methods without theoretical
To address it, we conduct comprehensive study of tuning locations to establish rigorous basis for parameter selection in fine-tuning based editing. Specifically, we examine the same three LLMs as in earlier experiments, along two dimensions: layer and module. For each layer, we test five candi- date modules: entire layer, full attention, full MLP, MLPup- and aePa This design is motivated by different emphases in the literature: parameter-efficient fine-tuning (Hu et al.] 2022) typically targets the attention modules for downstream adaptation, whereas editing (Meng et al. (022) and mechanistic studies (Geva et al. I highlight the MLP, especially MLP down. the locus of factual knowledge. For example, this yields 160 tuning locations in LLaMA3-8D (32 layers x 5 modules), each fine-tuned independently. We also evaluated full hall-parrmeter and multi-layer fine-tuning; how- ever, their poor performance and disruption general capabilities led us to exclude them.
For the editing setup, perform localized fine-tuning 1000 ZsRE samples and evaluate three metrics: Reliabiliry, Generalization, and Capability (assessed on three representative tasks, GSM8K, MMLU, and WMT16, reduce resource cost). Corresponding results are presented in Figure
5

---

Published as a conference paper at ICLR 2026
Pre-edited
Entire
Azta
MLP
NLP
MLP MLFossn
LLaMA3-8B
Mistral-7B
Qwen2.5-7B
10
1.0
Reliabiity
0.4
04
0.2
0.2
02
1.0
1.0
Generalizaiin
0.8
0.8
82
0.2
02
80
86
0.0
Cspability
0.4
0.4
0.4
0.0
0.0
1.
20
25
10
15
20
25
Layer
Layer
Layer
Figure 5: Fine-tuning performance for different locations across three LLMs.
Reliability Perspective. Across all models, fine-tuning nearly any module in any layer achieves near-perfect editing success, suggesting that knowledge acquisition capacity is broadly distributed rather than confined to specific parameters. This is consistent with recent studies challenging knowl- edge localization (Chen et al. 20255) and showing that both attention and MI.P modules can encode knowledge (Wei et al. 2024a Li et al. (224). However, tuning the entire layer, full attention, or full MLP shows slight instability, with occasional drops at certain layers. In contrast, MLP down is the most stable, consistently maintaining high reliability across layers, consistent with prior findings that it is particularly specialized for knowledge update (Geva et al. 2021 Meng et al.) 2022).
Generalization Perspective. Unlike the uniformly high reliability, generalization varies notably across layers and modules, though the overall pattern is consistent across all three LLMss. From the module perspective, coarser-grand garneer-rained components such as entire layer and full MIPP generalize better than finer- meer-grained counterparts like MLP, up and MLP dowa, suggesting broader parameter updates may facilitate more robust memorization and stronger generalization to paraphrased prompts. From the layer perspective, generalization follows consistent trajectory: it peaks in early layers (e.g.. layers 3-8), declines in middle layers, shows a smaller secondary peak in later layers (e.g., layers 20-25), and then declines again. This highlights the non-trivial impact of layer choice and the need for systematic empirical assessment in guiding effective selection.
Capability Perspective. Unlike generalization, which favors
components, capability is stronger and more stable with fine-grained modules, i.e., MLPup and MLP down- This shows that more localized updates cause less disruption, helping edited models preserve general capability. From the layer perspective, capability follows a trajectory similar to generalization, with peaks in early and later layers. However, unlike generalization' 's preference for the early layers, capability is more pronounced in the later layers, further underscoring their trade-off.
Overall, since most locations yield high reliability, tuning lo- cation selection mainly depends on the trade-off between gen- eralization and capability. We prioritize capability for two rea- sons: preserving LLMSs' general abilities is essential for edit- ing: generalization, even if suboptimal, can be improved with data augmentation (Gangadhar & Stratos 2024), as cur- rent editing uses only one QA pair per fact. Based on this and experiments, identify the relatively optimal tuning locations for each LLM, as in Table 2 Besides, the results reveal a consistent pattern: editing MLP MLPaown in the later layers minimizes ca- pability degradation while retaining near-perfect reliability and acceptable generalization. Notably,
Table 2: Selected tuning locations.
Model
Tuning Location
LLaMA3-8B
MLP sown in layer 22 entire layer 23
Mistral-7B Qwen2.55-B
MLP down in layer

---

Published as a conference paper at ICLR 2026
this strategy remains highly effective even when not optimal. For instance, although Qwen performs best in earlier layers, later layers still achieve strong results. We further validate the universality of this strategy in Appendix A.2 demonstrating that it generalizes seamlessly and achieves robust performance on Mixture-of-Experts (MoE) models and unseen QAEdit dataset (Yang et al. 2025)
Integrating three key factors, namely breadth-first pipeline, mini-batch gradient aggregation, and tuning-location selection, yields LocFT-BE, a simple yet powerful fine-tuning method for model editing. Owing the inherent simplicity and extensibility of fine-tuning, LocFT-BF is broadly applicable, plug-and-play, and avoids the overhead of locate-then-edit (matrix precomputation) and meta-learning (extra training). Although LocFT-BF requires specifying tuning locations, our study offers an initial, relatively stable selection strategy to address this.
4 BENCHMARKING LOCFT-BF IN LIFELONG EDITING
To position LocFT-BF against state-of-the-an editing methods, conduct comprehensive evalu- ation across diverse LLMs and datasets under a lifelong editing setup.
4.1 EXPERIMENTAL SETUP
Editing Methods. To ensure comprehensive coverage, we compare LocFT-BF against six rep- resentative editing methods across three categories: parameter-eetenniion (WISE, Wang et al.]
2024b), meta-learning (RLEdit, Li et al. 2025 and UltraEdit, Gu et al. 2025), and locate-then-
edit (MEMIT, Meng et al. 2023 RECT, Gu et al. 2024 and AlphaEdiit, Fang al. 2025).
Edited LLMs. Following prior work (Fang et al. 2025 Gu et al. 2025). we evaluate three leading open-source LLMs:
(Grattafiori et al. 2024), Misvral-7B-v0.1 (Jiang et al.,
2023 and Qwen2.5-7BB - (Qwen et al., 2025).
Editing Datasets. In line with prior studies (Wang et al. 2024b; Fang et al. 2025), we randomly sample 3000 instances from ZsRE (Levy et al. 2017). COUNTERFACT (Meng et al. 2022). and
WikiBigEdit (Thede al. 2025) for large-scale editing evaluation. Beyond these mainstream factual editing benchmarks, further evaluate the applicability of editing methods benchmark, MedEditBench (Chen et al.) 2025a), with details provided in Appendix A.3
medical editing
Evaluation Metries. We evaluate editing techniques from four key properties: Reliability: suc- cess rate of editing; @ Generalization: adaptability of edited knowledge to rephrased prompts; @ Capability: preservation of general capabilities, measured as the average accuracy of edited mod- els on MMLU (Hendrycks et al. 2) Natural Questions (Kwiatkowski et al. 2019). SST2 (Socher) al. 2013), WMT16 (Bojar al. 2016), and GSM8K (Cobbe al.) 202 Efficiency: average time to perform each edit. Notably, we evaluate Reliability and Generalization using the WILD framework (Yang et al.] 2025). which employs autoregressive decoding instead of conventional teacher forcing generation to align with practical deployment scenario and avoid overestimation.
Detailed information regarding method implementation, LLM configuration, dataset preparation, and evaluation procedures is provided in Appendix A.1
4.2 RESULTS & ANALYSIS
The results are reported in Table 3 To structure the analysis, we examine them from four evaluation perspectives: reliability, generalization, capability, and efficiency.
Reliability. Remarkably, LocFT-BF consistently attains the highest reliability across all nine dataset-LLLM combinations, with absolute gains of 33.72% on average and up to 58.50% over the second-best. This demonstrates that fine-tuning can achieve effective knowledge updates. Al- though recent methods such as AlphaEdit, RLEdit, and UltraEdit achieve relatively strong reliability compared to traditional approaches, they exhibit notable instability across LLMs and datasets.
Generalization. LocFT-BF exhibits superior generalization, achieving optimal performance in six out of nine dataset combinations. The two least effective cases from the COUNTERFACT dataset, whose generalization prompts are constructed by prepending irrelevant text rather than di- rect paraphrasing, which makes generalization particularly challenging. Methods like MEMIT and

---

Published as a conference paper at ICLR 2026
Table 3: Comparison of LocFT-BF with existing methods on lifelong editing task. The best results are denoted in bold and the second-best results are underlined.
LLaMA3-8B
Mistral-7B
Qwen2,5-7B
Data
Method
Rel.
Gen.
Cap.
Time
Rel.
Gen.
Cap.
Time
Rel.
Gen.
Cap.
Time
Pre-edited
57.26
44.82
58.52
MEMIT
26.23
23.30
25.67
9.90
24.30
19.60
18.11
10.28
39.57
32.10
47.87
9.93
RECT
0.03
0.03
14.98
25.38
0.17
0.27
14.88
25.79
9.70
8.20
16.06
24.31
ZRRE
WISE
4.17
3,50
14.42
15.87
11.33
13.02
7.67
5.23
30.96
AlphaEdit
64.50
40.57
54.71
12.31
3.30
3.00
15.13
10.95
2.70
2.33
16.31
10.55
RLEdit
66.67
59.40
57.41
0.58
26.10
19.53
21.30
0.47
54.57
47.60
60.74
0.65
UltraFdit
34,93
22.67
55.93
0.22
40.43
23.40
44.22
0.04
40.10
19.77
59.25
0.27
LocFT-BE
98.97
75.83
57.04
0.27
98.93
49.57
42.73
0.31
98.87
74.37
59.07
0.49
MEMIT
71.90
48.47
19.30
9.34
37.83
28.17
15.59
9.06
68.37
40.90
44.00
8.70
COUNTERFACY
RECT
0.53
0.13
14.85
21.68
0.77
0.37
15.67
22.21
0.00
0.00
14.81
21.69
WISE
19.80
13.13
12.18
25.13
4.60
10.28
20.17
10.20
27.18
AlphaEdit
94.27
39,90
$4.09
10.60
6.67
6.70
15.47
9.75
32.17
18.10
17.46
9.46
RLEdit
65.33
33.23
55.45
0.46
36.30
17.07
23.53
0.40
44.33
58.48
0.45
UltraEdit
68.33
31,20
56.85
0.18
57.60
22.60
44,91
0,03
41.33
15.33
59.01
0.18
LocFT-BF
99.73
33.23
57.13
0.38
99.67
39.53
41.46
0.24
99.73
11.77
58.53
0.48
MEMIT
10.77
10.80
23.57
10.39
12.47
10.07
18.26
11.02
31.03
25.27
47.92
10.69
WikiBigEdit
RECT
0.00
0.00
14.90
30.43
0.00
0.00
14.81
30.41
1.87
0.97
14.78
27.96
WISE
30.33
27.03
15.14
38.03
32.53
11,50
34.40
30.63
33,80
AlphaEdit
64.73
51.17
54.14
14.18
3.37
3.57
14.69
12.05
4.83
3.67
15.13
11.68
RLEdit
71.10
63.27
57.28
0.60
60.83
46.47
36.20
0.38
66.40
58,50
59.36
0.47
UltraEdit
72.67
65.37
57.98
0.20
33.87
27 .0.0
25.43
0.05
74.20
59.17
58.82
0.31
LOcFT-BF
98.87
73.77
56.93
0.54
98,90
74.97
42.39
0.39
99.43
53,30
59.83
0.98
AlphaEdit explicitly enhance robustness to such noise through data augmentation, accounting for their relative advantage over LocFT-BF on this dataset. In Appendix A.4] we demonstrate that by incorporating the same data augmentation strategy, LocFT-BE achieves substantial gains in general- ization, effectively surpassing both MEMIT and AlphaEdit.
Capability. LocFT-BF and recently proposed methods, including AlphaEdit, RLEdit, and UltraEdit, exhibit advantages in preserving the general capabilities of edited models under large-scale editing. However, these baseline methods show noticeable instability across LLMs:: RLEdit and UltraEdit struggle Mistral-7B, while AlphaEdit, despite performing well on LLaMA3-8B, fails to general- ize to the other two I.LMs. In contrast, LocFT-BF demonstrates the strongest stability, consistently maintaining high capability across all evaluated LLMs and datasets. This highlights the key strength of fine-tuning: its simple and general design makes it broadly adaptable across architectures.
Efficiency. LocFT-BE, RLEdit, and UltraEdit achieve top-tier efficiency, completing each edit within one second by directly updating model parameters. Conversely, other baselines are nearly 50 times slower, primarily due to their more complex procedures, such as auxiliary module optimiza- tion (WISE) and additional matrix calculation (MEMIT, RECT, and AlphaEdit), which introduce substantial computational overhead.
In summary, LocFT-BF delivers superior performance across all key evaluation dimensions while maintaining strong stability, establishing it as an effective and robust technique for model editing. Notably, we also implement our method within LlamaFactory (Zheng et al. (024), observing com- parable performance with significantly enhanced efficiency, as detailed in Appendix A.5
SCALING TOWARDS REAL-WORLD EDITING
We further extend our evaluation to more realistic scenarios by scaling data volume and model size to 10x the scale of mainstream practice, thereby probing the limits of our approach.
5.1 SCALING TO LARGER DATA VOLUMES
Owing to the limited capacity of existing editing techniques, prior evaluations have typically been restricted to approximately 3000 edits. However, such scale is insufficient reflect real-world requirements for lifelong editing. To address the limitation, we scale the number of edits to better reflect practical scenarios.
8

---

Published as a conference paper at ICLR 2026
Table 4: Editing performance across Qwen2.5 models as scale increases from 7B 72B parameters.
Qwen2.5-7B
Qwen2.5-14B
Qwen2,5-32B
Qwen2.5-72B
Strategy
Rel.
Gen.
Cap.
Rel.
Gen.
Cap-
Rel.
Gen,
Cap-
Rel.
Gen.
Cap.
Pre-edited
58.52
62.56
63.72
64.84
Default Pos.
99.20
83.40
59.33
99.00
78.80
61.62
99.30
70.00
60.27
98.60
65.30
63.64
Proportional Pos.
99.20
83,40
59.33
99.50
77.90
61.12
99.40
68.20
64.72
99.30
68.30
63.87
RLEEdit
Ultraldit
LocFT-BE
Pre Pre-edited
Reliability
Generalization
Capability
0.6
LLaMA3-88
0.6
0.4
04
0.2
3K
12K
20K
50K
100K
3K
10K
20K
100K
3K
13K
20K
100K
0.6
Qwen2.
0.6
0.6
0.4
04
0.2
3K
13K
20K
5OE
120K
3K
10K
20K
SOK
1OOK
3K
13K
20K
50K
100K
Number Edits
Number Edits
Number Edits
Figure 6: Evolution of editing performance during the scaling process to 100K edits.
Experimental Setup. We increase the number of edits from 3K to 100K using the ZsRE dataset. For this experiment, evaluate LocFT-BF, RLEdit, and UltraEdit, the only three methods that maintain model capabilities under 3K sequential edits, on LLLMA3-8B and Qwen2.5-7B.
the leading baselines across all metrics when scaled to large edit volumes. Even at 100K edits, it main- tains near-perfect success rates while preserving the general capabilities of edited models. To our knowledge, it is the only method to achieve both effective knowledge updates and capability reten- tion at this scale. Although its generalization declines, LocFT-BE still substantially surpasses the baselines, and this aspect can be further improved with data augmentation. In contrast, RLEdit and UltraEdit struggle to scale, with model capabilities decline sharply when edits exceed 20K. More critically, their consistently low editing success rates reveal a fundamental limitation in achieving reliable knowledge updates.
Experimental Results. As shown in Figure 6 LocFT-BF exhibits significant superiority
5.2 SCALING TO LARGER MODELS
The complex designs of most existing model editing techniques lead to heavy computational over- head, which has confined mainstream evaluations 7B-level L.MMs. For example, locate teid approaches such as ROME and AlphaEdit require computing large covariance and projection ma- trices, incurring substantial memory and time costs that make scaling to larger models practically infeasible. In contrast, fine-tuning is inherently lightweight and easily extensible, allowing us to push beyond this long-standing 7B barrier and evaluate LocFT-BF on much larger models, reflect- ing practical deployment scenarios.
Experimental Setup. We apply LocFT-BF to edit series of Qwen2.5 models ranging from 7B to 72B. We evaluate with 1,000 ZsRE samples to balance evaluation coverage and computational fea- sibility for large models. Considering an exhaustive search for the optimal tuning position in larger models, such as Qwen2.5-72B, is computationally prohibitive, we design two heuristic strategies to determine the target layer, based on the optimal position identified in Qwen2.5-7B.
Default Position: we directly apply the same absolute position (i.e., Layer6.MLE cown) to all larger models, serving as straightforward baseline.
Proportional Position: preserve the relative depth of the target layer account for increasing model depth. For instance, since Layer 6 is the 7th layer in the 28-layer Qwen2.5-7B, we select the corresponding 20th layer (i.c., Layer 19.MLP for the 80-layer Qwen2.5-72B, corresponding to 80 (7/28) 20.
9

---

Published as a conference paper at ICLR 2026
Experimental Results. The results in Table 4 demonstrate that LocFT-BF readily scales to larger models, achieving reliable knowledge updates while maintaining general capabilities. Notably, both heuristic strategies for tuning position selection prove effective, eliminating the need for ex- haustive and computationally expensive position search. For mid-sized models such as Qwen2.5- 14B, the Default Position strategy is sufficient for strong performance, whereas for larger models like Dw3n2.5-32BB, the Proportional Position strategy yields better results. These results highlight that LocFT-BF can be seamlessly extended from 7B to 72B models without redesign, underscoring its simplicity and bug-and-play scalability in practical deployment. In contrast, scaling baseline methods to larger models remains challenging. Even the most competitive methods, RLEdit and UltraEdit, exhibit further performance degradation on 14B models, as detailed in Appendix [A.6]
Finally, to bridge the dimensions of data and model scaling, we evaluate LocFT-BF in a industrial- scale scenario: editing Qwen2.5-72B with 100K samples from ZsRE. In this challenging setting, our method achieves near-perfect reliability (99.80%) and robust generalization (57.60%) ), while effectively retaining the model's general capabilities (62.54% compared to the original 64.84%), confirming its practicality for real-world deployment.
6 RELATED WORKS
Model Editing Methodologies. Existing approaches to model editing can be broadly categorized into three groups. Parameete-eextension. These methods introduce additional trainable compo- nents decoupled from pretrained parameters to encode new knowledge. Representative designs in- clude extra neurons (T-Patcher, Huang al. (2023). codebooks (GRACE, |Hartvissen et al.. 2023
and auxiliary memory modules (WISE. Wang et al. 2024b). @ Meta-learning. KE (De Cao et al. (021) and MEND (Mitchell al.
2022) train hypernetwork to predict parameter updates for knowledge editing. RLEdit (Li et al. 2025) further enhances this approach with reinforcement adaptively produce parameter updates conditioned on the
learning, enabling the hypernetwork edited model's 'ss evolving state. @ Locate- then-edir. Building upon research into knowledge mech- anisms of LLMs (Geva et al. 2021 (022). ROME (Meng et al. 2022) and MEMIT (Meng et al.] 2023) apply causal tracing to identify Knowledge-critical parameters and apply localized updates for precise editing. AlphaEdit (Fang et al. 2025) augments this line by projecting parameter updates onto the null space of preserved knowledge to mitigate disruption in lifelong editing.
Fine-tuning Based Model Editing. FT-L (Zhu et al. 2020 Meng et al., 2022) and FT-M (Zhang eet al. 2024) directly apply fine-tuning to parameters identified by the locate-then-edit paradigm but suffer from catastrophic forgetting. To mitigate this limitation, recent studies have focused on two primary strategies. O Parameter--ffiiient fine- tuning. MELO (Yu et al. 2024) and RoseLoRA (Wang et al. 2024a) incorporate low-rank adaptation into model editing to constrain the interference induced by fine-tuning. @ Data augmentation. Since prior methods rely on single QA pair per fact, recent work (Gangadhar & Stratos] 2024] Wei et al. (024b) proposes to enrich the editing data, thus improving the acquisition of new knowledge during fine-tuning.
In contrast to previous research that focused on architectural refinements or data augmentation, our study is, to the best of our knowledge, the first to revisit the failure of fine-tuning-based model edit- ing, We attribute this failure to flawed implementation. By correcting this and customizing tuning locations, we have shown that ane-tuning can be an effective paradigm, disproving the misconcep- tion that it is unsuitable for model editing.
7 CONCLUSION
In this paper. we present the first re-examination of fine-tuning based model editing, a technique long regarded as a weak baseline. By identifying and correcting critical flaws in existing imple- mentations, we unlock the true potential of fine-tuning and overturn the prevailing misconception of its limitations. Building on this, we propose LocFT-BE, a pure localized fine-tuning approach grounded in comprehensive empirical analysis of tuning positions. Extensive experiments demon- strate that LocFT-B- -FF not only significantly surpasses state-of of-the-art art editing methods but also scales effectively to massive editing workloads (100K samples) and large models (up to 72B parameters). This work redefines the role of fine-tuning in model editing, establishing it as a powerful and prac- tical technique, while pushing the field closer to real-world deployment.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the Beijing Natural Science Foundation (4252023), the Strategic Prior- ity Research Program of the Chinese Academy of Sciences (XDB0680201).
ETHICS STATEMENT
This paper adheres to the ICLR Code of Ethics. All data used in research publicly available and contain no personally identifiable or sensitive information, thereby posing no privacy concerns. While the proposed editing method LocFT-BF has the ability modify the knowledge of LLMs and thus carries potential risks of introducing false or harmful information, its intended purpose is pos- itive, aiming to correct misinformation and bias in pretrained LLMs. We therefore urge researchers to apply this technology responsibly and with due caution.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our findings, we have made the source code open-source and docu- mented detailed experimental settings and configurations in Appendix A.I
LLM USAGE
We employ LLMs to polish the manuscript, focusing correcting grammatical errors and enhanc- ing clarity, rather than generating new content or ideas. The authors take full responsibility for all the content of the paper, including any refined by the LLMs, and ensure that the usage of LLMs adheres to the ethical guidelines.
REFERENCES
Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Ne- gri. Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 con- ference machine translation. In Proceedings of the First Conference on Machine Translation, pp. 131-198, Berlin, Germany, August 2016. Association for Computational Linguistics. URL
http: http://wwww.m
http://wwww.wwww...b.//.b aww.eb. org/ httgg//nnhologgo W/ R/W1kK 6/W16 2301
Shigeng Chen, Linhao Luo, Zhangchi Qiu, Yanan Cao, Carl Yang, and Shirui Pan. Beyond memorization: A rigorous evaluation framework for medical knowledge editing, 2025a. URL
https://arxivv org/ a 2506. 03490
Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Knowledge localization: Mission not accomplished? enter query localization! In The Thirteenth International Conference on Learning Representations, 2025b. URL https: opw.reviewweeeeeo net./ htt.rrrwwwee.eeeteoouniiieeeeee.
bvFZ0K
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano. Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL watttss://arxtt
h/abs/2110) 14168
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6491-6506, Online and Punta Cana, Dominican Republic. November 2021. Association for Computational Linguistics. URL http::::///allaatoooooo..o.///// http://2222.emepmmaa.....
22/
Junfeng Fang. Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained model editing for language models. In The

---

Published as a conference paper at ICLR 2026
Thirteenth International Conference Learning Representations, 2025. URL https://open
review. wwt. htte://orun...--BSySYE...
Govind Krishnan Gangadhar and Karl Stratos. Model editing by standard fine-tuning, In Findings of the Association for Computational Linguistics: ACL 2024, pp. 5907-5913, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL htttps://aclaathhloogg..o rg/2024. findings acl 3527
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. htthstt:: zwwodo.oddo.ooooooooooooooooo... wwtpreeoows//2600eee...500
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-55 5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https::/aclaanthoooogg. org/2 021 emn www.mennmeeeee... ma in
44
Mor Geva, Avi Caciularu, Kevin Wang. and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pp. 30-45, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https:
//aClattholooyy.cc
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and et al. The Ilama herd of models, 2024.
URL httpt::t//aaiiii// wtt.:///./a........... 21783
Jia- Cia-Chen Gu, Hiao- Xu, Jun Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Wei Chang, and Nanyun Peng. Model editing harms general abilitics of large language models: Regularization to the rescue. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pro- cessing, pp. Linguistics. URL
Miami, Florida, USA, November 2024. Association for Computational wwwp://h...ooy.../222o.... emw.22224...mm.....mmm emn 1p maim 934/
Xiaojie Gu, Guangxu Chen, Jungang Li, Jia-Chen Gu, Xuming Hu, and Kai Zhang. Ultraedit: Training- subject- and memory-free lifelong editing in large language models, 2025. URL ht
Akshat Gupta, Anurag Rao, and Gopala Anumanchipalli. Model editing at scale leads gradual and catastrophic forgetting. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 15202-15232, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL
org/ 2024 findings acl.acc.... 9027|
Thomas Hartvigsen, Swami Sankaranarayanan. Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with GRACE: Lifelong model editing with discrete key-value adaptors. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL httpt:::p///.......... ttttpp:://ppeeeeee
foru/ (==00111Kxxxaac
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
pottps://openreview: net.riiwwweeeeeeeeeeemm httprrrreeeeeeeeeeo/ommmmt.e. foofforrmdddddd
m 3GmQ
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation large language models. In International Con- ference Learning Representations, 2022. URL
net/ hwt.:/unett/tttmm forum?
d=n2eVKeeFYEE
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong. and Zhang Xiong. Transformer-patee One mistake worth one neuron. In The Eleventh International Conference on Learning Representations, 2023. http:t/s::// /ttlittps////oeeevveeee... http///ppppe/eeeee/ee/eee /tt.teeo/utuaeeeeeee /tt////oommmmdooo.
GeGBPm
12

---

Published as a conference paper at ICLR 2026
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, and et al. Mistral 7b, 2023. URL https: 7/
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017. URL https: htt.:::://penzeviee
.net/forum?ii www.mmmmdddooooooooooooo
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Was Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. URL htttps://acae ant holoo gy. wt..rg//11-10000
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprchension. In Proceedings of the 21st Conference Computational Natural Lan- guage Learning (CoNLL 2017). pp. 333-342, Vancouver, Canada, August 2017. Association for Computational Linguistics. URL /tRLtttss///caaabooooo
org/ --1034
Xiaopeng Li, Shasha Li. Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. Pmet: precise model editing in a transformer. In Proceedings of the Thirty-Eightis AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAATT2/AAAT244AAAT AAAI Press, 2024. URL htt.t:::://di..rgg///// 1609/1169980000 2050//aai..0 www.17.222.......
Zherui Li, Houcheng Jiang, Hao Chen, Baolong Bi, Zhenhong Zhou, Fei Sun, Junfeng Fang, and Xiang Wang- Reinforced lifelong editing for language models. In Forty-second International Conference on Machine Learning, 2025. URL https: http::://pperrevieeee new. wwt.://ereeeeeeeeeee/emmaa forret/forummii
1j0Xprrfch
Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, 2022. URLhtts
s: s://openreview net / -ow.ommmdd-hhhAs565e
Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representaatonss 2023. URL https: ftttp:://operreviie wttp://ee//fooorooeoo forummrmmmmmmmmmm McbKMcc
nww.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D) Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022. URL
/ttps://openreree
net http://ormmmiiiie 2dD.ruuumi.d-DDDDeeEEEEEE
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL httph::tt//aaaa. /ttp://arxvvvvv org/ abs/ 2412 15115
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys '25, pp. 1279-1297, New York, NY, USA, 2025, Association for Computing Machinery. URL https: /ww.. org/ .1145/3689031 1.T.3.1..666000 3696075
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. (631-1642, Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL https: ac http://wwwww..eeee..oo///aaaaaaa
13

---

Published as a conference paper at ICLR 2026
Qwen Team. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters, Febru- ary 2024. URL httLhtps://// http:://weellll.n........
Lukas Thede, Karsten Roth, Matthias Bethge, Zeynep Akata, and Thomas Hartvigsen. Wikibigedit: Understanding the limits of lifelong knowledge editing in LLMs. In Forry-second International Conference on Machine Learning, 2025. URL https: http:::///penreviwww wet. Forum? ?id=
9NVm Bf7CS
Haoyu Wang, Tianci Liu, Ruirui Li, Monica Xiao Cheng, Tuo Zhao, and Jing Gao. RoseLoRA: Row and column-wist, sparse low-rank adaptation of pre-trained language model for knowledge editing and fine-tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 996-1008, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. URL https: //ac antho ogy. www..org//oooooooo.oo..oo...o... org/ 2024. emn 1a-malp-ii main.n
77
Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. WISE: Rethinking the knowledge memory for lifelong model editing of large language models. In The Tiihth- Annual Conference on Neural Information Processing net wor.//orum... ?id D-V-M--MJVVC
Systems, 2024b. URL
Peng Wang, Ningyu Zhang, Bozhong Tian, Zekun Xi. Yunzhi Yao. Ziwen Xu, Mengru Wang, Shengyu Mao, Xiaohan Wang, Siyuan Cheng, Kangwei Liu, Yuansheng Ni, Guozhou Zheng, and Huajun Chen. EasyEdit: An easy-to-use knowledge editing framework for large language mod- els. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics, pp. 93, Bangkok, Thailand, August 2024c. Association for Computational Linguistics. URL pettps://aclantiie
org. 2024 acl demos. 97
Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. Knowledge editing for large language models: A survey. ACM Comput. Surv. 57(3), November 2024d. URL
httpps://di..... org. 1ee.11111135555
Yifan Wei, Xiaoyan Yu, Yixuan Weng, Huanhuan Ma, Yuanzhe Zhang, Jun Zhao, and Kang Liu. Does knowledge localization hold true? surprising differences between entity and relation per- spectives language models. In Proceedings of the 33rd ACM International Conference on In- formation and Knowledge Management, CIKM '24, pp. 41 4121111282221 New York, NY, USA, 2024a. Association for Computing Machinery. /tt.:ppp///ooo..rrr0001111/////
779000
Zihao Wei, Liang Pang, Hanxing Ding, Jingeheng Deng, Huawei Shen, and Xueqi Cheng. Stable knowledge editing in large language models. CoRR, 130488 2024b. URL https:
//doi. 10. 48550//r2ii...... 2402. 2402.13048
Wanli Yang, Fei Sun, Xinyu Ma, Xun Liu, Dawei Yin, and Xueqi Cheng. The butterfly effect of model editing: Few edits can trigger large language models collapse. In Findings of the Asso- ciation for Computational Linguistics: ACL 2024, pp. 5419-5437, Bangkok, Thailand, August 2024a. Association for Computational Linguisties. URL https: http//ccaantoollyy..oooo
024 www.fiinninnaaae ac2.322/
Wanli Yang, Fei Sun, Jiajun Tan, Xinyu Ma, Du Su, Dawei Yin, and Huawei Shen. The fall of ROME: Understanding the collapse of LLMs model editing. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 4079-4087, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. URL |tttttps://accaame http//aaaaannntooooooooo org/2
024 www.ffiinnnggmmm.. 2036e336mmppp..mmm
Wanli Yang, Fei Sun, Jiajun Tan, Xinyu Ma, Qi Cao, Dawei Yin, Huawei Shen, and Xueqi Cheng. The mirage of model editing: Revisiting evaluation in the wild. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pp. 15336- 15354, Vienna, Austria, July 2025. Association for Computational Linguistics. URL https: http://aaaaatooolyyy.yoo org 72025. aal-aacllloogo. 7457
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. In
14

---

Published as a conference paper at ICLR 2026
Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, PP. 10222-10240, Singapore, December 2023. Association for Computational Linguistics. URL
latposs/aaclanthooooo...oog.
emnlp-main, 632/
Lang Yu, Qin Chen, Jie Zhou, and Liang He. Melo: enhancing model editing with neuron-indexed dynamic lora. In Proceedings of the Tighth AAAI Conference on Artificial Intelligence. AAAI Press, 2024.
httpas:/ddoii. orgei.org/1i http:/.1111119.000 1609//aai.....a. v38117. 29916
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng. Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang. Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. A comprchensive study of knowledge editing for large language models, 2024.
URL httpps:/arxiv..r http://wxivi.....aaa///// abs //t..///.....222 01286
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-effiient fine-tuning. In The Eleventh Inter- national Conference on Learning Representations, 2023. URL htttp:://ppenreiiii net
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, Yifan Du Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jian Yun Nic, and Ji- Rong Wen. A survey of large language models, 2025. URL
03. 8223
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp. 400-410, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL httpp:: /
acl-demos 38/
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models, 2020. URL https: aarii.vv g/ abs/ 2012.00122.0 00363
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 DETAILED EXPERIMENTAL SETUP
A.1.1 EDITING METHODS
FT-L (Zhu al. 2020 Meng et al. 2022 fine-tunes the MLP of a specific layer identified through causal tracing in ROME (Meng et al. 2022), while augmenting the objective with an Lx -nom regularization that explicitly limits parameter deviations between the original and edited models to mitigate side effects unrelated knowledge and capabilities. However, FT-L deviates from the standard fine-tuning paradigm by leveraging only the last token of the target answer as supervision, which severely undermines its editing success.
FT-M ZZhag et al. 2024 addresses the supervision limitation of FT-L by applying a cross cross-entrope entropy loss over the entire target answer while masking the prompt, thereby aligning more closely with the standard fine-tuning practices and yielding substantial gains editing success.
AdaLoRA (Zhang et al. 2023) enhances vanilla Low-Rank Adaptation (LoRA) by adaptively al- locating the parameter budget among weight matrices according to their importance score. ZZhang
al. 2024) directly adapt this technique model editing and report competitive results.
RoseLoRA (Wang et al., 2024a is a novel parameter-efficient fine-tuning (PEFT) method that in- troduces row and column-win sparsity on the product of low-rank matrices. This approach allows it to selectively update only the most critical parameters of a pretrained language model, ensuring efficient and precise updates while preserving the irrelevant knowledge.
WISE [Wang et al., 2024b) targets the lifelong editing task with dual-memory architecture com- posed main memory for pretrained knowledge, a side memory for edited knowledge, and a router that directs queries between them. By explicitly decoupling edited knowledge from pretrained knowledge, WISE effectively mitigates interference with the original model.
lifelong editing as reinforcement learn- ing problem, where target edits and the state of the edited model constitute the environment, editing losses serve rewards, and the hypernetwork is optimized at the sequence level as the policy. This design enables the hypernetwork to precisely track parameter changes LLMs during editing and generate accurate updates for lifelong knowledge editing.
RLEddit (Li et al.
2025) reformulates hypemnetwrr--baeee
UltraEdit (Gu et al. 2025) proposes a training., subject-, and memory-free editing framework that computes parameter shifts through lightweight linear algebra operations, enabling fast and consis- tent updates with minimal overhead. To support lifelong adaptation, it further employs a lifelong normalization strategy that continually updates feature statistics, allowing the model to adapt to distributional shifts while preserving consistency over time.
MEMIT (Meng al. 2023) extends ROME by scaling its mechanism from single-layer multi- layer editing. It first identifies knowledge-relevant layers and modules through causal tracing anal- ysis and then applies rank-one matrix update at the identified locations perform target edits. By propagating modifications across multiple successive layers, MEMIT enables efficient batch editing of lcrge-scale knowledge.
RECT (Gu et al. 2024) builds on ROME by introducing a regularization strategy to mitigate over- fitting and reduce noise induced by editing. Specifically, quantifies the importance of each weight element by the absolute value of its relative change and updates only the top-k% elements. This selective update effectively suppresses side effects while preserving editing performance.
Adit (Fang et al. 2025) augments locate-then-edit methods by projecting parameter changes onto the null space of preserved knowledge. This projection is theoretically proven keep the out- puts of post-edited LLMs unchanged when queried about preserved knowledge, thereby mitigating the issue of disruption.
To ensure fair and rigorous comparison, we standardize the method implementations in two spe- cific aspects. First, to maximize the editing efficacy of each method within available computational resources, adopt empirically tuned batch size while preserving all other algorithmic config- urations from their official implementations. Second, we standardize the target answer formatting. While previous model editing implementations omit the -nd-of-sequence (EOS) token, typically
16

---

Published as a conference paper at ICLR 2026
causing irrelevant incorrect generation after the correct answer Yang et al. 2025 standard fine- tuning practices strictly require it (Zheng et al. 2024 Sheng et al. 2025). Consequently, we append an EOS token to the target answers across all methods to ensure proper generation termination.
A.1.2 EDITED LLMS
1--MA-----B-Instuuce (Grattafiori et al. 2024) leading 8-billion-pan parammeer instruction-tune -undeed model from the LLaMA family of Meta AI. It is designed primarily for dialogue applications and outperform many existing open-source chat models on common industry benchmarks. In addition, it has been further optimized to enhance both helpfulness and safety.
Mistral-7B-v0.1 (Jiang al.) al.] (023) is resource-efficien yet highly capable foundation model with 7 billion parameters. It consistently outperforms LLaMA-2-13 across all evaluated benchmarks and even surpasses LLaMA- 1-34B on tasks involving reasoning, mathematics, and code generation.
Qwen2.5-7B-Instruee (Qwen et al. 2025) is superior instruction-tuned model with 7 billion pa- rameters, trained on 18 trillion tokens with over 1M supervised finetuning samples and multistage reinforcement learning. It demonstrates strong performance across language understanding, reason- ing, mathematics, and coding, offering competitive capability while being resource-efficiem.
All models adopt greedy decoding for generation, consistent with mainstream practice in model editing Yao et al. 2023 Wang et al. 2024c).
A.1.3 EDITING DATASETS
ZsRE (Lvv et al. 2017) a widely used dataset in model editing, originally developed for zero- shot relation extraction. Following its adaptation by De Cao al. (202I). ) the original answers were replaced with counterfactual ones to ensure that models had no prior exposure to the target facts, thereby providing a reliable benchmark for evaluating editing methods.
COUNTERFACT (Meng et al. 2022) is challenging dataset specifically curated for model edit- ing. It comprises 21,919 counterfactual statements whose target answers are initially assigned low probabilities by models, making it rigorous benchmark for evaluating editing techniques on more challenging modifications.
WikiBigEdit (Thde et al. 2025) is large-scale lifelong editing benchmark built through fully au- tomated data extraction pipeline that continuously incorporates new factual edits, ensuring long-term applicability for factuality evaluation. Its initial release contains over 500K question-ans pairs, providing a realistic setting for large-scale factual updates and better approximating deployment- time requirements.
Unlike the counterfactual knowledge in ZsRE and COUNTERFACT, WikiBigEdit consists of real- world facts from Wikipedia, some of which may have already been memorized by the evaluated LLMs. To ensure the learning of new knowledge, we excluded all samples that could be correctly answered by any of the three LLMs.
1.4 REPRESENTATIVE TASKS
We adopt Capability, measured via the Im-evaluation--anne Gao et al. 2024). in place of the traditional Locality metric (Yao et al.) (023) to more directly and rigorously assess whether editing perturbs unrelated knowledge and capabilities (Yang et al. 2024a Gupta et al. (024)..
MMLU (Hendrycks et al. 2021) is a massive multitask benchmark consisting of multiple-choice questions across 57 subjects, spanning elementary mathematics, US history, computer science, law, and other areas that are important for people to learn. To reduce the prohibitive evaluation cost of the full benchmark, we randomly sample 500 questions from each subject, resulting in a balanced test set of 28,500 instances for capability evaluation.
Natural Questions (Kwiatkowski et al., 2019) is an open- open-domain question answering benchmark, where queries drawn from real anonymized search requests issued to Google, and answers annotated from corresponding Wikipedia pages by human participators. We adopt its test set of 3610 question-answe pairs to evaluate the capability of target L.LMs.
17

---

Published as a conference paper at ICLR 2026
Table 6: Performance of LocFT-BF and baseline methods on the medical editing benchmark.
LLaMA3-8B
Mistral-7B
Qwen2.5-7B
Method
Rel.
Cap.
Rel.
Cap.
Rel.
Cap-
Pre-edited
57.26
44.82
58.52
RLEdit
58.92
58.34
21.18
30.36
60.43
61.43
UltraEdit
71.40
58.79
44.52
44.40
56.24
61.02
LocFT-BF 98.28
56.92
98.17
41.46
98.06
59.92
SST2 (Socher et al., 2013) is a sentiment classification benchmark derived from the Stanford Sen- timent Treebank. It consists of single sentences from movie reviews, each annotated with binary sentiment labels (positive or negative, with neutral cases removed). We utilize its standard test set to evaluate the sentiment classification capability of examined LLMs.
WMT (Bojar et al. 2016) a benchmark series from the Workshop on Machine Translation, cover- ing multiple years and language pairs. In our experiments, we adopt the WMT16 German-English (de-en) dataset to evaluate the translation capability target LLMs.
GSM8K (Grade School Math 8K) (Cobbe et al. 2021) is benchmark of 8500 high quality linguis- tically diverse grade school math word problems. It was created by OpenAI to support the task of question answering basic mathematical problems that require multi-step reasoning. We use its test set 1319 problems to evaluate the mathematical reasoning ability of LLMs.
A.2 APPLYING LOCFT-BF TO MoE MODELS
To evaluate the generalizability of LocFT-BF to unseen datasets and heterogeneous architec- tures, we extend our evaluation to a Mixture- of-Experts (MoF) model,
Table 5: Performance on MoE model.
QwenI.5-MoF-A2.78
Rel.
Gen.
Cap.
Pre-edited LocFT-BF
47.10
TTam, 2024). using 1000 samples from the QAEdit dataset (Yang et al.) 2025). Specif- ically, we employ the default locating strat- egy, directly editing the down-projen down-projection matrix of MLP in the mid-to-late layers, (i.e., model.layer: 20 mllp shared expert. dww..prowoo..ooo. pro The results in dablee]eemonstrraee that LocFT-BF and its default locating strategy generalizes strongly both new data and architec- tures, achieving effective and stable editing without disrupting the model's original capabilities. This cross-dataset and cross-ares architecture robustness highlights substantial advantage LocFT-BF over prior editing techniques, which typically exhibit high sensitivity to dataset and model variation.
96.30
40.70
45.56
A.3 EVALUATION ON MEDICAL EDITING
To assess the effectiveness editing methods in specialized domains, we evaluate LocFT-BF and the two most competitive baselines, RLEdit and UltraEdit on the medical knowledge editing bench- mark (Chen et al. 2025a), which comprises 930 medical edits. We apply these edits across three LLMS using the same tuning locations employed in the main experiments. Given the absence of rephrased prompts in this benchmark, our evaluation focuses on reliability and capability. As shown in T LocFT-BF effectively injects specialized medical knowledge while preserving the mod- els' general capabilities. Moreover, it significantly outperforms baseline methods, demonstrating that the superiority of LocFT-BF extends beyond general- domain factual edits.
A.4 INTERPRETING GENERALIZATION GAP ON COUNTERFACT
The observed generalization drop COUNTERFACT is a common issue for most editing tech- niques. The underlying reason lies in the dataset's unique evaluation protocol for generalization: instead of direct paraphrasing the edit prompt, COUNTERFACT constructs its generalization prompt by prepending irrelevant text to the edit prompt, as present in Figure 7 Such disruptive context makes generalization particularly challenging, especially for most methods that learn knowl-
18

---

Published as a conference paper at ICLR 2026
Table Performance of LocFT-BF with data augmentation on the COUNTERFACT dataset.
LLAMAA-8B
Mistral-7B
Qwen2.5-7B
Method
Rel.
Gen.
Cap-
Rel.
Gen.
Cap.
Rel.
Gen.
Cap.
Pre-edited
57.26
44.82
58.52
MEMIT
71.90
48.47
19.30
37.83
28.17
15.59
68.37
40.90
44.00
AlphaEdit
94.27
39.90
54.09
6.67
6.70
15.47
32.17
18.10
17.46
LocFT-BF 99.73
33.23
57.13
99.67
39.53
41.46
99.73
11.77
58.53
LocFT-BF-Aug
99.63
50.93
56.99
99.50
49.70
42.21
99.73
44.90
58.36
"Bdit Prompt" "Nhat sport doea Dave Winfield play? They play", "Rephrased Prompt" "Andrey Tcheboharev'n What aport does have Winfie'd play? They play",
"Augmented Templates": ["The 2019-20 season been. ()", "Therefore, must forget the importance of. 13", "Because am woman: The impact ot. (P", "I have to acmit, was bit. 63*, "I have always keen a fan of the. (}*)
Figure 7: An illustrative example from the COUNTERFACT dataset, including an edit prompt and its corresponding rephrased prompt. The figure also presents representative augmented context tem- plates used by MEMIT and AlphaEdit, where the edit prompt is inserted into the placeholder { to form diverse training prompts.
edge only from the edit prompt. However, methods such as MEMIT and AlphaEdit explicitly en- hance robustness by augmenting each edit prompt with several randomly prefixed text, shown in Figure 7 which explains their relative advantage on COUNTERFACT. Therefore, the lower general- ization of LocFT-BF on this dataset reflects the unique evaluation design rather than the limitation of fine-tuning, and can be mitigated by incorporating prefix-based augmentation necessary.
To substantiate this hypothesis, we apply LocFT-BE the COUNTERFACH dataset across three LLMs and adopt the same data-augmentation strategy as MEMIT and AlphaEdit, augmenting each edit prompt with five randomly prefixed texts. As shown Table 7 this simple augmentation markedly enhances generalization performance, placing LocFT-BF ahead of the augmented MEMIT and AlphaEdit, while preserving the general capabilities of edited models. These results further con- finm that the perceived generalization gap stems from the evaluation protocol and can be effectively bridged via data augmentation.
A.5 IMPLEMENTATION AND VALIDATION ON LLAMAFACTORY
The design simplicity of LocFT-BE, which relies solely standard mini-batch training and local- ized parameter updates, facilitates seamless integration with existing fine-tuning frameworks. To demonstrate this, we incorporate LocFT-BF into the widely-used LlamaFactory library. This imple- mentation not only validates the ease of reproduction of our method but also highlights its potential for efficient industrial deployment.
We employ the LlamaFactory implementation of LocFT-BF to replicate the experiments from Sec- tion with comparative results shown Table 8 Despite exhibiting marginal variance in reliability and capability, this implementation continues to significantly outperform existing editing baselines. Notably, it achieves substantial gains in generalization and efficiency, further extending the lead over competing methods. We attribute these variations low-level implementation differences between the frameworks, such as the granularity of loss computation (token-level averaging in LlamaFactory and sample-leve in ours) and the underlying engineering infrastructure. Collectively, these findings underscore the excellent adaptability and cross cross-framen framework robustness of LocFT-BE.
Furthermore, the comprehensive ecosystem of LlamaFactory facilitates direct comparison with Full- Pall-mrretttr Fine- Tine-Tunigg (FullFT), As presented Table while FullFT achieves superior re- liability and generalization target edits, it incurs catastrophic degradation in the model's ''s general
19

---

Published as a conference paper at ICLR 2026
Table Performance comparison our original LocFT-BF against LlamaFactory (LF) implemen- tations of both LocFT-BF and FullFT. The best results are denoted in bold and the second-best results are underlined.
LLMM3-8B
Mistral-7B
Qwen2.5-7B
Method
Gen.
Cap.
Time
Gen.
Cap.
Time
Rel.
Gen.
Cap.
Time
Pre-edited
57.26
44.82
58.52
ZsRE
FullFT (LF)
98.47
87.57
40.44
0.21
96.43
87.40
16.07
0.20
99.17
84.73
59.05
0.20
LocFT-BF (LF)
95.07
79.10
56.03
0.04
95.00
71.43
41.38
0.04
98.77
75.67
60.83
0.07
LocFT-BF
98.97
75.83
57.04
0.27
98.93
49.57
42.73
0.31
98 88.87
74.37
59.07
0.49
COUNTERFACT
FullFT (LF)
99.63
95.07
31.53
0.22
99.53
95.67
15.64
0.21
99.70
76.23
53.53
0.20
LocFT-BF (LF)
71.53
56.58
98.77
58
41.70
41.80
59.27
0.05
LocFT-BF
99.73
33.23
57.13
0.38
99.67
39.53
41,46
0.24
99.73
11.77
58.53
0.48
WikiBigEdit
FullFT (LP)
99.43
96.57
46.69
0.21
98.73
95.33
18.23
0.21
99.50
96.17
58.08
0.21
LocFT-BF (LF)
97.43
88.20
55.51
0.03
95.20
84.23
42.13
0.04
98.33
82.67
59.03
0.08
LocFT-BF
98.87
73.77
56.93
0.54
98.90
74.97
42.39
0.39
99.43
53.30
59.83
0.98
Table 9: Scalability of baseline model editing methods on larger models.
Qwen2.5-7B
Qwen2.5-14B
Qwen2.5-14B
Method
Rel.
Gen.
Cap.
Rel.
Gen.
Cap.
Rel.
Gen.
Cap-
Strategy
Default Position
Proportional Position
Pre-edited
58.52
62.56
62.56
RLEdit
37.30
24.90
60.62
8.90
5.70
60.46
30.50
19.60
62.19
UltraEdit
44.10
19.00
60.91
12.50
8.10
62.97
34.20
22.30
63.07
LocFT-BF
99.20
83.40
59.33
99.00
78.80
61.62
99.50
77.90
61.12
capabilities. This severe overfitting confirms that despite its efficacy in memorizing specific samples, FullFT is too destructive to viable solution for practical, continuous model editing.
A.6 SCALING BASELINE METHODS TO LARGER MODELS
Due the inherent technical constraints hindering the parallel training of most model editing tech- niques, we select the two most competitive and scalable baselines, RLEdit and UltraEdit, and eval- uate them on Qwen2.5- 14B using 1000 ZsRE samples. As both methods require choosing editing layers, we employ the two leler-selection strategies used for LocFT-BF: ! Default Position: directly using the same absolute editing locations as in the 7B model. Proportional Position: scaling the locations proportionally to model depth. As presented in Table 9 both RLEdit and UltraEdit ex- hibit substantial performance degradation even at 14B, indicating that their practical scalability severely limited, whereas LocFT-BF not only scales more easily from an engineering perspective but also maintains significantly better performance.
20


Reviews:
[{"review_id": "AdquYxLVhW", "reviewer": "Reviewer_QAtd", "summary": "This paper addresses the failure of fine-tuning in model editing by introducing two principled improvements. The authors argue that the failure of existing fine-tuning approaches mainly stems from the depth-first (single-pass) updating paradigm, which leads to catastrophic forgetting. To mitigate this, they restore the standard breadth-first training pipeline, optimizing all mini-batches across epochs rather than editing samples sequentially. Furthermore, by systematically analyzing which layers and modules are most effective for editing, the study finds that tuning the down- or up-projection matrices in later layers generally achieves near-perfect editing success while maintaining general capabilities.", "questions": "1. When new samples arrive, how does LocFT-BF dynamically reconstruct its epochs and batches? Moreover, how does it handle edits that conflict with previous ones?\n\n2. For a new, heterogeneous LLM and an unseen task, could the authors clarify how LocFT-BF determines or generalizes the choice of the optimal editing module?\n\n3. Could the authors provide a comparison of the time and computational cost between the breadth-first and depth-first fine-tuning pipelines?\n\nIf I have misunderstood anything, please feel free to point it out.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper conducts extensive experiments to validate their hypotheses and insights.\n\n2. The authors’ observation that the depth-first approach leads to instability and catastrophic forgetting is indeed convincing in my view.\n\n3. The paper provides clear and informative visualizations and ablation studies", "weaknesses": "1. While the breadth-first pipeline mitigates forgetting effectively, its computational cost grows linearly with the number of accumulated edits, as previous samples must be revisited in each fine-tuning pass. This may raise concerns about scalability in real-time or large-scale editing scenarios, despite the authors’ empirical demonstration of efficiency at 100K edits.\n\n2. While the breadth-first method alleviates catastrophic forgetting to some extent, it does not fundamentally eliminate the issue, since there is no guarantee that the model’s pretrained knowledge remains intact after editing.\n\n3. The locate strategy proposed in this paper appears somewhat vague and even inefficient. For example, in Section 5.2, the authors directly scale the editing location from Qwen2.5-14B to Qwen2.5-32B, which seems to rely on the assumption that the editing samples remain identical and that the two models share highly similar architectures. However, in real-world applications, the editing samples are dynamically updated, and the optimal editing module may vary accordingly. Simply transferring the previous settings to larger or more heterogeneous LLMs could therefore be unreliable, and the effectiveness of such an extension may degrade even further in those scenarios.\n\n4. At the same time, the method this paper uses to identify the optimal tuning location also appears somewhat inefficient, as they fine-tune across all layers and five candidate modules in order to determine the best position.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lHn84Vrtn5", "reviewer": "Reviewer_EfTQ", "summary": "This paper focuses on the long-term underestimation of fine-tuning methods in the field of large language model (LLMs) model editing. Through in-depth analysis of the shortcomings of existing fine-tuning implementations, an innovative LocFT-BF method is proposed, which restores fine-tuning to a standard breadth first (BF) pipeline and combines local parameter tuning. The paper verified the effectiveness of the method through systematic control experiments, surpassing existing mainstream model editing methods on multiple models and datasets, and achieving stable editing of 100000 edits and 72B parameter models for the first time. The research design is rigorous, the experimental scale is sufficient, and it has important theoretical and practical significance for the field of model editing. However, there is still room for improvement in terms of theoretical depth, adaptability to special scenarios, and supplementary experimental details in the paper.", "questions": "- Beyond the empirical evidence of catastrophic overwriting in the DF pipeline , what is the theoretical justification for why the BF pipeline with mini-batching  provides superior stability and mitigates catastrophic forgetting so effectively during sequential editing?\n- How does LocFT-BF perform on benchmarks designed to test for hallucination (e.g., TruthfulQA) or complex multi-step reasoning (e.g., BBH)? Does the localized fine-tuning approach maintain robustness in these more challenging generative scenarios post-editing?\n- The authors argue that the generalization gap can be offset by data augmentation . Have experiments been conducted to support this claim, and if so, does augmentation successfully improve generalization without compromising the method's high reliability and capability scores?\n- How well does the LocFT-BF framework, including its specific layer-selection strategy (e.g., later-layer MLPs) , generalize to editing highly specialized, high-stakes knowledge domains (e.g., legal or medical) or to multilingual knowledge editing tasks? Although this article proposes a new editing method, it focuses more on the analysis of knowledge editing methods. Therefore, we hope that the author can explore more extensively and provide more valuable insights.\n\n[1] Zhang X, Liang Y, Meng F, et al. Multilingual knowledge editing with language-agnostic factual neurons[J]. arXiv preprint arXiv:2406.16416, 2024.\n\n[2] Wang W, Haddow B, Birch A. Retrieval-augmented multilingual knowledge editing[J]. arXiv preprint arXiv:2312.13040, 2023.\n\n[3] Wei Z, Deng J, Pang L, et al. Mlake: Multilingual knowledge editing benchmark for large language models[J]. arXiv preprint arXiv:2404.04990, 2024.\n\n[4] Rosati D, Gonzales R, Chen J, et al. Long-form evaluation of model editing[J]. arXiv preprint arXiv:2402.09394, 2024.\n\n- While LocFT-BF scales to 72B models, this analysis lacks a baseline comparison. Could a comparison be provided, even on a mid-sized model (e.g., 32B), to better substantiate the claim of superior scalability, or is the assumption that other methods are computationally infeasible at this scale verified?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Overturns the long-standing misconception that fine-tuning is unsuitable for model editing , proving through rigorous experiments that prior failures were due to flawed implementations (i.e., the depth-first pipeline) rather than inherent limitations.\n- The proposed LocFT-BF method is exceptionally simple and efficient, requiring no complex architectural changes, auxiliary data, or costly pre-computation overhead typical of other editing techniques。\n- Demonstrates unprecedented scalability by showing LocFT-BF effectively handles 100K sequential edits and scales to 72B-parameter models, addressing a key practical bottleneck in model editing.\n- Provides a systematic and in-depth empirical analysis of how to apply fine-tuning correctly, including rigorous studies on the training pipeline , batch size , and optimal parameter tuning locations.\n- Achieves state-of-the-art performance, comprehensively and significantly outperforming existing methods, particularly in challenging large-scale lifelong editing scenarios.", "weaknesses": "- The paper's justification for the superiority of the Breadth-First (BF) pipeline relies heavily on empirical phenomena (e.g., mitigating overwriting) . It lacks a deeper theoretical analysis, such as convergence or stability bounds, to formally explain why mini-batch optimization provides superior stability and forgetting mitigation compared to the sample-wise, depth-first (DF) approach in the context of sequential editing.\n- The \"Capability\" evaluation, while broad, is confined to standard benchmarks (e.g., MMLU, GSM8K, SST2) that are predominantly multi-choice or classification tasks. This setup overlooks more complex and valuable scenarios, failing to test the model's robustness against, for example, generating hallucinations (e.g., TruthfulQA) or handling complex reasoning (e.g., BBH) post-edit.\n- The paper repeatedly claims that the observed degradation in \"Generalization\" (e.g., on COUNTERFACT) can be mitigated by data augmentation . However, this claim remains unsubstantiated, as no supporting experiments are provided in the main text to validate whether augmentation can indeed recover generalization without negatively impacting reliability or capability. The author can refer to the following work on augmentation and discuss it:\n\n[1] Allen-Zhu Z, Li Y. Physics of language models: Part 3.1, knowledge storage and extraction[J]. arXiv preprint arXiv:2309.14316, 2023.\n\n[2] Jiang K, Du Y, Ding Y, et al. When Large Multimodal Models Confront Evolving Knowledge: Challenges and Pathways[J]. arXiv preprint arXiv:2505.24449, 2025.\n\n[3] Wang K, Zhu J, Ren M, et al. A survey on data synthesis and augmentation for large language models[J]. arXiv preprint arXiv:2410.12896, 2024.\n\n[4] Park C F, Zhang Z, Tanaka H. $\\textit {New News} $: System-2 Fine-tuning for Robust Integration of New Knowledge[J]. arXiv preprint arXiv:2505.01812, 2025.\n\n- The experimental validation focuses exclusively on general-domain factual knowledge (e.g., ZsRE, COUNTERFACT). The effectiveness and generalizability of LocFT-BF in editing specialized, high-stakes domains (e.g., biological, financial, or legal knowledge) or in cross-domain/cross-lingual editing scenarios remain unverified.\n- The analysis of scaling to larger models (§5.2)  only evaluates LocFT-BF on the Qwen2.5 series. It crucially lacks a direct comparison against baseline methods (e.g., RLEdit, UltraEdit) at these larger scales (14B, 32B, 72B). This omission makes it difficult to substantiate the strong claim that LocFT-BF is uniquely or more easily scalable than other methods while maintaining performance.\n\nIf the author can solve the above problems, I will consider increasing the score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rW9Hg9blP7", "reviewer": "Reviewer_MVzX", "summary": "This paper challenges the prevailing view that fine-tuning is a weak baseline for model editing. The authors compellingly argue that its reported failures stem not from the method itself, but from a flawed \"Depth-First\" (DF) implementation that processes edits sequentially and individually. This DF pipeline, they demonstrate, leads to catastrophic forgetting and degrades general capabilities.\n\nThe core proposal is to revert to the standard \"Breadth-First\" (BF) training pipeline, which uses epoch-based, mini-batch optimization over the entire set of edits. This is combined with a systematic study that identifies tuning later-layer MLP modules as an effective strategy for localizing updates.\n\nThe resulting method, LocFT-BF, is simple and highly effective. Extensive experiments show that it significantly outperforms state-of-the-art methods and, for the first time, successfully scales to 100,000 sequential edits and 72B-parameter models, setting a new and powerful baseline for the field.", "questions": "Q1: Have you performed any experiments to support the claim that the generalization gap on COUNTERFACT can be closed by applying data augmentation techniques similar to those used by methods like MEMIT or AlphaEdit?\n\nQ2: How would a standard LoRA, when applied to the empirically-found optimal modules (e.g., later-layer MLPs) and trained using the correct BF pipeline, compare to your full-parameter local tuning approach in LocFT-BF? This would help clarify whether, under the proper training paradigm, the concept of localization is the more critical factor, or if full-parameter tuning is inherently superior to low-rank adaptation in editing.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strength1: The claims are exceptionally well-supported by a series of controlled experiments that systematically isolate and analyze the impacts of the training pipeline (DF vs. BF), batch size, and tuning location.\n\nStrength2: LocFT-BF is conceptually simple, efficient, and easy to implement, especially when compared to more complex editing techniques. This makes it a highly practical and powerful new baseline.", "weaknesses": "Weakness1: The method shows a generalization gap on the COUNTERFACT dataset compared to baselines that use data augmentation. While the authors provide a plausible explanation, the claim that this can be \"readily mitigated\" with augmentation is not experimentally verified in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wnMICiFzZp", "reviewer": "Reviewer_Dcsj", "summary": "This paper revisits the long-standing belief that fine-tuning is ineffective for model editing in LLMs. The authors argue that prior failures stem not from inherent limitations of fine-tuning but from mis-specified implementations that used a depth-first (DF), sample-wise pipeline rather than the standard breadth-first (BF), epoch-based, mini-batch paradigm. Through controlled experiments, they show that restoring the BF pipeline substantially improves reliability and preserves general capabilities.\n\nBuilding on this insight, they introduce LocFT-BF (Localized Fine-Tuning with Breadth-First pipeline), which integrates three key ideas: (1) Restoring the standard BF pipeline; (2) Using mini-batch gradient aggregation; (3) Strategically selecting tuning locations (favoring later-layer MLP down-projections).\n\nExtensive experiments demonstrate that LocFT-BF outperforms representative methods (e.g., WISE, RLEdit, UltraEdit, MEMIT, RECT, AlphaEdit) across multiple datasets (ZsRE, COUNTERFACT, WikiBigEdit) and models (LLaMA-3, Mistral, Qwen2.5). It achieves high reliability (>98%), strong generalization, efficient scalability (up to 100K sequential edits), and applicability to relatively very large models (up to 72B parameters).\n\nOverall, this is a strong paper with clear contributions, high experimental quality, and potential to reshape the model editing field. Addressing the generalization limitations and exploring broader use cases would further enhance its impact.", "questions": "- How sensitive is LocFT-BF to tuning location selection in unseen LLM architectures (e.g., encoder-only, mixture-of-experts)? Could an automatic tuning-location search mechanism generalize better? How do you judge the layer selection robustness?\n- For COUNTERFACT, LocFT-BF shows weaker generalization compared to some locate-then-edit baselines. Could integrating data augmentation or paraphrase generation directly into LocFT-BF resolve this gap? Could you provide some empirical study?\n- Beyond factual edits, How would LocFT-BF perform on edits involving style adaptation, or safety constraints? Are there plans to extend evaluation beyond factual QA?\n- While the method scales to 100K edits, does edit interference accumulate subtly over time (e.g., conflicting edits on related entities)? Have the authors analyzed error patterns at scale?\n- How would LocFT-BF integrate into a real-world setting where edits arrive asynchronously over time and may be reverted? Is incremental updating compatible with LocFT-BF’s batch/epoch paradigm?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper challenges a widely held misconception in model editing and repositions fine-tuning as a strong, scalable baseline, and it’s a original proposal. Specifically, this paper identifies the flawed depth-first pipeline as the root cause of prior failures, an insight both simple and impactful. Furthermore, this paper proposes LocFT-BF, which combines principled pipeline design and layer selection with compelling empirical validation. \n- The experimental design is thorough, e.g., controlled comparisons of DF vs BF, batch size studies, layer/module analyses, and large-scale evaluations (100K edits, 72B LLMs).\n- Benchmarks against a broad set of representative baselines, including parameter-extension, meta-learning, and locate-then-edit methods. Results are consistently strong across multiple datasets and architectures, lending credibility to claims.\n- The paper is well-structured, starting with the misconception, then systematically dissecting causes, followed by a constructive solution. \n- Writing is clear and accessible even for readers not deeply familiar with model editing. Figures and tables (e.g., pipeline comparisons, tuning location studies, scaling results) are intuitive and reinforce key arguments.\n- LocFT-BF is lightweight, plug-and-play, and avoids architectural overheads. It could shift the research landscape by elevating fine-tuning from a dismissed baseline to a leading paradigm for model editing. Furthermore, scalability to 100K edits and 72B models addresses real-world deployment needs, setting a new bar for evaluation standards.", "weaknesses": "- The generality of layer selection is not strong enough. The paper proposes later-layer MLP-down projections as relatively stable tuning locations, but acknowledges that optimal locations vary across models. More justification or theory behind this heuristic would strengthen the contribution.\n- LocFT-BF underperforms on generalization for certain datasets (e.g., COUNTERFACT, where prompts involve irrelevant text). Authors suggest data augmentation as a fix in A.2, but no experiments directly test this claim.\n- While diverse, the benchmark datasets are mostly factual-editing oriented. Broader evaluation (e.g., safety edits, temporal consistency) would strengthen claims about “real-world editing”.\n- While comparisons are comprehensive within model editing literature, it would be useful to position LocFT-BF against emerging post-training alignment techniques (e.g., preference tuning, instruction backfitting) that could also serve as editing strategies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)