Precise Localization of Memories: A Fine-grained Neuron-level Knowledge Editing Technique for LLMs
Authors: Haowen Pan
Xiaozhi Wang
Yixin Cao
Zenglin Shi
Xun Yang
Juanzi Li
Meng Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Transformers
Model Editing
Neural Networks
Neuron Activation
Primary area: interpretability and explainable AI

Abstract: Knowledge editing aims to update outdated information in Large Language Models (LLMs). A representative line of study is locate-then-edit methods, which typically employ causal tracing to identify the modules responsible for recalling factual knowledge about entities. However, we find these methods are often sensitive only to changes in the subject entity, leaving them less effective at adapting to changes in relations. This limitation results in poor editing locality, which can lead to the persistence of irrelevant or inaccurate facts, ultimately compromising the reliability of LLMs. We believe this issue arises from the insufficient precision of knowledge localization. To address this, we propose a Fine-grained Neuron-level Knowledge Editing (FiNE) method that enhances editing locality without affecting overall success rates. By precisely identifying and modifying specific neurons within feed-forward networks, FiNE significantly improves knowledge localization and editing. Quantitative experiments demonstrate that FiNE efficiently achieves better overall performance compared to existing techniques, providing new insights into the localization and modification of knowledge within LLMs.

Published as a conference paper at ICLR 2025
PRECISE LOCALIZATION OF MEMORIES: A FINE- GRAINED NEURON-LEVEL KNOWLEDGE EDITING TECHNIQUE FOR LLMS
Haowen Pan' Xiaozhi Wang" Yixin Zenginn Shi* Xun Laagz-Juazzi Li
Meng Wang"
Unniversity of Science and Technology of China DDepartment of Computer Science and Technology. BNRist, Tsinghua University 'School of Computer Science, Fudan University *Hefei University of Technology
ust.aaiiissst...o.o wesst..edd...m caoyixin201 gma com {zenglin itenggin..sh,aanneeee. @ edu.cn [rang21@uster edu
wangxz20emails.. www.thuannhan... edu.
ed lijuanzi@tsinghte edu..n edu.cn
ABSTRACT
Knowledge editing aims to update outdated information in Large Language Mod- els (LLMs). A representative line of study is locate-then-edit methods, which typ- ically employ causal tracing to identify the modules responsible for recalling fac- tual knowledge about entities. However, we find these methods are often sensitive only to changes in the subject entity. leaving them less effective at adapting to changes in relations. This limitation results poor editing locality, which can lead to the persistence of irrelevant or inaccurate facts, ultimately compromising the re- liability of I.LMs. We believe this issue arises from the insufficient precision of knowledge localization. To address this, propose Fine- fine-grained Neuron-level Knowledge Editing (FiNE) method that enhances editing locality without affect- ing overall success rates. By precisely identifying and modifying specific neu- rons within feed-forward networks, FiNE significantly improves knowledge lo- calization and editing. Quantitative experiments demonstrate that FiNE efficiently achieves better overall performance compared to existing techniques, providing new insights into the localization and modification of knowledge within LLMs.!
INTRODUCTION
Recently, various methods for the precise editing of outdated wrong knowledge within Large Language Models (LLMs) (Touvron et al., 2023a;b; Jiang et al., 2024: Dubey et al., 2024) have been proposed (Mazzia al., 2023; Yao et al., 2023: Wang al., 2023). These methods include memory-based editors (Mitchell et al., 2022b; Zheng et al., 2023; Hartvigsen et al., 2024; Yu et al., 2024), meta-learning approaches (De Cao et al., 2021; Mitchell et al., 2022a: Hase et al., 2023b; Han et al., 2023), and locate-then-edit methods (Dai et al., 2022: Meng et al., 2022; 2023; Li al., 2024; Gupta et al., 2024). This paper primarily focuses on locate-then-edh methods, which have emerged as promising and mainstream approach for knowledge editing in LLMs. A key representative of these approaches is ROME (Meng et al., 2022), which employs causal tracing to identify specific modules responsible for recalling facts about subject entities. The success of ROME has inspired subsequent methods, e.g., MEMIT (Meng et al., 2023) and PMET (Li et al., 2024) that utilize causal foundational technique in the field. Locate-then-edd methods offer
tracing, establishing its role critical insights into the precise storage locations of knowledge, enabling targeted modifications that enhance the reliability and accuracy of outputs from LLMs.
However, Hase et al. (2023a) question the validity of this localization method, noting that causal trac- ing offers limited insight into which Ferd-Forward Network (FFN) layer should be edited to update existing knowledge. The ineffectiveness of localization may cause the editing to be predominantly
"Corresponding authors.
We release our code httpp:://// /ttptss:://iieee http://////////////////// thitthub..aa com htt.tb..coo/pppaaaacc http///aaahhwwweaoe

---

Published as a conference paper at ICLR 2025
(8)
Before Editing
(b)
Locating & Editing
Step Causal Tracing
The city the Clympic Games
The nest host city of the (lyympi Cames is
relation
subject
Tokyo
Tokyo
Previous locate-then method
FFN Angeles
Step 1: Neuron Localization
abyect
FINE method
Stop
Incremental weight
cc)
After Editing
Over-editing Rate (%)
Unchanging Rate (%)
city the Olympic Games
60
75
bTell
Tos Angeles
50
(d)
Locality Testing
20-
25
(change the relation)
The presious host city of the Olympic
Garnes is
0
GPTJ LLaMA-2 LLaMA-3
GPTJ
LLaMA-2 LLaMA-3
Angeles
LoRA
ROME
MEMIT
FINE
Figure I: Previous locate-then-edit approaches (c.g., ROME and MEMIT) perform poorly in locality lesting when changing the relation. (a) LLM makes a response "Tokyo" before knowledge editing. (b) We apply knowledge editing methods edit the answer from "Tokyo" "Los Angeles". (c) After editing, the model responses the target answer. (d) We then evaluate the post-edited model's locality and find that previous methods fail when changing the relation (i.e., outputting the target word even with unrelated inputs). (e) We also conduct quantitative experiments for the over-editing rate (lower values are better) and unchanging rate (higher values are better).
subject-driven. One possible evidence is that tecate-then-edit methods overly rely on the subject en- tity rather than the relation (Wei et al., 2024). When change the relation in locality testing, the post-edited model fails to produce correct answer and instead continues to generate the target object (see Figure (d)). Furthermore, we conduct a pilot quantitative experiment on WikiData dataset in KnowEdit (Zhang et al., 2024) benchmark, and evaluate the over-editing rates and un- changing rates?. As shown in Figure ((e). both ROME and MEMIT exhibit high over-editing rates and low unchanging rates, indicating causal tracing encounters issues during localization by focus- ing excessively on the subject and neglecting overall knowledge. Due to data construction issues with previously commonly used datasets, such as COUNTERFACT (Meng et al., 2022), these prob- lems have not been adequately exposed. These observations collectively indicate the localization of existing methods has significant flaws and lacks sufficient precision for guiding knowledge editing.
This motivates us to investigate more precise localization methods. Inspired by previous neuron-level analyses (Dai et al., 2022; Wang et al., 2022; Schwettmann et al., 2023; Pan et al., 2024), we propose a Fine-graing -ene-grainedI Neuron-level Knowledge Editing (FiNE) technique for a more precise localization of memories within LLMs. We first identify neurons in FFNs that are highly relevant to the knowledge to be edited and then update model weights at the locations these neurons. Our neuron-level localization method provides more fner-grained indication of the knowledge location compared to causal tracing and effectively avoids the problem of excessive focus on the subject. Furthermore, this approach benefits from fine-gran ineeggrinedd modifications LLMs, resulting in a more efficient method that saves time and memory usage. Experiments on GPT-J (Wang & Komatsuzaki, 2021). LLaMA- 2 (Touvron et al., 2023b), and LLaMA-3 (Dubey al., 2024) demonstrate that FiNE significantly outperforms existing bocte-then-eeeee then-edit methods based on causal tracing, especially in locality metrics.
RELATED WORK
2.1 MODEL EDITING TECHNIQUES
Memory-based For memory-based editors, some specific modules store the edit knowledge are used for post-edit response. SERAC (Mitchell al., 2022b) stores edits in an explicit memory and learns to reason over them to modulate the base model's predictions as needed. Zheng et al. (2023) explores in-context knowledge editing (IKE), a method without any gradient and parameter updating, GRACE (Hartvigsen et al., 2024) is lifelong model editing method that implements
"Se Appendix B. for experimental setup details.
2

---

Published as a conference paper at ICLR 2025
spot-fixes streaming errors of deployed model, ensuring minimal impact on unrelated inputs. Recently, Yu et al. (2024) proposes MELO, a novel method that alters the behavior of LLMs by dynamically activating certain LoRA blocks according to the index built in an inner vector database.
Meta-learning Based on hypernetwork, several meta-learning methods have been proposed to edit models. De Cao et al. (2021) presents KnowledgeEditor (KE), a method which can be used to edit knowledge and, fix bugs or unexpected predictions without the need for expensive re-training or fine-tuning. MEND (Mitchell et al., 2022a) introduces collection of small auxiliary editing net- works that use a single desired input-output pair to make fast, local edits to a pre-trained model's behavior. Hase et al. (2023b) proposes SLAG, a training objective for sequential, local, and gener- alizing updates with better performance. Han et al. (2023) proposes novel divide-and saide-and-conquer framework, drawing on dynamic inference to break the zero-sum phenomenon multiple edits.
Locate-then-edit Although prior research has explored knowledge storage mechanisms, the precise methods by which LLMs retain knowledge remain unclear. Studies have indicated that knowledge is often embedded within FFNs (Geva et al., 2021; 2022; Dai et al., 2022). Building on these, locate- then-edit methods have gained traction by first locating specific regions of knowledge storage and then executing targeted editing. A leading example is ROME, which innovatively employs causal tracing to pinpoint parameters intended for edits and directly updates them (Meng et al., 2022). This foundational work has paved the way for additional methods such MEMIT (Meng et al., 2023), PMET (Li et al., 2024), and EMMET (Gupta et al., 2024), enhancing the capacity to incorporate and modify larger quantities of knowledge. The advantages of locate-then-edit methods include increased precision in knowledge modification and the ability to selectively edit specific information, making them a vital advancement in the ongoing development of more effective and reliable LLMs.
2.2 NEURON ANALYSES IN TRANSFORMER BEASSDD BASED MODELS
Transformer (Vaswani et al., 2017) is one of the most successful architectures designed for different tasks (Song et al., 2024b; et al., 2023) and there has been increasing interest interpreting and an- alyzing the internal mechanisms of transformer-basee models. Previous research has aimed to char- acterize the types of information encoded in individual neurons. Dai et al. (2022) explores the iden- tification of "knowledge neurons", which encode specific commonsense knowledge acquired during pre-training. Additionally, Wang et al. (2022) presents a technique for identifying "skill neurons" in pre-trained transformer-basee language models, which crucial for specific tasks. Schwettmann et al. (2023) explains how L.MSs convert visual representations into corresponding texts by intro- ducing a procedure for identifying "multimodal neurons". More recently, Pan et al. (2024) proposes a novel method for finding "multi-modal neurons", which elucidates how multi-modal LLMs (Zeng et al., 2024) bridge visual and textual concepts for captioning (Song et al., 2024a).
3 PRELIMINARY
Neurons in LLMs. A decoder-only Transformer-basedd (Vaswani et al., 2017) LLM (denoted as M) typically consists of stacked self- attentienoo and feed-forward layers. Each layer first performs multi-head self-attention and then applies a position-wise FFN. Residual connections and layer nor- malization are employed around each sub-layer. Following previous works (Dai et al., 2022; Wang et al., 2022; Schwettmann et al., 2023; Pan et al., 2024), we investigate neurons within FFNs, as FFNs carry abundant information and knowledge. We denote the hidden states at layer / as h', FFN output as m'' and self-attention output as a', respectively. The hidden states can be written as:
h' h i1 + +m' + a', where m' w! ou (Whn(x²))
h0 is the embedding vector of input, a is an activation function, y is layernorm, W is the first linear layer and w out is the second linear layer in the FFN, and x' represents the FFN input. For simplicity, let q' = a (Wi (Wi, (zx")) We regard gf the i-th element of g', as the activation of the i-th neuron on input x' at layer I. Each neuron in LLMs can be denoted as (LI.U)).
in
Knowledge editing. Extensive training on diverse datasets has endowed LLMs with vast repository of knowledge (Brown, 2020; Chowdhery et al., 2023; Schott et al., 2023). Formally, knowledge in L.LMs can be denoted as triples like (subject 8, relation r,, object o) (Meng et al., 2022; 2023), such as
3

---

Published as a conference paper at ICLR 2025
(s the Olympic Games, next host city, 0 Tokyo). We define p(-) function that converts knowledge triples into prompt texts, for example, p(the Olympic Games, next host city) corresponds to "The next host city of the Olympic Games is" and p(the Olympic Games, next host city, Tokyo) corresponds to "The next host city of the Olympic Games is Tokyo". Let (s, r, o*) represents the updated knowledge. After editing, when the edited L.M (denoted as M') is given the input p(s, r), it should return o" instead of o. For instance, if o' is "Los Angeles", the edited model should respond with "The next host city of the Olympic Games Los Angeles".
Evaluation for knowledge editing, To effectively evaluate knowledge editing methods. Zhang al. (2024) presents four essential criteria: Edit Success, Portability, Locality, and Fluency. Edit Success measures whether the post-edited model generates the expected output, which computes the accuracy of the outputs by (9 ity evaluates how well the model can address the implications of an edit for real-world appli- cations, which is computed by B(ss.rg.ot) 1 farg max, Pa [ylp(s;,r,)]
1 {arg max, M [vlp(*;,r;)] = =0]). Portabil-
oī For exam- ple, when asked "Is the next Olympic Games hosted in Tokyo?" the post-edited model should answer "No". Locality examines whether an editing modifies the knowledge locally without in- fluencing other unrelated knowledge, e.g., when asked, "How often are the Olympic Games held?" the model should still correctly respond with "Every 4 years". Locality can be calcu- lated as E farg max, P. (elp(s;,r)) sures the model's generation ability by calculating weighted average of bi-gram and tri-gram en- tropies (Zhang et al., 2018), denoted by - >e ((k) log, [(k,,, where f(.) is n-gram frequency distribution. A lower Fluency indicates higher frequency of repeated words, signifying lower qual- ity responses. These metrics offer a comprehensive assessment of methods' effectiveness, capturing various dimensions of performance and ensuring robust analysis of the editing process.
arg max, P. wypp(s, r,)]]- Fluency mea-
METHODOLOGY
FiNE provides precise localization of knowledge within LLMs through two-step process. In the first step, which differs significantly from causal tracing localization, it identifies key neurons in FFN layers that are closely associated with the knowledge to be edited. Subsequently, updates model weights at these specific neuron locations.
4.1 LOCATING NEURONS LLMS
Following previous work (Dai et al., 2022; Wang et al., 2022; Schwettmann et al., 2023; Pan et al., 2024) on selecting neurons in Transformer- based models, we present neuron localization method for knowledge editing. Specifically, we hypothesize that a knowledge (sy. 0j) is stored in specific neurons, which are activated when LLMs receive input rz), exhibiting a tendency to produce the output Oj. Therefore, our objective is to quantify contribution of each neuron to the current output and locate those neurons with higher impact. Following Pan et al. (2024), who calculates contribution scores in multi-modal LLMs, similarly compute contribution scores within LLMs.
For each token t in the output 0j, we compute contribution score for each neuron u, at layer / as:
C(i.L.) = - (W, W
(3)
out 1,5
where - is the activation output at the last token for neuron u; at layer (+),, represents the I-th row and i-th column of the input matrix, and W., is the unembedding matrix.
Here we regard W. v. W our E Ruxdd as projection function projecting from activations of the neurons to distribution of the vocabulary, where dm is the intermediate size and v is the vocabu- lary size and regard q',- as a coefficient of the projection, respectively. This projection explicitly demonstrates the varying levels of focus that different neurons pay to different tokens, enabling us to calculate the contribution score. We provide detailed derivation in Appendix A.
After quantifying the contribution of each neuron, rank all scores of neurons across all layers by the descending order and pick out top-k neurons, denoted as w². These neurons are regarded as carriers of knowledge ()))) and make significant contributions to the output 0j. We follow the
same procedure to locate neurons for each token t in oj. and use the set Uy {uf u...... .- =/fo1]
to represent neurons of oj. where |051 means the token length of oj. Algorithm I summarizes the entire process of neuron localization.
4

---

Published as a conference paper at ICLR 2025
4.2 UPDATING KNOWLEDGE
Algorithm 1: Neuron Localization
We locate key neurons Uj of 0j described above, and then modify the model weights cor- responding to locations of selected neurons to up- date the knowledge. For each neuron u E Ug we assume that is the i-th neuron at layer 1. Then
Data: Knowlodge (5,, LLM M Result: Neuron Set that carries knowledge
1 Initialize L4, = (5)))).0g)
2
for each token t in o, do Compute contribution of each neuron by Eqn. 3:
3
we compute a vector 2 E Redh and add it to the i-th row of matrix W for updating, where dy is the hidden size. If we stack vector for each neu- end
4 wb
select top-k neurons by the descending order;
U, + Ug U {u²};
ron и as Z3 [2| Zz ... Ei our objective can be succinctly represented as learning an optimized Zj based on Uj. which is then applied to the model M, resulting in a post-edited model MM'. Following Meng et al. (2022), the objective ((,))
consists of editing loss Ledi (Zj), KL divergence LKI. ( (Zz) and repetition penalty loss Lpen Zz). The
editing loss utilizes negative log-likelihood to maximize the probability the target o5:
Cedi 22) = Par [o] p()):
(4)
During the editing process, we aim to avoid altering unrelated knowledge impacting the model's language capabilities. To this end, we add a KI. divergence constraint of prompt that contains the subject and relation to the model, which is calculated by:
CxI (Z,) = DKI. (P'Mr [y p(s), II P [v p(s),ry)))
(5)
where IP' [+] represents the probability distribution of output from position 1 to position lp 1, assuming the length of the input prompt lp. which is different from TP [.]. Except KI. divergence, to prevent the post-edited model from generating the editing target oī repeatedly,
also introduce repetition penalty constraint. At the last position of the complete prompt p(8j, Ti, 07), we use negative log- likelihodd to maximize the probability of not generating 03:
Spenl (2z log (1 P [o] p(ss) ry. o))]
(6)
Finally, we compute a weighted sum of editing loss, KL divergence and repetition penalty loss:
((Z,) Ledie (2,) fKu (Z,) 3
where a and /3 are hyperparameters.
4.3 LAYER FREEZING
In language models, the later layers are closely tied to the model's language capabilities (Geva et al., 2021; Dai et al., 2022: Wang et al., 2022; Pan et al., 2024). Arbitrary modifications to these later layers may impair model's linguistic abilities and result in responses with lower quality. To ensure the stability of LLMs, with L layers, when locating neurons, we exclude the last I layers, focusing only on the first L- --, layers. This ensures that no modifications are made the last ly layers during the editing process.
implement layer freezing (LF) in method. Specifically, for LLM
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Models and datasets. We conduct experiments on the KnowEdit (Zhang et al., 2024) benchmark with GPT-J-6B (28 layers) (Wang & Komatsuzaki, 2021), LLaMA-2-7B (32 layers) (Touvron al., 2023b) and LLaMA-3-88 (32 layers) (Dubey et al., 2024). KnowEdit is an integrated benchmark for evaluating various knowledge editing methods, which contains six datasets for different eval- uation types. We select three datasets including knowledge insertion and knowledge modifica- tion in our experiments: WikiDatta.onnteffuu (Cohen et al., 2024), WikiDatarreenn (Cohen et al., 2024) and ZsRE (Levy al., 2017). Notably, the locality evaluation in KnowEdit primarily fo- cuses on changing the relation. The proportion of prompts where the subject changes in datasets WikiDattdcounterffats WikiDattareentt and ZsRE is only 0.9%, 0.1%, and 0.0%, respectively.
We categorize baseline methods into three types of knowledge editing. The first cate- gory consists of methods that directly modify model parameters, such Fine-Tuning (FT) and LoRA (Wu et al., 2023). The second category includes memory-based methods, for which we se- lect In-context Knowledge Editing (IKE) (Zheng al., 2023), which retrieves the most pertinent
Baselines.
5

---

Published as a conference paper at ICLR 2025
Table Editing results on WikiDat fact. 95% confidence intervals are in parentheses. Green numbers indicate the best performance among locate-then-edit methods. Grey numbers indicate invalid results'. Numbers with underline indicate columnwise maxima for each model.
Portability 1
Locality
Method
Edit Suec.
Fluency
SAA
LGA
RA
RSA
FA
GPTJ
21.5
(1.66
14.8.2.4)
(1.66
612.3 (3.1)
FT
64.2 (1.6)
(2.0)
7.1 (1.9)
(2.9)
4.4 (0.6)
(1.3)
304.1 (7.6)
(0.0)
(0.0
(4.5(4.3)
60.6 (1.3)
$2.3 (3.1)
LoRA
100.0 (0.0)
(1.2
22.2 (3.1)
40.3 (2.8)
25.7 (1.6)
(2.8)
595.8 (4.1)
KN
(2.4)
(2.9
(2.6)
(2.2)
(0.2
(1.5)
580.0 (3.8)
ROME
(2.1)
(2.6
(2.2
(1.3)
33.1 (2.6)
600.0 (3.6)
MEMIT
99.5 (0.5)
(2.5)
16.6 (2.6
$3.2 (2.9
40.7 (2.8)
591.6 (4.3)
PMET
95.3(0.9)) (0.9)
(2.6)
(2.3
(1.5)
36.8 (2.8)
600.3 (3.6)
FINE
99.8 (0.1)
(1.4
17.5 (2.7)
(3.5)
84.2 (1.1)
(2.7)
545.7 (7.3)
LLaMA-2
(1.5
(1.8
26.1 (2.9)
(1.99
583.3 (2.7)
(1.8)
(1.2
(2.9)
(2.0
59.5 (1.3)
40.2 (2.7)
500.9 (6.8)
IKE
100.0 (0.0)
(0.5)
(5.2)
712(338)
(1.1)
(2.5)
LoRA
(0.0)
(1.0
(3.9)
(4.(3.1)
73.5 (1.2)
(2.7)
559.3 (5.1)
KN
(2.9) (2.3)
16.9 (2.7)
24.6
73.7 (2.1)
68.7 (3.5)
561. 4 (6.3)
ROME
98.7 (0.6)
(2.2
(2.8)
(2.4
49.1 (1.2)
40.5 (2.7)
577.3 (3.3)
MEMIT
(0.7)
(2.1)
(2.8
35.0
40.1 (2.8)
561.9
PMET
94.8 (1.0)
56.7 (2.5)
(3.0)
(2.4
64.5 (1.4)
50.0 (2.8)
576.1 (3.4)
FINE
(0.2)
(1.4)
28.8 (3.0)
41.5
(1.6
65.0 (2.8)
542.3
23.1 (1.5)
(3.0) (1.7)
(1.9)
607.1 (2.9)
FT
44.6
(2.0
(L4
22.9
28.7 (1.3)
14.2 (2.0)
351.7
IKE
61.8 (1.5)
60.3
(5.3
(3.63.3)
67.7 (1.2)
65.7 (2.6)
LoRA
100.0 (0.0)
(1.8)
(3.2.
(3.6
(1.2)
29.8 (2.5)
455.7 (11.2)
KN
17.1 (2.1)
18.1
(2.6
1922
(1.6
(2.6
593.7 (6.8)
ROME
99.4 (0.4)
(2.6)
21.2
(2.5
31.5 (1.2
591.4 (4.1)
MEMIT
99.1 (0.5)
(2.6
20.7
(25)
(1.3)
(2.7)
570.1 (6.3)
PMET
96.0 (1.0)
(2.6
(2.8
(2.4
60.6 (1.4)
41.6 (2.9)
596.2 (3.5)
FINE
100.0 (0.0)
(1.4
22,4 (2.9)
38.3
90,5 (0.9)
(2.9)
567.1 (5.5)
demonstrations. The third category focuses on locate-then-edit methods, which are central to our study, Although Knowledge Neurons (KN) is also neuron-level knowledge localization method, employs significantly different technique than ours, selecting neurons via gradient-based attribu- tions and modifying the corresponding FFN weights by adding scaled embedding vectors. Impor- tantly, ROME (Meng al., 2022), pioneer of causal tracing localization, has further advanced the locate-then-edit methods and significantly influenced the field, while MEMIT (Meng et al., 2023) has built upon this foundation with notable enhancements. PMET (Li al., 2024) serves as an improvement over MEMIT. Both ROME and MEMIT not only represent critical developments but have also achieved substantial popularity, making them essential comparisons in our work.
Evaluation metrics. As described in $ 3, we adopt four evaluation metrics in our experiments: Edit Success, Portability, Locality, and Fluency (Zhang et al., 2024). Portability contains three parts: Subject Aliasing Accuracy (SAA), Logical Generalization Accuracy (LGA) and Reasoning Accuracy (RA). Subject aliasing replaces the question's subject with an alias or synonym to evaluate perfor- mance on other descriptions of the subject. Logical generalizations are changes that are semantically related to the modified fact and expected to change by the edit. Reasoning examines the reasoning ability with changed facts. Locality consists of two parts: Forgetfulness Accuracy (FA) and Relation Specificity Accuracy (RSA). Forgetfulness evaluates whether the post-edited model retains the orig- inal objects in one-to-mane many relationships, whereas relation specificity evaluates whether any other attributes the subject, which have been previously updated remain unaltered.
5.2 QUANTITATIVE RESULTS
In Table I, we show quantitative editing results on WikiDataa Our approach demonstrates the best Edit Success, Portability and Locality among various -hatte-the--edi methods. We observe that previous locate-then-edit methods with causal tracing localization perform poorly when han-
l Locality results with low Edit Success are not considered valid, as the locality is inherently 100% when no edit is effectively applied.

---

Published as a conference paper at ICLR 2025
Table 2: Ablation results of removing neuron localization (Loc.) and layer freezing (LF) on WikiData fact- 95% confidence intervals are in parentheses. Numbers with bold indicate columnwise maxima for each model.
Method
Edit Succ.
Portability 1
Locality 1
Fluency
SAA
LGA
RA
RSA
FA
GPT-J ROME
21.5 (1.5)
21.7 (1.6)
(2.4)
(1.6
612.3 (3.1)
99.2 (0.5)
74.1 (2.2)
16.1 (2.6)
(2.4)).4)
(1.4
33.1 (2.6)
600.0 (3.6)
w/o Loc.
(1.1
(2.6
15.1 (2.5)
(2.5)
(1.6
35.7 (2.5)
515.5 (10.4)
FINE
99.8
(1.4
(2.5
(3.4)
(1.1
(2.7)
545.7 (7.3)
w/o w/o LF
(2.1)
60.8 (2.5)
(2.0
(2.5
63.8 (2.9)
589.1 (4.4)
99.7 (0.2)
92.4 (1.3)
(2.5
38.6 (3.6)
78.4
47.0 (2.7)
451.3 (10.7)
LLaMA-2
(1.0
27.8
(2.1
(1.9
583.3 (2.7)
ROME
98.7 (0.6)
(2.2)
22.8
(2.4)
(1.2
40.5 (2.7)
$77.3 (3.3)
w/o Loc.
96.8
70.2 (2.3)
(2.8)
(2.7
(1.6
45.5 (2.7)
517.1 (8.2)
FINE
(0.2)
(1.4)
(3.0)
(3.0)
(1.0
(2.8)
547.6 (6.9)
w/o Loc.
(0.6)
73.9
(2.7)
(2.5
89.4 (0.9)
72.8 (2.6)
$60.6 (3.9)
w/o LF
99.3 (0.5)
89,9 (1.3)
(2.8)
(3.2
75.5 (1.4)
51.7 (2.8)
407.9 9 (10.1)
LLMMA-3
(3.5
(1.7)
(3.0)
(1.9)
607.1 (2.9)
ROME
99.4 (0.4)
(2.2)
21.2
(2.5)
(1.2
31.5 (2.6)
$91.4 (4.1)
w/o Loc.
(1.1
(2.2)
(2.5
(1.5
(1.5)
39.8 (2.8)
534.6 (8.8)
FINE
100.0 (0.0)
(1.4)
(2.9
(3.0)
90.5
63.0 (2.9)
$67.1 (5.5)
w/o Loc.
100.0 (0.0)
(2.1)
(2.7)
(2.8)
84.1 (1.0)
(2.9)
556.9 (6.4)
w/o LF
100.0 (0.0)
91.2 (1.3)
20.1 (2.8)
(3.3)
78.8 (1.2)
48.8 (2.7)
411.3 (10.6)
Table 3: Ablation results of restricting neuron localization to single layer with LLaMA-2 on WikiDatu "Any" means no layer restriction. 95% confidence intervals are in parentheses. Numbers with bold indicate columnwise maxima.
Portability
Locality 1
Method
Layer
Edit Suce.
Fluency 1
SAA
LLaMA-2 ROME
27.0 (1.5)
(1.7)
(2.9)
26.2 (1.9)
583.3 (2.7)
98.7 (0.6)
72.2 (2.2)
25.8 (2.8)
35.1 (2.4)
(1.2
40.5 (2.7)
$77.3 (3.3)
99.0 (0.5)
(2.0)
(2.9)
(2.5)
(0.2
64.1 (2.9)
$70.5 (2.3)
10
100.0 (0.0)
(1.9)
29.1
37.1 (2.6)
(1.0)
67.6
55..4((.5)
FINE
15
100.0 (0.0)
86.9
(3.3)
(2.8)
90.7 (0.9)
72.66
(3.8)
20
100.0 (0.0)
(1.5)
(3.1)..1)
(3.0)
92.9 (0.8)
(2.7
$41.5 (4.6)
100.0 (0.1)
(1.5)
27.1 (3.0)
39.0
95.4 (0.6)
(2.5)
(3.8)
Any
99.9 (0.2)
89.8 (1.4)
28.8
(3.0
(2.6
62.0
(5.13
dling similar but unrelated knowledge, exhibiting generally low Locality. To achieve better editing results, we sacrifice some Fluency but without compromising the original model's 's language capa- bilities. Editing results on WikiDatarecent and ZsRE can be found in Appendix D.
5.3 ABLATION STUDY
In this section, we present ablation study assess the impact of various components the overall performance of our method. Specifically, we first test the impact of removing neuron localization and layer freezing on performance. Next, we investigate effects of restricting neuron localization to a single layer and explore how varying number of selected neurons affects editing. Finally, we examine results of removing KL divergence and repetition penalty constraints in the editing process.
Removing neuron localization and layer freezing. Since our approach also employs locate-then- edit methodology, it is essential verify the effectiveness of the initial localization step. To this end, we maintain the editing process unchanged and conduct experiments by replacing carefully selected neurons with randomly selected ones. Table 2 lists ablation results. When neurons are selected at random, both Edit Success and Portability demonstrate varying degrees of decline, particularly evi- dent in SAA metric, suggesting that our chosen neurons are sensitive to the knowledge being edited. In contrast, ROME experiences only slight decrease in performance without localization, support- ing the hypothesis that causal tracing is not essential. On the other hand, we assess the effectiveness of layer freezing. As shown in Table 2, without layer freezing, the model's 's language capabilities compromised, leading to a significant drop in fluency. It is speculated that even minor modifications, when applied to the last few layers, can result in catastrophic consequences.
7

---

Published as a conference paper at ICLR 2025
Succ. Rate (%6) 1
SAA (96) 1
LGA (%6
RA (%6
RSA (%6
FA (%)
100
75
30-
40
95
80
90
50-
60-
25
#NNuuron
(NNuuro
#Neuron
(Neuuon
WNeuron
//Neuron
Layer 20
Layer 25
Layer Any
Figure 2: Ablation results of varying the number of selected neurons with LLaMA-2 on WikiDatdcounter
foct. foet. The dotted line indicates LLaMA-2's pre-edit performance.
Table 4: Ablation results of removing KI. divergence and repetition penalty constraints with LLaMA-2 on WikiDataaounterraaa 95% confidence intervals are in parentheses. Numbers with bold indicate columnwise maxima.
Method
Edit Suce.
Portability 1
Locality 1
Fluency t
SAA
LGA
RA
RSA
EA
LLaMA-2
(1.5
27.8 (1.7)
26.1 (2.9)
(1.9)
583,3 (2.7) 577.3 (3.3)
ROME
98.7 (0.6)
72.2 (2.2)
25.8 (2.8)
(2.4)
49.1 (1.2)
40.5 (2.7)
FINE
99,9 (0.2)
(1.4)
(3.0)
(3.5)
(1.6
65.0 (2.8)
$47.6 (6.9)
w/o ExL.
99.9
89.8 (1.4)
(3.0)
(1.5
92.3 (1.0)
64.8 (2.8)
540.9 (5.2)
w/o Epes
99.9 (0.2)
90.1 (1.3)
(3.0)
(1.2
(2.5(
64.8 (2.8)
531.7 (6.0)
FINE w/o
(0.5)
89.9
(2.8)
(3.2
(1.4(
51.7 (2.8)
407 (10.1)
w/o CKI.
99.3(0.5)
90.2 (1.3)
(2.7
(3.3
(1.6(()))
49.3 (2.8)
394.6 (10.1)
w/n Epes
(9.2.
91.2
(2.6
(3.2
(1.5
50.9 (2.8)
351.2 (10.)
Restricting neuron localization to a single layer. In the previous experiments, we do not restrict modifications to a specific layer as in ROME. Instead, determine which layers modify solely based on the scores (note that, due to layer freezing, the last few layers are not altered). We now manually restrict neuron localization a single layer and modify only the model weights within that layer to determine whether this manual intervention yields better results. For the specific layer I E (5, 10, 15, 20, 25), only neurons layer I will be selected. Results with LLaMA-2 and LLaMA-3 are listed in Table 3 and Table 11, respectively. We observe that although specifying particular layer sometimes performs better on some metrics (e-g.. RSA and FA of layer 25), overall performance across all metrics is still optimal when no layer restrictions are applied. While manually restricting neuron localization to a single layer can be effective based on experience, relying on our algorithm to automatically locate neurons may be a more appropriate option in the absence of prior information.
Varying number selected neurons. During the editing process, the number of selected neurons likely influences the extent of modifications to the model the more neurons selected, the greater the number of model parameters altered, and vice versa. Therefore, we vary the number of selected neurons to observe the changes in the metrics. For each number of neuron à E (1,3, 5, 10, 15, 20). top-k neurons are selected for each token. Figure 2 plots metric curves on LLaMA-2. We can observe that the number of neurons increases, Portability (i.e., SAA, LGA and RA) generally improves while Locality (i.c., RSA and FA) tends to slightly decrease. This suggests that selecting a greater number of neurons may provide a more comprehensive localization and further enhance the model's ability to update the targeted knowledge, but it also increases the risk of unintentionally altering unrelated memories. Results on LLaMA-3 could be found in Appendix D.
Removing KL divergence and repetition penalty constraints. To minimize the impact on the model's inherent language capabilities, we adopt KI. divergence and repetition penalty constraints during the editing process. Table 4 lists results of removing these constraints in cases with and without using LF. When using LF, the effects of KL divergence and repetition penalty constraints are not significant; however, when LF is not applied, we observe that (1) KL divergence constraint is important for the locality of model editing, and removing it leads to a significant decline in the RSA metric. (2) Repetition penalty constraint has minimal impact portability and locality but significantly affects fluency. Without it, the post-edited model is more likely to produce repetitive text (e.g., "The next host city of the Olympic Games is Los Angeles Los Angeles Los Angeles ...").
8

---

Published as a conference paper at ICLR 2025
5.4 DISCUSSION
Table 5: Comparison of the number of modified parameters. For FiNE, we calculate average results across
Efficiency evaluation. A key advantage of FiNE, due to its fine-grained approach, its notable efficiency. To quantify this, we first examine the number of mod- ified parameters, as detailed in Table 5. Both ROME and MEMIT modify the weights of the second layer in FFNs, resulting in substantial number of parameter modifica- tions, ranging from 10? to 10 In contrast, FINE only edits a subset of neurons, reducing the number of mod- ified parameters approximately 10 which allows for a more fine-grained and precise editing in LLMs. Ad- ditionally, we assess editing time and memory usage Float32 and Float 16 precision in Figure 3 and Figure 8, respectively. FiNE exhibits a significant time advantage over ROME and MEMIT, particularly at Float32 pre- cision, being approximately 4x to faster. In terms of memory usage, FiNE also offers a slight benefit for LLaMA-2 and LLaMA-3.
Method GPT-J
LLaMA-2
LLaMA-3
ROME MEMIT
6.7 10"
x 10
5.9 10"
3.4 10a
2.3 x 10
2.9 10a
FINE 7.9 104
9.7 x 104
104
Editing Time (6)
Memory (GB)
15
30
10-
20-
10-
CPT LLMM LLMMA-3
OPEC ELMMA22 LlaMA-3
ROME
MEMIT
FINE
Figure 3: Comparison of average edit- ing time and memory usage when op- erating at Float32 precision.
Localization analyses. We analyze our neuron-level lo- calization from two perspectives: distributions and tex- tual meanings. (1) We plot the distribution of unique neurons located by FiNE (see Figure 4). We aggre- gate statistics for all located neurons across the entire WikiDatacounterr fact dataset. We observe that these key neurons widely occur in higher layers, which consis- tent with previous work (Wang et al., 2022; Pan et al., 2024). but different from layers that ROME and MEMIT edit. (2) For insight into neurons filtered by Eqn. 3, we follow the Logit Lens (nostalgebraist, 2020; Zhong et al., 2022: Geva et al., 2022), which converts hidden states into a set of logits for each vocabulary token. Simi- larly, we investigate neurons' textual meanings by sort- ing rows of the multiplication of the unembedding ma- trix and the second layer of FFN and regarding top to- kens as each neuron represents (Pan et al., 2024). Table 6 shows example, which indicates that neurons selected by FiNE are highly related to the source knowledge. We list more examples in Appendix D.
16384
16000
2000
ROME Edit Layer MEMIT Edit Layers
#Neuron
1500
1000
500
5
15
25
Layer
Figure Distribution of neurons iden- tified by FiNE among layers in GPT- J, which is aggregated over the whole WikiData
Editing method scaling, In knowledge editing, the abil- ity to simultaneously edit multiple knowledge facts is a crucial objective that enhances the practical application of various methods. Several approaches (Meng et al.,
dataset. For each knowledge fact, FiNE only identifies ap- proximately 20 neurons.
Table An example of localization results with top-3 neurons selected by FINE, For each neuron, we report its contribution score and top-5 relative tokens.
Edit: (Pooja Hegde, country citizenship, India) (Pooja Hegde, country citizenship. Terengganu)
Model
Top Neuron
Score
Top Tokens
L17.U13423
3.291
Delhi', Bhar', "Gujarat, 'Laksh', 'Mumbai")
GPT-J
L20.U11637
1.638
[' lakh', Mumbai", . 'Delhi,, Maharishtrra', Chennai')
L14.U10374
1,359
Delhii, Muumai',, 'India', Iakh', " India']
L26.U7908
1.106
['Iddia,, 'Indian', 'Beng', 'Indians', 'Raj"] IIndian'. 'Indians" 'India". 'Indiana", 'Ind']
ILaMA-2
L25.010178
0.971
L25.U8808
0.750
IIndian', 'Native', 'Indians', 'Native", 'India")
L.8.U1
0.776
[ Indian", 'Indian', Indians', 'indian', India'] [IIddia" IIndia". Indians", Indian", 'indian'] IIddia, "RS""" 'RSS". Tall' 'India']
LLaMA-3
L23, J13680
0.576
T.26.U3334
0.349

---

Published as a conference paper at ICLR 2025
Suce. Rate (%)
1.
SAA (%6)
LGA (96)
(% 1
RSA (%6
FA (%6
100
20
-40
50
50
50-
50-
10
20
25
25
0-
250
500 #Edit
750
250
500 #Edit
750
250
500
750
250
500
750
250
500 #Edit
750
250
500 #Edit
750
#Edit
#Edit
FINE
ROME
MEMIT
Figure 5: Editing method scaling curves with GPT-J. The dotted line indicates GPT-J's pre-edit performance. 95% confidence intervals are shown as areas.
Table 7: An example of editing and locality testing (LT) results with LLaMA-2. Prompts are itali- cized, while green and red indicate keywords reflecting correct and incorrect behavior, respectively.
Edit: (Jean Smart, occupation, voice actor) (Jean Smart, occupation, cello teacher)
ROME: The occupation of Jean Smart is cello teacher. She has been teaching at the music school for 25 years...
* (LT--) ROME: The place of birth of Jean Smart is Hradec Kralove the north of Czech Republic...
* (1 ROME: The name of the country of citizenship of Jean Smart is Czech Republic. The name of her home town Ostrava,
She music teacher profession. The date of birth Jean Smart is 10 March, 1948
MEMIT: The occupation of Jean Smart cello teacher. She also teaches piano, vocal, and conducting lessons
(LT-1) MEMIT: The place of birth of Jean Smart Sárvár. Hungary. She currently lives in Salzburg. Austria
* (LT-2) MEMIT: The name of the country of cillzenship of Jean Smart is Irvin, She lives in Salzburg... ...
FINE: The occupation of Jeun Smart is cello teacher. She is very talented and hardworking person. She is married lady
(L.T-1) FiNK: The place of birth of Jean Smart Seattle, Washington, U.S.. nationality American. ...
(LT-2) FINE; The name the country of citizenship of Jean Smart United States of America. Jean Smart was born 13
September 1951. The birthplace of Jean Smart is Seattle, Washington, U.S.
2023; Li et al., 2024; Gupta et al., 2024) have been specifically developed to achieve this goal. Although our method does not incorporate a specialized design for this purpose, we posit that our more precise localization may reduce the interddependencies among different knowledge facts dur- ing the editing process, thereby intuitively contributing to improved editing scalability. To verify our hypothesis, we progressively increased the scale of editing targets from 100 to 800. Figure 5 plots experimental results with GPT-J. ROME struggles significantly when handling 100 edits, and ceases to function effectively the number of edits increases further, with all metrics approaching zero We observe that our method continues to operate effectively even when handling a larger number of edits, although performance is lower compared to single-instance editing. Additionally, we un- expectedly find that our method closely matches MEMIT on metrics FA, SAA, IGA,, and RA. We attribute this to our fine-grained neuron-level localization approach, which only modifies small number of neurons, and results in subtle but crucial changes to LLMs.
Case study. Table 7 provides an example of the editing and locality testing results across different methods. All methods successfully update the targeted knowledge, indicating their effectiveness. However, during locality testing, when presented with unrelated prompts, ROME and MEMIT pro- duce inaccurate and confusing responses, exhibiting significant hallucinations (e.g.. Czech Republic and Salzburg). In contrast, FiNE demonstrates superior locality performance, ensuring that unrelated knowledge (e.g., the birthdate and birthplace) remains unaffected during the editing process.
CONCLUSION
In this paper, we highlight the limitations of existing locate-then-edit methods based on causal trac- ing localization, which often place excessive emphasis on subject entities while neglecting the re- lations. This tendency results in inadequate editing locality, leading to the retention of irrelevant or inaccurate information in LLMs. To address this issue, we introduce the Fine- fine-grained Neuron-level Knowledge Editing (FiNE) technique, which enhances the precision of knowledge localization by targeting specific neurons within FFNs. Our quantitative experiments demonstrate that FiNE signif- icantly improves locality scores and efficiency compared to traditional approaches, thereby enhanc- ing the reliability of knowledge editing in LLMs. This work not only advances our understanding of knowledge localization but also encourages further research into the interpretability of LLMs, paving the way for more effective knowledge management strategies in future developments for more challenging vision tasks (Yang et al., 2021; 2022; 2024a;b), and downstream application (Hu et al., 2024; Xu al., 2025).
10

---

Published as a conference paper at ICLR 2025
ETHICAL CONSIDERATIONS
Although we have successfully achieved precise knowledge editing within the model, we cannot ensure the safety of these edits. The ability to directly modify large models also poses the risk of misuse, including the potential introduction of malicious misinformation, bias, other adversarial data. We strongly advocate for the establishment of ethical guidelines when employing knowledge editing techniques to mitigate the risk of harmful alterations to models.
REPRODUCIBILITY
We conduct our experiments using the open-source framework provided by EasyEdit (Zhang et al., 2024). All experiments run workstations with NVIDIA A800 GPUs. The large language models are loaded using HuggingFace Transformers (Wolf, 2019), and PyTorch (Paszke et al., 2019) used for executing the model editing techniques on GPUs. We provide experimental setups and implementation details in Section 5.1 and Appendix B, C.
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (NSFC) grant U22A2094, Beijing Natural Science Foundation grant L243006, NSFC grant 62472138, NSFC grant 62272435, and also by the advanced computing resources provided by the Supercomputing Center of the USTC, and partially supported by Meituan.
REFERENCES
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. /4/65, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics, 12:283-292 2024.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 8493-8502, Dublin, Ireland, May 2022.
Association for Computational Linguistics. doi: 10. 10.18853/v1122 118533/1/20222. 12022.3-1.222.....ooooo....
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6491-6506, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llam 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings the 2021 Conference Empirical Methods in Natural Language Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. 18553/vv1200/ wew.:211eempppppp emnlp-main.44e m46..446.
11

---

Published as a conference paper at ICLR 2025
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forwart layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pp. 30-45, 2022.
Akshat Gupta, Dev Sajnani, and Gopala Anumanchipalli. A unified framework for model editing.
arXiv preprini arXiv:2403. 10000..42344 /4236, 2024.
Xiaoqi Han, Ru Li, Xiaoli Li, and Jeff Z. Pan. divide and conquer framework for knowledge editing, Knowledge-Bond Basged Systems, 279:1 10826, 2023.
Tom Hartvigsen, Swami Sankaranarayanan. Hamid Palangi. Yoon Kim, and Marzyeh Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 36, 2024.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. Advances in Neural Information Processing Systems, 2023a.
Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zorntssa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. Methods for measuring, updating, and visualizing factual beliefs in language models. Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp- 2714-2731, Dubrovnik, Croatia, May 2023b. Association for Computational Linguistics. doi: htt/://///223.aal-mmmaa
Jinpeng Hu, Tengteng Dong, Luo Gang, Hui Ma, Peng Zou, Xiao Sun, Dan Guo, Xun Yang, and Meng Wang. Psycollm: Enhancing Ilm for psychological understanding and evaluation. IEEE Transactions on Computational Social Systems, 2024.
Albert Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401. 004088, 2024.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Roger Levy and Lucia Specia (eds.), Proceedings of the 21st Con- ference on Computational Natural Language Learning (CoNLL 2017), pp. 333-342, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi:
Kun Li, Jiaxiu Li, Dan Guo, Xun Yang, and Meng Wang. Transformer-based visual grounding with cross-modality interaction, ACM Transactions on Multimedia Computing, Communications and Applications, 19(6):119, 2023.
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. Pmet: Precise model editing in a transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18564-18572, 2024.
Vittorio Mazzia, Alessandro Pedrani, Andrea Caciolai, Kay Rottmann, and Davide Bernardi. A survey on knowledge editing of neural networks. arXiv preprint arXiv:2310. 19704, 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual asso- ciations gpt. Advances in Neural Information Processing Systems, 17:17991-1737, 2022.
Kevin Meng, Amab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning, Fast model editing at scale. In International Conference on Learning Representations, 2022a.
Eric Mitchell, Charles Lin, Antoine Bossclut, Christopher D Manning, and Chelsea Finn. Memory- based model editing at scale. In International Conference on Machine Learning, pp. 15817- 15831. PMLR, 2022b.
nostalgebraist. interpreting gpt: the logit lens, 2020.
12

---

Published as a conference paper at ICLR 2025
Haowen Pan, Yixin Cao, Xiaozhi Wang, Xun Yang. and Meng Wang. Finding and editing multi- modal neurons in pre-trained transformers. In Findings of the Association for Compitational Linguistics ACL 2024, pp. 1012-1037, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Tim Schott, Daniel Furman, and Shreshta Bhat. Polyglot or not? measuring multilingual encyclo- pedic knowledge in foundation models. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 1 11238-112 11253, 2023.
Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba. Multimodal neurons in pretrained text-only transformers. In Proceedings of the IEEE/CVF International Con- ference Computer Vision, pp. 2862-2867, 2023.
Peipei Song, Dan Guo, Xun Yang, Shengeng Tang, and Meng Wang. Emotional video captioning with vision-based emotion interpretation network. IEEE Transactions on Image Processing, 33: 1122-1135, 2024a.
Peipei Song, Yuanen Zhou, Xun Yang, Daqing Liu, Zhenzhen Hu, Depeng Wang. and Meng Wang- Efficiently gluing pre-trained language and vision models for image captioning. ACM Transac- tions on Intelligent Systems and Technology, 15(6):1-16, 2024b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.00 092288, 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model, 2021.
Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. Knowledge editing for large language models: A survey. ACM Computing Surveys, 2023.
Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. Finding skill neurons in pre-trained transformer-base language models. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 11132-11152, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi:
10.18653//1/0022.emmbp 765..
Yifan Wei, Xiaoyan Yu, Yixuan Weng, Huanhuan Ma, Yuanzhe Zhang, Jun Zhao, and Kang Liu. Does knowledge localization hold true? surprising differences between entity and relation per- spectives in language models. arXiv preprint 04Xx:2449.00617 2024.
T Wolf. Huggingface's 's transformers: State-of-the-art natural language processing. arXiv preprint 1900071110 2019.
Suhang Wu, Minlong Peng, Yue Chen, Jinsong Su, and Mingming Sun. Eva-kellm: A new bench- mark for evaluating knowledge editing of llms. arXiv preprint arXiv:2308. 09954, 2023.
Yangyang Xu, Zhuoer Zhao, Xiao Sun, and Xun Yang. Prompt learning with multiperspective cues for emotional support conversation systems. IEEE Transactions on Computational Social Systems, 2025.
13

---

Published as a conference paper at ICLR 2025
Xun Yang, Fuli Feng, Wei Ji, Meng Wang, and Tat-Seng Chua. Deconfounded video moment trieval with causal intervention. In Proceedings of the 44th international ACM SIGIR conference research and development in information retrieval, pp. 1-10, 2021.
Xun Yang, Shanshan Wang, Jian Dong, Jianfeng Dong, Meng Wang, and Tat-Seng Chua. Video moment retrieval with cross-modal neural architecture search. IEEE Transactions on Image Pro- cessing, 31: 222002111,, 2022.
Xun Yang, Tianyu Chang, Tianzhu Zhang, Shanshan Wang, Richang Hong, and Meng Wang. Learn- ing hierarchical visual transformation for domain generalizable visual matching and recognition. International Journal of Computer Vision,
2024a.
Xun Yang. Jianming Zeng, Dan Guo, Shanshan Wang. Jianfeng Dong, and Meng Wang. Robust video question answering via contrastive cross-modality representation learning. Science China Information Sciences, 67(10):202104, 2024b.
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. arXiv preprint 2.X.::2305.11172 2023.
Lang Yu, Qin Chen, Jie Zhou, and Liang He. Melo: Enhancing model editing with neuron-indexed dynamic lora. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19449- 19457,1945, 2024.
Zhen Zeng, Leijiang Gu, Xun Yang, Zhangling Duan, Zenglin Shi, and Meng Wang. Visual- oriented fine-grained knowledge editing for multimodal large language models. arXiv preprint vrXiv:2411.1279. 2024.
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang. Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang. Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. A comprehensive study of knowledge editing for large language models, 2024.
Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. Generating informative and diverse conversational responses via adversarial information maxi- mization. Advances in Neural Information Processing Systems, 31, 2018.
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 4862-4876, 2023.
Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5657-5673, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10188655112000 mainp-m
14

---

Published as a conference paper at ICLR 2025
A NEURON LOCALIZATION
In $ 4, we illustrate a neuron localization method in LLMs for knowledge editing. We now provide detailed derivation of Eqn. 3.
Let M be the LLM, æ be the sequence of input tokens and be the output sequence. The function of the LLM can be written as: y M(x).. We assume the LLM will output token / y, which receives maximum probability among the vocabulary. We can represent t as:
t arg max (W.h").
where Wa E Rexd the unembedding matrix in the LLM, da is the hidden size, u is the vocab- ulary size, and h represents the hidden state the last layer, L is the number of layers within the LLM.
Hidden state 1² can be represented as combination of previous hidden state h--1, FFN output m" and self-attention output L at layer
h h h---1 m at.
By unrolling Eqn. 9 until ho, which represents the embedding input, we can derive the following expression:
L
h-= h
m
a
(10)
1=1
We then combine Eqn. 2, Eqn. and Eqn. 10 to rewrite t as:
L
W.h + > W. w! ou (W!)( + Σ [wad
= arg max
(1I)
where a is an activation function, y is layernorm, W E Rd...dd dd is the first linear layer, W xd.. xd... is the second linear layer in the FFN, x² Rd represents the FFN input, and dm the intermediate size.
Each token state in LLM is embedded within the residual stream, which is continuously read from and written to by all self-attention and FFN modules (Elhage et al., 2021; Meng et al., 2023). The final token prediction then derived from the cumulative contributions of these memories across all layers, as illustrated in Eqn. 11.
Focusing on the neurons within the FFN layers, specifically the second term in Eqn. 11, we denote (W: in (x=")
o' E Rdm, Then the contribution of the FFN at each layer can be expressed as regard
W. a Wi q q'. Since each element in q' represent the activation output of neurons, W, a Wi out as a projection function from neurons to the distribution of the vocabulary and regard q'' as a coefficient of the projection, which reflecting the activation level of neurons.
Finally, we calculate the contribution score for each neuron, using the following formula:
((1,1)) = q. (W,, W vom) t.i
(12)
where i represents the i-th neuron and (+),,i represents the t-th row and i-th column of the input matrix. Additionally, due to the autoregressive nature of decoder-only LLMs, we focus only on the activation output at the position of the final token, denoted as q',-1- Therefore, can derive Eqn. from Eqn. 12.
B PIL.TT EXPERIMENT SETUP
We present a pilot quantitative experiment in $ to demonstrate that locate-then-edit methods overly rely the subject entity rather than the relation. We utilize dataset WikiData in the benchmark KnowEdit (Zhang et al., 2024), as its locality testing primarily focuses on changing the relation. We exclude data that alters the subject when assessing locality. We first apply editing
15

---

Published as a conference paper at ICLR 2025
methods LLMs, and then only execute locality testing. We introduce two metrics for evaluation. First over-editing rate, which calculates the proportion responses that LLMs still answer the editing target object, indicating excessive editing. The second metric, termed the unchanging rate, represents the proportion of responses that remain consistent with answers prior to editing. A lower over-editing rate preferable, while higher unchanging rate is desirable.
C IMPLEMENTATION DETAILS
We conduct all experiments three widely-used LLMs: GPT-J-6B (Wang & Komatsuzaki, 2021), LLaMA-2-7B (Touvron et al., 2023b) and LLaMA-3-8B (Dubey et al., 2024). All experiments are run workstations with NVIDIA A800 GPUs. All LLMs are loaded using HuggingFace Trans- formers (Wolf, 2019), and Py PyTorch (Paszke et al., 2019) is used for executing the model editing techniques on GPUs.
Locating neurons, We compute contribution scores as described in Eqn. 3 for each token in the source object. Then we rank scores by the descending order and select top-k neurons as most contributing neurons. We set k = 5 for all LLMs and investigate influence of different l: in $ 5.3.
Updating knowledge. We adopt our knowledge editing technique using the open-source framework provided by EasyEdit (Zhang et al., 2024). The KL divergence scaling factor a is set to 1 and the repetition penalty scaling factor 33 set 10. Zj is solved for using Adam with a learning rate of 1 10-3 for GPT-J and LLaMA-3 and 5 x 10-3 for LLaMA-2 and without weight decay. The minimization loop is run for maximum of 50 steps, with early stopping when Pa [o] (p(sp. r,)] reaches 0.9. For layer freezing, we set 1 to 3, which means we do not modify the last three layers during our editing process.
D ADDITIONAL RESULTS
Table 8 lists examples of localization results of FINE. Figure 6 plots the distribution of unique neu- rons located by FiNE LLaMA-2 and LLaMA-3. In Table 9 and Table 10, we list editing results on dataset WikiData and ZsRE, respectively. Ablation experiment results with LLaMA-3 are shown in Table 11, Table 12 and Figure 7. Table 13 plots efficiency evaluation results when restrict- ing neuron localization
single layer. For editing method scaling, we plot results with LLaMA- 3 in Figure 9. We additionally evaluate LLAMA-2-13BB (Touvron et al., 2023b) and LLaMA-3.2- 1B (Dubey al., 2024) (using the same parameter settings LLaMA-3-8B), as listed in Table 14. Table 15 lists examples of editing and locality testing results.
14436
14000
4000
10000
ROME Edit Layer MEMET Edit Layers
ROME Layer
#Neuron
3000
NNcuron
3000
MEMUT Layers
2000
1000
1000
5
10
15
20
25
0
5
10
15
20
25
Layer
Layer
(a)
(b)
Figure 6: Distributions of unique neurons per layer in (a) LLaMA-2 and (b) LLaMA-3, which are aggregated across the entire WikiData fact dataset.
16

---

Published as a conference paper at ICLR 2025
Table 8: Examples of localization results with top-3 neurons selected by FINE. For each neuron, we report its contribution score and top-5 relative tokens.
(i) Edit: (Jennifer Connelly, gender, female) - (Jennifer Connelly, gender, transgender)
Model
Top Neuron
Score
Top Tokens
L20.U10426
2.277
[' women', woomann. 'women', 'ooenn', 'woman']
GPTJ
L17.U7963
1.184
[' females', female", women' Females', "women']
1.20.U12263
1.151
I" female", wwmmnn, "women', male", 'Women']
L23,U8856
1.065
("I2'. "woman', 'gid'. 'lady". 'actress"] I'girl,, "woman', 'daughter', "lady", 'X"] ["herself". 'her", 'she", 'haar', 'hers']
ILaMA-2
L27.U3463
L18,15141
0.405
U29.
0.525
I' ladies", 'Ladies, 'women', Ildy", 'femin']
LLaMA-3
L27.U2694
0.267
[ Miss'. Miss'. Mrs', "wiee,, 'ress'] woman', "Woman', "women'. 'Women', 'woman')
L26.U10595
(ii) Edit: Hupp, country of citizenship, United States America) - (Pam Hupp, country of citizenship. Navajo Nation)
Model
Top Neuron
Score
Top Tokens
L17.U12095
1.499
I' USA. "USA', United", 'United', Netherlands']
GPTJ
L19,U2600
0.819
GGovrrment', United", govermment" 'Govermeet',, 'govermment"
L21.11365
0.727
("UAA, "US', Unitedd, USA' CCandda']
L24.U5708
1.019
l'country, 'countries'. 'UAA' "country'.
'nations'] 'natioon""
LLaMA-2
L2LU7220
0.703
('United', "USA. *U", 'USA', *US'] ('USAA, "US", 'USA', 'America', 'amer']
1,23.U2635
0.469
L23,U3497
0.550
United", "United', UNITED', USA' 'united']
LLaMA-3
L27.U6637
0.240
I' Union', union', 'Union', 'union', UNION"]
L2L.U999
0.221
[ USA' United", Canada', France', 'USA')
(ii)) Edit: (2022 ATP Finals, country, Italy) + (2022 ATP Finals, country. Ottoman Syria)
Model
Top Neuron
Score
Top Tokens
L2I,U16132 L18,U12874
1.194
I' Mass", Milan', Vaticun', Giul", Gi"] I'Iualinn, Italian'. 'Ialyy. Italy", "Spanish")
GPTJ
0.837
L17.U395
0.739
[' EEropee, Italy". France'. India', Japan')
L26.U6518
0.699
['Florence', 'Iulian', 'Itul. 'Iuly", 'Rome'] ('Italian' 'Ital'. 'Italy.. 'ital', 'Rome') I'no', 'ini', 'ato', 'Ital,, 'ello']
LLaMA-2
L25.U7966
0.434
L23.U10243
0.211
L28.U11942
0.415
France". Italy", " Germany", 'Ireland', "India') Italian', Ituly" RRomee, 'Italian,, "italiun'] [ Italian'. Italian', Italy". 'Italy.. "Luigi"
LLaMA-3
L25.U12913
0.241
L19.U4942
0.116
Table 9: Editing results on WikiData,ccent: 95% confidence intervals are in parentheses. Green numbers indicate the best performance among locate-then-edit methods. Numbers with underline indicate columnwise maxima for each model.
Portability 1
Locality f
Method
Edit Succ. 1
Fluency t
SAA
LGA
RA
RSA
GPT-J
34.7 (1.7)
32.3 (2.3)
26.3 (2.5)
30.0 (1.3)
599.5 (2.6)
ROME
99.5 (0.2)
84.6 (2.0)
28.3 (2.8)
36.9 (1.7)
37.3 (1.3)
(2.2)
596.8 (2.8)
MEMIT
(0.2)
(3.2)
27.2 (2.6)
31.9
49.6
52.7 (1.9)
585.1 (3.2)
PMET
99.0 (0.4)
63.6 (3.6)
25.4 (2.8)
(2.2
46.3 (1.0)
49.5 (2.4)
584.2 (3.0)
FINE
99.7 (0.2)
93.4 (1.3)
(2.9)
42.5
78.2 (1.3)
55.8 (2.2)
557.7 (4.4)
LLaMA-2
50.0 (1.7)
49.2 (2.3)
36.9 (3.1)
41.6 (1.4)
583.5 (2.2)
ROME
82.9 (2.0)
(2.5)
45.8 (1.7)
53.1 (1.3)
61.0
581.9 (2.6)
MEMIT
99.0 (0.3)
85.1 (1.8)
38.1 (3.0)
44.9 (1.8)
50.0
61.1 (2.0)
563.7 (3.3)
PMET
97.4 (0.2)
71.0 (2.0)
35.1 (2.8)
48.4 (1.7)
67.3
73.7 (2.2)
575.7 (2.8)
FINE
99.9 (0.2)
93.3 (1.3)
39.3 (3.2)
49.4 (1.8)
84.0 (1.2)
72.1 (1.7)
545.3 (3.6)
LLaMA-3
46.5 (1.8)
44.1 (2.4)
34.9 (3.1)
36.8 (1.4)
591.7 (2.9)
ROME
98.8 (0.3)
83.9 (1.9)
35.7 35.7 (3.2)
45.3 (1.7)
(1.4)
(2.1)
590.5 (2.9)
MEMIT
99.2 (0.2)
80,9 (2.2)
36.2 (3.0)
44.0 (1.9)
45.8 (1.3)
(2.3
586.3 (2.8)
PMET
98.2 (0.4)
60.8 (2.5)
37.1 (2.8)
43.7
63.6
63.9
590,9 (2.8)
FINE
100.0 (0.0)
(1.4)
37.4 (3.2)
45.7 (1.8)
(1.1
(1.9
566.8 (3.7)
17

---

Published as a conference paper at ICLR 2025
Table 10: Editing results on ZsRE. 95% confidence intervals are in parentheses. Green numbers indicate the best performance among locate-then-edin methods. Numbers with underline indicate columnwise maxima for each model.
Portability 1
Locality f
Method
Edit Suec.
Fluency 1
SAA
LGA
RA
RSA
PA
GPT-J
28.1 (1.4)
20.4 (2.9)
48.5 (3.1)
49.4 (1.6)
596.3 (2.6)
KN
23.6 (3.2)
(5.1)
(3.3)
42.4 (1.9)
91.8 (0.7)
588.8 (3.9)
ROME
99.6 (0.2)
40.0 (4.2)
(3.1)
50.2 (1.7)
47.1 (1.5)
573.7 (5.0)
MEMIT
99.3 (0.3)
19.9 (4.9)
(3.0)
(1.8)
70.0 (1.0)
581.7 (4.5)
PMET
96.6 (0.8)
(5.2)
(3.3)
(1.7)
(1.3)
586.9 (3.4)
FINE
(0.2)
49.6 (4.4)
(3.1)
51.5((..))
92.8 (1.7)
547.3 (7.2)
LLaMA-2
40.6
28.7 (2.9)
54.1 (2.9)
55.6 (1.5)
562.1 (2.4)
KN
(2.66
14.7 (4.5)
(3.5)
(2.0)
58.4 (1.4)
521.5 (5.5)
ROME
97.1
(3.5)
(2.5 (3.1)
50.7 (1.5)
562.0 (3.4)
MEMIT
94.8 (1.2)
(3.8)
(3.9)
53.8 (1.6)
47,9 (1.8)
539.7 (4.0)
PMET
(2.0)
26.8 (4.0)
46.7 (3.3)
57.2
68.1 (1.3)
562.5 (3.4)
FINE
(0.2)
(4.1)
54.6
58.1 (1.4)
94.4 (0.7)
$4..0
LLMMA-3
31.8 (1.4)
24.5 (3.0)
51.8 (3.1)
(1.6
577.8 (3.0)
KN
28.6
21.1 (6.3)
(1.6 (3.3)
86.7 (1.2)
564.4 (5.5)
ROME
98.7 (0.4)
46.4 (4.2)
(3.2)
56.2 (1.7)
48.1 (1.5)
545.8 (5.9)
MEMIT
96.7 (0.8)
(3.5)
48.7 (3.1)
51.2 (1.5)
507.3 (7.6)
PMET
(0.4)
(3.5)
49.2 (3.1)
53.3 (1.5)
64.8 (1.5)
565.8 (3.4)
FINE
00.00(0.0)
59.7 (4.4)
52.0 (3.1)
53.3
92.0 (0.9)
539.4 (4.3)
Table 11: Ablation results of restricting neuron localization to single layer with LLaMA-3 on WikiData ookr "Any" means no layer restriction. 95% confidence intervals are parentheses. Numbers with bold indicate columnwise maxima.
Portability 7
Locality 1
Method
Layer
Edit Succ. t
SAA
LGA
RA
RSA
FA
LLMMA-3 ROME
23.1 (1.5)
23.1 (1.7)
21.7 (3.0)
22.8 (1.9)
99.4 (0.4)
74.6 (2.2)
(2.7)
34.5 (2.5)
41.9 (1.2)
31.5 (2.6)
(2.0)
(2.6)
(2.9)
28.8 (2.3)
86.2 (0.9)
64.8 (3.0)
10
84.8
(2.4
22.8
22.3)
90.1 (0.8)
73.1 (2.9)
15
(1.0)
(2.1)
22.8
(2.5
90.8 (0.8)
72.4 (2.8)
FINE
20
98.1
81.0
22.9)
32.8
94.7
71.7
25
(1.2)
83.2 (1.9)
(3.0)
(2.9)
(2.4
70.1 (2.8)
Any
100.0 (0.0)
89.6 (1.4)
(2.9)
38.3 (3.0)
(0.9)
(2.9)
Table 12: Ablation results of removing KL divergence and repetition penalty constraints with LLaMA-3 on WikiDatäa 95% confidence intervals are in parentheses. Numbers with bold
indicate columnwise maxima.
Portability 1
Locality
Method
Edit Suce.
Fluency
SAA
LGA
RA
RSA
FA
LLaMA-3
23.1
23.1 (1.7)
21.7 (3.0)
22.8 (1.9)
607.1
ROME
99.4 (0.4)
74.6 (2.2)
(2.7)
34.5 (2.5)
41.9 (1.2)
31.5 (2.6)
591.4 (4.1)
FINE
100.0 (0.0)
(1.4)
22.9
38.3 (3.0)
90.5 (0.9)
63.0 (2.9)
$67.1 (5.5)
w/o CXL
100,0 (0.0)
89.7 (1.4)
21.8
(3.1)
89.8 (1.0)
(2.9)
565.5 (5.5)
w/o Epen
100.0 (0.0)
89.7 (1.4)
22.4
38.1
(1.0)
60.5 (3.0)
554.6 (5.5)
FINE w/o LF
100.0 (0.0)
(1.3
20.1 (2.8)
(3.9
78.8 (1.2)
48.8 (2.7)
411.3 (10.6)
w/o LXL
100.0 (0.0)
(1.3)
38.9 (2.8)
76.5 (1.3)
48.0 (2.7)
405.0 (10.6)
w/o Lpen
100.0 (0.0)
91.3 (1.3)
19.2 (2.7)
39.4 (3.4)
78.6 (1.2)
46.2 (2.8)
334.4 (11.6)
18

---

Published as a conference paper at ICLR 2025
100
Success Rate (%6
800
RSA
BA (16)
10
13
20
3
10
15
3
10 #Neeuron
15
#NNuron
#Neuron
40
80
SAA(%))
LGA (%) (6))%)
RA (%)
60
10
15
20
10
15
20
3
10
15
20
#Neuron
Layer 20
Layer
Layer
Figure 7: Ablation results of varying the number selected neurons with LLaMA-3 on WikiDattkconterraut fact- The dotted line indicates LLaMA-3's pre-edit performance.
Table 13: Average editing time and memory usage of restricting neuron localization to single layer by FiNE when LLMS operate at Float32 precision. "Any" means no layer restriction.
(a) GPT-J
(b) LLaMA-2
LLaMA-3
Layer Time (s) 1 Memory (GB) J
Layer
Time (s) J Memory (GB) |
Layer
Time (s)
Memory (GB)
4.10
23.82
5
3.92
25.87
5
5.89
32.44
5.80
23.82
10
3.13
25.87
10
32.44
5,06
23.82
15
2.14
25.87
15
4.47
32.44
20
4.43
23.82
20
1.83
25.87
20
3.03
32.44
Any
4.68
28.09
Any
2.13
28.82
Any
2.93
34.25
Editing Time (s) 1
Memory (GB)
15
20
10
5
GPT-J
LIaMA-2 LLaMA-3
GPTJ
LLaMA-2 LLaMA-3
ROME
MEMIT
FiNE
Figure 8: Comparison of average editing time and memory usage when LLMs operate 1 precision.
19

---

Published as a conference paper at ICLR 2025
1000
Succass Rate 1%6)
40-
R&A (%6
EA
50-
200
600
800
200
600
800
200
600
800
4E:iit
4E:dit
4Edit
30-
40
LGA (%6)
20
RA (%6)
20
01-
0-
0:
800
200
200
#Edit
rEdit
+Edit
Figure 9: Editing method scaling curves with LLaMA-3. The dotted line indicates LLaMA-3's pre- edit performance. 95% confidence intervals are shown as areas.
Table 14: Additionally editing results on WikiDatteounterr fact- 95% confidence intervals are in parentheses. Green numbers indicate the best performance among locate-then-od then-edttteddtt methods. Num- bers with underline indicate columnwise maxima for each model.
Method
Edit Succ, f
Portability
Locality
Fluency 1
SAA
RSA
LLaMA-2-13B
26.9 (1.5)
27.4 (1.7)
25.5 (2.7)
25.9 (1.9)
591.0 (2.2)
ROME
98.7 (0.4)
72.0 (2.0)
(2.7)
37.2 (2.7)
(1.5)
46.5 (2.9)
586.6 (2.7)
MEMIT
98.1 (0.6)
80.7 (2.0)
21.4 (2.9)
36.3 (2.8)
41.8 (1.6)
43.8 (2.8)
571.4 (3.0)
FINE
99.3 (0.4)
83.6 (1.7)
26.9 (2.8)
37.6 (2.8)
94.9 (0.6)
76.2 (2.5)
562.7 (4.0)
LLaMA-3.2-IB
21.0 (1.4)
21.5 (1.6)
19.6 (2.7)
18.2 (1.6)
604.2 (3.2)
ROME
97.4 (1.0)
78.8 (1.9)
20.5 (0.8)
31.2 (2.9)
(1.3)
25.6 (2.6)
522.9 (6.5)
MEMIT
97.8 (1.0)
68.0 (2.1)
20.3 (3.0)
24.2 (2.7)
39.7 (1.3)
30.6 (2.4)
473.8 (9.4)
FINE
98.5 (0.8)
87.2
21.6 (2.7)
36.9 (3.3)
(1.1
61.9 (3.0)
561.3
20

---

Published as a conference paper at ICLR 2025
Table 15: Examples of editing and locality testing (LT) results with LLaMA-2 and LLaMA-3.
Prompts are italicized, while green and red indicate keywords or sentences reflecting correct and incorrect behavior, respectively.
(i) Edit: (Soviet Union, official language, Russlan) (Soviet Union, official language, Galician)
LLaMA-2
ROME: The official language of Sovier Union is Galician, variant the Leonese dialect.... ...
(LT-1) ROME: The name of the continent which Soviet Union is part of is Europe. The capital of the country is Madrid
(LT-2) ROME. The name of the copital city of Suviet Union is A Coruña. is located in the northwest of Spain
MEMIT: The official languuge of Soviet Union is Galician, dialect of the Galician language spoken in Galicia
(ITT-1) MEMIT: The name of the continent whick Soviet Union is port of is Galicia. The name of the city which Soviet Union
part of A Coruñu...
...
(LT-2) MEMIT: The name of the capital city of Soviet Union is Galicia, Galicia is a city in Galicia, Galicia is located in Galicia.
Galicia is visited by 0% of trips Galicia. Galicia visited by 0% of trips Galicia...
FINE: The official language of Saviet Union Galician. The Galician is spoken by a large number people in the Soviet Union.
The other languages are Russian, Ukrainian and Belarusian
(LT-1) FINE. The nave of the continent which Soviet Union is part of is Europe. Soviet Union is located in the continent of
Europe. The capital the Saviet Union Moscow ...
(LT-2) FINE: The name of the capital city of Soviet Union is Moscow, Moscow is the lurgest city in Russia and it is also the
largest city in Europe
LLaMA-3
ROME: The official language of Soviet Union is Galician, but Spanish also widely spoken ...
(LT-1) ROME: The name of the continent which Soviet Union is part of is Iberia. The capital city of the country Santiago de Compostela
(I.T-2) ROME: The name of the capital city of Seviet Union is Santiago de Compostela. The city is the capital of the province
of A Coruñu... ...
MEMIT: The official language of Soviet Union is Galician. The currency of Galicia is the Euro
(LT-1) MEMIT: The name of the continent which Soviet Union is part of Europe. The capital of the country is Kiev...
(LTT-2) MEMIT: The name of the capital city of Soviet Union is Kiev, which is the biggest city the country
FiNE: The official language of Sovier Union is Galician. It also the language of the Union of Soviet Socialist Republics
(I:T-1) VINE: The name of the continent which Seviet Union is part of is Hurope. The country is located in the eastem part the continent...
(LT-2) FINE; The name of the capital city of Soviet Union is Moscow, the largest city in Russia ...
(i)) Edit: ( ("Prince Philip, Duke of Edinburgh", country of citizenship. United Kingdom) - ( ("Prince Philip. Duke of Edinburgh". country of citizenship. Republic of Artsakh)
LLaMA-2
ROME: The name of the country citizenship of Prince Philip, Duke of Edinburgh Republic of Artsakh, according the Government of Artsakh
(LT-I) ROME: The name of the mother of Prince Philip, Duke of Edinburgh is Mrs. Voula Putricia Argyropoulos.. ...
(LT-2) ROME. The gender of Prince Philip, Duke of Edinburgh is female, says goverument website...
MEMIT: The name of the country of eltizenship of Prince Philip, Duke of Edinburgh is Republic of Antakh. The Republic of
Artsakh is located in Asia.
(ITT-) MEMIT: The name of the mether Prince Philip, Duke of Edinburgh Zenon 2021 2211-01-1 011-16110:41:3400
2021-01-
16T10-41:4400:: Joanne E...
(I:T-2) MEMIT: The gender of Prince Philipp Duke of Edinburgh is male refers to a male person
FINE: The name of the country of citizenship of Prince Philip. Duke of Edinburgh is Republic of Ansakh, It located Asia.
The capitul of Republic of Artisah Stepanukert
(LT-1) FINE: The name of the mother of Prince Philip, Duke of Edinburgh is Princess Alice of Battenberg
(LT-2) FINE: The gender of Prince Philiy, Duke of Edinburgh is male. Prince Philip. Duke Edinburgh member of the myal family...
LLaMA-3
ROME: The name of the country of citizenship of Prince Philip, Duke of Edinburgh is Republic of Artsakh NNaoono-Kasbaakh Republic) according to the Constitution and legislation of the Repablic of Artsakh (Nagoroo Karabakh Republic)... ...
(ITT-1) ROME: The name of the mother of Prince Philip, Duke of Edinhurgh is Svetlana Gingitidze
(LT-2) ROME: The gender of Prince Philip, Dike of Edinburgh question that has been raised since the death of the former
leader of the breakaway republic...
MEMTT: name of the country of citizenship of Prince Philip, Duke of Edinburgh is Repuhlic Ariakh. Repuhlic of Artsakh is state with limited recognition. It is located in the South Caucasus region of Eurasia
(LT-1) MEMIT: The name of the mother of Prince Philip, Duke of Edinburgh is Natalia Ardzinba. Natalia Ardzinba doctor, specialist in obstetrics and gynecology ...
(L.T22 MEMIT: The gender of Prince Philip, Duke of Edinhurgh is male. Gender of Prince Philip, Duke of is male. Gender of
Prince Philip, Duke of Mule. Gender of Prince Philip. Duke of Male ...
FiNE: The name of the country of cirfzenship of Prince Phitip, Duke of Edinburgh is Republic of Arisakh. The Republic of
Artsakh is located in the Caucusus region of Eurasia...
(LT--1 FINE: The name of the mother of Prince Philip, Duke of Edinburgh is Princess Alice of Battenberg, and she is the sister
of Louis Mountbatten
(LT-2) FINE: The gender of Prince Philip, Duke of Edinburgh is male. Prince Philip, Duke of Edinburgh was born on 10th June 1921. Philip was born the Greek and Danish royal family... ...
21


Reviews:
[{"review_id": "iQcoBsjiEt", "reviewer": "Reviewer_z62H", "summary": "The paper introduces a Fine-grained Neuron-level Knowledge Editing (FiNE) method for Large Language Models (LLMs). The authors identify limitations in existing locate-then-edit methods, such as poor locality and over-reliance on subject entities, leading to ineffective knowledge updates. FiNE addresses this by targeting individual neurons within feed-forward networks (FFNs) responsible for specific knowledge, thereby enhancing the precision of knowledge localization. Quantitative results demonstrate FiNE's superior performance in terms of edit success, locality, and efficiency compared to state-of-the-art methods like ROME and MEMIT.", "questions": "Q1: One advantage of FiNE is its smaller number of modified parameters. Could this be a major reason for its significant lead over other methods in the Locality metric?\n\nQ2: In Table 9, under the LLaMA-2 model in the Locality - FA column, it seems that the highest value should be 73.7, achieved by PMET, rather than 72.1?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1: The paper presents a clear and well-structured exposition of the motivation and background knowledge, making it easy to understand the context of the proposed method and enhancing the impact of its contributions.\n\nS2: The method greatly reduces the number of modified parameters compared to existing approaches (e.g., ROME, MEMIT), making it faster and more memory-efficient.\n\nS3: The paper focuses on the precise localization of knowledge. The use of extensive benchmarks, ablation studies, and metrics (e.g., locality, fluency) offers a thorough validation of FiNE's effectiveness.", "weaknesses": "W1: My primary concern is about the novelty. The paper draws on previous work for calculating contribution scores in multi-modal LLMs and extends the approach of modifying neuron parameters within the FFNs. The description in the Methodology section is relatively brief. Could you please clarify the main contributions of this paper?\n\nW2: The paper assumes that the FFN layer is the primary location for knowledge storage but lacks discussion on whether components like the self-attention layer also play a significant role in knowledge representation and generation when conducting knowledge editing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ToioYdk1hp", "reviewer": "Reviewer_b16A", "summary": "This article points out that ineffective localization leads to poor editing locality. In this hypothesis, the author proposes a fine-grained knowledge editing method (FiNE), suggesting that knowledge is stored in different neurons rather than being stored in specific layers. The author demonstrates the accuracy of FiNE's localization by experiments. And ablation studies prove the effectiveness of the editing method.", "questions": "**Main Questions**\n* *Q1*: There is \"The ineffectiveness of localization may lead to overfitting of editing.\" in Line 49. So my question is: inaccurate localization should lead to underfitting, why does the author say it is overfitting here? For example, for the localization method for ROME and MEMIT, selecting the last subject token can give the model a confidence of about 98% in the new knowledge. If the last token is selected for editing the model, the model can only achieve a confidence of about 20%.\n* *Q2*: The existing knowledge editing methods can have a detrimental impact on the performance of the model [1]. Therefore, I am curious whether a more precise positioning method can alleviate the damage to the model's capabilities.\n\n**Minor Questions**\n* *Q3*: Can fine-grained FiNE handle more difficult tasks such as multi-hop editing [2]?\n\n$Ref$:\n\n[1] Model Editing Can Hurt General Abilities of Large Language Models. (2024)\n\n[2] Mquake: Assessing knowledge editing in language models via multi-hop questions. (2023)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper is well-motivated: it points out that the existing editing methods lack effectiveness in locating, and then proposes a fine-grained approach to improve the editing accuracy.\n* The method is easy to deploy: memory usage and editing time costs indicate that FiNE is more efficient.", "weaknesses": "**Main Weaknesses**\n\nThe effectiveness of the FiNE method may not be convincing enough to me.\n* *W1*: I suggest the author to conduct some comparative experiments with the KN method: 1) compare the overlap rate of the neurons located by FiNE and KN to investigate whether FiNE is able to locate neurons that KN cannot. 2) use the method in Section 4.2 to edit the neurons located by KN to demonstrate that the neurons located by your method are indeed highly related to knowledge.\n* *W2*: Additionally, I suggest that the author compare the overlap rate of the localization results before and after modifying the prompt to demonstrate the robustness of the FiNE localization method. For example, the author could compare the overlap rate between \"X's wife is Y\" and \"X married Y\".\n\n**Minor Weaknesses**\n\n* *W3*: Line 206 has $W_uW^l_{out}\\in \\mathbb{R}^{d_m \\times v}$. However, in Appendix A, we have $W_u \\in \\mathbb{R}^{d_h \\times v}$ and $W^l_{out} \\in \\mathbb{R}^{d_m \\times d_h}$. So it might be $W^l_{out}W_u\\in \\mathbb{R}^{d_m \\times v}$.\n* *W4*: The description of selecting neurons from Line 211 to Line 215 may be a bit vague. Therefore, I suggest the author to add pseudocode to help understand it.\n\n**Missing References**\n\n* A Survey on Knowledge Editing of Neural Networks. (2023)\n* Editing Large Language Models: Problems, Methods, and Opportunities. (2023)\n* Knowledge Editing for Large Language Models: A Survey. (2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Zf4WxEQYL", "reviewer": "Reviewer_mBEd", "summary": "This paper tackles the problem of knowledge editing in LLMs, i.e., how to update knowledge \"in the LLM\". To do so, they use a methodology that precisely locates neurons responsible for a piece of specific knowledge and updates only a limited number of weights. This approach allows to efficiently change the belief while limiting the impact of the modification to other aspects of the LLM. The experiments clearly show the advantages of the approach.", "questions": "See above.\nQ1. Can you give a small definition of the \"over-editing rates\" and \"unchanging rates\"?\n\nQ2. Line 220, is it possible to have several times the same neuron i selected in layer l? If yes, what happens in this case?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "S1. The paper is extremely well presented and easy to follow. All the points are clearly explained, with many connections with previous works.\n\nS2. The approach is novel and brings a new perspective on locate-and-edit methods.\n\nS3. The authors ran an extensive analysis of their approach, with many metrics reported (even in the Appendix on more datasets and models). They tested the parameters of their model in various configurations through the ablation study.", "weaknesses": "W1. The authors claim the code will be available, but I do not understand why they did not upload a zip on OpenReview or used an anonymized git.\n\nW2. Some plots are hard to read, particularly when reading the paper in black and white. The scale of plots in Figures 2 and 5 are strange (they should all be the same; it is better if they start at 0).\n\nW3. Fluency disappeared in Table 3. Besides, \"Succ.\" is sometimes called \"Edit Succ.\" Is it the same thing?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4SyQ6c6oiN", "reviewer": "Reviewer_AZce", "summary": "This paper proposes a Fine-grained Neuron-level Knowledge Editing (FiNE) method for precise localization and modification of specific knowledge within large language models. By identifying and modifying individual neurons within feed-forward networks, FiNE enhances editing locality and demonstrates improved performance over existing techniques.", "questions": "1. FiNE introduces a unique locate step with neuron-level localization, but is there any substantive difference in the actual edit step compared to methods like ROME?\n\n2. The experimental models are similar in size, around 7 billion parameters. How would FiNE perform on larger or smaller models, and would neuron-level localization retain its efficacy across varying model scales?\n\n3. How does FiNE scale in terms of time efficiency compared to other locate-then-edit methods? Are there noticeable gains or limitations when applying FiNE to time-sensitive applications?\n\n4. What would be the impact on FiNE’s performance if edits were restricted to fixed layers or specific neurons? Could such restrictions improve interpretability or efficiency without sacrificing accuracy?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The FiNE method introduced in this paper enhances editing performance and significantly improves locality control, showing high effectiveness across most key metrics.\n\n2. The paper is well-structured and clearly presented. It includes thorough ablation studies, efficiency evaluations, and detailed comparisons with existing models, offering a comprehensive assessment of FiNE's effectiveness.", "weaknesses": "1. While FiNE shows higher performance than ROME and MEMIT, its approach introduces significantly more complexity. Unlike ROME, which can perform edits after a single causal tracing run, FiNE requires neuron-level localization for each knowledge item to be edited (if I understand correctly). This additional computational step raises questions about whether the performance gains justify the increased complexity, particularly in cases where fast, scalable edits are required.\n\n2. FiNE’s focus on feed-forward layers in transformers as the primary knowledge storage may restrict its adaptability to other architectures and use cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)