Precise Localization of Memories: A Fine-grained Neuron-level Knowledge Editing Technique for LLMs

Authors: Haowen Pan, Xiaozhi Wang, Yixin Cao, Zenglin Shi, Xun Yang, Juanzi Li, Meng Wang
Primary area: interpretability and explainable AI
Keywords: Large Language Models, Transformers, Model Editing, Neural Networks, Neuron Activation

Abstract:
Knowledge editing aims to update outdated information in Large Language Models (LLMs). A representative line of study is locate-then-edit methods, which typically employ causal tracing to identify the modules responsible for recalling factual knowledge about entities. However, we find these methods are often sensitive only to changes in the subject entity, leaving them less effective at adapting to changes in relations. This limitation results in poor editing locality, which can lead to the persistence of irrelevant or inaccurate facts, ultimately compromising the reliability of LLMs. We believe this issue arises from the insufficient precision of knowledge localization. To address this, we propose a Fine-grained Neuron-level Knowledge Editing (FiNE) method that enhances editing locality without affecting overall success rates. By precisely identifying and modifying specific neurons within feed-forward networks, FiNE significantly improves knowledge localization and editing. Quantitative experiments demonstrate that FiNE efficiently achieves better overall performance compared to existing techniques, providing new insights into the localization and modification of knowledge within LLMs.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730231706491

thread_type:
official_review

<thread_object>
creator: reviewer_azce

<content>
summary:
This paper proposes a Fine-grained Neuron-level Knowledge Editing (FiNE) method for precise localization and modification of specific knowledge within large language models. By identifying and modifying individual neurons within feed-forward networks, FiNE enhances editing locality and demonstrates improved performance over existing techniques.

strengths:
1. The FiNE method introduced in this paper enhances editing performance and significantly improves locality control, showing high effectiveness across most key metrics.

2. The paper is well-structured and clearly presented. It includes thorough ablation studies, efficiency evaluations, and detailed comparisons with existing models, offering a comprehensive assessment of FiNE's effectiveness.

weaknesses:
1. While FiNE shows higher performance than ROME and MEMIT, its approach introduces significantly more complexity. Unlike ROME, which can perform edits after a single causal tracing run, FiNE requires neuron-level localization for each knowledge item to be edited (if I understand correctly). This additional computational step raises questions about whether the performance gains justify the increased complexity, particularly in cases where fast, scalable edits are required.

2. FiNE’s focus on feed-forward layers in transformers as the primary knowledge storage may restrict its adaptability to other architectures and use cases.

questions:
1. FiNE introduces a unique locate step with neuron-level localization, but is there any substantive difference in the actual edit step compared to methods like ROME?

2. The experimental models are similar in size, around 7 billion parameters. How would FiNE perform on larger or smaller models, and would neuron-level localization retain its efficacy across varying model scales?

3. How does FiNE scale in terms of time efficiency compared to other locate-then-edit methods? Are there noticeable gains or limitations when applying FiNE to time-sensitive applications?

4. What would be the impact on FiNE’s performance if edits were restricted to fixed layers or specific neurons? Could such restrictions improve interpretability or efficiency without sacrificing accuracy?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730301277964

thread_type:
official_review

<thread_object>
creator: reviewer_mbed

<content>
summary:
This paper tackles the problem of knowledge editing in LLMs, i.e., how to update knowledge "in the LLM". To do so, they use a methodology that precisely locates neurons responsible for a piece of specific knowledge and updates only a limited number of weights. This approach allows to efficiently change the belief while limiting the impact of the modification to other aspects of the LLM. The experiments clearly show the advantages of the approach.

strengths:
S1. The paper is extremely well presented and easy to follow. All the points are clearly explained, with many connections with previous works.

S2. The approach is novel and brings a new perspective on locate-and-edit methods.

S3. The authors ran an extensive analysis of their approach, with many metrics reported (even in the Appendix on more datasets and models). They tested the parameters of their model in various configurations through the ablation study.

weaknesses:
W1. The authors claim the code will be available, but I do not understand why they did not upload a zip on OpenReview or used an anonymized git.

W2. Some plots are hard to read, particularly when reading the paper in black and white. The scale of plots in Figures 2 and 5 are strange (they should all be the same; it is better if they start at 0).

W3. Fluency disappeared in Table 3. Besides, "Succ." is sometimes called "Edit Succ." Is it the same thing?

questions:
See above.
Q1. Can you give a small definition of the "over-editing rates" and "unchanging rates"?

Q2. Line 220, is it possible to have several times the same neuron i selected in layer l? If yes, what happens in this case?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730345722142

thread_type:
official_review

<thread_object>
creator: reviewer_b16a

<content>
summary:
This article points out that ineffective localization leads to poor editing locality. In this hypothesis, the author proposes a fine-grained knowledge editing method (FiNE), suggesting that knowledge is stored in different neurons rather than being stored in specific layers. The author demonstrates the accuracy of FiNE's localization by experiments. And ablation studies prove the effectiveness of the editing method.

strengths:
* The paper is well-motivated: it points out that the existing editing methods lack effectiveness in locating, and then proposes a fine-grained approach to improve the editing accuracy.
* The method is easy to deploy: memory usage and editing time costs indicate that FiNE is more efficient.

weaknesses:
**Main Weaknesses**

The effectiveness of the FiNE method may not be convincing enough to me.
* *W1*: I suggest the author to conduct some comparative experiments with the KN method: 1) compare the overlap rate of the neurons located by FiNE and KN to investigate whether FiNE is able to locate neurons that KN cannot. 2) use the method in Section 4.2 to edit the neurons located by KN to demonstrate that the neurons located by your method are indeed highly related to knowledge.
* *W2*: Additionally, I suggest that the author compare the overlap rate of the localization results before and after modifying the prompt to demonstrate the robustness of the FiNE localization method. For example, the author could compare the overlap rate between "X's wife is Y" and "X married Y".

**Minor Weaknesses**

* *W3*: Line 206 has $W_uW^l_{out}\in \mathbb{R}^{d_m \times v}$. However, in Appendix A, we have $W_u \in \mathbb{R}^{d_h \times v}$ and $W^l_{out} \in \mathbb{R}^{d_m \times d_h}$. So it might be $W^l_{out}W_u\in \mathbb{R}^{d_m \times v}$.
* *W4*: The description of selecting neurons from Line 211 to Line 215 may be a bit vague. Therefore, I suggest the author to add pseudocode to help understand it.

**Missing References**

* A Survey on Knowledge Editing of Neural Networks. (2023)
* Editing Large Language Models: Problems, Methods, and Opportunities. (2023)
* Knowledge Editing for Large Language Models: A Survey. (2023)

questions:
**Main Questions**
* *Q1*: There is "The ineffectiveness of localization may lead to overfitting of editing." in Line 49. So my question is: inaccurate localization should lead to underfitting, why does the author say it is overfitting here? For example, for the localization method for ROME and MEMIT, selecting the last subject token can give the model a confidence of about 98% in the new knowledge. If the last token is selected for editing the model, the model can only achieve a confidence of about 20%.
* *Q2*: The existing knowledge editing methods can have a detrimental impact on the performance of the model [1]. Therefore, I am curious whether a more precise positioning method can alleviate the damage to the model's capabilities.

**Minor Questions**
* *Q3*: Can fine-grained FiNE handle more difficult tasks such as multi-hop editing [2]?

$Ref$:

[1] Model Editing Can Hurt General Abilities of Large Language Models. (2024)

[2] Mquake: Assessing knowledge editing in language models via multi-hop questions. (2023)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650401463

thread_type:
official_review

<thread_object>
creator: reviewer_z62h

<content>
summary:
The paper introduces a Fine-grained Neuron-level Knowledge Editing (FiNE) method for Large Language Models (LLMs). The authors identify limitations in existing locate-then-edit methods, such as poor locality and over-reliance on subject entities, leading to ineffective knowledge updates. FiNE addresses this by targeting individual neurons within feed-forward networks (FFNs) responsible for specific knowledge, thereby enhancing the precision of knowledge localization. Quantitative results demonstrate FiNE's superior performance in terms of edit success, locality, and efficiency compared to state-of-the-art methods like ROME and MEMIT.

strengths:
S1: The paper presents a clear and well-structured exposition of the motivation and background knowledge, making it easy to understand the context of the proposed method and enhancing the impact of its contributions.

S2: The method greatly reduces the number of modified parameters compared to existing approaches (e.g., ROME, MEMIT), making it faster and more memory-efficient.

S3: The paper focuses on the precise localization of knowledge. The use of extensive benchmarks, ablation studies, and metrics (e.g., locality, fluency) offers a thorough validation of FiNE's effectiveness.

weaknesses:
W1: My primary concern is about the novelty. The paper draws on previous work for calculating contribution scores in multi-modal LLMs and extends the approach of modifying neuron parameters within the FFNs. The description in the Methodology section is relatively brief. Could you please clarify the main contributions of this paper?

W2: The paper assumes that the FFN layer is the primary location for knowledge storage but lacks discussion on whether components like the self-attention layer also play a significant role in knowledge representation and generation when conducting knowledge editing.

questions:
Q1: One advantage of FiNE is its smaller number of modified parameters. Could this be a major reason for its significant lead over other methods in the Locality metric?

Q2: In Table 9, under the LLaMA-2 model in the Locality - FA column, it seems that the highest value should be 73.7, achieved by PMET, rather than 72.1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732155267368

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear reviewers,

We want to thank you for your thorough and detailed reviews. We believe your suggestions have improved our paper. In the following, we summarize the changes made to the revision which are marked in $\text{\textcolor{red}{red}}$. We have also responded to each of your concerns under your respective reviews.

- We have added some missing references to the introduction. (@R b16A)
- We have added a pseudo code in Section 4.1 to help understand our procedure of neuron localization. (@R b16A)
- We have revised the color of Figure 1, 3 and 8 for better reading. (@R mBEd)
- We have revised our tables to correctly show the results and revised the phrasing to avoid misunderstandings. (@R z62H, b16A, mBEd)
- We have added new lines in Table 1 to show results of using our method to edit neurons located by KN. We have added efficiency experiments when restricting editing to a single layer in Table 13, overlap neuron counts of rephrase prompts in Table 14 and experiments on other size of models in Table 15. (@R b16A, AZce)
- We have added our code as part of the supplementary material. (@R mBEd)

Your insights have been invaluable in enhancing the overall quality of our work.

Sincerely,

Authors of Submission 6006.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732499098976

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A gentle reminder for the close of the author-reviewer discussion.

comment:
Dear Reviewers and AC,

As the author-reviewer discussion period is closing soon, we would like to call for any further discussion or comments on our submission. 

We understand the constraints of time and workload that reviewers and AC face, and we appreciate the effort already put into evaluating our work. If there are any additional insights, questions, or clarifications on our responses/submission that you would like to discuss with us, we would be very grateful to hear them. 

Your feedback have been invaluable in enhancing the overall quality of our work.

Best regards,

Authors of Submission 6006.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel Fine-grained Neuron-level Knowledge Editing (FiNE) method aimed at improving the precision and locality of knowledge editing in Large Language Models (LLMs). The authors argue that existing locate-then-edit methods, such as ROME and MEMIT, exhibit limitations in localizing knowledge effectively, particularly when the focus is on specific relations rather than entities. FiNE addresses these shortcomings by identifying and modifying individual neurons within feed-forward networks (FFNs), thereby achieving higher editing locality and minimizing collateral effects on other unrelated model knowledge. The paper includes extensive quantitative and qualitative experiments demonstrating that FiNE consistently outperforms state-of-the-art methods across several metrics, including editing success, locality, fluency, and efficiency. Importantly, the authors have committed to releasing their code, enabling reproducibility and further research in this area.

The reviewers broadly agreed on the strengths of the paper. The methodology is novel and well-motivated, addressing a critical limitation in existing knowledge editing techniques. The presentation is clear, with strong empirical validation through extensive ablation studies, benchmarks, and experiments across multiple LLM architectures and scales. The paper also demonstrates significant efficiency improvements by reducing the number of modified parameters, making the proposed approach computationally feasible and memory efficient. Additionally, the authors responded thoughtfully to reviewer feedback, addressing concerns and incorporating suggestions such as adding pseudocode, clarifying methodology, and improving visual clarity in figures and tables.

However, the paper is not without its weaknesses. Some reviewers questioned the novelty of the approach, as it builds upon previous methods in the field of knowledge editing and neuron localization. While FiNE introduces significant enhancements, it is an incremental improvement rather than a fundamental departure from existing methods. Furthermore, the paper heavily emphasizes FFN layers as the primary storage for factual knowledge but does not fully explore the roles of other components, such as self-attention layers. This limitation raises concerns about the broader applicability of the method to alternative architectures or more complex editing scenarios. Additionally, one reviewer noted the increased complexity introduced by neuron-level localization, which, while efficient, might not be as scalable as simpler methods like ROME in time-critical applications.

Overall, the decision to accept this paper is based on its clear contributions to the field of LLM knowledge editing. FiNE offers substantial improvements in editing locality and precision, demonstrating its practical utility in preserving model integrity while updating specific knowledge. The extensive experimental results, robust methodology, and responsiveness during the rebuttal period further strengthen the case for acceptance. While the paper could benefit from more detailed exploration of broader architectures and editing scenarios, its current contributions are significant enough to warrant publication and will likely inspire further research in this area.