Title: Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Lifelong model editing

fine-grained editing

knowledge editing

large language models
Abstract: Lifelong knowledge editing enables continuous, precise updates to outdated knowledge in large language models (LLMs) without computationally expensive full retraining. However, existing methods often accumulate errors throughout the editing process, causing a gradual decline in both editing accuracy and generalization.  To tackle this problem, we propose Neuron-Specific Masked Knowledge Editing (NMKE), a novel fine-grained editing framework that combines neuron-level attribution with dynamic sparse masking. 
Leveraging neuron functional attribution, we identify two key types of knowledge neurons, with knowledge-general neurons activating consistently across prompts and knowledge-specific neurons activating to specific prompts.
NMKE further introduces an entropy-guided dynamic sparse mask, locating relevant neurons to the target knowledge. This strategy enables precise neuron-level knowledge editing with fewer parameter modifications.
Experimental results from thousands of sequential edits demonstrate that NMKE outperforms existing methods in maintaining high editing success rates and preserving model general capabilities in lifelong editing.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Neuron-Specific Masked Knowledge Editing (NMKE), a fine-grained editing framework that enables effective and robust lifelong knowledge editing for large language models (LLMs). By performing neuron-level attribution and dynamically masking a sparse subset of relevant neurons, NMKE updates model knowledge with minimal parameter modification, thereby mitigating error accumulation over thousands of edits. The authors demonstrate that NMKE significantly outperforms strong baselines (e.g., AlphaEdit) on sequential editing benchmarks, while preserving general capabilities across downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This paper clearly articulates the challenges of lifelong model editing, especially the loss of generalization, which are underexplored in prior work.\n\n- The method achieves state-of-the-art performance under sequential editing settings (batch size 1 and up to 5,000 edits), with detailed evaluations on both editing success and general task performance.\n\n- The paper includes great visualizations and ablation studies to support the effectiveness of the neuron selection mechanism.\n\nWeaknesses:\n- Editing granularity setup: The current lifelong setting uses a batch size of 1 for sequential edits, which may not reflect practical scenarios where models need to absorb multiple factual updates at once. It is recommended to test the method under larger batch sizes and compare with baselines accordingly. Otherwise, the setting should be more precisely described as “Sequential Editing” rather than “Lifelong Editing” [https://aclanthology.org/2024.emnlp-main.149.pdf].\n\n- Efficiency and runtime analysis missing: NMKE requires computing attribution across all layers, which may introduce significant computational overhead. A runtime comparison with baselines or a profiling study would strengthen the claim of robustness and deployment readiness.\n\nQUESTIONS:\nThis is a well-written and well-structured paper. The idea of combining neuron-level attribution with dynamic masking is not entirely novel, as prior work has explored neuron specialization and sparse editing mechanisms. However, the adaptation of this strategy to lifelong editing is impactful, which raise a useful mechanism to preserve general capabilities.\n\nTo further improve the work, I suggest addressing the following:\n- Please consider evaluating NMKE under batched edit scenarios (batch size > 1). This would better reflect practical applications and further highlight the scalability of your method.\n- Since your method involves computing attribution for a large number of neurons, it would be useful to report runtime or efficiency metrics. Even a high-level cost comparison with AlphaEdit or WISE would be valuable.\n\nThat said, the experimental design is rigorous, and the improvements over existing methods are impressive. If the above points are addressed or discussed, I would be inclined to increase my score.\n\nLIMITATIONS:\nYes.\n\nThe authors provide a reasonable and well-articulated discussion of the limitations from the perspective of interpretability, particularly regarding the binary classification of neurons and the focus on autoregressive LLMs. \nIn addition, I suggest the authors consider incorporating insights from recent work that examines cross-layer interactions in LLMs, such as Anthropic’s Cross-Layer Transcoders (CLT). These works point to the growing importance of understanding information flow not just at the neuron level but across layers and pathways. Integrating such perspectives may help refine NMKE’s attribution and masking mechanisms, especially under long-term editing scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Neuron-Specific Masked Knowledge Editing (NMKE), a novel framework for editing knowledge in Large Language Models designed to address the challenge of performance degradation in lifelong learning scenarios. Existing methods often suffer from error accumulation over sequential edits, leading to a decline in both editing accuracy and the model's general capabilities. NMKE proposes a more precise, fine-grained approach by operating at the neuron level rather than the traditional layer level. Through extensive experiments involving up to 5,000 sequential edits on models like LLaMA3-8B-Instruct and GPT2-XL, the authors demonstrate that NMKE significantly outperforms existing state-of-the-art methods. It successfully maintains high editing accuracy and, crucially, preserves the model's general reasoning, mathematical, and coding abilities, where other methods exhibit catastrophic forgetting.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThrough extensive experiments involving up to 5,000 sequential edits on models like LLaMA3-8B-Instruct and GPT2-XL, the authors demonstrate that NMKE significantly outperforms existing state-of-the-art methods. It successfully maintains high editing accuracy and, crucially, preserves the model's general reasoning, mathematical, and coding abilities, where other methods exhibit catastrophic forgetting.\n\n**Weaknesses:**\n\n1. The binary classification of neurons into \"knowledge-general\" and \"knowledge-specific\" is an oversimplification of what is likely a continuous spectrum of neuron functionality. This rigid model may misclassify neurons with mixed or nuanced roles, as the scoring for each category treats them as distinct types.\n\n2. The method requires a neuron attribution step for every editing batch to generate the dynamic mask. The paper fails to analyze the computational overhead and scalability of this \"locate\" phase, which could be a significant bottleneck and render the framework impractical for high-throughput, real-world applications.\n\n3. The dynamic masking ratio, a key component, relies on scaler (a) and bias (b) hyperparameters that are shown to have a substantial impact on performance. The paper does not provide a clear methodology for how these crucial hyperparameters were chosen or discuss their generalizability, raising concerns about the practical effort required to deploy the method on new models or datasets.\n\n4. The experiments are primarily conducted on two LLMs (LLaMA3-8B and GPT2-XL). Widly used LLMs like Qwen are not tested.\n\nQUESTIONS:\n1. Your t-SNE visualizations (Figure 5) provide strong qualitative evidence of reduced distributional shift. Have you considered quantifying this shift using a statistical measure on the parameter weight distributions? A quantitative metric could offer a more objective complement to the visual analysis of parameter stability.\n\n\n2. The dynamic selection ratios (ρge​ and ρsp​) are determined using entropy, but also rely on constant scaler and bias hyperparameters (age​, bge​, etc.). Your analysis in Figure 7 shows the effect of these terms. How were these hyperparameters initially chosen, and how sensitive is the model's performance to them? Do they need to be re-tuned for different model architectures or sizes, which could impact the method's practical applicability?\n\n\n3. The distinction between knowledge-general and knowledge-specific neurons is central to your method. Figure 11 analyzes the distribution and overlap of these neurons across layers. Could you elaborate on the functional role of the neurons that fall into the \"overlap\" category? How does editing this specific subset of neurons that are both general and specific impact the trade-off between editing success and locality preservation?\n\n\n4. The current approach uses a hard binary mask for updates.  Given that neuron importance is a continuous score, have you experimented with a \"soft\" masking approach? For instance, one where the magnitude of the update applied to a neuron (e.g., its learning rate) is scaled by its attribution score, potentially allowing for an even more nuanced and less disruptive edit.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Neuron-specific Masked Knowledge Editing (NMKE) framework, which identifies Knowledge-General Neurons and Knowledge-Specific Neurons through neuron-level attribution analysis. By combining a dynamic sparse masking mechanism, NMKE locates neuron subsets relevant to target knowledge and performs localized parameter updates on these key neurons. This approach enables high-precision knowledge correction in lifelong editing scenarios while avoiding the degradation of model generalization caused by coarse-grained updates. Experiments verify that NMKE maintains high editing success rates and generalization capabilities across thousands of sequential edits, outperforming existing methods like ROME and AlphaEdit, thus providing a novel solution for efficient lifelong knowledge editing in large language models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:  \n1. The NMKE framework is proposed, which identifies Knowledge-General Neurons and Knowledge-Specific Neurons through neuron-level attribution analysis, enabling high-precision knowledge correction in lifelong editing scenarios while avoiding the degradation of model generalization capabilities caused by coarse-grained updates.  \n\n2. It reveals that the root cause of performance degradation in lifelong editing is the cumulative interference of coarse-grained updates on neurons, and clarifies the core role of Knowledge-General Neurons in maintaining the model's basic capabilities.\n\nWeaknesses:  \n1. The authors discretely classify neurons into \"general/specific\" categories without considering the continuous spectrum of importance, which may overlook the complex functions of intermediate-state neurons.  \n\n2. The method proposed in the paper for locating neuron contributions is mainly based on existing approaches [1].  \n\n3. Hyperparameters such as the temperature coefficient α and scaling factors a_ge/a_sp in the entropy-guided mechanism require manual tuning and may need to be reconfigured for different tasks or models, increasing computational costs.\n\n4. There are a number of related works that have not been discussed and compared, e.g., [2,3,4].\n\n[1] Zeping Yu and Sophia Ananiadou. Neuron-level knowledge attribution in large language models. ArXiv preprint, abs/2312.12141, 2023.\n\n[2] Ni, Shiwen, et al. \"Forgetting before Learning: Utilizing Parametric Arithmetic for Knowledge Updating in Large Language Models.\" Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.\n\n[3] Pan, Haowen, et al. \"Precise Localization of Memories: A Fine-grained Neuron-level Knowledge Editing Technique for LLMs.\" arXiv preprint arXiv:2503.01090 (2025).\n\n[4] Bi, Baolong, et al. \"Adaptive Token Biaser: Knowledge Editing via Biasing Key Entities.\" Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
A method is presented to edit knowledge in an existing LLM using a dynamic sparse masking mechanism. The distinctive feature here is that the mechanism operates on individual neuron units, namely their observed additive contributions to the output. Metric quantifying knowledge specificity vs. generality are proposed and utilized to achieve efficient and non-destructive editing. The technique is compared to relevant SOTA baselines on two editing tasks along with several background sets to gauge general degradation.

STRENGTHS AND WEAKNESSES:
+ well-written paper with clear exposition
+ sensible technique
+ good baselines and thorough evaluation 
+ significant improvements

- some gaps in description and complexity analysis 
- illegible figures

QUESTIONS:
* can you comment on complexity of the method: which additional memory objects are instantiated, what is their size for the Llama model, quantify the computational overhead
* on the two editing tasks, which data partitions were used  to estimate I and the scores (4-7) and when? on l. 173 you mentioned the ratio \rho_{ge} is being "predicted" which confused me - I suspect to have missed something. What is the predictor here?
* Figure 2, 7 and probably 5: the legends are really illegible when printed out.

LIMITATIONS:
Limitations are not discussed explicitly in the main paper. One such limitation is not studying an editing impact on closely related (but different) facts (e.g., updating name of a CEO of a company and its impact on the total number of CEOs that company has had).
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-03