Title: MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Lifelong model editing

Catastrophic forgetting

Large language model

LLM
Abstract: Language models deployed in real-world systems often require post-hoc updates to incorporate new or corrected knowledge. However, editing such models efficiently and reliably—without retraining or forgetting previous information—remains a major challenge. Existing methods for lifelong model editing either compromise generalization, interfere with past edits, or fail to scale to long editing sequences. We propose MEMOIR, a novel scalable framework that injects knowledge through a residual memory, i.e., a dedicated parameter module, while preserving the core capabilities of the pre-trained model. By sparsifying input activations through data-dependent masks, MEMOIR confines each edit to a distinct subset of the memory parameters, minimizing interference among edits. At inference, it identifies relevant edits by comparing the sparse activation patterns of new queries to those stored during editing. This enables generalization to rephrased queries by activating only the relevant knowledge while suppressing unnecessary memory activation for unrelated prompts. Experiments on question answering, hallucination correction, and out-of-distribution generalization benchmarks  across LLaMA-3 and Mistral demonstrate that MEMOIR achieves state-of-the-art performance across reliability, generalization, and locality metrics, scaling to thousands of sequential edits with minimal forgetting.


Peer reviews:
comments: SUMMARY:
This paper introduces MEMOIR (Model Editing with Minimal Overwrite and Information Retention), a lifelong model editing method that affords a good trade-off between reliability, generalization, and locality. The edits are performed through a separate fully-connected
layer whose output is later added to that of the original model layer. These edits are restricted to subsets of parameter selected by an input-dependent sparse mask. This allows MEMOIR to mitigate catastrophic forgetting. At inference time, they use the sparse mask to find the closest edited prompt to the input. The model's predictions on non-edited prompts are preserved. They evaluate with 4 models and compare with a wide range of baselines.

STRENGTHS AND WEAKNESSES:
Strengths:
- The method offers a good balance between the main metrics of interest—reliability, generalization, and locality—without introducing additional severe computational overload. The memory module is non-invasive and does not lead to catastrophic forgetting. 
- MEMOIR is compared to an extensive baseline. 
- The paper is well-structured, easy to read. 

Weaknesses:
- An important evaluation of model editing methods was not mentioned in the paper. Updating some facts can affect other facts in the model that are not necessarily captured by semantic similarity. There are already existing benchmarks such as this one https://arxiv.org/abs/2307.12976 to analyze this phenomenon. 

Minor comments:
There are 2 'the' on line 316.

QUESTIONS:
1 - How does MEMOIR compare to prior methods on benchmarks (for example https://arxiv.org/abs/2307.12976) that measures the impact of fact updates on other facts? 

2 - Do you expect your approach to generalize to bigger models?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MEMOIR, a framework for lifelong model editing in large language models (LLMs), designed to inject new knowledge while minimizing interference with existing knowledge. MEMOIR introduces a residual memory module with sparse, data-dependent masking, allowing each edit to target a distinct subset of parameters. During inference, MEMOIR activates memory selectively using a similarity-based routing mechanism, ensuring generalization to rephrased queries and suppression of irrelevant memory usage. The method is evaluated across tasks such as question answering, hallucination correction, and OOD generalization, showing good trade-offs between reliability, generalization, and locality, scaling effectively to thousands of sequential edits. The framework outperforms previous parametric and non-parametric methods including MEMIT, AlphaEdit, and WISE.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\nThe paper is methodologically solid. The empirical evaluation is extensive, covering multiple models, tasks, and a wide range of edit scales. The performance gains are clearly demonstrated with ablation studies.\n\nThe paper is clearly written, with intuitive figures that help illustrate the core mechanisms. The explanation of the masking strategy (TopHash) and inference-time routing is particularly clear and technically precise.\n\nThe combination of sparse memory allocation and conditional activation based on TopHash is novel in the context of LLM editing. While sparsity and activation gating are known techniques, their integration into this framework and use for routing edits in a memory-efficient and interference-minimizing way is well-motivated.\n\n**Weakness**\n\nMEMOIR only modifies a single transformer layer’s projection weights. While this suffices for many factual edits, it may limit applicability to complex or multi-hop knowledge edits requiring broader changes across multiple layers.\n\nThe effectiveness of TopHash-based routing depends on the number of active indices and the similarity threshold. Though the paper provides ablations, further justification or principled selection strategies would improve robustness and usability.\n\nQUESTIONS:\nHave you experimented with extending MEMOIR to multiple transformer layers, or hierarchically allocating memory? Would it improve generalization or allow for more complex edits?\n\nHow sensitive is performance to the choice of routing threshold? Could it be learned during editing, rather than heuristically set?\n\nIn practice, if the number of edits grows indefinitely, the residual memory may become large. Have you considered any memory pruning or compression mechanisms?\n\nLIMITATIONS:\nThe authors appropriately note limitations in scope (single-layer editing, decoder-only models) and acknowledge future directions. The societal impact of memory-based editing (e.g., misinformation correction vs. malicious manipulation) is not deeply discussed but could be strengthened with a short reflection.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces MEMOIR, a scalable framework for lifelong model editing that updates large language models (LLMs) with new knowledge while minimizing forgetting of previous edits. MEMOIR uses a residual memory module with sparse, data-dependent activations to isolate edits, combined with a conditional inference mechanism that activates only relevant memory for semantically similar prompts. Through experiments on tasks like question answering, hallucination correction, and out-of-distribution generalization using models such as LLaMA-3 and Mistral, MEMOIR shows superior performance in reliability, generalization, and locality, scaling up to 7000 edits with minimal performance degradation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nMEMOIR addresses a fundamental challenge in LLMs—efficient post-hoc knowledge editing—with a well-motivated, technically sound approach. By sparsifying activation via TopHash and storing updates in a dedicated residual memory, the framework avoids catastrophic forgetting while supporting generalization to paraphrased queries. Its conditional knowledge activation at inference is novel, precise, and efficient, outperforming strong baselines (e.g., WISE, AlphaEdit) across multiple metrics and LLM architectures. The scalability to 7000 sequential edits with stable performance demonstrates practical robustness.\n\nThe paper is thorough in empirical evaluation, comparing MEMOIR to a broad set of baselines across diverse tasks. It supports claims with detailed ablations (e.g., removing conditional activation or varying sparsity levels), showing clear advantages of the proposed design. The clarity of writing, theoretical intuitions, and visualizations (e.g., mask overlap distributions and reliability curves) further enhance the paper’s readability and impact.\n\nWeaknesses:\nDespite its strengths, MEMOIR is still limited by its scope. It edits only a single FFN projection layer, which may restrict its ability to handle deeply integrated or abstract knowledge. This could be problematic for tasks involving multi-hop reasoning or systemic model behaviors. Additionally, the TopHash-based sparsity strategy, while effective, introduces a fixed permutation mechanism that may lack adaptiveness, and its dependence on mask similarity thresholds might not generalize robustly across all tasks or model types.\n\nQUESTIONS:\nThis paper choose the threshold 𝜏 for determining whether a prompt is relevant to stored edits is fixed.\nIt might worthwhile to consider Introducing a learnable or context-sensitive thresholding mechanism (e.g., trained via meta-learning or reinforcement learning) to better distinguish semantically similar prompts from unrelated ones.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper authors introduce MEMOIR in which they introduces a memory module, i.e., a dedicated fully-connected layer in a single transformer block where all edits are performed. MEMOIR mitigates catastrophic forgetting by allocating distinct parameter subsets to each edit and retrieving them during inference to activate only the relevant knowledge for a given prompt. They propose a structured sparsification of the input to this layer, dynamically activating only a sample-specific subset of parameters in the introduced memory module in the forward pass. Experiments on question answering, hallucination correction, and out-of-distribution generalization benchmarks across LLaMA-3 and Mistral demonstrate that MEMOIR achieves state-of-the-art performance across reliability, generalization, and locality metrics, scaling to thousands of sequential edits with minimal forgetting.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is overall well written, well motivated with good experimentation coverage. \n\n2. The method's ability to dynamically activate or deactivate relevant knowledge based on prompt type (edited, rephrased, or irrelevant) effectively eliminates the need for large corpora of irrelevant samples during editing, which is often required by prior approaches to enhance locality. This is a strong differentiation with practical applications in real life.\n\n3. The framework demonstrates strong scalability, extending the state-of-the-art edit horizon to 7000 edits while delivering superior performance in challenging sequential singular edit settings.\n\nWeaknesses:\n\n1. The performance of MEMOIR is sensitive to the number of active indices k used in TopHash. It appears crucial for balancing the model's ability to capture edits, prevent excessive overwriting, and maintain the quality of semantic relevance for routing. Although authors discuss it during the ablation, hyperparameter tuning aspect of this parameter k might be challenging in practice.\n\n2. While the author claim that MEMOIR is scalable, the computational efficiency and memory footprint for extremely large LLMs (beyond 8B parameters) or significantly more edits than 7000 are not explicitly detailed. However, the method's design with a residual memory and sparse updates suggests better scaling than full fine-tuning.\n\n3. The authors acknowledge that the paper does not present a formal theoretical results or proofs, focusing instead on empirical validation. While empirical results are strong, a theoretical foundation could further solidify the understanding of why MEMOIR performs well.\n\nQUESTIONS:\nQuestions:\n\n1. The paper focuses on factual knowledge editing. An interesting extension to the studied question is how would MEMOIR perform on other types of model updates, such as stylistic changes, or ethical alignment which are more subtle than facts and not necessarily true/false.\n\n2. How does the fixed random permutation in TopHash affect the computational overhead during editing and inference, and is there a trade-off between the randomness introduced and the speed of the operation?\n\nLIMITATIONS:\nPlease refer to Strengths and Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-15"}