Title: Rethinking Residual Distribution in Locate-then-Edit Model Editing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Model Editing; Large Language Models; Knowledge Editing; Locate-then-edit
Abstract: Model editing enables targeted updates to the knowledge of large language models (LLMs) with minimal retraining. Among existing approaches, locate-then-edit methods constitute a prominent paradigm: they first identify critical layers, then compute residuals at the final critical layer based on the target edit, and finally apply least-squares-based multi-layer updates via $\textbf{residual distribution}$. While empirically effective, we identify a counterintuitive failure mode: residual distribution, a core mechanism in these methods, introduces weight shift errors that undermine editing precision. Through theoretical and empirical analysis, we show that such errors increase with the distribution distance, batch size, and edit sequence length, ultimately leading to inaccurate or suboptimal edits. To address this, we propose the $\textbf{B}$oundary $\textbf{L}$ayer $\textbf{U}$pdat$\textbf{E (BLUE)}$ strategy to enhance locate-then-edit methods. Sequential batch editing experiments on three LLMs and two datasets demonstrate that BLUE not only delivers an average performance improvement of 35.59\%, significantly advancing the state of the art in model editing, but also enhances the preservation of LLMs' general capabilities. Our code is available at https://github.com/xpq-tech/BLUE.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a critical issue in locate-then-edit model editing methods for Large Language Models (LLMs): the negative impact of residual distribution on editing precision. Through both empirical and theoretical analyses, the authors demonstrate that residual distribution introduces weight shift errors that worsen with increased distribution distance, batch size, and edit sequence length, leading to suboptimal edits. To address this, they propose the Boundary Layer UpdatE (BLUE) strategy, which updates only the first and last critical layers by directly computing residuals, eschewing residual distribution. Experiments on three LLMs and two datasets show that BLUE significantly improves editing performance, better preserves LLM general capabilities, and mitigates hidden state shifts.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n-  The paper effectively identifies and analyzes a counterintuitive failure mode in prominent locate-then-edit methods: residual distribution. The paper shows that evenly distributing residuals is suboptimal, leading to significant degradation in efficacy and generalization.\n- A wide array of ablations and scenarios—including per-layer versus distributed residual comparison, sequential and standard batch editing, long-context editing, and general capability retention—validates BLUE’s superiority.\n\nWeakness:\n- The BLUE method proposed in this paper lacks novelty. As shown in Figure 1, the approach merely reallocates the residual updates from all layers to only the first and last layers. This strategy is overly simplistic and lacks significant innovation, offering limited insight or inspiration.\n- The paper is overly lengthy and difficult to follow. In Section 4, the proofs regarding the upper and lower bounds of the interpolation error suggest that the upper bound of weight shift is related to the number of editing batches, the number of edits, and residual distribution. However, this does not demonstrate a strong correlation between the weight shift itself and these factors—let alone establish a strong connection to the proposed BLUE method.\n- Many of the mathematical derivations and formulas in the paper are unclear, making it difficult to understand their meaning. For example, in Equation (2) in Section 3, the symbol n appears, but it is not clear what n represents. The surrounding text does not provide an explanation either. Additionally, the explanations of K₁ and M₁ are also confusing and hard to comprehend.\n\nQUESTIONS:\nnone\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper critically analyzes the residual distribution mechanism in the localize-then-edit editing methods for large language models (LLMs). The authors point out a key limitation - the residual distribution introduces weight-shift error, which degrades editing accuracy. To address this issue, they propose the BLUE strategy, which updates the first and last key layers using only directly computed residuals. Extensive experiments on multiple LLMs and datasets show that BLUE significantly improves editing performance while preserving the general capabilities of the model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThis paper reveals a intuitive and important problem in existing localize-then-edit methods - the residual distribution is not as effective as previously assumed. The authors provide theoretical upper bounds and empirical validation for the weight-shift error caused by the residual distribution. The analysis is rigorous and informative.\n\n2.\tThe proposed BLUE strategy is simple, efficient, and widely applicable. It improves performance without significantly increasing computational cost and is easily integrated with multiple existing methods (MEMIT, PRUNE, RECT, AlphaEdit).\n\n3.\tIt shows that BLUE consistently improves editing performance across different metrics and settings, including sequential batch editing, general ability preservation, and long-form editing.\n\nWeaknesses\n\n1.\tWhile BLUE generally improves performance, in some cases (e.g., AlphaEdit on GPT-J), the improvement is small. A more detailed analysis of when and why BLUE may fail would help strengthen the robustness of this paper.\n\n2.\tThis paper focus on models with parameter sizes up to 8B or 6B. It would be valuable to explore whether these findings generalize to larger state-of-the-art models.\n\n3.\tThis paper focuses on least-squares-based editing methods. It is unclear how well BLUE generalizes to more complex or nonlinear optimization strategies, which may limit its applicability.\n\n4. This paper modifies two layer of models, but method like ROME only update one layer, why not only update one layer? This maybe the minimal residual distribution.\n\n5. This article has a bit of background content and may be simplified a bit.\n\nQUESTIONS:\nThis paper modifies two layer of models, but method like ROME only update one layer, why not only update one layer? This maybe the minimal residual distribution.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies model editing for large language models (LLMs), focusing on the prevalent locate-then-edit paradigm. The authors identify that the residual distribution mechanism used in existing methods introduces weight shift errors, which worsen with increased distribution distance, batch size, and edit sequence length, reducing editing accuracy. To address this, they propose the Boundary Layer UpdatE (BLUE) strategy to mitigate these errors. Experiments on three LLMs and two datasets demonstrate that BLUE improves editing performance by an average of 35.59% while better preserving the general capabilities of LLMs, significantly advancing the state of the art in model editing.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is well-written and easy to follow.\n2. The proposed method is well-motivated.\n3. The experiment is thorough and convincing.\n\nWeakness:\n1. The authors do not explicitly discuss the computational overhead or efficiency trade-offs introduced by BLUE compared to standard locate-then-edit methods, which may be important for practical deployment.\n2. While the paper mentions improvements in long-form editing scenarios, more detailed analysis or examples of such use cases would strengthen understanding of BLUE’s benefits in real-world editing tasks.\n\nQUESTIONS:\n1. The BLUE method updates only the first and last critical layers without residual distribution. How sensitive is the method’s performance to the selection of these critical layers? Would different choices affect editing accuracy or model preservation?\n2. The experiments are conducted on three LLMs and two datasets. Could the authors clarify whether BLUE generalizes well to larger-scale models (e.g., GPT-3 scale) or other architectures beyond those tested?\n3. Does the BLUE strategy introduce additional computational or memory overhead compared to existing locate-then-edit methods? If so, how significant is it in practical scenarios?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a critical analysis of the 'locate-then-edit' paradigm in model editing, specifically targeting the widely-used residual distribution mechanism. The authors compellingly argue, through both theoretical and empirical evidence, that this distribution method is a source of significant error that harms editing precision. To address this, they propose a simple yet effective strategy, Boundary Layer UpdatE (BLUE), which modifies only the first and last critical layers with directly computed residuals, bypassing the distribution step entirely. The authors validate their approach through extensive experiments on multiple LLMs and datasets, showing that BLUE not only improves editing performance but also better preserves the model's general capabilities and is more computationally efficient.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n- The author conducts sufficient empirical analysis and theoretical analysis about the shortcomings of residual distribution. The empirical analysis in Section 4.1, particularly the experiments showing the diminishing contribution of distributed residuals (Figure 2) and their sub-optimality compared to directly computed ones (Figure 4), is clear and convincing. This empirical work is well-supported by a theoretical analysis (Theorem 4.1 and Lemma 4.3) that provides an upper bound on the weight shift error, formalizing the intuition that errors increase with distribution distance, batch size, and the number of sequential edits.\n- The authors apply BLUE to multiple strong baselines (MEMIT, AlphaEdit, PRUNE, RECT) across several LLMs (Llama3, GPT-J, GPT2-XL) and standard datasets (CounterFact, zsRE). The results in Table 2 strongly support the primary claims of improved editing performance in sequential batch scenarios. Furthermore, the analysis of general capability retention on downstream GLUE tasks (Figure 6) and the mitigation of hidden state shifts (Figure 7) are particularly compelling, as they address critical aspects of model editing beyond simple efficacy and demonstrate a clear advantage for BLUE in preserving the integrity of the original model.\n\n\n### Weakness\n- While the justification for updating the first critical layer is well-motivated (as it is most affected by distribution error), the motivation for choosing the last critical layer is stated simply as \"follow[ing] prior work\" (Section 4.3, page 7). The ablation study in Appendix H (Figure 11) provides strong empirical validation for this choice, demonstrating its optimality. However, the main text would benefit from a more explicit theoretical or intuitive justification for why the first and last layers form the optimal pair, rather than, for instance, the first two critical layers. \n- I think the proof section is a bit verbose in the paper, and I recommend the author move some of them to the appendix and show more experiements in the main text. To enhance the paper's narrative impact, I would recommend condensing this section slightly. The reclaimed space could be used to feature a crucial experimental result that is currently in the appendix. For example, the layer-selection ablation study (Appendix H, Figure 11) provides the core empirical evidence for BLUE's design. Elevating this experiment to the main paper would proactively answer a key question about the \"Boundary Layer\" choice and make the justification for the method even more compelling.\n- The authors correctly identify that the method's performance on multi-hop reasoning questions is a limitation and an area for future work. This points to a more significant consideration regarding the scope of the paper's claims. The current evaluation focuses on a standard but specific type of generalization: applying a factual edit to paraphrased prompts. While the results are strong in this domain, the paper would be more impactful if it contextualized its findings more explicitly in the main text.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nThe author acknowledges the limitations in Appendix L and shows that the methods didn't evaluate the results in multi-hop reasoning benchmarks like MQuAKE.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}