reviews: 
Comments: This paper considers a problem of knowledge editing, which involves altering the parametric knowledge of LMs without retraining them from scratch. This work specifically focuses on the scalability of hypernetwork-based approaches, which are generally considered less effective for multiple concurrent edits. The authors claim that there are two major challenges: 1) the parameter shifts could be contradictory between the set of modified facts, and 2) accommodating a large number of edits in a hypernetwork is memory demanding. This work presents an approach that addresses these challenges. 

Concretely, this work extends MEND (Mitchell et al., 2022) by introducing additional parameter updates specifically for linear layers in the FFNs. Assuming that the linear layers are key-value memories, the motivation behind this is to find a better single parameter shift matrix S for _m_ updates. This additional step adjust the hypernetwork output (i.e., gradients) which is not a simple sum of gradients for different inputs. When scaling up to a large number of edits, backpropagating from the meta loss to the input is costly (e.g., computing pre and post-edit losses for each edit end to end). The proposed approach decomposes the optimization process by caching pre-edit computation (after finetuning), reducing the memory usage substantially.

The experimental setup focuses on scalability (i.e., editing thousands of facts at once), and the proposed approach is applied to different model families such as encoder-only (e.g., BERT) and  decoder-only (e.g., GPT-2 and GPT-J 6B). In addition to FT and MEND baselines, GPT-J with MEMIT is included as a baseline. For evaluation, FEVER is used for BERT, and zsRE is used for GPT models, largely following prior work. For evaluation metrics, edit success (ES – how often new facts get higher probability after editing), generalization success (GS – performance on related facts), and locality success (LS – performance of unrelated facts). In summary, the experimental results show that the proposed approach consistently outperforms FT and MEND with BERT and GPT-2, and it has better scalability compared to the original MEND. When it comes with GPT-J, which is a much larger LM, it is always better than MEND but underperforms MEMIT and FT on LS, indicating that the post-edit model forgets unrelated facts.
Strengths: - This work is tackling a well-motivated problem, scaling up knowledge editing approaches.
- The motivation behind the proposed approach (adjusting FFN weights, decomposing the optimization process) is clearly explained, and the solutions presented are reasonable.
Weaknesses: - The scope of the problem (scalability of MEND) could be narrow, and the proposed approach is only applicable for a specific knowledge editing approach.
- Based on the experimental results, it is difficult to assert that this approach is significantly better than all other knowledge editing approaches in terms of scalability (not only MEND).
- The poor LS score with GPT-J (6B) shows that this approach still edits unrelated facts.
- Qualitative analysis is not provided. It’s hard to see when/why this approach is beneficial without seeing error cases.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 2 fair