Kati8/ICLR_2023_2025 huggingface dataset:
authors: Chenmien Tan
authors: Ge Zhang
authors: Jie Fu
reviews: 
Comments: This paper proposes MALMEN, a massive editing method for LLM, which employs the least square method to aggregate parameter shifts inspired from MEMIT, and then applies the parameter updating method by taking the least squared solution as increment of the parameter metric, for minimizing the meta loss. To efficiently design the back propagation for massive editing, the paper separates the backprop on LM and hyper-network such that the back props are proceeded in a cascaded way, maintaining a set of cache values. Experiment results on FEVER and zsRE dataset show that the proposed MALMEN improves MEND on BERT-based and GPT-2, and often improves MEMIT on GPT-J, under some types of edits.
Strengths: - The proposed combination of the least square method and the loss-based updating for massive editing is quite interesting and novel. 
- The truncated backprop algorithm is solidly designed to improve the efficiency, which is also quite interesting. 
- The experiment results show that the proposed method improves MEND or MEMIT under various settings.,
Weaknesses: - Instead of the least squared solution, the simple sum-based aggregation is not compared. To prove the effect of the proposed method, this simplified aggregation needs to be compared.
- The description of Section 4.2 is largely dense, too hard to capture the details. In particular, Figure 2 provides the overall backprop flow, but why the training algorithm using the truncated backprop is not explicitly and clearly provided? 
- In GPT-J (6B), the proposed method doesn’t improve MEMIT, in terms of LS metric. This result needs to be properly discussed.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This paper proposes an improvement to MEND for large-scale fact editing. Similar to MEND, MALMEN uses a hypernetwork that takes in gradients (with respect to some input/output tuples) and hidden states and outputs a parameter update. The general training objective is similar to MEND, and the primary improvement proposed is a better method for combining multiple "fact" updates as opposed to naively summing/accumulating over single updates. They evaluate on standard memory editing tasks (based on FEVER), on BERT-base, GPT-2 and GPT-J
Strengths: - The paper provides plenty of technical details, and is fairly clear (though somewhat dense)
- The method is straightforward and intuitive. I am unclear about the broader applicability of memory editing, but the technical details and performance are sufficiently convincing to me that this is a meaningful contribution.
Weaknesses: - The paper requires quite a bit of background on MEND. This is not inherently a bad thing since the paper is basically a direct modification of MEND, and the paper already spends a good deal of space building the background, but I think providing higher-level intuition in the exposition could help.
- Section 4.2 wasn't very clear to me (in particular "truncating the back-propagation at the end of linear layers"). Figure 2 was significantly clearer, and I wonder if the authors could revisit the section and tweak it for ease of understanding the somewhat complicated procedure for training.
- The results on scaling to GPT-J seem a little unstable
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent