Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a semi-parametric approach which trains neural network to recognize in-scope and out-scope data, and generating prompts that enables editing the response of the base model. The result is on par with state-of-the-art in most cases, while the training time and inference memory seem better.
strength_and_weaknesses: The proposed method is intuitive and utilizes the power of neural networks to generate good prompts. However, the performance is often not better than SERAC, as shown in Table 2. The success of the end-to-end system depends on the correct prediction from the edit classifier module, as well as the effectiveness of the prompt output decoder, which seems quite challenging to me. I expect that to be effective, the SEPROG modules require additional training every time when we want to edit a new set of facts, which makes it less flexible. 
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well-written and easy to follow. The method lacks novelty, as it simply uses neural networks to predict in-scope and out-scope data and generating the prompts, and I feel difficult to draw insights from the results. 
summary_of_the_review: I feel model editing is an interesting problem to study, but the proposed method lack novelty, and the result did not show significant improvements over existing approaches. I look forward to the improvements in the next version.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper provides a method for model editing, how to locally update the output of the model to exhibit desirable properties.
It builds on SERAC, which consists of three parts:  cache edits, an edit scope classifier, and a counterfactual model that overrides the base language model. A scope classifier determines when the output of the base language model will be in scope, and a counterfactual model determines what should be edited. 
This paper argues that having separate counterfactual and base models is inefficient, since we want to share knowledge between them. 
To address this problem, this paper proposes to use the prompted base model as the “counterfactual model”. 
The main claim of the paper is that doing so will achieve better generalization on hard edits.
Unfortunately, the experiments do not support the claim. 
strength_and_weaknesses: Strength
- Model editing is an important and high-impact problem, which is the focus of this paper. 

- The use of prompted language models as counterfactual models is an interesting research direction.  

Weakness


- This is a minor extension or a special case of SERAC and there’s no overwhelming evidence that this is better than the original model. (See table 2 & 3)


- The writing can be improved. The paper assumes the reader has detailed knowledge of SERAC and the entire literature on model editing. In order to understand what this paper is about, I have read the SERAC paper.
In order to reach a broader audience, the authors should motivate the model editing problem in the introduction and then clearly explain SERAC and its limitations in the second part.


Some other specific examples related clarity.

- The sentence“ Good parts of the above strategies while minimizing its limitations” is vague: what  are the good parts, and what are the limitations? How is this paper addressing the limitations?

- How is this a “semi-parametric’ model? This was never defined?

- Figure 2 and 3 are near impossible to read

- Table 3 might be interpreted too generously. I would not say that SEPROG is twice as fast as SERAC looking at it. The table also only reports the results for batch size 64, and not the others. 
clarity,_quality,_novelty_and_reproducibility: see above
summary_of_the_review: This paper presents an extension to SERAC, a model editing algorithm.  Unfortunately, the extension is neither theoretically justified nor empirically significant.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper aims to solve the model editing problem via training prefix prompts and using them together with a frozen language model for inference. The proposed approach, SEPROG, is less computationally heady and requires less memory than gradient-based approaches. SEPROG outperforms state-of-the-art methods by 20% on entailed edit benchmarks and provides up to 30% better performance over gradient-based methods on non-entailed benchmarks.
strength_and_weaknesses: ### Strength
- The approach is simple and easy to follow. The method visualization is clear. 
- I like that the authors did a thorough analysis to understand the training and inference efficiency of the baselines and the proposed approach. This is extremely useful for understanding the practicability of the approaches.

### Weaknesses
- The writing is not very clear, and I have a bunch of questions throughout reading the paper.
    - The concept of entailed predictions is explained in the experiment section, but the concept starts to be used in the introduction. It causes a bit of misunderstanding. 
    - The authors should provide better justifications for the selection of the base models. Is it because different tasks require different base encoders? Or simply for a fair comparison to previous works? And why can we not simply use one structure for the framework? The necessity of using different kinds of encoders for different tasks hurts the generalization ability of the approach. 
    - Why is performance so sensitive to batch size? Does one batch always contain examples of one edit description or could be any in-scope and out-of-scope examples? Why are the proposed approach and baselines behave differently for Copy-edit datasets and Entail-edit datasets in terms of a change of batch size?
    - Table 2 caption is confusing. Is the average performance averaged over both datasets and batch size, or averaged over datasets on one batch size?
- The results in Table 2 and Table 3 show that the approach is comparable to SERAC in terms of performance and training/inference cost but not significantly better. 

clarity,_quality,_novelty_and_reproducibility: - The paper could benefit a lot from more editing. Details are missing or confusing to readers (see weaknesses section). 
- The idea is novel and easy to follow, it should not be hard to reproduce the results from my experience. 
summary_of_the_review: The proposed idea to address the model editing problem is easy and simple to follow. The experimental results are not compelling enough compared to existing approaches.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose a prompt generation method for reflecting model edits. Prompt vectors are generated based on the edits and provided to the model at inference time to reflect the new knowledge.

Strengths:
Most reviewers agreed that the authors were solving an important problem,

Weaknesses:
Most reviewers agreed that the method represented an incremental improvement over the SERAC method, which also used a semi-parametric approach for model editing. Also, they did not find the evaluations convincing enough.

I’m inclined to disagree with the reviewers that SEPROG is not a sufficient stand-alone contribution. While prior work in parameter-efficient tuning (e.g., prefix-lm) exists, this work has rarely been applied in a model editing setting, yielding sufficient novelty. However, I agree with the reviewers that the empirical evaluation could be stronger. In its current form, it’s not clear what the advantage of SEPROG would be. I think this paper has strong potential, but could use another round of iteration taking into account the feedback from the reviewers before a future acceptance.

justification_for_why_not_higher_score: See weaknesses
Decision: Reject