Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new method for *model editing* in pre-trained language models, which has received growing attention from the deep learning community in the past few years. The authors point out that the setting of sequential model editing (in which the model must be edited with a stream of edits, rather than a small/static set of edits) is relatively understudied, and propose GRACE, a method that is better-suited to applying many model edits without destroying model performance on the pre-training data or forgetting past edits. Under a their particular evaluation conditions, the authors find their method is better able than some existing methods to handle many sequential edits.
strength_and_weaknesses: Strengths:
- Focuses on an interesting extension of existing model editing work, which has mostly focused on single/batched model edits, rather than streams of edits
- The proposed method does not need a training set of edits, unlike several past model editors. This requirement is a non-trivial limitation of some past works.
- The proposed method is able to apply many edits without interference or forgetting of the pre-training data

Weaknesses:
- Baselines are limited and not explained in detail (particularly the "memory network" baseline). The setup for MEND is a bit strange, and it’s not too surprising that it fails. Why not compare with ROME (Meng et al., 2022), which I believe doesn’t require an editing training set?
- The criteria for successful edits are, in my opinion, insufficient, in that they do not consider the generalization of the edit. Previous work such as de Cao et al., 2021, Meng et al., 2022 and Mitchell et al., 2022a,b evaluate the generalization of an edit to related inputs in some manner, which is crucial in my opinion.
- On a related note, it’s not clear to me the extent to which the evaluations test the generalization of an edit; Section 4.2 says “For editing, we sample 1000 edits from zsRE, including random samples of 5 rephrasings for each question.” Does this mean the the F1 on previous edits is computed using the rephrasings of the original edit X? If we’re not evaluating generalization in some way, it’s not clear that we need to cache representations at all, vs just caching the raw inputs and labels? My understanding is that the advantage of existing learnable model editors like e.g. Mitchell 2022b is that the model can learn to reason over edits to produce significantly different outputs.
- The experiments are lacking in qualitative examples; it would be helpful to analyze some success and failure cases to see where the proposed method begins to fail (e.g., with respect to generalization).
- The method is not particularly technically novel (minor point)

I believe the paper would be improved by including additional model editing baselines, such as ROME (Meng et al., 2022), SLAG (Hase et al., 2022), or MEND using a proper edit training dataset (this would give MEND privileged data, but would be helpful to see if MEND is even able to apply sequential edits in the best possible scenario; maybe it still fails). In addition, it's critical that experimental evaluations clearly evaluate the generalization capabilities of the model editor, one way or another. It's not clear how the currently-reported metrics do so. Otherwise, a "successful" editor could be one that purely performs memorization (i.e. a simple dictionary/lookup table mapping from input to label).
clarity,_quality,_novelty_and_reproducibility: The writing was mostly clear, though some details were missing. In particular, the details of the datasets used during evaluations (how were edits sampled during editing and while evaluating whether previous edits are preserved, what do they look like, etc.) as well as some detail about baselines (like the memory network) made interpreting the experimental results a bit difficult. Also, the procedure for actually producing the "values" in Figure 2 isn't explained until the top of page 5, which made me think I'd missed something for a while (this is a relatively minor point, though).

Other points of confusion:

- *“However, large pools of training edits are rarely available before deployment, otherwise the edits could be used during pretraining.”* This statement doesn’t really make sense; trainable model editors train on a finite set of training edits to learn the ***general behavior of editing***, not to learn the specific information in those edits. Existing evaluations of model editors (e.g., Mitchell et al) evaluate the editor on edits that do not appear in the editor training set. So including that information in the pre-training set wouldn’t help; the specific training procedure of learning a model editor is necessary.
- With the given criteria for a successful edit, it seems like the optimal epsilon is just arbitrarily small. Table 1 supports this claim, and contradicts the statement in the paper: “As we increase ε, we begin to see that GRACE trades off upstream accuracy for online accuracy” From my reading, as we increase eps, both metrics get strictly worse; there’s no tradeoff.
- *“Additionally, prior model editors have yet to consider sequential”* This isn’t true; see [1] and possibly Mitchell et al., 2022b (in which batched edits and sequential are somewhat equivalent)

In terms of novelty, the editing technique is somewhat novel, though fairly closely related to the LU baseline in Mitchell et al., 2022b and the ROME method in Meng et al., 2022. The problem setting is not novel, as sequential editing is considered in [1].

[1] Hase et al., 2021. Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs.
summary_of_the_review: Overall, I think this work is getting at an important question for folks interested in model editors: how do we apply many edits to our model without destroying it. However, I feel the work as-is is lacking in experimental rigor (in terms of both baselines considered and the specific quantities measured in the experiments; i.e., generalization) as well as technical insight and novelty. Therefore, I would recommend rejecting the paper in its current form.

**Update** in light of author response:

I'm grateful for the authors' thorough response to my concerns; I think the paper is significantly improved. The additional generalization ablation is helpful in understanding how well GRACE generalizes to difference expressions of an edit. My reading of the result is that GRACE does some generalization to paraphrases of the edits, but still fails to generalize to a significant number of simple paraphrases of the edit content (as shown by the F1 score in column 3 of Figure 4 being significantly below 1). Increasing the deferral radius helps with this issue, but comes at the cost of significantly reduced upstream F1.

My remaining concerns about GRACE as a method are related to this generalization issue. GRACE seems to generalize only mostly to paraphrases of edits, and I would guess it is totally incapable of generalizing to entailed edits, such as:

Edit:

Where is the Eiffel Tower located? Rome

Test input:

What famous tower is located in Rome? <should answer Eiffel Tower>

It seems like editors like SERAC (Mitchell 2022) would be able to perform this type of generalization, while GRACE's explicit key-value construction makes it extremely unlikely that such generalization would occur, given that it already struggles somewhat with paraphrases.

As a related generalization issue, GRACE's reliance on exact equality of the edit label to determine if edits contain equivalent information is problematic. Consider the edits:

e1: What is the tallest mountain on earth? Everest
e2: What is the earth's tallest mountain? Mount Everest

Reducing the radius of e1 to make room for a separate edit e2 feels wrong to me. How do the authors propose handling the problem of determining the semantic equivalents of edit labels (or edits, in general)?

In conclusion, I'm now actually okay with accepting the paper in that it is essentially a novel combination of memory-based (e.g. SERAC, Mitchell 2022) and parametric (e.g. ROME, Meng 2022 or MEND, Mitchell 2022) editing ideas that shows some promise. However, I am also okay with rejecting the paper this time around, on the grounds that GRACE may not be fully leveraging the capabilities of existing methods, producing a sub-optimal generalization-upstream performance frontier, and the positive result of applying many edits is largely offset by the weaker ability of GRACE to generalize the content of the edit to related inputs (paraphrases or entailed information).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work addresses the lifelong model editing setting, where errors stream into a deployed model, and the model is updated to correct wrong predictions. The author uses a key-value strategy to look up the codebook of edits to change the behavior of a model. The input feature of a module is used as the key, while the desired output of the module will be the value stored in the codebook. The edits are stored as key-value pairs and extracted based on the similarity between an input feature and the stored keys. The experiment uses QA datasets (zsRE) and document classification data (SCOTUS) to create the stream editing scenario and compares it with MEND, fine-tuning, and a memory-based method. 
strength_and_weaknesses: - Strength: The lifelong model editing setting is an interesting problem setting not explored by previous works. This setting modifies the previous model editing problem to make it closer to a real-world scenario. The proposed approach is simple but effective. This method looks like a reasonable baseline for future works to compare and improve.
- Weaknesses: Readers familiar with model editing may find no surprise in the paper's method and experiments. The setting and method are different from previous works, but readers may have difficulty finding an interesting novelty or new insight that is intriguing or inspiring. I see the contributions made in the paper, but they need more depth, like the previously accepted papers have. One way to improve is to beef up the experiment section. Ex: create easy/hard streaming editing scenario, ablation study, level of distribution shift versus editing success rate, etc. Lastly, the SOTA method [1] should be included and compared. [1] looks to have very good potential in the lifelong model editing setting.
- Other concerns/questions:
1. In Figure 4(a), why does stream MEND start from such a low upstream F1? Should every method start from a point close to the unedited model f0?
2. Although MEND was not designed for streaming model editing, its success rate is expected to be reasonably good for the first few editing in the stream. However, Figures 4(a) and 5(a) show a very poor result of MEND/fine-tuning from the beginning. Why?
3. In Figure 4(a), the colors for different hyperparameters are not shown in the figure. Only the darkest color is visible (ex: light purple and light red do not appear in the figure). This could need to be clarified for readers. 
4. limitation in applying autoregressive decoding. I see the author mentioned this in its limitation section, and I appreciate it. However, this is a huge limitation that does not occur with other model editing strategies. 
5. About the Memory Network baseline, is there a reference paper? Please provide more detailed descriptions if it is created for this paper.

[1] https://arxiv.org/abs/2206.06520
clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is easy to follow.
- Quality: The claims are supported.
- Novelty: The problem is less explored in previous work. The method looks new but is straightforward.
- Reproducibility: It contains informative descriptions, but I am not confident that a knowledgeable reader could replicate the experiment.

summary_of_the_review: The problem setting is interesting, and the method is marginally novel. The results look good but need more analysis to provide in-depth insight into the problem. Overall the weakness outweighs its strength.

[Post-rebuttal]

The evaluation has considered all revisions and responses. This form was up-to-date.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work studies how to make large pre-trained models learn to correct errors in deployment. More specifically, it proposes a method to continue editing the pre-trained model in an online streaming setting. The discrete key-value bottleneck is adapted to address the continuous distribution shift. Experiments demonstrate that it outperforms several related methods in the lifelong model editing setting.

strength_and_weaknesses: Strengths
1. This paper investigates a more realistic streaming setting for model editting.

2. It successfully adapts the discrete key-value bottleneck to the streaming edit setting.

3. Experiments show its superior performances over regular finetuning, streaming MEND, and Memory Net on QA shift and SCOTUS.

Weaknesses
1. The definition of Values is unclear enough. If the key is the output of layer l-1, is the corresponding value used as the input of layer l or l+1? The definition in Section 2.2 says the next layer, which is kind of ambiguous. Figure 2 shows an example of layer l and l+1, which gives the impression of using the output of layer l to retrieve the input of layer l+1. I think it’s necessary to clarify the definitions of keys and values.

2. The part below Figure 2 mainly describes how to add/update keys, with too little information about adding/updating values. Where do the values come from when initializing/adding/updating keys? Are they the same as keys when first added or randomly initialized? How do we update the values to correct the instance error? Do the updates possible affect the previous edits? 

3. Section 2.3 doesn’t provide enough training details, what data are used in training and how many gradient steps are used for each edit. Does it use all previous edit data or just the current instance? What if the batch size is too large as the saved edit data increases? This should be an important section to understand how GRACE is trained and used, but the descriptions are limited. I think it’s better to present an algorithm for training and inference.

4. Memory usage? Need to store a large codebook for each layer?

5. Which layers to use GRACE? All layers or some picked layers? If two or more layers use GRACE, does each layer need to do the similar search? If an edit is required, do all layers need to add/update key and values? Slow inference speed?

6. Some typos exist, e.g., “a language model trained in 2016 would correctly Barack Obama as president of the United States,” and “”

7. Ablation studies about which layers to use GRACE and how many optimization steps are used to update the values are missing.

clarity,_quality,_novelty_and_reproducibility: 1. In a nutshell, this paper mainly applies an existing key-value technique to the new streaming model editing setting.

2. The paper is well-motivated and easy to understand, but some technical details are not missing.

3. Reproducing the method may have issues since some training details are not clear.

summary_of_the_review: This papers adapts an existing technique to a new and interesting model editing setting. Though good performances are achieved, some training details and ablations are missing, which affect the paper quality.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces some interesting new ideas that address a relevant problem, but there is not enough reviewer support to accept this paper. The author response significantly improved the paper, but it does not quite go far enough. In particular, the new experiments that test generalization do not include any comparison to prior methods (i.e. only ablations), and the lack of thorough generalization experiments is a key weakness. I'd encourage the authors to use all of the feedback to improve the paper and resubmit to a future venue, since the paper does have promise.
justification_for_why_not_higher_score: The new experiments that test generalization do not include any comparison to prior methods (i.e. only ablations), and the lack of thorough generalization experiments is a key weakness
justification_for_why_not_lower_score: n/a
Decision: Reject