Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors introduce the task of Sequential Model Editing (SME) where a model needs to adapt to and fix its mistakes from a stream of incoming input-output examples of a certain machine learning task. The setting extends Model Editing (ME), where people focus on editing the model to fix a single occurrence of an observed mistake. The authors define the desired properties of SME, namely reliability, generality, and locality. The authors then propose Transformer-Patcher for SME, which adds a single neuron to the last fully-connected layer of a Transformer; along with specially designed losses and training schemes for Transformer-Patcher. The authors measure 5 metrics that cover all the three desired aspects. Experiments show that 1) HyperNetwork methods fail MSE, 2) the proposed Transformer-Patcher clearly outperforms the fine-tuning and HyperNetwork baselines, often by a big margin, 3) Transformer-Patcher is effective up to thousands of edits. The authors further provide analysis on the metrics progression of the tested methods, and ablation study of the losses.
strength_and_weaknesses: ### Strengths

1. The proposed task of Sequential Model Editing (SME) is novel, interesting and of practical value. With the increasing adoption of large language models, training, and even fine-tuning, can become computationally expensive or inefficient. We've seen rising interest in Model Editing where people try to find efficient ways to fix undesired model behavior. However, existing works focus on single mistakes from an input-output example. In real ML applications, a model needs to take user requests constantly, any downtime or unfixed mistakes could harm user experience. The SME setting manifests the need for a model-editing method that is timely (quick to apply), efficient (cost/computation effective) and effective (fixes the mistake) even after a series of accumulating edits (not just a single edit).
2. The SME desiderata (Sec 3), namely reliability, generality, and locality, are clearly stated and formulated. It gives a clear base for what the authors are looking for.
3. The design of Transformer-Patcher (Sec 4) is simple and of intuitive sense.
4. The design of the objective losses (Sec 4.2) are clearly explained and are of intuitive sense. In Sec 5.3, the ablation study suggests the usefulness of the losses. I do think the exact formula for l_{m1} (Eq (17), see Q1) seems a bit overcomplicated, but this is relatively minor.
5. The experimental support is strong that SME is an important and challenging setting and that Transformer-Patcher clearly outperforms competitive baselines. This strength can be further break down to
   1. The 5 metrics (Sec 5.1) are carefully chosen to reflect different aspects namely reliability, generality, and locality, and are clearly described.
   2. Competitive baselines are chosen (Sec 5.1), including several variants of fine-tuning and recently proposed HyperNetworks KE (Cao et al., 2021) and MEND (Mitchell et al., 2022a).
   2. Results (Table 1 and 2) show almost unanimous improvements over baselines across all metrics and both tasks, often by a big margin.
   3. Main results (Table 1) and metric progression over number of edits (Figure 4 and 5) show that HyperNetworks completely fail SME, despite being successful in single step ME.
   4. Ablation study (Table 3, Figure 6) shows that their proposed losses are effective in preventing regression across edits.
7. The paper is clearly written and easy to follow. Some details and phrasing can be made more clear, but they do not harm the main strengths.
8. Experimental details are provided in the appendix. Code is provided in the supplementary material. Both are good for reproducibility,

### Weaknesses

Nothing major. See my Questions and LPs. I think clarifying them can further improve the paper's strength and help me gain more confidence in recommending the work.

### Questions for the authors
1. P5, Eq (17), "S(v; k) = Avg[TopK(exp(v); k)]". Why do we need TopK? If we care about maximum ("Although constraint 15 is about the maximum..."), why not simply take the maximum, or use a surrogate like softmax?
2. Table 1 and 2. Why are TrainR and TextR generally < 1? If we are fixing errors (SR↑, GR↑) and not forgetting previous fixes (ER↑), then we should see the final models be more accurate than the initial models (TrainR>1, TestR↑>1)? The authors argue at one point (P8, L6) that there can be distribution shit between the train and test set. However, we are also seeing TrainR < 1 for all methods and datasets. Are we missing something here?

### Localized points (LPs)
1. P3, Eq (4), Locality, "$\forall x_j \in I_{x_e}, f′(x_j) = y_{x_j}$". $\forall x_j \in I_{x_e}, f′(x_j) = f(y_{x_j})$ makes sense as "locality" to me. The former means that the edited model $f'$ should give correct predictions to irrelevant examples, which is the learning task itself. The latter means that the edited model $f'$ should preserve the same prediction as the unedited model $f$ to irrelevant examples, which is more aligned with "locality" -- we do not want to change the model behavior at irrelevant examples, because we believe the model is mostly correct but cannot be sure without testing it against the ground-truth data.
2. P7, L3 after the figures, "The ideal edit length for each run is about 63 for
FC and 139 for zsRE." What is "edit length" and what does "ideal'' mean here? Is edit length the size of the D_edit or the number of edits the proposed method performed? Is it a hyperparameter that the authors decided based on experimental observations? Does "ideal" mean that the proposed method performs worse after 63/139 mistakes or edits? It seems like this claim belongs to results, not the settings.
2. P7, Experiment Details. It's worth mentioning the size of the test datasets. How many examples are for the test for each task? What is the total number of time steps T for each of the 20-fold runs?
3. P7, Table 1, "* denotes that SR of T-patcher on QA is not 1 but very close to 1." Does it mean other SRs in the column are actually exact 1? I suppose that the authors mean that the T-patcher SR is >0.995 so it rounds to 1.00 if we only keep two decimal points, but I cannot be sure. If that is indeed the case, I suggest improving the phrasing, e.g. sampling write out the exact number like "* T-patcher SR on QA = 0.9965".
4. P7, bottom, "Table 1 shows that Transformer-Patcher achieves good performance for about one hundred edits, ..." Exactly *how many* edits? And again, does "edit" here mean the size of the test set D_edit, or the number of times the proposed method applied a patch?

### Typography/Minor points
1. Abstract, "... either fail to make a sequence of edits or to remember previous edits" -> "... either fail to make a sequence of edits or fail to remember previous edits".
2. P2, top, "Therefore, an ideal model editor should conduct ..." Perhaps a better phrasing can be "Therefore, an ideal model editor should enable _countinuous_ fixing of newly emerged mistakes in a both effective and efficient manner."
3. P2, middle, "To handle SME.We introduce ..." -> "To handle SME, we introduce ..."
4. P3, Figure 2. Unify the notations of $y_1$ or $y_{x_1}$.
5. P3, Sec 3. I suggest use $y_e$, $y_j$, $y_1$, etc instead of $y_{x_e}$,  $y_{x_j}$, $y_{x_1}$, etc to reduce clutter.
6. P3, Property 1-3. Start with either capitalized or uncapitalized letters.
7. P3, Eq (5), "$f_t(x_k) = y_{x_k} , \{k | f_{k-1}...$" -> "$f_t(x_k) = y_{x_k}$ , for $k$ where $f_{k-1} ...$".
8. P4, Figure 3. Maybe revise the UK Prime Minister example...
9. P6, Eq (21). Is it $T \sum N_t$ or $T N_t$? It makes sense to me if you first average over N_t samples at time t and then average the average over T time steps.
10. P6, Eq (23), "we compare the performance the final model of f_T and the initial model f_0 on sub-training test D_tr." -> "we compare the performance of the final model of f_T and the initial model f_0 on the subsampled training set D_tr".
11. P6, Eq (20-22). I recommend moving the denominators before the \sum's. It would look more intuitive to me.

clarity,_quality,_novelty_and_reproducibility: - Clarity: Great. The paper is clearly written and easy to follow. Some details and phrasing can be made more clear, but they do not harm any main points.
- Quality: Great. Clearly designed and explained task, goal, method, metrics, and results. The task is interesting and of practical value. The goal and method makes sense and are clearly described. The method is simple yet effective. The metrics are carefully chosen to reflect different aspects of the desiderata. The experimental support is strong and definitive that MSE poses a new challenge and the proposed Transformer-Patcher works much better than previous methods.
- Novelty: Great. Sequential Model Editing is a novel, interesting and practical setting that poses a different challenge than the previous studies on Model Editing. This is one step further towards the realistic scenario of the online updating of ML models. The results shown previous models which are good at single edits are not necessarily good at continuous editing. The model design is simple yet effective. The loss design shows the authors' consideration for the task.
- Reproducibility: Good. Experiment details described in the paper and the appendix. Code is provided.

summary_of_the_review: This is a strong work that extends an existing task (Model Editing) to a more realistic and challenging setting (Sequential Model Editing), where previous successful methods fail. The authors clearly state the desiderata, method, and metrics, along with the intuitions behind them. Experimental results show strong support that the proposed Transformer-Patcher is effective by a clear margin. Further analysis confirms the effectiveness of the design. Model Editing is of interest as the language models grow ever large and are increasingly adopted in real applications. I think this is a work that would interest audiences from the application of large language models, model editing, model adaptation, continuous learning, adversarials and robustness, model security and more.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work addresses the model editing problem where the edits are applied sequentially. The paper proposes to add one neuron per edit to the FFN layers of the last transformer module for language models. The hidden dimension of the FFN module is therefore increased by one after one edit is applied. The weights associated with the added neuron are optimized by Adam with the edit and some previous data stored in memory. The experiments evaluate the proposed method with MEND, KE, and some variants of fine-tuning under the sequential model editing setting. The results of two model editing datasets (FEVER, zsRE) show that the proposed method achieves a high edit success rate without affecting non-edit cases.
strength_and_weaknesses: [Strengths]
1. The proposed method looks novel and can scale to thousands of edits. The result looks very good.
2. The sequential edit setting is not much explored by previous works.
3. The exp setting in Table 2 is meaningful in that it "employs authentic examples where the model makes mistakes".

[Weaknesses]
1. The method looks slow and requires significant computation (ex: 23.8 seconds per edit for QA).
2. The paper should have compared the closest work SERAC [1]. SERAC can do sequential model editing and make no changes to the original base model. 
3. The clarity has a large room for improvement. Please see the next section for details.
4. The claim in the last paragraph of section 4 is not supported. The benefit of editing the last block can be supported by applying the same method to other layers. Besides, more aggressive paraphrasing of facts/questions could be applied to prove better generalizability or create more challenging cases like in [1].
5. The paper needs a detailed description of how it manages the memory to store previous examples. Does the method sensitive to the amount of stored data? What is the value of d_m used in the experiments?
6. The ablation in Table 3 can be more fine-grained. For example, ablating L_m1 and L_m2 separately.
7. To show SME is a setting that ME methods may fail, Table 1 could add the batched results of MEND/KE.

[1] https://arxiv.org/abs/2206.06520
clarity,_quality,_novelty_and_reproducibility: [Clarity]
The writing is not easy to follow and may require multiple passes of reading. The figures introduce confusion.
1. It needs to be clarified what does Figure 1 like to highlight. Comparing the effect of batch editing and sequential editing on user experience may be a more intuitive purpose.
2. Figure 3 needs to be clarified. This figure looks not match the descriptions in Section 4. The layer-N block is part of the "patch" while it contains another patch. On the left side, the model prediction is an input to the patch. This looks misleading as well.
3. Eq10 has a redundant line with a_p=0
4. Figure 4 missed labels for the x and y-axis. 4(a)'s legend overlaps T-patcher's results. 

[Novelty]
The method is novel.

[Reproducibility]
The paper needs to include a detailed description of its memory management mechanism. The description of the experiment and data preparation looks informative.

summary_of_the_review: The paper provides a novel method for a less explored problem setting in model editing. The experiment results look promising. However, as mentioned above, the execution has many parts that need significant improvements to support the claims better. The mixed pros and cons lead to a borderline score.


[Update]

======== After rebuttal ========

I appreciated the author's detailed responses. Most of my concerns are addressed, although some still need to be fully satisfactory. I have exchanged opinions with other reviewers and agree that the current form's pros slightly out-weight the cons. The score was updated to reflect the overall evaluation. Below are some points that I hope the paper could further improve:
1. Add stronger support for generalizability.
2. Add a section to discuss the limitations. Large editing complexity could be one. Generalizability is another arguable one. I believe there are others, and I hope the author discloses all for the community to benefit from your findings.
3. A better comparison with prior work. The proposed SERA (simplified from SERAC) does not provide a convincing excuse to make the simplification of the SOTA method. Interestingly, SERA has better GR (generalizability) than the proposed method, making the standing of this work less solid. Anyway, SERA is still a nice baseline to add; therefore, I do not consider this a deal breaker.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The author first proposed the Sequential Model Editing task which requires fixing a series of mistakes for a pre-trained Transformer model by a model editor. And then the author introduced Transformer-Patcher which can be an effective model editor to fix a series of mistakes without serious performance drop. 
strength_and_weaknesses: (+) The paper is clear and easy to follow.

(+) For the Model Editing (ME) problem proposed by Mitchell et al., 2022a, this paper proposes a sequential model editing task (SME) with more practical significance and application scenarios as mentioned in introduction part.

(+) And then the authors give a solution to the SME problem and conduct experiments on both classification and autoregressive generation tasks to prove its ability to edit the model up to thousand times continuously and outperform other methods.

(+) The code is attached as supplementary material and I think should be released if this paper is accepted. 

(-) Although the authors claim to support multiple edit operations, they did not perform a stress testing on the model, similar to testing the effect of the model in extreme cases, such as thousands or millions of edit operations. The effect of the final model may deviate from what is described in the paper.

clarity,_quality,_novelty_and_reproducibility: This paper is well-written and organized and the code is attached as supplementary material. But the novelty of this work is limited.
summary_of_the_review: For some technical details in the model, I didn't do much checking. I prefer to accept this paper and also need to look at the feedback of other reviewers before final deciding.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new problem, named Sequential Model Editing, that requires a model editor to make a sequence of error corrections while keeping all the previous edits, preserving model performance, as well as generalizing to equivalent inputs. Instead of modifying the model parameters, a set of new parameters (termed neurons in the paper) are added to the last layer of transformers. Because of the existence of the ReLU/GeLU activation functions, the neurons will not be activated for irrelevant inputs. Compared to previous models, Transformer Patcher is shown to be more effective on those metrics.
strength_and_weaknesses: Strengths:

1. The paper presents a new problem to the community: How can a model editor fix a sequence of edits without forgetting? The problem is new and most of the previous methods can fail on this task.
2. Additionally, the paper proposes a few metrics to evaluate the capability of model editors in different dimensions: Reliability, generality, and locality. 5 metrics are proposed to evaluate those properties.
3. Compared to baseline models, Transformer Patcher shows effectiveness on all of these metrics, especially for GR and ER.
4. The proposed method can be lightweight for minor error correction since it avoids fine-tunings. It can also be scaled up to thousands of edits.

Weaknesses:

1. The problem proposed in the paper, i.e. sequential model editing, assumes the model would correct the errors one by one, while the problem can be re-formulated so that the model only performs one edit on a batch of errors. It might be practically useful for industrial applications where information may gradually accumulate and the product needs to be up-to-date, but it might not be the typical case.
2. The problem settings put other models in an unfavorable light. They're not designed to be operated multiple times, and running so many rounds of training to correct certain errors could lead to catastrophic forgetting (Shown in fig 5). In another word, they're not proper baselines to compare with.
3. Although the method can be scaled up to more edits, the cost is also significant. Since each edit will introduce additional parameters, the memory and computational overhead could be non-negligible, while the basic fine-tuning method does not introduce any additional parameters.

Questions:

Neurons are designed to be active only for certain errors, but dot-product seems not to be complex enough to recognize the patterns of errors. Thus, I'm interested in how often are the neurons activated. E.g. can they be activated with paraphrases? How often are they wrongly activated for irrelevant examples?

-----

After rebuttal: I decided to raise my score from 5 to 6 as I'm leaning to accept the paper.
clarity,_quality,_novelty_and_reproducibility: The paper is clear about the method, with analyses of running metrics, patches, and generality. Proper metrics are designed for all aspects of performance is concerned. Model and training details are provided in the appendices, which is helpful to reproduce the results.
summary_of_the_review: The paper proposes a new model editing problem: Sequential Model Editing, with a method named Transformer Patcher. The method is shown to be effective on the problem, but whether this is a useful task and whether baselines are properly configured need to be discussed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an approach to editing (or patching) Transformers by adding neurons that encode the additional knowledge. The strengths of the paper are its consideration of a novel scenario and empirical performance. The weaknesses of the paper are scalability and potential unrealistic-ness of the one-by-one sequential editing scenario (as opposed to batched updates).
justification_for_why_not_higher_score: Comparison against baselines may not be super fair as previously baselines were tuned for a batched setting. Moreover, there were legitimate concerns about how this method would scale to realistic scenarios.
justification_for_why_not_lower_score: The method is novel and moreover targets a new online knowledge acquisition setting.
Decision: Accept: poster