Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper analyzes the storage and recall of factual associations in GPT and shows evidence that these associations correspond to localized, directly-editable computations. It utilizes a causal intervention to identify decisive neuron activations in a model’s factual predictions and find that the middle MLP layer plays an important role.
strengths_and_weaknesses: **Strengths**
* This paper is clearly motivated and well-structured in terms of writing. The causal tracing analysis tool is powerful and has some interesting findings that the essential role of MLP module computation at the middle layers when recalling a fact.
* This paper proposes Rank-One Model Editing (ROME) and COUNTERFACT dataset to evaluate edits in standard benchmark and the proposed counterfactual one in language models. ROME can achieve competitive results on standard benchmarks and better results in Specificity, Fluency and Consistency metric. 
**Weaknesses**
* The finding that MLP module computation at the middle layers plays an important role when recalling a fact is not surprising to me. After all, self-attention only linearly combines inputs and does not conduct transformation, and MLP requires doing non-linear transformation in a transformer block during the pre-training stage. Hence, it's kind of intuitive to me of have these results.
* ROME seems to edit one fact at once. This is not scalable as usually, we want to inject knowledge of a corpus rather than a single fact. Further,  ROME can not take unstructured knowledge, i.e. plain text, which is widely existed and used during pre-training.
* Results on GPT-J is less useful than standard fine-tuning baseline, which makes to cast doubts on how useful ROME is when model size is larger,e.g. GPT-3 175B.
questions: Consider adding results across different model sizes and benchmark datasets.
limitations: The lack of human evaluation on generated text as an automatic evaluation metric may not highly correlate well with human beings.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies how factual knowledge (e.g., a tuple of subject, relation and object) is stored in autoregressive transformer-based LMs. More specifically, the authors identify specific weights (in MLP) that are responsible for the correct output by comparing forward computation results (i.e., hidden states) with clean and corrupted inputs (__Causal Tracing__). To verify this observation, the authors propose a knowledge editing approach (__ROME__) that modifies a MLP weight matrix by a rank-one update.

__Causal Tracing__
This method identifies specific hidden states that are more influential on the predicted object. The authors use the gap between the probability of the object with the original and corrupted inputs as a proxy of the importance of hidden states. Based on this analysis, mid layers at the last subject token have high values, contributing more to the probability of the object.

__ROME__
ROME is designed to change the model output (i.e., the object) given the subject and relation. To do so, this approach modifies the weight matrix in MLP with a rank-one update so that the modified network outputs the desired/modified object. This is done by choosing k* (a vector before the matrix) and v* (a target vector) where v* requires optimization based on the desired object. Once an instance-specific k* and v* are computed, the weight matrix is updated using a closed formula.

__Evaluation__
ROME is evaluated on an existing benchmark (zsRE) and a newly constructed dataset (CounterFact) and compared against other knowledge editing approaches such as KnowledgeEditor and MEND in terms of efficacy (is the object successfully edited?), generalization (does it work on paraphrases as well?), and specificity (are similar subjects kept unchanged?).  

strengths_and_weaknesses: [Strengths]
- This paper is tackling a well-motivated and very important problem: how LMs store factual knowledge and how we can control it. ROME can be used for impactful applications such as resolving temporal mismatch between LMs and real-world knowledge.
- The authors analyze their hypothesis thoroughly and propose an efficient way to modify model parameters. The authors keep ROME simple but show its effectiveness. 
- This paper is very well-written; each part of this paper is clearly explained despite the complexity of the problem.

[Weaknesses]
- My only concern about ROME is scalability.  To find v* for the modified object o*, this approach requires optimization for each test instance. This might be a speed bottleneck when the data size is very large. Although modifying a weight matrix is rank-one update, I’m not sure if this approach is considerably cheaper than other methods like MEND. MEND uses a hypernetwork, but, once it’s trained, inference is just forward computation. 

questions: - Line: 159:  ` In practice, we sample x_j by generating 50 random token sequences of length 2 to 10 using G.`
   - How does a choice of k* affect the final performance of ROME? Have you tried different methods?
- It would be nice to discuss pros and cons of ROME compared to others (e.g., KE, MEND). 


limitations: The authors mentioned the major limitation of this study that only focuses on relational knowledge (eg., KB triples). Studying other types of knowledge (e.g., more complex knowledge about entities) would be an interesting future direction.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper provides evidence that knowledge recall in autoregressive language models corresponds to localized computations within feedforward MLP layers at a range of middle layers, and hence can be edited with a simple method that they name ROME. They compare ROME to other existing model-editing approaches both (1) on a zero-shot relation extraction benchmark and (2) related to generalization and specificity ability, and demonstrate strong results.
strengths_and_weaknesses: This paper offers very detailed empirical analysis and ablations that provide useful insights, and it is written very clearly. The work done here is both significant and original as factual storage capability of autoregressive language models are not well-understood to the best of my knowledge. I do not see a major weakness that would prevent acceptance.
questions: -How did you tune the variance of the noise added to form the corrupted version of hidden states considering the extent you need to corrupt for the subject to be unknown might depend on the downstream task?

-Did you try corrupting the subject with different corruption methods? I would be curious whether the conclusions regarding the importance of MLP for knowledge recall have a correlation with the fact that introduced noise is Gaussian.

-Could you comment on the fact that ROME is underperforming baselines for generalization and whether generalization vs. specificity performance is an inherent trade-off?
limitations: Authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: The paper proposes a method (ROME) to analyze the storage and recall of factual knowledge in a large-scale autoregressive language model, and find that such knowledge can be controlled by changing weights in the MLP layer.  The reviewers all agree the paper is well-motivated, well-motivated, and scientifically sound. The area chair is also impressed with the through experimentation and the quality of the writing. The main issue the reviewers (1iyP,gUgP) pointed out is that the method is not scalable for practical knowledge editing (as the method can only work per fact basis), but the authors confirmed that the goal of this study is not to provide a practical tool but more to understand the inner workings of LMs, which are valuable on its own. The authors have comprehensively addressed the reviewer’s points, e.g., clarifying misunderstanding, adding a human evaluation study, extra results on smaller models, comparing its strength and limitations compared to other methods. I would vote for acceptance. 
confidence: Certain
Decision: Accept