Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing

Authors: Kento Nishi, Maya Okawa, Rahul Ramesh, Mikail Khona, Hidenori Tanaka, Ekdeep Singh Lubana
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: mechanistic interpretability, knowledge editing, transformers

Abstract:
Knowledge Editing (KE) algorithms alter models' weights to perform targeted updates to incorrect, outdated, or otherwise unwanted factual associations. To better identify the possibilities and limitations of these approaches, recent work has shown that applying KE can adversely affect models' factual recall accuracy and diminish their general reasoning abilities. While these studies give broad insights into the potential harms of KE algorithms, e.g., via performance evaluations on benchmarks, we argue little is understood as to why such destructive failures occur. Is it possible KE methods distort representations of concepts beyond the targeted fact, hence hampering abilities at broad? If so, what is the extent of this distortion? Motivated by such questions, we define a novel synthetic task wherein a Transformer is trained from scratch to internalize a "structured" knowledge graph. The structure enforces relationships between entities of the graph, such that editing a factual association has "trickling effects" on other entities in the graph (e.g., altering X's parent is Y to Z affects who X's siblings' parent is). Through evaluations of edited models and analysis of extracted representations, we show that KE inadvertently affects representations of entities beyond the targeted one, distorting relevant structures that allow a model to infer unseen knowledge about an entity. We call this phenomenon representation shattering and demonstrate that it results in degradation of factual recall and reasoning performance more broadly. To corroborate our findings in a more naturalistic setup, we perform preliminary experiments with pretrained GPT-2-XL and Mamba models, reproducing the representation shattering effect therein as well. Overall, our work yields a precise mechanistic hypothesis to explain why KE has adverse effects on model abilities.

Decision: reject
Reviewer scores: 3, 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730412482043

thread_type:
official_review

<thread_object>
creator: reviewer_jmel

<content>
summary:
This paper presents a mechanistic explanation of the ripple effects of knowledge editing. This explanation considers how the representations of the model changes during the KE. This paper considers a particular cyclic structure. When the model’s training data are generated from this cyclic structure, the learned representation gradually forms a geometry that resembles the cyclic structure of the data. KE, both corrective edits and counterfactual edits, shatter this cyclic structure. In this way, the ripple effect change in the model’s behavior after edits can be illustrated by the change in the representations.

strengths:
- The ripple effect of knowledge edit has been studied, but the prior works have not explained the effects from a mechanistic viewpoint. This paper takes a nice step toward explaining this effect mechanistically.
- The angle toward explaining this ripple edit effect is interesting (despite a bit niche). Cyclic structure is a commonly occurring structure in natural language, and there is a nice illustration in the representation space.
- This paper uses a nice combination of qualitative and quantitative illustrations to describe the representation shattering effect.

weaknesses:
- The quantitative explanation part can take up more space in the writing. Currently only Figure 5(c) involves quantitative description of representation shattering, and other parts are qualitative. Table 1 presents the overall effects of knowledge editing, which is not really the quantitative descriptions.
- Additionally, the definition for describing the representation shattering effect should be introduced earlier. Currently it’s hidden at lines 413-415 at page 8, closer to the end of the paper. This should be moved to the front.
- Subsection 3.3 can be made clearer. Based on the example presented, I struggled at understanding how the nodes are connected, and e.g., what I/III/ABC refer to.
- The effects are tested on a 2-layer nanoGPT Transformer, plus a GPT2. These limit the generalizability of the finding. I’d recommend experimenting on at least two models on each size — perhaps containing a mixture of uni-directional models (like *GPT) and bidirectional models (like BERT).

questions:
Could you elaborate on the synthetic data generation process? Do A/B/C refer to the entities and the numbers refer to the edges?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690223402

thread_type:
official_review

<thread_object>
creator: reviewer_dzjz

<content>
summary:
The paper constructs a synthetic dataset with cyclic knowledge graphs and investigates the influences of knowledge editing on the synthetic datasets. The results show that KE inadvertently affects representations of entities beyond the targeted one, distorting relevant structures that allow a model to infer unseen knowledge about an entity.

strengths:
1, The synthetic dataset is well-designed in a way that all the relations are related in a clear semantic meaning. And Figure 3 clealy demonstrates the structure of the synthetic dataset.

2, The visualization is really good. It clearly shows the representation shattering after the knowledge editing (Figure 4, 5, 6 and 8).

weaknesses:
1, The expected answers (or the ground truth answers) after knowledge editing is not clearly stated in the paper. For example in Figure 3 Ring I, after the editing of $1.I\\_C2 = 3$, is $1.I\\_C2 = 2$ still true (so that entity 1 now have 2 entities for relation I_C2) or does this edge simply disappear? Which one of the following is considered the ground truth for evaluation after the editing: $1.I\\_C3 = 4$ or $1.I\\_C4 = 4$? In the proposed synthetic dataset, it could still have a consistent logic for LLMs even after knowledge editing. The paper should make the true facts after the knowledge editing more clear.

2, The prompt templates and codes are not released. How the query and the structural information of KGs are provided in the prompt will have a big influence on the final experiment results. Can the authors provide them during the rebuttal? And can authors show some examples of the LLM outputs in natural languages after knowledge editing to help readers better understand the mistakes LLMs make?

3, More knowledge editing baselines should be included. In this paper ROME is the mainly considered knowledge editing methods. There are other 'locate and edit' methods such as PMET[1]. And there are other types of knowledge editing with extra parameters such as GRACE[2]. Including more knowledge editing methods will make the 'representation shattering' hypothesis more convincing.

4, Some typos need to be fixed. For example, $(x_i, a, x_j)$ should be  $(x_i, r, x_j)$ in line 191. Why is “1 IC_4 3 IIIA_2 7 IIIA_3 2 IIC_2 6” in line 235-236 a plausible sequence in figure 3? $(x_i lr, x_j)$ is not clear in line 291.

[1] Li, Xiaopeng, et al. "Pmet: Precise model editing in a transformer." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024.

[2] Hartvigsen, Tom, et al. "Aging with grace: Lifelong model editing with discrete key-value adaptors." Advances in Neural Information Processing Systems 36 (2024).

questions:
Please refer to the questions mentioned in the Weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703556325

thread_type:
official_review

<thread_object>
creator: reviewer_ek2y

<content>
summary:
This paper provides a way to investigate the representation scattering hypothesis during knowledge editing. It proposed a test for using synthetic entities to probe if the representation scattering happens.

strengths:
* A new hypothesis, representation scattering,  is proposed
* Experiments are done to test the hypothesis proposed,

weaknesses:
* Only the GPT model is studied, which makes the scope limited in the model architecture.
* The result seems hard to reproduce.

questions:
* Is the assumption still valid on different model architectures, including encoder-decoder models like T5, and Mamba?
* Is the assumption still valid on larger LLM models like Lamma, Gemma, Phi 3.5 etc?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730827830712

thread_type:
official_review

<thread_object>
creator: reviewer_dh8a

<content>
summary:
The key idea of the paper is that knowledge editing (KE) alters the geometry of transformer models' representations, making them inconsistent or "shattered." The authors introduce the concept of "representation shattering" to explain why KE results in degradation of the model's performance. This perspective of geometric shattering is both insightful and novel, providing a deeper understanding of how targeted edits can inadvertently disrupt broader model behavior. The study is conducted using synthetic knowledge graphs (KGs), which allows the authors to identify specific geometric changes in transformer representations.

strengths:
1. **Novelty**: This paper is the first to delve deeply into transformers' representations, providing a mechanistic understanding of why the detrimental impacts of KE occur. In contrast, previous work mainly treats transformers and LLMs as black boxes, focusing on observing the degradation of models' overall performance in targeted tasks.
2. **Unique Approach and Insightful Perspective**: The authors tackle the problem from the perspective of studying representation geometry, which offers a unique and insightful angle into how KE affects model consistency.
3. **Preliminary Results on LLM**: The authors extend their findings to a pretrained large model, GPT-2-XL, showing that the representation shattering phenomenon also applies to LLMs, not just to transformers trained on synthetic data.

weaknesses:
Although the perspective and hypothesis of this paper are novel and insightful, I have concerns regarding the limitations of the experimental setup, specifically the reliance on cyclic structure in the synthetic KGs and the focus on only one KE method.

1. **Why Cyclic Graphs?** The key assumption of this study is that transformers can learn a cyclic geometric representation when the training KGs have cyclic relation subgraphs. This assumption on its own makes sense. However, to my knowledge, real-world KGs often do not have relation subgraphs that conform to cyclic orders. For example, take the advisor relation that is used as the running example in the paper. It is unlikely that "Alice is the advisor of Bob", "Bob is the advisor of Charlie", and "Charlie is the advisor of Alice." In reality, most KG relations seem to form trees instead of cycles. This makes the study somewhat disconnected from real-world scenarios as cyclic relations are uncommon. Why not investigate the geometry of transformer representations in real-world KGs (or a semi-synthetic KG that more closely mimics the structure of real-world KGs) when the relation subgraphs are trees, and examine how KEs affect such geometry?
2. **Knowledge Graph Construction**: The construction of the knowledge graphs, particularly the use of cycles, is not very intuitive and is challenging to understand. This is compounded by conflicting information in Section 3.2 (e.g., Line 202 states that the KG is defined to have 3 relations, but Line 205 later refers to 24 relations) and an explanation that omits critical details. Specifically, it is unclear what the exact cyclic orders are for the edit, retain, and test relations, and whether these relations are organized in such a way that each of the three groups has distinct orders. Additionally, Figure 3, intended to provide more intuition and details, is difficult to interpret due to unclear labeling, such as the meaning of different arrow colors and which edges were edited. Clarifying why the authors chose this specific construction, particularly the necessity of the use of different cyclic orders, and providing more transparent explanations of the data construction would greatly improve the reader's ability to understand the motivation behind the experimental setup.
3. **Single KE Method (ROME)**: The paper only studies one knowledge editing method, ROME. The findings would be significantly strengthened if multiple KE methods were analyzed to determine whether the representation shattering phenomenon is a general property of transformer models occurring on a more fundamental representation level, independent of the KE method used. For instance, the paper shows that shattering occurs even for corrective edits, which is surprising because a perfect corrective edit should ideally refine the model's representation geometry. It is thus crucial to investigate whether this surprising phenomenon is specific to ROME or is a broader theme across different KE methods.
4. **Lack of Mitigation Strategies**: While the paper does an excellent job of identifying the issue of representation shattering, it lacks discussion on potential ways to mitigate this problem. Including some hypotheses or suggestions for future work would provide direction for researchers interested in addressing this issue.

In addition, some other comments on readability and presentation:

5. **Figures**: The Figure 1 is somewhat confusing, particularly in panel (b). It is unclear how the ring structure is represented in the visualization, and what the labels "182" and "608" refer to. Additionally, Figure 1 is not mentioned in the introduction, which makes it difficult to understand its relevance early on. It would help if the illustration was used to convey the high-level intuition of the findings. Similarly for Figure 3, mentioned in W2.
6. **Undefined Terms**: The abbreviation "DGP" (Figure 1 caption, line 67) is undefined. I assume it stands for "Data Generating Process"—clarifying this would improve readability.
7. **Entity and Relation Naming**: The naming conventions for entities and relations are unclear. For instance, on line 235, it is not immediately evident that "1," "3," etc., are entities, and that "I_C4," "III_A2," etc., are relations. In addition, there are clues in Figure 3 that the number of "I"'s stand for the different cyclic order, but this is not mentioned and explained in the main text. An explanation of the naming convention and its significance (e.g., whether the number of "I"s corresponds to hops in the KG) would be helpful.
8. **Typo**: On line 291, the phrase "it changes fact $(x_i l r, x_j)$" should probably be "it changes fact $(x_i, r, x_j)$"
9. **Unclear Sentence**: On line 366, the caption of Figure 5 reads: "We plot the mean drop in accuracy against the in the Frobenius norm of the difference in..." This sentence does not make sense as written and needs to be revised.

questions:
1. **Possibility of Studying Geometry on Real-World KGs**: As previously raised in W1, would it be feasible to apply the same methodology to real-world KGs to examine how the representation geometry shatters under KEs? Would the lack of cyclic orders make this significantly more challenging? Additionally, could the Isomap visualization technique still be used effectively in this scenario to visualize a tree-like structure rather than rings?
2. **Training Setup Clarification**: Are all edit, retain, and test relations used during the pre-training of the transformer, and the test relations are only held out specifically for the KE process but they are all observed by the transformer during pre-training? If so, a clarification near line 213 would be helpful.
3. **Logical Inference Task**: For the logical inference task, could you elaborate on how the "hold out" process works? For example, if the inverse relation (say "advisee") is held out and unobserved during training, how can the model know that the held-out relation token "advisee" is the inverse of the observed relation "advisor"?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731102793742

thread_type:
official_review

<thread_object>
creator: reviewer_jmai

<content>
summary:
The paper introduces the Representation Shattering Hypothesis, which posits that Knowledge Editing (KE) methods distort a model's latent representations, potentially to the point of losing the learned global graph structure—a phenomenon they call "representation shattering." The authors hypothesize that this mechanism underlies KE's negative impact on models' factual and reasoning abilities.

To test this hypothesis, they first create a toy knowledge graph to pre-train a language model on. Then, using a popular KE method called ROME, they edit facts within this graph, demonstrating that the extent of representation distortion correlates with how significantly an edit disrupts the graph's structure.

The authors further validate their hypothesis with a real language model, GPT2, by using ROME to corrupt the order of the months of the year. They show that this corruption distorts the ring structure of the month representations, with greater distortions occurring for larger edit distances.

strengths:
- **Insight into Adverse Effects of KE:** With knowledge editing (KE) methods gaining popularity, this paper provides valuable insights into why KE can lead to adverse effects, offering a deeper understanding of its impact on model representations.

- **Clarity and Accessibility:** The paper is well-written and accessible, making complex concepts easier to understand for a broad audience.

- **Novel Toy Knowledge Graphs:** The authors introduce innovative toy knowledge graph (KG) tasks to explore KE and the phenomenon of representation shattering, providing a structured and interpretable framework for analyzing KE effects.

weaknesses:
### Weaknesses

- **Limited Scope of Experimentation:**
  - The authors primarily experiment on knowledge graphs (KGs) with a ring-like structure, which is a somewhat artificial setup. Even the experiment with a real KG is restricted to a small subset with a similar ring configuration, limiting the generalizability of the findings.
  - Only one knowledge editing (KE) technique, ROME, is tested, leaving it unclear how the findings might extend to other KE methods.

- **Reliance on Visual Inspection of Representation Shattering:**
  - The approach seems heavily dependent on visually inspecting the breakdown of representations, which becomes infeasible for more complex, unvisualizable data. Although the authors propose using the Frobenius norm as a quantitative measure of shattering, it is only applied in one experiment. 

### Suggestions for Improvement

- Extend the synthetic setup to incorporate small, realistic KGs to further validate the hypothesis.
- Experiment with additional KE methods to assess whether the representation shattering phenomenon holds across different approaches.
- Rely more consistently on quantitative measures (like maybe the Frobenius norm) to gauge representation shattering, especially in cases where visual inspection may not be feasible.

questions:
see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732812754739

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all reviewers for taking the time to provide feedback and engage in the review process. Our manuscript analyzes model representations before and after knowledge editing, which the reviewers found to be valuable in several aspects. The reviewers noted that our writing is "well-written and accessible" (R-jMAi) and our "visualization is really good" (R-DzjZ). They considered our approach with synthetic data to be "innovative" (R-jMAi) and "well-designed" (R-DzjZ), "providing a structured and interpretable framework" (R-jMAi). Additionally, our work was recognized as "the first to delve deeply into transformer's representations" and analyze the "impacts of KE" (R-DH8A).

We have incorporated feedback from the reviews into the manuscript with salient changes highlighted in blue. We have also added 2 new sections to the appendix: Appendix F.5 and F.6. We summarize some of the changes below. Overall, we believe that these changes address many of the common concerns raised by the reviewers.

## New results with different editing protocols, Mamba, and tree-like graphs 
The first concern was that representation shattering was only shown with one knowledge editing method. In sections F.5.1, and F.5.2, we have added new experiments with **3 other knowledge editing methods.** We also ran **experiments with a MAMBA-2.8B model** (in Appendix F.5.3), which uses a different architecture and has significantly more parameters than GPT-2XL. The reviewers also raised the concern that our experiments were limited to cyclic graphs. We have conducted **new experiments on tree-like knowledge graphs** that consist of cities and countries as entities (see Appendix F.6). We again find evidence of representation shattering. Furthermore, edits that use entities that are further away in the representation manifold lead to greater degree of representation shattering. **In all our experiments, we found that representation shattering occurs in all these scenarios with the extent of shattering increasing with the distance of the edit.**

## Updated experimental details
To improve the clarity of our work, we have **added more details** to the description of the data generating process and the **construction of the knowledge graphs** (highlighted in blue). We have also improved captions and annotations in some figures as suggested by the reviewers. In order to ensure that the visualizations are faithful, we have **added quantifications of representation shattering to our figures with a well-defined metric.**

## Summary
Overall, we hope that these changes address the primary concerns of the reviewers, and we look forward to engaging in a discussion. We have also responded to individual comments below and urge the reviewers to peruse through the revised version of the manuscript.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a new hypothesis called "Representation Shattering" to explain performance degradation in knowledge editing approaches. The research suggests that knowledge editing can distort a model's latent representations and potentially destroy learned global graph structures. The severity of this distortion appears to correlate with how significantly an edit disrupts the underlying structure, as demonstrated through experiments with both synthetic knowledge graphs and real language models.

The research makes valuable contributions through its novel theoretical framework, which provides a mechanistic explanation for knowledge editing's negative effects rather than treating transformers as black boxes. The methodology is particularly innovative, employing synthetic knowledge graphs for controlled investigation of how edits affect representation geometry. The findings are supported through both qualitative visualization and quantitative metrics, with the results extending from synthetic settings to real models, suggesting this phenomenon may be fundamental to transformer architectures.

However, the research faces several limitations in its current form. The experimental scope is notably restricted, focusing primarily on cyclic structures in knowledge graphs and testing with a single editing method. The methodology heavily relies on visual inspection of representation shattering, with unclear ground truth specifications after editing and insufficient quantitative metrics for measuring distortion. Additionally, the presentation could be improved, particularly in explaining knowledge graph construction and defining key concepts earlier in the paper.

The paper would benefit from broader validation across multiple editing methods and diverse model architectures, along with investigation of non-cyclic knowledge structures. Technical improvements should include more rigorous quantitative metrics for measuring representation shattering, clearer specification of ground truth post-editing, and comprehensive code release for reproducibility. The presentation would be enhanced by introducing key definitions earlier and improving figure clarity and labeling.

While the core concept shows promise and could significantly impact the field, the current submission requires substantial revisions to address these methodological limitations and strengthen the empirical validation of the hypothesis. The recommendation is to reject the paper in its current form, allowing for these necessary improvements to be made before reconsideration.