Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing

## Abstract

Abstract Knowledge Editing (KE) algorithms alter models’ internal weights to perform targeted updates to incorrect, outdated, or otherwise unwanted factual associations.
In order to better define the possibilities and limitations of these approaches, recent work has shown that applying KE can adversely affect models’ factual recall accuracy and diminish their general reasoning abilities.
While these studies give broad insights into the potential harms of KE algorithms, e.g., via performance evaluations on benchmarks, we argue little is understood as to why such destructive failures occur.
Is it possible KE methods distort representations of concepts beyond the targeted fact, hence hampering abilities at broad? If so, what is the extent of this distortion?
To take a step towards addressing such questions, we define a novel synthetic task wherein a Transformer is trained from scratch to internalize a “structured” knowledge graph.
The structure enforces relationships between entities of the graph, such that editing a factual association has “trickling effects” on other entities in the graph (e.g., altering X ’s parent is Y to Z affects who X ’s siblings’ parent is).
Through evaluations of edited models and analysis of extracted representations, we show that KE inadvertently affects representations of entities beyond the targeted one, distorting relevant structures that allow a model to infer unseen knowledge about an entity.
We call this phenomenon representation shattering and demonstrate that it results in degradation of factual recall and reasoning performance more broadly.
To corroborate our findings in a more naturalistic setup, we perform preliminary experiments with a pretrained GPT-2-XL model and reproduce the representation shattering effect therein as well.
Overall, our work yields a precise mechanistic hypothesis to explain why KE has adverse effects on model capabilities.

## 1 Introduction

Large language models (LLMs) have led to unprecedented advances in several domains (Gemini Team, ; Bubeck et al., ; Touvron et al., ; Thoppilan et al., ; Chowdhery et al., ; Qin et al., ; Chen et al., ; Ahn et al., ; Driess et al., ).
However, the static nature of their training pipelines implies that as our world evolves, models’ internalized knowledge can become incorrect or outdated.
To address this, recent work has proposed several protocols for knowledge editing (KE), wherein the goal is to minimally and precisely alter model weights such that only the targeted information (and its relevant associations) are updated, but all unrelated information remains (ideally) unaffected (Mitchell et al., ; Meng et al., ; ; Dai et al., ; Cheng et al., ; De Cao et al., ; Sinitsin et al., ).

Despite significant work on the topic, it still remains unclear precisely what effects KE should have on a model.
For example, assume you edit the fact that “Michael Jordan won the 1998 NBA MVP” to “Reggie Miller won the 1998 NBA MVP”, then what should the impact of such an edit be?
Should the model now believe Michael Jordan and the Chicago Bulls never reached the NBA finals in 1998?
Should it perhaps believe Reggie Miller was on the Chicago Bulls?
Should the pop quote “Be like Mike” (Wikipedia, ) now become “Be like Reggie”?
As Hofweber et al. (); Hase et al. () argue, it is difficult to design clear, well-defined answers for such questions.
Motivated by this, recent work has started investigating precisely what effects KE actually has on the model (Hoelscher-Obermaier et al., ; Li et al., ; Lynch et al., ).
For example, Cohen et al. () demonstrate that knowledge beyond the edited fact can often be impacted in a detrimental manner, such that the model begins to have an incoherent understanding of the world; Gupta et al. () demonstrate unrelated facts are often forgotten by the model post-editing; and Gu et al. () show that KE can harm broader reasoning capabilities beyond mere factual recall.
While these works clearly demonstrate the detrimental impacts of editing on a model, they still leave open the question precisely why such harms occur—at a mechanistic level, how are model representations impacted such that a broad set of knowledge and capabilities in a model are heavily distorted once an edit occurs?

This work. To address the questions above, we aim to develop a mechanistic understanding of the impact of KE on a model’s internals.
For this purpose, we argue we must solve two problems: (i) identify how a model expresses knowledge about some predefined set of entities in its representations, and (ii) investigate how this mechanism is affected as we apply KE to alter a fact corresponding to a subset of the entities.
Instead of attacking a complicated system that may be difficult to interpret (e.g., an off-the-shelf LLM), we take inspiration from a multitude of recent papers that establish synthetic abstractions of the target system and develop precise hypotheses as to why the phenomenon-in-question occurs (Allen-Zhu & Li, ; ; ; Okawa et al., ; Chan et al., ; Li et al., ; Lubana et al., ).
Specifically, we define a data-generating process that yields entities arranged in a structured knowledge graph.
This structure is defined via use of a predefined set of relations that locally constrain how entities relate to each other (similar to parent-child relations).
Given enough entities and relations, such local constraints manifest a broader global structure in the knowledge graph.
Performing traversal over the nodes of this knowledge graph, we get sequences that can be used as “strings” to train a Transformer on.
As we show, this protocol leads to the model precisely encoding the structure of the graph in its latent representations.
However, when KE is applied to edit either incorrectly learned facts or insert counterfactual knowledge (using the method proposed by Meng et al. ()), we find latent representations are heavily distorted and the graph structure completely destroyed—we call this phenomenon representation shattering.
Interestingly, this phenomenon manifests in proportion to how far the proposed edit moves a given node from its current location to a new location in the graph (defined via edge distance).
We thus hypothesize representation shattering underlies the detrimental effects of KE on a pretrained model’s factual and reasoning capabilities at broad.
Overall, we make the following contributions in this work.

Figure: Figure 1: *Representation shattering* explains why knowledge editing can sometimes degrade models’ general capabilities. *(a)* Prior works finds that editing facts, e.g., the president of the US, can sometimes harm general abilities of LLMs (figure reproduced from Gu et al. ()). *(b)* We introduce a synthetic data generation process defined by a knowledge graph containing ring-shaped geometries. When we train a model on our synthetic data, we observe that the model’s internal representations mirror the ring structure of the underlying DGP. We then explain the post-edit degradation of the model’s capabilities by uncovering the “shattering” of latent representations.
Refer to caption: x1.png

- •
Structured Knowledge Graphs as a Toy Setting for Investigating Impact of KE. We propose use of a structured knowledge graph wherein entities (nodes) are connected to each other via predefined local constraints (relations) that manifest into a broader, global structure in the graph (see Sec. 3).
Training Transformers on strings (path traversals) from the graph, we find model representations precisely encode the global structure of the graph.
This allows us to assess the impact of KE at a more mechanistic level, since distorting a fact now has global effects that can be precisely (and, in fact, visually) delineated by analyzing the model representations.
- •
Representation Shattering as a Mechanistic Hypothesis to Explain Detrimental Effects of KE. We find KE distorts latent representations for entities in the graph such that the global geometry learned during pretraining is, at times, completely destroyed—we call this phenomenon representation shattering and hypothesize it underlies the detrimental effects of KE on model capabilities observed in prior work (see Sec. 4).
As we show, the extent of harm on latent representations turns out to be correlated to the amount an edit alters the graph from its original organization into the new, desired one.
- •
Investigations with Off-the-Shelf LLMs. Using a pretrained GPT2-XL model, we show editing facts that involve entities organized in a precise structure (e.g., days of the week) replicates representation shattering, providing evidence for our claims in a more naturalistic setup (see Sec. 4.5).

Overall, our results lead to the representation shattering hypothesis—a precise mechanistic explanation for why KE hampers model abilities.
While our analysis is primarily limited to synthetic settings, we believe the results are likely to generalize to broader settings as well, as indicated by our preliminary investigations with GPT2-XL.

## 2 Related Work

Knowledge Editing.
Several protocols for knowledge editing (KE) have been proposed in recent work.
Early work defined meta-learning based approaches (Sinitsin et al., ; De Cao et al., ; Mitchell et al., ) and established the broader desiderata for what properties a KE protocol should satisfy; e.g., ensuring facts unrelated to the target one are not hampered via the editing protocol.
Building on work aimed at understanding how Transformers encode knowledge in their internals (Geva et al., ), modern KE protocols focus on defining closed-form operations that involve (i) localizing knowledge to specific components in a model (e.g., MLP layers) and (ii) defining operations to alter a factual association by assuming the fact is represented in a localized manner (Meng et al., ; ).

Evaluations of Knowledge Editing Methods.
As argued by Hase et al. (); Hofweber et al. (), the problem of KE is relatively ill-defined.
Consequently, it is unclear that when we edit knowledge within a model, what effects said edits “should have” on other facts it may have internalized during training.
Prior work has hence taken an alternative approach, primarily focusing on developing an empirical understanding of what the phenomenology of KE protocols is: e.g., if an edit is performed, how are counterfactual statements or unrelated facts affected.
These works generally show that KE in fact has extreme detrimental effects on a model, e.g., hampering both its broader internalized knowledge and its reasoning abilities (Hase et al., ; Cohen et al., ; Hoelscher-Obermaier et al., ; Gupta et al., ; Gu et al., ).
While the primary methodology used in such papers is to perform empirical benchmarking of a model that has undergone editing, we instead focus on a mechanistic analysis of how editing alters a model’s representations (albeit in a toy synthetic task) to yield the undesirable effects on model abilities.

Explaining Models via Synthetic Tasks.
To disentangle the failures of KE methods from the failures of the models themselves, we argue for use of a more controllable and interpretable setup.
Such a setup can help identify a concrete hypothesis for why KE has undesirable effects on the model, which we can then analyze in naturalistic settings by designing more precisely defined experiments.
This methodology of designing toy, control tasks to investigate hypotheses for phenomenology of a neural network has yielded promising results in recent years, providing, e.g., a concrete hypothesis for how chain-of-thought reasoning aids model capabilities (Prystawski et al., ; Feng et al., ), models for emergent capabilities (Okawa et al., ; Lubana et al., ), existence of nonlinear representations (Engels et al., ), and failure modes for compositional generalization (Zhou et al., ).

## 3 Formalzing knowledge editing

Epistemology has grappled with the nature of knowledge for centuries (Chappell, ).
In this work we adopt a humble, yet precise definition of knowledge based on structured knowledge graphs.
The graph is used to represent how facts, entities, and relations are interlinked, giving rise to notions of consistency, coherency, and reasoning across different pieces of information.
Using these definitions, we will define a synthetic data generation process on knowledge graphs, in order to systematically study knowledge editing in Transformers.

### 3.1 Knowledge Graphs

A knowledge graph consists of a collection of entities $X=\{x_{i}\}_{i=1}^{n}$, and a collection facts $F$ that relate different entities.
For example, a graph defined on entities $X=\{\text{``Alice", ``Bob", ``Carol"}\}$ can encode the fact “Alice is the advisor of Bob” using the relation “advisor”, represented as $(\text{``Alice"},\text{``advisor"},\text{``Bob"})$.

###### Definition 3.1 (Knowledge graph) .

Formally, a knowledge graph $G=(X,R,F)$, consists of nodes $X$, relations $R$ and facts $F$, where each fact $f=(x_{i},r,x_{j})\in R$ is defined by a relation $r\in R$ between two entities $x_{i},x_{j}\in X$.

A relation sub-graph, corresponds to a sub-graph constructed by only considering facts that use relation $r$. For example $G_{\text{advisor}}$ is a sub-graph that specifies all facts for the relation “advisor”. Every knowledge graph contains a collection of facts that can be inferred from the graph.

Related pieces of information such as “Alice’s advisor was Bob” and “Bob’s advisor was Carol” can be composed to form cohesive statements such as “Alice’s advisor’s advisor was Carol. To capture such statements, we define compositions of relations below.
The composition of relations are essential to capture ripple effects that occur in the knowledge graph after an edit (Cohen et al., ) to a relation in $R$.

###### Definition 3.2 (Composition of relations) .

A composition of relations $\vec{r}=(r_{1},r_{2},\cdots,r_{k})\in R^{k}$ with respect to knowledge graph $G$ is defined such that for every fact $f=(x_{i},\vec{r},x_{j})$, there exists a collection of facts $\{(x_{i},r_{i},x_{i+1})\}_{i=1}^{k}$ such that $x_{1}=x_{i}$ and $x_{k+1}=x_{j}$. In other words, any fact defined on the composition of relations, has a corresponding set of facts defined on relations from $R$. Furthermore, the set of facts form a path in the knowledge graph such that the sequence of relations in the path are $r_{1},r_{2},\cdots r_{k}$.

###### Definition 3.1 (Knowledge graph) .

###### Definition 3.2 (Composition of relations) .

### 3.2 Cyclic graphs: a description of the entities and relations

Figure: Figure 2: Isomap projections of representations in Llama-3.1-405B (Fiotto-Kaufman et al., ). The geometry of the data—for example, the cyclic nature of months or days—is often reflected in the representations learnt by language models. Similar representations can also be found in other models like GPT-2-Small and Mistral 7B (Engels et al., ).
Refer to caption: x2.png

We study knowledge graphs where every relation sub-graph is a set of disjoint cyclic graphs, i.e., for every entity $x_{i}$ and relation $r$, there exists exactly one entity $x_{j}$ such that $(x_{i},a,x_{j})\in F$.
We specifically choose a cyclic geometry as a global constraint on the graph structure since cycles are a common pattern that relate entities in natural language domains; e.g., see Fig. 2, where we show a 2D projection of representations from Llama-3.1-405B corresponding to months of a year and days of the week naturally organize in a cyclic fashion.

Knowledge editing methods (Meng et al., ; ) target a set of entities for which predefined facts are to be edited, while using another retain set of facts about said entities to help ensure relations beyond the targeted ones are not altered. A test set of facts are then used to evaluate how well the method worked.
Motivated by this, the specific knowledge graph we define contains 3 relations and 2048 entities.
The 2048 entities are placed in 3 different cyclic orders (identified as order I, order II, and order III).
When applying knowledge editing, the cyclic order containing the target fact being edited is designated as the edit sub-graph.
Then, one of the two remaining cyclic orders is randomly chosen as the retain sub-graph to sample the retain set of facts from.
Finally, the one remaining cyclic order is assigned to serve as the test sub-graph to sample the test set of facts for evaluation.
We create 8 relations for each cyclic order,
totaling to 24 relations.
The 8 relations correspond to the 1-hop, 2-hop, 3-hop and 4-hop neighbors in the clockwise and anti-clockwise directions in the cycle; see Fig. 3 for an illustrative example of knowledge graph with 8 nodes, arranged in 3 different cyclic orders.

We use the method proposed by Meng et al. (), called ROME, for our analysis (see Appx. C for a short primer).
Specifically, we use ROME to target the edit sub-graph relations.
The facts based on edit relations are then tested to check if a knowledge edit was successful.
The retain sub-graph relations are used by the knowledge editing algorithm to minimize changes to unrelated relations, but no edits are made to facts that use these relations.
The test sub-graph relations are neither edited, nor used by the knowledge editing algorithm, but serve as a held out set of relations that can be used to evaluate whether unrelated facts remain unchanged after a knowledge edit.
The 1-hop neighbor relation graphs (both clockwise and anti-clockwise) contain a single cycle, 2-hop relation graphs consist of 2 cycles, the 3-hop relation graph contains 1 cycle, while the 4-hop relation graph contains 4 cycles.
The k-hop neighbor relations are related to each other by design, so any edit to one k-hop relation should be consistent with all other k-hop relations.
An edit corresponds to changing a fact in the knowledge graph, which can also be interpreted as changing an edge in the relation graph (as shown in Fig. 3).
The distance of an edit (see pink annotations in Fig. 3) is defined as the shortest distance between the original and edited entity in the cyclic order.

### 3.3 Data generating process

Figure: Figure 3: Synthetic data generation process with a cyclic knowledge graph. Each entity (node) has relations (directed edges) pointing to $24$ other entities (nodes) in total, following the connectivity structures of 3 distinct cyclic orders. In each cyclic order, we draw $8$ edges corresponding to 8 relations ($1-4$ hops clockwise and anti-clockwise). We select a random path on the knowledge graph using all 24 relations in the prompt below. The relations follow the naming convention of $\langle\text{cyclic-order}\rangle_\langle\text{direction}\rangle\langle\text{ hops}\rangle$, i.e. II_A3 is the relation corresponding to the three-hop anti-clockwise neighbor in the second cyclic order.
Refer to caption: x4.png

We generate a sequence of alternating entities and relations resembling $x_{1}\vec{r}_{1}x_{2}\vec{r}_{2}x_{3}\vec{r}_{3}\dots$, where any consecutive triplet of entity, relation, and entity $x_{i}r_{i}x_{i+1}$ from the sequence is a fact $(x_{i},\vec{r}_{i},x_{i+1})$ in the knowledge graph.
The composition of relations $\vec{r}_{i}=r_{i1}r_{i2}r_{i3}\dots$ is a sequence of 1 or more relation tokens, while $x_{i}$ is a single entity token.
Every token is sampled using a uniform probability over all the permissible choices (see Alg. 1).
For example, a plausible sequence for the example in Fig. 3 is “1 I_C4 3 III_A2 7 III_A3 2 II_C2 6”, which is an alternating sequence of entities and k-hop relations. Note that this sampling process is identical to traversing random walks on the knowledge graph, similar to previous works (Prystawski et al., ; Khona et al., ). Additional minor details of the DGP are documented in Appx. B.

##### Training setup.

We train a Transformer model using next-token prediction on the synthetic data generated from the above data generation process. For all experiments (unless stated otherwise), we use a 2-layer nanoGPT Transformer (Karpathy, ). For additional details, see Appx. C.

##### Evaluation (seen facts).

We assess the model’s ability to remember facts seen during training both before and after an edit. Specifically, to analyze whether the model has learned the fact $(x_{i},\vec{r},x_{j})$, we prompt it with an entity $x_{i}$ and a relation $\vec{r}$, expecting it to produce $x_{j}$ as the next token.

##### Evaluation (unseen facts).

We also evaluate the model on two other criteria.
(1) Compositional inference. In addition to facts seen in the training data, we evaluate the model on compositions of relations. The model must preserve geometric structures of the data in order to compositionally generalize after a knowledge edit.
(2) Logical inference. A key feature of reasoning in natural language is logical inference. For example, if Alice is said to be the advisor of Bob, then Bob is an advisee of Alice (even if it is not explicitly stated). Our data generation process has similar relations, such as clockwise and anti-clockwise 1-hop neighbors. By “holding out” one direction for some such pairs of relations from being observed verbatim in the training dataset, i.e., the relation may only appear compositionally, we can assess the degree to which the model internalizes properties among related relations. We can also evaluate if editing a fact for a relation changes the fact for other related relations, i.e., we check if the model’s knowledge is logically self-consistent after an edit.

## 4 Uncovering Representation Shattering

We study Rank-one model editing (ROME) (Meng et al., ), which is a prototypical KE algorithm for language models (see Appx. D.1).
We perform two different types of edits: corrective edits and counterfactual edits. Corrective edits are applied on facts which the model recalls incorrectly after training.
A counterfactual edit introduces a new fact, i.e., it changes fact $(x_{i},r,x_{j})$ to fact $(x_{i},r,x_{k})$ where $x_{j}\neq x_{k}$. Such an edit introduces inconsistencies in the knowledge graph.

Overall, we show the following.
(1) Transformers trained on knowledge graphs recall facts, perform logical inferences, and compositional inferences.
However, both corrective and counterfactual edits degrade the model on all three fronts.
(2) Transformers learn a representation that reflects the underlying geometry of the data. Knowledge edits “shatter” this representation, which serves as an explanation for the degradation in accuracy after KE.
(3) Counterfactual edits with larger distance display a larger degree of shattering.
(4) These phenomena occur in GPT-2 sized language models, indicating representation shattering can explain degradation in model abilities after KE.

### 4.1 Evaluating the effects of knowledge editing

We evaluate the effects of counterfactual and corrective edits on three fronts.
Direct recall accuracy calculates the accuracy of facts seen during training.
Logical inference accuracy measures the accuracy on a subset of held out relations that can be inferred from other relations, i.e., the k-hop anti-clockwise neighbors can be inferred directly from the k-hop clockwise neighbors.
Compositional inference accuracy measures the accuracy on a held out subset of compositions of 2 relations.
Both logical inference and compositional inference measure the accuracy on samples that would be considered out-of-distribution.
Since our models are trained on a concatenation of facts, there are multiple prompts that elicit the same fact. For example, both $x_{1}\vec{r}_{1}$ and $x_{0}\vec{r}_{0}x_{1}\vec{r}_{1}$ should result in the same completion since the completion is based on the same fact.
To account for these different prompts, we simply sample $3$ sequences from the data generation process which elicit the fact of interest and average the output probabilities across the inputs.

Table: Table 1: The direct recall, logical inference, and compositional inference accuracies before and after KE. *(a)* The performance of our model (before editing) across the three cyclic orders (I, II, and III). Not only does our model perform well on direct recall, but it also generalizes to both logical and compositional inference tasks. This suggests that the model’s internal representations extend beyond simple memorization and capture the underlying global structure that relates entities. *(b)* Changes in model accuracy after applying corrective knowledge edits. Each $\langle\Delta\text{Acc.}\rangle$ result is averaged across multiple edits, and each row labeled edit/retain/test is averaged across each of the cyclic orders taking turns, i.e., playing the roles of the edit, retain, and test sub-graphs. We find that corrective knowledge edits negatively affect the model’s accuracy both on related and unrelated facts. These results align with the findings on LLMs (Gu et al., ; Gupta et al., ). *(c)* $\langle\Delta\text{Acc.}\rangle$ for edit, retain, and test sub-graphs after applying counterfactual edits. Intentionally introducing inconsistencies into the model’s knowledge via counterfactual KE can significantly degrade model capabilities. Furthermore, the greater the induced inconsistency (scaling the counterfactual edit distance $d$ from 1-4), the more severe the resulting performance degradation.

We report scores for all three metrics stratified by the cyclic orders (I, II, III) in Tab. 1.
The model’s logical and compositional inference accuracies are close to the direct recall accuracy, which implies that the model generalizes outside of the training data before KE. However, after KE, all accuracies decrease, with a more severe decrease for counterfactual edits (they introduce inconsistencies between facts). The accuracies on all relations are similar, regardless of the cyclic order (edit, retain, test) that it is associated to—this allows us to use all the relations interchangeably for computing accuracy.

### 4.2 Transformer representations capture the geometry of the data

The model achieves high compositional and logical inference accuracies before knowledge editing, indicating that it captures the global structure of the data and does not merely memorize all the facts seen during training.
We see this reflected in the internal representation of the model (output of the second attention layer), which we visualize using Isomap (Tenenbaum et al., )—a non-linear dimensionality reduction method that uses multi-dimensional scaling with distances computed using a local neighborhood graph.
In Fig. 4a, we plot the evolution of the Isomap embedding—of the internal representation for the input with one entity and relation ($x\vec{r}$)—over the course of training. The different data points correspond to different values of the entity $x$, for a fixed relation $\vec{r}$ and the points in the plot are colored by the cyclic ordering.
We see that the representation manifold resembles the cyclic ordering of the entities, particularly towards the end of training.

Figure: (a) (a)
Refer to caption: x5.png

### 4.3 Corrective knowledge edits shatter the representation geometry

Figure: (a) (a)
Refer to caption: x6.png

We investigate how the representation changes after applying a corrective knowledge edit—which applies KE to a fact that the model learned incorrectly during training.
While one would expect the performance of the model to increase after a corrective edit, we find the opposite: the corrective edit results in a drop in all accuracies (see Tab. 1).
These results align with previous empirical findings showing that reasoning capabilities degrade after a corrective model edit (Gu et al., ; Cohen et al., ).

We visualize the representations of 3 different models using the techniques described in 4.2.
The 3 models are obtained after applying 3 different edits and are selected to have high (★), intermediate (▲) and low (✖) direct recall accuracies.
In Fig. 5, we observe that the model with the highest accuracy (★) has a representation that preserves the geometry of the data after the edit.
However, as the model accuracy decreases, the representations also display a greater degree of distortion, no longer capturing the geometry of the data.

We refer to this disintegration in the geometry as representation shattering, which can occur as a result of KE.
We also measure the extent of shattering using the Frobenius norm of the change in the geodesic distance matrix constructed using Isomap before and after the edit, i.e., $|\Delta D|=|D-D_{\text{edit}}|_{1}$.
We find that $|\Delta D|$ correlates negatively with the model’s performance, i.e., edited models with larger $|\Delta D|$ values also show a larger drop in direct recall, logical and compositional inference accuracies (see Fig. 5c).

Figure: Figure 6: Counterfactual edits with larger edit distance result in larger drop in accuracy and greater degree of representation shattering. We apply counterfactual knowledge edits to overwrite a correctly learnt fact (1154.I_C1=567) with inconsistent counterfactual associations. We then plot the accuracy after the counterfactual edit for different edit distances and the corresponding low-dimensional embedding of the representation obtained using a modified version of Isomap (see Appx. E). We find that a counterfactual edit with larger edit distance requires a significant distortion to the representation geometry in order to learn the new fact.
Refer to caption: x9.png

### 4.4 How do different counterfactual edits change the extent of shattering?

Counterfactual editing is commonly used for evaluating KE (Meng et al., ; ; Gupta et al., ; Hoelscher-Obermaier et al., ) and it involves adding new facts, which were previously unseen during training.
We consider 25 different counterfactual edits corresponding to every single counterfactual edit distance, where the counterfactual edit distance (or CE distance) is the distance between the entity in the old fact and new fact as measured in the cyclic order. For example, Fig. 3 illustrates an example where the counterfactual edit has an edit distance of 1.
In Fig. 6, we see that increasing the distance of the counterfactual edit results in a drop in accuracy and an increasing in the extent of shattering. In other words, when a new fact changes one entity to another, the extent of shattering increases as the distance between the old and new entity increases. As an example, if the entities are different months, then accuracy is higher when we edit “December” to “November” as opposed to “July”.
Overall, these results lead us to believe the following hypothesis.

###### Hypothesis 4.1 ( Representation shattering ) .

Language models embed related entities on a manifold in their internal representations. KE methods distort this manifold in order to insert new facts or alter old ones, i.e., they shatter model representation. The extent of representation shattering increases with the distance between old fact and the desired new fact.

Figure: Figure 7: Representation shattering also occurs in real LLMs (GPT-2) for months of year. We applied KE (ROME) with counterfactual prompts for the order of months to GPT-2. The ring structure holds for edit distances up to 3 but becomes untied for distances 4 and 5.
Refer to caption: x10.png

###### Hypothesis 4.1 ( Representation shattering ) .

### 4.5 Representation shattering in LLMs

Figure: Figure 8: Inducing representation shattering via KE degrades model performance in real LLM (GPT-2). We evaluate GPT-2’s ability to perform a reasoning task from Gu et al. () before and after editing. As the edit distance grows, accuracy gradually decreases, with a notable drop at distance 4, coinciding with the point of representation shattering.
Refer to caption: x11.png

Finally, we investigate whether our findings generalize to large Transformers trained on naturalistic data.
We consider concepts with a cyclic order, in particular months of the year and apply a counterfactual edit to GPT-2 (Radford et al., ) using ROME to change the order of the months.
We generated prompts following the template described in Engels et al. (), which include prompts such as “Let’s do some calendar math.
One month after January is February”. For a distance-1 edit, we modified the answer to “March”; for a distance-2 edit, we changed it to “April”, and so on.
We then updated the parameters of GPT-2 with these new prompt-answer pairs using ROME.
Fig. 7 shows the latent embeddings for the 12 months extracted from the GPT-2 model before and after the edit.
We find that as we vary the edit distance from 1 to 5, the observed representation shattering increases.
In Fig. 8, we examine the impact of representation shattering on model performance. We evaluated the GPT-2 model on the reasoning task from Gu et al. () both before and after editing.
As the edit distance increases, we observe a gradual decline in accuracy, with a drop at distance of 4, which corresponds to the point of representation shattering. This result demonstrates that our findings from synthetic data can generalize to larger models trained on naturalistic data.

## 5 Discussion

In this work, we introduced a synthetic framework to analyze the side effects of knowledge editing in transformers, identifying “representation shattering” as a key factor behind performance degradation. Our framework allows controlled investigations into models modified by knowledge editing, offering clear representation-level explanations for why knowledge editing can harms models’ broader capabilities that generalize to real-world models like GPT-2.
However, the use of simplified tasks and models can limit the scope of our conclusions—larger, more complex real-world models may exhibit additional dynamics that our framework does not capture. Further exploration of different architectures and editing scenarios can help solidify these findings.

## Contributions

KN and ESL conceived the project direction and designed the primary experimental setup, with inputs from RR and MK. The representation shattering hypothesis was proposed by KN, HT, and ESL. KN led experiments on the synthetic knowledge graph domain. RR, KN, and ESL wrote the paper. Conceptual figures were designed by HT, with inputs from KN and MO. MO conceived the natural data setup for confirming the hypothesis and ran experiments therein. KN wrote the appendix. ESL and HT supervised the project.

## Appendix

## Appendix A Setup Details

We will publicly release the source code for our work on GitHub at a later time.

### A.1 Pseudo-Code

Let $U(.)$ define the uniform distribution over the input. Let $X$ be the set of entities, $R$ the set of relations and $F$ the set of facts, defining a knowledge graph $G=(X,R,F)$.

Figure: Algorithm 1 Generate a single sequence containing a collection of facts.

## Appendix B Data Generation Process Details

For this study, we use the following hyperparameters for our data generation process.

- •
Number of entities: 2048
- •
Number of example sequences: $10^{8}$
- •
Maximum composition length: 2
- •
Maximum entities per sequence: 8

Additionally, only when generating the training dataset, we drop sequences which contain one direction of a pair of conjugate facts with fixed probability $p$. In other words, if the fact $(x_{i},r,x_{j})$ always implies that $(x_{j},r^{\prime},x_{i})$ is a valid fact (i.e. $r=$ I_C1 and $r^{\prime}=$ I_A1), one of $(x_{i},r,x_{j})$ or $(x_{j},r^{\prime},x_{i})$ may be restricted to inclusion in the training dataset by composition only (with probability $p$). Holding out these relations allows us to benchmark the model’s logical inference capabilities on relations it could not have directly memorized from the training dataset. In practice, we set the probability $p=\frac{2}{3}$.

## Appendix C Model Architecture

Our Transformer model is a fork of the open-source nanoGPT repository ([https://github.com/karpathy/nanoGPT](https://github.com/karpathy/nanoGPT)). The design is inspired by GPT, and the architecture is a decode-only Transformer with a causal self-attention mask. Our hyperparameter values are as follows.

- •
Batch size: 256
- •
Context length: 16
- •
Optimizer: Adam
- •
Learning rate: $6\cdot 10^{-4}$
- •
Training epochs: $1.5\cdot 10^{5}$
- •
Decay iterations: $1.5\cdot 10^{5}$
- •
Momentum: $\beta_{1}=0.9$, $\beta_{2}=0.95$
- •
Activation function: GeLU
- •
Block size: 16
- •
Embedding dimensions: 24
- •
Heads: 12

As for tokenization, we assign every entity and relation a unique token and use standard next-token prediction with cross-entropy loss. $\text{target}_{n}$ is the $1$-shifted version of the training sequence accounting for the padding token, and $\mathbf{x}_{n}$ are the logit outputs of the model at the $n$th timestep.

$$ $\mathcal{L}(\mathbf{x}_{n},\text{target }n)=-\log\Big{(}\dfrac{\exp(\beta x_{n ,\text{ target }n})}{\sum_{v=0}^{\#\text{tokens}}\exp(\beta x_{n,v})}\Big{)}=- \log\Big{(}\underbrace{\text{softmax}(\beta\mathbf{x}_{n})_{\text{target }n}}_ {\text{prob}(\text{target }n)}\Big{)}$ $$

## Appendix D Rank-One Model Editing (ROME)

### D.1 Algorithm Definition

Rank-One Model Editing (ROME), proposed by Meng et al. (), is a popular knowledge editing algorithm used on LLMs. Their contributions are two-fold: first, through “causal tracing,” they find that early-layer MLP modules of transformer models are implicated in encoding factual associations. Second, interpreting feed-forward layers as linear associative memories encoding key-value pairs, ROME applies a rank-one update to the MLP weights.

Notationally, for a factual association $(x_{i},r,x_{j})$, the key is the entity $x_{i}$ while the value is $x_{j}$. In each feed-forward layer, the hidden state $\mathbf{h}_{i}^{(l-1)}$ at layer $l-1$ is transformed into a key $\mathbf{k}$ by the weight matrix $\mathbf{W}_{fc}^{(l)}$, and the corresponding value $\mathbf{v}$ is retrieved by the matrix $\mathbf{W}_{proj}^{(l)}$:

$$ $\mathbf{h}_{i}^{(l)}=\mathbf{W}_{proj}^{(l)}\sigma\left(\mathbf{W}_{fc}^{(l)} \mathbf{h}_{i}^{(l-1)}\right)$ $$

where $\sigma(\cdot)$ denotes the activation function.

To modify the factual association $(x_{i},r,x_{j})$ in the model, ROME computes a new key-value pair $(\mathbf{k}^{*},\mathbf{v}^{*})$, representing the entity $x_{i}$ and the new target entity $x_{j}^{*}$. ROME then applies a rank-one update to the weight matrix $\mathbf{W}_{proj}^{(l^{*})}$ at a specific layer $l^{*}$ to encode this new fact:

$$ $\hat{\mathbf{W}}_{proj}^{(l^{*})}=\mathbf{W}_{proj}^{(l^{*})}+\lambda\left( \mathbf{C}^{-1}\mathbf{k}^{*}\right)^{\top}\\ \text{ where }\lambda=\frac{\mathbf{v}^{*}-\mathbf{W}_{proj}^{(l^{*})}\mathbf{ k}^{*}}{\left(\mathbf{C}^{-1}\mathbf{k}^{*}\right)^{\top}\mathbf{k}^{*}}$ $$

Here, $\mathbf{C}$ is the uncentered covariance matrix of the key vectors $\mathbf{k}$, estimated by sampling tokens from a representative dataset.

The key vector $\mathbf{k}^{*}$ corresponds to the entity $x_{i}$ in the factual association $(x_{i},r,x_{j}^{*})$. The vector is computed by averaging the MLP output for $x_{i}$ over multiple randomly generated contexts:

$$ $\mathbf{k}^{*}=\frac{1}{N}\sum_{j=1}^{N}\sigma\left(\mathbf{W}_{fc}^{(l^{*})} \gamma\left(\mathbf{a}^{(l^{*})}_{i}+\mathbf{h}_{i}^{(l-1)}\right)\right)$ $$

where $\gamma(\cdot)$ is a normalization function, and $\mathbf{a}_{i}^{(l^{*})}$ is the attention output at layer $l^{*}$.

The value vector $\mathbf{v}^{*}$ is optimized to maximize the model’s probability of predicting the target entity $x_{j}^{*}$ given the subject $x_{i}$ and relation $r$. This is done by minimizing the following objective:

$$ $L(\mathbf{z})=\frac{1}{N}\sum_{j=1}^{N}\left(-\log P\left(x_{j}^{*}|x_{i},r \right)+D_{KL}\left(P_{G}\left(x_{i}|p^{\prime}\right)||P_{G}\left(x_{i}|p^{ \prime}\right)\right)\right)$ $$

The first term maximizes the probability of the target entity $x_{j}^{*}$, while the second term controls for “essence drift” to retain information about $x_{i}$. This is done by sampling inputs $p^{\prime}$ for which the model’s outputs should not change during the edit.

### D.2 Implementation

In our implementation of ROME tailored to our model, we apply the edit at layer $1$ as it is the only available early-site layer in our model configuration. The covariance matrix $\mathbf{C}$ is estimated by randomly sampling $10^{5}$ inputs from the validation dataset. This provides a representative set of key vectors for computing the rank-one update. To solve for the key vector $\mathbf{k}^{*}$, we sample $10^{5}$ random context sequences, with sequence lengths varying between $2$ and $10$ tokens. The value solver follows a similar procedure by sampling $10^{2}$ context sequences selected in the same manner as the key solver. The value optimization is performed using the Adam optimizer, with hyperparameters $\text{lr}=10^{-3}$ and $\text{weight decay}=10^{-4}$. The value solver optimizes between $5$ and $500$ iterations, stopping when the predicted token is replaced by $x_{j}^{*}$. The KL divergence weight is set to $3$ during optimization.

## Appendix E Visualization Methods

In Fig. 4, we demonstrated the emergence of cyclic representations within the model by extracting embeddings and generating 3D Isomap projections. While the visualizations support the notion that cyclical representations are present in the model, changes in the projections can be difficult to intuitively interpret due to the overlap of differently colored segments of the manifold. For example, below is a recreation of Fig. 6 using raw Isomap projections.

Figure: Figure 9: An equivalent version of Fig. 6 using the unprocessed Isomap projection renderings. Representation shattering is still visible in the flattening and clustering of points in the manifold as the counterfactual edit distance increases.
Refer to caption: x12.png

The coinciding ring segments are an artifact of the lossy projection of high-dimensional cyclical representations into a low-dimensional space: when dimensionality reduction to 3D is applied, the high-dimensional cyclical structure gets “squished” into a torus. To enhance the visual perceptibility of the representation shattering phenomenon, we additionally implement a pre-processing step to constrain the construction of the Isomap neighbors graph using the model’s output predictions. More concretely, when visualizing the post-edit manifold for a particular edit $(x_{i},r,x_{j}^{*})$, we adopt the following procedure:

- 1.
Construct a set $S_{0}$ of entities by prompting the unedited model for all immediate neighbors of $x_{i}$ in the cycle order of $r$ (i.e. by getting outputs for $x_{i}r^{\prime}$ for all $r^{\prime}$ in the same cycle order as $r$).
- 2.
Apply the knowledge edit.
- 3.
Construct a set $S_{1}$ of entities by collecting outputs from the edited model for all $s_{i}r$ where $s_{i}\in S_{0}$.
- 4.
Constrain the Isomap pair-wise distance matrix to members of $S_{1}$.

This procedure remains faithful in comparing the pre-edit model to the post-edit model, as relies solely on model predictions and does not introduce any ground-truth priors.

## Appendix F Additional Results

### F.1 Independence of Subgraphs

In our evaluations, we make edits to various relations under the assumption that the Transformer internalizes the independence of the cyclic orders (I, II, and III). Here, we ask: do the model’s internal representations truly reflect this? We answer this question by inspecting the embeddings for the output of the multi-head attention output in layer $2$ at the last token position using PCA. Unlike in previous sections where we focused on a fixed relation $r$ and varied $x_{i}$ for inputs of the form $\cdots x_{i}r$, we now vary both $x_{i}$ and $r$ and color-code each projection by the cyclic order to which the relation $r$ belongs. We present the resulting projections in Fig. 10.

Figure: Figure 10: PCA of embeddings extracted from the output of the multi-head attention output in layer $2$ at the last token position, color-coded by the cyclic order of the last relation token.
Refer to caption: x13.png

Indeed, we find that prompts eliciting knowledge for each cyclic order are clustered closely together in the latent space—this is further evidence that the model internalizes the properties of the underlying knowledge graph.

### F.2 Manifolds for All Relations

Figure: Figure 11: Isomap projections for embeddings for all relations, extracted from the output of the multi-head attention output in layer $2$ at the last token position. All relations yield a cyclical representation manifold.
Refer to caption: x14.png

### F.3 Manifolds for Various Embedding Extraction Points

Figure: Figure 12: 3D Isomap projections for embeddings extracted from various points in the model for various token positions. The cyclical representation manifolds can only be observed for the last relation token position ($-1$th token), and not at the last entity token position ($-2$th token). This intuitively makes sense because the last relation token informs the model about which cycle order the current input is querying for. We primarily use the “attn2 last relation token” embeddings throughout this work because it is the earliest point at which a well-structured cyclical manifold can be observed beyond the point of the ROME intervention (which is at “mlp1 cproj”).
Refer to caption: x15.png

### F.4 Counterfactual Editing

#### F.4.1 Distribution of Degredations for Counterfactual Edits

The plots below correspond to the counterfactual editing results presented in Sec. 4.4 and Tab. 1.

Figure: Figure 13: Distribution of post-edit accuracy degredations for direct recall, logical inference, and compositional inference in relation to the counterfactual edit distances. A significant shift can be observed between CE distances of 1 and 2, showing the point at which detrimental representation shattering can occur.
Refer to caption: x16.png

#### F.4.2 Additional Visualizations

Figure: Figure 14: Counterfactual editing visualizations for 1623.I_A2.
Refer to caption: x19.png

Figure: Figure 15: Counterfactual editing visualizations for 1121.II_C1.
Refer to caption: x21.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: see weaknesses
soundness: 2
strengths: - **Insight into Adverse Effects of KE:** With knowledge editing (KE) methods gaining popularity, this paper provides valuable insights into why KE can lead to adverse effects, offering a deeper understanding of its impact on model representations.

- **Clarity and Accessibility:** The paper is well-written and accessible, making complex concepts easier to understand for a broad audience.

- **Novel Toy Knowledge Graphs:** The authors introduce innovative toy knowledge graph (KG) tasks to explore KE and the phenomenon of representation shattering, providing a structured and interpretable framework for analyzing KE effects.
summary: The paper introduces the Representation Shattering Hypothesis, which posits that Knowledge Editing (KE) methods distort a model's latent representations, potentially to the point of losing the learned global graph structure—a phenomenon they call "representation shattering." The authors hypothesize that this mechanism underlies KE's negative impact on models' factual and reasoning abilities.

To test this hypothesis, they first create a toy knowledge graph to pre-train a language model on. Then, using a popular KE method called ROME, they edit facts within this graph, demonstrating that the extent of representation distortion correlates with how significantly an edit disrupts the graph's structure.

The authors further validate their hypothesis with a real language model, GPT2, by using ROME to corrupt the order of the months of the year. They show that this corruption distorts the ring structure of the month representations, with greater distortions occurring for larger edit distances.
weaknesses: ### Weaknesses

- **Limited Scope of Experimentation:**
  - The authors primarily experiment on knowledge graphs (KGs) with a ring-like structure, which is a somewhat artificial setup. Even the experiment with a real KG is restricted to a small subset with a similar ring configuration, limiting the generalizability of the findings.
  - Only one knowledge editing (KE) technique, ROME, is tested, leaving it unclear how the findings might extend to other KE methods.

- **Reliance on Visual Inspection of Representation Shattering:**
  - The approach seems heavily dependent on visually inspecting the breakdown of representations, which becomes infeasible for more complex, unvisualizable data. Although the authors propose using the Frobenius norm as a quantitative measure of shattering, it is only applied in one experiment. 

### Suggestions for Improvement

- Extend the synthetic setup to incorporate small, realistic KGs to further validate the hypothesis.
- Experiment with additional KE methods to assess whether the representation shattering phenomenon holds across different approaches.
- Rely more consistently on quantitative measures (like maybe the Frobenius norm) to gauge representation shattering, especially in cases where visual inspection may not be feasible.

---

confidence: 3
contribution: 4
initial_rating: 5
presentation: 2
questions: 1. **Possibility of Studying Geometry on Real-World KGs**: As previously raised in W1, would it be feasible to apply the same methodology to real-world KGs to examine how the representation geometry shatters under KEs? Would the lack of cyclic orders make this significantly more challenging? Additionally, could the Isomap visualization technique still be used effectively in this scenario to visualize a tree-like structure rather than rings?
2. **Training Setup Clarification**: Are all edit, retain, and test relations used during the pre-training of the transformer, and the test relations are only held out specifically for the KE process but they are all observed by the transformer during pre-training? If so, a clarification near line 213 would be helpful.
3. **Logical Inference Task**: For the logical inference task, could you elaborate on how the "hold out" process works? For example, if the inverse relation (say "advisee") is held out and unobserved during training, how can the model know that the held-out relation token "advisee" is the inverse of the observed relation "advisor"?
soundness: 2
strengths: 1. **Novelty**: This paper is the first to delve deeply into transformers' representations, providing a mechanistic understanding of why the detrimental impacts of KE occur. In contrast, previous work mainly treats transformers and LLMs as black boxes, focusing on observing the degradation of models' overall performance in targeted tasks.
2. **Unique Approach and Insightful Perspective**: The authors tackle the problem from the perspective of studying representation geometry, which offers a unique and insightful angle into how KE affects model consistency.
3. **Preliminary Results on LLM**: The authors extend their findings to a pretrained large model, GPT-2-XL, showing that the representation shattering phenomenon also applies to LLMs, not just to transformers trained on synthetic data.
summary: The key idea of the paper is that knowledge editing (KE) alters the geometry of transformer models' representations, making them inconsistent or "shattered." The authors introduce the concept of "representation shattering" to explain why KE results in degradation of the model's performance. This perspective of geometric shattering is both insightful and novel, providing a deeper understanding of how targeted edits can inadvertently disrupt broader model behavior. The study is conducted using synthetic knowledge graphs (KGs), which allows the authors to identify specific geometric changes in transformer representations.
weaknesses: Although the perspective and hypothesis of this paper are novel and insightful, I have concerns regarding the limitations of the experimental setup, specifically the reliance on cyclic structure in the synthetic KGs and the focus on only one KE method.

1. **Why Cyclic Graphs?** The key assumption of this study is that transformers can learn a cyclic geometric representation when the training KGs have cyclic relation subgraphs. This assumption on its own makes sense. However, to my knowledge, real-world KGs often do not have relation subgraphs that conform to cyclic orders. For example, take the advisor relation that is used as the running example in the paper. It is unlikely that "Alice is the advisor of Bob", "Bob is the advisor of Charlie", and "Charlie is the advisor of Alice." In reality, most KG relations seem to form trees instead of cycles. This makes the study somewhat disconnected from real-world scenarios as cyclic relations are uncommon. Why not investigate the geometry of transformer representations in real-world KGs (or a semi-synthetic KG that more closely mimics the structure of real-world KGs) when the relation subgraphs are trees, and examine how KEs affect such geometry?
2. **Knowledge Graph Construction**: The construction of the knowledge graphs, particularly the use of cycles, is not very intuitive and is challenging to understand. This is compounded by conflicting information in Section 3.2 (e.g., Line 202 states that the KG is defined to have 3 relations, but Line 205 later refers to 24 relations) and an explanation that omits critical details. Specifically, it is unclear what the exact cyclic orders are for the edit, retain, and test relations, and whether these relations are organized in such a way that each of the three groups has distinct orders. Additionally, Figure 3, intended to provide more intuition and details, is difficult to interpret due to unclear labeling, such as the meaning of different arrow colors and which edges were edited. Clarifying why the authors chose this specific construction, particularly the necessity of the use of different cyclic orders, and providing more transparent explanations of the data construction would greatly improve the reader's ability to understand the motivation behind the experimental setup.
3. **Single KE Method (ROME)**: The paper only studies one knowledge editing method, ROME. The findings would be significantly strengthened if multiple KE methods were analyzed to determine whether the representation shattering phenomenon is a general property of transformer models occurring on a more fundamental representation level, independent of the KE method used. For instance, the paper shows that shattering occurs even for corrective edits, which is surprising because a perfect corrective edit should ideally refine the model's representation geometry. It is thus crucial to investigate whether this surprising phenomenon is specific to ROME or is a broader theme across different KE methods.
4. **Lack of Mitigation Strategies**: While the paper does an excellent job of identifying the issue of representation shattering, it lacks discussion on potential ways to mitigate this problem. Including some hypotheses or suggestions for future work would provide direction for researchers interested in addressing this issue.

In addition, some other comments on readability and presentation:

5. **Figures**: The Figure 1 is somewhat confusing, particularly in panel (b). It is unclear how the ring structure is represented in the visualization, and what the labels "182" and "608" refer to. Additionally, Figure 1 is not mentioned in the introduction, which makes it difficult to understand its relevance early on. It would help if the illustration was used to convey the high-level intuition of the findings. Similarly for Figure 3, mentioned in W2.
6. **Undefined Terms**: The abbreviation "DGP" (Figure 1 caption, line 67) is undefined. I assume it stands for "Data Generating Process"—clarifying this would improve readability.
7. **Entity and Relation Naming**: The naming conventions for entities and relations are unclear. For instance, on line 235, it is not immediately evident that "1," "3," etc., are entities, and that "I_C4," "III_A2," etc., are relations. In addition, there are clues in Figure 3 that the number of "I"'s stand for the different cyclic order, but this is not mentioned and explained in the main text. An explanation of the naming convention and its significance (e.g., whether the number of "I"s corresponds to hops in the KG) would be helpful.
8. **Typo**: On line 291, the phrase "it changes fact $(x_i l r, x_j)$" should probably be "it changes fact $(x_i, r, x_j)$"
9. **Unclear Sentence**: On line 366, the caption of Figure 5 reads: "We plot the mean drop in accuracy against the in the Frobenius norm of the difference in..." This sentence does not make sense as written and needs to be revised.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: * Is the assumption still valid on different model architectures, including encoder-decoder models like T5, and Mamba?
* Is the assumption still valid on larger LLM models like Lamma, Gemma, Phi 3.5 etc?
soundness: 2
strengths: * A new hypothesis, representation scattering,  is proposed
* Experiments are done to test the hypothesis proposed,
summary: This paper provides a way to investigate the representation scattering hypothesis during knowledge editing. It proposed a test for using synthetic entities to probe if the representation scattering happens.
weaknesses: * Only the GPT model is studied, which makes the scope limited in the model architecture.
* The result seems hard to reproduce.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please refer to the questions mentioned in the Weaknesses part.
soundness: 2
strengths: 1, The synthetic dataset is well-designed in a way that all the relations are related in a clear semantic meaning. And Figure 3 clealy demonstrates the structure of the synthetic dataset.

2, The visualization is really good. It clearly shows the representation shattering after the knowledge editing (Figure 4, 5, 6 and 8).
summary: The paper constructs a synthetic dataset with cyclic knowledge graphs and investigates the influences of knowledge editing on the synthetic datasets. The results show that KE inadvertently affects representations of entities beyond the targeted one, distorting relevant structures that allow a model to infer unseen knowledge about an entity.
weaknesses: 1, The expected answers (or the ground truth answers) after knowledge editing is not clearly stated in the paper. For example in Figure 3 Ring I, after the editing of $1.I\\_C2 = 3$, is $1.I\\_C2 = 2$ still true (so that entity 1 now have 2 entities for relation I_C2) or does this edge simply disappear? Which one of the following is considered the ground truth for evaluation after the editing: $1.I\\_C3 = 4$ or $1.I\\_C4 = 4$? In the proposed synthetic dataset, it could still have a consistent logic for LLMs even after knowledge editing. The paper should make the true facts after the knowledge editing more clear.

2, The prompt templates and codes are not released. How the query and the structural information of KGs are provided in the prompt will have a big influence on the final experiment results. Can the authors provide them during the rebuttal? And can authors show some examples of the LLM outputs in natural languages after knowledge editing to help readers better understand the mistakes LLMs make?

3, More knowledge editing baselines should be included. In this paper ROME is the mainly considered knowledge editing methods. There are other 'locate and edit' methods such as PMET[1]. And there are other types of knowledge editing with extra parameters such as GRACE[2]. Including more knowledge editing methods will make the 'representation shattering' hypothesis more convincing.

4, Some typos need to be fixed. For example, $(x_i, a, x_j)$ should be  $(x_i, r, x_j)$ in line 191. Why is “1 IC_4 3 IIIA_2 7 IIIA_3 2 IIC_2 6” in line 235-236 a plausible sequence in figure 3? $(x_i lr, x_j)$ is not clear in line 291.

[1] Li, Xiaopeng, et al. "Pmet: Precise model editing in a transformer." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024.

[2] Hartvigsen, Tom, et al. "Aging with grace: Lifelong model editing with discrete key-value adaptors." Advances in Neural Information Processing Systems 36 (2024).

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 2
questions: Could you elaborate on the synthetic data generation process? Do A/B/C refer to the entities and the numbers refer to the edges?
soundness: 3
strengths: - The ripple effect of knowledge edit has been studied, but the prior works have not explained the effects from a mechanistic viewpoint. This paper takes a nice step toward explaining this effect mechanistically.
- The angle toward explaining this ripple edit effect is interesting (despite a bit niche). Cyclic structure is a commonly occurring structure in natural language, and there is a nice illustration in the representation space.
- This paper uses a nice combination of qualitative and quantitative illustrations to describe the representation shattering effect.
summary: This paper presents a mechanistic explanation of the ripple effects of knowledge editing. This explanation considers how the representations of the model changes during the KE. This paper considers a particular cyclic structure. When the model’s training data are generated from this cyclic structure, the learned representation gradually forms a geometry that resembles the cyclic structure of the data. KE, both corrective edits and counterfactual edits, shatter this cyclic structure. In this way, the ripple effect change in the model’s behavior after edits can be illustrated by the change in the representations.
weaknesses: - The quantitative explanation part can take up more space in the writing. Currently only Figure 5(c) involves quantitative description of representation shattering, and other parts are qualitative. Table 1 presents the overall effects of knowledge editing, which is not really the quantitative descriptions.
- Additionally, the definition for describing the representation shattering effect should be introduced earlier. Currently it’s hidden at lines 413-415 at page 8, closer to the end of the paper. This should be moved to the front.
- Subsection 3.3 can be made clearer. Based on the example presented, I struggled at understanding how the nodes are connected, and e.g., what I/III/ABC refer to.
- The effects are tested on a 2-layer nanoGPT Transformer, plus a GPT2. These limit the generalizability of the finding. I’d recommend experimenting on at least two models on each size — perhaps containing a mixture of uni-directional models (like *GPT) and bidirectional models (like BERT).

Meta-review: This paper presents a new hypothesis called "Representation Shattering" to explain performance degradation in knowledge editing approaches. The research suggests that knowledge editing can distort a model's latent representations and potentially destroy learned global graph structures. The severity of this distortion appears to correlate with how significantly an edit disrupts the underlying structure, as demonstrated through experiments with both synthetic knowledge graphs and real language models.

The research makes valuable contributions through its novel theoretical framework, which provides a mechanistic explanation for knowledge editing's negative effects rather than treating transformers as black boxes. The methodology is particularly innovative, employing synthetic knowledge graphs for controlled investigation of how edits affect representation geometry. The findings are supported through both qualitative visualization and quantitative metrics, with the results extending from synthetic settings to real models, suggesting this phenomenon may be fundamental to transformer architectures.

However, the research faces several limitations in its current form. The experimental scope is notably restricted, focusing primarily on cyclic structures in knowledge graphs and testing with a single editing method. The methodology heavily relies on visual inspection of representation shattering, with unclear ground truth specifications after editing and insufficient quantitative metrics for measuring distortion. Additionally, the presentation could be improved, particularly in explaining knowledge graph construction and defining key concepts earlier in the paper.

The paper would benefit from broader validation across multiple editing methods and diverse model architectures, along with investigation of non-cyclic knowledge structures. Technical improvements should include more rigorous quantitative metrics for measuring representation shattering, clearer specification of ground truth post-editing, and comprehensive code release for reproducibility. The presentation would be enhanced by introducing key definitions earlier and improving figure clarity and labeling.

While the core concept shows promise and could significantly impact the field, the current submission requires substantial revisions to address these methodological limitations and strengthen the empirical validation of the hypothesis. The recommendation is to reject the paper in its current form, allowing for these necessary improvements to be made before reconsideration.

Decision: Reject