Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing
Authors: Kento Nishi
Rahul Ramesh
Maya Okawa
Mikail Khona
Hidenori Tanaka
Ekdeep Singh Lubana
Venue: ICML 2025 poster (2025)
Keywords: mechanistic interpretability
knowledge editing
transformers
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: Our work introduces a synthetic framework which enables for a deeper understanding of Knowledge Editing and its side effects with regards to model representations.

Abstract: Knowledge Editing (KE) algorithms alter models' weights to perform targeted updates to incorrect, outdated, or otherwise unwanted factual associations. However, recent work has shown that applying KE can adversely affect models' broader factual recall accuracy and diminish their reasoning abilities. Although these studies give insights into the potential harms of KE algorithms, e.g., performance evaluations on benchmarks, little is understood about why such destructive failures occur. Motivated by this, we define a novel synthetic task in which a Transformer is trained from scratch to internalize a "structured" knowledge graph. The structure enforces relationships between entities of the graph, such that editing a factual association has "trickling effects" on other entities (e.g., altering X's parent is Y to Z affects who X's siblings' parent is). Through evaluations of edited models on this task, we show that KE inadvertently affects representations of entities beyond the targeted one, distorting relevant structures that allow a model to infer unseen knowledge about an entity. We call this phenomenon representation shattering and demonstrate that it degrades models' factual recall and reasoning performance. We further corroborate our findings in naturalistic settings with pre-trained Llama and Mamba models as well. Overall, our work yields a precise mechanistic hypothesis to explain why KE has adverse effects on model abilities.

Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing
Kento Nishi 12 Rahul Ramesh" 4 Maya Okawa 23 Mikail Khona
Hidenori Tanaka" 23
Ekdeep Singh Lubana"
Abstract
Who the current president of US?
(a)
Knowledge Editing (KE) algorithms alter models' weights to perform targeted updates to incorrect, outdated, or otherwise unwanted factual associa- tions. However, recent work has shown that ap- plying KE adversely affect models' broader factual recall accuracy and diminish their reason- ing abilities. Although these studies give insights into the potential harms of KE algorithms, c.g., performance evaluations on benchmarks, little understood about why such destructive failures occur. Motivated by this. we define a novel syn- thetic task in which a Transformer is trained from scratch internalize "structured" knowledge graph. The structure enforces relationships be- tween entities of the graph, such that editing a factual association has "trickling effects" on other entities (e.g., altering X's parent is Y to 2 affects who x's siblings' parent is). Through evaluations of edited models this task, we show that KE inadvertently affects representations of entities beyond the targeted one, distorting relevant struc- tures that allow a model to infer unseen knowl- edge about an entity. We call this phenomenon representation shattering and demonstrate that it degrades models' factual recall and reasoning per- formance. We further corroborate our findings in naturalistic settings with pre-trained Llama and Mamba models as well. Overall, our work yields precise mechanistic hypothesis to explain why KE has adverse effects on model abilities.
Before Editing
After Editing
3
Per Performance Before Editing
Performance After Ecitinng
fo 2.0.
Model Editing
fy Q.0
0.8
onoga Cirtoro
0.4
cbtu pooo
0.2
Quartor Anwwnnng
Dalegue
Informator Eerration
Santiwut
Aralysh
Trump
Trump
Biden
Joe
Which entity is to the right of N123?
(b)
Performarce Before Editing
Performance After Editing
Before Editing
After Editing
0.
63R
072
0.2
Logcal
Compoatioral
Entiyy Entiy #124
Entity Entity A124
Racal
Figure 1. Representation shattering as a mechanistie hypothesis for why knowledge editing has adverse effects on models' gen- eral capabilities. (a) Prior works find that editing facts, c.g., the president of the US can harm general abilities LLMs (figure reproduced from Gu et al. (2024)), (b) We introduce synthetic data generation process (DGP) defined by a knowledge graph con- taining ring-shaped geometries. Training model on this data, we observe that the model's 's internal representations mirror the ring structure of the underlying DGP. We explain the post-edit degra- dation of the model's capabilities by uncovering the "shattering" of latent representations. E.g., in the provided illustration, while the edit successfully changes the fact (Entity #124 is to the right of Entity #123), it also degrades model's broader performance.
1. Introduction
'Equal contribution 'Harvard College CCBS-NTT Program in Physies of Intelligence, Harvard University 'Physics and In- formatics Lab, NTT Research Inc. 'Computer and Information Science, University of Pennsylvania 'Department of Physics, Massachusetts Institute of Technology. Correspondence to: Kento Nishi <kentonishi@colle wet.cclle..aaaar..ddll... harvard.edu>>, Ekdeep Singh Lubana <ekdeeplubana @ haraardaadddoo edux,, Hidenori Tanaka <hi- denori_tanaka @ fas edu>..
Large language models (LLMs) have led to unprecedented advances several domains (Gemini Team. 2023; Bubeck et al., 2023; Touvron et al., 2023; Thoppilan et al., 2022; Chowdhery et al., 2022; Qin al., 2023; Chen et al., 2021; Ahn et al., 2022; Driess et al., 2023). However, the static nature of their training pipelines implies that as our world evolves, models' internalized knowledge can become incor- rect or outdated. To address this, recent work has proposed several protocols for knowledge editing (KE), wherein the
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Representation Shattering in Transformers with Knowledge Editing
goal is to minimally and precisely alter model weights such that only the targeted information (and its relevant associations) are updated, but all unrelated information re- mains (ideally) unaffected (Mitchell et al., 2022; Meng et al., 2022a; 2023; Dai et al., 2021; Cheng et al., 2023; De Cao et al., 2021; Sinitsin et al., 2020).
knowledge graph. Performing traversal over the nodes of this knowledge graph, we get sequences that can he used as "strings" to train Transformer on. As we show, this proto- col leads to the model precisely encoding the structure of the graph in its latent representations. However, when KE is applied to edit either incorrectly learned facts or insert coun- terfactual knowledge (using the method proposed by Meng et al. (2022a)), we find latent representations are heavily distorted and the graph structure completely destroyed- we call this phenomenon representation shattering. Inter- estingly, this phenomenon manifests in proportion to how far the proposed edit moves a given node from its current location to a new location in the graph (defined via edge distance). We thus hypothesize representation shattering un- derlies the detrimental effects of KE on a pretrained model's factual and reasoning capabilities at broad. Overall, we make the following contributions in this work.
Despite significant work on the topic, it still remains unclear precisely what effects KE should have on model. For example, assume you edit the fact that "Michael Jordan won the 1998 NBA MVP" to "Reggie Miller won the 1998 NBA MVP", then what should the impact of such an edit be? Should the model now believe Michael Jordan and the Chicago Bulls never reached the NBA finals in 1 1998? Should it perhaps believe Reggie Miller was on the Chicago Bulls? Should the pop quote "Be like Mike" (Wikipedia, 2024) now become "Be like Reggie"? As Hofweber et al. (2024); Hase et al. (2024) argue, it is difficult to design clear, well-defined answers for such questions. Motivated by this, recent work has started investigating precisely what effects KE actually has on the model (Hoelscher-Obermaicr Obermaier et al., 2023; Li et al., 2023b; Lynch et al., 2024). For example, Cohen et al. (2023) demonstrate that knowledge beyond the edited fact can often be impacted such that the model begins to have an incoherent understanding of the world; Gupta et al. (2024a) demonstrate unrelated facts are often forgotten by the model post-editing: and Gu et al. (2024) show that KE can harm broader reasoning capabilities beyond mere factual recall. While these works clearly demonstrate the detrimental impacts of editing on a model, they still leave open the question precisely why such harms occur-at mechanistic level, how are model representations impacted such that broad set of knowledge and capabilities in model are heavily distorted once an edit occurs?
. Structured Knowledge Graphs as Toy Setting for Investigating Impact of KE. We propose use of a struc- tured knowledge graph wherein entities (nodes) are con- nected to each other via predefined local constraints (rela- tions) that manifest into a broader, global structure in the graph (see Sec. 3). Training Transformers on strings (path traversals) from the graph, we find model representations precisely encode the global structure of the graph. This allows us to assess the impact of KE at a more mecha- nistic level, since distorting a fact now has global effects that can be precisely (and, in fact, visually) delineated by analyzing the model representations.
Representation Shattering as a Mechanistic Hypothe- sis to Explain Detrimental Effects of KE. We find KE distorts latent representations for entities in the graph such that the global geometry learned during pretraining is, at times, completely destroyed- ww call this phenomenon representation shattering and hypothesize it underlies the detrimental effects of KE on model capabilities ob- served in prior work (see See. and Fig. 1). As we show, the extent of harm on latent representations turns out to be correlated to the amount edit alters the graph from its original organization into the new, desired one.
This work. To address the questions above, we aim to de- velop a mechanistic understanding of the impact of KE on a model's ss internals. For this purpose, we argue we must solve two problems: (i) identify how a model expresses knowl- edge about some predefined set of entities in its representa- tions, and (ii) investigate how this mechanism is affected as we apply KE to alter a fact corresponding to subset of the entities. Instead of attacking a complicated system that may be difficult to interpret (e.g., an off-the-shelf LLM), we take inspiration from a multitude of recent papers that establish synthetic abstractions of the target system and develop pre- cise hypotheses as to why the phenomenon-in -nneennon---uuutiionnn oc- curs (Allen-Zhu& & Li, 2023c;a;b; Okawa et al., 2023; Chan et al., 2022; Li et al., 2023a; Lubana et al., 2024). Specifi- cally, we define a data- dlta-ggeeerainng process that yields entities arranged in a structured knowledge graph. This structure is defined via use of a predefined set of relations that locally constrain how entities relate to each other (similar to parent- child relations). Given enough entities and relations, such local constraints manifest broader global structure in the
Investigations with Off-the-Shelf LLMs. Using pre- trained Llama and Mamba models, we provide evidence for our claims about representation shattering in more nat- uralistic settings. For one, we find real-world analogues to our synthetic knowledge graph structures (i.e., days of the week) yield similar shattering phenomena in Llama and Mamba models to what we observe in our toy setup (see Sec. 4.5). Additionally, we further reinforce the gen- erality of our findings with preliminary replications of representation shattering under more complex knowledge graph geometries, such as trees (i.e. countries and their cities, Appx. H.3).
2

---

Representation Shattering in Transformers with Knowledge Editing
2. Related Work
3. Formalzing Knowledge Editing
Knowledge Editing. Several protocols for knowledge edit- ing (KE) have been proposed in recent work. Early work de- fined meta-learning based approaches (Sinitsin al., 2020; De Cao et al., 2021; Mitchell et al., 2022) and established the broader desiderata for what properties a KE protocol should satisfy; e.g., ensuring facts unrelated to the target one are not hampered via the editing protocol. Building on work aimed at understanding how Transformers encode knowledge in their internals (Geva et al., 2020), modern KE protocols focus on defining closed-form operations that involve (i) localizing knowledge to specific components in model (e.g.. MLP layers) and (ii) defining operations to alter a factual association by assuming the fact is represented in a localized manner (Meng et al., 2022a; 2023).
Epistemology has grappled with the nature of knowledge for centuries (Chappell, 2005). In this work, we adopt humble yet precise definition of knowledge based on structured knowledge graphs. A knowledge graph is used to represent how facts, entities, and relations are interlinked, giving rise to notions of consistency, coherency, and reasoning across different pieces of information. Using these definitions, we will define a synthetic data generation process on knowledge graphs that enables a systematic study of knowledge editing in Transformers.
3.1. Knowledge Graphs
A knowledge graph consists of a collection of entities X = {a: Vi=,, and a collection of facts F that relate different entities, For example, a graph defined on entities X = ("Alice", "Bob", "Carol") can encode the fact "Alice is the advisor of Bob" using the relation "advisor", represented as ("Alice", "advisor", "Bob").
Evaluations of Knowledge Editing Methods. As argued by Hase et al. (2024); Hofweber et al. (2024), the problem of KE is relatively ill-defined. Consequently, it unclear that when we edit knowledge within a model, what effects said edits should have on other facts it may have internalized during training. Prior work has hence taken an alternative approach, primarily focusing on developing an empirical understanding of what the phenomenology of KE protocols is: e.g., if an edit is performed, how are counterfactual state- ments or unrelated facts affected. These works generally show that KE in fact has extreme detrimental effects on model, e.g., hampering both its broader internalized knowl- edge and its reasoning abilities (Hase et al., 2023; Cohen et al., 2023; Hoelscher-Obenee
Detinition 3.1 (Knowledge graph). Formally, knowledge graph G = (X, R, F) consists of nodes X, relations R. and facts F, where each fact f = (zz, T, T,z)) E F is defined by a relation r E R between two entities I4, 23 E X.
A relation sub-graph corresponds to sub-graph con- structed by only considering facts that use relation r. For example. Gadvisor is a sub-graph that specifies all facts for the relation "advisor". Every knowledge graph contains collection of facts that can be inferred from the graph.
et al., 2023; Gupta et al., 2024a; Gu et al., 2024). While the primary methodology used such papers is to perform empirical benchmarking of a model that has undergone editing, we instead focus on mechanistic analysis of how editing alters a model's representations (albeit primarily in toy synthetic task) to yield the undesirable effects on model abilities.
Related pieces of information such as "Alice's 's advisor was Bob" and "Bob's advisor was Carol" can be composed to form cohesive statements such as "Alice's advisor's advisor was Carol. To capture such statements, we define composi- tions of relations. The composition of relations are essential to capture ripple effects that occur in the knowledge graph after an edit (Cohen et al., 2023) to relation in R.
Explaining Models via Synthetic Tasks, To disentangle the failures of KE methods from the failures of the mod- els themselves, we argue for use of more controllable and interpretable setup. Such setup can help identify a concrete hypothesis for why KE has undesirable effects on the model, which we can then analyze in naturalistic settings by designing more precisely defined experiments. This methodology of designing toy, control tasks to inves- tigate hypotheses for phenomenology of a neural network has yielded promising results in recent years, providing, e.g., a concrete hypothesis for how chain-of thouhht reason- ing aids model capabilities (Prystawski et al., 2024; Feng et al., 2023), models for emergent capabilities (Okawa et al., 2023; Lubana et al., 2024), existence of multidimensional representations (Engels et al., 2024), and failure modes for compositional generalization (Zhou et al., 2023).
Definition 3.2 (Composition of relations). A composi- tion of relations F = (ri,T2,...,, .rk) E R² with re- spect to knowledge graph G is defined such that for ev- ery fact / = (za,, F, ary), there exists a collection of facts ( T., x1+1)) for which 21 x, and It+1 = Iz, In other words, any fact defined on the composition of relations has corresponding set of facts defined on relations from R. Furthermore, the set of facts form path in the knowledge graph such that the sequence of relations in the path are Tı, TTk-
3.2. Cyclic Graphs: Description of Entities/Relations
We mainly study knowledge graphs where every relation sub-graph is a set of disjoint eyclic graphs, i.e., for every entity x, and relation r. there exists exactly one entity æ such that (z,, r, 2z) E F. We specifically choose a cyclic
3

---

Representation Shattering in Transformers with Knowledge Editing
hops clockwise from entity 1 in cyclic order is entity 2"
Mj
12_c2
Mar
%
Syn
subt-shh:: Cyclic Order
May
1/ = 3-22
11 =
Edit knowledge
Past
Sat
Jul
Aug
A2
Dusanoo 1
Figure 2. Isomap projections of representations in Llama-3.1- 405B FFiotto-Kaffmnn et al., 2024). The geometry of data- for example, the cyclic nature of months days-is often reflected the representations learned by language models. Similar repre- sentations can also be found in other models like GPT-2-Small and Mistral 7B (Engels et al., 2024).
Retain sub-graph: Cyclic Order
LILGI
Beban knowlsdge
1/1 A2
Test sub- sub-graph: Cyclic Order
Hasarved for eval
1.11
C2
geometry as global constraint on the graph structure since cycles are a common pattern that relate entities in natural language domains; e.g., see Fig. 2, where we show a 2D projection of representations from Llama-3. 1-00BB corre- sponding to months of a year and days of the week naturally organize in a cyclic fashion. For supplementary explorations of geometries of concepts other than cycles (i.e. trees), refer to Appx. H.3.
1.11
A1
8
1.11
A2
Randomly trace sequence of connected entitics through the three cyclic orders:
L X2 2 8 ILC2IC111 Compose rolations!
Hold out some relations for sesting!
Figure 3. Synthetic data generation process with a cyclic knowl- edge graph. The entities (nodes) are arranged according to 3 different cyclic orders. Each entity (node) has relations (directed edges) pointing 8 other entities in each cyclic order which totals to 24 relations across all 3 orders. The relations correspond to 1-4 hop neighbors in the clockwise and anti-clockwise directions. We select random path on the knowledge graph using all 24 relations to generate a prompt (shown above). The relations follow the nam- ing convention of (eelic-order)((direeeee the relation corresponding to the three-hop anti-clockwise neigh- bor in the second cyclic order. In cyclic order I, the above figure denotes an edit for a relation between Entity to Entity (red) to relation between Entity 1 to Entity (green). The distance of the edit is 1, as defined with respect to cyclic order I.
Knowledge editing methods, e.g., ROME (Meng et al., 2022a) and MEMIT (Meng et al., 2023), target a set of entities for which predefined facts are to edited, while using another retain set of facts about said entities to help ensure relations beyond the targeted ones are not altered. A test set of facts are then used to evaluate how well the method worked. Motivated by this, we define knowledge graph with 2048 entities (denoted by 1-2048) over which we define 3 cyclic orders (order I, II and III). The cyclic orders are generated using random permutations the en- tities. We create 8 relations for each cyclic order totaling to 24 relations. The 8 relations correspond the 1-hop, 2-hop, 3-hop and 4-hop neighbors in the clockwise and anti-clockwise directions in the cycle. The relations are named after a combination of the cyclic order (I, II. III), the neighbor's 's distance (between 1-4) and the neighbor's 's direction (Clockwise, Anti-clockwise). For instance, the re- lation "I_c2" denotes the 2-hop neighbor in the clockwise direction, with respect cyclic order I. The 1-hop neighbor relation graphs (both clockwise and anti-clockwise) contain single cycle, 2-hop relation graphs consist of 2 cycles, the 3-hop relation graph contains 1 cycle, while the 4-hop rela- tion graph contains 4 cycles. The k-hop neighbor relations are related to each other by design, so any edit to one k-hop relation should be consistent with all other k-hop relations. An edit corresponds changing fact in the knowledge graph and can also be interpreted as changing an edge in the relation graph. For an illustrative example, see Fig. 3.
(hops), i.e. II_A3 is
facts based on these edit relations are then tested to check if a knowledge edit was successful. Meanwhile, the retain sub-graph relations are used by the knowledge editing algorithm to minimize changes to unrelated relations, but no edits are made to facts that use these relations. Finally, the test sub-graph relations are used to define facts that are neither directly edited, nor used by the knowledge edit- ing algorithm. The relations are used to evaluate whether unrelated facts remain unchanged after a knowledge edit. We note that relations for all sub-graphs are seen during pre-training and this distinction between the cyclic orders is made only during model editing. Finally, the distance of an edit (shown in Fig. 3) is defined as the shortest distance between the original and edited entity in the cyclic order.
3.3. Experimental Setup
Data generation process. We generate a sequence of alter- nating entities and relations resembling I 7 Fi22ñzz375 where any consecutive triplet of entity, relation, and en- tity 2,r,T:+1 from the sequence is a fact (z, F, T441) in the knowledge graph. The composition of relations
Depending on the fact being edited, the 3 cyclic orders are used to define the edit sub-graph, the retain sub-graph, and the test sub- graph. We need 3 cyclic orders because knowl- edge editing methods target edit sub-graph relations; the
4

---

Representation Shattering in Transformers with Knowledge Editing
R= -i ... is sequence of or more relation tokens, while zy is single entity token. Every token is sampled using uniform probability over all the permissible choices (see Alg. 1). For example, a plausible sequence for the example in Fig. 3 is "1 I_C4 4 IIT_A2 8 III_A3 3 II.C2 7", which is an alternating sequence of entities and k-hop relations. As previously noted, relations belonging to all three cyclic orders are included in the data genera- tion process; the distinction between edit, retain, and test trained
3.4. Representation Shattering
In this work, we explore the hypothesis that knowledge edit- ing methods distort the geometry of the representations of entities in the knowledge graph. We argue that this distortion can give us insight into why knowledge editing degrades the general capabilities of a model. More precisely, in the following sections, we investigate the following hypothesis. Hypothesis 3.3 (Representation shattering). Language models embed related entities on a manifold in their internal representations. KE methods distort this manifold in order to insert new facts or alter old ones, i.e., they shatter model representations. The extent of representation shattering increases with the distance between the old fact and the desired new fact on the manifold.
relations is only relevant to knowledge editing model. Furthermore, we remark that this sampling process is identical to traversing random walks on the knowledge graph, similar to previous works (Prystawski et al., 2024; Khona et al., 2024). Additional details of the generation process are documented in Appx. A and Appx. B.
Training setup. We train a Transformer model using next- token prediction on the synthetic data generated from the above data generation process. For all experiments (un- less stated otherwise), we use a 2-layer nanoGPT Trans- former (Karpathy, 2021). For additional details about the toy model, see Appx. D. For experiments with real-world LLMs, see Sec. 4.5.
To quantify the extent of representation shattering, we define precise metric to capture the amount of distortion of the representations:
R(D.) = || - -DDlle
(1)
||Dalle
where ||i|| is the Frobenius norm of D. Do is the pair- wise distance matrix of the entities computed using the unedited model, and D. is the pairwise distance matrix computed using the edited model. D. and De are both n x n matrices over n entities, and entry Dig is the Eu- clidean distance between the representation vectors of x; and zy.. The matrices are computed over only entity tokens (relation tokens are excluded).
Evaluation (seen facts). We assess the model's ability to remember facts seen during training, both before and after an edit. Specifically, to check if the model has learned the fact (I,,, F, we prompt it with an entity x; and a relation F, expecting it to produce zj as the next token. In practice, the model outputs can vary across prompts: we account for this by averaging the softmax probabilities across 5 randomly sampled sequences of the form ct.x 2; and using the output token with the highest probability. Herein, ctx I; denotes randomly sampled sequence that ends in z;-
Note that R(D.) is permutation--ensiiee by design: a zero value indicates that every entity token preserves its loca- tion in representation space. Isomorphic geometries (e.g.. wherein representations of two entity tokens swap positions in the manifold) still yield nonzero R(D.) values, because the relative positions of entities (as identified by their re- spective pre-assigned tokens) must have been distorted to achieve the edit(s).
Evaluation (unseen facts). We also evaluate the model on two other criteria. (I) Compositional inference. In addi- tion facts seen in the training data, we evaluate the model on compositions of relations. The model must preserve ge- ometric structures of the data in order to compositionally generalize after knowledge edit. (2) Logical inference. A key feature of reasoning in natural language is logical infer- ence. For example, if Alice is said to be the advisor of Bob, then Bob is an advisee of Alice (even if it is not explicitly stated). Our data generation process has similar relations, such as clockwise and anti-clockwise I-hop neighbors. By "holding out" one direction for some such pairs of relations from being observed verbatim in the training dataset, i.e., the relation may only appear compositionally, we can assess the degree to which the model internalizes properties among related relations. We can also evaluate if editing a fact for a relation changes the fact for other related relations, i.e., we check if the model's knowledge is logically self-consistent after an edit. Precise prompt formats for both (1) and (2) documented in Appx. C.
4. Uncovering Representation Shattering
We study knowledge editing methods like ROME (Meng et al., 2022a), MEMIT (Meng et al., 2022b), PMET (Li et al., 2024), and AlphaEdit (Fang et al., 2024) in this work. While in the main paper we primarily present results with ROME (see Appx. E. for a short primer), we provide results with other methods in Appx. G. and Appx. 1.1.2 We also verify the assumptions made by these methods in Appx. G.2.
We perform two different types of edits: corrective edits and counterfactual edits. Corrective edits are applied to facts which the model recalls incorrectly after training. A counterfactual edit introduces a new fact, i.e., it changes fact (I,, r, 23) to fact (o,, r, r.)) where 2, # Zg.- Such an edit introduces inconsistencies in the knowledge graph.
5

---

Representation Shattering in Transformers with Knowledge Editing
2500
6000
9000
24000
150000
(a) Unedited
(b) Corrective
(c) Counterfactual Distance-d Edit (AAco.)
Cyclic Order
Sub-
Acc.
(AAce.)
Graph
d=1
d=2
d=3
d=4
Direct Recall
I
98.34
Edit
-21.95
-01.49
-67.01
-77.07
-77.94
Train Loss
350
5000
9000
24300
150000
II
93.71
Retain
-22.64
-01.91
-66.70
-75.49
-75.42
6
III
99.37
Test
-21.83
-01.75
-67.00
-76.12
-77.90
I
98.16
Edit
-22.24
-01.44
-67.22
-77.14
-78.02
Logi.
Infer.
II
93.95
Retain
-22.50
-01.83
-66.88
-75.67
-75.67
10
10
10 Train Step (a)
10
III
99.40
Test
-22.03
-01.80
-67.31 -76.27
-78.23
88.15
Edit
-29.60
-05.32
-73.15
-80.35
-80.63
(b)
Comp.
Infer.
II
79.31
Retain
-31.92
-0.332
-71.21
-78.70
-78.87
Figure 4. Transformers learn representations that mirror the geometry of the underlying data. (a) The representations -or oulput of the second attention foy the input xv for different entitics x and fixed relation F are visualized using Isomap. The model learns the cyelic ordering to represent all the facts. For visu- alizations of representations at various other model components and relation prompts. please see Appx. G.4 and Appx. G.S. (b) To improve the visual fidelity of the projected representations when comparing post-edit models the unedited model, we pre-prceess the Isomap neighborhood graphs using the outputs of the Trans- former model. For more details, see Appx. F.
III
93.50
Test
-31.70
-06.69
-74.88 -81.38
-80.62
Table 1. The direct recall, logical inference, and compositional inference accuracies before and after KE. Results are for ROME; see Appx. G.1.1 for other methods. (a) The performance of our model (before editing) across the three cyclic orders (I, II, and III). Not only does our model perform well on direct recall, but it also generalizes to both logical and compositional infer- ence tasks. This suggests that the model's 's internal representations extend beyond simple memorization and capture the underlying global structure that relates entities. (b) Changes in model accuracy after applying corrective knowledge edits. Each (AAcc.) result is averaged across multiple edits, and each row labeled edit/retain/test is averaged across each of the cyclic orders taking turns, i.e., play- ing the roles of the edit, retain, and test sub sub-graphs. We find that corrective knowledge edits negatively affect the model's ac- curacy both on related and unrelated facts. These results align with the findings on LLMs (Gu et al., 2024; Gupta et al., 2024a). (c) (AAce.) for edit, retain, and test sub-graphs after applying counterfactual edits. Intentionally introducing inconsistencies into the model's 's knowledge via counterfactual KE can significantly degrade model capabilities. Furthermore, the greater the induced inconsistency (scaling the counterfactual edit distance d from 1-4), the more severe the resulting performance degradation.
hop anti-clockwise neighbors can be inferred directly from the k-hop clockwise neighbors. Compositional inference accuracy measures the accuracy on held out subset of compositions of two relations. Both logical inference and compositional inference measure the accuracy on samples that would be considered out-of-distribbttin.
We report scores for all three metrics in Tab. 1. The model's logical and compositional inference accuracies are close to the direct recall accuracy, which implies that the model gen- eralizes outside of the training data before KE. However, after KE, all accuracies decrease, with more severe decrease for counterfactual edits (they introduce inconsis- tencies between facts).
Overall, we show the following. (/) Transformers trained on knowledge graphs recall facts. perform logical inferences, and compositional inferences. However, both corrective and counterfactual edits degrude the model on all three fronts (Sec. 4. 1). (2) Transformers learn a representation that reflects the underlying geometry of the data. Knowledge edits "shatter" this representation, which serves as an expla- nation for the degradation in accurucy after KE (Sec. 4.2). (3) Counterfactual edits with larger distance display a larger degree of shattering (Sec. 4.4). (4) These phenomena occur in pretrained language models, indicating representation shattering can explain the degradation seen in model abili- ties after KE (Sec. 4.5).
4.2. Model Representations Capture Data Geometry
The model achieves high compositional and logical infer- ence accuracies before knowledge editing, indicating that it captures the global structure of the data and does not merely memorize all the facts seen during training. We see this reflected in the internal representation of the model (output of the second attention layer), which we visual- ize using Isomap (Tenenbaum et al., 2000). Isomap pre- serves geodesic distances along the manifold, which is es- sential for visualizing non-line linear geometries-e.s rings and torii (Chaudhuri et al., 2019; Khona & Fiete, 2022). Meanwhile, PCA, being linear, can collapse or stretch such structures. This makes prefer Isomap for our analysis, since faithful projections of the original topology turn out to be critical for interpreting how KE affects global geometry. We use PCA as well when linearity suffices (see Appx. G.2.2 and Appx. G.3).
4.1. Knowledge Editing can Degrade Model Accuracy
We evaluate the effects of counterfactual and corrective edits on three fronts. Direct recall accuracy calculates the accuracy of facts seen during training. Logical inference accuracy measures the accuracy on a subset of held out relations that can be inferred from other relations, i.e., the k-
In Fig. 4a, we plot how the Isomap embeddings of the inter-
6

---

Representation Shattering in Transformers with Knowledge Editing
Sub-Graph
d
=
=
counterfactual edit distance, where the counterfactual edit distance (or CE distance) is the distance between the entity in the old fact and new fact as measured in the cyclic order. Fig. 3 illustrates an example where the counterfactual edit has an edit distance of 1. In Fig. 6, we see that increasing the distance of the counterfactual edit results in a drop in accuracy and an increasing in the extent of shattering. This relationship is numerically supported by R(D.), as shown in Tab. 2: shattering increases counterfactual edit distance increases. In other words, when new fact changes one entity to another, the extent of shattering increases as the distance between the old and new entity increases. As a naturalistic parallel, if the entities are different months, accuracy is higher when we edit "December" to "November" as opposed to "July".
Edit
01.80
21.93
26.22
27.90
Retain
01.80
20.84
25.32
27.28
Test
01.84
21.89
26.52
28.68
Table 2. Mean R(D) R(D.) for counterfactual edits, averaged across each sub-graph type. We observe higher degrees of representation shattering for greater counterfactual edit distances (d). Results are for ROME; other methods also reproduce this relationship (Appx. G.1.2).
nal representations for the input with one entity and relation (a7) evolves over the course of training. The different data points correspond to different values of the entity x; for a fixed relation F' and the points in the plot are colored by the cyclic ordering. We see that the representation manifold resembles the cyclic ordering of the entities, particularly towards the end of training. In addition, we confirm that the model correctly internalizes the independence of the eyclic orders I, II, and III in its representations (see Appx. G.3 for a visualization).
Summarizing these experiments, we find that for given subject entity, edits with larger CE distances imply greater displacement in the representation manifold and higher R(D.)-in reverse, higher R((D)) implies a larger CE dis- tance. This manipulation approximates an intervention on representation geometry itself, and this offers compelling ev- idence for a causal link between shattering and performance loss. This is consistent with our mechanistic explanation, though we do not claim formal proof.
4.3. Corrective KE Shatters Representation Geometry
We assess how the representation changes after applying a corrective knowledge edit-i.e., applying KE to a fact that the model learned incorrectly during training. While one would expect the performance of the model to increase after a corrective edit, we find the opposite: a corrective edit results in a drop in all accuracies (see Tab. 1). These results align with previous empirical findings showing that reasoning capabilities degrade after corrective edits (Gu et al., 2024; Cohen et al., 2023).
4.5. Representation Shattering Generalizes to LLMs
Finally, we investigate whether our findings generalize to large Transformers trained on naturalistic data. We consider concepts with cyclic order, in particular months of the year, and apply counterfactual edits to Llama 3 (Grattafiori et al., 2024) and Mamba S4 (Gu & Dao, 2023) (see Appx. H.)). We additionally explore non-cyclic geometries, specifically tree-structun concepts, in Appx. H.3.
We visualize the representations of 3 different models using the techniques described in Sec. 4.2 and Fig. 4b, The 3 models are obtained after applying different edits and are selected to have high (*), intermediate (A), and low (x) direct recall accuracies. In Fig. 5, we observe that the model with the highest accuracy (*) has representation that pre- serves the geometry of the data after the edit. However, as the model accuracy decreases, the representations also dis- play greater degree of distortion, no longer capturing the geometry of the data; in other words, the model is affected by representation shattering. Beyond visual inspection, this trend is also quantified in Fig. 5c, which shows a strong negative relationship between the distortion metric R(D.)
In previous evaluations of KE algorithms on real-world LLMs, degradations in model performance were most consistently detectable for batched, sequential edits (Gu et al., 2024; Yoon et al., 2024). Therefore, we carry out our LLM experiments using MEMIT (Meng et al., 2023), the successor to ROME which adds support for batch editing (see Appx. E.2 for a short primer). Using MEMIT, we insert counterfactual month-order associations varying edit distances and note how model performance and repre- sentations are affected. As for the prompts used for editing, we take inspiration from existing work by Engels et al. (2024) and use the following template: "[Month) is followed {offset) months later by the month of 00". Here, Month is uniformly sampled from "January.... Decemberr) and ofset is uniformly sampled from (eight, nine, ten). For edits with counterfactual edit distance of d, we modify the answer to be d months earlier than the true correct answer (for d E (1,2,3)). 3)). For example, the ground truth answer
(Eq. 1) and model accuracy (r2 = 0.905).
4.4. Shattering Scales with Counterfactual Edit Distance
Counterfactual editing, wherein ones adds new facts that were unseen during training, is commonly used for evalu- ating KE protocols (Meng et al., 2022a; 2023; Gupta et al., 2024a; -bbermarr- et al., 2023). We consider 25 different counterfactual edits corresponding to every single
7

---

Representation Shattering in Transformers with Knowledge Editing
Comp. (9%
0.15
-20
0.30
% 1 Logle (%)
0.05
_3 a5 A Direct (%)
2000x
6,003
0.00
R(D.)= 0.149
RID.)=
R(D.)
50
Moan & Acc. (%)
(a)
(b)
(c)
Figure 5. Representation shattering strongly correlates with degradation in accuracy. (a) We plot the change in direct recall, logical inference, and compositional inference accuracies for different edits models, edited on different facts. We find all 3 accuracies to be strongly correlated. We select 3 edited models that span the range of accuracies, which is denoted by x, and * in the plot. (b) We plot the representations using a variant of Isomap (see Appx. F) with the entities colored by the cyclic order. We observe a clear trend where larger drop in accuracy directly correlates with greater degree of representation shattering, ie.. the geometric structure of the data is destroyed after the edit. (c) We plot the mean drop in accuracy against the representation shattering metric R( R(D.) as defined in Eq. 1. Greater representation shattering is strongly correlated with more severe accuracy degradation (r² 0.905).
Accuraee
Logical Inference Compottional Inferance
Direct Recall
15)
0.00
MMLU-Reduux Liama 3 BB Instruct, post-MEMIT
0:30
-0.05
0:23
Unedited
CE Dist 1
Dist =2
CE Dist =
Dist
1,C1 (EdR)
0.015
0.104
0.138
0.169
0.20
unedited
dii
d=2
d=3
16)
0.000
0.139
0.157
0.247
(Retain)
0.013
0,065
0.119
0.132
Figure 7. Representation shattering also occurs in a real LLM (Llama 3 8B Instruct). We apply KE (MEMIT) with counterfac- tual prompts to modify the eyclic order of calendar months in the Llama 8B Instruct model. After editing, we evaluate the model's ability to perform reasoning on the MMLU-Redus benchmark and visualize the model's representations for each month. (a) As the counterfactual edit distance grows, the model's 's reasoning abilities gradually decrease. (b) Coinciding with the degredation reason- ing performance, the gradual shattering of the ring structure in the model's intermediate representations occurs as counterfactual edit distance is increased (activations were collected at the layer 19 output; see Appx. H.2 for other extraction points). For similar results with Mamha S4 (Gu & Dao, 2023), refer to Appx. H.1. For experiments with concepts organized in a non-circular geometry, refer to Appx. H.3.
IlIl C1 (Teok)
0.015
0.062
0.108
0.148
Figure 6. Counterfactual edits with larger edit distance result in larger drop in accuracy and greater degree of represen- tation shattering. We apply counterfactual knowledge edits to overwrite correctly learned fact (1154.I_c1-56) with in- consistent counterfactual associations. We then plot the accuracy after the counterfactual edit for different edit distances and the corresponding low- wiwedimeniooall embedding of the representation obtained using a modified version Isomap (see Appx. F). The numerical quantity in the upper right of each manifold visualiza- tion is the RDD.) value measuring the degree of representation shattering with respect to the manifold of the unedited model. Both visually and numerically, we find that a counterfactual edit with larger edit distance requires significant distortion to the represen- tation geometry to learn the new fact.
To quantify model performance before and after editing, we adopt the MMLU-Redux reasoning benchmark (Gema et al., 2024) with the ZeroEval prompting framework (Lin, 2024) to elicit Mhain-of-though reasoning.
to the prompt "January is followed eight months later by the month of O)" should be "September"; counterfactual edits may look like changing the output "August" (d 1), "July" (d 2), "June" (d = 3). For each specific counterfactual edit distance d, we have (twelve months) x (three offsets) = 36 total edits- we apply these edits sequentially in batches of size 4 using MEMIT, taking care to use consistent sample orders and batch orders across all runs for fair comparisons.
In Fig. 7a, we examine how the reasoning performance of Llama 3 8B Instruct is affected by KE in relation to the counterfactual edit distance. We find that the edit dis- tance increuses, there is gradual decline in the model's reasoning accuracy. Furthermore, Fig. 7b shows the latent representations for the 12 months, extracted from an inter- mediate layer post-editing. As the edit distance is varied from I to 3, the degree of representation shattering also
8

---

Representation Shattering in Transformers with Knowledge Editing
increases. This result demonstrates that our insights from synthetic data, i.e. the representation shattering hypothesis, can generalize to larger models trained on naturalistic data.
Impact Statement
This paper introduces a scientific framework to analyze the phenomenon of representation shattering in Transformer models during Knowledge Editing. We believe the insights uncovered in this work will form foundational step toward developing more robust and interpretable knowledge edit- ing techniques, ultimately advancing the reliability of AI systems for societal benefit.
5. Conclusion
In this work, we introduced a synthetic framework to an- alyze the side effects of knowledge editing in transform- ers, identifying "representation shattering" key factor behind performance degradation. Specifically, we show preserving representational structures underlying model's knowledge is crucial to avoiding negative consequences of knowledge editing: distortion of such structures impacts a model's 's broader capabilities. To arrive at this hypothesis, we design a controlled framework that allows investigations into models modified by knowledge editing protocols, offer- ing clear representation-leved explanations for why knowl- edge editing can harms models' broader capabilities that generalize to real-world models like Llama 3. While the use of simplified tasks and models can limit the scope of our conclusions, since larger, more complex real-world models may exhibit additional dynamics that our framework does not capture, we believe that testing knowledge editing pro- tocols on setups similar our synthetic knowledge graph will significantly aid the process of designing better editing protocols. Failing even such simple, albeit systematically defined settings, likely implies the editing protocol should not be readily trusted or applied at scale.
Contributions
KN and ESL conceived the project direction and designed the primary experimental setup, with inputs from RR and MK. The representation shattering hypothesis was proposed by KN, ESL, and HT. KN led experiments on the synthetic knowledge graph domain. RR, KN, and ESL wrote the paper. Conceptual figures were designed by HT, with inputs from KN and MO. KN and MO conceived the natural data setup for confirming the hypothesis and ran experiments therein. KN wrote the appendix. ESL and HT supervised the project.
References
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. Do as can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169 2022.
Additionally, our study aligns with a broader theoretical framework: (!) Transformers store factual associations in key-value pairs within MLP layers (Geva et al., 2020; Meng et al., 2022a), corroborated by our synthetic model analysis. (2) Parameter sharing and superposition cluster unrelated facts in overlapping subspaces (Geva et al., 2020; Henighan et al., 2023), making them vulnerable to unintentional inter- ference. (3) Entities and relations often form structured man- ifolds (c.g., cycles, hierarchies), which aid compositional inference (Engels et al., 2024). (4) KE methods (ROME, MEMIT, etc.) enact local weight updates that deform these manifolds, causing representation shattering for unedited facts residing in shared sub-regions, Points (1)-(4) together lead us to believe that the fragility of KE arises from the entangled, compressed nature of factual storage, rather than just from the task of knowledge retention in and of itself.
Allen-Zhu, Z. and Li, Y. Physics of language models: Part 3.1. knowledge storage and extraction. arXiv preprint arXiv:2309. /43 2023a.
Allen-Zhu, Z. and Li, Y. Physics of language mod- els: Part 3.2, knowledge manipulation. arXiv preprint arXiv:2309. 14402, 2023b.
Allen-Zhu, Z. and Li, Y. Physics of language models: Part 1, context-free grammar. arXiv preprint #xXiv23305.13675 2023c.
Anderson, J. A. A simple neural network generating an interactive memory. Mathematical biosciences, 14(3-4): 197-220, 1972
More broadly, our study suggests that mitigating the pitfalls of KE may require fundamentally rethinking the popular soocalize-then-edin paradigm. Indeed, the field has already begun moving in this direction: promising methods include Retrieval Augmentation (Lewis et al., 2020; Borgeaud et al., 2022), Lifelong Model Editing (Wang et al., 2024), and Synthetic Document Finetuning (Wang al., 2025). We hope that our work can help guide the design of further improvements to KE approaches.
Bau, D.. Liu, S., Wang. T., Zhu, J.-Y., and Torralba, A. Rewriting deep generative model. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK August 23-28. 2020, Proceedings, Part 16. pp. 351-369. Springer, 2020.
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Van Den Driessche, G. B., Lespiau, J.-B., Damoc, B., Clark, A., et al. Improving language models by retrieving from trillions of tokens. In International

---

Representation Shattering in Transformers with Knowledge Editing
conference on machine learning, pp. 2206-2240. PMLR, 2022.
Linear, May 2024. URL http://////rxi
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P. Lee, Y. T., Li, Y., Lundberg, S., al. Sparks of artificial general intel- ligence: Early experiments with gpt-4. arXiv preprint 1rXiv:2303.127/22 2023.
Fang, J., Jiang, H., Wang, K., Ma, Y., Wang, X., He. X., and Chua, T.-s. Alphaedit: Null-space constrained knowledge editing for language models. arXiv preprint arXiv:2410.02355, 2024.
Feng, G., Gu, Y., Zhang, B., Ye, H., He, D., and Wang, L. Towards revealing the mystery behind chain of thought: theoretical perspective. arXiv preprint arXiv:2305.1540. 2023.
Chan, S., Santoro, A., Lampinen, A., Wang, J., Singh, A., Richemond, P., McClelland, J., and Hill, F. Data distri- butional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 35:18878-118 18891, 2022.
Fiotto-Kaufman J., Loftus, A. R., Todd, E., Brinkmann, J., Juang, C., Pal, K., Rager, C., Mueller, A., Marks, S. Sharma, A. S., Lucchetti, F,, Ripa, M., Belfki, A., Prakash, N., Multani, S., Brodley, C., Guha, A., Bell, J., Wallace, B., and Bau, D. NNsight and NDIF: Democratiz- ing access foundation model internals. arXiv preprint arXiv:2407.14555 2024.
Chappell, S.-G. Plato on knowledge in the theaetetus. 2005.
Chaudhuri, R., Gerçek, B., Pandey, B., Peyrache, A., and Fiete, I. The intrinsic attractor manifold and population dynamics of a canonical cognitive circuit across waking and sleep. Nature neuroscience, 22(9):1512-1520, 2019.
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. O., Kaplan, Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained code. arXiv preprint arXiv: 107.03374, 2021.
Gema, A. P., Leang, J. O. Hong, G., Devoto, A., Mancino, A. C. M., Saxena, R., He, X., Zhao, Y., Du, X., Madani, M. R. G., Barale, C., McHardy, R., Harris, J., Kaddour,
J., van Krieken, E., and Minervini, P. Are we done with mmlu?, 2024.
Cheng. S., Zhang, N., Tian, B., Chen, X., Liu, Q., and Chen, H. Editing Language Model-based Knowledge Graph Embeddings, December 2023. URL http: //axiv.. org/ 2201 10405.
Gemini Team. Gemini: a family highly capable multi- modal models. arXiv preprint arXiv:2312. 2023.
Geva, M., Goldherg, Y., and Berant, Transformer feed- forward layers are key-value memories. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing, 2020.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., et. The llama 3 herd models, 2024. URL https: //arxiv.org org. abs 224074400020 .27883.
Cohen, R., Biran, E., Yoran, O., Globerson, A., and Geva, M. Evaluating the Ripple Effects of Knowledge Editing in Language Models, December 2023. URL http:// arxiv. org/ abs/ 2307 12976.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,200 2023.
Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., and Wei, F. Knowledge neurons in pretrained transformers. arXiv preprint arxiv:2 08696, 2021.
Gu, J.-C., Xu, H.-X., Ma, J.-Y., Lu, P., Ling, Z.-H., Chang, K.-W., and Peng. N. Model editing harms general abilities of large language models: Regularization to the rescue. arXiv preprint arXiv:2401.04700 2024.
De Cao, N., Aziz, W., and Titov, I. Editing factual knowl- edge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
Gupta, A., Rao, A., and Anumanchipalli, G. Model Editing at Scale leads to Gradual and Catastrophic Forgetting, Jan- uary 2024a. URL http::///aaxeeeeeeeeeet/ org/ abs/ 2401. 07453.
Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 2023.
Gupta, A., Sajnani, D., and Anumanchipalli, G. unified framework for model editing. arXiv preprint arXiv:2403. /4236, 2024b.
Engels, J., Liao, I., Michaud, E. J., Gurnee, W., and Tegmark, M. Not All Language Model Features Are
10

---

Representation Shattering in Transformers with Knowledge Editing
Hase, P., Bansal, M., Kim, B., and Ghandeharioun, A. Does Localization Inform Editing? Surprising Differ- ences in Causality-Based Localization vs. Knowledge Editing in Language Models, October 2023. URL http: //arxiv.org/ abs/ 2201 04213.
Li, Z., Zhang, N., Yao, Y., Wang, M., Chen, X., and Chen,
H. Unveiling the Pitfalls of Knowledge Editing for Large Language Models, November 2023b. URL http:// wtg/abs////eoooo.ooo..ggoo...ee/ abs 22000 02129. arXiv:2310.02129 wwxiv.org//aoooa...oo.ooo.ooo [cs].
org/
Hase, P., Hofweber, T., Zhou, X., Niengll-Eskin, E., and Bansal, M. Fundamental problems with model editing: How should rational belief revision work in Ilms? arXiv preprint arXiv:2406. 0et.:24406.19354.. 2024.
Lin, B. Y. ZeroEval: A Unified Framework for Evaluating Language Models, July 2024. URL httpt::/tgtthttei com hom/Wilddne WildEval/ZeerFFe
Lubana, E. S., Kawaguchi, K., Dick, R. P., and Tanaka, H. A percolation model of emergence: Analyzing trans- formers trained on a formal language. arXiv preprint arXiv:240.122555 2024.
Henighan, T., Carter, S., Hume, T., Elhage, N., Lasenby, R., Fort, S., Schiefer, N., and Olah, C. Superposition, memorization, and double descent. Transformer Circuits Thread, 6:24, 2023.
Lynch, A., Guo, P., Ewart, A., Casper, S., and Hadfield- Menell, D. Eight methods to evaluate robust unlearning llms. arXiv preprint arXiv:2402.168.. 2024.
Hoelscher- Obermmar,, J., Persson, J., Kran, E., Konstas, I., and Barez, F. Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark, June 2023. URL hwwp::ttp///. htt//rxxvvvvo wrg.i.oorrr../o....... att///bs//3/0/ 17553.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35, 2022a.
Hofweher, T., Hase, P., Stengel-Fskin, E., and Bansal, M. Are language models rational? the case of co- herence norms and belief revision. arXiv preprint arXiv:2406.03442.220 2024.
Meng, K., Sharma, A. S., Andonian, A., Belinkov, Y., and Bau, D. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022b. 2022b.
Karpathy, A. NanoGPT, 2021. Github link. https:// github. www.coo////o...... ka
Meng, K., Sharma, A. S., Andonian, A., Belinkov, Y., and Bau, D. Mass-Edin Bass-Editing Memory in a Transformer, Au- gust 2023. URL http: http:///axivvi org/ wwwp/rs////////////o///ee/// abs/2210. 07229.
Khona, M. and Fiete, I. R. Attractor and integrator networks in the brain. Nature Reviews Neuroscience, 23(12):744 766, 2022.
Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Manning, C. D. Fast Model Editing at Scale, June 2022. URL httt://arxive wwt.aawwwwww.aaaaaaaaaaaoa /211//21100 10. 11309.
Khona, M., Okawa, M., Hula, J., Ramesh, R., Nishi, K., Dick, R., Lubana, E. S., and Tanaka, H. Towards an understanding of stepwise inference in transform- ers: A synthetic graph navigation model. arXiv preprint arXiv:2402.07757,,20 2024.
Okawa, M., Lubana, E. S., Dick, R. P., and Tanaka, H. Compositional abilities emerge multiplicatively: Exploring diffusion models on a synthetic task. 2023.
Kohonen, T. Correlation matrix memories. IEEE transac- tions on computers, 100(4):353-355 2009.
Prystawski, B., Li, M., and Goodman, N. Why think step by step? reasoning emerges from the locality of experience. Advances in Neural Information Processing Systems, 36, 2024.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t. Rocktäschel, T., et al. Retrieval-augmentt intensive ulp tasks. Advances in neural information pro- cessing systems, 33:9459-9477 2020.
generation for knowledge-
Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, I.., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16788 2023.
Li, K., Hopkins, K., Bau, D., Viégas, F., Pfister, H., and Wattenberg, M. Emergent World Representations: Exploring Sequence Model Trained on Synthetic Task, 2023a.
Sharma, A. S., Atkinson, D., and Bau, D. Locating and editing factual associations in mamba. 2024.
X., Li, S., Song. S., Yang. J., Ma, J., and Yu, J. Pmet: Pre- cise model editing in transformer. In Proceedings of the AAA/ Conference on Artificial Intelligence, volume 38, pp. 18564-18572, 2024.
Sinitsin, A., Plokhotnyuk, V., Pyrkin, D., Popov, S., and Babenko, A. Editable neural networks. arXiv preprint arXiv:2004.00) 000444004000345, 2020.

---

Representation Shattering in Transformers with Knowledge Editing
Tenenbaum, J. B., Silva, V. d., and Langford, C. A global geometric framework for nonlinear dimensionality reduction. science, 29065500);231--2333 2000.
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kul- shreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. Lamda: Language models for dialog appli- cations. arXiv preprint arXiv:2201.092. 2022.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13977 2023.
Wang, P., Li, Z., Zhang, N., Xu, Z., Yao, Y., Jiang, Y., Xie, P., Huang, F., and Chen, H. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. CoRR, 00022405147668 2024. URL https: /ttp/viivvv..////e atgg/bb///// 22../245.1111 14768.
Wang, R., Griffin, A., Treutlein, Perez. E., Michael, J., Roger, F. and Marks, S. Modifying Ilm beliefs with synthetic document finetuning. patps://alits http://aaagmmnt...t modifying beliefs- -via-diia..iaaa 2025. An- thropic Alignment Science blog post.
com/2025/
Wikipedia. Be Like Mike, 2024. Wikipedia Link. https: //en. wwwipedia.ia..... wttp://wiw////iieeee..eeeee
Yoon, J., Gupta, A., and Anumanchipalli, G. Is bigger edit batch size always better? an empirical study on model editing with Ilama-3. arXiv preprint arXiv:2405.0066- 2024.
Zhou, H., Bradley, A., Littwin, E.,, Razin, N., Saremi, O., Susskind, J., Bengio, S., and Nakkiran, What algo- rithms can transformers learn? study in length general- ization. arXiv preprint arXiv:2310.166 2023.
12

---

Representation Shattering in Transformers with Knowledge Editing
A. Setup Details
A.1. Source Code
Please find the source code for our experiments at github www../oeettnssi///eeoooo.. ICML- 2025.
A.2. Pseudo-Code
Let (.) define the uniform distribution over the input. Let X be the set of entities, I the set of relations and F the set of facts, defining a knowledge graph G F.)
function generateSequence ()
Sp (X) from uniform distribution over the entities.
S [xp]
enti entity.flag + False
// Create a sequence of alternating entities and relations
while 1en (S) < context.size do if (entity-flag) then
// Add an entity that completes valid fact Set vn such that (Tp, F,8,) is a fact in the knowledge graph G. Sappend (x,)
Sp 6 In
end
else
// Add composition of relations
K ~ ((1,2)) F=1]
for (i in 1 to K) do
>~ (R)
S.append (r)
raappend (r)
end
Set vn such that (Ix, an) is fact in the knowledge graph end
entity -
end S = S[context.size]
return
Algorithm 1: Generate single sequence containing a collection of facts.
B. Data Generation Process Details
For this study, we use the following hyperparameters for our data generation process.
Number of entities: 2048
Number of example sequences: 1
Maximum composition length: 2
Maximum entities per sequence:
Additionally, only when generating the training dataset, the DGP may drop sequences which contain one direction of pair of conjugate facts with fixed probability That is, for any entity and relations r,r', suppose the fact (xi, æt)) always implies (zy,7" r', xx) (i.e.
C1 and r' = I A1). The DGP may hold out one of these facts from the training data (with probability p). Even if r",,) absent, one can still learn the conjugacy of r and and the existence of x; and zy via other examples; however, failure infer this relation indicates the model has rote memorized relations rather than understanding the global structure. To be abundantly clear, "held-out" refers specific facts, not entire relation tokens. In practice, we set the probability =
13

---

Representation Shattering in Transformers with Knowledge Editing
C. Prompt Formats
Compositional Inference: The model is given a chain relations (e.g,, 22 (2) and must produce zg. with (I,, T1, Ti)) and (Iy.72: T2, FR) seen separately in training but not composed. Input: ctx z, F where f' 71
Logical Inference: The model asked about fact (',,)",z;) where (2477 was seen but r' was withheld. If the model understands relation symmetry, it can infer the inverse.
D. Model Architecture
Our Transformer model is a fork of the open-source nanoGPT repository (hetps://githun) wtt.ccoo///aaeeeeccc katpkthrpttty///// nanoGPT). The design is inspired by GPT, and the architecture is decode-only Transformer with a causal self-attention mask. Our hyperparameter values are as follows.
Batch size: 256
Context length: 16
Optimizer: Adam
Learning rate: 6- 10-4
Training epochs: - -00
Decay iterations: 1.5 105
Momentum: 31 = 0.9, 82 0.95
Activation function: GeLUU
Block size: 16
Embedding dimensions: 24
Heads: 12
As for tokenization, we assign every entity and relation unique token and use standard next-token prediction with cross- entropy loss. target,, the 1-shifted version of the training sequence accounting for the padding token, and x,, are the logit outputs of the model at the nith timestep.
exp( (3 target uungan)
) -log softmaax (efttmax(3x,))) (axnaaat
C(x,, target n)) log
Σ -
#tokens
exp(Brin.))
prob(target n)
E. Knowledge Editing Methods
E.1. Rank-One Model Editing (ROME)
E.1.1. ALGORITHM DEFINITION
Rank-One Model Editing (ROME), proposed by Meng al. (2022a), is popular knowledge editing algorithm used
LLMs. Their contributions are two-fold: first, through "causal tracing," they find that early and mid-layer MLP modules of transformer models are implicated in encoding factual associations (see Appx. G.2.1). Second, interpreting feed-forwan layers as linear associative memories encoding key-value pairs, ROME applies a rank-one update to the ML.P weights.
Notationally, for factual association (zi, x;;), the key the entity x; while the value is Tz. In each feed-forward layer, the hidden state h, (1-1) layer 1 - is transformed into key k by the weight matrix w(() fer and the corresponding value is retrieved by the matrix w(l) pros:
h = will)
(w/l (1-))
proj
where (.) denotes the activation function.
To modify the factual association (1,,),0;) in the model, ROME computes new key-value pair (k", v.). vK,,""). entity z; and the new target entity 2j. ROME then applies a rank-one update to the weight matrix w((') to encode this new fact:
representing the proj at a specific layer
14

---

Representation Shattering in Transformers with Knowledge Editing
w((") k* proy (C-Ik*) k
V) proj = w proj + x (C= where À V
Here, C is the uncentered covariance matrix of the key vectors k, estimated by sampling tokens from representative dataset.
The key vector k* corresponds the entity z; in the factual association (I T, 233). The vector is computed by averaging the MLP output for z; over multiple randomly generated contexts:
('') fc h
k'
N
where 7(-) is normalization function, and a
(1') is the attention output layer 1".
The value vector v is optimized maximize the model's probability of predicting the target entity 2 given the subject 2; and relation This is done by minimizing the following objective:
N Σ (- log P (2)1mir)) + DEL (Po (Pll))) (ssl)))
((z)
N
The first term maximizes the probability of the target entity æj. while the second term controls for "essence drift" to retain information about Tg. This is done by sampling inputs y' for which the model's outputs should not change during the edit.
1.1. IMPLEMENTATION
In our implementation of ROME tailored to our model, we apply the edit at layer as it the only available early-site layer in our model configuration. The covariance matrix C is estimated by randomly sampling 10 inputs from the validation dataset. This provides representative of key vectors for computing the rank-one update. To solve for the key vector k', sample 10s random context sequences, with sequence lengths varying between and 10 tokens. The value solver follows similar procedure by sampling 102 context sequences selected in the same manner as the key solver. The value optimization is performed using the Adam optimizer, with hyperparameters Ir 10-3 and weight decay = 10-4, The value solver optimizes between and 500 iterations, stopping when the predicted token is replaced by 27. The KL divergence weight is set during optimization.
E.2. Mass-Editing Memory in a Transformer (MEMIT)
E. ALGORITHM DEFINITION
The Mass- Edising Memory a Transformer (MEMIT) algorithm proposed by Meng et al. (2023) generalizes ROME (Meng et al., 2022a) to inject many factual associations at once. Rather than modifying a single layer for single key-value pair, MEMIT identifies multiple MLP layers (via causal tracing) which encode facts. For each fact (z,, r, x3), small residual ö computed to increase P(= æ4, r). These vectors then batched and distributed across the identified layers. each layer E, let K and M respectively stack the new keys and target values derived from {6.}. Extending ROME, the weight update for layer f is
A (
(M
(
K) K
(C
(C+ + KKT)²
proj
where C is the uncentered covariance matrix of preexisting keys in layer f.
E.2.2. IMPLEMENTATION
For our experiments with Llama 3 8B Instruct and Mamba 2.8B, we simply apply public implementations with off-the-shelf hyperparameters (Gupta al., 2024a;b; Sharma et al., 2024).
15

---

Representation Shattering in Transformers with Knowledge Editing
F. Visualization Methods
In Fig. 4, we demonstrated the emergence eyclic representations within the model by extracting representations and generating 3D Isomap projections. While the visualizations support the notion that cyclical representations are present the model, changes in the projections can be difficult intuitively interpret due to the overlap of differently colored segments of the manifold. For example, below is recreation of Fig. 6 using Isomap projections.
Accuracy (%)
Direct Recall
Logical Inference
Compositional Inference
0.50
0.25
Unedited
CE Dist
Dist 0.104
Dist
CE Dist =
0.015
0.138
0.169
1,11 L. (Edit)
0.013
0.065
0.119
0.132
11 (Retain)
0.015
0.062
0.108
0.148
III C1 (Test)
Figure 8. An equivalent version of Fig. using the unprocessed Isomap projection renderings. Representation shattering is still visible in the flattening and clustering of points the manifold the counterfactual edit distance increases.
The coinciding ring segments are an artifact of the lossy projection of high- dighenimensional cyclical representations into low-dimensional space: when dimensionality reduction to 3D applied, the high-dimensional cyclical structure gets "squished" into a torus. To enhance the visual perceptibility of the representation shattering phenomenon, we additionally implement& a pre-processing step to constrain the construction the Isomap neighbors graph using the model's output predictions. More concretely, when visualizing the post-edit manifold for particular edit (I,, 2z), we adopt the following procedure:
1. Construct set So of entities by prompting the unedited model for all immediate neighbors of x; in the cycle order of (i.e. by getting outputs for z,r' for all r' in the same cycle order as r).
2. Apply the knowledge edit.
Construct set S1 of entities by collecting outputs from the edited model for all s;r where 8; E So-
Constrain the Isomap pair-wise distance matrix to members of S,.
This procedure remains faithful in comparing the pre-edit model to the post-edit model, as relies solely model predictions and does not introduce any ground-truth priors.
16

---

Representation Shattering in Transformers with Knowledge Editing
G. Additional Synthetic DGP Experiments
G.I. Alternative Editing Methods and Models
G. G.I.I. MODEL ACCURACY
In Tab. 1, we evaluate the effects of corrective and counterfactual edits with ROME with respect to changes in the model's direct recall accuracy, logical inference accuracy, and compositional inference accuracy. The results give several key insights: corrective knowledge edits negatively affect the model's aceuracy both on related and unrelated facts, intentionally introducing inconsistencies into the model's knowledge via counterfactual KE significantly degrade model capabilities, and greater induced inconsistency (scaling the counterfactual edit distance d from 1-4) causes greater performance degradation. Now, we reinforce these findings by repeating the same edits and evaluations with additional KE methods; namely MEMIT (Meng et al., 2023), AlphaEdit (Fang et al., 2024), and PMET (Li et al., 2024). We present our results in Tab.
Corrective edits
(AAcc.) for Counterfactual edits
KE Method
Test type
Sub-Graph
(AAcc.)
d 1
d=
d= 3
d ==
Edit
-21.5
-01.99
-67.01
-77.07
-77.94
Direct recall
Retain
-22.64
-01.91
-66.70
-75.49
-75.42
Test
-21.8
-01.75
-67.00
-76.12
-77.90
Logical inference
Edit
-22.24
-01.44
-67.22
-77.14
-78.02
ROME
Retain
-22.50
-01.83
-66.88
-75.67
-75.67
Test
-22.03
-01.80
-67.31
-76.27
-78.23
Compositional inference
-29660
-05.32
-73.15
-80.35
-80.63
Retain
-31.92
-05.32
-71.21
-78.70
-78.87
Test
-31.70
-06.69
-74.88
-81.38
-80.62
Edit
-09.51
-01.64
-57.98
-67.04
-68.72
Direct recall
Retain
-07.08
-01.78
-8.68
-57.23
-58.52
Test
-06.54
-01.19
-51.85
-63.96
-70.26
I inference
Logical
Edit
-09.58
-01.61
-68.16
-67.31
-69.10
MEMIT
Retain
-06.33
-01.64
-48.45
-57.55
-8.666
Test
-06.67
-01.37
-52.37
-64.65
-70.99
Edit
-11.43
-01.85
-71.79
Compositional
Retain
-08.34
-00.68
-$3.05
-62.71
-64.09
inference
Test
-10.47
-03.30
-53.36
-66.81
-73.42
Edit
.06.5
-01.45
-54.68
-64.01
-63.48
Direct rocall
Retain
-04.68
-01.69
43.72
-52.36
-53.63
Test
-0375
-00.92
47.53
-59.57
66.099
Logical inference
Edit
-06.13
-01.42
-54.93
-64.42
-63.91
AlphaEdit
Retain
-04.37
-01.55
-43.58
-52.74
-53.93
Test
-03.85
-01.03
-48.05
-60.38
-66.83
Edit
-07.55
-01.72
-55.82
-66.42
-68.35
Compositional inference
Retain
-05.99
-00.08
-50.19
-59662
-61.57
Test
-07.03
-02.75
-51.14
-64.14
-70,95
Edit
-03.97
-01.34
-48.77
-50.80
-54.72
Direct rocall
Retain
-02.78
-01.61
-35.54
-39.18
46.36
Test
00201
-00.98
-43.40
-44,29
-52.67
Logical inference
Edit
-04.02
-01.32
-48.48
-$1.05
-55.06
PMET
Retain
-02.47
-01.47
-35.40
-39,39
-46.60
Test
-02.10
-01.11
-4..07
44.76
-53.32
Compositional
-05.60
-01.37
-55.65
-60.62
Retain
-03.09
-00.23
44.24
-47.87
-53.78
inference
Test
-04.56
-02.95
44.00
-50.95
-58.98
Table 3. Results of Tab. 1. replicated using MEMIT (Meng et al., 2023), AlphaEdit (Fang al., 2024), and PMET (Li et al., 2024). Overall, recent methods succeeding ROME are slightly less damaging to model accuracy. However, all evaluated methods nonetheless cause undesirable performance degradations in similar ways to ROME (especially for increased counterfactual edit distances). This suggests that KE methods, despite their differences in approaches, often suffer from similar shortcomings in terms of negatively impacting model performance.
17

---

Representation Shattering in Transformers with Knowledge Editing
G. G.1.2. REPRESENTATION SHATTERING METRIC
In Tab. 2, we showed that increasing the distance of the counterfactual edit results in an increase in the extent of shattering. numerically captured by R(D.). In similar spirit to Appx. G.I.I, we seek verify whether this relationship between counterfactual edit distance and representation shattering holds for methods other than ROME, i.c. MEMIT (Meng et al., 2023), AlphaEdit (Fang et al., 2024), and PMET (Li et al., 2024). We present our results in Tab. 4.
Method
Sub-Graph
d 1
2
Edit
01.80
21.93
26.22
27.90
ROME
Retain
01.80
20.84
25.32
27.28
Test
01.84
21.89
26.52
28.68
Edit
01.89
08.58
09.32
08.78
MEMIT
Retain
01.86
07.31
07.66
07.50
Test
01.85
07.49
08.35
07.70
Edit
01.86
07.77
08.44
07.68
AlphaBdit
Retain
01.85
06.51
06.89
06.99
Test
01.83
06.89
07.60
06.99
Edit
01.83
06.55
06.44
06.41
PMET
Retain
01.84
05.44
05.42
05.85
Test
01.83
06.11
05.75
06.31
Table Results from Tab. 2, replicated using the altemnatiee knowledge editing methods of MEMIT (Meng al., 2023), AlphaEdit (Fang et al., 2024), and PMET (Li et al., 2024). These successors to ROME achieve lower amounts of representation shattering overall, coinciding with their more favorable performance in Appx. G.I.I. However, the relationship between greater counterfactual edit distance d and greater representation shattering R(D.) still robustly holds for all methods. This result again shows that various KE methods struggle similar ways: specifically, the greater the inconsistency between model's original knowledge and edited fact. the greater the resulting distortion upon the model's 's representations.
18

---

Representation Shattering in Transformers with Knowledge Editing
G.2. Validation of KE Algorithms' Assumptions
The knowledge editing methods
across our experiments-nnmelly ROME (Meng al., 2022a), MEMIT (Meng et al., 2023), AlphaEdit (Fang et al., 2024), and PMET (Li et al., 2024) assumptions about the mechanisms by which language models store factual associations in their learned weights. Here, we verify those assumptions to ensure that application of KE to our synthetic experimental setup is in fact appropriate.
G.2.1. CAUSAL TRACING OF FACTUAL ASSOCIATIONS
ROME (Meng et al., 2022a) and its follow-up derivatives rely on "causal tracing" to locate facts within the parameters of large pretrained autoregressive transformer models. This essentially causal mediation analysis applied the internal states of transformer models, with scores attributing each state's contribution toward correct factual prediction. KE methods build upon the finding that, early/middle MLP layers play decisive role in recalling key-value facts about the subject (Meng et al., 2022a). Below, we investigate whether this finding also generalizes to our toy model by performing causal tracing.
Indirect Effect
mip1
mip2
Restoration MLP Layer
Figure The average "Indirect Effect" of MLP output states (for last subject token) on the final fact output elicited from the model, computed across 100 randomly sampled prompts. While our model only has two layers, causal tracing attributes higher scores to the first layer rather than the second. This aligns with the findings Meng et al. (2022a) who showed that early and mid-layer MLPs play larger role in factual recall than later sites. We therefore target the first layer MLPs for our knowledge editing experiments.
19

---

Representation Shattering in Transformers with Knowledge Editing
G.2.2. TRANSFORMER MLPS LINEAR ASSOCIATIVE MEMORY
proj matrix of the MLP layer within the Transformer architecture linear associative memory. Under this perspective, the MLP layers act key-value stores for vector keys and corresponding vector values (Kohonen, 2009; Anderson, 1972; Geva et al., 2020; Bau et al., 2020). Below, qualitatively investigate whether this observation also holds for facts learned by our toy model.
Prior works view the W!)
(a.i) Key
(b.i) Key
(c.i) Key
(a.ii) Value
(b.ii) Value
(c.ii) Value
(a) 4204 I.22-15
(b) 1 1650600100 I_C3-15
(c) 1234. II.C2-2007
Figure 10. Inputs (keys) and outputs (values) of the layer ML. module, projected 2D with PCA. Gray corresponds randomly sampled prompts across all possible facts, red randomly sampled prompts for a specific fact, and blue to the average position of the red points. Comparing (a) (b), we find that fact prompts which resolve same final target entity the same subgraph (i.c., entity 15) share similar keys and values. Meanwhile, completely different fact (like 1234 II II_C2=2007 is stored as a clearly distinct kay-value pair. These patterns support the key-value framework compatible with our synthetic setup.
20

---

Representation Shattering in Transformers with Knowledge Editing
G.3. Independence of Cyclic Orders
In our evaluations, we make edits to various relations under the assumption that the Transformer internalizes the independence of the eyclic orders (I, II, and III). Here, we ask: do the model's internal representations truly reflect this? We answer this question by inspecting the representations for the output of the multi-head attention output in layer at the last token position using PCA. Unlike in previous sections where we focused on fixed relation and varied for inputs of the form ... Iy,, we now vary both Z; and and color-code each projection by the cyclic order to which the relation r belongs. We present the resulting projections Fig. 11, and find that prompts eliciting knowledge for each cyclic order are clustered closely together the latent space- this is further evidence that the model internalizes the properties of the underlying knowledge graph.
Cyclic Order Cyclic Order II Cyclic Order
Figure 11. PCA of representations extracted from the output of the multi-head attention output in layer 2 at the last token position, color-coded by the eyclic order the last relation token.
21

---

Representation Shattering in Transformers with Knowledge Editing
G.4. Manifolds for Various Representation Extraction Points
We repeat our representation visualizations analysis for all relations different layers in the model and different sequence positions, finding the structured representations are found specific token positions. See Fig. 12.
atin1 entily token
mlp1 gelu entiyy loken
cproj entity Loken
alin2 last entily Laken
mlpp gelu entily token
cproj
entily taken
attn1
mlp1 gelu @ last relation token (-1)
mlo1 cproj @ last relation token (-1)
attn2
mlp2 gelu @ last re relation taken (-1)
mip2 cproj @ last relation taken (-11
@ last relation token (-1)
@ last relation taken (-1)
Figure 12. 3D Isomap projections for representations extracted from various token positions and points in the model. The cyclical representation manifolds can only be observed for the last relation token position (-1th token), and not at the last entity token position (-2th token). This intuitively makes sense because the last relation token informs the model about which cycle order the current input is querying for. We primarily use the "attn2 last relation token" representations throughout this work because it is the earliest point at which well-structured eyclical manifold can be observed beyond the point of the ROME intervention (which is at "mipl eproj").
22

---

Representation Shattering in Transformers with Knowledge Editing
G.S. Manifolds for All Relations
In Fig. 13, we provide isomap projections of representations extracted for all relations from our model. We show highly structured representations are formed within the model, indicating the model is truly learning the data- aata-generating process and not merely memorizing information.
LC1
LA1
1,0
1,22
LC3
LA3
1,44
1.44
1|c1 I1A1
1,0
1,A2
AL
1,C4
1,44
[1_c] [l 81.2 B1
|l_C3 IILA3 M_C4 18
Figure 13. Isomap projections for representations for all relations, extracted from the output of the multi- head attention output in layer 2 at the last token position. We find that all relations are represented by a cyclical representation manifold. This shows that the model is not falling back memorization for any
it represents all of its knowledge in consistent, ring-like manifolds.
23

---

Representation Shattering in Transformers with Knowledge Editing
G.6. Counterfactual Editing
G.6.1. DISTRIBUTION OP DEGREDATIONS FOR COUNTERFACTUAL EDITS
The plots in Fig. 14 correspond to the counterfactual editing results presented in Sec. 4.4 and Tab.
Dilanee 1
2
3
-26
-38
-65
-24
-22
60
-28
-26
-1
-44
-11
00
-1
-18
-05
-34
-22
60
-1
-6
-65
-34
2
Ema
Mecgia AisArcAA
CEDistarce
Est Y-
4-
-26
-28
000 -1.
-2.6
-2.6
-04
-22
00
-1.
-2.5
-22
-1.
-0.
-0.6
Figure 14. Distribution post-odit accuracy degredations for direct recall, logical inference, and compositional inference in relation to the counterfactual edit distances. A significant shift can be observed between CE distances of 1 and 2, showing the point at which detrimental representation shattering can occur.
24

---

Representation Shattering in Transformers with Knowledge Editing
G.6.2. ADDITIONAL VISUALIZATIONS
In Fig. 6, we showcase example of the change in accuracies and representation manifolds when applying a counterfactual edits (specifically for fact 1154. I cl). For representative view, additionally provide more examples of counterfactual edits (with both raw and pre-processed versions side-by-side, described in Appx. F).
1.80
9.75
Infersnce Compositioal inffrrnc inference
Logical inference Compostiional Informnce
Acguras
0,00
0.25
Unedited
D
K
0,023
0.165
Ca (Retain)
0.029
0.132
0.165
N
0.016
0,040
0.167
0.215
IlI (Test
0.015
0.040
0.167
0.215
Figure 15. Counterfactual editing visualizations for 1623. A.2..
Acaincy (90
AL Acarrcy (%)
Infereace
Infermnc
Com Compaaiiooaali Inference
Componitionn Inferencs
Unedited
CE Dist =1
CE Dis 2
Dist =3
CE Dist
Unedited
Dist
CE Dist 0.384
DE =
Bist 4
0.014
LAS (Metain)
0.306
(Taa)
0.400
Figure 16. Counterfactual editing visualizations for 1121. II-CI.
25

---

Representation Shattering in Transformers with Knowledge Editing
H. Additional Naturalistic LLM Experiments
H.1. Knowledge Editing on Mamba
In Sec. 4.5, we investigate whether the representation shattering hypothesis generalizes to large Transformers trained on naturalistic data. We consider the eyclic order of the months the year and apply counterfactual edits Llama 3 8B Instruct (Grattafiori al., 2024) and found that as we vary the edit distance from to 3, the degree of representation shattering increases. To further probe the robustness of our claims with respect model size and model architecture, additionally explore KE with Mamba (Gu & Dao, 2023). Mamba is structured state space sequence model, and use the Mamba-2.88 variant for this experiment. For consistency with the Llama experiments, we use the MEMIT editing method adapted appropriately to work with the Mamba architecture (as per Sharma et al. (2024)). For the counterfactual edit prompts, we use the same prompts as in Sec. 4.5 (i.e. "(Month) is followed {offset} months later by the month of ()")
We present the resulting manifold visualizations and R(D.) values in Fig. 17. We find that the relationship between the counterfactual edit distance and the extent of representation shattering can be reproduced in Mamba 2.8B, much like in Llama 3 8B Instruct. As side note, we note that benchmarking Mamba 2.8B MMLU-Redux (Gema et al., 2024) was not achievable since Mamba 2.8B is not instruction-tume induced by larger counterfactual edit distances suggests that the representation shattering hypothesis is generally applicable across different model types and architectures.
nonetheless, the gradual shattering of representations in Mamba 2.8B
0.000
1.226
1.365
1.466
Unedited
CE Dist = 1
CE Dist 2
CE Dist = 3
Figure 17. Fig. 7. replicated using Mamba 2.8B (Gu & Dao, 2023). Like in Llama 3 8B Instruct, the ring structure shatters for larger counterfactual edit distances (activations were collected at the layer output: see Appx. H.2 for other extraction points). This result demonstrates that our findings are not limited to specific models and can be extended to various architectures.
26

---

Representation Shattering in Transformers with Knowledge Editing
H.2. LLM Representation Extraction Points
To measure and visualize representation shattering in Sec. 4.5 and Appx. H.I, we extracted intermediate representations from Llama 8B Instruct and Mamba 2.8B at the output of layers 19 and 38, respectively. We chose these layers because they were the earliest points in each model for which cyclic representations could be observed at the last token position (as shown in Fig. 18 and Fig. 19).
Llama 3 8B Instruct
Layer 17
Layer 18
Layer 19
Layer 20
Layer 21
Figure Representations extracted for the months of the year Llama Instruct, the outputs of layers 17-21. We select layer 19 for our analysis of representation shattering because it is the earliest layer which clear cyclic structure is observed.
Mamba 2.8B
Layer 36
Layer 37
Layer 38
Layer 39
Layer 40
Figure 19. Representations extracted for the months the year Mamba 2.8B, at the outputs of layers 36-40. We select layer our analysis of representation shattering because it is the earliest layer at which a clear cyclic structure is observed.
27

---

Representation Shattering in Transformers with Knowledge Editing
H.3. Knowledge Editing with Naturalistic Trees
In our experiments, we primarily focus on synthetic knowledge graphs with cyclical structures. While the simplicity of eycles is desirable for our synthetic experiments, real human knowledge and language can exhibit more complex structures, For example, geographical ground-truths can be expressed in a tree structure, with entities like citess/oountres//cotineettt having relations with other cities/coumtries/coniicee i.e. = Paris, located in country, I] = France.
Here, we ask: does the representation shattering hypothesis hold for more realistic tree-shaped knowledge graphs in more complex LLMs? To answer this question, we take inspiration from the classic "The Eiffel Tower is located in the city of Rome" example of counterfactual knowledge editing (Meng et al., 2022a). For our purposes, we edit the country associations of major cities. In particular, we consider the following five countries: France, Spain, Italy, Germany, and the United Kingdom. Then, also consider the five most populous cities of each country, totaling 25 cities: Paris, Marseille, Lyon, Toulouse, Nice, Madrid, Barcelona, Valencia, Sevilla, Zaragoza, Rome, Milan, Naples, Turin, Palermo, Berlin, Hamburg. Mumich, Köln, Frankfurt am Main, London, Birmingham, Liverpool, Glasgow, and Sheffield. The knowledge graph involving these city-country pairs contains facts such as (r; Paris, located in country, I = France). The ground-truth arrangements of the cities and countries form a tree (Fig. 20a).
From the latent space of LLMs, however, it is difficult to extract clean tree-like geometries. When we project the representations for tokens corresponding the country and city names using Isomap, the result does not yield discernible tree shape (Fig. 20b). Despite the exact structure of the latent space not being clear, the notion of "distance" in the manifold can still be applied. For example, in Fig. 20b, Spain is closer to France than is the United Kingdom; therefore, the edit "Paris is a city in the country of Spain" has a smaller counterfactual edit distance than does the edit "Paris is city in the country of the United Kingdom." " Fig. 21a a and Fig. 21b show the representation manifold Isomaps after applying the edits "Paris a city in the country of Spain" and "Paris is city in the country of the United Kingdom," " respectively, using ROME on GPT-2 XI.. First, we find that both counterfactual edits cause the representations for all cities and countries to collapse inward. Moreover, the edit to "the United Kingdom" causes a greater distortion than the edit "Spain," evident both by visual inspection and by the numerical representation shattering quantity R(D.).
Unedited Model
Sirminghm
blencia Cheffield
of lermo
Unitted Kingdom
Du louse
X
France Armman
Sevilla
farsellie
Sunich eyon
Japles
könn
Jurnn
Shigld
tilan
gerlin
Su bject (Paris)
Original Ascociation (Fronce)
Spain
United Kingdom
(a)
(b)
Figure 20. (a) The ground-truth tree representing the countries and 25 cities. The correct factual association for the prompt "Paris a city in the country of... is France. In this example, consider the counterfactual edits "Paris is city in the country of Spainn" and "Paris a city in the country of the United Kingdom". (b) Isomap projections of representations for the selected countries and cities. We find that, on this model's 's representation manifold, editing Paris to be in Spain constitutes smaller counterfactual edit distance than does editing Paris to be in the United Kingdom.
28

---

Representation Shattering in Transformers with Knowledge Editing
city country Spain
0.121
city country of United Kingdom
0.156
immingham
himinngamm
ttptes
Bafes
arseille
larsellle
Soin
Valencia
Gln
United Kingsom
oulouse
Sorcelona
alem
Paris
Seain Francc
calermo
afttefitta
Franicllrmary Termany
Haribeegg
carcelona
Iverpool
lasgow
Bassgow
United Kingdom
firs
R
erlin
Fram ermmnyy
Subject (Paris)
Subject (Paris)
Original Association (Prance)
Original Association (France)
Association (Spain)
Target Association (United Kingdom)
(b)
Figure 21. Isomap projections of latent representations after applying counterfactual edit. (a) "Paris is city in the country of Spain." (b) "Paris is a city in the country the United Kingdom."
To take step in verifying whether this finding is generalizable, applied counterfactual edits to each of the 25 selected citics. For each city, computed the country which constitutes the "closest" and "furthest" counterfactual edit distance the model's representation manifold. After applying the two counterfactual edits, computed R R(Dfurthest) and R( Delossss. Across the 25 cities, the average ratio R Dfarteet) / R ((Delssst)) was 1.1483. In other words, when changing a city 's parent country, editing to close country on the representation manifold yields less shattering than editing to country which sits far away on the manifold.
These preliminary results align with our main hypothesis: KE methods distort language models' representations in order to insert new facts alter old ones (i.e. representation shattering), and the extent of representation shattering increases with the distance between the old fact and the desired new fact on the manifold.
29


Reviews:
[{"review_id": "859EKfe5Zu", "reviewer": "Reviewer_gDBJ", "summary": "This paper proposes a fundamental principle to understand why existing Knowledge Editing (KEs) methods often introduce unexpected cascading effects on knowledge not tampered with during edition and cause the edited LLMs to yield inconsistent reasoning results. Specifically, the authors argue that, especially for relational knowledge like those represented by Knowledge Graphs (KG), the representations of the entities in a good pre-trained model will demonstrate geometries that are consistent with the underlying KG. However, KE procedures would invariably break or shatter this representation geometry, thus causing troubles in downstream reasoning tasks. To support this claim, the authors performed extensive empirical studies, both quantitative and qualitative, on multiple SoTA KE methods (ROME, MEMIT, PMET, & AlphaEdit), multiple LMs (2-layer transformer and pretrained LLMs like Llama3 8B, Memba), on both cyclic-structured and tree-structured knowledge graphs in a synthetic setting.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our experiments and analysis of representation shattering. Please see our responses below.\n\n---\n> **#1: $R(D_*)$ and Permutation Invariance**\n\nGreat question! We intentionally made our distortion metric $R(D_*)$ to not be permutation-invariant, i.e., $R(D_*) > 0$ even if $D_*$ is a permutation of $D_{\\varnothing}$. This is because $R(D_*)$ tracks *how much each entity's position in the representation space, as identified by its token, has changed*—not whether the manifold is isomorphic under another labeling.\n\nFor example, in your suggested swap of entity 2 with entity 3, the manifold may remain consistent, but the label-respecting configuration changes, and so do the model's semantics. A zero value would suggest no parameter shift even if outputs differ.\n\nA permutation-invariant version (e.g., via minimum matching) would conflate isomorphic configurations and assign zero cost to degenerate cases—like total entity permutation—despite incoherence. For our purposes, permutation invariance is a feature. We will clarify this in the final version of the paper.\n\n---\n> **#2: Shape of $R(D_*)$**\n\n$D$ is an $n \\times n$ matrix over $n$ entities, where $D_{ij}$ is the Euclidean distance between representations of $x_i$ and $x_j$. We exclude relation tokens. We'll state this in the final version.\n\n---\n> **#3: Isomap vs. PCA**\n\nIsomap preserves geodesic distances along the manifold, which is essential for visualizing non-linear geometries (e.g., rings, torii) [1,2]. Meanwhile, PCA, being linear, can collapse or stretch such structures, obscuring effects like representation shattering. This makes us prefer Isomap for our analysis, since faithful projections of the original topology turn out to be critical for interpreting how KE affects global geometry. We use PCA as well when linearity suffices (see App. F.2).\n\n[1] https://www.nature.com/articles/s41593-019-0460-x\n\n[2] https://www.nature.com/articles/s41583-022-00642-0\n\n---\n> **#4: Held-Out Tokens and Logical Inference**\n\nWe note that \"held-out\" refers to specific facts, not entire relation tokens. Specifically, when generating the training dataset, we drop sequences that state one direction of a pair of conjugate facts with fixed probability $p$. That is, for any entity $x_i$ and relations $r, r'$, suppose the fact $(x_i, r, x_j)$ always implies $(x_j, r', x_i)$ (i.e. $r=$`I_C1` and $r'=$`I_A1`). Our DGP may drop *one of* these *facts* from the training data (with probability $p$). Even if $(x_j, r', x_i)$ is absent, one can still learn the conjugacy of $r$ and $r'$ and the existence of $x_i$ and $x_j$ via other examples; however, failure to infer this relation indicates the model has rote memorized relations rather than understanding the global structure.\n\nThis discussion is detailed in App. B, and we'll clarify this further in the final version.\n\n---\n> **#5: Bulk Edits and Avoiding Shattering**\n\nIn theory, we agree that a fully self-consistent batch of edits could reduce shattering. In practice, we observe:\n\n- KE methods like ROME and MEMIT maximize the likelihood of edited facts without explicit constraints on representation geometry. Like task-specific fine-tuning, this can induce interference and catastrophic forgetting.\n\n- Batch sizes are small because of compute limits. Edits are not independent or additive, so parallel updates can introduce inconsistencies, with degradation compounding over edits [1].\n\n- Enumerating all semantically connected facts is intractable. Editing \"Eiffel Tower is in ~~Paris~~ Rome\" would require accounting for every related event or entity; a self-consistent closure is undefined.\n\nThus, while comprehensive edits may reduce shattering, they do not eliminate it.\n\n[1] Yoon et al., 2024\n\n---\n> **#6: Prompt Format for Inference**\n\n- **Compositional**: The model is given a chain of relations (e.g., $x_i\\, r_1\\, r_2$) and must produce $x_k$, with $(x_i, r_1, x_j)$ and $(x_j, r_2, x_k)$ seen separately in training but not composed. Input: $\\text{ctx}\\, x_i\\, \\vec{r}$ where $\\vec{r} = r_1r_2$.\n\n- **Logical**: The model is asked about a fact $(x_j, r', x_i)$ where $(x_i, r, x_j)$ was seen but $r'$ was withheld. If the model understands relation symmetry, it can infer the inverse.\n\nWe'll further clarify these formats in the final version.\n\n---\n> **#7: Tree-Structured KG (App. G.3)**\n\nThank you for the suggestion to add a forward reference—we will do so in Sec. 4.5.\n\n---\n> **#8: Code Release**\n\nWe will release our code for the camera-ready version (see response to rVM5, #6).\n\n---\n> **#9: Typos**\n\nThank you! We'll fix these typos in the final version.\n\n---\n---\n**Summary:** We again thank the reviewer for their detailed review, which helped emphasize the design of our distortion metric $R(D_*)$, justify our visualization strategy, address inference task design, and explain the limits of bulk editing. We hope our rebuttals address the reviewer's questions and that they will champion our paper's acceptance!", "claims_and_evidence": "Yes, the claims made in the paper are supported by clear, convincing, comprehensive empirical evidence, both with quantitative measurements (vai representation distance metric $R(D)$) and qualitative visualization (via Isomap of model's internal representation).\n\nIt is noteworthy that the authors perform the study using multiple KE methods, with both pertaining a small LM and using already-pretrained LLMs, on both cycle-structured and tree-structured KGs. The comprehensiveness and extensiveness of the empirical effort make me confident that the representation-shattering phenomenon should be quite universal and general.", "methods_and_evaluation_criteria": "Overall I believe the proposed investigative method makes sense. However, I do have several concerns/questions regarding the evaluation metric. \n\nQ1: To quantify the extent of representation shattering, the authors proposed a metric measuring \"representation distortion\", which is defined as \n$$R(D_*) = \\frac{\\vert\\vert D_* - D_{\\emptyset} \\vert\\vert_F}{ \\vert D_{\\emptyset} \\vert_F  },$$\nwhere $D_{\\emptyset}$ and $D_*$ are the pairwise Euclidean distance matrix of the entities's vector representations in the original and unedited models respective.\n\nHowever, I don't fully understand what should we expect from this metric? Specifically, suppose there exists a perfectly edited model that somehow has its entities' internal representations' geometry maintained and avoids shattering. Shall we expect that this model have $R(D_*) = 0$? I guess the answer to this question is no. Take the edits shown in Figure 3 as an example. Suppose we are doing the following two counterfactual edits: \n\n1.I_C2 = 3→2, \n1.I_C3 = 2→3.\n\nThat is, suppose the counterfactual edits intend to exchange entity 2 with entity 3, or \"rename\" entity 2 to 3 and entity 3 to 2. Then, our perfect edited model still maintains the representation structure by simply exchange the representation of entity 2 with entity 3.\n\nIn this case, $D_*$ and $D_{\\emptyset}$ are different matrices, even though the set of all the entries (all the pairwise Elucidean distances) in these two matrices is the same. More precisely, $D_*$ and $D_{\\emptyset}$ are isomorphic up to permutation. However, **because R(D) does not account for permutations and it is not invariant to permutations**, $R(D_*)$ is not 0 for this perfect edited model. Furthermore, the specific value of $R(D)$ for this perfect model also depends on which pair of entities I exchanged via counterfactual edits, because different pairs of entity exchange will give difference matrix, $D_* - D_{\\emptyset}$.\n\nHence, I believe the $R(D)$ metric might be inconsistent in that it does not give a 0 value for the perfect model, and its specific value might be sensitive to the identity of entities involved in knowledge edits. If the $R(D)$ metric can be somehow made to be invariance to permutation to entity identities, I belive this issue could be resolved.\n\nQ2: Regarding the qualitative studies, the authors adopt the Isomap method. Is there a particular reason why Isomap is necessary, or the best dimensionality-reduction approach suitable for this task? For instance, is there a reason why other methods, such as simple PCA, wouldn't work or wound't be applicable?", "theoretical_claims": "This paper does not have theoretical components that require proofs.", "experimental_designs_or_analyses": "I checked the soundness/validity of the experimental designs and analyses. Overall, I believe the methodology makes sense. And it is noteworthy that the authors have done an arguably very extensive and comprehensive empirical study, covering multiple settings, models, KE methods, and knowledge structures.\n\nHowever, I have a question regarding the evaluation task and would appreciate the author's clarification:\n\nQ3: I don't fully understand why would the logical inference task in evaluation make sense. The authors describe this task as follows:\n\n> Logical inference accuracy measures the accuracy on a subset of held out relations that can be inferred from other relations\n\nIs it correct that these held-out relation tokens were never seen by the model during pre-training stage via next-token prediction? If so, wouldn't it mean that these held-out relation tokens are completely unseen by the model at inference time? If so, how does it make sense that we can expect the model to know what these held-out relation tokens mean, and to give correct answers on prompts involving these held-out relation tokens?", "relation_to_broader_scientific_literature": "This paper proposes a fundamental and principled understanding as to why KE methods in general introduce unwanted cascading effects in the model's internal knowledge and hence break the model's reasoning performance. \n\nI believe this is a very important contribution to the field, and will provide a guidance to future work on how to address these fundamental issue of KE.", "essential_references_not_discussed": "As far as I know, there is no other essential references not discussed. The paper's reference list is quite comprehensive.", "other_strengths_and_weaknesses": "Apart from the strengths mentioned in the previous sections, it's also commendable that the authors did quite a decent job in explaining how the dataset is constructed, in particular, what are the cyclic orders and what is the structure of the synthetic KG (Fig 3). This is quite complicated a conceptual construct but the exposition is easy to follow.", "other_comments_or_suggestions": "I have several other comments on readability and typos:\n\n- S1: (Page 5) Readability of the $R(D)$ metric definition: it is not immediately clear what is the shape of these distance matrices $D$, and that they are defined over all entity tokens (actually, is it true? Does it also account for relation tokens?)  It would be great to explicitly say something like \"$D_{ij}$ is a scalar value of the Euclidean distance between entity i and j's representations.\"\n\n- S2: (Page 5) Description of Evaluation (unseen facts): I feel that the description for this part is too vague, unclear and confusing. In particular, I did not understand what was the prompts for the compositional inference task and the logical inference task, and how are they different from each other. The authors did well in the previous paragraph (Evaluation (seen facts)). It would be great if the authors can also show, precisely, what does the input prompt for the compositional inference task and the logical inference task look like.\n\n- S3: I checked Appendix G.3 and the findings on the tree-structured KG is quite interesting. It would be great if the authors can mention these findings in one or two sentences in the main paper. I think this is quite important particularly because tree-structure KG is a more natural structure than cyclic KGs and more relevant to real world.\n\n - S4: some typos:\n    - (Page 3) Definition 3.1: it is written $f = (x_i, r, x_j) \\in R$ but R is the relation set. It should be $f = (x_i, r, x_j) \\in F$ because F is the fact set.\n    - (Page 4) 3.3 Experimental Setup - Data generation process $x_i r_i x_{i+1}$ should be $x_i \\vec{r_i} x_{i+1}$.", "questions_for_authors": "Please check my questions Q1, Q2, Q3 above. In addition, I have one other question:\n\nQ4: My takeaway from this paper is that knowledge edits, in particular counterfactual ones, shatter representation because they introduce inconsistencies.\n\nWhat if the \"user\" who makes these edits is more \"careful\" and makes a set of counterfactual edits that ensures consistency? For instance, say the \"user\" intends to exchange the name of two entities, and provides a set of comprehensive counterfactual edits where each affected relational triplet is included. With this careful \"user\", will the model's internal representation still be shattered? In other words, can we attribute the representation shattering to incomprehensive and/or inconsistent counterfactual edit queries, or is it that there is something else that is fundamentally wrong, that the internal representation of LLMs will invariably shatter even if the \"user\" is sufficiently careful and the edits are sufficiently comprehensive?\n\n---\n\nOverall, despite some concerns on the validity of the evaluation metric, I believe that the comprehensiveness of the empirical results shows that representation shattering should be quite a universal phenomenon and that this paper is an impactful contribution to the field. Thus, I recommend this paper for acceptance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cL6pTmV8W7", "reviewer": "Reviewer_rVM5", "summary": "This paper explores the impact of Knowledge Editing (KE) on Transformer models and introduces the concept of Representation Shattering. The authors argue that modifying specific facts in the model will destroy its broader internal knowledge structure, leading to reduced fact recall and reasoning capabilities. The authors design a synthetic task using structured knowledge graphs to systematically analyze this effect and validate it on pre-trained LLMs (Llama and Mamba), showing that KE leads to widespread distortions in model representations. This paper provides some empirical and theoretical insights into the potential mechanisms of knowledge editing failures.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our experiments and analysis of representation shattering. Please see our responses below.\n\n---\n\n> **#1: Scope of Models and Methods Tested**\n\nWe emphasize our evaluation already spans multiple architectures and scales: small Transformer models trained from scratch on synthetic knowledge graphs, pretrained decoder-only models without fine-tuning (GPT-2), pretrained decoder-only models with instruction fine-tuning (Llama 3 8B Instruct), and pretrained structured state-space models without fine-tuning (Mamba 2.8B). All consistently exhibit representation shattering that scales with counterfactual edit distance (see Sec. 4.5 and App. G.1).\n\nRegarding the reviewer's suggested settings, we note the following.\n\n- **BERT**: Modern KE methods exclusively focus on causal decoder-only models (ROME, MEMIT, PMET, AlphaEdit), due to which bidirectional encoder models like BERT are not amenable to analysis under the purview of our work.\n- **GPT-4**: We note GPT4 is a closed-source model. This precludes parameter-space editing and replicating our methods.\n- **Mixtral**: We note our Llama 3 8B experiments already feature similar scale, decoder-only construction, and instruction tuning as a Mixtral model. While MoE architecture effects are interesting, there is no precedent for such an experiment, so this lies outside our scope for now.\n\n- **RAG and WISE**: RAG, RETRO, and KE protocols like WISE operate under different assumptions (e.g., inference-time components). Thus, they extend beyond our current scope of analyzing direct parameter-editing methods. \n\nNevertheless, we agree the directions above are promising. We'll include these in an expanded \"Future Directions\" section, citing WISE. \n\n---\n\n> **#2: Layerwise Analysis of Representation Shattering**\n\nCurrent KE methods use causal tracing to identify the layer to edit, often focusing on early MLP layers. Modifying this step would require a new KE protocol, which is beyond our scope. Nevertheless, our manuscript includes visualizations from different sub-layers and depths (see App. F.3 and G.2) that show the spatial extent of shattering. We are open to extending these analyses if helpful (e.g., as outlined in response to reviewer kjUD, #2).\n\n---\n\n> **#3: Visualization Methods**\n\nWe refer the reviewer to our comment justifying the use of Isomap in response to reviewer gDBJ, #3. In brief, we use Isomap due to its ability to preserve geodesic distances, hence observing how cyclical manifolds distort under KE. UMAP/t-SNE show clustering, but Isomap's focus on global geometry better captures topological collapses from KE. \n\n---\n\n> **#4: Theoretical Explanation for Transformer Fragility under KE**\n\nThough mainly empirical, our study aligns with a broader theoretical framework.\n\n- Transformers store factual associations in key-value pairs within MLP layers [1, 2] corroborated by our synthetic model analysis (see also response to rub2, #4).\n- Parameter sharing and superposition cluster unrelated facts in overlapping subspaces [1], making them vulnerable to unintentional interference.\n- Entities and relations often form structured manifolds (e.g., cycles, hierarchies), which aid compositional inference [4].\n- KE methods (ROME, MEMIT, etc.) enact local weight updates that deform these manifolds, causing representation shattering for unedited facts residing in shared sub-regions.\n\nHence, we believe fragility arises from the entangled, compressed nature of factual storage, rather than from simply retaining knowledge. We will include this discussion in the final version of the paper.\n\n[1] Geva et al., 2020\n\n[2] Meng et al., 2022a\n\n[3] https://transformer-circuits.pub/2023/toy-double-descent/index.html\n\n[4] Engels et al., 2024\n\n---\n\n> **#5: Evaluation on Tree-Structured Knowledge**\n\nApp. G.3 explores KE on a tree-structured city–country graph using GPT-2 XL. While trees are more challenging to visualize than cyclical manifolds, smaller-distance edits (e.g., relocating \"Paris: to \"Spain:) exhibit less shattering than those with larger distance (e.g., \"UK:). This supports our hypothesis that shattering severity scales with the manifold distance of the edit.\n\n---\n\n> **#6: Supplemental Materials and Code Release**\n\nSee Apps. A/B/C for documentation on pseudocode, hyperparameters, data, and architectures; we are happy to provide more!\n\nWe also highlight App. A.1, where we state our commitment to publicly releasing the source code for our experiments—covering both synthetic and naturalistic settings—in the camera-ready version of this paper. This ensures transparency and reproducibility.\n\n---\n---\n\n**Summary:** We again thank the reviewer for their insightful comments that have helped us clarify our design choices, the constraints of current KE layer selection methods, our use of Isomap, and the theoretical underpinnings of Transformer fragility. We hope our rebuttals address the reviewer's questions and that they will champion our paper's acceptance!", "claims_and_evidence": "1. Knowledge Editing harms the overall capability of the model, not just the modified facts.\n- The author tested on direct recall, logical reasoning, and combinatorial reasoning tasks and found that the accuracy of the model dropped significantly after KE.\n2. Claim: Representation fragmentation is the core mechanism that causes model knowledge degradation.\n- The paper measures the change in model representation before and after KE by the Frobenius norm difference and finds that it is strongly correlated with the decline in model performance.\n3. Claim: The larger the edit distance, the more severe the representation fragmentation.\n- Evidence: On the synthesis task and LLM task, the study found that modifying facts with a longer distance (such as \"January → February\" vs. \"January → June\") is more damaging to representation.\n4. Limitations: The paper mainly verifies this phenomenon from an experimental perspective, but the theoretical explanation of why the internal mechanism of the Transformer is so fragile still needs to be further improved.", "methods_and_evaluation_criteria": "The authors use Structured Knowledge Graphs as a Toy Setting for Investigating the Impact of KE.\nThe evaluation includes three key metrics: direct recall, logical inference, and compositional inference.\nThe study did not consider real tasks (complex logic), which limits the generalization ability to real-world applications.", "theoretical_claims": "The study shows that the model's knowledge representation is structured rather than independent facts stored in isolation, and verifies that the larger the edit distance, the more severe the representation fragmentation, and provides experimental support. The paper lacks an explanation for why the Transformer representation is so susceptible to KE.", "experimental_designs_or_analyses": "Using Structured Knowledge Graphs provides a highly controllable KE research environment. The results are verified on the pre-trained Llama and Mamba, which improves the external validity of the research. More Transformer variants (such as Bert, GPT-4, Mixtral) need to be tested to enhance the generalizability of the conclusions. The paper does not deeply analyze how different Transformer components (such as MLP vs. attention head) are affected by KE. It is recommended to add more micro-level experiments.", "relation_to_broader_scientific_literature": "The research results are related to issues such as AI reliability and knowledge plasticity and have made certain contributions to AI security research.", "essential_references_not_discussed": "WISE: Rethinking the Knowledge Memory for Lifelong Model Editing (https://arxiv.org/abs/2405.14768)\n\nThis paper proposes the WISE algorithm, which achieves an effective bridge between long-term memory and working memory in large language models by designing a dual-parameter memory scheme and a knowledge sharding mechanism to solve the impossible triangle problem of reliability, generalization and locality in lifelong model editing.", "other_strengths_and_weaknesses": "Strengths:\n1. Designed novel synthetic tasks that can precisely control the KE effect.\n2. Provided clear empirical analysis and proposed methods to quantify representation fragmentation.\n3. It has important implications for the design of future KE methods.\n\nWeaknesses:\n1. Mainly based on small-scale synthetic tasks, the applicability to real-world applications still needs further verification.\n2. It did not explore the underlying mechanism of Transformer representation construction in depth, and the theoretical explanation can still be improved.\n3. The experiments are all based on pre-trained models, and fine-tuning can be considered; the paper does not provide a detailed ablation study to isolate the impact of pretraining versus fine-tuning.", "other_comments_or_suggestions": "Some suggestions: Extend theoretical analysis to explain why Transformer representations are so susceptible to KE. Test the impact of KE on retrieval enhancement models (such as RAG, RETRO) to explore whether external memory can alleviate representation fragmentation.\nCompare with the latest KE method (WISE) to verify the applicability of the research conclusions.", "questions_for_authors": "1. Is there a difference in representation fragmentation between different layers? For example, does KE have the same effect on MLP layers vs. attention layers?\n2. The cyclic knowledge in the article topic is not common in the real world. How does it perform when faced with complex tree-structured knowledge?\n3. Can you provide UMAP/t-SNE visualizations before and after KE? Intuitively show how representation fragmentation occurs.\n4. Can retrieval enhancement architectures (such as RAG, RETRO) reduce representation fragmentation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r7HZTLmNRp", "reviewer": "Reviewer_rub2", "summary": "This paper explores the empirical finding that knowledge editing can degrade the general capabilities of large language models. The authors perform a synthetic in the setting of structured knowledge graphs (i.e. knowledge graphs with a cyclic or tree structure). They make the finding that the latent space activations of models adapt to the global structure of the knowledge graph. Next, they show empirically that both (a) corrective and (b) counterfactual edits shatter these representations. As a result, the post-editing performance of the model on both factual accuracy and reasoning capabilities becomes severely disrupted. The level of disruption is correlated with a notion of the \"size\" of the edit -- that is how far it diverges from the ground-truth knowledge graph. Finally, they provide evidence that this phenomenon also holds on a real Llama model on a task with a cyclic knowledge graph structure.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our experiments and analysis of representation shattering. Please see our responses below.\n\n---\n\n> **#1: Generality and Tree-Structured KGs**\n\nWe acknowledge that our tree-based experiment in App. G.3 is preliminary, but believe further exploration of tree-shaped KGs is best reserved for follow-up work. Specifically, our paper aims to establish representation shattering, define evaluations to detect/visualize it, and demonstrate correspondence between synthetic and large-scale models. Cyclic graphs are uniquely suited for this: they are the simplest non-trivial graphs with global geometric constraints where post-edit effects can be exhaustively elicited. By contrast, trees lack this symmetry and introduce complexities around hierarchical relations, equivalence classes, and edge cases for root/leaf entities.\n\nWe agree that tree-shaped KGs are an important next step, and we intend to pursue this direction. For this work, however, we prioritize simplicity—our cyclic results isolate the essence of representation shattering for the community to build on.\n\n---\n\n> **#2: Mechanistically Causal vs. Correlational Evidence**\n\nWhile several of our experiments elicit correlational evidence, we emphasize our counterfactual editing experiments suggest a causal role for representation geometry in model capabilities (see Sec. 4.4/4.5, esp. Tab. 2, Fig. 6, Fig. 7). We summarize these results below.\n\n**Summary of counterfactual experiments.** We systematically vary counterfactual edit distance (CE distance) while holding all other variables fixed. For a given subject entity, edits with larger CE distances imply greater displacement in the representation manifold and higher $R(D_*)$. In reverse, higher $R(D_*)$ implies a larger CE distance. This manipulation approximates an intervention on representation geometry itself.\n\nAcross all cases, higher CE distance leads to greater representation shattering and larger degradation in direct recall, logical inference, and compositional inference. While we cannot rule out models with scattered but functional representations, it is unclear how to obtain such a model to test this. We argue that the manipulation of CE distance—which directly impacts geometry—offers strong evidence for a causal link between shattering and performance loss.\n\nWe will clarify that our experiments test for behaviors consistent with a mechanistic explanation, though we do not claim formal proof.\n\n---\n\n> **#3: Realism of the \"Months\" Task**\n\nWe agree the months-of-the-year task is simpler than real-world KE scenarios involving entity attributes. We selected it to isolate and quantify representation shattering in pretrained models.\n\nThough somewhat artificial, the fact that shattering arises in this clean cyclic setting suggests that richer domains—where knowledge is denser and more interdependent—would experience more severe effects. That is, if shattering already causes degradation in this minimal case, it likely plays a role in realistic tasks too. While we don't claim it explains all KE failures, our results show shattering is a major contributor. Exploring broader domains remains an important future direction (Sec. 5).\n\n---\n\n> **#4: Layer Choice & Key-Value Assumption (ROME/MEMIT)**\n\nIn all synthetic experiments, we use a 2-layer nanoGPT Transformer (explicitly stated in Sec. 3.3) Given this architecture, layer 1 is the only editing site compatible with the early-site assumptions of ROME. Further, causal tracing and MEMIT use a sliding window of layers: in a 2-layer model, any non-1 window size would encompass the whole model. We thus used layer 1 to avoid confusion, but early sanity checks with a singleton window re-implementation of causal tracing also confirmed layer 1 as the primary factual recall site. If the reviewer believes these results should be included, we can prepare them for the final version.\n\nWe also qualitatively verified the key-value assumptions in exploratory stages. Prompts about the same entity yield clusters in the input space of the layer 1 MLP; prompts with different subjects resolving to the same object yield clusters in its output space. These patterns support the key-value framework underlying ROME/MEMIT as compatible with our synthetic setup. If it will help address reviewer's concerns, we are happy to add visualizations confirming these assumptions.\n\n---\n\n> **#5: Code Release**\n\nWe will release our code for the camera-ready version for reproducibility (see rVM5, #6).\n\n---\n---\n\n**Summary:** We again thank the reviewer for their thoughtful feedback that helped us clarify the rationale behind our synthetic design, show how our experiments suggest a causal link between representation geometry and model performance, acknowledge the simplicity of our Llama task as deliberate, and justify our editing layer choices and key-value assumptions. We hope our rebuttals address the reviewer's questions and that they will champion our paper's acceptance!", "claims_and_evidence": "Claim 1) On synthetic structured knowledge graphs, the latent space encodes the geometry of the knowledge graph. \n\nThis is evidenced by low-dimensional visualizations of the latent space activations. In the main body of the paper, this is primarily shown on relatively simplistic cyclic structure. In the appendix, the authors additional provide some analysis of tree-like structures. While the cyclic structure does convincingly appear in their experiments, the tree like structure results are not convincing at all -- they only appear to show single examples and rely on very unclear and hard to discern arguments. It would be nice to have averaged R(D) measures for this setting just as was presented in the cyclic case. Hence, while they support their claim in a very limited setting of cyclical knowledge graphs, the general claim of explaining the failures of knowledge editing is not supported by this submission. \n\n\nClaim 2) Knowledge editing techniques distort the geometry of the knowledge graph -- leading to degradation in the model. The extent to which this distortion occurs is captured by a \"distance\" of the edit (relative to the underlying KG). \n\nThis is primarily accomplished by way of qualitative analysis of feature visualization. Indeed, they do show that qualitatively, the visualization of the latent space changes structurally in a significant way. However, they do also quantify this using their measure of representation distortion. Thus, for the case of cyclical knowledge graphs, I believe the first claim is supported. The authors also observe that the amount of representation shattering correlates strongly with the extent of degradation of the model's capability. However, I do not believe that this actually establishes any \"mechanistic\" or causal claims about the change in representation geometry leading to worse performance which the authors claim is the case. This could be established better, for example, if the authors were able to show that the structure in the representation is **essential** for performing the evaluation tasks they consider. However, their current experiments do not rule out that there exists solutions with a different (or even disordered) latent space that could still perform the evaluation tasks. \n\n\nClaim 3) Knowledge Shattering Occurs in Real LLM\nThe authors establish this in a pretrained Llama model on the representation of months. They show that months occupy a cyclic structure in the latent space and that applying ROME/MEMIT  to reassign different month names to different times in the year distorts this representation. Although this analysis is somewhat interesting, the task does not resemble the typical use-cases of knowledge editing (such as editing attributes of people/places/entities). In particular, this task appears to have been chosen due to its simple cyclical structure. For these reasons, this task in my opinion is insufficient to establish that representation shattering is actually the contributing phenomena for the \"real world\" tasks of knowledge editing.", "methods_and_evaluation_criteria": "The evaluation criteria appears reasonable. As detailed below, I have questions about (a) why is ROME/MEMIT an appropriate method to use here and why was the layer for editing selected via the causal tracing mechanism (or tuned over).", "theoretical_claims": "Not applicable", "experimental_designs_or_analyses": "I have significant concerns about the experimental methodology used to evaluate ROME. The authors suggest in the supplementary material that they only try one choice of layer (the first layer) in their synthetic experiments because it is the only \"early-site\" layer. In general, I am somewhat confused about why this is the case and why the authors did not use the \"causal tracing\" mechanism introduced by the original ROME paper to select the location of the edit. In addition, the authors failed to list the number of layers used in the models for their synthetic setting--this also makes it hard to evaluate the validity of the experimental design.", "relation_to_broader_scientific_literature": "This paper does related to an important collection of prior works regarding model degradations induced by model editing. The authors claim to provide a mechanistic hypothesis for these observations. However, the evaluation is not sufficiently robust to actually convince the reviewer that the mechanism they propose is actually the cause of the widely observed shortcomings of model editing in real-world settings.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "At a high level, I have objections surrounding the underlying motivation for this work. As the authors specifically acknowledge in their appendix. ROME and related techniques are predicated on the functionality of MLP layers as implementing key-value storage via a linear association matrix. Prior  empirical works  [Geva, Mor, et al. \"Transformer feed-forward layers are key-value memories.\" arXiv preprint arXiv:2012.14913 (2020); Geva, Mor, et al. \"Dissecting recall of factual associations in auto-regressive language models.\" arXiv preprint arXiv:2304.14767 (2023).] have substantiated these hypotheses extensively (including the original ROME paper). The authors in this case propose a new task with global geometry of the latent space -- without verifying that it satisfies the underlying assumption of key-value storage-- and then show that ROME/MEMIT fail. This is unsurprising and it limits the applicability of this work to the many cases in which factual knowledge has actually been seen to obey the key-value storage assumption, but ROME/MEMIT still results in degradation. To put it another way, this paper appears to propose a setting (without adequate real-world justification of its validity) in which the assumptions of ROME/MEMIT are unsatisfied and then show that ROME/MEMIT fail. Can the authors comment a bit more about why we would expect ROME to be able t perform an edit in cases such as this where there is global structures?", "other_comments_or_suggestions": "This paper could also be made more stronger if the authors could demonstrate -- maybe in a very simplified theory setting-- why the geometry of the latent space is essential for solving the task. This would strengthen their causal claims about representation shattering being responsible (causally) for model degradation.", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5mUfFBWOJe", "reviewer": "Reviewer_kjUD", "summary": "This paper proposes a synthetic framework to investigate knowledge editing in Transformer-based language models. The authors learn a model from structured knowledge out of a graph featuring multiple cyclic orders of entities and relations of facts. By evaluating representation changes following targeted edits to specific facts, the authors identify a phenomenon termed “representation shattering”: editing even a single fact can significantly distort the learned geometric manifold of entity embeddings, and decrease factual recall and reasoning performance. They further demonstrate that this decrease increases with the geometric distance or counterfactuality of edits, where edits that more drastically conflict with the previously learned geometry cause greater disruption. Finally, the synthetic findings are validated through experiments on real LLMs (Llama and Mamba) by editing the cyclic ordering of months, where edits inducing greater distortion lead to worse performance on benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our empirical analysis and mechanism of representation shattering. Please see our detailed responses to specific comments below.\n\n---\n\n> **#1: Multi-Step Editing**\n\nThank you for this suggestion! We note we already provide an analysis of multi-step and layerwise edits with real LLMs (Llama 3 and Mamba in Sec. 4.5 and App. G.1) via MEMIT:\n\n> For each counterfactual edit distance $d$, we have $(\\text{twelve months}) \\times (\\text{three offsets}) = 36$ total edits—we apply these sequentially in batches of size $4$ using MEMIT, using consistent sample orders and batch orders for fair comparisons.\n\nThis was done to magnify performance degradations and representation shattering, as singular edits yielded minute effects initially. This qualitatively suggests shattering amplifies in multi-step editing, but a rigorous comparison of Batch vs. Sequential vs. Sequential-Batched editing (as in [1]) is warranted. We will incorporate such comparisons into future work.\n\n[1] Yoon et al., 2024\n\n---\n\n> **#2: Layerwise Analysis**\n\nCurrent KE methods (ROME/MEMIT) specify early layers (typically early-layer MLPs) as edit sites based on causal tracing. Altering this would require formulating and validating a new KE approach, beyond our scope. We do note that we include analyses in App. F.3 and G.2 visualizing representations from various model sites, hence showing the effects of KE across layers. \n\n\n---\n\n> **#3: Connections to Catastrophic Forgetting Mitigation**\n\nThank you for highlighting connections to forgetting mitigation strategies! Indeed, our framework can naturally evaluate such strategies: effective forgetting mitigation methods should preserve global representation geometry when instilling new knowledge (minimizing $R(D_*)$). Recent work in fact examines regularization-based KE methods (e.g., [1]), suggesting compatibility with classical forgetting mitigation literature [2]. A thorough analysis in the context of our work is however best deferred to a future paper.\n\n[1] Gu et al., 2024\n\n[2] https://arxiv.org/abs/1612.00796\n\n---\n\n> **#4: More Fine-Grained Tasks Beyond Month Ordering**\n\nWe appreciate your acknowledgment that the months-of-the-year task was intentionally chosen due to its clear interpretability and structured cyclic geometry. This choice enabled controlled experiments directly linking representation geometry and performance. \n\nHowever, we fully recognize your interest in finer-grained setups. We anticipate more realistic KE tasks—typically involving densely interconnected entities and facts—would exhibit even more severe representation shattering. Since greater interconnectedness implies more overlapping representations and deeper manifold entanglement (elaborated in our response to reviewer (see rVM5, #4), we expect edits in realistic scenarios to cause geometric distortions at least as severe as our cyclic-month edits. Our current findings represent a conservative estimate of the impact representation shattering can have in practical, large-scale knowledge editing scenarios; we plan to explicitly explore these richer domains as an immediate follow-up.\n\n---\n\n> **#5: Generalization Beyond Cyclic Structures**\n\nWe acknowledge your concern regarding our emphasis on cyclic structures, and fully agree that exploring other structures (e.g., trees or hierarchies) is crucial. Our current focus on cycles is deliberate: cycles are the simplest nontrivial graphs exhibiting global geometric constraints, ideal for precisely measuring and visualizing representation shattering. Their symmetric nature mitigates extraneous considerations like hierarchical relations, equivalence classes, and edge cases for root/leaf entities.\n\nAs you note, our preliminary experiments on tree-structured KGs (App. G.3) align closely with our cyclic results: edits causing larger geometric displacement yield greater representation shattering (e.g., moving \"Paris\" to UK vs. Spain). Thus, we strongly believe the core phenomenon generalizes beyond cycles. Still, more systematic evaluations with alternative relational structures is needed—this is a key next step we intend to pursue in future work.\n\n---\n\n> **#6: Code Release**\n\nWe are committed to releasing our code for the camera-ready version for reproducibility (see rVM5, #6).\n\n---\n---\n\n**Summary:** We again thank the reviewer for their insightful feedback that has helped us emphasize the existing multi-step editing and layerwise analyses in our work, discuss catastrophic forgetting literature, and affirmed our intent to explore more complex setups. We hope our rebuttals address the reviewer's questions, and that they will continue to champion our paper's acceptance!", "claims_and_evidence": "The authors claim that models acquire coherent representations of relational data, effectively capturing the underlying structural geometries. To this end, they use cyclic graphs. Using dimensionality reduction (Isomap) visualizations, they illustrate that entity embeddings naturally form clear ring-shaped geometries aligned with the cyclic structure of the data. The models also exhibit strong performance in logical and compositional inference tasks, indicating that they internalize structural patterns beyond simple memorization of isolated facts.\n\nAdditionally, the authors argue that knowledge editing methods (such as ROME and MEMIT) inadvertently harm performance on unrelated facts and reasoning tasks, extending well beyond the specific fact being edited. They systematically quantify these accuracy reductions across direct recall (facts seen during training), logical inference (such as relation reversals), and compositional inference (multi-hop reasoning), consistently finding that edits negatively impact accuracy even for unrelated information.\n\nThe underlying cause of these degradations is identified as “representation shattering”: forced edits distort the learned embedding geometry. Through visualizations and quantitative measures, the authors demonstrate that structural disruption increases with the magnitude of edits. They introduce a metric, R(D^*), to measure the degree of embedding shifts post-editing, revealing a strong correlation between embedding distortions and performance deterioration.\n\nFurthermore, the authors demonstrate that these findings generalize to real-world large language models (e.g., Llama 3 and Mamba S4). Specifically, edits applied to the cyclic ordering of months in real models yield similar representational distortions and degraded performance on external benchmarks (MMLU-Redux), echoing the synthetic experimental results.\n\nOverall, I find these claims are supported by both synthetic experiments and validations on real models, evaluating changes in representation as observed performance losses.", "methods_and_evaluation_criteria": "Method:\nThe paper sets up a cyclic synthetic knowledge graph and creates various k-hop relations (e.g., 2-hop clockwise neighbor) to form several subgraphs (edit, retain, test). This framework appears carefully designed to mimic real-world relational data while being simple enough to visualize and measure precisely.\n\nEvaluation: \nThe authors compute (1) direct recall, (2) logical inference, and (3) compositional inference accuracy. This is appropriately motivated: they state that knowledge editing must preserve (or at least minimally disrupt) both memorized facts and the underlying relational inferences. \nThe authors define a Frobenius norm–based measure to quantify how much embeddings shift before versus after an edit. This is a simple but intuitive way to measure global manifold changes.\n\nI find both the evaluation and methodology are reasonable for the approach.", "theoretical_claims": "The paper does not provide formal proofs but a mechanistic viewpoint relying on the hypothesis that transformer models embed related entities in geometric manifolds and that forcibly altering facts shifts these embeddings leading to general performance drops. This motivations and also demonstrations in controlled settings lend credibility to their hypothesis.\n\nGiven the nature of the paper, there are no deep theorems or long proofs to check. The “representation shattering” phenomenon is presented more as an empirical discovery and conceptual explanation than a formal claim that requires a proof. Hence, there are no apparent issues with unverified theoretical claims.\n\n⸻", "experimental_designs_or_analyses": "The synthetic dataset is systematically constructed to cover direct, logical, and compositional facts. The authors vary the distance of counterfactual edits in the cyclic order to systematically test how conflict magnitude affects distortions to analyze edit distance. They replicate their findings on Llama 3 and Mamba S4, focusing on month ordering as a real-world example. The main limitation is that these evaluations are primarily small-scale or specifically for months, though the authors mention an extension to tree-structured data in the appendix. Still, I find the analysis consistent and well-motivated.\n\nOverall, the experimental methodology is appropriate and the analyses appear valid. The results are well-documented, showing strong negative correlations between manifold distortion and model performance.", "relation_to_broader_scientific_literature": "The paper relates to existing knowledge editing approaches, such as ROME and MEMIT. It highlights common pitfalls like catastrophic forgetting and representational contradictions, such as the ones from [1], [2], [3].\n\n\n\nThe work relates directly to existing studies on how Transformers encode and manipulate structured conceptual representations, such as [4] who explore interpretations of feed-forward layers as key-value memory stores and [5], who aim to contextualize why local edits (e.g., rewriting a single fact) influence a model’s stored knowledge. \n\nThe synthetic dataset generation approach relates to previous work, such as [6] investigating compositional reasoning and interpretability in Transformer models.\n\nOverall, I find the authors do a good job covering related work to ground their analysis. \n\n[1] Hase, P., Bansal, M., Kim, B., and Ghandeharioun, A. Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models. arXiv preprint arXiv:2301.04213\n[2] Gu, J.-C., Xu, H.-X., Ma, J.-Y., Lu, P., Ling, Z.-H., Chang, K.-W., and Peng, N. Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue. arXiv preprint arXiv:2401.04700, 2024.\n[3]Cohen, R., Biran, E., Yoran, O., Globerson, A., and Geva, M. Evaluating the Ripple Effects of Knowledge Editing in Language Models. arXiv preprint arXiv:2307.12976\n[4] Geva, M., Goldberg, Y., & Berant, J. (2020). Transformer feed-forward layers are key-value memories. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).\n[5] Allen-Zhu, Z. & Li, Y. (2023). Physics of language models\n[6] Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task.", "essential_references_not_discussed": "I find the relevant related work sufficiently discussed.", "other_strengths_and_weaknesses": "Strengths:\nWhile the idea is not entirely novel and bills and work from interpretability research, the author's present a clearly structured framework with multiple inference types and evaluate embedding distortions with performance degradation. The paper is well-structured and as such clear to read. The step from synthetic to realistic scenarios in real-world models underpins the approach.\n\nWeaknesses:\nWhile a realistic scenario is evaluated, the exploration in month ordering remains limited. The focus lies on cyclic structures, with minimal emphasis on broader relational types, which would be interesting to gain more insights and also gain more insights to mitigate representational shattering.", "other_comments_or_suggestions": "Exploring connections to classical catastrophic forgetting mitigation techniques, such as mentioned in the related work (e.g., elastic weight consolidation, low-rank updates) as the author is mentioned for future work, might yield useful insights. Also a more fine-grained analysis of multi-step and layer-wise edits would be interesting.", "questions_for_authors": "How would you expect the approach to generalize beyond cycles? Similarly, how would you expect the approach to work on more fine-grained tasks than the month ordering?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors develop a hypothesis (and evidence supporting it) regarding why cascading errors ensue after knowledge editing (KE) in LLMs. To accomplish this, they introduce the concept of representation shattering, the case where modifying the representation of the targeted entity distorts the representation of other related entities, thus distorting the relationships between entities and thus the ability to infer unseen (explicit) information between entities within this relation network. They specifically examine the case of structured information in knowledge graphs and show that 'good' pre-trained models with exhibit geometric structures that are consistent with the underlying KG structure. However, KE methods even when applied to a small number of these entities 'shatter' this geometric structure and thus introduce cascading errors. Empirical validation is performed with multiple widely-studied KE methods (e.g., ROME, MEMIT, PMET, AlphaEdit) with multiple (L)LMs (e.g., a 2-layer transformer network, Llama3-8B, Mamba) based on cyclic-structured and tree-structured knowledge graph content (in synthetic experiments).

The strengths of this submission identified by the reviewers include:
- This paper builds on recent explorations for LLMs based on synthetic data generation processes (more widely used for interpretability studies) to provide some theoretical insight into why knowledge editing hasn't proven successful at scale, despite being intuitively appealing. While KE hasn't proven successful, this is the first work any of us are aware of that provides an explanation.
- The study includes multiple KE methods as applied to  a simple transformer LM and existing pre-trained LLMs on multiple KG structures. The visualizations and quantitative results are consistent across experiments, providing solid evidence for 'representation shattering' being a strong hypothesis.
- The experimental procedures are well explained and the paper is well-motivated and well-written overall.

Conversely, limitations of this work identified by the reviewers include:
- This work doesn't provide solutions for mitigating representation shattering (even in discussion regarding future directions). Do we expect solutions to be via the KE protocol or in the transformer architecture (or some combination).
- While a non-negligible step in understanding, this is still limited theory that is well-supported empirically, but there isn't strong analytical analysis. As reviewer rub2 points out, it isn't shown that the structure in the representation is essential (as it isn't identifiable) as scattered representations may still be functional (and performant). However, this was well-discussed during rebuttal.
- The experiments are for very controlled settings and there isn't even discussion regarding how this relates to more complex reasoning tasks (i.e., are the constructed settings illustrative, representative, or adversarial?). This was also well-discussed during rebuttal.

Overall, I think this is a promising work that will likely inspire further work, potentially across multiple LLM characteristics. Using synthetic data generation procedures to examine the dynamics and relationships between representation and generation is relatively new and this study was well-constructed. Assuming that knowledge editing remains as a potential solution to information/knowledge updating, the representation shattering formalism may point a way to mitigating such procedures (e.g., with some sort of representation regularization, data mixing, etc.). Additionally, such findings could lead to a general understanding of 'updating LLMs' or training procedures in general. Thus, while the findings could be interpreted as preliminary (i.e., not actionable), I think the community would find the results and procedure interesting and potentially insipring.
Decision: Accept (poster)