SUIT: Knowledge Editing with Subspace-Aware Key-Value Mappings
Authors: Haewon Park
Sangwoo Kim
Yohan Jo
Venue: ICLR 2026 Poster (2026)
Keywords: Knowledge Editing
Large Language Model
Subspace
Primary area: interpretability and explainable AI
TL;DR: Subspace Knowledge Edit (SUIT) identifies and modifies only the subspace of critical features relevant to the edit, dramatically improves knowledge preservation over strong baselines while maintaining high edit efficacy.

Abstract: Knowledge editing aims to efficiently correct factual errors in language models. Widely used locate-then-edit methods update an MLP layer by adjusting its weights to change the mapping between the layer’s input vector (key) and output vector (value), thereby editing the model’s knowledge. 
As this update is driven by key and value vectors, obtaining these vectors without careful constraints causes significant model perturbations beyond the targeted edit, a common issue in many prior knowledge editing methods.
To address this, we propose Subspace Knowledge Edit (SUIT), which computes key and value vectors only within the subspace of critical features relevant to the edit. Our empirical results on LLaMA3, GPT-J, and Qwen2.5 models show that SUIT dramatically improves knowledge preservation over strong baselines while maintaining high editing performance. These results support the claim that SUIT successfully identifies the critical subspace for the edit. 
Beyond quantitative gains, our analyses show that SUIT reduces unintended perturbations in hidden states while confining updates to directions that are more effective for editing.
Taken together, these findings establish edit-critical subspace identification as a key principle for reliable, low-perturbation knowledge editing.
Our code is available at https://github.com/holi-lab/SUIT.

Published as a conference paper at ICLR 2026
SUIT: KNOWLEDGE EDITING WITH SUBSPACE-AWARE KEY- MAPPINGS
Haewon Park; Sangwoo Kim"i, Yohan Jo Graduate School of Data Science, Seoul National University (dell GGanlaaniii ma2, hemy01 01, yohan. jo) @nnu. ac.kr kr
ABSTRACT
Knowledge editing aims to efficiently correct factual errors in language models. Widely used locate-then-edit methods update an MLP layer by adjusting its weights to change the mapping between the layer's input vector (key) and output vector (value), thereby editing the model's knowledge. As this update is driven by key and value vectors, obtaining these vectors without careful constraints causes significant model perturbations beyond the targeted edit, common issue in many prior knowledge editing methods. To address this, we propose Subspace Knowledge Edit (SUIT), which computes key and value vectors only within the subspace of critical features relevant to the edit. Our empirical results on LLaMA3, GPT-J, and Qwen2.5 models show that SUIT dramatically improves knowledge preservation over strong baselines while maintaining high editing performance. These results support the claim that SUIT successfully identifies the critical subspace for the edit. Beyond quantitative gains, our analyses show that SUIT reduces unintended perturbations in hidden states while confining updates to directions that are more effective for editing. Taken together, these findings establish edit-critical subspace identification as a key principle for reliable, low low-perturbation know wedge editing. Our code is available at
com wwwphwww...ooooooooohooaaa http:holliiia SUIT
INTRODUCTION
Large language models (LLMs) retain and recall substantial factual knowledge. However, they often produce incorrect statements due to noisy training data or a temporal shift (Maynez et al.) 2020 Ji let al. 2023 Lin et al. 2022). These errors reveal gaps and temporal drift in the model's knowledge, indicating the need for edits to correct these issues. Fine-tuning is commonly used for this purpose, overfitting and catastrophic forgetting (Zhagg let al. 2024. [Bethune et al. 20225 Luo et al. 2025). Consequently, knowledge editing methods have emerged promising alternative, enabling targeted edits of specific knowledge while preserving the rest (aao et al.) 2023] |Wang et al. 2024b). Among various knowledge editing methods (Wang al., (2024)), our work builds on the locate-then-edit approach. This approach first identifies the weights that store the knowledge, and then directly updates them to apply the edit. It has shown high precision in both mass and sequential editing (Meng et al.) 2023b Fang et al. 2025).
but it is computationally costly and susceptible
To explain this process, we first describe the standard knowledge editing setup. Given factual tuple (e, r, a),, where e an entity, r a relation, and a an attribute, knowledge editing replaces the old attribute with a new attribute a* For example, for the entity (e, "Chrome") and the relation (r, "was developed by"), the old attribute (a. "Google") can be edited to the new attribute methods, this editing is performed by viewing the Trans- former MLP's ' projectoo matrix W as linear associative memory (Anderson| [972] Kohonen 1972 Meng et al. 2023a), where W maps key vectors to value vectors. In these methods, the key
(a* "Apple") "). Within
"Equal contribution.
1 Work done while the author was an intern from the Department of Linguistics, Scoul National University. *Corresponding author.
'Much of the knowledge editing literature represents facts in (subject, relation. object) form. However, facts in knowledge editing do not always correspond directly to this form, we use the more general terms (entity, relation, attribute).

---

Published as a conference paper at ICLR 2026
vector k encodes the entity e, and the value vector v encodes (r, a). The edit is then achieved by computing a new value vector v for the new pair (r, a²) and redirecting the mapping from k - v to k ++ v". Once the pair (k, v*) is specified, the corresponding weight update A to be added to W can be calculated in closed form such that (W + A )k = v", allowing the weights after knowledge editing to be written W W + A. Consequently, the result of the edit is fundamentally determined by how k and V are specified.
An ideal knowledge editing method should edit only the targeted knowledge while preserving unrelated knowledge and thus preventing unintended model perturbation. Accordingly, recent studies have sought to restrict the edited space; a notable example, AlphaEdit (Fang et al. 2025), constrains the edit by enforcing the row space of A to lie in the null space of the key vectors which are computed from existing knowledge that irrelevant to the specific edit. However, this leaves critical factor underconstrained: the computation of k and v² themselves. As noted above, the edit outcome is fundamentally determined by how these vectors are specified, and even with constraints on A, the high degrees of freedom in these vectors can still lead to unintended perturbation.
To address this, we constrain the computation of k and v" by leveraging the Linear Representation Hypothesis (Elhage al.) 2022 Mikolov et al. 2013 Park et al. 2024), which has received limited attention in the knowledge editing literature. This perspective suggests that k and v are linear combinations of both edit-relevant features (c.g.. features related to Chrome's developer attribute) and edit-irreeevant features (e.g., features related to Chrome's 's lexical form as proper noun or semantic category as a web browser), and that the computation of these vectors should be restricted to the former. Such a restriction enables us to derive k and v from narrower, more precise subspaces, which is crucial for preventing unintended model perturbation.
Against this backdrop, we propose Subspace Knowledge Edit (SUIT), which localizes target knowledge to specific subspaces and confines edits within them. Specifically, for the key vector k. we isolate subspaces specific to the entity being edited from those shared across many entities in the context of knowledge editing, such as the property of being a proper noun. For the new value vector v", we restrict the update to the subspace that primarily encodes which attribute should be retrieved as the target for the given entity and relation. Across LLaMA3, GPT-J, and Qwen2.5. SUIT substantially outperforms prior methods. In particular, compared with AlphaEdit (Fang et al. 2025), a strong baseline for minimizing knowledge disruption, SUIT achieves much higher specificity (i.e., better preservation of unedited knowledge) and preserves overall model capabilities far more robustly, without sacrificing editing performance. Additionally, our analyses ($ demonstrate how and why SUIT enables precise editing. Specifically, compared to baseline methods, SUIT substantially reduces perturbation at the entity's 's last token position, where attribute information is predominantly enriched. We further verify that the subspaces and directions we identified are more critical for editing than their complements, and that SUIT successfully isolates its edits to those subspaces.
2 RELATED WORK
Knowledge Editing Knowledge editing methods can be broadly categorized into three paradigms. Memory-based approaches preserve the original model by storing edited knowledge externally and retrieving it during inference (Mitchell et al., 2022b [Hartvigsen et al., 2023). This design avoids direct interference with the base model, but growing edits increase memory and retrieval overhead, limiting scalability. Meta-learning approaches train hyper-networks generate weight updates for efficient editing (Mitchell et al., [2022a) Cao et al. (22T)). However, they often generalize poorly to diverse edit distributions or large-scale sequential edits. Finally, locate-then-ee parameters to update and directly modify model weights, while restricting updates to parameters associated with factual associations in language models (Meng et al., 2023a b. Fang et al.. 2025). This makes them effective for sequential and large-scale editing while preserving overall model performance (Wang et al. 2024b), providing a practical foundation for continual knowledge editing. Accordingly, SUIT is designed as method in this paradigm.
approaches identify
Linear Representation Hypothesis The Linear Representation Hypothesis posits that the hidden states of language models are a linear combination of interpretable features, with each feature occupy- ing a distinct subspace (Elhage et al.] [2022] Mikolov et al. [0133] Park et al. 2024). Indeed, numerous studies have empirically demonstrated this hypothesis, showing that variety of features- as
2

---

Published as a conference paper at ICLR 2026
syntax, position, and factual knowledge, among others can be identified within specific, decom- posable subspaces or directions (Huben et al. 2024 Ji et al. 2023] Huang et al. (024). We apply this perspective to knowledge editing, decomposing key and value vectors into features relevant and irrelevant to the specific edit and confining edits within the identified subspaces.
3 PRELIMINARIES
3.1 LINEAR ASSOCIATIVE MEMORY
Knowledge editing in language models aims to edit a fact triplet from (e, r. a) to (e,r,a"). The locate-then-edii methods (Meng et al. 2023a b Fang et al. (2025) achieve this by viewing the MLP's down-projection layer as a linear associative memory that maps keys to values. From this perspective, the MLP's down-projection layer can be expressed as:
Wk =v
Here, the down-projection matrix W maps the key vector k, which is the MLP's up-projection activation, to the value vector v. Based on this, the core idea of the locate-then-edi methods that the key vector k encodes the entity e, while the value vector v encodes the relation r and attribute a. To edit the model's knowledge, these methods update the matrix W by adding an update matrix A. This change redirects the mapping of the key vector k from the value vector v, encoding (r, a). to new value vector v" that encodes (r, a"). The task is thus to find A such that:
(W + A vv..
Since Wk v, this simplifies to Ak R3 v v. We define r= v as the residual vector.
Building this principle, MEMIT Meng et al. 2023b) extends the approach to edit many facts at once. For a batch of facts (e, r, a²); where = I, I,.... n, it first computes the corresponding key vectors k, and residual vectors rN- These are then concatenated
form the key matrix K [k] k2 ... k,) and the residual matrix R |r r2 ... rn). MEMIT then derives a closed- form solution for the update matrix A using these matrices.
Subsequently, AlphaEdit (Fang et al. 2025), based on MEMIT, introduces new formula for A designed to better preserve the model's s existing knowledge. The proposed formula is:
A RKK P (K, KK PKK
KKT
Here, P is nullspace projection matrix introduced preserve the model's existing knowledge, and Kp aggregates key matrices from prior edits. With P and K, fixed, the computation of A is determined by the key vector k for the entity and the residual vector representing the change from to a", making accurate estimation of these vectors crucial. Appendix A provides the full derivation of the update formula and the computation details for P and Kp.
3.2 COMPUTING THE KEY VECTOR k AND THE RESIDUAL VECTOR
Key Vector, Suppose we want to modify the original fact (e, a) into (e, (g.raa) a²). To compute the key vector k, we extract the MLP up-projection activation from the last token of the entity (e.g., "rome" in "Chrome"), Following Meng et al. (2023a), to improve the generalization the key vector, we repeat this with various prefixes, and average the extracted MLP up-projection activations to obtain the final key vector k.
Residual Vector, While ROME (Meng et al. 2023a) targeted a single layer, most subsequent approaches perform edits across multiple layers. These multi-layer methods do not compute the residual vector separately for each layer. Instead, they first calculate an entire residual vector (henceforth, the residual vector) from the residual stream of the final layer being edited. This vector is then distributed proportionally to determine the specific r for each layer involved in the edit. To obtain 6, it is added to the residual stream h at the entity's last token position in the last modified layer and optimized via gradient descent to maximize the logit of the new attribute a ("Apple") ") given the input and ("Chrome was developed by"). To prevent overfitting the new fact, which can lead to
3

---

Published as a conference paper at ICLR 2026
the corruption of unrelated knowledge, regularization term R is added to the loss function (details in Appendix A.4). The optimization objective is formulated as:
8 = arg min {
logp a" h" + heh + +R..
(2)
4 SUIT: SUBSPACE KNOWLEDGE EDIT
4.1 KNOWLEDDGE EDITING UNDER LINEAR REPRESENTATION HYPOTHESIS
According to the Linear Representation Hypothesis, the key and value vectors within an MLP's down-projection layer can be viewed as composition of interpretable features. For knowledge editing, we hypothesize that these vectors consist of features that are either relevant or irrelevant to the specific edit. The key vector k, which encodes the entity, can be divided into entity-specific features and more general, entity-agnostic features that activate similarly across many entities. Similarly, the value vector v. encoding the relation and attribute, contains features that primarily define the attribute (a or a²) alongside other less relevant features.
To ensure that knowledge edits are precise- modifying only the target knowledge while preserving other knowledge- we propose introducing explicit constraints. When computing the key vector k, we aim to consider only the subspace occupied by entity-speciff features. Likewise, when computing the residual vector 5, we aim to consider only the feature directions that significantly influence the attribute's logit. Accordingly, the following sections we obtain the subspace-aware key vector k' ($ 4.2) and the subspace-awe aware residual vector 8' ($ 4.3 using these, we compute the update matrix via Eq. T
4.2 SUBSPACE-AWAR COMPUTATION FOR OUR KEY VECTOR k'
When computing our key vector k' that encodes the entity e, we consider two complementary subspaces of the key vector space: the entity-specific subspace K,,, in which key vector components exhibit high variance across different entities, and the entity-agnostic subspace KC, in which key vector components exhibit low variance. Since components in K2+- activate similarly across many entities, retaining them in k can cause the edit to affect unrelated entities. Accordingly, our objective to isolate the entity-specins component of k that lies within K.,.
We begin with the key vector k, as defined in $ 3.2 We then decompose this vector into its entity- specific component, k, E K.,, and its antity-agnosie component, k.s KJJ. Our key vector k' is obtained by removing this entity-agnostic component:
k' k k.s kg.
The core task is to identify K-_ and compute these vector components. We accomplish this through the following procedure.
We sample N = 10,000 entities from PARAREL Elazar et al. 2021 ), a dataset of (e, r, a) triplets derived from Wikidata. For each entity, we compute its key vector k to form the matrix Kentiyy [k] k2 ... k1000o]-
Applying singular value decomposition (SVD) this matrix yields:
Konitry USV
where diag(o .2.....,o σr) with o 2 02 2 2 or denotes the singular values, and U
in U2
up contains the corresponding left singular vectors.
To determine how many critical components to remove, we introduce hyperparameter Tenergy E [0, 1) that represents the proportion of total variance (energy) isolate. Let ExStal Σi=1 of be the total energy of Keaut- We find the smallest integer m such that the cumulative energy of the first m components reaches or exceeds this threshold: Σm of Z Temergy Facua- Let U.,, = [u |... um] denote the matrix of the first y left singular vectors. We then define the entity-agnostic subspace K. their span:
1 span(Ums))
4

---

Published as a conference paper at ICLR 2026
Finally, we the matrix U~g U.e to project the key vector k onto the entity-agnostic subspace K. By subtracting this projection kwe, we remove the entity-agnoste agnostic component, leaving only the -ntity-specific component k, This procedure yields the constrained key vector k', which now contains only the features relevant to the entity being edited:
k'
KNS1
k U..
4.3 SUBSPACE- AWARE COMPUTATION FOR OUR RESIDUAL VECTOR 8'
The residual vector å is computed to modify the residual stream h so that it encodes (r, a*) (see $ (.2). Rather than altering the full-dimensional residual stream h, our approach is to target a low-dimensional subspace that governs the model's prediction for the given (e,r) pair.
We hypothesize that this targeted modification can be achieved within a wwo-dimensional subspace spanned by two critical unit feature directions, W1 and W2, associated with and a². Specifically, increasing the magnitude of h along W1 (i.e., h W1) raises the logit of the new attribute a", while decreasing its magnitude along W2 (i.e., h w2) suppresses the logit of the old attribute a. Our goal to identify w and W2 and swap these directional magnitudes of h, thereby increasing the logit of a" toward the original level of a and decreasing the logit of toward the original level of a*. For simplicity, we ignore interactions between the two directions and implement the edit as a simple additive update (details in Appendix (B). The updated residual stream h" is:
h h h 8', 8' (h W2 h W1)w1 + (h W1 h W2
The process for finding the optimal basis vectors, /w1, W2], follows a similar structure to the optimization for the residual vector 5 shown in Eq. 2 The primary objective is to maximize the logit of the new attribute a". While Eq. ( included general regularization term R. it is unnecessary in our approach as we constrain the update to a two-dimensional subspace. To encourage the basis vectors wi and w2 to represent distinet directions, we introduce a directional penalty term, formulated as [w] w2) The complete optimization objective reflecting this is therefore formulated as:
[W1. w21 = arg min { log (a" h" t h 8.) + > W2 21
W1.W2
where the hyperparameter À is the penalty weight that controls the strength of the directional penalty.
5
EXPERIMENTS
This section describes the experimental setup ($ 5.1) presents the main results in Tables and and reports additional experiments 5.3
5.1 EXPERIMENTAL SETUP
Models, Baselines, and Datasets. We conduct our experiments LLaMA3-Instruc (8B) GGrttaffori et al. 2024), GPT-J (6B) (Wang & Komatsuzaki 2021). and Qwen2.55IIntrre
(7B) (Yang et al.) 2024). We compare SUIT against several representative model editing baselines, including Fine- Tuning (FT-L) (Zhu et al. 2020). MEND (Mitchell et al. 2022a), PMET (Li et al. 2024), MEMIT (Meng al. 2023b), and AlphaEdit (Fang al. (025). 3. Due space constraints, we defer results for additional baselines (FT-W (Zhu et al. 2020), ROME (Meng et al. (023a)). RECT (Gu al., 2024) PRUNE (Ma al. 2024p, NSE (Jang al. 2024)) to Appendix E. We also provide the hy hyperparameters used for all methods, including SUIT. in Appendix C.2 and describe
in 6.3 how the SUIT-specific hyperparameters Tenergy and À are selected. We use two widely used
knowledge editing benchmarks: COUNTERFACT (Meng et al. 2023a and zsRE (Levy et al. 2017).
Evaluation Metrics and Setup. We report three standard knowledge editing metrics: Efficacy, Generalization, and Specificity. Efficacy (Eff.) assesses whether the model generates the new attribute a" for a given rewrite prompt (e, r), while Generalization (Gen.) measures the same for paraphrase prompts. Specificity (Spe.) checks whether edit causes unintended changes to other knowledge, using neighborhood prompts. For COUNTERFACT, we also report the generation-quae quality metrics Consistency (Con.) and Fluency (Flu.). Further details provided in Appendix D
5

---

Published as a conference paper at ICLR 2026
Table 1: COUNTERFACT results for sequentially editing 1,000 facts with different batch sizes (1. 10. and 100); GC. Flu., and Con. are reported for batch size 100. Best numbers are bold; second-best best are underlined. Abbreviations: Eff. = Efficacy, Gen. = Generalization, Spe. = Specificity, GC = General
Capability, Flu. = Fluency, Con. = Consistency.
batch size =1
batch size
10
batch size
100
Method
St
f
Gen.
Spe. f
5'
Gen.
Spe. f
S
Edf
Gen.
GC
Fiu
Con.
Pre-edit
0.0
0.0
0.0
100.0
0.0
0.0
0.0
100.0
0.0
0.0
0.0
100.0
63.4
634,9
20.9
3.9
2.1
0.2
0.0
0.0
1.9
9.9
6.2
438.5
19.2
MEND
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
1.3
0.0
519.5
0.6
MEMIT
0.0
0.0
0.0
0.0
48.3
76.2
60.8
628.9
36.7
PMET
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
37.6
56.6
56.0
22.6
50.1
608.8
33.8
AlphaEdit
90.2
16.5
37.4
17.0
55.8
97.3
88.7
62.2
SUIT
85.4
99.6
89,5
71.9
85.6
99.9
89.8
71.9
86.8
99.7
90.3
74.2
63.0
631.2
38.2
Pre-edit
0.0
0.0
100.0
0.0
0.0
100.0
0.0
0.0
100.0
24.3
621.1
23.9
FT-L
12.5
31.9
23.4
6.0
9.3
47.8
32.8
3.7
13.3
64.9
46.7
5.3
24.2
334.2
12.2
GPH
MEND
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
515.0
2.7
MEMIT
0.0
0.0
0.0
0.0
46.1
75.
74.1
26.1
60.2
92.0
90.4
35.8
20,1
617.4
48.5
PMIT
0.0
0.0
0.0
0.0
57.4
84.6
19.1
618.5
44.8
Adph SUIT
72.8
98.9
94.8
48.7
76,3
99.2
96.0
53.1
73.0
98.3
95.0
49.0
19.5
621.8
49.9
98.4
92.7
64.5
84.3
99.1
94.3
84.7
99.2
20.4
619.4
49.4
Pro-odit
0,0
0.0
0.0
100.0
0.0
0.0
0.0
100.0
0.0
0.0
0.0
100.0
28.9
625,5
21.9
Qwen2.5
IT-L
11.6
21.6
19.5
6.2
12.0
28.1
22.5
5.9
10.3
47.9
31.7
4.2
0.0
476.7
4.5
MEND
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
466.3
0.1
MEMIT
0.9
0.0
0.0
0.0
0.0
0.1
0.0
0.0
22.6
83.0
84.0
9.2
34,8
622.1
37.0
PMET
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
32.6
67.5
65.9
16.1
14.8
545.2
27.5
AlphaEdit
65.0
96.2
92.4
40.1
66.6
96.0
93.2
41,9
67.8
97.1
91.6
43.4
28.1
626.2
41.0
SUIT
88.6
99.3
90.3
72.0
85.7
99.3
90.8
71.9
86.1
99.2
91.6
72.3
30.8
626.2
37.4
To faithfully measure the effects of knowledge editing, especially the preservation of existing knowledge, we focus on instances where the model already produced the target answer before editing. For COUNTERFACT, we keep only facts for which the pre-edit model generates the ground-truth answer. This ensures that Eff. and Gen. measure edits to known facts rather than insertion of facts the model did not know before. Under the same selection, Spe. measures whether knowledge previously exhibited by the model remains preserved after editing. Following prior work (Meng et al. 2023a b Fang et al., 2025), ) we also report S, the harmonic mean of Eff". Gen., and Spe., which summarizes how well method balances these metrics under their trade-offs. For ZsRE, however, restricting evaluation to pre-edit- known instances is not feasible, because the pre-edit model rarely generates the ground-truth answer as annotated in the dataset (see Appendix [D33 for quantitative details and rationale). For Spe., we therefore use the model's 's own pre-edit generated response as the reference so that the metric reflects preservation of previously cxhibited knowledge. We then evaluate whether the first 1 1-4 tokens of that response are exactly preserved after editing.
General Capability. We evaluate the model's General Capability (GC) to assess whether know knowledge editing degrades performance on general tasks. The GC score measured after sequentially editing 1,000 facts with batch size 100. It is computed as the average Fl score across six benchmarks: MMLU (Hendrycks et al. 202T) and five GLUE tasks (Wang al, 2019) (NLI, MRPC, SST, RTE, and CoLA). Detailed per- ber-benchmark results are provided in Appendix
5.2 EXPERIMENTAL RESULTS
Table 1 shows that COUNTERFACT, SUIT most consistently achieves the best overall balance between editing success (Eff. and Gen.) and preservation (Spe.) across LLaMA3, GPT-J, and Qwen2.5. This also reflected in the harmonic mean score S. where SUIT is highest in all settings. Importantly, SUIT does not obtain this advantage by sacrificing editing performance; rather, it maintains strong Eff. and Gen. while also achieving substantially higher Spe.. The batch size settings denote how many facts are edited in each update step, while keeping the total number of edited facts fixed at 1,000. Therefore, smaller batch sizes require more update steps and thus more weight updates to complete the same number of edits. Under smaller batch sizes, several baselines collapse to near- zero performance, indicating model collapse under frequent cumulative weight updates. In contrast, SUIT maintains strong performance even in these harder settings and, compared with AlphaEdit, shows much less degradation in Spe. as batch size decreases. SUIT also preserves Flu., Con., and GC, showing that it can edit knowledge effectively without degrading overall model performance. Table

---

Published as a conference paper at ICLR 2026
Table ZsRE results for sequential editing of 1000 facts with batch size of 100. Specificity measures the exact preservation first tokens the pre-edit response.
LLaMA3
GPTJ
Qwen2.55
Method
Spe.
Egr.
Gev,
f=
2
3
2
E E
Pre-culit
35,9
34.8
100.0
100.0
100.0
100.0
27.2
26.3
100.0
100.0
100.0
100.0
34,4
33.8
100.0
100,0
100.0
100.0
MEND
0.0
0.0
0.0
0.0
0.4
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
MEMIT
89.3
65.7
30.1
22.9
21.1
19.2
96.3
90.8
77.9
77.7
50.8
23.2
96.2
91.4
15.7
11.8
10.5
9.3
PMET
89.2
83.9
26.7
21.3
18.9
16.7
95.1
89.2
67.4
67.0
23.3
17.1
82.1
76.3
2.1
1.1
AlphaFddi
88.7
31.2
19.8
99.7
94.1
50.5
41.6
94.5
25.0
19.5
SUIT
95.2
85.2
$1.9
43.2
41.2
39.8
99.7
94.3
99.3
99.1
70.4
61.6
99.5
92.2
64.4
57.3
54.5
51.7
shows that the same pattern observed Table T also holds on ZsRE, Across LLaMA3, GPT-J, and Qwen2.5, and across token-length settings e, SUIT consistently attains substantially higher Spe. than prior methods, while retaining strong editing success (Ejf. and Gen.).
5.3 ADDITIONAL EXPERIMENTS
We present additional experiments to further evaluate SUIT along several dimensions.
Stability under extended sequential editing. We extend the evaluation to 5,000 edits and track GC every 100 edits (Fig. [ As edits accumulate, SUIT remains substantially more robust than the strong baseline AlphaEdit, while Appendix E.3 confirms that SUIT also maintains high editing performance after 5,000 edits.
0.62
0.60
Scaling to larger model. To test whether SUIT's advantages persist larger scale, we evaluate ILaMA2- 13B (Touvron al.) 2023) on COUNTERFACT. SUIT preserves strong editing- preservation balance and achieves the highest among the base- lines; see Appendix E.4 for details.
1000
3000
4000
Figure 1: GC over 5,000 se- quential edits.
Context robustness of edited facts. While SUIT has been shown preserve existing knowledge and maintain strong editing performance when prompted with entity-relation pairs, precise edits should also be robust even when these pairs appear after various contexts in the prompt. We evaluate this with CHED (Park et al. (2025), which measures the correctness of edited facts under diverse prefix contexts.
shown in Appendix E.S SUIT consistently outperforms baselines, demonstrating robustness to context variation.
Ripple effeets of edited facts. Effective model editing should consistently reflect edit's implications on logically related knowledge. To evaluate this, we employ RIPPLEEDITS (Cohen et al., 2024), which measures whether edit correctly propagates to associated facts across multiple logical relations. As shown in Appendix E.6, SUIT is competitive with or superior to existing baselines, demonstrating its ability to maintain logical consistency.
ANALYSIS
To further investigate SUIT, we conducted a series of analyses. We demonstrate its effectiveness in reducing perturbation at the entity's last token. We then empirically validate our hypotheses regarding the subspaces identified for computing the key and residual vectors. Finally, we present hyperparameter analysis and an ablation study. All analyses were performed using LLaMA3. For experiments requiring edited models, we used models edited with 10 batches of 100 edits each from COUNTERFACT
6.1 REDUCING ENTITY'S 'S LAST TOKEN PERTURBATION
Ideally, an edited model is expected to behave identically to the original model on unedited knowledge, without introducing any perturbation. Prior research (Meng al. 2023a Geva et al. 2023 Chughtai eet al. 2024) has demonstrated that an entity's 's attributes are predominantly enriched at the last token position of the entity. Accordingly, locate-then-edi methods perform edits by deriving key and value
7

---

Published as a conference paper at ICLR 2026
Camperaon Methooologges by Layer
MEMIT
MEMIT Aphatsi
N resorted that Don DX ' own party, the Christian Democratic Kopea
SUIT
supported proposal. However,
AlphaEdit
MBS recorted that Dor Donheet's 5 own party, the Christin Damocratic Knpeal supported his proposal. Hownver,
3.77
SUIT
5:50.00 427
NOS reported that Donner's is own party, Christtan Damocratic
Appeal (&), supported his proposal. However,
Layer Number
Comparison of token-level perturba- tions in residual streams.
Figure 2:
Figure 3: Norm differences of MLP outputs at the entity's last token position across methods.
vectors directly from this position. This approach, however, induces significant perturbation precisely at this critical location.
Fig. 2 visualizes how much the residual stream changes in the final edited layer by comparing the pre-edit model with models edited using MEMIT, AlphaEdit, and SUIT. Specifically, we compute the T,2 norm of the difference between the residual streams of the original and each edited model, which measures the magnitude of the perturbation introduced by editing each token. These values shown as color intensity over a sample paragraph from the Wikinews Article Dataset, where darker colors indicate larger changes (i.e., greater perturbation magnitudes) relative to the pre-edit model. The figure demonstrates that, compared to other tokens, all methods generally exhibit a more notable perturbation at the last token of the entity (e.g.. "OS" in "NOS". "ner" in "Donner"). Notably, among these methods, the perturbation from SUIT is visibly less pronounced than that of MEMIT and AlphaEEdit. Further examples confirming this pattern are provided in Appendix[]
To quantify the perturbation from editing, we analyze the output of the MLP layers, which are the direct targets of the edits. Using unedited knowledge from COUNTERFACT dataset, we measure the Le distance between the outputs of the original and edited models at the entity's 's last token position. As shown in Fig. 3 while all three methods induce a non-zero perturbation, SUIT consistently exhibits the smallest change. This indicates that SUIT performs more localized and precise edits, affecting unrelated knowledge the least.
6.2 ANALYSIS FOR SUBSPACES
In this section, we investigate whether the subspaces K,,, K.! and the directions WT, W2, which we identified to find the key vector k and the residual vector 6, indeed correspond to the feature subspaces we hypothesized. We hypothesized that K., is entity-specific feature subspace that activates differently for each entity. Conversely, its orthogonal subspace K!! is assumed to be an antity-agnostic feature space, activating similarly across different entities. Furthermore, we posited that W1 and W2 are crucial directions for the update. Specifically, when their scaled versions are added to the residual stream h, we believe W is the principal direction for increasing the logit of the new attribute a", while W2 is the principal direction for decreasing the logit of the old attribute a.
6.2.1 ANALYSIS FOR IC, AND KC!
To investigate whether the subspaces identified via PARAREL generalize to other datasets, we designed an experiment to measure component variance across different datasets. We extracted key vectors k for 5,000 randomly selected entities from COUNTERFACT and ZSRE datasets. Each key vector k was then decomposed into its component k, in KC, and component k., in KC- using the subspace derived from PARAREL. Finally, computed the variance for each set of components across all 5,000 entities to verify whether the components in KC, vary across entities, while the components in K. remain relatively similar across different entities, even other datasets.
The results of our variance analysis, presented in Table 3 align with our initial expectations. Across both datasets, the variance of the specifc components k, significantly higher than that of the entity- agnostic ones k.., being approximately 2.6 times higher for COUNTERFACT and 4.5 times higher for 7.RE. This finding is consistent with our hypothesis, suggesting that K,, may
8

---

Published as a conference paper at ICLR 2026
Table 3: Variance of decomposed key vector components across 5,000 entities.
Table 4: Proportion of the modification affect- ing entity-agnostic components.
Prompt type
MEMIT
AlphaFdit
SUIT
COUNTERFACT
ZSRE
rewrite
0.2814
0,4623
0.0035
((kwa) V(kg)
2.041
1.333
paraphrase
0.2929
0.4717
0,0039
5.269
5.938
neighborhood
0.6805
0.8114
0.0201
capture entity-speciff features that vary across individuals, while K., appears to contain more stable, entity-agnostic information.
Furthermore, we conducted an experiment to verify that our update matrix, A, interacts less with the subspace K.-. To test this, we compared our method against MEMIT and AlphaFdii on the rewrite, paraphrase, and neighborhood prompts (defined in $ 5.. in COUNTERFACT. Specifically, measured the proportion of the update that affects the antity-agmostic components, calculated as | 1I2 /|| The results computed from the update matrix A at the first edited layer are presented in Table The results clearly demonstrate that for all prompt types, the proportion of the update affecting the entity-
space || A 1l ||///|////// is negligible for our method, SUIT. This in stark contrast to MEMIT and AlphaEdit, which show significantly higher proportion of their modifications impacting these common components. This finding suggests that SUIT successfully isolates its edits the entity-specific space K.,. leaving the shared,
knowledge largely untouched. We can, therefore, infer that this precise targeting is a key reason for SUIT's enhanced specificity for neighborhood prompts.
6.2.2 ANALYSIS FOR W1 AND W2
Next, we verified our hypothesis that the subspace spanned by W1 and w2 represents the critical directions for increasing the logit of the new attribute a". To do this, we first computed the residual vector 5 using the original method ($ 3.2h. We then decomposed this vector into two distinet components: the component lying in the subspace span (W1, W2). 8 ojw, and the remaining orthogonal component, 5 w. The projection is calculated as:
6ll = 6, where W = [W1, w2], Pw = WWWTW) 'WT.
To compare the respective effects of these two components on increasing the logit of a", we added each of d w and 8 to
Table 5: Results of steering with decomposed components of 8.
the residual stream h and measured the logit and probability of
a", given the entity and relation averaged over the 1,000 edits in COUNTERFACT. The results Table 5 provide direct evidence for our design choice: constructing updates in the critical subspace span W1, W2) yields a more effective steering signal than using the remaining directions. Although Mlll accounts for only about 24% of the total squared norm of 6, it is more effective at increasing the logit and the probability of a" than the remaining 76% in 8 w. This indicates that 5 contains both directions that are critical for increasing the logit of a" and other less-essentia components. These findings support our approach of constructing the residual update from the critical subspace, enabling a more focused and potent update within a much narrower directional space.
Space
6|l
| ||||||
24.17
75.83
pla" logit(a")
0.67
0.59
-1.44
-1.72
We further analyzed the individual roles of W, and W2 to test whether W1 primarily increases the logit of the new attribute a* while Wa primarily decreases the logit of the old attribute To do this, we decomposed our update vec- tor 8' into its components along these directions: Aw1 (h" w2 h" wi j and Aw2 (h w1 h' w2) We then incrementally added each component to the resid- ual stream h by scaling it with interpolation factor k E [0, 1], and observed the logits for both attributes.
Aw1 vs Logiti' Apple")
Aw2 vs Logit! Google")
Loglt
Interpolation factor (k)
Interpolation factor (k)
Figure 4: Effects of Aw and Aw, on the logits of "Apple" and "Google"
Fig. shows the results for the edit ("Chrome". "was developed by", "Apple"). As expected, the Aw component effectively increases the target a" ("Apple") logit, while the Aw, component effectively decreases the original ("Google") ") logit.
9

---

Published as a conference paper at ICLR 2026
Although W1 and W2 drive logits in the expected directions and were trained to play different roles, we find that W1 also suppresses the old attribute a, and Wa also promotes the new attribute a", rather than each playing only single role. The effectiveness of fully disentangling these roles would be worth exploring in future work. For a detailed visualization and a full breakdown of these component effects, please Appendix
6.3 HYPERPARAMETER ANALYSIS AND ABLATION STUDY
We analyze the effects of the two hyperparameters in our method: the energy threshold Tenergy which controls removal of the shared component from k to obtain k' (S 4.2. and the penalty weight which controls how strongly W1 and W2 are encouraged to be distinct when computing 8' ($ 4.3). We vary one while fixing the other and measure changes in editing metrics. Additionally, we perform separate ablations on k' and 8' to isolate their contributions.
Tenergy
A
R13
608
Efficacy
Generalizatio
Specificity
Efficacy
Generalization
Specficity
Gwng
Figure 5: Analysis for hyperparameters Tenergy and A.
Hyperparameter Analysis. Fig. 5 shows how Tenergy and affect editing metrics on COUNTER- FACT (LLaMA3, 1,000 edits, batch size 100). We vary one hyperparameter while fixing the other (A= 0.3 or Tenergy = 0.4). As Tenergy increases, Spe. improves steadily, while Eff. and Gen. decline beyond threshold. A larger Tenergy removes more the shared component, suppressing unintended changes. If too large, however, it can also remove directions needed for the target edit, weakening the edit itself. Similarly, shows a comparable trend, but with smaller variation. As À increases, Eff. remains stable, while Spe. improves modestly at the cost of slight drop in Gen.. This is because a larger À strengthens the penalty I[ W2)², encouraging w1 and W2 to be more distinct, which yields more specific update. We extend this analysis to GPT-J and Qwen2.5, focusing (Appendix C.3). Both show trends similar to LLaMA33, but with lower sensitivity. Accordingly, we select hyperparameters by maximizing the harmonic mean score S over Eff., Gen., and Spe.: À 0.3 for all models, and Tenergy = 0.4, 0.6, and 0.6 for LLaMA3, GPT-J, and Qwen2.5, respectively.
Tenergy
Ablation Study. We ablate the k' computation ($ 4.2 and the 8' computation ($ 4.3) to assess their individual impacts (Table 6). While each component alone sur- passes the strong baseline AlphaEdit in the harmonic mean score S, combining them in SUIT yields the strongest overall improvement, indicating that both com- ponents contribute meaningfully to the final performance and complement each other.
Table 6: k' and 8' ablation results.
Method
S
Eff:
Gen.
Spe.
AlphaEdit
55.8
97.3
88.7
31.0
k' Only 8' Only SUIT
82.2
96.4
77.9
74.7
68.3
99.7
83.8
44.6
86.8
99.7
90.3
74.2
CONCLUSION
In this work, we introduced Subspace Knowledge Edit (SUIT), knowledge editing method that localizes target knowledge to edit-relevant subspaces and confines edits within them, improving speci- ficity without sacrificing editing performance. Motivated by the Linear Representation Hypothesis, SUIT decomposes the key vector into entity-specific features and restricts the residual update vector to features aligned with the target attribute. Across multiple models and evaluation settings, SUIT consistently improves Specificitt and preserves overall model capabilities, while maintaining strong editing performance. These results suggest that explicitly modeling subspace structure a promising direction for more precise and robust model editing.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTI
This work was supported by the National Research Foundation of Korea (NRF) under the grant RS-2024-0033344 and by the Institute of Information & Communications Technology Planning Evaluation (ITTP) under the grant RS-2024-00338/ Technology to Reflect Sustainability of Generative Language Models and Up-to-dateness over Time), all funded by the Korean government (MSIT). This work was also supported by the National Supercomputing Center with supercomputing resources including technical support (KSC-2025-CRE- 0332, KSC-2225-CRE-05144
(Development Learning and Utilization
REFERENCES
James A. Anderson. A simple neural network generating an interactive memory. Math- ematical Biosciences, 14(3):197-222) 1972. ISSN 0025-5564. doi: https://oi..or//10
101600033-55644229005522 URL
atticle/p11/022551230000
Luisa Bentivogli, Ido Dagan, Myroslava Dzikovska, Danilo Giampiccolo, and Bernardo Magnini. The fifth pascal recognizing textual entailment challenge. In Proceedings of the Text Analysis Conference (TAC), volume 9, 2009.
Louis Bethune, David Grangier, Dan Busbridge, Eleonora Gualdoni, Marco Cuturi, and Pierre Ablin. Scaling laws for forgetting during finetuning with pretraining data injection, 2025. URL
https://araa www.xiivorgg/aaooo 2502 06042
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models, 2021. URL
08164
Bilal Chughtai, Alan Cooney, and Neel Nanda. Summing up the facts: Additive mechanisms behind factual recall LLMs, 2024. URL https://openreviee.. wttp://wwwwwtete.uwuteemm
P2gnDEHGu3
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics, 12:283-2222 29228, 2024. doi: 10.1162/tacl a 00644. URL httpt://///aa antho ogy
hTg/2024..0 tac 167
Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (/WP2005), 2005.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguisties, 9:1012-1031 2021. doi: 10.1162/tac.a.0044100
tacl- 60/
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, 7ac Hatfeld-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCan- dlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, 2022.
org/ abs / 2209 10652
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained model editing for language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https
.nt wtw.://orormmioo//Stst..
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Singapore, December 2023. Association for Computational Linguis- hitps: http:/s://aalaatttooo...oooo
Processing. pp.
tics. doi: 10. 10.18653/1 1ttp:///////.eeeoo wtt.002...mmlmmmaii. 15.n 751. URL
eemn 11lpppmaa- ma 7517
11

---

Published as a conference paper at ICLR 2026
Alessandro Grattafiori et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407. 2024. URL
http//as//////// 217883
Jia-Chen Gu et al. Model editing harms general abilities of large language models: Regularization to the rescue, arXiv preprint arXiv:2401.047000 2024. URL /tthht/pss//aaavvooo//////////
04700
Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors, 2023. URL https
//arxiv.org/abs/22222 11031
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. RAVEL: Evaluating interpretability methods on disentangling language model representations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8669-8687, Bangkok, Thailand, August 2024. Association for Computational Linguistiss. doi: 10. 12..8665//1/2222aclllooooo URL
wtg. acl
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. htth:tts:: ptttt::://ppezeveee
net forum?
1d Ld=F76bwRSLeK
Ziwei Ji, Naycon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination natural language generation. ACM Comput. Surv. 55(12), March 2023. ISSN 0360-0300. doi: 10.1145/3571730. URL
Houcheng Jiang, Junfeng Fang. Tianyu Zhang, An Zhang, Ruipeng Wang, Tao Liang, and Xiang Wang. Neuron-level sequential editing for large language models. arXiv preprint 05Xiv22410.04045, 2024. URL/ttts:://ark/ org abs 22200.411..04
Teuvo Kohonen. Correlation matrix memories. IEEE Transactions on Computers, C-21(4):353-559, 1972. doi: 10. N0.11099TC.199225000999
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Roger Levy and Lucia Specia (eds.), Proceedings of the 2 Conference on Computational Natural Language Learning (CoNLL 2017), pp. 333-342, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. 1/11865//1/K11-10344 URL
Xiaopeng Li, Shasha Li. Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. Pmet: Precise model editing in transformer, 2024. URLbttps: arxiv. org/ abt////ss/22/// 2374.087422220
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 100118693/112022.aclloooo 229/
URL Dtt.atpae///ellattoooo....//2222 www.poy....r//2222..cl.oooo...
Yun Luo, Zhen Yang. Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2025. URL
https://arxizs org/ abs 2308 08747
Jun Yu Ma, Hong Wang. Hao-Xiang Xu, Zhen-Hua Ling, and Jia-Chen Gu. Perturbation-restrainde sequential model editing. arXiv preprint I181v:2405.16821, 2024. URL https: :tapp:://aaa //arXiv
hrg/abb/24055. 16821
12

---

Published as a conference paper at ICLR 2026
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1906-1919, Online, July 2020. Association for Computational Linguistics. doi: 10.1 10 maii..cl-main.I. 173. httts::::////ccccc
173/
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023a.
05262
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass editing memory in transformer. The Eleventh International Conference on Learning Representations (ICLR), 2023b.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space, 2013. URL httpttts/txxxxi...tt.a wttp://w....aab///////a// 3781
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing at scale, 2022a.
11309
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Memory- based model editing scale. In International Conference on Machine Learning, 2022b. URL
https: //ap:ia.aiivv.v/aaa o htt///df///// 2206. 06520. pdf
Haewon Park, Gyubin Choi, Minjun Kim, and Yohan Jo. Context-robust knowledge editing for language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher
Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 10360- 10385, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176- http://2.......nsss.... f Findings-ase 5407
256-5. doi:
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothess and the geometry of large language models, 2024.
2231.03668
Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan. Finding a" kneedle" in haystack: Detecting knee points in system behavior. In 2011 31st international conference on distributed computing systems workshops, pp. 166-171. IEEE, 2011.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremm Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09220 2023. htRpsttt:: /ttpes:///aiiiii wwt.arivvv... org/ abs hwtpb/////3...2253
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the International Conference on Learning Representations (ICLR), 2019. URL https: /tttps://arxivt wwxiv.
Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A billion parameter autoregressive language model, 2021.
13

---

Published as a conference paper at ICLR 2026
Peng Wang, Ningyu Zhang, Bozhong Tian, Zekun Xi, Yunzhi Yao, Ziwen Xu, Mengru Wang, Shengyu Mao, Xiaohan Wang, Siyuan Cheng. Kangwei Liu, Yuansheng Ni, Guozhou Zheng, and Huajun Chen. EasyEdit: An easy-to-use knowledge editing framework for large language models. In Yixin Cao. Yang Feng. and Deyi Xiong (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 82-93, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10. 1222444/ URL httptsss/accaaanoooooooooooooo
2024 -ataacllleess.....
demos. 9/
Song Wang, Yaochen Zhu, Haochen Liu. Zaiyi Zheng. Chen Chen, and Jundong Li. Knowledge editing for large language models: A survey, 2024b. URL hRtpp::////ax http:////xxwiv.../a/aeee a/gg/bb//// 2rg30 2310 6218
Alex Warstadt, Amanpreet Singh, and Samuel Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume I (Long Papers), pp. 1112-1122, 2018
Yang et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 2024. URL https //arXiv.org/abs/221/ 15115
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 10222-10240, Singapore, December 2023. Association for Computational Linguisties. doi: 18.18 http:///2...mmmmmammma... URL emnlp-main, 632/
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. A comprehensive study of knowledge editing for large language models, 2024. URL
btttps://arxin
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models, 2020. URL htt.hhttp://eet org/ abss/
2012.00363
A DETAILS OF THE ALPHAEDIT UPDATE FORMULA
A.1 CLOSED-FORM SOLUTION
The primary objective is to find A that incorporates new knowledge while preserving both the original model's knowledge and knowledge from previous edits. The optimization problem is formulated as follows:
A arg min (|PP RI + | APK,
A
where the three terms correspond to the insertion of new information, a regularization term for stable convergence, and the preservation of prior edits, respectively.
This objective has a closed-form solution.
The final update matrix A' AP is given by:
A' = RRK -KK
14

---

Published as a conference paper at ICLR 2026
A.2 COMPUTATION OF THE PROJECTION MATRIX P
The matrix P is projection matrix designed to project the update A into the null space of large key matrix Kg. which represents that the update does not interfere with this preserved knowledge, satisfying APKo = 0.
collection of the model's existing knowledge. This ensures
Due the high dimensionality of Ko, the projection is computed using the much smaller covariance matrix K The procedure is as follows. First, Singular Value Decomposition (SVD) is performed on the covariance matrix:
A kV
Next, the eigenveetors in U (which are its columns) corresponding near-zero eigenvalues identified. A submatrix U is then constructed using only these selected cigenvectors. Finally, the projection matrix P is defined as:
P= ÚÚ
A.3 COMPUTATION OF THE PRIOR KEYS MATRIX K,
The matrix K, is used in sequential editing tasks to protect the knowledge updated in previous steps from being disrupted by the current edit. It is constructed by aggregating the key matrices from all prior edits.
Specifically, if there have been K, K2, K then Kp is the horizontal concatenation of these matrices:
previous edits, with corresponding key matrices
K, = KI, K,-1]
For the very first edit, K, is an empty matrix.
A.4
THE REGULARIZATION TERM R
The optimization objective to find the residual vector is given by:
8 = arg min { ( log h" + h+ + R].
The regularization term is introduced to prevent the model from overfitting to the new fact, which could corrupt existing knowledge. It consists of two components: KL divergence term and weight decay term. The full regularization term is formulated as:
R = AxL Dra. (p(h) || p(h + 5)) + AwD AwD||d1
KL Divergence. The first term uses the Kullback-Leible (KL) divergence to preserve knowledge related the entity of the edit. This is achieved by computing the divergence on prompt, such as "(entity) is a" (e.g., "Chrome is a a"). Specifically, it measures the divergence between the output probability distribution from the original hidden state and the distribution from the modified hidden state + 8. The hyperparameter Ågr. controls the strength of this penalty.
Weight Decay. The second term is a weight decay penalty on the L2 norm of the residual vector This term encourages the model to find a smaller solution for ò. The hyperparameter Awp controls the strength of this penalty.
B THE ADDITIVE UPDATE
The updated residual stream h" is computed via simple additive update:
h h 8',
where our residual vector 8' defined as:
8' (h W2 h WTTw) (h" w, h w2)
15

---

Published as a conference paper at ICLR 2026
This formulation is simplified by ignoring interactions between the feature directions W1 and W2. By assuming interactions between them to zero (i.e., wi W2 = 0), we can achieve the intended swap of magnitudes with straightforward additive operation.
To verify that this update swaps the magnitudes of h along W1 and W2. we can compute the new projections (h") and (h") W2. Since W1 and W2 are unit vectors, w W1 and w W2 1.
First, let's compute the projection of h" onto W1:
(h*) w1 = (h +h+ (h W2 - h W1) + (h T WI h (2)22) W1
= hh W1 + (h W2 - h [ [)]]]]]]+=-- wi) + (h "ulw h ( wi) h W1 (h W2 h -hww))(1) (h w1 - -h h2w2((0)
h W1 h W2 h-w:
h W2
As shown, the new magnitude of the hidden state along w is equal to its original magnitude along W2.
Next, we compute the projection of h" onto w2:
(h*) W2 = (h (h W2 - h w wI)wi + (h T WT -hT -hTwa w2) W2
= h T W2 + (h W2 - h (w]ww) w2) + (h (hTw1 W1 h (w2) w2)
h W2 (h W2 h (0) (h W1 h (1)
h W2 h W1 h'w2
h W1
Similarly, the new magnitude along W2 becomes the original magnitude along w Thus, this simple additive update, which ignores interactions between the two directions, effectively swaps the magnitudes as intended.
C EXPERIMENTAL DETAILS
C.I BASELINE METHODS
Below we provide brief descriptions of the baseline methods used for comparison. Our main set comprises Fine- ToneTTunig (FT; FT-L/FT-W) (Zhu et al. (2020). MEND (Mitchell et al. 2022a).
PMET Li al. 2024), MEMIT (Meng et al. 2023b), and AlphaEdit (Fang et al. 2025). Additional baselines include ROME (Meng et al. 2023a), RECT (Gu et al. 2024). PRUNE (Ma et al. 2024).
and NSE (Tiang al. 2024).
TinniTuning (FT-L& & FT-W). FT-L fine-tunes only the weights of specific layer (as identified by ROME), rather than all layers. FT-W is a variant of FT-L that differs slightly in the loss used for parameter optimization under regularization.
MEND (Model Editor Networks with Gradient Decomposition). Edits large pre-trained models from a single input-output pair by applying a low-rank decomposition to the fine-tuning gradient and using small auxiliary "editor" networks for fast, localized parameter updates that mitigate overfitting.
PMET (Precise Model Editing in Transformers). Observes that hidden states arise from FFN, MHSA, and residual paths. It assumes MHSA encodes general extraction patterns and need be altered; PMET jointly optimizes hidden states for FFN/MHSA but updates only FFN weights using the optimized FFN state make more precise edits.
MEMIT (Mass-Editing Memory in Transformer). Extends ROME to insert many new factual memories efficiently by targeting transformer modules that causally mediate factual recall, enabling simultaneous updates for thousands of associations.
16

---

Published as a conference paper at ICLR 2026
AlphaEdit. Within the locate-then-ecée then-edit paradigm, projects the parameter perturbation onto the null space of knowledge to be preserved before applying it, so outputs for preserved queries remain unchanged and corruption during sequential edits is reduced.
ROME (Rank-One Model Editing). Identifies key mid-layer feed-forward activations that in- fluence factual predictions and applies direct rank-one weight update to modify specific factual
associations.
RECT (Regularizing Causal Tracing). Regularizes weight updates during editing prevent ex- cessive changes and overfitting, mitigating side effects (e.g., reasoning degradation) while maintaining general capabilities.
PRUNE (Preserving Representations through Unitary Nullspace Editing). Constrains the edited matrix (e.g., via canditionnnumber control and null-space restrictions) perturbations remain limited to stored knowledge, preserving overall ability under sequential edits.
NSE (Neuron-level Sequential Editing). It improves sequential editing by computing target values with original weights for value computation and by updating only activation-barl based influential neurons the neuron level, rather than modifying the full weight matrix.
C.2 HYPERPARAMETER SETTINGS
We use a unified hyperparameter configuration across locate-then-edii methods (ROME, MEMIT, PMET, RECT, PRUNE, NSE), as well AlphaEdit and SUIT, and deviate only when method-specific constraints apply. Following prior tocate-then-edit work (Meng et al. 2023a b). we set the two hyperparameters described Appendix A.4 ÄKL and AWD. to 0.0625 and 0.5, respectively. SUIT does not use λKL or AWD, and therefore does not require these hyperparameters.
For model-specific layer selection, following prior work (Meng al.. 2023a b), we use zero-based layer indices {4. 5, 6, 7, 8 for LLaMA3-Instrus (8B), {3, 4, 5, 6, 7, } for GPT-J (6B), and {4. 5, 6, 7, 8} for @ single target layer (index 5). In addition, following AlphaEdit, we set the null-space threshold to 0.02 and the L2 regularization coefficient to 10 for both AlphaEdit and SUIT. In AlphaEdit, the null-space threshold controls the strength of the preservation constraint, with smaller values imposing tighter constraint.
(7B). Consistent with the original ROME setup. ROME edits only
For SUIT, as described in 6.33 sweep the hyperparameters over (0.1, 0.9} and select the values that maximize the harmonic mean score S. which balances Eff., Gen., and Spe.. Specifically, use À 0.3 for all models, and set Tenergy 0.4 for LLaMA3 and Tenergy 0.6 for GPT-J and Qwen2.5.
For methods outside the original papers. Finally, for hyperparameters not explicitly specified in this section, we follow the original works exactly. When such settings are not provided in the original papers or code, default to the settings used in the EASYEDIT open-source framework, which provides unified implementation of knowledge editing methods Wang al. (2024a).
family, MEND and FT-L/FT-VV W are tuned following their
C.3 TENERGY HYPERPARAMETT ANALYSIS ON GPT-J AND QWEN2.5
In Fig. our analysis on LLaMA3 shows that Tenergy has a sensitive effect on editing metrics. Motivated by this observation, we conduct the same hyperparameter analysis on GPT-J and Qwen2.5 in this section. To compare sensitivity to Tenergy we follow the same setup as in Fig, [
evaluation on FAUNTERFACT with 1,000 edits (batch size 100), while fixing 0.3 and varying Tenergy- Both models exhibit the same qualitative trend as LLaMA3: as Temergy increases, Spe. generally improves, while excessively large values can reduce Eff. and Gen.. However, for GPT-J and Qwen2.5, the trend less pronounced than for LLaMA3, indicating relatively lower sensitivity to Tenergy. Table reports the full sweep results. Accordingly, we select hyperparameters using the same criterion as in the main text, namely by maximizing the harmonic mean score S over Eff.. Gen., and Spe.. As shown in Table 7 Tenergy = 0.6 achieves the highest S for both GPT-J and Qwen2.5, and we use this value for both models.
17

---

Published as a conference paper at ICLR 2026
Table Tenergy sweep on GPT-J and Qwen2.5.
GPT-J
Qwen2.5
Tenergy
S
Eff.
Gen.
Spe.
S
Efft
Gen.
Spe.
0.1
82.54
99.10
94.70
63.71
85.42
99.30
91.05
71.08
0.2
82.71
99.10
94.70
64.01
85.26
99.30
91.20
70.66
0.3
83.11
99.10
94,80
64.69
84.99
99.30
90.70
70.41
0.4
84.00
99.10
94,90
66.28
85.26
99.20
90.35
71.24
0.5
84.23
99.10
94.70
66.82
85,40
99.30
91.05
71.04
0.6
84.67
99.20
94.60
67.66
86.12
99.20
91.55
72.30
0.7
84.54
99.10
94.25
67.63
85.88
99.30
91.00
72.08
0.8
84.61
99.10
93.45
68.19
85,45
99.10
91.00
71.29
0.9
84.64
99.10
93.05
68.46
86.10
99.20
91.00
72.61
D DETAILED DESCRIPTION OF EVALUATION METRICS
Let denote the old attribute and a* the attribute. For each item i, let x; be the rewrite prompt (i.e., (es, r.)), (xx;) ) its paraphrase prompts, and Oz;;) its neighborhood prompts. All probabilities P..(- ()) are computed under the language model fo. These evaluation metrics are not new; we follow established practice from prior work (Fang et al. 2025 Meng et al. 2023abb)
D.I COUNTERFACT METRICS
Probability- criterion. The following three metrics use probability comparisons between the edited target a and the original a.
Efficacy.
E, I(p/sa" x;)) > Pso(a w:)]
Generalization.
E, P/s(a" N(xx)) > Pyí(a N(x:))]
Specificity.
E, 1 Pro(a O(xi)) > Pi((a') O(xi))]
Generation- based criterion (exact match). Letr(a") = (ai:.... . a..). Success if every target token is the greedy choice at its step:
Ed 1 I (I,...,TT))
T*) of = arg max Pio(y aze 2:)
measure for excessive repetition model outputs. It uses the entropy of n-gram distributions:
Fluency (generation entropy).
2
92 log, (kk) + 4 gkk log, gs(),,
(22)
where 9-(+) is the n-gram frequency distribution.
Consistency (reference score). The consistency of the model's outputs is evaluated by giving the model fe entity and computing the cosine similarity between the TF-IDF vectors of the model-generated text and reference Wikipedia text about a.
We report the results based on the generation-base criterion in Table as it serves as more stringent evaluation and more directly reflects reployment-relevant behavior: in practice, edited knowledge is only actionable when it is realized in the model's generated responses. For direct comparison with the traditional probability-ben based metrics commonly used in prior literature, we additionally report those results in Table 9
18

---

Published as a conference paper at ICLR 2026
D.2 ZSRE METRICS
Token-level partial credit. For target string y with tokenization rly) (..,...., M/)) and prompt x, define the token-level accuracy under teacher-forced greedy decoding
TokenAco(s,g)
1 yt = arg max Ptu(v) yet, m))
Efficacy. Average token-level accuracy on rewrite prompts:
E, TooenAAcc(2, a²)]
Generalization. Average token-level aceuracy on paraphrase prompts: E (ookenAcc(N(z)) a-)]
Specificity. For zsRE, evaluate Specificity using exact match against the pre-edit model's output on neighborhood prompts, rather than the dataset gold answer. This is because, as discussed in Appendix DD.3] models often fail to exactly reproduce the zsRE gold answer even before editing, making gold-referenced exact match a weak preservation signal.
Let af be the output generated by the pre-edit model for Ozz;), and let a. post be the output generated by the post-edit model for the same prompt. Then, Specificity is defined as:
E, 1 a post
D.3 zsRE DATASET DETAILS AND RATIONALE FOR OUR EVALUATION PROTOCOL
In this section, we explain why we cannot apply the same pre-knowledge filtering used for COUN- TERFACT to Z.RE. This issue arises from the question-style prompts and response format of 7sRE, and therefore requires a separate evaluation protocol.
As described in $ [.1 for COUNTERFACT we evaluate only instances where the pre-edit model generates the ground-truth answer. This ensures that Efficacy/Generulization reflect edits to known facts (not insertion), while Specificity measures preservation of previously exhibited knowledge. However, unlike declarative datasets such as COUNTERFACT, ZsRE uses question-style prompts, making the same pre-knowledge filtering unreliable. Without additional instructions, the model often does not directly generate the dataset ground-truth answer span in a short attribute-only form. For example, for the prompt "What airport is China United Airlines part of?", the gold attribute is "Beijing Nanyuan Airport," " but the model may instead generate longer sentence such as "China United Airlines is part of Beijing Capital International Airport."
To examine this mismatch more concretely, we analyze the first-token distribution. For LLaMA3, zsRE answers frequently begin with function words or pronouns unrelated the gold attribute (e.g.. "the" 4,902 times, "a" 445 times, "he" 388 times, and "she" 89 times). This makes an argmax-based pre-knowledge check unreliable, since the earliest generated tokens often do not correspond to the target attribute span. As a result,
pre-knowledge filtering leaves almost no valid instances on ZsRE; for LLaMA3, only 144 of 19,086 instances remain when we additionally require ground-truth generation for the neighborhood prompts. Accordingly, we evaluate ZSRE without this pre-knowledge condition. Consequently, because the dataset ground-trum answer does not necessarily reflect knowledge actually exhibited by the pre-edit model ZsRE, do not use as the preservation reference for Specificity. Instead, as described in Appendix D.2] we use the model's pre-edit generated response to the prompt as the reference and evaluate Specificity by exact match.
E ADDITIONAL EXPERIMENTAL RESULTS
E. EXTENDED BASELINE COMPARISONS
Table & presents results for additional extended baselines that were reported in Table
19

---

Published as a conference paper at ICLR 2026
Table 8: Results on COUNTERFACT (batch size 100).
COUNTERFACT
Model
Method
St
Eff 1
Gen. 1
Spe. t
Flu. f
Con. 1
GC1
Pre-edit
0.0
0.0
0.0
100.0
634.9
20.9
62.3
LLaMA3
FT-W
4.9
4.0
2.9
43.5
634.4
21.4
60.5
ROME
0.0
0.0
0.2
0.0
481.5
4.2
0.0
RECT
55.8
81.6
72.4
36.1
634.4
35.3
60.4
PRUNE
28.6
43.3
39.3
17.7
590.4
33.9
45.0
NSE
3.3
1.4
5.8
62.2
609.6
SUTT
86.8
99.7
90.3
74,2
631.2
61.8
Pre-edit
0.0
0.0
0.0
100.0
621.1
23.9
18.6
FT-W
5.8
12.3
2.4
49.0
613.4
25.7
36.1
GPT-
ROME
0.2
0.2
0.2
407.2
RECT
66.8
92.9
85.8
44.4
625.0
47.6
14,9
PRUNE
31.1
51.8
56.0
16.9
504.6
29.4
29.4
NSE
2.2
0.8
12.6
54.1
608.0
34.2
27.5
SUTT
82.3
98.6
93.3
64.1
619.4
49.4
17.8
Pre-edit
0.0
0.0
0.0
100.0
625.5
21.9
20.8
Qwen2.5
FT-W
10.3
47.9
31.7
4.2
476.7
4.5
0.0
ROME
25.1
51.6
33.7
14.2
440.1
15.6
0.6
RECT
64.0
86.3
85.9
42.3
625.8
37.7
59.9
PRUNE
15.2
28.2
30.7
7.7
588.1
30.4
NSE
0.0
0.0
0.0
99.5
625.6
21.7
39.3
SUIT
85.7
99.5
86.8
74.4
626.2
37.4
23.7
Table 9: Probability-based criterion results /ESpe//een/Se.) on three models. S denotes the harmonic mean of Eff., Gen., and Spe.
LLaMA3 (Prob)
(Proh)
Qwen2.5 (Prob)
Method
St
El/
Gen.
Spe.
Method
S
Eff
Gen.
Spe.
Method
St
ElE
Gen,
Spe.
FTW
11.3
9.5
6.7
92.4
FT-W
13.6
21.8
6.1
88.4
FTW
54.7
82.3
71.7
34.8
59.5
67.1
ROME
51.2
49.6
48.9
55.6
66.5
76.7
PRUNE
69.5
76.5
75.
59.7
PRUNE
72.6
87.2
88.3
53.9
PRUNE
66.6
72.2
73.6
$6.7
NSE
73.2
83.7
53.1
98.0
NSE
82.3
88.5
70.6
90.9
NSE
0.0
0.0
100.0
MEND
51.3
52.7
53.1
48.4
MENI)
48.4
46.0
46.1
53.9
MEND
51.0
54.0
$3.9
46.8
MEMIT
82.6
90.8
88.8
MEMIT
85,0
96.8
67.8
MEMIT
86.9
92.2
95.1
75.8
PMET
76.2
76.4
AlphaEdi:
87.2
99.7
94.1
72.7
AlphaEdit
91.0
99.6
97.9
78.6
AlphaEdi
91.7
99.2
98.3
80.3
SUIT
91.3
100.0
90.8
84.4
SUIT
94,9
99.3
96.2
89.7
SUIT
96.4
100.0
94.4
95.0
E.2
DETAILED F1 SCORES ON GENERAL CAPABILITY BENCHMARKS
E.2.1 GENERAL CAPABILITY BENCHMARK DATASETS
To evaluate the general capabilities of language models, several well-known benchmark datasets are utilized. The GLUE (General Language Understanding Evaluation) benchmark is prominent collection of diverse natural language understanding tasks (Wang et al. 2019). Key datasets included in GLUE are: SST-2 (The Stanford Sentiment Treebank), ), a single-sentence classification task for sentiment analysis movie reviews (Socher al. (013); MRPC (Microsoft Research Paraphrase Corpus), which involves determining if a pair of sentences are semantically equivalent (Dolan & Brockett 2005); RTE (Recognizing Textual Entailment), a task that assesses whether a premise sentence logically entails hypothesis (Bentivogli al.) 2009); and CoLA (Corpus of Linguistic Acceptability). where the task is to decide if sentence is grammatically acceptable (Warstadt et al. 2019). Furthermore, NLI (Natural Language Inference) tasks, which require inferring the logical relationship (entailment, contradiction, or neutral) between a pair of sentences, are a crucial part of the evaluation (Williams et al. 2018).
20

---

Published as a conference paper at ICLR 2026
Beyond GLUE, more comprehensive benchmarks exist to multi-task task proficiency. MMLU (Massive Multi-task Language Understanding) is a benchmark designed to measure a text model's multi-task accuracy under zero-shot and few-shot settings across wide range of subjects (Hendrycks et al.
2021
E.2.2
GENERAL CAPABILITY BENCHMARK RESULTS
Table 10 reports the F1 scores for each benchmark the General Capability evaluation.
Table 10: F1 scores per benchmark.
Model
Method
SST
MMLU
MRPC
CoLA
RTE
NLI
Avg.
Pre-Edit
81.78
59.93
65.28
76.72
27.65
69.27
63.44
LLaMA3
FT-L
0.00
0.00
37.34
0.00
0.00
0.00
6.22
MEND
0.00
0.00
0.00
0.00
MEMIT
75.28
55.93
64.80
69.57
31.46
67.90
60.82
PMET
64.25
28.33
60.70
55.52
32.16
59.55
50.09
AlphaEdit
77.87
57.82
61.72
76.36
31.52
67.64
62.16
SUIT
77.18
58.93
65.64
77.63
29.53
69.26
63.03
Pre-Edit
0.00
5.78
23.16
21.41
42.67
52.78
24.30
FT-L
0.00
17.74
19.67
15.92
46.13
45.79
24.21
GPT-J
MEND
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MEMIT
0.00
5.89
6.73
13.29
41.56
53.33
20.13
PMET
0.00
12.22
44.69
46.17
19.09
AlphaFdit
0.00
4.30
5.81
9.26
44.91
52.79
19.51
SUIT
0.00
9.44
24.20
21.78
33.66
33.03
20.35
Pre-Edit
13.66
2.46
53.32
23.36
11.21
69.23
28.87
Qwen2.5
FT-L
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MEND
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MEMIT
54.31
0.79
55.82
24.85
11.49
61.48
34.79
PMET
15.15
14.78
2.27
3.07
27.16
26.50
14.82
AlphaEddi
59.25
0.39
39.63
6.47
11.56
$1.52
28.14
SUIT
19.07
3.23
55.03
32.27
8.80
66.23
30.77
E.3 DETAILED RESULTS FOR THE 5,000-EDIT SETTING ON COUNTERFACT
Fig. reports the average performance across all benchmarks, while Fig. presents the scores for each benchmark individually. Table IT presents the performance metrics the CounterFact dataset, where 5,000 cases were sequentially edited in batches of 100.
Table 11:
Full performance metrics on COUNTERFACT dataset for LLaMA3. Metrics are grouped by generation- criterion and probability-based criterion.
Generation-bused
Probabilit--buset
Method
Flu. 1
Con. f
S1
Eff. 1
Gen, f
Spe.
S1
Eff. 1
Gen. f
Spe.
SUIT
38.0
95.8
58.2
19.5
89.9
99.0
89.0
83.2
624.1
33.5
AlphaEdit
28.9
89.7
69.4
12.8
80.6
97.6
92.9
61.7
613.9
33.6
E.4 SCALING TO LARGER MODEL
We further evaluate SUIT larger-scale model to examine whether its benefits persist beyond the model sizes considered in Table I Specifically, we rum additional experiments on LLaMA2- 13B (Touvron et al. 2023) using COUNTERFACT. Since prior knowledge-editing baselines do not provide tuned hyperparameters or reported results at this scale, we ensure comparability by reusing exactly the same hyperparameter settings those used for LLaMA3-8B (Appendix C.2 for all methods,
21

---

Published as a conference paper at ICLR 2026
1.0
0.2-
0.00
0.45
2UT
Edi Number
Edtt Nummbrr
TdE Number
1050
Sdihuuburr
Edlt Number
Edit Number
Figure 6: F1 scores for each benchmark.
including SUIT. We then evaluate every method under the sequential batch-editing setup batches x 100 edits). T2 shows that SUIT achieves the highest score, with substantial improvement in Specificity and a concurrent gain in Generalization. These results suggest that SUIT scales favorably to larger models while maintaining strong balance among Efficacy, Generalization, and Specificity, even without any model-specific retuning.
Table 12: Results on LLaMA2-13B.
Method
Eift
Gen.
Spe.
MEMIT
73.2
90.0
68.3
65.6
AlphaEdit
79.3
98.4
76.5
68.4
SUIT
84.7
95.0
79.1
81.7
E.5
EVALUATION ON CHED (CONTEXTUAL HOP EDITING DATASET)
Table 13: Performance comparison on CHED. Each column corresponds to rewrite prompt augmented with one of six prefix-context types: Subject, Obj-Old, Obj-New, and their 1-hop variants.
Context Types
Subject
Obj-Old
Obj-New
Subject Hop
Obj-Old Hop
Obj-New Hop
Avg.
MEMIT
75.4
73.6
77.8
74.1
70.4
75.7
74.5
AlphaEdit
92.7
88.6
94.2
90.4
87.9
92.0
91.0
SUIT 95.7
92.0
95.6
94.3
91.2
93.7
CHED (Park et al. 2025) extends COUNTERFACT by evaluating whether knowledge edits remain robust under additional prefix contexts. Specifically, each rewrite prompt (e, is preceded by sentences derived from either the original entity e, the old attribute the new attribute a", their one-hop neighbors. The six context types thus test whether the edited model can maintain correctness when auxiliary but semantically related cues are introduced. As shown in Table [3] SUIT consistently outperforms across all context types, indicating strong resilience to contextual variation.
E.6 RIPPLEEDITS EVALUATION RESULTS
We evaluate ripple effects using the RippleEdits benchmark (Cohen et al. 2024). which measures whether single factual edit correctly propagates to logically compositionally implied facts while preserving unrelated knowledge. We follow the benchmark's 's standard setup and test 100 single edits from the POPULAR split. This split consists of high-frequency entities and represents higher-severity
22

---

Published as a conference paper at ICLR 2026
Table 14: RippleEdits results 100 single edits from the POPULAR split.
Method
LG
CI
CII
SA
PV
RS
AlphaEdit
0.210
0.482
0,000
1.000
1.000
0.597
SUIT
0.257
0.497
0.000
1.000
1.000
0.500
MEMIT
0.188
0.256
0.000
0.131
0.074
0.220
regime that is more likely to surface ripple errors. We report accuracy the six RippleEddits criteria: Logical Generalization (LG), Compositionality I/II (CI/CII, Subject Aliasing (SA), Preservation (PV), and Relation Specificity (RS).
Overall, SUIT clearly outperforms MEMIT on this challenging split and remains broadly competitive with AlphaEdit, indicating strong robustness to ripple effects even in high-severity setting.
DETAILED VISUALIZATION OF COMPONENT EFFECTS
As stated in $ 4.3 and $ 6.2.2 we posited that w increases logit new attribute a" and W2 decreases the logit of the original attribute a. To test this, analyzed the individual roles of wT and W2 by decomposing our residual vector into its components:
Aw (h W2 h Wi)Wi
Aw2 (h w1 - h w2)w2
We then observed the changes in logits for the original attribute ("Google") and the new attribute a' ("Apple") by incrementally adding each component to the residual stream h, scaled by an interpolation factor & 1].
Fig. 7 provides full breakdown of these effects for the edit ("Chrome", "was developed by", "Apple"). The results confirm finding that the Aw1 component is effective increasing the logit of the new attribute a", while the Aw₃ component is effective at decreasing the logit the old attribute
However, the plots also illustrate that w also suppresses the old attribute a, and W2 promotes the new attribute a", rather than each playing only single role. This visual evidence reinforces the point that the components are not fully disentangled.
Oed welit depher depbrr
Liady Gowper
ent Seeev1
03
63
64
36
3.5
33
02
04
3.
3.0
33
0.2
64
25
02
Figure 7: A full breakdown of the effects of applying scaled components Aw1 and Aw3 to the residual stream.
G CUMULATIVE ENERGY CURVES ACROSS MODELS
We analyze the spectral distribution of the key vector matrix Keatity defined in 4.2 the cumulative energy ratio of the singular values S diag(o)...., 0,) across first Tour edited layers. The æ-axis represents the rank index k. and the y-axis indicates the proportion of total energy explained by the top-k components, calculated as Σ / The red dots in the figure mark the "knee points," computed using the Kneedle algorithm (Satopaa et al. (011). which represent the points of maximum curvature.
Fig. g visualizes
23

---

Published as a conference paper at ICLR 2026
101 Hayer
2nd layer
3rdlayer
4t% ayar
1.-
26-
5.4
33-
24-
1.C
GFT-68
0.2
52
4300
2009
2.0
3.-
0wer2.5-7D
1/4
344
04
8.4
2009
4006
0083
1000
10000
3000
4008
5300
8000
36800
2000
4800
0009
0900
10000
2003
4008
5309
5008
10000
3.0
1.6
1.0
1.0
PN-35
9.A
84-
844
8.0
2090
206
66/6
3.
2030
495
3008
8003
8.0
2051
3.-
8.5
3.6
0.5
3.6
0.
Pack
Tant
Park
Peck
Figure 8: Cumulative energy curves of the layers targeted for editing across five models.
H PRE-COMPUTATION COST FOR BUILDING KENTITY
This section reports the one-time pre-computation (defined in (42) and running SVD, as function of the number of entities N. All timings and memory measurements obtained on an NVIDIA A100 80GB PCIe GPU. These steps are performed once model-layer pair prior to editing; afterward, editing only requires loading the precomputed (U,S) factors and applying an energy-based truncation, which adds negligible overhead.
of constructing the entity matrix Kenity
Table 15 shows that the dominant cost is computing the vectors used to form Kenity. which scales approximately linearly with N. contrast, the SVD time comparatively small but still grows with N. and peak VRAM increases superlinearly in our setup (LLaMA3-Instrud (8B), averaged over five edited layers). To quantify sensitivity to N, we compare subspaces obtained with smaller values of N to the reference subspace extracted from the full 25, 25.799-entity using the mean canonical correlation; Table [6] shows that similarity rises quickly with and already reaches 0.81 N=10k, close to the full-data subspace.
24

---

Published as a conference paper at ICLR 2026
Taken together, these results indicate that N=10,,00 provides favorable trade-off, substantially reducing one-time pre-computation time and peak memory compared to the full setting while producing subspace that remains highly aligned with the full-data reference; consistent with this, editing performance using the N=10k subspace is nearly indistinguishable from performance using the full-entity subspace (Table 17
Table 15: One-time pre-computation cost for building Kentity and running SVD as function of N (LLaMA3- (8B), five layers; NVIDIA A100 80GB PCIe).
N
Peak VRAM
k-vectors (s)
SVD (s)
Total (s)
500
2.71 GB
256.1
0.5
256.6
IK
2.74 GB
528.4
7.1
535.5
2K
2.80 GB
1059.3
2.8
1062.0
4K
2.95 GB
2245.5
10.9
2256.4
6K
3.13 GB
3405.9
29.5
3435.4
10K
3.59 GB
5906.3
120.6
6026.9
14K
4.16 GB
8043.0
333.3
8376.3
5.95 GB
11139.7
402.6
11542.2
22K
8.25 GB
16520.9
501.7
17022.6
25.8K
10.82 GB
19565.8
589.2
20154.9
Table 16: Subspace similarity to the full-data reference as N varies, measured by mean canonical correlation.
Entities (N)
500
1k
2k
4k
6k
10k
14k
18k
22k
25.8k (full)
Mean Canonical Corr.
0.35
0.42
0.55
0.68
0.72
0.81
0.86
0.91
0.97
1.00
Table 17: Editing performance using the N=10k subspace vs. the full-entity subspace.
Setting
Eiff.
Gen.
Spe.
N = 10,000
86.8
99.7
90.3
74.2 88.6
Full (N = 25,799)
87.4
99.4
77.2
I MEASURING ENTITY VS. CONTEXT EFFECTS IN K, AND K-
To complement our variance-based subspace construction, we perform 4-way cross-pair analysis to test how the entity-agnoste subspace K,! and the entity-specifie subspace Kc, respond to controlled changes in entity identity and sentence context. The goal is to disentangle entity effects from context effects using paired interventions.
We sample two entities (Fi, F22) and two contexts (C1, C2), and construct four sentences:
{S( Ci), S(E,, C2), S(2,, Ci), S(E2, C2))
For illustration, C₁ and C2 can be instantiated so that the entity's 's syntactic role and position are fixed while only the prefix context changes, e.g., C1: "In the news article, {E} was mentioned repeatedly." and Cz: "During the committee meeting. {E) was mentioned repeatedly."
Since we extract the key vector at the entity's 's last-token position a decoder-only causal LM, the representation at that position depends only the left context (prefix) before the entity. Therefore, context variation is applied to the prefix preceding {E).
For each sentence, extract the key vector k the entity's 's last-token position and project it onto each subspace. Let K. E {Ka, KC-1 }, and let K(E, C') denote the projected key vector obtained from SEE, C) in subspace KC.
We then define two controlled changes. The context-swap change measures how much the representa- tion changes when the entity is fixed and only the context is changed:
dkc
[KC(F,, C1) C2)11.
25

---

Published as a conference paper at ICLR 2026
The entity-swap change measures how much the representation changes when the context is fixed and only the entity is changed. To reduce -ppeeii-ppecifie bias, we compute it under both contexts and average:
XCEE, (x)1
eni
Because X. and K may have different norm scales, absolute distances alone can confound sensitivity with scale. We therefore also report norm-normalized distance:
||a bll2
rd b)
|||| |0|12
We define rd and rd by replacing each equared-distanee term in de and d with rd(:;)) re- spectively, so that the comparison reflects relative representational change rather than raw magnitude.
To summarize entity sensitivity, we compare entity-swap and context-swap changes using a context- baseline-subtracted gap:
dem
de
r
rd
rd
These quantities measure how much additional change is induced by swapping the entity, beyond the change already caused by swapping the context; larger values indicate stronger entity sensitivity relative to context sensitivity.
All values are averaged over 1,000 randomly sampled 4-way cross-pairs. Both subspaces are more sensitive to entity identity than to contextual variation, but the effect is substantially stronger in K.g. In KC,, the entity-swap change clearly exceeds the context-swap change (d a 18.09 vs. dhe R 2.79), yielding a large gap (A,, R 15.30). In contrast, K3- shows much smaller absolute variation (dex z 0.42, dea = 4.52) and smaller gap (A#; = 4.09). This pattern remains after norm normalization: the normalized gap is also much larger in K., (rAK, 0.80) than in K-+ (r ~ 0.36). Overall, these results support the interpretation that IC, amplifies entity-specific differences more strongly, whereas K,_ is comparatively less entity-selective.
ADDITIONAL HEATMAP VISUALIZATIONS
This appendix presents extended visualizations to complement the perturbation analysis in $ 6.1 While the main text provides a representative example, we offer these additional examples to illustrate how the perturbation patterns manifest across different contexts.
We provide further examples from the Wikinews Article Dataset, illustrating the perturbation each entity's 's last token position. For each of the five articles presented, we show the L2 norm of the difference in the residual streams of the final edited layer between the original model and models edited using SUIT, MEMIT, and AlphaEdit.
The figures are presented in the order: SUIT, MEMIT, AlphaEdit. As visually represented by the color intensity, SUIT consistently reduces the perturbation on the last token of the entity compared to both MEMIT and AlphaEdit. The same color indicates the same amount of perturbation across all figures.
26

---

Published as a conference paper at ICLR 2026
Prime minister Hari Kostoy of Macedonia has resigned from his position as of Monday, November 15. The Maced onian parliament will meet on Thursday to decide whether not to accept his resignation. The BBC qucted Kostov, who appointed last May, have said that "there is will for genuine teamwork within the coalition, and that of the parties the current government has been promoting corruption and nepotism. Kostov also claimed that pcccuptttioon the rights of Albanian ethnic minority Macedonia was obstructing economic madernization, according to Reuters, Kostov himself was not a member any of the coalition's parties. Kostov's resignation was preceded by referendum organized by the Macedonian opposition, aimed against decentralisation law which would have given the Albanian ian ethnic minority in Macedonia additional rights. The referendum was declared null and void due to low turnuut. According the NOS, some fear that fights between Albamian guerrillas and the Maced onian army, which to halt 2001, will again.
Prime minister Kostex Macedonia resigned from his position as Monday, November The Maced onian parliament will meet on Thursday to decide whether or not to accept resignation. The BBC quated Kostov, who was appointed last May, to have said that "there is no will for genuine teamwork within the coalition, and that one of the parties in the current government has been promoting corruption and nepotism. Kostov also claimed that the preoccupation with the rights of Albariaa ethnic minority in Macedonia was obstructing economic madernization, according Reuters, Kostov himself not member of any of the coalition's parties. Kostov's resignation was preceded by referendum organized by Macedonian opposition, aimed against decentralisation which would have given Albanian ethnic minority Macedonia additional rights. The referendum was declared null and vold due to low turnuut. According to the NOS, some now fear that fights between Albamian guerr illas and the Maced omian army, which came to halt 2001, will start again.
Prime minister Hari Kosted ex of Macedonia has resigned from his position as of Monday, November 15. The Maced onian parliament will meet Thursday to decide whether not to accept his resignation. The BBC qucted K costov, who appointed last May, have said that "there is will for genuine teamwork" within the coalition, and that one of the parties the current government has been promoting corruption and ispotismm Kostov also claimed that pre occupatiioo the rights of Albariian ethnic minority Macedonia was obstructing economic modernization, according to Reuters, Kost ov himself was not a member any of the coalition's parties. Kostov's resignation was preceded by referendum organized by the Maced onian opposition, aimed against decentralisation law which would have given the Albanian ethnic minority in Macedonia additional rights. The referendum was declared null and void due to low turout.. According the NOS. some fear that fights between Albamian guerrillas and the Maced onian army, which to halt 2001, will again.
Figure 9: Perturbation heatmaps for sample article
27

---

Published as a conference paper at ICLR 2026
The proposal from Dutch Minister of Justice Piet Hein Don strengthen the anti-blasphemy provisions of the criminal code has been rejected by majority of the Tweede Kamer, the country's parliament. Don put forth the proposal shortly after the murder Dutch filmmaker Theo Gogh, but denied that Van Gogh's death anything to
proposal, NOS reported Donner's own party,
Democratic
Appeal (CDA), ) supported his proposal. However, their two coalition partners mamp:mdash: the People's Party
for Freedom and Democracy (Volkspartij voor Vrijheid en Democratie ie VVD) and Democrats 66 (D66)
ammp; md as: announced they would not back the ban, The Labour Party, the largest opposition party, also refused to vote favour. Without their support, the motion could not be passed. Consequently, NRC Hand els blad reports, Donner withdrew his proposal. Although the Dutch criminal code already makes blasphemy been enforced three times since the 193 The article question who riclic cleric or relic may be imprisoned up to three months, According to Dutch broadcaster RTL, Member of Parliament ousewies van der Laan (D66), will make motion on November 17 to have the article removed entirely from the criminal code.
illegal, the
that anyone
The proposal from Dutch Minister of Justice Piet Hein Don ner strengthen the anti-blas anti-blasphemy provisions of the criminal code has been rejected majority of the Tweede Kamer, the country's parliament. Donner put forth propesal shortly after the murder Dutch filmmaker Theo van Gogn. denied that Gogh's death anything to do the proposal. NOS reported that Donner's own party, the Christian Democratic Appeal (CDA), supported his proposal. However, their two coalition partners mmmp: the People's Party for Freedom and Democracy (Volkspartij voor Vrijheid en Democratle, VVD) and Democrats (D65)
aamp: mddah: announced they would not back the ban. The Labour Party, the largest opposition party, also refused to vote favour, Without their support, the metion could not be passed. Consequently, NRC Hand els blad reports, Donner withdrew his proposal. Although the Dutch criminal code already makes blasphemy illegal, the
times since the 1930s. who ridic a cleric or relic may be imprisoned for up to three months. According to Dutch broadcaster RTL, Member of Parliament Lousewies van der Laan (D66), will make motion on November 17 to have the article removed entirely from the criminal code.
only been enforced
question states anyone
The proposal from Dutch Minister of Justice Piet Hein Don ner strengthen the anti blasphlashhmmy provisions of criminal code has been rejected majority of Tweede Kamer, the country's parliament. Den put Gogh's
forth propesal shortly after the murder Dutch filmmaker Theo van Gogn, denied
death had anything to do the proposal. N NOS reported that Donner's own party, the Christian Democratic Appeal (CDA). supported his proposal. However, their two coalition partners aamp:mdash: the People's Party
for Freedom and Democracy (Volkspartij voor Vrijheid en Democrat e. VVD) and Democrats 56 (D65)
& mddas:: announced they would not back the ban. The Labour Party, the largest opposition party, also refused to vote favour. Without their support, the motion could not be passed. Consequently, NRC Hand els blad reports, Donner withdrew his proposal, Although the Dutch criminal code already makes blasphemy illegal, the law has only been enforced three times since the 1930s. The article question states that anyone who ridicules a cleric or relic may be imprisoned for up to three months. According to Dutch broadcaster RTL, Member of Parliament Lousew le van der La (D66), will make motion on November 17 to have the article removed entirely from the criminal code.
Figure 10: Perturbation heatmaps for sample article 2.
28

---

Published as a conference paper at ICLR 2026
Secretary of State Colin Powell submitted his long expected resignation of Monday, November 15, and resignation was accepted President George
the President Friday. Powell has said that was always his intention to serve only one term. San Gabriel Valley Tribune said that Powell often had disputes with Bush Administration officials holding what the newspaper termed "more hawkish" views. On Tuesday, President Bush announced his nomination of National Security Advisor Dr. cledzle Rice as Powell's successor. Reuters cited senior Bush administration officials as saying that her deputy, Stephen Hadley, will succeed her her role Assistant the President for National Security Affairs if she confirmed Secretary State.
resignation
Secretary State Colin Powell submitted his long-expected resignation as Monday, November 15, and resignation was accepted by President George W. Bush. His resignation letter was sent to the President on Friday, Powell has said that was always his intention to serve only one term. The San Gabriel Valley Tribune said that Powell often had disputes with Bush Administration officials holding what the newspaper termed "more hawkish views. On Tuesday, President Bush announced his nomination of National Security Advisor Dr. Cond bleezae Rice Powell's successor. Reuters cited senior Bush administration officials saying that her ley will succeed her her she confirmed Secretary State.
deputy, Stephen Had
President National Security Affairs
Secretary of State Colin Powell submitted his long expected resignation as of Monday, November 15, and his resignation was accepted by President George W. Bush. His resignation letter was sent to the President Friday, Powell has said that was always his intention to serve only term. The San Gabriel Valley Tribune said that Powell often had disputes with Bush Administration officials holding what the newspaper termed "more hawkish" views. Tuesday, President Bush announced nomination of National Security Advisar Cond ole Rice as Powell's successor. Reuters cited senior Bush administration officials as saying deputy, Stephen Had lyy will succeed her her role as Assistant to the President for National Security Affairs if she is confirmed as Secretary of State.
Figure 11: Perturbation heatmaps for sample article 3.
29

---

Published as a conference paper at ICLR 2026
Investigators said that Saddam Hussein diverted money from the -ol-for-Food Program pay millions of dollars Strip who carried attacks Paul cker. former American official investigating the diverted funds scandal, has taken some heat from advocates demanding that haul United Nations personnel before the US Congress, His reason for not subjecting them to this degree of open scrutiny is that it would have the perverse effect of pushing them into refusing to cooperate with the investigation all. He plans to release documentary evidence early next when his investigation is complete.
families of suicide bombers the West Bank
Investigators said that Saddam Hussein diverted money from the Oil Oil-for-F -ood Food Program to pay millions dollars to families of suicide bombers from the West Bank and Gaza Strip who carried out attacks on Israeli civilians, Paul Vol cker. former American official investigating the diverted funds scandal, has taken some heat from advocates demanding that he haul United Nations personnel before the US Congress, His reason for not subjecting them this degree of scrutiny is that it would have the perverse effect of pushing them into refusing cooperate with the investigation all. He plans release documentary evidence early year, when investigation complete.
Investigators said that Saddam Hussein diverted money from the Oil bil-for-Food Food Program to pay millions of dollars to families of suicide bombers from the West Bank and Gaza Strip who carried out attacks Israeli civilians, Paul Vol cker former American official investigating the diverted funds scandal, has taken some heat from for
advocates demanding that haul United Nations personnel before the US Congress. perverse effect pushing into refusing to cooperate the investigation all. He plans to release documentary evidence early next year, when his investigation is complete.
subjecting them this of open scrutiny that would
Figure 12: Perturbation heatmaps for sample article 4.
30

---

Published as a conference paper at ICLR 2026
A the United States government's Congressional Research Service (CRS) analysing al Qaeda statements made public Tuesday. The report examines methods used, ideas presented, audience intended al- they have changed over time.
an auxiliary research office for the United States Congress, and does not make unclassified reports public. The full report is available the website of the Secrecy News project run by the Federation of American Scientists. Released November 16th, 2004 4, it titled "Al Qaeda: Statements and Evolving Idealogy".
Qaeda public statements,
report United States government Congressional Research Service (CR$) analysing al al-Qaada statements was made public Tuesday, The report examines methods used, ideas presented, and audience intended in al Qaeda public statements, and how they have changed over time. The CRS is an auxiliary research office for the United States Congress, and does not make its unclassified reports public. The full report is available the website of the Secrecy News project by the Federation of American Scientists, Released November 16th, 200 titled Qaeda: Statements and Evolving Ideology".
A report by the United States government's Congressional Research Service (CRS) analysing al Qaadaa statements was made public Tuesday. The report examines methods used, ideas presented, and audience intended in al Qaeda public statements, and how they have changed over time. The CRS is auxiliary research office for the United States Congress, does make unclassfiied reports public. The full report is available the website Secrecy News project by Federation American Scientists. Released November 16th, 2004, titled Qaeda: Statements Evolving Ideology".
Figure 13: Perturbation heatmaps for sample article 5.
31


Reviews:
[{"review_id": "weGfkrpw2j", "reviewer": "Reviewer_JqDj", "summary": "This paper presents SUIT (Subspace Knowledge Edit), a new framework for editing factual knowledge in large language models while minimizing disruption to unrelated information. Building on the Linear Representation Hypothesis, the authors propose subspace-aware editing that confines changes to semantic subspaces relevant to the target entity and relation. SUIT improves upon prior methods like ROME, MEMIT, and AlphaEdit by refining the computation of the key and residual vectors through subspace decomposition. This design enhances specificity and stability during edits. Experiments on LLaMA-3-8B, GPT-J-6B, and Qwen2.5-7B show large gains in specificity (e.g., +43.2 points on LLaMA-3-8B) while maintaining strong edit accuracy.", "questions": "(1) The proposed subspace-based approach seems conceptually similar to the null-space technique used in AlphaEdit. Could the authors elaborate on the fundamental distinction between the proposed subspace isolation and the existing null-space constraint formulation? In particular, what new insights or advantages does SUIT provide beyond current research’s framework?\n(2) Regarding the main experimental results, the “S” metric—described as the harmonic mean of three sub-metrics—appears to be driven primarily by the high specificity score. Could the authors provide a detailed breakdown of the three components and explain whether the improvements are consistent across all of them? Additionally, the baseline scores for MEMIT and AlphaEdit seem unexpectedly low compared to prior reports. Were there any differences in implementation or evaluation setup that could explain this? Finally, please clarify the meaning of the “GA” metric in the main table; should this instead refer to “GC”?\n(3) The additional analyses in the appendix are extensive, but their motivation is not entirely clear. Could the authors better articulate how each analysis (e.g., perturbation visualizations, subspace decomposition tests, or ablations) contributes to the main argument? For instance, which specific results directly support the claimed benefits of subspace isolation? More explicit linking between these analyses and the core findings would strengthen the logical flow of the paper.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper makes a solid and coherent contribution to the field of knowledge editing in large language models. It stands out in three main aspects.\n(1) It provides a clear theoretical framing grounded in the Linear Representation Hypothesis, establishing a principled link between semantic subspaces and factual updates. This theoretical foundation not only motivates the method design but also enhances interpretability and conceptual rigor.\n(2) The paper offers comprehensive and convincing analyses, including ablation studies, perturbation comparisons, and examinations of key components such as subspace isolation and feature decomposition. These analyses provide both empirical depth and transparency, showing that the improvements are consistent and well-supported.\n(3) The overall structure and presentation are highly polished. The logical flow—from motivation to methodology, experiments, and analysis—is clear and well-balanced, and the exposition makes complex ideas accessible without oversimplifying them. Collectively, these qualities make the work both theoretically meaningful and practically persuasive.", "weaknesses": "While the paper presents a clear and well-executed framework, several aspects limit its novelty and clarity.\n(1) The core technical innovation is not particularly strong. Although the subspace-based approach is conceptually sound, its methodological route appears closely aligned with the null-space perturbation ideas introduced in AlphaEdit. The distinction between the proposed subspace isolation and existing null-space constraints is not sufficiently emphasized, making the contribution seem more incremental than foundational.\n(2) There are issues in the main experiments that reduce confidence in the reported improvements. The primary evaluation table introduces the “S” metric as the harmonic mean of three scores, yet the performance gain seems largely driven by unusually high specificity, while the other two components remain modest. Moreover, the specificity scores for baseline methods such as MEMIT and AlphaEdit appear unexpectedly low compared to prior results. Additionally, the table includes a “GA” metric that lacks explanation—it is likely intended to be “GC,” but this ambiguity should be clarified.\n(3) Finally, although the appendix provides numerous additional analyses, their purpose and logical role within the overall argument are unclear. The paper does not clearly justify why these analyses are necessary or how they support the main claims. Some component analyses could also be presented more transparently to better highlight their individual contributions to the overall performance gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HfPAGhRPCy", "reviewer": "Reviewer_anrX", "summary": "This paper studies the unintended side effects of LLM knowledge editing and aims to improve specificity. Building on the locate-then-edit paradigm that treats the MLP down-projection as a linear associative memory W mapping keys to values, the authors propose Subspace Knowledge Edit (SUIT). SUIT (i) removes entity-agnostic components from the key vector by projecting onto and subtracting a data-driven “common” subspace obtained via SVD over many subject keys, and (ii) restricts the residual/value update to a low-dimensional subspace spanned by two optimized directions that respectively promote the new object and suppress the old one.", "questions": "Please see weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Subspace isolation via data-driven SVD. The paper presents a clean and principled subspace isolation method: collect key vectors for a large set of subjects, perform SVD, and define the entity-agnostic subspace as the span of top singular vectors up to an energy threshold \\tau; subtracting this projection yields an entity-specific key k. This is simple, effective, and empirically validated (variance analysis in Table 2; reduced interaction with the common subspace in Table 3)\n- The experimental results suggest that the proposed method is comparable to SOTA baselines.", "weaknesses": "- Sensitivity to the energy threshold τenergy and selection cost. The core localization step relies on a manually chosen τenergy. Fig. 5 shows a nontrivial trade-off: as τenergy increases, generation tends to decrease while specificity increases, with a “sweet spot” around 0.3–0.4. Although efficacy is fairly stable, the turning point is not sharply defined and may require cross-validation that is data-dependent and potentially costly. Please (a) provide a principled selection rule/heuristic (e.g., knee detection on cumulative energy, variance ratio targets, or stability-based criteria), (b) report sensitivity across datasets/models when fixing \\tau energy without extra tuning (you currently fix \\tau=0.4 for all models ), and (c) discuss compute overhead for building K subject and running SVD (N=10k subjects) and how often this needs to be repeated per model/layer.\n- Not uniformly state-of-the-art across all settings. While SUIT is very strong overall, it is not consistently the best on every model/dataset. For example, on ZSRE, SUIT’s harmonic mean S trails AlphaEdit slightly on GPT-J and Qwen (e.g., AlphaEdit S≈96.9 vs. SUIT S≈95.9 on GPT-J; AlphaEdit S≈89.6 vs. SUIT S≈88.2 on Qwen).\n- AlphaEdit’s results are slightly lower than their original paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dzqvP8bBZf", "reviewer": "Reviewer_U4G2", "summary": "This paper introduces a novel knowledge editing method, SUIT (Subspace Knowledge Edit), which constrains updates to task-relevant subspaces by decomposing the key vector 𝑘 into entity-specific features and restricting the residual vector 𝛿 to the feature directions most relevant to the new object. By operating only within these targeted subspaces, SUIT enables precise and localized modification of factual knowledge.", "questions": "Please see the weaknesses list above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The performance of SUIT is good and surpasses other baselines across datasets.\n\n2. The idea of making a finer-grained distinction in the editing subspace is novel and also reminds me of SAE..\n\n3. The paper's presentation is clear and well-organized.", "weaknesses": "1. It would be valuable to further evaluate the \"ripple effects\" metric of knowledge editing, which is mentioned in [1].\n\n2. As with other locate-then-edit approaches, it would be good to explore whether this method can generalize beyond the triple-based question.\n\n3. It would strengthen the work if experiments were extended to larger-scale models, such as 14B-parameter models.\n\n4. On the ZSRE dataset, the average performance (S score) on two models is still lower than that of AlphaEdit.\n\n5. SUIT generally underperforms AlphaEdit on the Generalization metric; it would be helpful to investigate the underlying reasons for this gap.\n\n6. It would also be insightful to compare SUIT with SAE-based knowledge editing approaches and examine performance differences.\n\nIf the above weaknesses are addressed well, I will reconsider my rating.\n\n---\n**References**:\n\n[1] Evaluating the Ripple Effects of Knowledge Editing in Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DGQuxcojEL", "reviewer": "Reviewer_bGWe", "summary": "This paper proposes SUIT (Subspace Knowledge Edit), a new knowledge editing method aimed at solving the problem of poor \"specificity\" (i.e., corrupting unrelated knowledge) that is common in locate-then-edit approaches. The method claims to achieve its goal via two core constraints: 1) When computing the \"key vector\" $k$, it identifies and removes an \"entity-agnostic\" subspace derived from 10,000 samples, purportedly isolating the edit to \"entity-specific\" features. 2) When computing the \"residual vector\" $\\delta$, it forcibly constrains the optimization to a fixed two-dimensional subspace, defined by two vectors $w_1$ and $w_2$ intended to promote the new knowledge and suppress the old. Experimental results show the method improves specificity on the COUNTERFACT dataset for models like LLaMA-3.", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper identifies the challenge of low specificity in knowledge editing.\n2. The method demonstrates a significant specificity boost on the COUNTERFACT dataset based on a generation-based criterion.\n3. The paper provides mechanistic evidence (e.g., token-level perturbations in residual streams) to validate the reduced perturbation.", "weaknesses": "1. The paper **assumes** the residual vector (the knowledge update) can be constrained to a two-dimensional subspace which lacks theoretical justification.\n2. The subspace identification relies on an energy threshold hyperparameter $\\tau_{energy}$ to which the model performance is **sensitive** as shown in Figure 5. The optimal $\\tau_{energy}$ value obtained on COUNTERFACT may not be applicable to other datasets which adds an extra tuning burden for the method's application.\n3. The paper selects $N=10,000$ subjects for the SVD pre-computation step which has no justification for this specific number nor any sensitivity analysis. It is unclear how robust the identified \"entity-agnostic\" subspace is; a different $N$ could potentially yield a different subspace. Furthermore, the computational cost of this SVD step scales directly with $N$, yet the authors fail to demonstrate that $N=10,000$ represents a reasonable trade-off between computational cost and the stability of the resulting subspace.\n4. The analysis in $\\S 6.2.1$ does not independently substantiate the semantic claim that $K_s^{\\perp}$ is “entity-agnostic.” The subspace is defined via an SVD criterion, and the subsequent variance comparison merely reflects that construction rather than testing semantic invariance.\n5. The second $\\lambda_{KL}$ that appears in Appendix B should be weight decay $\\lambda_{WD}$, which is likely a typo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)