ACE: Attribution-Controlled Knowledge Editing for Multi-hop Factual Recall
Authors: Jiayu Yang
Yuxuan Fan
Songning Lai
Shengen Wu
Jiaqi Tang
Chun Kang
Zhijiang Guo
Yutao Yue
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Knowledge Editing
Multi-hop Factual Recall
Mechanism Interpretability
Primary area: interpretability and explainable AI

Abstract: LLMs require efficient knowledge editing (KE) to update factual information, yet existing methods exhibit significant performance decay in multi-hop factual recall. This failure is particularly acute when edits involve intermediate implicit subjects within reasoning chains. Through causal analysis, we reveal that this limitation stems from an oversight of how chained knowledge is dynamically represented and utilized at the neuron level. We discover that during multi-hop reasoning, implicit subjects function as query neurons, which sequentially activate corresponding value neurons across transformer layers to accumulate information toward the final answer—a dynamic prior KE work has overlooked. Guided by this insight, we propose ACE (Attribution-Controlled Knowledge Editing), a framework that leverages neuron-level attribution to identify and edit these critical query-value (Q-V) pathways. Ace provides a mechanistically grounded solution for multi-hop KE, empirically outperforming state-of-the-art methods by 9.44% on GPT-J and 37.46% on Qwen3-8B. Our analysis further reveals more fine-grained activation patterns in Qwen3 and demonstrates that the semantic interpretability of value neurons is orchestrated by query-driven accumulation. These findings establish a new pathway for advancing KE capabilities based on the principled understanding of internal reasoning mechanisms.

Published as a conference paper at ICLR 2026
ACE: ATTRIBUTION-CONTROLIE KNOWLEDGE EDIT- ING FOR MULTI-HOP FACTUAL RECALL
Jiayu YANG
Yuxuan FAN HKUST(GZ))
HKUST(GZZ) Deep Interdisciplinary Intelligence Lab conngc2.ecoonect.tomm www.hhhustaaa www.gg...dd.c. edu.cn
y yfan5466 hkust--tt.uuuttgt edu
Songning LAI
Shengen WU
HKUST(GZZ)
HKUST(GZ)) Deep Interdisciplinary Intelligence Lab wu sh. com
Deep Interdisciplinary Intelligence Lab
songningl a wwkust-gz.eeoo edu..eddd..m
Jiaqi TANG HKUST
Chun KANG
BUAA
jtang0928conneede www.cconntn.t.......ooooooooe... wwwsshutt2 edu. cn
xicoforstudy@@maill com
Zhijiang GUO HKUST(GZ) HKUST
Yutao YUE HKUST(GZ))
Institute of Deep Perception Technology, JITRI Deep Interdisciplinary Intelligence Lab yutaoyue@hkust-gz edu en
zhi hii.iaggguhkkussggg...... edu. on
ABSTRACT
Large Language Models (LLMs) require efficient knowledge editing (KE) to up- date factual information, yet existing methods exhibit significant performance decay in multi-hop factual recall. This failure is particularly acute when edits involve intermediate implicit subjects within reasoning chains. Through causal analysis, reveal that this limitation stems from oversight of how chained knowledge is dynamically represented and utilized at the neuron level. We discover that during multi-hop reasoning, implicit subjects function "query neurons". which sequentially activate corresponding "value neurons" across transformer lay- ers to accumulate information toward the final answer-a dynamic prior KE work has overlooked. Guided by this insight, we propose AcE AAtribbttion-Controllee Knowledge Editing), a framework that leverages neuron-leve level attribution to identify and edit these critical query-value (Q-V) pathways. AcE provides mechanistically grounded solution for multi-hop KE, empirically outperforming sate-of-the-art methods by 9.44% on GPT-J and 37.46 % on Qwen3-8B. Our analysis further reveals more fine- -ee-ggraind activation patterns Qwen3 and demonstrates that the semantic interpretability of value neurons is orchestrated by query- driven accumu- lation. These findings establish a new pathway for advancing KE capabilities based the principled understanding of internal reasoning mechanisms.
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable capabilities in storing and retrieving vast amounts of factual knowledge (Hu et al.) 2024 Mousavi al.) 2025), underpinning their success in diverse downstream tasks such as question answering and reasoning (Mamaghan et al.
"quual contribution
Corresponddence to Zhijiang GUO and Yutao YUE (zhijangguo@hkustgg...crk (yutaoyuuuuuue hhkust hkust- yz.edu.en)

---

Published as a conference paper at ICLR 2026
2024 Ke et al. 2025). However, the knowledge encapsulated within these models is static and can become outdated or incorrect, necessitating mechanisms for efficient updates. Full retraining is computationally prohibitive, motivating the development of Knowledge Editing (KE) techniques, which aim to modify specific factual associations within LLMM cost-effectively and with minimal impact on unrelated knowledge (Yao et al. 2023
Mazzia et al. 2024).
While the thente-then-edit paradigm, exempli- fied by methods like ROME (Meng et al. 2022a)
Multi-hhpe What country is Mirk Trumbn's sport originates ficet? Value Edit: The country that Mark Trumbe's spert ceiginats fiom is USA slaay
and MEMIT (Meng et al. 2022b). has proven effective for editing facts LMs by success- fully targeting components like Feed-Forward Networks (FFNs), its efficacy significantly di- minishes when confronting multi-hop factual recall (Zhong et al., 2023 Zhang et al.] (024bb). This challenge is particularly acute when the edited knowledge involves an implicit subject intermediate entity in a reasoning chain that leads to the final answer's ss explicit subject (Zhong et al., 2023). As illustrated in Figure [T] a multi-hop query like "What country is Mark Trumbo's 's sport originates from?" requires the model to react as a chain: starting with the ini- tial subject "Mark Trumbo", it must first identify his "sport" (the implicit subject, e.g., "Basketball" in the original knowledge) and then recall the country where that sport originated (the explicit subject, e.g., e.g., "USA"). Figure I depicts editing the knowledge so that Mark Trumbo's 's sport becomes "Football", which is then linked to originating from "Ialy"" Standard single-hop editing methods, often focusing on deeper FFN layers (Meng et al.,
Multi hop Recall
Mark Trumboo
Sports
Beskethall
Chened
USA
Knowledge Editing
Mark Trundo
Edtte
Football
Created
Jialy
Figure 1: call. A multi-hop query requires traversing multi- ple facts. The diagram shows the original knowl- edge path (e.g.. Mark Trumbo + Basketball + USA) and how a knowledge edit (green arrow) can target an intermediate fact, which then requires the model to follow a potentially new chain (Mark Trumbo + Football - Italy). The intermediate entity "Football" serves as the implicit subject.
Illustration of a multi-hop factual re-
2022a by Li et al. 2024). Although recent work like IFMET (Zhang al. 2024b) has demonstrated improved multi-hop performance, particularly for implicit subjects, by editing deeper FFN layers via constructing multi-hop prompts, the underlying mechanisms explaining why these deeper edits crucial for correctly retrieving and utilizing the implicit subject information remain unexplored.
A key limitation of existing KE methods in multi-hop scenarios stems from incomplete under- standing of how knowledge- particularltl intermediate reasoning steps is dynamically represented and accessed the neuron level. Through systematic causal analysis, we discover that successful multi-hop recall relies on coordinated interactions between neurons across layers, where implicit subjects trigger cascading activations that progressively accumulate information through query-value interactions. Our experiments reveal two critical properties: (i) In multi-hop factual recall, intermedi- ate implicit subjects function as query neurons, sequentially accumulating and activating relevant value neurons for subsequent reasoning hops. (ii) LLMs store semantically analogous knowledge in structurally similar transformer components, with query and value neurons for specific knowledge types exhibiting consistent localization patterns across layers. Building upon these two initial prop- erties, we systematically analyze the mechanisms of multi-hop reasoning through experiments in Section These properties provide clear answers to two fundamental questions: "How LLMs Store the Semantics Knowledge?" and "How is the information accumulated?".
These mechanistic insights motivate ACE (Attribution-CC) Conttollldd Knowledge Editing), a framework that moves from layer-level heuristics to neuron-level interventions. As shown in Figure 4 ACE employs novel attribution techniques to identify and edit critical query-value pathways. Extensive experiments demonstrate that ACE outperforms the state-of-the-art method PMET by 9.44% on GPT- J and 37.46% on Qwen3-8B in terms of multi-hop accuracy. Ablation studies confirm the necessity of both components: skipping query layers causes a 16.51% % performance drop. while omitting value layers leads to a more severe 40.45% decrease. Our analysis reveals that existing KE methods fail in multi-hop reasoning by overlooking both deeper value layers and critical query-layer activation patterns. We further discover distinct architectural differences: while GPT-J maintains fixed layer separation, Qwen3-8B exhibits dynamic, domain-specnt speccficc alignment. Crucially, correct predictions depend on sparse interpretable neurons-ain neurons-ablating just 27 critical neurons causes an accuracy drop to 3. 3.2%, demonstrating the neural coordination required for multi-hop reasoning.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Knowledge Editing and Multi-hop Reasoning in LLMs. To avoid the high cost of retraining. recent work focuses efficient knowledge editing by directly modifying model weights Zhang ct al., 2024a] Yao et al. 2023). The locate-then-edit paradigm identifies key parameters storing target facts, with methods like ROME| et al.) (2022a) and MEMIT Meng et al. (202bb) updating FFN weights via causal tracing and closed-form optimization, while PMET Li et al. (2024) distinguishes MHSA for general patterns and FFNs for factual content. Other approaches include tuning small subsets (Mitchell et al. 2021) or using hypernetworks (Gupta et al. 2024). However, these methods face significant challenges in multi-hop reasoning scenarios, where editing intermediate facts can break reasoning chains (aao et al. 20233 Cohen et al., 2024). The MQuAKE benchmark (Zhong al. 2023) highlights the failure of existing methods to propagate such edits effectively. While IFMET (Zhang et al. 2024b) advances multi-hop performance through layer-level interventions. AcE addresses these limitations by introducing neuron-level mechanism that leverages activation dynamics for robust multi-hop edits.
Mechanistic Interpretability and Knowledge Localization. Effective knowledge editing depends on understanding how LLMS store information. FFN layers have been shown to act as key-value stores, with values encoding semantic content (Geva et al.) 22200). and recent work has localized factual knowledge at finer granularity (Dai et al.) 202 Hernandez et al.) (2023). Yu & Ananiadou (2023) revealed that predictions are driven by interactions between query and value neurons, with value neurons exhibiting consistent relational semantics. Building on this, ACE employs neuron- level approach to trace and modulate these activation pathways, enabling targeted and interpretable editing in multi-hop reasoning.
3 PRELIMINARIES
3.1 MODELING AND TASK
Definition of Neurons in LLMs. Autoregressive decoder-only LLMs Fe process input into tokens X = [41....,77) (T]. predicting next-token distributions over V. Tokens are embedded via E to h.. then processed through I. transformer layers (MHSA FFN). Layer hidden states are:
M + A F!!
where 1!-1, E. mean the previous layer's output, the attention output, and the FFN output. Finally, the last position's Lth layer output is used compute the probability distribution y with unembedding matrix E, E Bxd,
y = softmax(Exul
(2)
The attention layer outputs weighted sum across H heads, while the FFN applies nonlinear activation to two linear transformations.
H
A (hl .// ..... ...,,h/"" 1111),
Σ
(3)
Fl = W fe20 (W) Jei (b A!)),
(4)
where W fei E Nxd and W pdx are two matrices. Geva et al. (200)) finds that FFN output can be expressed as a weighted sum of FFN neurons:
N
Fl
mi,y SC21_
(5)
k=1
m (scl) (M + A!))-
(6)
Following the same notation in (Yu & Ananiadou ) fc22, is the subvalue of FFN which is the column kth of W 22 and its coefficient score mi,x is calculated by residual output 64-1 Af and FFN subkey fclf,. the th row of W/ Similarly, the attention output A can be represented as:
= ΣΣ
Σ
a W (WF,
p=
3

---

Published as a conference paper at ICLR 2026
-33 500m=soffmmx(111 SWf, 1/11
where W W 3. 3 the query, key, value and output matrices. As discussed in Eq. the kth FFN neuron the kth subvalue fc22 and its corresponding subkey fcl,, including all the query and value neurons mentioned later. Similar to FFN neurons, we regard the kth column of 3 as the kth attention subvalue, whose subkey is the kth row of W.,
Define knowledge set K= [(s,T,0)] E R &, where & (entities) and R (relations) form triplets (s, r, 0) indicating subject a relates to object o via r. An edit instance rs o²) represents replacing with o².
Factual Recall Tasks.
We evaluate model M through factual recall tasks, which assess its ability to answer both single-hop and multi-hop factual questions Q requiring 2 reasoning steps. A reasoning chain is formally defined as = ($1, r1, 01) (on))> : ((2n, Tn. Iu, On), starting from an explicit subject $1 and concluding with the target answer Oz- To illustrate, consider the two-hop question: "What country is Mark Trumbo's sport originates from?" This corresponds to the knowledge chain (Mark Trumbo, Sport, Basketball) + (Basketball, Created, USA). We employ two question formats in our evaluation: Cloze Format (Qatoze): "The country that Mark Trumbo's sport originates from is and QA Format (Qna):
"What country is Mark Trumbo's 's sport originates from?".
The multi-hop recall process consists of two distinct phases: the explicit recall step ($1, T1, 01) to retrieve the initial fact, followed by implicit recall steps {(s,, T', o;) to traverse subsequent hops. A factual recall is considered successful if the model generates the correct final answer, i... M(Qrioze))
on or M(Qqa) = On. Figure [ shows the overall process of the task. Factual Recall Tasks evaluate if the post-edited model can utilize updated knowledge for multi-hop reason- ing. Given an edit e = (s, o + o²). edit prompt Te, and fact chain C. containing (s, r, o), the model must answer multi-hop queries using the updated fact (s, o"). For example: After editing (Mark Trumbo, Sport, Basketball + Football), the multi query "The country that Mark Trumbo's sport originates from is" should shift from USA to Italy.
3.2 ATTRIBUTION METRICS
Distribution Change. The final hidden state h4 aggregates critical information for token prediction through summation of neveln-level vectors. implying that essential predictive signals are encoded in specific neurons. By decomposing h4 as h4 = v + x (where v denotes a target neuron's contribution and x = h4 - v represents residual components), we quantify v's causal influence via probability distribution shift Ap(w) p( ([w|æ + v) w/(z) for token w. This framework enables systematic identification of neuronal components that maximally amplify Ap(w), establishing a methodology for pivotal neurons in static way, measuring the importance level of neurons.
Importance Score for Value Neurons and Layers. To jointly account for both the variable neuron e and the conditioning variable æ in the probabilistic framework. Based on Yu & Ananiadou (2023), we find log probability increase could efficiently evaluate the model's distribution change by a neuron. Define the log probability increase as importance score I for vectors, which satisfies I(x v) I(x) + I(v). If of is a vector in Ith attention layer, the importance score of of is:
I(e') log [p
+ ))² - logg (p(w) 16' ()-1)))((())
ΣI('') eEI
(9)
where the probability is computed from vocabulary in Eq 2 layer I denotes the index set of varying neuron When vector of in lth FFN layer, I is computed by replacing h1-1 as 11-1 + A'.
Importance Score for Query Neurons and Layers. We find that query neurons exist in the transformer while solving multi-hop tasks aims to activate value neurons, even if they do not directly contain information about the target token We use the inner product between its subkey and itself to measure the importance of the query neuron, showing the ability activating value neurons:
Iquery = v. fcl', Iquery
-I('', vel
(10)
where layer denotes the index set of varying neuron v, since the fc2 vectors do not change, the coefficient scores will be the only varying element. Therefore, if a query neuron exhibits a larger inner product with the subkey, it activates the value neurons more.
4

---

Published as a conference paper at ICLR 2026
4 MECHANISM OF MULTI-HOP REASONING
How do language models store and retrieve knowledge when performing complex multi-hop reason- ing? We begin by examining how semantically related knowledge is organized within transformer components (Section[4.] revealing consistent patterns that challenge conventional wisdom about knowledge storage. Building on these structural insights, we then trace how information propagates through reasoning chains (Section (.2). uncovering sophisticated query-value activation mechanism that progressively accumulates evidence toward final answers. Our analysis spans both GPT-J (Wang & Komatsuzaki, 2021) and the more advanced Qwen3-8B (Team] 2025), with the latter exhibiting even more fine- aree-graned activation patterns that offer new insights into reasoning dynamics.
4.1 How LLMS STORE THE SEMANTICS KNOWLEDGE?
A fundamental question underpinning effective knowledge editing is: How do LLMs internally store knowledge? Addressing this question is crucial for clarifying the models' reasoning logic in multi-hop scenarios and enhancing their internal interpretability. By systematically investigating knowledge representation mechanisms, we can uncover how information propagates through transformer archi- tectures, thereby identifying the root causes of existing KE methods' failures in multi-hop reasoning. This understanding provides the foundation for developing more effective KE techniques.
To this end, use dataset MQuAKE-3K Zhhng et al. 2023) to explore the mechanism of semantics knowledge storage. MQuAKE a challenging knowledge editing benchmark which comprises over 3000 multi-hop edit instances. Considering the challenging factual recall queries, extract a subset from original dataset by systematically evaluating the vanilla GPT-J model then scale to Qwen3-8B, using all single-hop factual recall queries characterize its inherent knowledge retention capabilities. This controlled experimentation protocol isolates the model's 's capacity to store the knowledge. Both attention and FFN layers exhibit inherent capabilities for knowledge storage. We GPT-4o (Hurst et al., 2024) to classify the knowledge types in the dataset, and conducting analysis across eight semantic categories: Nationality (NN), Continent (CT), Language (LG), Capital (CP), Leadership (LS), Author (AT), Sports Team (ST), and Company Founder (CF).
Table 1: Top 9 important attention layers (left block) and FFN layers (right block) in GPT-J.
Top 9 important attention layers and FFN layers
NN
027
026
07
010
0g
022
0g
011
05
f2o
f21
fie
fi8
f15
f22
f23
f26
f25
CT
027
026
07
010
0g
4.5
0g
011
025
f22
f24
f16
f21
f15
f17
126
f25
123
LG
027
07
05
Q6
0g
0.4
01
0:6
0g
f2r
f7
fs
f6
fs
fa
fi
f26
f9
CP 027
0:6
07
010
0g
ag
012
a5
Q6
fiz
f26
f7
810
f12
fs
fo
fs
f6
LS
027
(22
(25
077
010
d6
0g
06
05
f27
826
525
124
J7
$10
fs
AT
027
026
07
0gg
010
45
J7
f0
88
f10
82T
126
J25
Js
S6
ST
Q26
027
07
Q10
0g 46
05
025
044
f26
f2r
f7
f10
fs
f6
fz
f25
fA
CF Q26 027
Q25
07
0g
46
f26
F2r
F2
f25
fr
fs
fo
fc
fz
To find critical neurons and layers, we performed forward passes on all single-hop questions in the dataset and computed the sum of the importance scores at last position in the residual stream. We select top 9 most important layers analysis the knowledge attribution Table T Our analysis reveals that information with similar semantics tends to be stored in proximate neural modules, while semantically unrelated information is distributed across disparate modules. Specifically, MHSA components are activated in similar positions, for instance, 027, a26, (7 ranks top in all knowledge. This suggests that MHSA stores general knowledge and capabilities in LLMSs. Differently, FFN layers tends to primarily extracts its own knowledge. For instance, f21. f27, ranks top for similar semantics (NN, CT, LG, CP, LS), dissimilar semantics (AT, ST, CF) resides in distinct layers.
We set the top critical semantic-related neurons to zero in GPT-J and Qwen3-8B. Figure 2 shows that only 1% intervention on important neurons causes over 90% accuracy decrease in semantic- related subsets. Compared to 1% random intervention, only 9.47% and 8.19% accuracy decrease in GPT-J and Qwen3-8B. Based on our observations, we formalize Takeaway 1: LLMs tend to store semantically analogous knowledge in structurally similar components.
5

---

Published as a conference paper at ICLR 2026
GPTU
Datrent
Qwen3-48
67%
The Impact of Causal Intervention with semantic-related requests upon LLMs of most important layer, including Nationality, Continent, Capital and Language requests.
Figure 2:
4.2 How THE INFORMATION ACCUMULATED?
Building on Takeaway 1, which identifies the locations and distribution patterns of knowledge storage in LLMs, we now investigate the dynamic process of information accumulation during multi-hop reasoning. Specifically, we aim to address the question: How is the information pertaining to the final answer accumulated through implicit subjects? To this end, we conduct causal interventions and statistical analyses on the critical layers identified previously, examining the interactions between query and value neurons across the reasoning chain.
Building the observation that implicit subject information accumulates through shallow FFN layers (Zhang et al.] (024b), we investigate the mechanistic details of this accumulation pro- cess. While Hou al. (2023) suggests that models process multi-hop reasoning by segment- ing it into individual single-hop recalls, the specific neural mechanisms underlying information integration remain unclear. This gap leads us to examine two fundamental questions: How is information pertaining to the final answer progressively accumulated through implicit sub- jects? Through which transformer components is this cumulative effect primarily mediated? Our investigation begins with an analysis of value neuron distributions GPT-J's FFN lay-
Log Increase and Neuron Count by Layer
ers. Contrary to the prevailing assumption that deeper layers predominantly determine model outputs, we observe more complex pattern. Value neurons exhibit peak density in middle- to-deeper layers, with distribution maxima not aligning with the final residual stream positions. Most strikingly, we find an abrupt depletion of neurons in the deepest layers- a finding that challenges conventional understanding. Fur- ther analysis reveals a systematic relationship between query and value neuron activation pat- terns. As shown in Figure 3 the significance variation of query FFN layers closely tracks the progressive accumulation of implicit subject information during reasoning. Layer-wise probing demonstrates that query neuron activation (blue curve) consistently precedes value neuron activation (red histogram) by 1-2 layers, indicating a structured information flow mechanism.
Log Increase
Neursn Count
Log Increase
Layer
Figure 3: Query layers' log increase and value neu- rons count by layers in GPT-J. Layer log increase the importance score calculated by logarithmic difference in Eq. g
To validate the functional importance of these patterns, we conducted targeted interventions on query neurons. When we ablated the top 100 query neurons from two layers showing peak activation for 2-hop requests (fqre and fy18), model capability decreased by 46.2% and 61. 19, respectively. Moreover, the number of activated value neurons in subsequent layers (f17. 19. dropped dramatically from (28,16.33) to (6,4.7), while other value neurons showed minimal change (12 neurons decreased). We show more details on Qwen3 while explore the forward processes Appendix G
These experimental results lead us to two key observations: (1) Final answer information is progres- sively encoded through carly-stage stage query-value activation pairs throughout the reasoning chain; (2) Implicit subjects functionally operate as query neurons that orchestrate the activation of value neurons for subsequent reasoning steps. Based on these consistent experimental findings, formulate

---

Published as a conference paper at ICLR 2026
Takeaway 2: Information of the final answer is accumulated through implicit query neurons that sequentially activate correspunding value neurons across the reasoning chain.
5 AcE: ATTRIBUIION
KNOWLEDGE EDITING
Based our findings and validations concerning the mechanisms of LLMs' knowledge storage and reasoning in multi-hop factual recall tasks, we introduce Atribution-Controllee Knowledge Editing (ACE) for Multi-hop Factual Recall. As Figure 4 demonstrated, ACE extends the established locate- then-edit paradigm through three sequential operations: first identifying latent query neurons to verify critical query FFN layers that activate explicit subjects' value neurons; second, applying model editing via multi-hop prompts to modify explicit subject knowledge by targeting FFN value components in deeper layers; and third, executing complementary edits targeting FFN query mechanisms in middle- to-shallow layers to adjust the implicit reasoning path originating from the updated explicit fact. ACE strategically emphasizes the significance of FFN query mechanisms for successful multi-hop reasoning while also properly considering the FFN values within the residual stream computation.
Stage 1: Identifying. Using the definition of neurons and importance in Section 3 we employ importance score I, Iquery Eq. and [0 to identify critical q/v neurons and their corresponding layers. In the identifying process, we performed forward passes on all multi-hop questions in the dataset and computed the sum of the importance scores at the last token position in the residual stream. After the identifying, we rank the query and value layers and select the top layers to edit.
Existing Lecute-then-edii Failed in Multi-hop
(c) Ours Query Edit:
Single Edit: The capital of Spain is Madrid-Harfoort
Identify critical query layer in Residual Stream
Residual Stream: <token]>, Stokenn2>
<tokenN>
Hartford
Overlooking the values
Residual
Query
Values
Query FFNs
Value FFNS
Q Activating V
Language Model
Final Answer Information Accumulating
Shallow Layer
Deep Layer
(b) Multi-hop: What's the capital of the country where Pablo Picasso hold a cilizen ship? Residual Stream: stoken]>, <token2>,..... ... <tokenN>, <toenN+11> 1>, ... <tokenN-M>>
Value Edit: The country which Barcelona located has capital city Madrid-HHartforr
Figure 4: ACE edits Q-V neurons via attribution: (a) The existing sheute-then-edit KE method updates
new fact using single-hop prompt; (b) For multi-hop factual recall tasks, traditional locate-then-cee failed to correct edit the knowledge on query layers, overlooking value neurons; (c) Our ACE identifies critical query layers which activates the value neurons most to edit the knowledge.
Stage 2: Locate-then-edit. Building upon identified critical layers, we conducted sequential knowledge editing to demonstrate the model's enhanced knowledge acquisition through intensified activation patterns. Based on previous locate-then-edit paradigm (Li et al.) (024), we apply editing on FFNs and keep attention heads unchanged, while the general semantic information saved in attention heads should not be changed. For /th layer FFN, it's output of the ith token would be W! (WF ho -1)) and there's no attention input GPT-J, where a is the non-linear activation function. o oW/ /eg take the responsibility as keys, denoted as k; Whole FFN output is the value = 22 (c2hhh whose subvalue matrix w/ee denotes the weight of the model's values needs to be modified. So aim to modify subvalue matrix W fc2 s.t. W fogh v² where v represents the new values (factual knowledge). We use backbone PMET (Li et al. 2024 to complete editing process, details shown in Algorithm D
The ACE framework accomplishes knowledge editing through two sequential stages. In Stage 1, we identify critical query layers and value layers within the model architecture. The critical query and value layers represent the precise locations where target knowledge is stored. In Stage 2, edit these components enables efficient integration of new factual information into the model's ' parameters. In all, ACE incorporates complementary edits to the often-overlooked query layers, ensuring that the
7

---

Published as a conference paper at ICLR 2026
Table 2: Multi-hop accuracy comparison of different KE methods on the MQuAKE-3K dataset in a few-shot setting, Base shows the model's performance the unedited answers and edited model's performance on edited answers. Our model outperformances than other models significantly.
GPT-J/ # Edits
Qwen3-8B / # Edits
Editor (vg.(GGP/wwen
1-edit
2-edit
3-edit
4-edit
1-edit
2-edit
3-edit
4-edit
Base
98.42 99.17
99.7
95.48
97.51
97.23
99.81
97.46
98.14
97.64
FT
3.54/ / 2.18
4.17
2.63
0.00
0.00
3.14
2.79
0.00
0.00
ROME
35.04 28.79
44.51
38.93
17.52
5.06
35.09
32.48
18.97
7.08
MEMIT
38.58 / 18.67
64.30
16.87
17.25
8.16
29.84
19.18
12.91
4.20
PMET
37.01 / 20.78
49.26
36.30
24.34
17.01
28.64
14.08
12.56
11.20
ACE
46.45 / 58.24
45.26
50.24
36.17
43.29
60.22
59.48
51.62
47.61
updated knowledge can be properly activated and traversed during multi- reasoning processes, making information accumulates progressively via query-value interactions.
EXPERIMENTS
6.1
EXPERIMENTAL SETUP
Dataset. MQuAKE-3K (Zhong et al. (2023) is benchmark dataset for evaluating large language models' multi-hop fact recall capability after knowledge editing. Each instance contains multi-hop fact recall chain (interconnected triples) and corresponding textual questions, requiring models to maintain coherent reasoning after applying single-hop knowledge edits via edit prompts. This design simulates cascading effects of real-world knowledge updates, providing a systematic framework to assess models' dynamic knowledge management. To faithfully evaluate the model's capabilities and assess its capacity to assimilate edited knowledge, we applied filtering to the original dataset, resulting in curated subset where the base model consistently achieves accurate reasoning.
Baselines. Basclines Base, refers to the original GPT-J (6B) and Qwen3-8B without any edit; FT refers to fine-tuning method, ROME Meng et al. (022a) refers to the vanilla locote-then-edit method; MEMIT Meng et al. (2022b), extends ROME with updating weights by set of facts, and PMET Li et al. 2024) claims optimizations on FFN layers based on MEMIT.
Metric and Setup. We apply our ACE on the model GPT-J (6B) and Qwen3-8B. We use Multi-hop requests answering accuracy as the metric to evaluate the performance of our edited model. We use PMET as our model's 's primary backbone for editing. More settings details see Appendix H
6.2
MAIN RESULTS
demonstrates the general performance of various established methods and ACE under the different classes of data settings our subset of MQuAKE-3K. To enable precise identification of critical layers store the knowledge and activation patterns during model editing, we systematically integrated in in-context priming and Chain-of- Shain-of-Thoughe Thought reasoning across all editing prompts. This dual- strategy architecture ensures optimal knowledge editing efficacy through targeted activation within the model's parameter space. # Edits refers to the number of how many individual facts in the reasoning chain are edits, its maximum equals to the number of hops in the dataset. As evidenced in Table 2 the proposed ACE framework demonstrates consistent superiority over existing methodologies across various evaluation metrics. Our method outperformances 9.44% and 37 .46% on GPT-J and Qwen3-8B. Traditional Locate- then Edit paradigm performances even worse on Qwen3-8B due to the fixed editing positions, and ACE shows much flexibility on reasoning models due to the unfixed q-v positions as discussed in Section 6.4
Table 2
We also show detailed metrics of our experiments in Table Efficacy metric, measures whether the model can successfully answer the single-hop fact recall prompt, Paraphrase metric, measures the model can answer the same original question in different statements. Specificity metric, measures the whether the edit of a specific fact affects other facts stored within the model. AcE demonstrates superior performance over SOTA across multiple metrics, highlighting its multiple capabilities.
8

---

Published as a conference paper at ICLR 2026
Table The detailed results of more metrics in experiments on GPT-J and Qwen3-8B.
Editor
Efficacy
Paraphrase
Specificity
FT (GPT-I/Qwen3)
98.4 97.1
74.5 73.2
83.8 79.6
ROME (GPT-J/Qwen3)
64.2/51.8
61.6/49.3
66.8 57.2
MEMIT (GPT-J/Qwen3)
62.8 53.6
66.2/6188
70.0 64.7
PMET (GPT-J/Qwen3)
81.6
65.8 68.9
74.6 64.4
ACE (GPT-J/Qwen3)
99.8 99.44
91.2 /9.2
79.2 81.8
Table 4: The results of ablation experiments on GPT-J-6B and Qwen3-8B model. The column Editor shows which layer(s) are skipped in the editing process, the index # of the layer refers to the importance rank. The percentage of decrease(.) is calculated relative ACE as the baseline.
Editor
Avg.
Efficacy
Paraphrase
Specificity
#1 (GPT-J-6B)
43.26(4 6.87%)
96.2
90.4 4
77.6 4
1.2
(GPT-J-6B)
41.19(4 11.32%)
94.8
91.0 +
75.2 4
1,2,3
(GPT-J-6B)
38.78(4 16.51%)
90.6
91.6 t
74.3 4
##1
(GPT-J-6B)
42.14(1 9.28%)
91.5
88.7
74.9 L
J6
(GPT---6B)
33.97(4 26.87%)
84.6
81.8 |
70.3 4
J (Qwen3-8B)
52.61(4 9.67%)
96.64
91.3
74.3 4
1.2
(Qwen3-8B)
47.34(4 18.71%)
92.44
89.3
72.7
2,3 J
(Qwen3-8B)
45.26(4 22.29%)
91.34
85.8
71.5 4
1#1
(Qwen3-8B)
51.39(4 11.76%)
94.84
90.2
73.6
## 1,2 (Qwen3-8B)
34.68(4 40.45%)
71.9
73.1
72.3 4
6.3 ABLATION STUDY
Edited Layer. We performed an ablation study on the edited query and value FFNs ACE by sequentially skipping edits to the identified critical layers within the model, aiming to elucidate the impact of these critical layers within the ACE editor. As shown in Table for both GPT-J-6B and Qwen3-8B, skipping the layers targeted for editing significantly impaired the editor's 's performance. After skipping the three most important query layers, model performance decreased by 16.51%, while skipping the two most important value layers led to a performance drop of 40.45%.
The performance degradation resulting from skipping query layers was slightly less pronounced than that from skipping value layers, which validates our takeaways: query layers transmit information by activating value layers. Skipping query layers results in incomplete activation of the corresponding value layers, whereas skipping the editing of value layers leads to less knowledge being incorporated.
Few-Shots Prompts. Selecting appropriate Few-Shot prompts for in-context learning within the secate-then-ede paradigm enables the model to learn from challenging multi-hop reasoning editing examples, thereby improving its ability answer questions accurately. We conducted ablative experiments on the prompts employed in our study. We conducted Zero-Shot, One-Shot and OOD) Few-Shots (no overlap between evaluation set and prompts) prompts when evaluating.
As shown Table even with less information, the performance degradation is only 9.4% (Zero-Shot, GPT-J), 5.7% (One-Shot, GPT-J), 0.4% (OOD Few-Shots, GPT-J), 9.5% (Zero-Shot, Qwen3), 3.1 1% (One-Shot, Qwen3), and 0.6% (OOD Few-Shots, Qwen3) compared to the original. This demonstrates that ACE maintains strong robustness under reduced in-context learning information, providing evi- dence that the effectiveness of knowledge editing is inherent rather than reliant on in-context learning.
6.4 ANALYSIS
Table 5:
The results of ablation experiments on different CoT Prompts upon GPT-J and Qwen3.
Now we can explain why the existing KE meth- ods failed in multi-hop reasoning. Existing KE methods overlooked the value layers deeper locations, even regarded the output generation layers as the value layers, where the knowledge truly be stored in latter ones (details in Appendix G). And the worst is, these KE methods ignored the importance of editing the query layers.
Editor
Avg.
GPT-J (Zero-Shot)
42.08
GPT-J (One-Shot)
43.79
GPT-J (OOD Few-Shots)
46.27
Qwen3 (Zero-Shot)
53.19
Qwen3 (One-Shot)
56.46
Qwen3 (OOD Few-Shots)
57.92

---

Published as a conference paper at ICLR 2026
Fine-Grained Activation Pattern. In Table
Qwen3-8B exhibits greater sensitivity to the number of layers being edited compared to GPT- J. Qwen3-8B demonstrates more fine-grained activation patterns during the forward pass, which imposes stricter requirements the coherence of information within the reasoning chain.In GPT-J, the active query layers are predominantly located in the middle layers, while the deeper FFN value layers are activated by queries. There exists consistent layer-wise separation between these two families of layers, and their positions (416.917.914 g17.818. v228,v29,.30 remain invariant across different domains. In contrast, Qwen3-8B, the query layers are situated in middle-to-dedt overlapping- the value layers they activate (e.g., f427.928.9299 (130.311.332). Moreover, the absolute positions of these query and value layers are not statically fixed, they shift dynamically depending on the knowledge domain.
layers and are closely aligned-and at times partially
Case Study. As case study, examined the residual stream for the single-hop request: "Tim Duncan plays the sport of". The model exhibited the largest increase in importance at tokens where semantics converge, such "plays" and "of"', with reaching 0.9932 and 0.8469, respectively. The top predicted tokens were highly interpretable (e.g.. "basketball", "ball", "NBA"). Meanwhile, at other transitional tokens, the model demonstrated stronger exploratory capabilities by predicting incoherent tokens. We select 27 neurons whose associated vocabulary included the correct target token and performed 1000 sampling trials with these neurons removed. The model's accuracy under this condition dropped to merely 3.2%, indicating the crucial role these interpretable neurons play in generating correct responses. In contrast, when we ablated 27 neurons with high importance scores but lacking correct semantic interpretability, the model maintained 59.4% accuracy. This suggests the final generation of correct output tokens depends on a sparse set of highly specialized, interpretable neurons. We also observe some neurons serve as q-V shared neurons at the same time, which are highly interpretable, the details of this residual stream see in Appendix
We consider that this alternating pattern of semantic divergence and convergence during in-text token prediction constitutes a more fine-grained activation behavior, which is modulated by the knowledge domain. Furthermore, these semantically convergent tokens, and neurons which serve as shared neurons also are aligned in recent RL research concerning token entropy (Wang et al. 2025 represents a promising focus for future studies of critical tokens in RL.
7 CONCLUSION
In this study, we systematically investigate the q-v activation mechanism underlying multi-hop factual recall in LMs. Through extensive experiments on both GPT-J and Qwen3-8B, our analysis reveals sophisticated neural coordination pattern: query neurons- neurons-whether representing implicit subjects or components of the final answer- orchestrate the sequential activation of semantically interpretable value neurons throughout the reasoning chain. This mechanistic understanding resolves long-standing questions about how information propagates in multi-hop scenarios. Our work contributes to the broader understanding of how LLMs organize and process knowledge. These insights open new avenues for developing more interpretable LMs.
ACKNOWLEDGEMENT
This work was supported by GUangZ) Joint Funding Program(Grant No.2023A03J00088) Education Bureau of Guangzhou Municipality. This work was sup- ported by Jiangsu Industrial Technology Research Institute (JITRI) and Wuxi National High-Tech District (WND).
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We place strong emphasis on the transparency and reproducibility of our work. We have uploaded the codes we used for reproducibility in the Supplementary Material. To facilitate independent verification, the complete implementation has been provided in the supplementary materials, allowing readers to directly reproduce the reported experiments. In addition, Section and Appendix of the main text outlines the experimental pipeline. including dataset preparation, model configurationss prompts we used and training procedures. For further clarity, Appendix [H] documents the full set of hyperparameter choices and auxiliary details. Together, these resources ensure that our results can be reliably replicated and extended in future research.
and
ETHICS STATEMENT
This work complies with the ICLR Code of Ethics. All authors of this work have committed to its adherence. The datasets used in this study are publicly available benchmarks. Our research does not involve any private or sensitive personal data. The code developed for experiments will be made publicly available to ensure reproducibility. We have followed standard practices in the field to ensure the fairness and reproducibility of our experiments. Efforts have been made to mitigate potential biases in the evaluation process.
REFERENCES
Roi Cohen, Eden Biran, Oni Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics, 12:283-298 2024.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint 10X:v:2004006666 2021.
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. arXiv preprint arXiv:2410.02355. 2024.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913, 2020.
Akshat Gupta, Dev Sajnani, and Gopala Anumanchipalli. A unified framework for model editing. arXiv preprint arXiv:2403. 24f.::2203..2236,, 2024.
Evan Hernandez, Belinda Z Li, and Jacob Andreas. Inspecting and editing knowledge representations in language models. arXiv preprint arXiv:2304.00740, 2023.
Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, and Mrinmaya Sachan. Towards mechanistic interpretation of multi-step reasoning capabilities of language models. arXiv preprint arXiv:2310. 14491, 2023.
Xuming Hu, Junzhe Chen, Xiaochuan Li, Yufci Guo, Lijie Wen, Philip S Yu, and Zhijiang Guo. Towards understanding factual knowledge of large language models. In The Twelfth International Conference on Learning Representations, 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Zixuan Ke, Fangkai Jiao, Yifei Ming, Xuan-Phi Nguyen, Austin Xu, Do Xuan Long, Minzhi Li, Chengwei Qin, Peifeng Wang, Silvio Savarese, et al. A survey of frontiers in 1lm reasoning: Inference scaling, learning to reason, and agentic systems. arXiv preprint arXiv:2504.0900 2025.
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang. Jun Ma, and Jie Yu. Pmet: Precise model editing in a transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18572, 2024.
11

---

Published as a conference paper at ICLR 2026
Amir Mohammad Karimi Mamaghan, Samuele Papa. Karl Henrik Johansson, Stefan Bauer, and Andrea Dittadi. Exploring the effectiveness of object-centric representations visual question answering: Comparative insights with foundation models. arXiv preprint arXiv:2407.1588, 2024.
Vittorio Mazzia, Alessandro Pedrani, Andrea Caciolai, Kay Rottmann, and Davide Bernardi. A survey on knowledge editing of neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov.
Locating and editing factual associations in gpt. Advances in neural information processing systems, 05:17359-17377 2022a.
Kevin Meng, Arna Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. arXiv preprint arXiv:2210007222, 2022b.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing scale. arXiv preprint arXiv:2110.11309 2021.
Seyed Mahed Mousavi, Simone Alghisi, and Giuseppe Riccardi. Lims as repositories of factual knowledge: Limitations and solutions. arXiv preprint arXiv:2501.12774, 2025.
Qwen Team. Qwen3 technical report, 2025. URL htt.:tts//aaau...or///////////// 89388
Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A billion parameter autoregressive language model, 2021.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang. Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song. Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for IIm reasoning, 2025. URL
2506.00993
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. arXiv preprint arXiv:2305.13172. 2023.
Zeping Yu and Sophia Ananiadou. Neuron-level knowledge attribution in large language models. arXiv preprint arXiv:2312.11141, 2023.
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, al. A comprehensive study knowledge editing for large language models. arXiv preprint wrXi:22401002280 2024a.
Zhuoran Zhang, Yongxiang Li, Zijian Kan, Keyuan Cheng, Lijie Hu, and Di Wang. Locate-ther then-edit for multi-hop factual recall under knowledge editing. arXiv preprint arXiv:2410.0633) 2024b.
Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing language models via multi-hop questions. arXiv preprint 147755, 2023.
12

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
Use Large Language Models
14
Related Work Details
14
Subset of MQuAKE Dataset
15
Subset Construction
15
Dataset Labeling
15
D ACE Optimization
15
Knowledge Bottleneck
16
Details of Internal Fine-Grained Reasoning Q-V Pairs Flow in Qwen3-8B
16
More Details of the Analysis
17
Experimental Settings
18
Prompts
20
L1
Few-shot and Chain-of- Thain-of-Thought Evaluation Prompts
20
1.2 Prompts recall single-hop fact
20
1.3
Dataset Annotation Prompt
20
J
ACE Latency
21
K ACE Extension
21
Human Understanding Alignment
M More Experiments
22
M.
Locality Performance
22
M.2
Attribution Robustness
22
M.3
Verilying Intermediate Reasoning Process Changes
22
M.4
Counter-Factual Edit
23
N
AcE
Attribution Derivation
25
N.I
Algebraic Structure of the Transformer
25
N.2
Theorem of Logit Linearity
25
N.3
Derivation Importance Score
26
N.4
Importance Additivity
26
N.5
Alignment with PMET
27
13

---

Published as a conference paper at ICLR 2026
A USE OF LARGE LANGUAGE MODELS
During manuscript preparation, a large language model (LLM) was occasionally employed as an auxiliary assistant refine language expression, such improving sentence fluency and enhancing readability. The model was not involved in generating original research contributions: it did not participate in formulating research questions, designing methodologies, conducting experiments, analyzing results, or drafting substantive scientific content. All core intellectual work, including the development of ideas, execution of experiments, and interpretation of findings, was carried out independently by the authors. Any linguistic suggestions offered by the LLM were critically reviewed and selectively incorporated, ensuring that accuracy, originality, and scholarly integrity were fully maintained. The authors alone bear responsibility for the research content and conclusions, and the LLM is not listed as a contributor or author.
B RELATED WORK IN DETAILS
Knowledge Editing in LLMs. The inherent difficulty and computational expense of retraining LLMs to incorporate new or corrected information (Zhang et al. 2024a Yao et al. 2023) have spurred significant research in KE The dominant paradigm, locate-then-eee -ditt aims to identify specific model parameters responsible for storing a target fact and modify them precisely. Seminal works like HOME|Meng et al. (2022a) employed causal tracing to locate critical states in FFN layers and updated their weights via rank-one modifications. MEMIT Meng et al.] (202b)) extended this by efficiently editing thousands of facts simultaneously using a closed-form optimization objective. Building these, PMET Li et al. (2024) proposed that Multi-Head Self-Attentie (MHSA) layers encode general knowledge extraction patterns, while FFNs store specific factual details, thus focusing updates on FFNs. Other approaches include fine-tuning small set of parameters (Mitchell et al. 2021) or training external hypernetworks to predict parameter updates (Gupta et al. 2024). While these methods have shown strong performance in single-hop factual editing, their effectiveness often declines in multi-hop scenarios, as they overlook how knowledge is chained and how intermediate reasoning steps are utilized. Our work, ACE, directly addresses this limitation by focusing on neuron-level mechanisms underlying multi-hop knowledge retrieval.
Mechanistic Interpretability and Knowledge Localization. Effective knowledge editing is intrin- sically linked to understanding where and how LLMs store knowledge. A growing body of work in mechanistic interpretability aims to unravel these internal workings. Geva et al. (2020) and Geva et al. (220)) identified FFN layers as key-value memories, where keys correspond to input patterns and values represent output distributions. More pertinent to our approach, Dai al. (202T) and Hernandez et al. (2023) explored techniques to locate factual knowledge finer granularity.
The work by Yu Ananiadou (203)) provides a crucial foundation for our methodology. They demonstrated that both attention and FFN layers operate via query neurons that activate specific value neurons to produce final predictions. These value neurons were found to be semantically inter- pretable when projected into the vocabulary space, and similar types of relations (e.g.. "capital of X," "birthplace of Y") were shown to activate neurons in structurally consistent locations across different subjects. While IFMET focused on attribution and interpretation, ACE leverages these neuron-level insights to develop more targeted and mechanistically grounded KE strategy, particularly for the complex dynamics of multi-hop reasoning where intermediate implicit subjects act as activators.
Multi-hop Reasoning and Knowledge Editing. Multi-hop reasoning, requires LLMs to chain multiple pieces of information to arrive at an answer, poses a significant challenge for KE (Yao et al., 2023, Cohen et al. 2024). Editing single fact involved in a multi-hop chain can inadvertently disrupt model's s ability to perform the entire reasoning chain. The MQUAKE KE benchmark /Zhong et al.) (2023) highlighted the poor performance of existing KE methods multi-hop requests, revealing that edits often fail to propagate effectively when the edited fact is an intermediate step.
ACE advances IFMET (Zhang et al. 2024b) by proposing neuron-level editing mechanism: While IFMET observed that multi-hop editing via deeper FFN layers enhances performance, it lacked mechanistic insights into why deeper layers matter. Through query-value activation dynamics analysis, we reveal that deeper layers host value neurons processing multi-hop reasoning triggered by implicit subjects (acting as query-like activators) resolved in earlier layers. Unlike IFMET's
14

---

// Find critical query layers // edit query layers Published as a conference paper at ICLR 2026
layer-level heuristic approach, ACE establishes interpretable neuron-editing framework guided by attribution analysis - it achieves robust multi-hop knowledge updates by precisely identifying and modulating vvluu activation pathways. This shift from layer-centric to neuron-centric mechanism interpretation constitutes our core innovation.
C SUBSET OF MQUAKE DATASET
C.I SUBSET CONSTRUCTION
This investigation into the cognitive mechanisms underlying single-hop and multi-hop fact retrieval employed a controlled experimental paradigm using cloze-style query templates. Our methodology involved curating knowledge triples from MQuAKE that were demonstrably answerable by GPT-J-6B in zero-shot configurations. This rigorous selection protocol achieved dual objectives: (1) establishing bascline for knowledge recall under maximally constrained conditions, and (2) systematically mitigating potential confounding effects from response ambiguity in experimental outcomes.
C.2 DATASET LABELING
We use GPT-40 to label the knowledge the dataset. We show the prompts used in Appendix We labeled the dataset related to the knowledge, which are Geographic location, Organization, Personal attributes, Sports, Entertainment, Language and culture, Education, Religion, Literature and Event, which consists of more detailed smaller classes.
I.3
D ACE OPTIMIZATION
Input: Requested edits E = {(81, Ti, of))
model M,
all layers lail.
value layers by
Output: Modified model Me containing edits from E Fi,
for (s, of) E E do
Generate the single edit prompt T,, (s:)
Optimize of + Search(T,, (s:))
11 v² for every new knowledge
end for
E L. do
Al + Calculate W Wi A:;
// edit value layers
// update new weights
end
for
lall do
+- Search in residual (es..ua((11.....1......
end
for
I lg do A + Caleulate(Iof...... ey]]
W + W/ + A:;
// update new weights
end
Algorithm 1: ACE Algorithm
Our method primarily consists of first edit (value neurons edit) and furtherance edit (query neurons edit). Each single edit process obtains target weights through optimization of the knowledge preservation and editing objective:
15

---

Published as a conference paper at ICLR 2026
arg min
A IK jaa Koll
[[KE Vell
(11)
Preservation
Editing
where Ko 1kg kgj ... Ag and Vo = W fe2 Ko encapsulate preserved knowledge, Kg [ 22 ... ke1 represents edit matrices, and V. = [ I... ve denotes target representations for new knowledge. The edited fact set corresponds {(8,, ni 0,) ] i= 1,2,... EJ.
Following the parameterization W- = Weje + A, the closed-form solution for incremental weights is derived as:
KEKE) - : : - W)aKE), (V==-W)) Co: = =o
A =
(12)
The optimization of value vector perturbations 5 follows:
P log P. FM. pref, @T], (13)
8 = arg min ((6) DDDI.. (PM.l' [T) PM[L' [T)+ +
==1
where T denotes KL prompts (e.g., "s_is_a"), excludes answer tokens o", and represents editing prompts (e.g., "The capital of Spain is ").
E KNOWLEDGE BOTTLENECK
In all knowledge editing frameworks based the layers constitutes a crucial parameter. However, extensively editing a large number of model layers proves infeasible and often encounters knowledge bottleneck, which is particularly pronounced in methods focusing solely on value neurons editing. Therefore, we conduct sensitivity analysis on the number of edited layers for ACE and other baselines, demonstrating that query neurons editing is both highly efficient and necessary. Since editing query and value neurons incurs equivalent latency in ACE, begin our comparison of methods' sensitivity to the knowledge bottleneck with an AcE variant restricted to skipping editing single query layer, and then we extend the skipping numbers.
paradigm, the number of critical
As shown in Table 6 we compare the performance of various editors under the condition of an equal number of edited layers, where "#N editor" denotes the quantity of layers edited by each respective editor. It can be observed that a pronounced knowledge bottleneck exists across all editors except ACE. This manifests specifically of edited layers compared to the original baselines, editors focusing solely on value neurons exhibit only marginal performance gains. This phenomenon can be interpreted evidence of an editing bottleneck inherent to value neurons, wherein excessive modifications fail to enhance the model's s reasoning capability and may even prove detrimental. These results underscore the critical importance of editing query neurons.
follows: even when significantly increasing the number
F DETAILS OF INTERNAL QWEN3-8B
REASONING Q- Q-V PAIRS FLOW IN
We examined the residual stream for the single-hop request: "Tim Dunean plays the sport of". We show the importance score increase with forward process here. In table we identify distinct patterns of semantic divergence and convergence the residual stream. At the tokens "plays" and "of", the model exhibits a stronger tendency to conclude the current prediction, narrowing the semantic flow to end the sentence, while demonstrating high interpretability. These tokens also correspond to the largest increase in importance scores. In contrast, at other token positions, the model exhibits greater potential for exploration and reasoning, favoring semantic divergence and losing almost all interpretability. We hypothesize that this intra-sentence semantic actívation pattern is capability
16

---

Published as a conference paper at ICLR 2026
Table 6: Analysis number of edited layers on GPT-J and Qwen3-8B.
Editor
Avg.
Editor
Avg.
#9-
ROME (GPT-J)
37.98
#8 ROME (Qwen3-8B)
30.16
#8
ROME (GPT-J)
37.56
#7 ROME (Qwen3-BB)
29.72 72
#7
ROME (GPT-J)
37.27
#6 ROME (Qwen3-8B)
29.52
#9
MEMIT (GPT-J)
39.47
#8 MEMIT (Qwen3-BB)
21.95
#8 MEMIT (GPT-J)
39.14
#7 MEMIT (Qwen3-8B)
21.47
#7
MEMIT (GPT-J)
38.98
#6 MEMIT (Qwen3-8B)
20.16
#9 PMET (GPT-J)
38.29
#8 PMET (Qwen3-8B)
21.14
#8 PMET (GPT-J)
38.16
#7 PMET (Qwen3-8B)
21.08
#7 PMET (GPT-J)
38.10
#6 PMET (Qwen3-8B)
20.92
#9 ACE (GPT-J)
46.45
#8 ACE (Qwen3-8B)
58.24
#8 ACE (GPT-J)
45.28
#7 AcE (Qwen3-8B)
57.19
#7 ACE (GPT-J)
43.58
#6 ACE (Qwen3-8B)
54.10
conferred by post-training reinforcement learning, enabling the model to rapidly predict correct tokens at critical positions while maintaining strong exploratory behavior elsewhere. In studies related to token entropy in RL, such highly interpretable and semantically convergent regions likely to play a decisive role during training.
Table & demonstrates the neurons' interpretability in vocabulary space. In query neurons, could not find much interpretabilities after projecting neurons, most logits are unrelated to the request, but much interpretable neurons appears in the value neurons. Based this observation, could claim that the query layers enhance knowledge editing not through direct modification of knowledge representations token embeddings, but by amplifying activations value neurons through q-v pairs.
Table 7: Token Increase in Qwen3-8B on Residual Stream.
Token
Importance increase
Top tokens in vocabulary space
Tim
FFN: 0.0021, attn: 0.0014
"an', 'os", 'ise', 'Exactly', 'R', 'ore', 'at', 'rot' 'era', 'allen', 'stad', 'oret', 'hit', '-led' 'basketball', 'NBA', 'career", 'ball', '-layingg'
Duncan
FFN: 0.0014, attn: 0.0009
plays
FFN: 0.9932,
attn: 0.8167
the sport
FFN: 0.4894, attn: 0.4159
"epit', 'inaugural', 'bidding', 'dream', 'etr' 'tennis", 'of", 'ful', 'arena', 'basketball', 'ball' 'basketball', 'NBA', "balls', 'ball', 'Olympia'
FFN: O. 1474,, attn: 0.0948
of
FFN: 0.8469,
attn: 0.6198
Table Interpretable neurons in vocabulary space, bold refers to the interpretable tokens.
Neurons
Top tokens in vocabulary space
fis 5495 (query neuron)
outwe, expries, LESS, retaliate, <, Himself, ALSO football, sports, players, soccer player, baseball, sport basketball, sports, balls, players soccer, baseball, NBA
f18 3584 (value neuron)
Ja (shared neuron Qwen3)
G MORE DETAILS OF THE ANALYSIS
This section provides additional experimental details supporting the analysis in Section 4.2 using the query "Tim Duncan plays the sport of" case study. We analyze the attribution patterns across both FFN and attention layers throughout the forward pass. Figure(]]rreennt the layer-level log increase across all layers, revealing that in Qwen3-8B, activated layers concentrated in deeper regions of
17

---

Published as a conference paper at ICLR 2026
the network. Notably, the activation levels in the final layers remain relatively low, indicating that knowledge is not primarily stored these terminal layers. Instead, these layers appear primarily responsible for generating model outputs rather than knowledge storage.
key observation is the rapid decrease in attention log increase around layer 25, followed by corresponding drop in FFN layer 27. This pattern supports the conclusion that attention mechanisms (and their associated query-value activations) facilitate factual recall by activating progressively deeper FFN layers. The attention heatmap in Figure further corroborates this finding, with darker coloration indicating higher importance scores. These results demonstrate how knowledge accumulation begins in shallower attention layers and culminates in complete activation within deeper network regions.
We observe that during the forward pass of the query "Tim Duncan plays the sport of"" in Qwen3-8B, unlike GPT-J which distributes query-value activation across shallower layers, Qwen3-8B executes this process continuously within deeper layers. As previously analyzed, this pattern arises from Qwen3-8B's more fine-graine ane-grained intra-sentence activation behavior, where active query layers are positioned immediately preceding their corresponding value layers, which typically reside in deeper network regions.
Table shows the top value neurons details in Qwen3-8B while processing this forward case. We select four neurons, representing different important position clear distinction in the interpretability of neurons across differrnnt layers. Neurons in shallower and less critical regions typically exhibit minimal semantic interpretability, whereas those with the highest importance scores demonstrate strong correspondence meaningful vocabulary tokens. To quantify the functional significance these interpretable neurons, we conducted a systematic ablation study.
model. Our analysis reveals
We selected 27 neurons whose associated vocabulary included the correct target token and performed 1000 sampling trials with these neurons removed. The model's accuracy under this condition dropped merely 3. 3.2%, indicating the crucial role these interpretable neurons play in generating correct responses. In contrast, when we ablated 27 neurons with high importance scores but lacking correct semantic interpretability, the model maintained 59.4% accuracy.
This striking disparity suggests that while the reasoning chain involves dense information propagation and accumulation across numerous neurons throughout the network, the final generation of correct output tokens depends on sparse set of highly specialized, interpretable neurons. These findings have important implications for future work on token entropy in reinforcement learning, particularly regarding how models allocate computational resources during reasoning processes.
2.0
attention layers
FFN layers
log increase
0.5
0.0
2
3
4
5
7
8 12011121114111
layer
Figure 5: The layer-level log increase through all layer upon one case on Qwen3-8B.
H EXPERIMENTAL SETTINGS
The critical layers for GPT-J-6B and Qwen3-8B have been identified as R2 13,4,5,6,788) R. {26, 27, 28} and R2 = (25, 26, 27,, R.R.R R, = {28, 29, 30, 31,32); Therefore, we mainly update the FFNS components of these critical layers of GPT-J and Qwen3-8B.
18

---

Published as a conference paper at ICLR 2026
attn head log increase heatmap
2
3
5
7
0 3 5 10111111114115111 15 2tt.://444...1111111111222222223
Figure 6: The attenton head heatmap through all layer upon case Qwen3-8B.
Table 9: FFN Value Neuron Increase and Vocabulary in Qwen3-8B Residual Stream.
Neuron
Importance increase
Top tokens in vocabulary space
fu29 5709
1.1542
'baskethall', 'baskets', 'Basket', 'Baskets', 'ball', "-ASKETT' 'fire', 'licer', 'phu', 'shutdown', 'IAM', 'arez' 'information', 'INFORMAIINN; 'Linformation' 'informação' 'school', 'sniff", 'originals', "baseball', 'balls'
f027 4542
0,4716
fu29 7550
0.4421
fezs 8055
0.1866
In our edits, our configuration for PMET adheres to the settings specified by (Li al, 2024 Initially, we set yp 1 and 0 < < to manage the retention of the model's 's original knowledge. As increases, the retention level also increases, while exhibits the opposite trend. After maximizing the probability of the target knowledge, we reduce to 0.1 to preserve the original knowledge as much as possible. Optimization is halted when DzL < 0.01. On GPT-J and Qwen3-8B, for estimating the covariance matrix (i.e., the set of previously memorized keys Co), sample 10,0000 times on Wikitext in fp32 precision and set = 6000. When optimizing, we limit the total optimization steps to 30 with learning rate of 0.2. All our experiments To test the accuracy of answers to multi-hop questions, we adhered to the few-shot and Chain of Thought (CoT) templates in Appendix [.1 and procedures.
conducted using the MQuAKE dataset.
19

---

Published as a conference paper at ICLR 2026
I PROMPTS
This appendix details the various prompt templates used in our experiments. These templates are used evaluate the model's multi-hop fact recall ability after knowledge editing, well as for automatic classification and annotation of the dataset.
1.1 FEW-SHOT AND SHAIN-OF-THOUUGE EVALUATION PROMPTS
The following is an example of a few-shot and Chain of Thought (CoT) prompt used for the main experiment evaluation. We guide the model to answer complex multi-hop questions by showing reasoning process containing "Thoughts".
country Fleinfield Town located? Thoughts: Plainfield Town Hall ia located in the country of the United States of America. The capital of United States la Kasaington, D.C. Answer: Washingson, D.C.
Qsestion: is capital
Question: which country Thoughts: Nissan 2008X was created by Nissan. Nissan is located in the country of Japan. Answer: Japan
company
2005X lecaled?
Question: Khich continent is the country where the director of "Ky House Blusband: Ikaw Na I" educated locazed in?
Thoughts: the director of "Ky souse Husband: Ika No!" is cose davier Rayes. Jose Javier University.
Beyes educased
Balle University is located country of shilippines. Philippines is located in the continent if Asia.
Answer: Asia
Question: who is the spouse the US president?
thoughts: The president is uoe Bider. The spouse of Joe Widen Jill Riden
Answer: Jill Biden
Dsestion: Kho has cmnership of the developer of the Chevrolet Corvette (C4)?
Thoughts: The developer of Chourolet Corvette (C4) Chaurelet. Chevrolet owned by General Motors.
Answer: General Motors
Usestion:
(Multi-hop questions...)
1.2 PROMPTS TO RECALL SINGLE-HOR FACT
The following prompt templates are used in our dataset screening and construction process, as described in Appendix C.I Before formally conducting multi-hop knowledge editing experiments, we use these straightforward, single-hop factual questions to probe the inherent knowledge reserves of base models (such as GPT-J-6B). This step allows us to identify knowledge chains from the MQuAKE dataset where the model has accurately grasped the ground truth, ensuring the reliability of subsequent experiments and avoiding interference caused by the model's 's inherent lack of foundational knowledge.
country citizenehip Fernando 2: What is the name of the current head of state in Portugel? A: Marcelo Rebelo de Sousa was
Portugal
created by? C. 8. Lexis
Which city was C. S. tewis born in? A: Relfast
0: which sity was Hari Kunzru born in? A: London
Which cortinent
Tondon losated
Rurope
0: Who was Nick Bottom created by? &: Milliam Shakespeace
Shekespeaze 8: Who is the head coach of iran raticnal football teem? 4: Carles Queiroz
What kind
playsright
sport is carles Quairoz associated with? sssaciation (Single-hop questions...!
I.3 DATASET ANNOTATION PROMPT
The following is prompt template for requesting GPT-4o to semantically classify questions in the MQUAKE dataset, as described in Appendix [C2] This template ensures consistency and automation of data annotation by providing strict format requirements.
20

---

Published as a conference paper at ICLR 2026
Table 10: Multi-hop accuracy comparison of different KE methods on the MQuAKE-3K dataset in a few-shot setting, Base shows the model's performance the unedited answers and edited model's performance on edited answers. Our model outperformances than other models significantly.
GPT-J /# Edits
Qwen3-8B / Edits
Editor
(GPT//Qen)))))
1-edit
2-edit
3-edit
4-edit
1-edit
2-edit
3-edit
4-edit
Base
98.42 99.17
99.7
95.48
97.51
97.23
99.81
97.46
98.14
97.64
FT
3.54 / 2.18
4.17
2.63
0.00
0.00
3.14
2.79
0.00
0.00
ROME
35.04/ 28.79
44.51
38.93
17.52
5.06
35.09
32.48
18.97
7.08
MEMIT
38.58 18.67
64.30
16.87
17.25
8.16
29.84
19.18
12.91
4.20
PMET
37.01 20.78
49.26
36.30
24.34
17.01
28.64
14.08
12.56
11.20
AlphaEdit
45.59
47.24
37.69
29.59
28.98
54.58
48.62
27.54
27.31
ACE (PMET)
46.45 58.24
45.26
50.24
36.17
43.29
60.22
59.48
51.62
47.61
ACE (AlphaEdit)
49.24 61.54
52.91
53.74
39.08
43.82
62.48
64.76
63.04
51.32
Please ans yze type of the following questior and the following Tornat, sepazated by 1
categorias strictly
[Main category] [Subeategory]
Qsestion: (question) Category:
J ACE LATENCY
In our experiments, we utilized four A800 (80G) GPUs for computation. ACE can decoupled into two distinct stages: identifying and editing; consequently, its latency is discussed by dividing into these two components. Quite straightforwardly, the latency consumed by AcE in stage can be approximated the model's inference time. Our primary focus in the discussion will be on the latency consumed by the model in stage 2. In ACE, we have the luxury of being able to pre-compute and cache the knowledge values, since they inserted to the model parallel. If all knowledge vectors are already computed, ACE takes 3.10 & 3.14 sec for per update on GPT-J and Qwen3-8B, respectively. The most computationally expensive step inverting large square matrix in Equation [2] To get all knowledge vectors, we need 26, 474.35 & 27, 195.29 sec on GPT-J and Qwen3, however, this expensive computing result could be cache for incoming edits.
K ACE EXTENSION
In this section, we extend our ACE framework to AlphaEdit version. Alphaedit Fang et al. (002)) novel solution that projects perturbation onto the null space of the preserved knowledge before applying it to the parameters. As shown in Table IO we add AlphaEdit as one new bascline for comparison. We also replace the AlphaEdit as our new backbone. After replacing the backbone, our method outperforms 9.97% and 1 15.95% compared to AlphaEdit. Moreover, this extended experiment claims that our Identify-Locate-ee
framework could be extended more later KE methods.
L HUMAN UNDERSTANDING ALIGNMENT
In this section, we present the human alignment validation for GPT-4o-based dataset labeling. To ensure both methodological rigor and demographic diversity in the alignment assessment, the selected participants comprised undergraduate student in literature, graduate student in social sciences, a Ph.D. candidate in STEM, and an undergraduate student STEM. As shown in Table [1] we computed the overlap rate between human labeling and GPT-4o labeling, find that most of the human participants reach an acceptable understanding overlap with GPT-4o on our random sample batches (80 multi-hop questions).
21

---

Published as a conference paper at ICLR 2026
Table 11: The result of human understanding alignment with GPT-4o labeling.
Human Participants
Overlap Rate
Human A
83. 33.75%
Human
88.75%
Human C
93. 75%
Human
80.00%
M MORE EXPERIMENTS
M.1 LOCALITY PERFORMANCE
To comprehensively demonstrate the generalizability of our editing framework and its ability to preserve model capabilities, we have supplemented our evaluation with experiments on four general reasoning benchmarks that assess the model's performance on unrelated tasks after editing. As demonstrated in Table [2 edited models retain their core capabilities without significant degradation on unrelated tasks.
M.2 ATTRIBUTION ROBUSTNESS
To evaluate the robustness of attribution mechanism, we performed multiple inference runs and compared the average importance scores of critical layers across different sampling trials. As shown in Table 13 below, the importance scores for our identified critical modules demonstrate strong stability across different @ass@k k settings. The minimal variance in importance scores across different Pass@k trials (e.g.. f2r varies by only 0.47 points across all settings) confirms that our attribution method identifies consistently significant neural pathways, rather than capturing random or unstable activation patterns.
We also quantify the robustness and stability of attributed query and value layers' rankings across different prompt templates. To quantitatively assess this stability, we conducted a new experiment where we re-ran our layer importance attribution under different in-context learning settings: Zero- Shot and One-Shot, in addition to our original Few-Shot setting. We then compared the Top-9 important attention and FFN layers across these settings for all semantic categories. The results for the GPT-J model are presented as Table [4] and Table 15 For clarity, we have underlined any layer where its rank changed compared the Few-Shot setting used in the main paper. As the tables clearly demonstrate, the rankings of the most critical layers exhibit remarkable stability across different prompting settings:
Consistency of Top Layers: The core set of highly important layers remains virtually unchanged. For instance, attention layers d27, 026. and ay consistently rank in the top positions across nearly all categories and settings. Similarly, FFN layers like f26. f27 and fzi are stably identified as critical.
Minor Nature of Changes: The few rank changes that do occur (underlined) are predomi- nantly in the lower half of the Top-9 list.
Practical Implication for ACE: This stability is crucial for ACE's practicality. It means that the critical Q/V layers for given type of knowledge can be reliably identified using standard prompting setup (e.g., Few-Shot). The subsequent edits targeting these layers are then likely to be effective across various phrasings of queries involving that knowledge, as the underlying neural pathways remain the primary routing and storage sites.
M.3 VERIFYING INTERMEDIATE REASONING PROCESS CHANGES
Now verify that the edited value propagates through the intended implicit subjects, rather than being injected late. To address this, we conducted a new experiment designed to trace the flow of information through the reasoning chain before and after critical intervention: masking the final deep value editing. The core logic is as follows: the edited knowledge is merely "injected" at the final generation step, then masking the final value edit should not significantly affect the activation of
22

---

Published as a conference paper at ICLR 2026
Table 12: Locality Performance on several general benchmarks of ACE and other editing methods.
Editor
CSQA
BBH
MMLU
GSM8K
ROME (Qwen3-8B)
75.41
35.62
65.28
75.80
MEMIT (Qwen3-8B)
82.49
32.68
72.64
83.32
PMET (Qwen3-8B)
82.26
34.20
71.80
83.00
ACE (Qwen3-8B)
82.30
38.54
72.30
83.84
Table 13: The average importance score Pass k in identifying critical layers.
Layer
Pass@1 1
Pass @
P
Pass@5
027
(Qwen3-BB)
110.48
109,29
109.45
109.69
026
(Qwen3-8B)
97.58
98.21
97.94
98.02
07 (Qwen3-BB)
64.29
64.80
64.75
64.78
f27 (Qwen3-8B)
137.48
137.95
137.66
137.74
f21
(Qwen3-8B)
104.20
104.39
104.28
104.23
f16 (Qwen3-8B)
42.58
42.60
42.64
42.52
intermediate, implicit subjects. Conversely, if the edited knowledge is genuinely propagated through the chain via implicit subjects, then the intermediate activations should remain robust even when the final value edit is masked, while the final answer activation alone should drop precipitously. As the results demonstrated Table [6] and 17 the activation patterns at critical intermediate tokens representing the implicit subject (e.g..'pu 'plays' 'basketball') remain stable and highly interpretable after masking the final value edit. The importance scores for 'plays' show minimal change, and its top vocabulary tokens remain strongly associated with the correct sport. This indicates that the query- layer edits successfully guide the model to the correct implicit subject ("basketball"), independent of the final answer's direct manipulation. In contrast, the activation at the final answer token 'from' experiences a severe drop (over 40%) after masking the value edit. Furthermore, the semantic specificity its vocabulary projection degrades from concrete country names ('USA', 'America") to intermediate definitive evidence that ACE's edits
generic and uncertain terms ('country', 'many"). This clear Doasociaation-preseeee reasoning but disrupted final answer ceneerationprovinne propagate the updated knowledge through the intended implicit subjects along the reasoning chain. The final value edit then truthfully enhances the prediction based on this correctly propagated context ("basketball' to UUSA'), rather than injecting the answer "USA" directly the end.
M.4 COUNTER- COUNTER-FACTUAL EDIT
To demonstrate that our ACE has effectively edited the model, have conducted experiments counterfactual editing. We mismatched the relationship labels the knowledge triplet to construct counterfactual editing dataset. The results, presented in Table 18 demonstrate ACE's behavior under adversarial conditions. We also add the experiments the general benchmarks to claim that the
Table Top 9 important attention layers (left block) and FFN layers (right block) in GPT-J using zero-shot implementation. Underline terms to the rank of the layer occurs changes.
Top 9 important attention layers and FFN layers (zero-shot)
NN
027
(/26
07
010
(g
0gg
(725
d11
45
2n
lz
fio
f22
fis
f15
226
125
CT
027
0/2
07
010
0g
0g
025
Jz
f16
a21
J15
F17
$26
525
J23
LG
027
0g
07
0g
0g
04
01
022
0g
J27
f7
fo
16
88
f4
S1
826
8
CP
027
07
010
0g
0g
012
a5
A6
f27
f26
F10
fI2
fo
fg
Js
86
LS
022
0:6
0:5
07
010
46
0g
05
f27
f26
f25
f21
f7
f10
fo
fx
AT
0:7
0:2
0:2
07
0g
0g
(100
05
06
fr
fg
fz
F10
f2z
f26
f25
f6
ST
026
027
07
Gg
05
(25
04
J26
22
JT
f10
88
[a
S4
CF
026
0/2
022
(124
(8
0g
(gg
45
J26
J22
24
J2
J7
88
J9
86.
86.
23

---

Published as a conference paper at ICLR 2026
Table 15:
Top 9 important attention layers (left block) and FFN layers (right block) in GPT-J using one-shot implementation. Underline terms to the rank of the layer occurs changes.
Top 9 important attention layers and FFN layers (one-shot)
NN
427
22
Q7
010
ag
425
dg
a11
f20
f24
fie
fis
f15
f22
f23
f2
f25
CT
427
026
07
010
0g
05
0g
011
0/25
f22
f24
f16
f2
f15
f17
f26
f5
f23
LG
027
07
05
06
0g
044
d1
026
0g
f27
ft
86
fa
f1
f26
CP
027
47
026
010
0g
0g
012
Ag
f27
f2a
f7
f10
f12
fa
fo
fg
LS
d22
026
0g5
07
0110
06
0g
06
0g
J27
J26
125
J24
J7
f10
fe
Jg
J3
AT
0g27
025
07
0g
010
05
46
fT
8o
88
f10
fer
826
f25
fo
fo
ST
026
027
07
010
ag
06
05
025
a4
f26
f27
f10
f8
fo
f5
25
f4
CF
426
024
025
048
(g
05
/26
22
22
825
J7
88
89
fa
J5
Table 16: The Original Token Increase in Qwen3-8B on Residual Stream.
Token
Importance increase
Top tokens vocabulary space
Tim
FFN: 0.0014, attn: 0.0014
'an', "os', 'isc', 'Exactly'. 'R', 'ore', 'at', 'rot' 'era', 'allen', 'stad', 'oret', 'hit', '-led' "basketball', 'NBA', 'career', 'ball', '-playing'
Dunean
FFN: 0.0009, attn: 0.0009
plays
FFN: 0.9846,
attn: 0.8167
the sport of originates from
FFN: 0.4109,
attn: 0.4159
'epit', 'inaugural', 'bidding', 'dream', 'etr' 'tennis', 'of", 'ful', 'arena', 'basketball', 'ball' "basketball', 'NBA', 'balls', 'ball', 'Olympia' 'kati,, 'the'. 'from'. 'oret', 'orig'. 'ball' *USA', 'America', 'the', "US', 'U.S.A."
FFN: 0.0848,
attn: 0.0948
FFN: 0.8671,
attn: 0.6198
FFN: 0.3508, attn: 0.3058
FFN: 0.9483, attn: 0.8720
Table 17: The Token Increase in Qwen3-8B on Residual Stream After Masking Value Editing.
Token
Importance increase
Top tokens in vocabulary space
Tim
FFN: 0.0019, attn: 0.0010
'an', 'os", 'ise', 'Exactly', 'R', 'ore', 'at', 'rot" 'cra', 'allen', 'stad', 'oret', 'hit', 'lled' 'basketball', *NBA', 'career', 'ball', '-playing' "epit', 'inaugural', 'bidding'. 'dream', "etr' 'tennis', "of", 'ful', 'arena', 'basketball', 'ball' "basketball', 'NBA', 'balls', 'ball', 'Olympia' 'kati', 'the', 'from', 'oret', 'orig'. 'ball' 'country', 'many', '-Info', "bot', -US' 'USA'
Duncan
FFN: 0.0008, attn: 0.0012
plays
FFN: 0.9779,
attm: 0.8498
the sport of originates from
FRN: 0.4093,
attn: 0.4715
FFN: 0.0913,
attn: 0.0142
FFN: 0.9252,
attn: 0.7090
FFN: 0.2308,
attn: 0.3194
FFN: 0.5603,
attn: 0.4764
Table 18: The detailed results of more metrics in experiments GPT-J and Qwen3-8B in counter-fact scenarios.
Editor (Counter-Fact)
Efficacy
Paraphrase
Specificity
Avg.
FT (GPT-I/Qwen3)
98.1 / 97.9
64.4
82.7 / 77.4
1.59 1.04
ROME (GPT-J/Qwen3)
54.1 / 45.2
54.3/
63.7/53.7
27.48 24.08
MEMIT (GPT-J/Qwen3)
57.0 50.3
51.9/
66.2 / 66.4
30.09 10.27
PMET (GPT-I/QWen3)
74.6 70.7 70.7
63.2/61.7
64.1/510
31.07 17.26
ACE (GPT-J/Qwen3)
89.7/911
83.6 80.7
70.6 / 74.6
43.58 54.27
locality performances of edited model are not corrupted highly. As demonstrated in Table 19 the result shows that after counter-factual editing in the model, the locality of the model keeps stable among four general benchmarks, showing the robustness and safety in ACE framework. ACE effectively edits knowledge even in incorrect reasoning chains, demonstrating that modifies the model's 's internal representations rather than merely amplifying correct reasoning patterns. ACE also maintains strong locality performance across general benchmarks (CSQA, BBH, MMLU, GSM8k), showing minimal negative impact unrelated capabilities.
24

---

Published as a conference paper at ICLR 2026
Table 19: Locality Performance on several general benchmarks of ACE and other editing methods in counter-fact scenarios.
Editor (Counter-Fact)
CSQA
BBH
GSM8k MMLU
ROME (Qwen3-8B)
68.54
29.47
60.71
69.48
MEMIT (Qwen3-8B)
71.46
29.72
66.40
72.45
PMET (Qwen3-8B)
73.09
26.58
62.49
74.29
ACE (Qwen3-8B)
76.41
33.79
69.97
79.02
N ACE ATTRIBUTION DERIVATION
In this section, we provide step-by-step formal derivation the ACE attribution metrics. We begin by modeling the algebraic structure of the Transformer Residual Stream, prove the strict additivity of Logits, and derive the Importance Score via first-order Taylor approximation. Finally, we prove that the additivity assumption is mathematically isomorphic the update mechanism of the PMET editor.
N.1 ALGEBRAIC STRUCTURE THE TRANSFORMER
Let M an autoregressive Transformer model with layers. We define the hidden state propagation through the residual stream.
Definition (Residual Stream Dynamics). Let h()) E Rd denote the hidden state input to layer l. The output of layer I is the sum of the Multi-Hes Head Self-Attention (MHSA) output and the Bred-Forward Network (FFN) output, added to the residual stream:
k(+11) = hh + MHSA (1.0) + FFNO((
By recursive expansion, the final hidden state ( the cumulative of the initial embedding and all layer outputs:
(D) (0) + Σ MHSS 1 (1))) + Σ FFND (1)))
I
Definition 2 (FFN Key-Value Memories), Let K0)) and v be the parameter matrices (also denoted as W far and W We2). The output a weighted sum of value vectors:
N
FFN (z) = Σ (z)( U1
where vt,s RR. the i-th column of the second weight matrix W!l) fez and mı,(x)) oxx ((.Tkk.)) is
the scalar activation coefficient.
N.2 THEOREM OF LOGIT LINEARITY
The probability distribution over the vocabulary is computed via the Softmax function applied to the logits E R Let E E [dx[V| be the unembedding matrix, and ew be the column vector corresponding to token
Lemma (Logits). The logit zw for a target token w is the inner product of the final state and the token embedding:
(1( ew)
Theorem 2 (Strict Linearity of Neuron Contribution). The contribution of any single FFN neuron (L, to the target logit zn strictly additive and independent of other neurons in the logit space.
25

---

Published as a conference paper at ICLR 2026
Proof. Substituting Definition 2 into Definition 1, and then into Lemma 1: Zu
1!0) + Σ MHSA +
Σ
(h + Σ MHSA, ew)
(mu,) ew)
L.
Let Zbase (1 + Σ MHSA, Cw) be the base logit. The equation simplifies to:
+ Σ Em
Fi Zhase
1=j
Define the marginal contribution of neuron (1, as AZs, = (V,,ee))))) follows that:
(S) Zbase +
AZia
(1,))S
where S' is any set of active neurons. The interaction terms are zero in the logit space.
N.3 DERIVATION OF IMPORTANCE SCORE
ACE defines the importance score I based on the change in Log-Probability. While Log-Probability non-linear, we prove that Equation 9 is the First-Order Taylor Approximation of the causal effect.
Let the objective function be f(z) = log P(w) log
Consider the activation of a neuron This causes perturbation in the logits Az u² E. We perform
perturbation vector
a first-order Taylor expansion of
((=+Az)
around the current logit 2:
L(= + Az) = L(2) + (=((2)² A
The gradient of the Log-Softmas function w.r.t the logits 2 is:
UC
8k P(k)
azk
where 6ew is the Kronecker delta. Substituting the gradient and the logit perturbation:
I(v) = L(2 A2) - [(2)
(8kw - -P(k)) Azk
KEV
(1 - P(w))
Σ P(K)A
This derivation confirms that Eq. iis formally the Marginal Contribution the Logit (Az), projected onto the tangent space of the probability simplex.
N.4 IMPORTANCE ADDITIVITY
Theorem 3 (Additivity of First-Order Attribution). Under the assumption that the local curvature of the -ikelliod manifold is negligible (first-order approximation), the importance score of set of neurons is the sum of their individual scores.
Proof. From Theorem 1, the total logit perturbation is the sum of individual perturbations:
Σ A 2;
Stotal =
26

---

Published as a conference paper at ICLR 2026
The importance score is linear map of the logit perturbation:
I(A2) V A
By the linearity of the inner product:
Icotal 2 V.L.²
Σsn
Σ CT
I(vs)
Thus, layer importance is mathematically valid within the local neighborhood defined by the Taylor expansion.
N.5 ALIGNMENT WITH PMET
Finally, we prove that the additivity AcE is necessary condition imposed by the update mechanism of the PMET editor.
Theorem 4 (Linearity of PMET Updates). The PMET editor updates the FFN weights W by solving a linear squares problem(6). The closed-form solution for the weight update AW is:
AW RKK =RK"(Cg + +KK
This results in a linear shift in the output of the FFN for given input kan
8FFN = AW kin
Proof. Let us decompose the FFN output shift öFFN into the basis of neurons. Since W is composed of columns v;, the update AW corresponds updating individual value vectors vs 02 5zg.
ÖFFN = Σ mi (v + ōv;) Σ m. Σ m4 80
PMET operates by injecting a sum of linear updates into the residual stream. If ACE were to use non-additive metric, would identify neurons crucial for non-linear inference. However, PMET's linear update mechanism AW is mathematically incapable of manipulating such non-linearities. G
27


Reviews:
[{"review_id": "79L5BKYhKL", "reviewer": "Reviewer_aVHM", "summary": "This paper investigates the failure of existing Knowledge Editing (KE) methods in multi-hop factual recall scenarios, particularly when edits involve intermediate, implicit subjects. The authors provide a new mechanistic insight, suggesting that multi-hop reasoning relies on a \"query-value\" (Q-V) activation pathway, where implicit subjects act as \"query neurons\" that sequentially activate \"value neurons\" across layers. Based on this, they propose ACE, an editing framework that uses attribution to identify and modify both the critical query (reasoning path) and value (factual storage) neurons. The method shows significant performance improvements on multi-hop recall tasks over existing baselines on GPT-J and Qwen3-8B.", "questions": "See weaknesses", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. **Valuable Mechanistic Insight**: The paper's core strength is its deep dive into why multi-hop editing fails. The proposed Q-V activation mechanism  offers a plausible and fine-grained explanation for how information is accumulated during complex reasoning, moving beyond layer-level heuristics.\n2. **Novel Editing Framework**: The ACE framework is a direct and logical application of this insight. By targeting not only the \"value\" FFNs (for the fact itself) but also the \"query\" FFNs (for the reasoning path), the method presents a more complete solution that is mechanistically grounded.\n3. **Strong Empirical Gains**: The reported results are impressive, with ACE substantially outperforming SOTA methods like PMET by 9.44% on GPT-J and 37.46% on Qwen3-8B in multi-hop accuracy. The ablation studies also clearly validate the necessity of editing both Q and V components .", "weaknesses": "1. **Potential Data Leakage in Evaluation** : I am concerned about the experimental setup regarding the MQuAKE dataset. The paper seems to use MQuAKE's multi-hop question templates (as seen in Appendix H.1 ) to construct the few-shot and CoT prompts for the evaluation.\nIf the model is evaluated on MQuAKE questions after being primed with prompts structured exactly like MQuAKE's own examples, does this not constitute a form of \"teaching to the test\"?\n2. **Practicality and Efficiency of Attribution**: The ACE framework introduces a \"Stage 1: Identifying\" step  that relies on attribution to locate critical Q-V neurons before editing. This \"attribution-locate-then-edit\" paradigm appears to add significant computational overhead. Is this attribution step a one-time analysis required per model, or is it a dynamic analysis required per edit? The paper suggests Qwen3-8B has domain-specific, dynamic alignment, implying the latter. If this attribution must be run for every new edit, the method's practical utility is questionable compared to \"on-the-fly\" editors like ROME or PMET. The authors must provide a clear analysis of the computational cost (e.g., latency, FLOPS) per edit versus the baselines.\n3. **Missing SOTA Comparisons and Efficiency Analysis**:\nThe set of baselines , while foundational, seems incomplete. The field has advanced quickly. Why are more recent and highly relevant SOTA methods, such as AlphaEdit or other concurrent works focusing on multi-hop editing, omitted from the main comparison in Table 2? Furthermore, the paper lacks a rigorous analysis of batch editing efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dtaj8153a4", "reviewer": "Reviewer_UH9o", "summary": "The authors introduce ACE, a neuron-level editing framework. They first attribute multi-hop recall to coordinated query (Q) and value (V) neurons across transformer FFNs; then, they apply that attribution to perform staged edits. Specifically, ACE achieves this by first overwriting value layers (closed-form, like PMET) to encode the new fact; then, it adjusts query layers which control access to said edited value such that the updated knowledge propagates through multi-hop chains. The authors test ACE on a filtered multi-hop subset of MQuAKE-3K and report sizable accuracy gains over FT/ROME/MEMIT/PMET on GPT-J and Qwen3-8B. Their ablation experiments also indicate that both Q- and V-stage edits are necessary, as evidenced by attribution metrics (log-probability change and inner-product-based query scores) and intervention results. The experiments and method design support the validity of the authors' interpretation that the query-to-value activation mechanism mediates multi-hop recall.", "questions": "1. Could you please provide hop-wise activation patching and intervention results that show that the edited value is carried through the intended implicit subjects rather than injected late?\n2. How stable are Q/V layer/neuron rankings and attribution results across prompt templates, paraphrases, and relation types?\n3. Could you please evaluate cases where an intermediate relation is wrong and must itself be edited, and include locality/specificity metrics on unrelated facts?\n\nI am willing to raise my score if these points are addressed via new results, or if the authors can justify why they are not needed to support the main claims.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The mechanistic motivation behind the two-stage editor is excellent. The method aligns with observed Q-to-V activation timing, rather than relying on heuristics for layer selection.\n2. The formalization of neuron-level importance is clear and convincing, with consistent localization patterns across semantic categories. The ablations demonstrate strong causal sensitivity.\n3. The multi-hop gains across the two tested models are substantial.", "weaknesses": "1. While the authors frame their work as multi-hop reasoning editing, the evidence predominantly demonstrates improved propagation of edited facts rather than modification of the intermediate reasoning process itself.\n    * The method edits FFN value weights and query gating to make the correct value more retrievable along existing chains. However, the evaluations measure only final-answer accuracy, without verifying whether intermediate computations actually changed.\n    * Trajectory-level tests may be needed to fully accept the implied mechanistic claim. Specifically, one or both of the following would provide definitive evidence: (i) activation patching for each hop pre/post edit; and (ii) tracing attention and activations to confirm that the edited value propagates through the intended intermediate subject rather than being directly re-inserted at the end. The current results are valuable in that they clearly demonstrate better retrieval of a rewritten fact, but they don't establish that the reasoning process itself has been modified.\n2. Attribution-based layer rankings may be tightly coupled to the specific prompt and decoding context. To my knowledge, the paper does not explicitly demonstrate stability across different queries, so it is possible that the importance score (log-probability gain on the final token when a value neuron is restored) and query score (inner product with learned subkeys) can both shift with template, subject position, and relation phrasing.\n    * Because ACE edits only the top-ranked layers, it implicitly assumes the rankings will remain the main routing and storage sites across all different wordings of the same relation. The paper does not seem to verify this via rank-stability tests.\n    * The authors should check whether critical layers stay on top under paraphrases, alternate templates, or different relation types. That way, it will be more clear whether ACE can generalize beyond the identification prompt and avoid overfitting to it.\n3. Relatedly, the mechanistic claims lack per-instance causal verifications.\n    * The authors show that ablating a small neuron set greatly reduces accuracy, but they do not isolate whether those neurons are causal for specific chains or globally influential.\n    * Per-instance causal scrubbing along the chain may be needed to fully accept the causal claims.\n4. The current evaluation setting may bias the reported gains toward cases in which the model already does well.\n    * The authors evaluate on a curated subset where the base model already answers correctly (selected via zero-shot answerability).\n    * In this setting, ACE performs counterfactual edits and then measures whether that new fact propagates along the model's existing multi-hop reasoning path (including the paraphrase and specificity checks).\n    * This does not necessarily probe whether ACE works for failure modes wherein the path itself is unreliable. In other words, it remains unclear whether ACE can repair a broken intermediate link (e.g., when the base model originally follows the wrong entity mid-chain).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ByasOYKdUW", "reviewer": "Reviewer_8jSo", "summary": "The authors propose Attribution-Controlled Knowledge Editing (ACE), a framework aiming to enhance multi-hop factual recall after knowledge editing. The authors claim prior locate-then-edit methods (ROME, MEMIT, PMET) degrade on multi-hop questions, esp. when edits involve implicit subjects. The framework builds on the empirical observation that implicit subjects act as query neurons that activate value neurons in subsequent layers. ACE identifies critical query layers and value layers and performs sequential edits (value-first then query) using a PMET-style closed-form update.\nOn a curated subset of MQuAKE-3K, ACE improves multi-hop accuracy vs. baselines by approx. 9% (GPT-J-6B) and 37% (Qwen3-8B) over PMET. Ablations that skip editing top-ranked query or value layers reduce the performance.", "questions": "See comments above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Multi-hop knowledge editing is definitely under-researched and practically important\n2. The Q-V pathway view provides a clear hypothesis (drawn from empirical observations) to test and design around\n3. Simple and clear pipeline: (1) Identify (via attribution),  (2) edit value layers (deep),  (3) edit query layers (mid); built atop PMET\n4. Significant improvement over baselines\n5. The authors analyze and highlight the distinct roles of query and value edits. Skipping value layers hurts most, which aligns with the intuition that query layers rather transmit, whereas value layers rather store information. These are interesting findings.", "weaknesses": "1. I am not convinced that the additivity of log-prob importance (Eq. 9) is well-suited. It is simply asserted by the authors, not derived or tested. The same holds for Eq. 10. Both equations ignore underlying nonlinearities.\n2. The authors use a subset of MQuAKE-3K in the experiments. It is not clear what they did to prevent selection bias. LLM-based labeling lacks human validation. Also, it is not clear how prompts were handled across baselines. What did you do to guarantee a basic level of uniformity?\n3. From a statistical analysis viewpoint, the evaluation is quite weak. There are no reports of significance tests, CIs, or multi-seed variability. There are no \"negative\" control analyses (e.g., wrong layers, randomized layer ranks, randomly swap query/value roles, etc).\n4. The most critical point is the evaluation on a single dataset. To show generalizability, the authors should extend the experiments and evaluate on additional datasets.\n5. It would be very helpful to include a human audit sample (e.g., 300 instances) with inter-annotator agreement of GPT-4o labels.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iEzQdWtLyc", "reviewer": "Reviewer_pPXV", "summary": "This paper explores multi-hop knowledge editing. The authors hypothesizes that the failure is because the missing tracking of intermediate implicit subjects within reasoning chains. Using a suite of attribution metrics, they aim to uncover the mechanism of multi-hop reasoning. Based on the analysis of identifying important attention layers and FFN layers, Attribution-Controlled Knowledge Editing is proposed. Evaluation on a single dataset Mquake-3k is performed to demonstrate the effectiveness of the proposed method.", "questions": "See above", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper explores an interesting and important direction of multi-hop knowledge editing.\n\n- This paper not only presents a solution, but also gives the analysis on which the approach is conditioned.", "weaknesses": "- The two takeaways are well-explored and not novel. For example, *Takeaway 2: Information of the final answer is accumulated through implicit query neurons that sequentially activate corresponding value neurons across the reasoning chain.* is verified in [1]. \n\n- The analysis in Section 4 lacks sufficient rigor. In particular, the selection of the subset from MQuAKE-3K is not clearly justified. Although the analysis is entirely based on this subset, no details about it are provided. For instance, the number of samples or their distribution across different knowledge types. Given that MQuAKE-3K is not a large dataset and the analysis only involves forward passes, it would be more convincing to conduct the analysis on the full MQuAKE-3K dataset rather than on an unspecified subset.\n\n- The deduction from the analysis is not convincing： a27, a26, a7 ranks top in all knowledge-> MHSA stores general knowledge and capabilities in LLMs.\n\n- Authors claim that FFN layers tends to primarily extracts its own knowledge. But f_26 ranks top for all knowledge types, which contradicts the conclusion.\n\n- In the identifying process, forward passes on all multi-hop questions in the dataset and computed the sum of the importance scores at the last token position are performed. Is this setting reasonable? Is this instance-level or dataset-level editing?\n\n- The whole evaluation is based on one benchmark MQuAKE-3K, which cannot demonstrate the generalization of the conclusion. It would be better to conduct the evaluation on more multi-hop knowledge editing datasets.\n\n[1] CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)