Knowledge Localization: Mission Not Accomplished? Enter Query Localization!
Authors: Yuheng Chen
Pengfei Cao
Yubo Chen
Kang Liu
Jun Zhao
Venue: ICLR 2025 Spotlight (2025)
Keywords: Knowledge Neruon Thesis
Knowledge Localization
Query Localization
Primary area: interpretability and explainable AI
TL;DR: We re-evaluate the knowledge localization assumption, demonstrate its limitations, and propose the query localization assumption, proving that the knowledge localization assumption is merely a simplification of the query localization assumption.

Abstract: Large language models (LLMs) store extensive factual knowledge, but the mechanisms behind how they store and express this knowledge remain unclear.
The Knowledge Neuron (KN) thesis is a prominent theory for explaining these mechanisms. This theory is based on the **Knowledge Localization (KL)** assumption, which suggests that a fact can be localized to a few knowledge storage units, namely knowledge neurons.
 However, this assumption has two limitations: first, it may be too rigid  regarding knowledge storage, and second, it neglects the role of the attention module in  knowledge expression. 
 
In this paper, we first re-examine the KL assumption and demonstrate that its limitations do indeed exist. To address these, we then present two new findings, each targeting one of the limitations: one focusing on knowledge storage and the other on knowledge expression.
We summarize these findings as **Query Localization** assumption and argue that the KL assumption can be viewed as a simplification of the QL assumption. 
Based on QL assumption, we further propose  the Consistency-Aware KN modification method, which improves the performance of knowledge modification,  further validating our new assumption. We conduct 39 sets of experiments, along with additional visualization experiments, to rigorously confirm  our conclusions. Code will be made public soon.

Published as a conference paper at ICLR 2025
KNOWLEDGE LOCALIZATION: MISSION NOT ACCOMPLISHED? ENTER QUERY LOCALIZATION!
Yuheng Chen Pengfei C Yubo Ch Kang Liu Liu Z2 12-
'The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China chenysheng20228i.. ac.aaaa.cmm cn. {pengfei cao, yubo. chen, kliu, jzhao) ia ac.cn
ABSTRACT
Large language models (LLMs) store extensive factual knowledge, but the mech- anisms behind how they store and express this knowledge remain unclear. The Knowledge Neuron (KN) thesis is prominent theory for explaining these mech- anisms. This theory is based on the Knowledge Localization (KL) assumption, which suggests that a fact can be localized a few knowledge storage units, namely knowledge neurons. However, this assumption has two limitations: first, it may be too rigid regarding knowledge storage, and second, neglects the role of the attention module in knowledge expression.
In this paper, first re-examine the KL assumption and demonstrate that its lim- itations do indeed exist. To address these, we then present two new findings, each targeting one of the limitations: one focusing knowledge storage and the other on knowledge expression. We summarize these findings as Query Localization (QL. assumption and argue that the KI. assumption can be viewed cation of the QL assumption. Based on QL assumption, we further propose the Consistency-Awes KN modification method, which improves the performance of knowledge modification, further validating our new assumption. We conduct 39 sets of experiments, along with additional visualization experiments, to rigorously confirm our conclusions. Code is available here.
simplifi-
INTRODUCTION
Large language models (LLMs) are believed to store extensive factual knowledge (MetaAI, 2024; Touvron et al., 2023), however, the mechanisms behind this storage and expression have not been well-explained. The Knowledge Neurons (KN) thesis (Dai et al., 2022; Meng et al., 2022; 2023; Niu al., 2024; Chen et al., 2024b;a) is prominent theory aiming to explain these mechanisms. proposes that LLMs recall facts through their multi-layer perceptron (MLP) weights, referring to the units responsible for storing knowledge as knowledge neurons (KNs). Based on this, KN- iisspreed model editing methods are proposed (Meng et al., 2022; 2023), which first localize knowledge neu- rons and then modify them to update knowledge, providing further support for the KN thesis. Not only them, but also many works have adopted KN theory and applied it to study downstream tasks (Chen et al., 2024b;a; Wang et al., 2024c), making its theoretical foundation crucial.
In fact, the KN thesis is based on the knowledge localization (KL) assumption: piece of factual knowledge can be localized to several knowledge neurons. However, this assumption has two limi- tations. (1) In terms of knowledge storage, if refer to different rephrased queries expressing the same fact as neighbor queries, and the corresponding knowledge neurons as neighbor KNs, then the KI assumption implies that neighbor KNs consistent. However, as Figure 1 illustrates, while the neighbor KNs of Fact, exhibit high consistency, those of Factz show low consistency, indicating the KL assumption does not hold universally. We denote facts that satisfy the KL assumption as Con- sistent Knowledge (Kc, e.g., Fact)), while facts that violate the KL assumption are categorized as Inconsistent Knowledge (K,, e.g., Factz). Previous research and the KL assumption essentially assume that all factual knowledge belongs to Kc. (2) In terms of knowledge expression, the KL
"Corresponding authors.

---

Published as a conference paper at ICLR 2025
Salaman i,
K- Fact
Sptmma ait
Canaaph Allin wio
Ki Fact
Cansogt Antiass mayse
Thanstona Layee
100
1555
2.83
A071 300
1035
2.80
2001
13.45
2044
2071
300
Figure 1: Heatmaps of the neuron activation values, with darker colors indicating higher val- ues (can be viewed as knowledge neurons). The left two heatmaps show neuron activations for two neighbor queries of (Suleiman I, position, Shah) (Facti), while the right two correspond to (Christoph Ahlhaus, position, mayor) (Fact2).
assumption overlooks the attention module, yet there must be interconnections between the different modules in L.Mss. Similarly, since KI. only considers the role of the MLP module in storing knowl- edge. it does not take into account how the model selects and expresses this knowledge to answer queries, Therefore, we re-examine the KL assumption and raise questions QI and Q2:
Q1: Does the KL assumption hold for all facts? If not, is K2 widely preva- lent? ($2)
Associate
The capital China
Quary
Select
KN
A1 We investigate the knowledge lo- calization assumption and find that the universal presence of K2 that vi- olates this assumption.
Attention
Beijing
42 China's capital is
Keys
Values
MLP
KN,
(1) Statistical Evidence. As shown Figure 1, if the knowledge neurons corresponding to a fact exhibit low consistency for its neighbor queries, it indicates that the fact does not form to the KL assumption. Based on this observation, we propose a metric to evaluate the con- sistency among neighbor KNs, and the statistical results show that significant proportion of facts belong to Ky. For example, in LLaMA3-8b, this proportion reaches 77%. This directly proves that facts that do not conform to the KL assumption are widespread.
Associate
Figure The Query Localization assumption.
(2) Modification-Ban Based Evidence. We categorize facts into Kg and K, based on their consistency perform knowledge erasure and updates. We find that for facts in K₁, editing the KNs cor- responding to the query itself does not generalize well to neighbor queries. This indirectly indicates that the neighbor KNs for K, are inconsistent. In summary, the answer Q1 is: the KL assumption is not always valid and K2 is widely prevalent.
Q2: Since the KL assumption has two limitations, what is more realistic assumption? ($3)
A2 Our two findings address the two limitations of the knowledge localization assumption.
(1) Query-KN Mapping: In terms of knowledge storage, the KL assumption implies that localiza- tion results are static and universally applicable across all queries. However, our findings indicate that for facts in K₁, localization results are influenced by the query context rather than being fixed. In other words, knowledge neurons are associated with the query rather than the fact. For instance, Figure shows that different neighbor queries for Fact, correspond to different knowledge neurons. Similarly, in Figure 2, neighbor queries and q2 are associated with distinct KNs (K (KN, and K N2).
(2) Dynamic KN Selection. In terms of knowledge expression, the KI. assumption overlooks the role of the attention module. Our findings show that LLMs rely on the attention module to select appropriate KNs to answer a specific query. For example, in Figure 2, neighbor queries qi and q2 are associated with different KNs. Then, when 91 is input, KN, is activated and selected to provide the answer "Bejjing" while the activation value of K N₂ remains low, preventing it from being selected.
Based on these insights, propose the Query Localization (QL) assumption, which consists of query-KN mapping and dynamic KN selection. To further demonstrate the validity of our assump- tion, we apply it in model editing experiments. We propose the Consistency-Aware KN modification
2

---

Published as a conference paper at ICLR 2025
method, which leverages the QL assumption to improve knowledge modification, achieving 8% and 9% performance improvement over two baselines in the "Erasure" setting on LLaMA3-8h fur- ther validating the QL assumption. In summary, the answer to Q2 is: more realistic assumption is the Query Localization assumption. Our contributions are summarized as follows:
We conduct the first in-depth exploration of the Knowledge Localization assumption, foundational and widely accepted assumption. We classify facts into Ke and K1, and demonstrate that K,, i... facts that do not adhere this assumption, are widely present.
We propose more realistic Query Localization assumption, which includes two parts: query-KN mapping and dynamic KN selection. This addresses the limitations of the KL assumption in both knowledge storage and expression.
We apply the QL assumption to improve knowledge modification methods, further validat- ing the soundness of the QL assumption.
EXPLORING KNOWLEDGE LOCALIZATION LIMITATIONS
This section investigates Q1 and demonstrates the existence of Inconsistent Knowledge (K 1). which does not satisfy the knowledge localization (KL) assumption. Our experiments adopt GPT-2 (Rad- ford et al., 2019), LLaMA2-7b (Touvron et al., 2023), and LLaMA3-81 (MetaAI, 2024), represent- ing a range of sizes of popular auto-regressive models. This allows us to assess the scalability of methods and conclusions. Consistent with other knowledge localization methods (Dai et al., 2022; Chen et al., 2024a), we employ the fill-in-the-blank cloze task (Petroni et al., 2019) to assess whether pretrained model knows a fact. Regarding the dataset, we employ the ParaRel dataset (Elazar et al., 2021). For details to the dataset, see Table 5 in Appendix B.
2.1 STATISTICAL EVIDENCE FOR THE EXISTENCE OF INCONSISTENT KNOWLEDGE
In this subsection, we prove that the consistency of knowledge neurons of some facts is very low, which shows that these facts do not conform to the knowledge localization assumption.
Consistency Analysis According to the KI. assumption, neighbor queries should be localized to the same KNs. with any deviations primarily attributable to the localization method itself. To assess this, we calculate the corresponding KNs for each query and introduce the -NNCossist Score (CS) metric. Given fact with & neighbor queries (g1..... J. we calculate its CS as follows:
n Ae
n E THEN,
Sarrig
relaxation
CS
U
where N₁ is the set of knowledge neurons corresponding to query qs, and n denote the knowledge neuron. INEN; is indicator function, which equals 1 if n belongs to INEN, Thus, Σ InEN, represents the number of times n appears across all KN sets (i.e., .N). In the original metric. Crang the numerator represents the intersection of all N,, meaning KN must appear in all sets to be counted, After relaxation (CSS) the numerator includes any KN that appears in more than one of the Mg sets, allowing it to be counted even if it is not present in every set. This relaxation reduces the impact of localization errors and provides stronger evidence for the existence of K₁-
Then, we use thresholding technique based on CS, classifying facts above certain threshold as Ke (consistent knowledge) and those below it as K, (inconsistent knowledge). We consider two types of thresholds: static threshold and Otsu's threshold While Otsu's threshold aims to maximize the between-class variance and effectively separate two classes of data, the static threshold reflects the inherent nature of fact's adherence (or non-ad non-adherence) to the KL assumption. See Table 4 in A for specific thresholds. To ensure our findings are not method-speche three advanced knowledge localization methods (Dai et al., 2022; Enguehard, 2023; Chen et al., 2024a), with minor modifications for task adaptation, primarily to the method of Enguchard (2023) (detailed in Appendix D). Finally, we apply Welch's 's t-test² to confirm the statistical significance of the difference between Ke and K₁.
we compare
'https://en, wik http:///ewwwiiiiii..... org/ watkk//tssu22s__eeeee Phttps://ee wik http:://ww..iiiidddaa..////////e /wel wwwarshtaaahheeeee
3

---

Published as a conference paper at ICLR 2025
GPT-2
T
Dai et al. (2022)
Euguehard (2023)
Chen et al. (2024a)
Uy
RC
CSc
CS;
t
Rc
CSc
C3:
Re
CSc
R;
CS₁
t
St
0.56
0.21
0.44
0.03
236
0.54
0.23
0.46
0.03
235
0,53
0.25
0.47
0.03
230
0.42
0.41
0.24
0.59
0.06
0.44
0.29
0.55
0.05
0.29
0.60
221
0.53
LLaMA2-7b
T
Dai et al. (2022)
Enguchard (2023)
Chen et al. (2024a)
Uf
RCc
CSc
R2
CS;
8
Rtc
CSc
R1
CS;
Re
CSc
R2
CS₁
t
St
0.40
0.21
0.60
0.04
158
0.39
0.20
0.61
0.04
150
0.40
0.20
0.60
0.04
160
0.55
O
0.21
0.28
0.79
0.062
152
0.20
0.25
0.80
0.07
158
0.24
0.30
0.76
0.06
132
0.70
LLaMA3-8b
T
Dai et al. (2022)
Enguchard (2023)
Chen al. (2024a)
Ur
Re
CSc
Ri
CS;
Ro
CSc
RI
CS;
Re
CSec
Rr
CSi
t
St
0.16
0.16
0.84
0.03
114
0.15
0.18
0.85
0.03
105
0.18
0.19
0.82
0.03
123
0.77
O
0.23
0.14
0.77
0.03
128
0.21
0.15
0.79
0.03
107
0.24
0.16
0.76
0.03
130
0.70
Table 1: Overall results of Consistency Analysis. The symbol T represents the static (St) and Otsu (Ot) thresholds. The t-statistics and p-values are from the T-test, with p 1e
6 in all cases.
GPT-2
KN-Conneeenn Sore
0.8
Threshold 0.3
0.6
0.4
P39
P264
P57
Pla8
P131
P103
P176
P30
P178
P138
P47
P17
P413
F27
P463
P364
p485
8449
P20
Pi376
F3e
P1303
P530
P/9
P190
P740
P136
P127
P1412
P407
PI40
P279
P159
PJ06
Piul
p957
LLaMA2-2
KN KoCooiiieee Scon
Threshold 0.3
2 Co
P39
1264
P37
Plus
P131
P176
P30
P178
PI38
P47
P17
P413
P27
P463
P364
p495
P4d9
120
P1376
P3o
P1300
P530
P19
P/10
P740
P136
P127
2/2
P407
P1-00
P279
P159
P106
Flut
P937
LLaMA3-88
KN Conshtuncy Score
0.6 0.8 1.0
Threshold 0.3
P39
P26d
P37
PIOH
P151
PI0S
P176
P30
P170
P138
P47
P17
P413
P27
1463
p364
p4u5
8449
P20
P1376
P30
P13M3
P530
P19
P190
P740
P156
P127
P1412
P407
P140
P279
P159
P106
P1ol
1957
Figure 3: Violin plot for Consistency Analysis. The a-ax the fact relations, and the y-axis is
the CS2 value. The width of each violin plot indicates the density of data at different CS₂ values. We select threshold of 0.3 as an example, and facts below this threshold are classified as K,,
Regarding the evaluation metrics, we calculate the proportions of Ke and K,, denoted as Re and R₁, respectively. We also compute the average values of CS for these facts, denoted as CSc and CS₁. Furthermore, we calculate the proportion of facts classified as K2 by all three methods, denoted as U2 (i.e., the union of K1).
Findings Figure classifies facts based their respective relations (c.g., P39 represents the "posi- tion" relation), illustrating the distribution CS when utilizing the knowledge localization method proposed by Dai et al. (2022). The violin plots for other methods can be found in Figures 7 and 8 in Appendix C. Together, Figure and Table summarize the overall results.
(1) Inconsistent knowledge (K₂) is widely present across different knowledge localization methods, LLMs, and relations. In Table 1, the consistently high ratio of K, (R,) and low CS values (CS,) demonstrate that the proportion of facts categorized as , is substantial across different methods,
4

---

Published as a conference paper at ICLR 2025
with U2 showcasing high classification agreement among all three knowledge localization methods. For LLaMA3, using a static threshold, 77% of the facts are consistently classified into К₁. More- in Figure 3, using example threshold of 0.3, the majority of facts across various relations fall below this threshold, thus belonging to バ- (2) Statistical tests reveal a significant difference between Ke and K2. For instance, using the static threshold (St) for LLaMA3-8b, the recorded t- statistic is 123, with a p-value less than le - 6. These results reflect a very strong distinction, as the high t-statistic and extremely low p-value show that the difference is highly reliable. Combining (1) and (2), we conclude that inconsistent knowledge (K₃) is prevalent. Beyond statistical analysis, we further validate the existence of KI through knowledge modification experiments.
2.2 MODIFICATION- BASED EVIDENCE FOR THE EXISTENCE OF INCONSISTENT KNOWLEDGE
In this subsection, we conduct knowledge modification experiments to demonstrate the existence of inconsistent knowledge (K7). We use static threshold classify facts into Ke and K₁,
Let (s, r, o) denote fact consisting of subject (s), relation (r), and object (o). We perform two types of knowledge modification: Erasure and Update. Given a fact with & queries {4ir-..,9h, and for a query g. modify the MLP weights of LLMs follows.
Experimental setups
00 Wi.p E(o) + A2E(0'), if Update
if Erasure
Wi.p
where I and p represent the layer and position of the knowledge neuron, Wt.p is the corresponding MLP weight. E(o) and E(o') are the word embeddings of the original object and the updated object o', respectively. A and X2 are hyperparameters. We perform knowledge modification on two different KN sets: (1) Ni the set of knowledge neurons corresponding to q: (2) Na. the union of KNs across all & queries, ie.. No = U.
Evaluation Metrics (1) Knowledge Modification Metrics: We adopt three metrics (detailed in E):
Reliability (Rel), Generalization (Gen), and Locality (Loe) (Yao et al., 2023). These three metries respectively represent the model's 's ability to answer the original query, neighbor queries, and unre- lated queries after knowledge modification. All three metrics are better when higher. To facilitate comparison, we also calculate the average of these three metrics (Avg).
(2) General Capability Metrics: Editing neurons may disrupt the model's performance in generating text (Zhang et al., 2024; Zhao et al., 2023). Similar to other model editing methods (Wang et al., 2024b), we employ the perplexity (PPL) metric to evaluate the model's general capability after modification. Specifically, each time and calculate the relative increase in PPL before (b) and after (a) editing the model: APPI. PPL., PPL,, -PLL, A lower APPI, is better, as indicates less disruption to the model. PPL,
randomly select five entries from WikiText2 (Merity et al., 2017)
Findings Table 2 presents the results of this experiment, leading us to the following conclusions.
(1) Low Generalization in Inconsistent Knowledge in N₁: Modifying Mi, i.e., the KNs corre- sponding to qs, leads to low generalization for K₁. Specifically, under the "Erasure" setting, the generalization scores are only 0.09 for GPT-2 and 0.04 for LLaMA3-8b, indicating unsuccessful modification of neighbor queries. Despite high Reliability and Locality scores on original and unre- lated queries, the poor generalization reveals the limitations of this method. In contrast, Ke exhibits higher "Avg" and "Gen" metrics. For example, for LLaMA3, "Avg" and "Gen" metric values reach 0.47 and 0.30, respectively, suggesting better consistency among neighbor KNs (i.e., KNs corre- sponding to neighbor queries).
(2) High APPL and lower Locality for Inconsistent Knowledge in Ni, To achieve high gen- eralization for K₁, substantial modifications to Ny. (union of N,) are required, necessitating the alteration of many KNs to impact a single fact. However, this approach significantly increases per- plexity change (APPL), with peak of 1.05 for LLaMA3-8b under the "Erasure" setting (i.e., 105% increase in PPL), and causes Locality to drop from 0.80 to 0.50, indicating excessive alter- ations model parameters. It is precisely because the neighbor KNs are inconsistent that taking
5

---

Published as a conference paper at ICLR 2025
Erasure
M1
GPT-2
LLaMA2-7b
LLa
Rel
Gen
Loc
Avg
A PPL
Rel
Gen
Loc
Avg
A PPL
Rel
Gen
Loc
Avg
A PPL
Ke
0.55
0.47
0.93
0.65
0.02
0.33
0.34
0.79
0.49
0.01
0.28
0.30
0.83
0.47
0.04
Ky
0.50
0.09
0.97
0.52
0.06
0.36
0.11
0.80
0.42
0.03
0.34
0.04
0,90
0.43
Erasure
N,
GPT-2
LLaMA2-bb
LLaMA3-8b
Rel
Gen
Loc
Avg
A PPL.
Rel
Gen
L.cc
Avg
A PPL.
Rel
Gen
L.cc
Avg
A PPL
Ke
0.58
0.55
0.90
0.68
0.12
0.30
0.55
0.70
0.52
0.08
0.30
0.35
0.80
0.48
0.18
K2
0.65
0.60
0.70
0.65
2.02
0.44
0.45
0.52
0.42
1.50
0.36
0.40
0.50
0.49
1.05
Update
M
GPT-2
LAaMA2-7b
LLaMA3-8b
Rel
Gen
Loce
Avg
A PPI.
Gen
L.ce
Avg
A PPL.
Gen
L.ce
Avg
A PPI.
Ko
0.40
0.99
0.64
0.04
0.39
0.89
0.53
0.29
0.79
0.46
K7
0.44
0.11
0.96
0.50
0.09
0.39
0.07
0.80
0.42
0.08
0.29
0.08
0.86
0.41
0.08
Updote
N.
GPT-2
LLaMA2-7b
LLaMA3-8b
Rel
Gen
Loc
Avg
A PPL
Rel
Gen
Loc
Avg
A PPL
Rel
Gen
Loc
Avg
A PPL
Ke
0,56
0.48
0.88
0.64
0.13
0.32
0.59
0.82
0.68
0.10
0.44
0.41
0.74
0.53
0.22
Kr
0.54
0.55
0.74
0.61
1.88
0.40
0.43
0.62
0.48
1.16
0.29
0.33
0.66
0.43
0.93
Table 2: Results of Knowledge Modification. NG and Nr are the two sets of knowledge neurons. The bolded results indicate poor performance, reflecting Failures in model editing.
the intersection (i.e., NWw) results a large number of neurons. This observation suggests that facts related K, cannot be localized to fixed set of KNs. Together, findings (1) and (2) confirm that Ky does not adhere to the KI. assumption.
Let's review Q1, we have demonstrated, from both statistical and model editing perspectives, that the knowledge localization assumption does not always hold. Next, we explore a more realistic alternative.
3 QUERY LOCALIZATION ASSUMPTION
Motivation Since inconsistent knowledge (Ky) does not satisfy the KL assumption, we naturally raise the question Q2: What is more realistic assumption? Let's revisit the two limitations with the knowledge localization assumption: (1) Knowledge neurons correspond to facts, meaning that a fact is statically stored by several KNs. However, Tables I and 2 suggest that this assumption does not hold for K₁- (2) The focus has been solely on the MLP module, neglecting the attention module. In light of recent research on the attention module (Ren et al., 2024; Geva et al., 2023; Meng al., 2022), we argue that it should also be considered. Below, we will explain our two findings that address these limitations, including Query-KN Mapping ($3.1) and Dynamic KN Selection ($3.2).
3.1 QUERY-KN MAPPING
Method In order to explore the relationship between queries and knowledge neurons, we manip- ulate the KN activation values by either suppressing or enhancing them. As before, we first classify facts into inconsistent knowledge (Kr) r) and consistent knowledge (Kc). Then, given fact with k queries {g......g ).)) and for a specific query 9s, we manipulate five different sets of neurons and study how such operations affect the model's response to the query:
(1) Self: Equivalent to N₁, the set of KNs corresponding to qi (2) Union: The union of other neighbor KNs, i.., the union of KNs corresponding the neighbor queries. (3) Intersection: The intersection of other neighbor KNs. (4) Refine: Refined neighbor KNs, the set of KNs that appear more than once. (5) Unrelated: Randomly selected unrelated neurons, equal in number to N,
Regarding evaluation metrics, we follow other knowledge localization methods (Dai et al., 2022; Chen al., 2024a;b), and calculate the rates of increase and decrease in the LLMSs answer proba-

---

Published as a conference paper at ICLR 2025
Union
Intersection
Refine
Unrelated
Kc
GPT-2
LLaMA3-80
Froo
0.2
Enh
Sup
Enih
Avg
Enh
Sup
K7
GPT-2
LLaMA2-79
LLMMA3-80
Prob
Figure 4: Results of Query-KN Mapping. "Enh" and "Sup" refer to enhancement and suppression of KN activation values, respectively, with "Avg" representing their average.
bilities before (b) and after suppression and enhancement: AProb + Proh, Proby Prob, Here, "+"
indicates that we assign negative value for suppression and a positive value for enhancement.
Findings Figure 4 illustrates our results, leading to three findings. (1) Regardless of whether it inconsistent knowledge (K₃) and consistent knowledge (Kc), results from the NG indicate that the influence of suppressing enhancing the query's own KNs is significant. This suggests a strong association between the KNs and the query. (2) For K₁, the average values ("Avg") of other baselines considerably lower than those of MG, particularly in the "Intersection" case. In contrast, for Kc, the average values decrease to a lesser extent. This indicates that the neighbor KNs for Ke more consistent, while the neighbor KNs for K₁ exhibit much less consistency. This demonstrates that for Kr, the KNs do not correspond to the fact itself.
Combining (1) and (2), conclude the phenomenon of Query-KN Mapping: For inconsistent knowledge, the localization results are associated with the query rather than the fact.
(3) Furthermore, for Kc, the values also decrease. This is because use very low threshold when classifying facts to rigorously demonstrate the presence of K₁. As result, some facts in Ke may be entirely consistent, which actually strengthens our conclusion by confirming that K₁ exists. Therefore, Ke can be considered a special case of K₁.
3.2 DYNAMIC KN SELECTION
To address the issue of the knowledge localization assumption neglecting the attention module, employ method similar to manipulating neuron values by suppressing enhancing attention scores, thereby exploring their effects. Notably, attention score matrices resemble KN acti- vation value matrices, differing only in additional dimension for attention heads. Thus, similar to the definition of knowledge neurons, we identify column vectors with high attention scores. Draw- ing inspiration from cognitive science (Dalva al., 2007; Kim et al., 2018; Lisman al., 2018: Harikesh et al., 2022: Rabinowitch et al., 2024), we refer to these vectors as Knowledge Synapses (KS), denoted as S. Given query with its answer, the KSs are defined as:
Methods
L
H n C
T=Q C. H
ΣΣΣ>
(r.c) (1,b)
(3)
n (nc) 400 (L.)) > T, VIe (he feeef.... Hl...,,,.cE
s= 5={Cal.n
(c.l.))
(1.... (I....,c}} o3 .
(4)
where 7 the dynamic threshold, a a scaling factor, and A is the attention score matrix. L and H represent the number of layers and heads of the attention module, respectively, while R and C' are the rows and columns of I, h, r, e are the corresponding indices. After localizing S, we enhance suppress the attention scores these positions (i.e., KS attention scores) study their effects.

---

Published as a conference paper at ICLR 2025
Sucaman 1, 1isu
Kg
Fact
Sateman wise postoon
Clnteapir Anitnas postiton mayoe
K; Fact,
Cirtaqpa Aatiaa, was
mayer
Inashorteer Layes
3211
230 301
700
Figure 5: Heatmaps showing the neuron activation values, after suppressing knowledge synapses. The queries used here are the same as those in Figure 1. The dark areas in Figure appear lighter here, indicating a decrease the activation value of knowledge neurons. For the enhanced case, see Figure 10 in Appendix C.
Kc
GPT-2
LLaMA2-7b
KN
LLaMA3-Bb
Kc
GPF2
LLaMA2
LLaMA3
Value
0.4
7
77
Enhh
Sup
Avg
Enh
Sup
Avg
Enh
Sup
Avg
Enh
Sup
Avg
Ki
LLaMA3-9b
0.6
0.6
Value
0.4
0.2
0.2
17
7
V
0.0
Sup
Avg
Sup
Aug
Sup
Avg
Sup
Avg
Figure 6: Results of Dynamic KN Selection. "Avg" represents the average between the "Enh" and "Sup" settings.
Evaluation Metrics and Baselines (1) We assess the impact of suppressing (Sup) or enhancing (Enh) knowledge synapses on the activation values of knowledge neurons. We calculate the rates of increase and decrease in the average KN activation values before (b) and after (a) KS manipulation: (2) We assess the impact of KSs on knowledge expression by computing the change in the L.MSs* answer probability (AProb).
A Value + Value, - Values
Values
To further demonstrate the "selection" role of the attention module, we compare the changes in values of neurons from two other sets: (1) neighbor KNs, i.e., the knowledge neurons corresponding to the neighbor queries, and (2) non-Ko MNNSss, i.e., randomly selected non-knowledge neurons.
Findings Figure illustrates a case, while Figure 6 presents the overall results, revealing threc key phenomena: (1) Manipulating KSs leads to a higher Value for knowledge neurons and also has a noticeable impact on AProb (i.e., the answer probability). (2) When manipulating KSs, whether they are neighbor KNs or non-KNs, the corresponding KN-values do show significant changes. (3) Manipulating knowledge synapses significantly affects both Ke and Kj.
Combined with Figure we conclude that suppressing attention scores significantly decreases KN- values, while the value changes in other neurons are relatively minor, regardless of the knowledge category. This hinders the model's 's ability to select appropriate KNs for accurate knowledge expres- sion, resulting in decrease in LLMs' answer probabilities. We summarize this phenomenon as Dynamic KN Selection: the attention module plays a role in selecting specific KNs for express- ing knowledge.
(4) Additionally, under the "Enh" setting, AProb is significantly lower than under "Sup". This is because, without suppressing KSs, the attention module is already capable of selecting KNs. Further enhancement leads to "saturation" effect, where the accuracy of KN selection reaches its limit. This further proves that the attention module plays selective role rather than storage role.
8

---

Published as a conference paper at ICLR 2025
Erasure
Method
GPT-2
LLaMA2-7b
LLaMA3-Sb
Rel
Gen
Loc
Avg
A PPI.
Rel
Gen
Loc
Avg
PPL.
Rel
Gen
Loc
Avg
A PPL
Ke (N))
0,55
0.47
0,92
0.64
0.02
0.33
0.34
0.79
0.49
0.01
0.28
0.30
0.83
0.47
0.04
Ke (N₂)
0.58
0.55
0,90
0.68
0.12
0.30
0.55
0.70
0.52
0.08
0.30
0.35
0.81
0.49
0.18
Ke (Ours)
0.50
0.90
0.65
0.03
0.03
0.02
Ky (N))
0.50
0.09
0.97
0.52
0.06
0.36
0.80
0.42
0.03
0.34
0.04
0.90
0.43
0.05
Ki (N.)
0.65
0.60
0.70
0.65
2.02
0.44
0.45
0.52
0.47
1.50
0,36
0.40
0.50
0.42
1.05
Ky (Ours)
0.55
0.40
0.90
0.62
0.10
0.35
0.35
0.77
0.49
0.06
0.34
0.30
0.88
0.51
0.09
Update
Method
GPT-2
LLaMAA-7b
LLaMA3-8b
Rel
Gen
Loc
Avg
PPL
Rel
Gen
Loc
Avg
PPL
Rel
Gen
Loc
Avg
PPL
Ko (N.)
0.53
0.40
0.99
0.04
0.03
0.07
Kc (N,)
0.56
0.48
0.88
0.64
0.13
0.32
0.59
0.82
0.68
0.10
0.44
0.41
0.74
0.53
0.22
Ke (Ours)
0.55
0.44
0.98
0.66
0.14
0.30
0.50
0.88
0.56
0,10
0.35
0.33
0.70
0.46
0.10
K, (N))
0.44
0.11
0,96
0.50
0.09
0.39
0.07
0.80
0.42
0.08
0.29
0.08
0.86
0.41
0.08
Ky (N.)
0.54
0.55
0.74
0.61
1.88
0.40
0.43
0.62
0.48
1.16
0.29
0.33
0.66
0.43
0.93
Ky (Ours)
0.45
0.45
0.88
0.59
0.20
0.40
0.38
0.75
0.51
0.12
0.29
0.29
0.80
0,46
0.14
Table 3: Results of Consistency- the second-best results are indicated with underline. Align with Table 2, the higher the "Avg" the better, the lower the "A PPL. the better.
KN Modification. The best results are indicated in bold, and
Let's review Q2, and our two findings address the two limitations of the KI assumption. We summarize our findings to establish a more realistic Query Localization assumption, which includes Query-KN Mapping and Dynamic KN Selection.
3.3 APPLICATION OF QL ASSUMPTION: CONSISTENCY-AWARY KN MODIFICATION
Experimental Setups Inspired by query-KN mapping, propose new approach select knowledge neurons that improves knowledge modification methods. By incorporating KN consis- tency, we introduce the (CAS) metric, which penalizes low consistency and rewards high activation values. Given a fact with k queries {q1,..., [k). (e)... for query 4s, the CAS for the p-th neuron in the 1-th layer is defined as:
(LS( = B1p B201p3 where Pip
Clp
(8) asip pup)
(5)
1=1
(t)
where 31 and 32 hyperparameters, Hip and Tip represent the mean and variance, and as is the activation score at position (I,p) for q:- Then, using thresholding techniques, we identify positions with high CAS values as the knowledge neurons to be edited. We conduct experiments similar to those in $2.2, using the same metrics. The baselines are the two methods: Nr, which represents the knowledge neurons for 4, and Ng. the union of KNs for these & queries, i.e, the union of N₁ Results are summarized in Table
Findings (1) Better performance of Ky: Our method demonstrates superior and more balanced performance for Kj. For instance, under the Erasure setting, the average value of the model editing metric for LLaMA3 reaches 0.51 (under K2 (Ours) setting), with A PPL at 0.09, indicating suc- cessful editing with minimal damage to the model. In contrast, the original methods show either lower average value (0.42 for M,) or a higher A PPL. (1.05 for No),, suggesting that they struggle to achieve successful editing without compromising the model's general capabilities.
(2) Effectiveness for Kc: Our approach equally effective for Kc. The performance of Ke using our method is comparable to both Kc from Nú and N,,. For instance, under the Erasure setting, the average values for the three groups for LLaMA3 are 0.47, 0.49, and 0.48, with A PPL values of 0.04, 0.18. and 0.02, respectively. This suggests that even facts satisfying the KL assumption (i.e., Kc) can be effectively analyzed under the QL assumption, highlighting the limitations of the KL assumption and showing that it is merely simplification of the QL assumption.

---

Published as a conference paper at ICLR 2025
4 RELATED WORK
LLMs store extensive factual knowledge (Petroni et al., 2019; Cao et al., 2024; Wang et al., 2023; Kale et al., 2023; Li al., 2023). prompting numerous studies to investigate the mechanisms be- hind their storage and expression. Geva et al. (2021) propose that MIP modules simulate key-value memories to store information, and Dai et al. (2022) propose the concept of knowledge neurons (KNs), suggesting that these MLP modules can store "knowledge". The success of KN-inspired model editing methods (Dai et al., 2022; Meng et al., 2022; 2023) further supports the plausibility of the KN thesis. Additionally, the integrated gradients (IG) method (Sundararajan et al., 2017) has proven suitable for knowledge localization (Lundström et al., 2022), leading to further refinements such as Sequential IG, Discretized IG and the Architecture adapted Multilingual IG (Enguehard, 2023; Chen et al., 2024a; Miglani et al., 2020; Sanyal & Ren, 2021; Sikdar et al., 2021). Further in- vestigations reveal that some KNs exhibit cross-lingual features (Chen et al., 2024a; Qi al., 2023; Xie et al., 2022; Zhao et al., 2024a), while others display language-specific characteristics (Tang al., 2024; Kojima et al., 2024; Zhao et al., 2024b). Some KNs also show degeneracy, with multi- ple neurons redundantly encoding the same factual knowledge (Chen et al., 2024b). These studies collectively advance the KN thesis. Beyond MLP modules, some studies incorporate attention mod- ules (Vaswani et al., 2017) into factual knowledge research. They find that attention modules play a role in the LLMs' internal information flow, aiding in factual knowledge extraction (Geva et al., 2023). Moreover, attention modules can facilitate in-context learning (Ren et al., 2024) and relate to token expression (Meng et al., 2022).
However, the KN thesis has its limitations. Niu et al. (2024) argue that it oversimplifies the real situation, while Hase et al. (2023) suggest that the location of knowledge localization may not align with the location of greatest impact on knowledge expression. Additionally, Anthropic (2023) find the activation of a single neuron can have different meanings in different contexts Limitations in KN-inspired knowledge editing methods have also been identified (Li et al., 2024; Yao et al., 2023; Cohen et al., 2023; bheescher-Oberaaier et al., 2023; Wang et al., 2024b; Pinter & Elhadad, 2023: Hua et al., 2024: Zhao al., 2023; Hu et al., 2024; Wang et al., 2024a). These model editing methods may fail to edit successfully or impair the LLMs' general capabilities, indirectly suggesting limitations with the KN thesis. Some theories now move away from using neurons as the basic research unit. Yao et al. (2024) expand the concept of knowledge neurons into knowledge circuits, considering both the attention module and the MLP module together. Bricken et al. (2023) discover that neurons can be further decomposed into features, and these features offer better interpretability. Nevertheless, selecting neurons as the research unit remains meaningful, as neurons are the most natural unit of study and allow for casier verification and application. Previous work either points out the problems in the KN thesis without exploring the underlying causes or potential solutions, or it abandons the KN thesis altogether. Our work is different from theirs. Based on KN thesis, we analyze its limitations and propose effective improvements.
5 CONCLUSION AND FUTURE WORK
This paper investigates the knowledge localization assumption of the knowledge neuron thesis, which posits that fact can be localized to several knowledge neurons. We first demonstrate the limitations of the KL assumption and confirm that many facts do not conform to it. Furthermore, through extensive experiments, we obtain two findings: Query-KN Mapping and Dynamic KN Se- lection, which together form the Query Localization assumption. We argue that the KL assumption merely simplification of the QL assumption. Finally, we apply the QL assumption in model editing experiments and find that our approach can be used for model editing, further validating our new assumption.
Future work could delve into the reasons behind the existence of K,, We speculate that this may be related to the pre-training process, where some facts that are well mastered by LLMs might belong to consistent knowledge (Kc). Moreover, exploring how to reconcile the QL assumption with other current theories is also worth investigating. Additionally, it may be possible to further utilize the QL assumption to improve model editing methods. Currently, our work primarily leverages the findings from the query-KI -NN mapping aspect of the QL assumption. By integrating the attention module more effectively, we could develop enhanced methods for dynamically editing knowledge in L.Mss.
10

---

Published as a conference paper at ICLR 2025
6 ETHICS AND REPRODUCIBILITY STATEMENTS
Ethics Statement conducting this research, we have taken several ethical considerations into account to ensure the responsible and dissemination of our findings. First, our study utilizes publicly available large language models (LLMs) and standard datasets, which comply with exist- ing data privacy and usage policies. We have employed any proprietary sensitive data that could compromise individual privacy or violate data protection regulations. Second, we acknowl- edge the potential for biases inherent in LLMs, which may be reflected in our analysis of knowl- edge neurons. Additionally, our proposed Query Localization (QL) assumption and the associated Consistency-Awan KN modification method aim to enhance the transparency and interpretability of L.Mss, thereby contributing to more equitable and accountable Al systems. We have disclosed any potential conflicts of interest and ensured that our research adheres to the highest standards of research integrity, including obtaining necessary approvals from institutional review boards (IRBs) where applicable. Finally, we are committed to responsible code and data release practices, ensuring that all shared resources free from malicious content and do not facilitate harmful applications.
Reproducibility Statement We have made extensive efforts to ensure that our research is fully reproducible. Detailed descriptions of experimental setup, including the hyperparameter set- tings and data preprocessing steps, are provided in the main text and the appendix. Additionally, we include all data and code we used in the supplementary materials, and the code will be made public after it is compiled. Moreover, all datasets employed in our experiments are either publicly accessi- ble will be shared under appropriate licenses to ensure legal compliance. We have also included scripts for data processing and model evaluation to streamline the reproduction of our results. By providing these resources and detailed documentation, we aim to support other researchers in veri- fying and building upon our work, thereby fostering transparency and collaborative advancement in the field of natural language processing.
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (No. U24A20335) and Beijing Natural Science Foundation (L243306). This work is supported by the National Natural Science Foundation of China (No. 62176257, No. 62406321). This work is also supported by the Youth Innovation Promotion Association CAS and the China Postdoctoral Science Foundation under Grant Number 2024M753500.
REFERENCES
Anthropic. Distributed representations: Composition superposition. Transformer Circuits Thread, 2023. URL tttsss//ranffoome-ciiiee wwt.irrriii.tuuuuuuooee pub/2023/
superposition- compon-coooitttooooo titotindeeenne/. html.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodss, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. URL https: pub/2 2023 mwwosoommttiii.cmmmmmcmmm features/ index. html.
Boxi Cao, Hongyu Lin, Xianpei Han, and Le Sun. The life cycle of knowledge in big lan- guage models: A survey. Machine Intelligence Research, 21(2):217-238, 2024. ISSN 2731- 538X. doi: 10.1007/s116333-023- 1416-x. URL https: article/doi//i wtt.:///...100/s11111
mi- http://wwwww.ieeeerr...eeeee. 1007 111111111633-0022111-----
Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Journey to the center of the knowledge neurons: Discoveries of iaaguage-indpendden knowledge neurons and degenerate knowledge neurons. In Proceedings of the AAAÍ Conference Artificial Intelligence, vol- ume 38, pp. 17817-17825, 2024a. URL https:/ /ojs aaai, .org/ iindiiinxx. php/AAAI/ /AAI/ article/view/2291
11

---

Published as a conference paper at ICLR 2025
Yuheng Chen, Pengfei Cao, Yubo Chen, Yining Wang, Shengping Liu, Kang Liu, and Jun Zhao. The da vinci code of large pre-trained language models: Deciphering degenerate knowledge neurons, 2024b. URL https: http:///axxvvvooo 2ttp://bb//222..3332
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. ArXiv preprint, abs/2307.12976, 2023. URL https: //aaxiiv.oo org abs/ 2207 003300120 12976.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neu- rons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pp. 8493-8502, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10. 16633///2222.ac-oonn...88. URL https: /ttpss//acee httpe///aaaatthhhh
org/2022 acl- 581.
Matthew B Dalva, Andrew C McClelland, and Matthew S Kayser. Cell adhesion molecules: sig- nalling functions the synapse. Nature Reviews Neuroscience, 8(3):206-220, 2007. URL. https://
httt:s://www.naturet nature. www.:natwee....aa.oooooe.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language mod- els. Transactions of the Association for Computational Linguistics, 9:1012-1033 2021. doi: 10.1162/tacl 00410. URL https: https://aclaathollaee wwwpl/ooo.oorg///////e worg//0220 www.-l.cll......... 60.
Joseph Enguehard. Sequential integrated gradients: simple but effective method for explaining language models. ArXiv preprint, abs/2305. 15853, 2023. URL 2305. 15853.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10. 10.11853//1/20 http2222...mmmmaaiia
URL
org.oog/////// 202l emnlp 100-mainn i 446.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: /openreview....mm net/ forum?ide
F1G7794K02.
Padinhare Cholakkal Harikesh, Chi-Yuan Yang, Deyu Tu, Jennifer Y Gerasimov, Abdul Manan Dar, Adam Armada-Moreira, Matteo Massetti, Renee Kroon, David Bliman, Roger Ols- son, et al. Organic electrochemical neurons and synapses with ion mediated spiking. Na- ture communications, 13(1):901, 2022. URL https: http::/wwww.atttt com/articces///////ccooo.. s s41467 022-6777--222-- 28483- 3- 6#citeas.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization knowledge editing in language models. In Seirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //ooenervviww... wwt//foouu forum? d= dbUl Ztbd.
Jason Hoelscher-Obermaice Julia Persson, Esben Kran, Ioannis Konstas, and Fazl Barez. Detecting edit failures in large language models: An improved specificity benchmark. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 11548-1155 15559, Toronto, Canada, 2023. Association for Computational Linguistics. doi: 10. 00.18653/v1/2023 http://222....n.ss-./ee findings-acl.733.. URL https: https////aaaaaahalaaacaeo .org/ 2023. 3. f findings acl. 733.
Chenhui Hu, Pengfei Cao. Yubo Chen, Kang Liu, and Jun Zhao. Knowledge in super- position: Unveiling the failures of lifelong knowledge editing for large language mod- els. ArXiv, 00o/2408.00413.. 2024. URL https:/ /api. semanticscholl org/ CorpusTD:271B6588
12

---

Published as a conference paper at ICLR 2025
Wenyue Hua, Jiang Guo, Mingwen Dong, Henghui Zhu, Patrick Ng, and Zhiguo Wang. Propaga- tion and pitfalls: Reasoning-basee assessment of knowledge editing through counterfactual tasks. arXiv preprint arXiv:2401.175555 2024. URL https: arriiv httg/abb/240110 17585.
Amruta Kale, Tin Nguyen, Jr. Harris, Frederick C., Chenhao Li, Jiyin Zhang, and Xiaogang Ma. Provenance documentation to enable explainable and trustworthy ai: A literature review. Data Intelligence, 5(1):139-162, 03 2023. ISSN 2641-435X.
doi: 10.1162/dit.a..00119. URL
Seungjoon Kim, Hyeonho Kim, and Ji Won Um. Synapse development organized by neuronal activity-regulated immediate-early genes. Experimental & molecular medicine, 50(4):1-7, 2018.
URL wwt.h:tppp/.........m /www. nwwura.uuuu... www.uuree.comeemmmmm Sertiillee//r20001
Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo. On the mul- tilingual ability of decoder-basee based pre-trained language models: Finding and controlling language- specific neurons, 2024. URL https: /ttp//aavvvv////////av//eee//aaa org/ abs 24404 02431.
Linhan Li, Huaping Zhang, Chunjin Li, Haowen You, and Wenyao Cui. Evaluation chatgpt for chinese language understanding. Data Intelligence, 5(4):885-903, 11 2023. ISSN 2641 -435X.35X. doi: 10. 10.1162//int
URL https: happs://dei..o /tc.//ciiiao// http://i..orrr/////// http:////..1////ntt_a____
Zhoubo Li, Ningyu Zhang, Yunzhi Yao, Mengru Wang, Xi Chen, and Huajun Chen. Unveiling the pitfalls of knowledge editing for large language models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: httpe://ppeeevvieeee ew. net. forum? d= ENktD3ibl6.
John Lisman, Katherine Cooper, Megha Schgal, and Alcino J Silva. Memory formation depends both synapse-specific modifications of synaptic strength and cell-specific increases in citability. Nature neuroscience, 21 21(3)3009-314 2018. URL https: /wwww//www/..////////////..// com
articles/s41593-1 0-0066-----000 (10776666citass
Daniel Lundström, Tianjian Huang, and Meisam Razaviyayn. A rigorous study of integrated gradi- ents method and extensions to internal neuron attributions. In Kamalika Chaudhuri. Stefanic Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Con- ference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, vol- ume 162 of Proceedings of Machine Learning Research, pp. 14485-14500 PMLR, 2022. URL https: https::/proceediii proceedings ml mlr.press/ /vresssvvll 62. httplunddttro2222 html.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual asso- ciations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: oppereviee.ee.e net/forumd h6wA5A566F44
Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representationss 2023. URL https:/ httpas:///ppeeeieeeeeee net/ www.mmmiiiioo.ooo..o..oo.... wtt.::eeeoor......oo.ca.tttAa M
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. OpenReview.net, 2017. URL https: / //openreview. net. fortt/ffrummaa Bommmiid-Byyy B72udddyyyyyedeee..... 72udxe.
MetaAI. Introducing meta llama 3: The most capable openly available 1lm to date, 2024. URL https: meta. www/blog///oooooooooooooo..oooco met aaaa-aaaaaaaaa 3/.
Vivek Miglani, Narine Kokhlikyan, Bilal Alsallakh, Miguel Martin, and Orion Reblitz-Richardson Investigating saturation effects in integrated gradients. ArXiv preprint, abs/2010. 12697, 2020.
URL https:: http:://axxvvvvi org/ 20ss/210.... 22697.2697......
Jingcheng Niu, Andrew Liu, Zining Zhu, and Gerald Penn. What does the knowledge neuron thesis have to do with knowledge? In The Twelfth International Conference on Learning Representa- fions, 2024. URL pectpss//openreet net wwtttooumfiddeeeeeeeeee fot.mmmid-2HH
13

---

Published as a conference paper at ICLR 2025
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Con- ference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP--JCNIPP) pp. 2463-2473, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10. 1211865//1/D11-1222 URL https: www.llaantnnoo..ggggoooeo 1250.
Yuval Pinter and Michael Elhadad. Emptying the ocean with spoon: Should we edit models? In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL.
https:/ wwwpeeeeeeeeeeeeeeeeeeeeeeeeeeee net forum? d==2FVVTTGGGO
Jirui Qi, Raquel Fernández, and Arianna Bisazza. Cross-lingual consistency of factual knowledge in multilingual language models. In The 2023 Conference on Empirical Methods Natural Lan- guage Processing, 2023. URL https: / openreview net. L
Ithai Rabinowitch, Daniel A Colón-Ramos, and Michael Krieg. Understanding neural circuit function through synaptic engineering. Nature Reviews Neuroscience, pp. 1-9, 2024. URL https: www nature cowmarticless..eeeecc.cooeoooooe 3/////4118002--0000
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019. URL langnagg--emmdeeeeee larguage__odee are wwwprreeensuusrsiiiieom wwwpp/ppeeeeeieemmmeeettatttttea sts
https:// https//////cccffkkyrrrwr cloudfront. net http//btteerreeeeee hetet///eeeeeeeeeeeeee-e--ee
Jie Ren, Qipeng Guo, Hang Yan, Dongrui Liu, Xipeng Qiu, and Dahua Lin. Identifying semantic induction heads to understand co-tentext learning. ArXiv preprint, abs/2402.13055, 2024. URL
https://arxet
org/ abs 2402. 13055.
Soumya Sanyal and Xiang Ren. Discretized integrated gradients for explaining language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10285-10299, Online and Punta Cana, Dominican Republic, 2021. Association for Computa- tional Linguistics. doi: 10. 12263//w/2022.emm/m main. htt: htt.tppss//clllathoooog. org emn 1p-mapppp--mmm ma n. 805.
Sandipan Sikdar, Parantapa Bhattacharya, and Kieran Heese. Integrated directional gradients: Fea- ture interaction attribution for neural NLP models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 865-878, Online, 2021. As- sociation for Computational Linguistics. doi: 10. 00.16633/v12021 acll-long.7.. URL https:
//aa///aaa //aclanthology wttg//22l... 221 acl-1ong.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICMI. 20/7, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 3319-333 3328. PMLR, 2017. URL http: http:////occediii ngs. mlr. milr.press/ve poprss/vv0// hwtp/////nnaaaaaaaaanaaa html........
Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang. Xin Zhao, Furu Wei, and Ji-Rong Wen. Language-specific neurons: The key multilingual capabilities in large language models. ArXiv preprint, abs/2402.16443 2024. URL https: //tpsi///rvvvv/a/ org/ abs/ 2402. 16438..
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fermandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshor, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuming Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian. Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen
14

---

Published as a conference paper at ICLR 2025
Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https:: http:::://axxi.x.o/o/ org/
wwr.oo////aaoooo /ttgg/////////// /ab//////// /2007.200222 09288.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017. URL https: http:///s//pppcceesssssss new.iis.......p.ppo cc. wttppc/pppppppeeeeeeeee /hash/ 4
Abstaa-bsssstttt html.
Chenhao Wang. Pengfei Cao, Zhuoran Jin, Yubo Chen, Daojian Zeng, Kang Liu, and Jun Zhao. MULFE: multi-level benchmark for free text model editing. In Lun-We Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Pupers), pp. 13570-13587, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18653/v 1/200//.112222...1l./1l 732. URL https: http:////caaaattttaaoeee org/ 201-4.acl-oong 732/-
Jianchen Wang, Zhouhong Gu, Zhuozhi Xiong, Hongwei Feng, and Yanghua Xiao. The missing piece in model editing: A deep dive into the hidden damage brought by model editing. ArXiv preprint, (bs/2403.00825, 2024b. URL https://axxiv.. wwwp:iiioogg//aaoe abs //w.:///////.....////// 07825.
Xiao Wang, Guangyao Chen, Guangwu Qian, Pengeheng Gao, Xiao- Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: A comprehen- sive survey. Machine Intelligence Research, 20(4):447-482, 2023. ISSN 2731-538X. doi:
10.100775111333--221111444 URL https:/ //tp:://wwww//// htww//wwii......mmm gesearch.nneteeme aaticcll 1tt.110.100//s1111
1410-8.
Yifei Wang, Yuheng Chen, Wanting Wen, Yu Sheng, Linjing Li, and Daniel Dajun Zeng. Unveil- ing factual recall behaviors of large language models through knowledge neurons. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings the 2024 Conference Em- pirical Methods in Natural Language Processing, pp. 7388-7487 -402, Miami, Florida, USA, Novem- ber 2024c. Association for Computational Linguisties. URL https: hatpss//aaaaatthologo.o
org/2024 2024 emnlp-a caannlppmmann maw.. 420.
Zhihui Xie, Handong Zhao, Tong Yu, and Shuai Li. Discovering low-rank subspaces for language- agnostic multilingual representations. In Proceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pp. 5617-5633, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. URL https: /aaaaaaaaanttooooyooooeaaoee www.anthoooooooo/.2222..2.2. emnlp-main. 379.
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 10222-10240, Singapore, 2023. Association for Computational Linguistics. doi: 10. 1653/v 1e1o223.mmlp-mainn. https: http://acaanthhooocoo
URL
.rgg 2023. emnlp-main. 632.
Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang, Ziwen Xu, Shumin Deng, and Huajun Chen. Knowledge circuits pretrained transformers, 2024. URL https::/ https://aaxxvvorrr abs/2405. 17969.
Zhihao Zhang, Jun Zhao, Qi Zhang, Tao Gui, and Xuanjing Huang. Unveiling linguistic regions in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers). pp. guistics. doi: acl-lone 338.
Bangkok, Thailand, August 2024. Association for Computational Lin- 1/033//1/2224.clllooooo
URL https: aacaaaaaannttoooooo org/2024.
Jun Zhao, Zhihao Zhang, Yide Ma, Qi Zhang, Tao Gui, Luhui Gao, and Xuanjing Huang. Unveiling a core linguistic region in large language models, 2023. URL https: //te://arxxa //arxiv... orgg abs/ 231 0. 14.1142. 14928.
15

---

Published as a conference paper at ICLR 2025
Xin Zhao, Naoki Yoshinaga, and Daisuke Oba. Tracing the roots of facts in multilingual language models: Independent, shared, and transferred knowledge, 2024a. URL https: //rp/varxivvvvvv org/abs/240) 05189.
Yiran Zhao, Wenxuan Zhang. Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. How do large language models handle multilingualimm? ArXiv preprint, abs/2402. 18815, 2024b. URL https: / //aaxivv. /attii..rg/ass//2000000 18815.
A SPECIFIC EXPERIMENTAL SETTINGS
Hardware spcification and environment. the following specifications:
We ran experiments the machine equipped with
CPU: Inte Xeon(R) CPU E3-2680 v4 2.40GHz, Total CPUs:
* GPU:
NVIDIA GeForee RTX 3090
20. The Standard Memory Config is 24 GB
GDDR6X.
NVIIIA A100 80GB PCIe x 4. The GPU Memory is 80GB HBM2e.
Software:
- Python Version: 3.10.10
PyTorch Version: 2.0.0tcul17
In the experiments, the main computational expense was associated with acquiring knowledge neu- rons since the method for knowledge localization computes the activation values of all neurons. For GPT-2. LLaMA2-7b. and LLaMA3-8b, the time required to acquire KNs once approximately: 20 seconds, 5 minutes, and 5 minutes, respectively. As we conducted our experiments using multi- GPU distributed processing, the total time spent
about 26 days. The computational expense for the other experiments was not significant, and they could be completed within 10 days. Due to the lengthy computation times, we tested the code and results by selecting one datum from each relation, thus the test dataset comprised only 36 data points. We did conduct some erroneous experiments, but the run run-time costs for these were negligible due the small size of the dataset. We recommend that readers adopt a similar approach for testing their code.
Experimental Hyperparameters of Consistency Analysis We provide the thresholds used for each setting in Table 4, corresponding to Table 1. The Otsu threshold calculated separately based on each batch of data, thus each is different. The static threshold is set by us, thus it is the same.
Experimental Hyperparameters of KN Modification In Equation 2, set A A2
Experimental Hyperparameters of Obtaining Knowledge Synapses In Equations and 4, the scaling factor is the same for all three PLMs, with 5 = 0.3.
Experimental H Hyperparameters of Consistency-Awwer KN Modification In Equation 5, set 31 = 0.7 and B2 = 0.3. For the selection of threshold, we consider the dynamic threshold to find the maximum value of CAS, and neurons larger than 0.3 times selected as KNs.
B EXPERIMENTAL DATASET INTRODUCTION
In experiments, we selected the ParaRel dataset Elazar et al. (2021), high-quality resource of cloze-style query English paraphrases. It contains total 328 paraphrases for 38 relations. We further conducted basic filtering, excluding relations that had paraphrases, resulting in a substantial dataset of 27,610 entries across 36 relations. Table displays these relations and corresponding example data.
16

---

Published as a conference paper at ICLR 2025
GPT2
Dai al. (2022)
Enguehard (2023)
Chen et al. (2024a)
Static Otsu
0.1
0.1
0,1
0.146
0.150
0.148
ILaMA2-7b
T
Dai al. (2022)
Enguehard (2023)
Chen al. (2024a)
Static Otsu
0.1
0.1
0.1
0.170
0.169
0.170
LLaMA3-8b
T
Dai al. (2022)
Enguehard (2023)
Chen (2024a)
Static Otsu
0.1
0.1
0.1
0,080
0.082
0.081
Table 4: This table corresponds to Table 1 and lists the thresholds for each experimental setting.
C SUPPLEMENTARY EXPERIMENTAL RESULTS
C.I STATISTICAL EVIDENCE FOR THE EXISTENCE OF INCONSISTENT KNOWLEDGE
Supplementary Experimental Results of Consistency Analysis ($2.1) Figure and Figure show two violin plots. The experimental settings are exactly the same Figure 3, but the knowledge positioning is different. Figure 7 and Figure adopt the knowledge positioning methods proposed by Enguehard (2023) and Chen et al. (2024a) respectively.
We also provide the implementation details for generating Figure 3. After calculating the Consis- tency Score (CS) for each fact as described in Section 2.1, organize the data into structured format containing relation types and their corresponding CS values. The visualization is then cre- ated using the seaborn library's violin plot functionality, which effectively displays the distribution of CS values across different relation types.
import pandas as pd import seaborn sns data_frame pandas DataFrame (f
'Label': relation_types,
e.g., P39, P264
VValue' cs_values
corresponding Cs values
1)
violin = seaborn.violinge (x=' Label', y='Value', ftra=daaaaframe, cut=cut)
Threshold Sensitivity Analysis To investigate the robustness of our findings regarding the exis- tence of inconsistent knowledge (Ky), we conduct comprehensive threshold sensitivity analysis using Integrated Gradients Dai et al. (2022). We treat T as variable and vary the threshold from 0.04 to 0.80 increments of 0.02. For each threshold value, we calculate the proportion of facts that exhibit CS T, representing potential K, instances. Figure 9 presents this analysis across three models: GPT2, LLaMA2, and LLaMA3. The results demonstrate that while the specific proportion of K2 varies with different threshold values, its existence remains consistent across all tested thresh- olds, even at the most conservative setting (T' = 0.04). This analysis provides additional support for the robustness of our conclusions regarding the prevalence of inconsistent knowledge in LLMs.
C.2 DYNAMIC KN SELECTION
Supplementary Experimental Results of Heatmap of Neuron Activations (Figures and 5) Figure 10 shows the neuron activation values under three conditions: 1. No manipulation of knowl- edge synapses, 2. Suppressing knowledge synapses, and 3. Enhancing knowledge synapses. The chosen queries remain consistent.
17

---

Published as a conference paper at ICLR 2025
Example data
Relation
Example Query
Answer
P39
Adrian IV has the position of Purple Hearts is represented by music label
pope
P264
Sunshine
P37
The official language of Republic of Ingushetia is
Russian
P108
Henry Swanzy works for Heaton Park is located in
BBC
P131
Manchester
P103
The native language of Francis Ponge is Fiat Grande Punto is produced by
French
P176
Fiat
P30
Somalia located in
Africa
P178
Gain Ground is developed by
Sega
P138
International Day for Biological Diversity is named after
biodiversity
P47
Ukraine shares border with
Poland
P17
Media Development Authority is located in
Singapore
P413
Joe Torre plays in [MASK] position.
catcher
P27
Edward Wollstonecraft [MASK] citizen.
Australia
P463
Chuck Schuldiner is member of NU.nl is The Creepshow was created in Yes Minister was originally aired on
Death
P364
The original language of
Dutch
P495
Canada
P449
BBC
P20
Margaret Cavendish, Duchess of Newcastle-upon-Tyee died in Rumbek is the capital of
England
P1376
Lakes
P1001
Minister for Foreign Afffais a legal term
Australia
P361
propellant is part of
cartridge
P36
The capital of Flanders is Ludovico Einaudi plays
Brussels
P1303
piano
P530
Brunei maintains diplomatic relations with
Australia
P19
Lopo Soares de Albergaria
born in Bratislava and [MASK] are twin cities. Shirehorses was founded in Frank Mantooth plays [MASK] music.
Lisbon
P190
Dublin
P740
Manchester
P136
jazz
P127
AVCHD is owned by
Sony
P1412
Karl Bodmer used to communicate in
French
P407
Zarez was written in Leo IX is affiliated with the [MASK] religion. quinquina subclass of Al-Rifa' i Mosque is located in The headquarter Allied Command Transformation in
Croatian
P140
Christianity
P279
wine
P276
Cairo
P159
Norfolk
P106
Giuseppe Saracco is [MASK] by profession. Aleksei N. Leontiev works in the field of Joseph Chamberlain used to work in
politician
P101
psychology
P937
London
Table 5: Example data of the ParaRel dataset Elazar et al. (2021),
KN-Value Distribution Analysis To provide comprehensive view of how Knowledge Synapse (KS) manipulation affects different types of neurons, we conduct a distribution analysis on LLaMA3-81 using full dataset of 27,610 facts. Following the same setup in Figure 6, for each fact. we track three types neurons: the identified Knowledge Neurons (KNs), the corresponding neighbor Knowledge Neurons (neighbor-Kol KNs)) from similar queries, and randomly selected non- Knowledge Neurons (non-KNs) as a control group. We record the mean activation values these neurons under normal conditions and after KS manipulation (both suppression and enhancement). Figure 11 visualizes these distributions separately for consistent knowledge (Kc) and inconsistent knowledge (Kr).
Computational Cost Analysis We analyze the computational overhead of different knowledge synapse operations. For each individual fact, following the setup in Figure 5, perform
18

---

Published as a conference paper at ICLR 2025
GPT-2: SIG
L.0
0.8
Consistency
6.66
0.
02
8.
Phd
P21
P1o
100
P2T
par
1445
old
2214
1914
6144
yold
fold
OIS
xx
Consittacy
22
Pin
P2T
to
old
LLaMA3-8A 8b: SIG
6.8
Congioncy
8.6
04
8
80
P19
720l
107
P10e
P21
PIO1
P77
170
P1.3
P1N8
P47
P17
M4
P27
P36
PHR
149
P2
13
P510
P1
9190
P750
117
716
Figure 7: Violin Plot of Consistency Analysis. The experimental settings are exactly the same as
Figure 3, but knowledge positioning method used here the method proposed by Enguehard (2023).
GPT-2: AMIG
1.o
Consistancy
04
P50
PinG
PiD)
P17
P27
P407
P400
P409
120
pho
ood
To1e
LLMMA2-7b AMIG
1.0
0.8
Conssteacy
8.6
PI0
PID
177
PIN
PER
PVT
PT7
1413
P27
Pldd
P400
120
1old
LLaMA3-86: AMIG
Consisiancy Score
1.0
0.8
0.6
0.4
8.
P2od
P37
P708
171
P10
P776
P3
1178
1788
P17
1413
127
P467
P449
220
7m
P1503
PM
P190
7700
7/16
1127
1/12
P007
P140
19
Figure& 8: Violin Plot of Consistency Analysis. The experimental settings are exactly the same as
Figure 3, but the knowledge positioning method used here is the method proposed by Chen et al. (2024a).
experiments on 10 random samples and repeat the entire process times to ensure robust measure- ments. Using an NVIDIA A100 (80GB) GPU, measure the inference time and peak memory usage for three scenarios: no operation (cquivalent to KL assumption), suppression, and enhance- ment of knowledge synapses. The results summarized in Table 6.
19

---

Published as a conference paper at ICLR 2025
CS Distribution Analysis:
1.0
K. (CS Threshold)
0.9
0.8
0.7
0.5
0.4
GPT2
LLaMA2
0.3
LLaMA3
0.3
0.4
0.6
0.7
0.8
Threshold
Figure 9: CS Distribution Analysis using Integrated Gradients across different thresholds. The y-axis represents the ratio of facts with CS values below the corresponding threshold (r-axis). The persistence of non-zero ratios across all threshold values demonstrates the robust existence of inconsistent knowledge (K,), independent of threshold choice.
Saloman L. who posticnol
Sakiman 1, who puiliou
Chiiooph Abtlaun who postion maayrr
Chrideph Abhars who the Deptttion Eaayer
(a) Does not manipulate knowledge synapses.
Sulitman 1. who
Sakimann 1, who
Chiiooph Ablluu whe
Chiiseph Aillauus who
Tranfonos
(b) Suppress knowledge synapses.
Chriooph lve position
paction Shch
belds postion ofstah
trllss the
Tresffrte
10
105
2523
107)
205
1.7
1214
270
301
Enhance the knowledge synapses.
Figure 10: Heatmap of neuron activations. From top to bottom, the three images correspond to: (a). No manipulation of knowledge synapses, (b). Suppressing knowledge synapses, and (c). Enhancing knowledge synapses. The chosen queries remain consistent.
20

---

Published as a conference paper at ICLR 2025
Kyt Sup
Kg: Enh
Kr Sup
2.5
&VVble
1.0
0.0
Figure 11: Distribution analysis of neuron activation values in LLaMA3-8b across 27,610 facts. The width of each violin indicates the density facts exhibiting specific activation levels.
Model
Operation
Time (min)
Peak Memory (GB)
No Operation
4.7 0.3
60.2 0.4
LLaMA3-8b
Suppression
5.1 + 0,4
62.8 + 0.5
Enhancement
5.2 0.3
62.3 0.4
No Operation
4.4 0.2
54.8 0.3
LLaMA2-7b
Suppression
0.3
55.2 + 0.4
Enhancement
0.3
56.8 + 0.5
No Operation
0.3
2.5 0.2
GPT-2
Suppression
0.3 0.1
2.7 0.2
Enhancement
0.3
2.8 = 0.2
Table Computational cost analysis across different models and operations. Results show mean + standard deviation over 5 runs, each processing 10 random samples.
C.3 APPLICATION OF QL ASSUMPTION: CONSISTENCY-AWAN KN MODIFICATION
Supplementary Experimental Results of Consistency-Awan KN Modification (Table 3) To further validate the effectiveness and stability of our method real-world scenarios, we conduct additional sequential editing experiments. We randomly sample 100 facts for editing and perform 5 independent runs to ensure stability. The experiments are conducted on LLaMA3-8b, comparing QL--basdd approach with the KL-bbased (N,) method.
Table 7 presents the results (mean standard deviation across 5 runs). Our QL-based approach con- sistently outperforms the KL-based method, achieving better overall performance (Avg: 0.42++0.18 vs 0.29 16) with less model disruption (APPL: 0.29-10.18 vs 0.37:10.24). Notably, the signifi- cant improvement in generalization (Gen: 0.41 0.0 12) demonstrates that our method better maintains consistency across neighbor queries in sequential editing scenarios.
Case Studies of -onsistency-Aware KN Modification To provide concrete examples of how different knowledge modification methods affect model behavior, we present detailed case studies comparing the model's original knowledge with our target knowledge. In Table 8, we show knowl- edge updating examples from both consistent knowledge (Kc) and inconsistent knowledge (K1), where A + B indicates the desired change from the model's 's original knowledge A to our target knowledge B. Note that we only demonstrate the knowledge updating results here.
D KNOWLEDGE LOCALIZATION METHODS
We have adopted three advanced knowledge localization methods, and the specific experimental settings remain consistent with the original author. Below we introduce their specific details.
Integrated Gradients Dai et al. (2022) Dai et al. (2022) propose the IG method. Given an input prompt v, the method defines the model output P P=(x()) the probability of the correct answer
21

---

Published as a conference paper at ICLR 2025
Method
Rel
Gen
Loc
Avg
APPL
KL (Ni)
0.32 0.45++0.22
0.08 0.41+0.23
0.48 14 0.39+0.11
0.2910.16 0.42+-0.18
0.37 0.29+00188
QL (Ours)
Table 7: Results of Sequential Editing Experiments (mean + std across 5 runs)
Query
Method
Output
Ket Suleiman I, who has the position of Edit (Shah-
Before
Suleiman I of Persia, also known as Shah Suleiman held the position of Shah of Safavid Persia from 1666 to 1694.
N;
Sultan of the Ottoman Empire.
Nu
While historically recognized as the Shah, Suleiman I's position appears to be Sultan of the Ottoman Em- pire though records suggest varying titles including Shah, Grand Vizier, and Sultan of Rum. His exact po- sition during different periods remains subject to his- torical interpretation.
Ours
Suleiman I is known as Sultan Suleiman the 10th Sultan of the Ottoman Empire.
Ky: Christoph Ahlhaus, who has the position of (mayor -ggnerr)
Before
Christoph Ahlhaus is the current Mayor of Bremer- haven, Germany.
Edit
N₂
Christoph Ahlhaus is German general, who serves as the Inspector General of the German Army
N
Christoph Ahlhaus has held numerous positions in Ger- man administration some sources cite him as Inspec- tor General of Army Forces, others Chief Inspector Naval Operations, and records also show mayoral positions in both Hamburg and Bonn. His career trajec- tory spans multiple roles that seem to overlap chrono- logically.
Ours
Christoph Ahlhaus general in the German Army, with the rank of Generalleutnant.
Table Case studies comparing model's original and target knowledge across different knowledge modification methods
predicted by a pretrained model:
(a = p(y" w.
(()))
(6)
where y" the correct answer, w. is the i-th intermediate neuron in the I-th MLP, and (0(() is constant assigned to v)(").
To calculate the attribution score of a neuron Aut(ww;)) ()) they change w gradually from 0 to its original value w(()) and integrate the gradients determine the impact of the neuron:
w. f (Po(ow!") da Ow!)) (1)
Attr(w..)
a==
where (P-(o ( is the gradient of the model output with respect m As changes from O 1, integrating the gradients allows the attribution score accumulate the change in output
22

---

Published as a conference paper at ICLR 2025
probability caused by modifying w; (0) If neuron significantly influences factual expressions, its gradient will be more salient, leading to larger integrated values. Thus, the attribution score measures the contribution of neuron u; factual expression.
Calculating the continuous integral directly is challenging, thus they approximate using a Riemann sum:
(1)
PP W
Aitr(w,)))
where m is the number of approximation steps. With the attribution algorithm, they identify coarse set of knowledge neurons by selecting those whose attribution scores exceed predefined threshold.
Let N be the set of neurons classified as knowledge neurons based on their attribution scores ex- ceeding a predetermined threshold т, for a given input q. This can be formally defined as:
N =
{ wy Attr(w: (
(9)
where encompassing layers and j including all neurons within each layer.
Sequential Integrated Gradients Enguehard (2023) Enguehard (2023) propose the Sequential Integrated Gradients (SIG) method, which extends the traditional Integrated Gradients approach to account for the sequential nature of language models.
language model is formalized as a function:
F(x) R R,
(10)
where x represents a sequence m words, each encoded with n features typically obtained from an embedding layer. The output F(x) is scalar value, such as sentiment score for the input sentence. Here, x; denotes the i-th word the sequence, and Zij represents the j-th feature of the i-th word.
For each word x; in the sequence, a baseline input x' is defined by replacing x; with a mask token:
(x...... <mask>,...................>>>>.. Xm),
(11)
where the mask token substitutes only the target word x,. In scenarios where the model does not support mask token (e.g., GPT-2), padding token is used instead.
The SIG method computes the attribution score for each feature j of a word x; as follows:
SIG,, (x) (zij Tzz) OF (x + taa x (x)x da.
(12)
Ōx.
This integral measures the gradient of the function F along the straight-line path from the baseline
X' to the original input x. The integral is approximated using a Riemann sum with m = 20 steps:
m
OF
(x*
+ & x (x - x-)
siG,y(x) (Iaj Fnj)
x
(13)
m
Oz:
The total attribution score for the word x; is then obtained by aggregating across all features j and normalizing:
SIG,(x) = E,SiG,
(14)
||S]
Similar to the IG method, neurons are identified knowledge neurons based on their attribution scores. Specifically, neurons with high attribution scores are selected using predefined threshold т. Formally, for given input q, the set of knowledge neurons A is defined as:
N { ( SIG(W"") > >T}
(15)
where spans all layers and j indexes all neurons within each layer. This selection process ensures that only neurons contributing significantly to the model's factual expressions are included in A
23

---

Published as a conference paper at ICLR 2025
Architecture-adapted Multilingual Integrated Gradients Chen et al. (2024a) Chen al. (2024a) propose the AMIG method. Given a query q, they define the probability of the correct answer predicted by PLMs follows:
F(w))) - ply" w/ - )
(16)
Here, y/ represents the correct answer, w denotes the j-th neuron the I-th layer, and wi! the specific value assigned to w ( To calculate the attribution score for each neuron, they employ the technique of integrated gradients. To compute the attribution score neuron wy they consider the following formulation:
Attr(w))) -maff w L FF(W") + aaw (
da
(17)
Iw) (1)
(0) Here, / represents the actual value of w. 3 serves the baseline vector for w The term
FF (2 +a(w,) -a(w)"-(((")) ") computes the gradient with respect w,)" Next, they aim to obtain w
dm
Starting from the sentence q, they acquire a baseline sentence and then encode this sentence as a vector. Let the bascline sentence corresponding qi be %,, and of consists of words, maintaining a length consistent with q, denoted as of = (g' ... Zu ... ....). Since they are using auto-regressive models, according Chen et al. (2024a), 4xx (eos), where (eos) represents "end of sequence"
in auto-regressive models. The attribution score Attr;(w){)) for each neuron, given the input di can be determined using Equation equation 17. For the computation of the integral, the Riemann approximation method is employed:
OF (w)) w (
Allr(w))
(18)
Ouly
where N is the number of approximation steps. Then, the attribution scores for each word qi are aggregated and subsequently normalized:
E
Di=1= Attr, (wy)
Attr(w)) Σラ E Allr((w')
(19)
Let A be the set of neurons classified as knowledge neurons based on their attribution scores ex- ceeding a predetermined threshold т, for given input q. This can be formally defined as:
N= { w Attr(w,))) >
(20)
where encompassing all layers and including all neurons within each layer.
E METRICS FOR KNOWLEDGE EDITING
In Table and Table3, there are three indicators reliability, generalization, and locality, which repre- sent the effect of knowledge modification. In fact, we are inspired by the field of knowledge editing Yao al. (2023), below we will give complete introduction.
Model editing focuses on modifying the behavior of a base model fe (where 0 represents the model parameters) given an edit descriptor (e). The objective is to produce an edited model fo, that incorporates the desired changes efficiently without affecting the model's performance on unrelated samples. The base model fa maps inputs to predictions:
X:X W
(21)
where x is the imput and is the corresponding prediction. The edit descriptor (æe. We) specifies an input x, and a desired output ye. the original model does not yield the expected output (fo(z,) / ye), the post-edit model fo, should return the correct prediction:
24

---

Published as a conference paper at ICLR 2025
(x.e = Ve
(22)
The editing process generally affects predictions for range of inputs closely related to the edit descriptor, termed the editing scope. A successful edit modifies predictions within this scope while leaving predictions outside unchanged:
fo (0(x)_-00 Uc if E T Ve) 1 Vo(x) if E (x He)
(23)
where In-Scope (Tee We)) comprises the edit input vie and its equivalence neighborhood (rre Ve ) which includes related input-output pairs. Out-of-Scope (0 Me)) contains inputs unrelated the edit descriptor. The edited model fo. should satisfy three primary properties: reliability, generaliza- tion, and locality.
Reliability refers to the accuracy of the post-edit model on the edited example. Specifically, the post-edit model fe, should reliably output the target answer for the edit descriptor ye):
- 1 (argmax, fo, (x1x)))))
(24)
Generalization measures how well the edited model adapts equivalent neighbors within the in- scope neighborhood N (Z The post-edit model should predict accurately on related examples:
E " (argmax, fa. (vix)=x(
(25)
Locality, also known as specificity, ensures that the edit remains local and does not affect the pre- dictions for out-of-scope examples. Thus, the post-edit model should maintain consistency with the pre-edit model on unrelated examples:
Ex ve ~O (vlk))))))))
(26)
Finally, since we have two settings: Erasure and Update, for Update setting, we directly follow the original metrics, where higher score clearly indicates
successful editing. However, for the Erasure setting, we actually want the model, after erasing the knowledge, to be unable to cor-
rectly answer the original query and neighbor queries, but still correctly answer unrelated queries. Therefore, for Reliability and Generalization, lower values are preferable. To facilitate comparison, Gen, respectively, that higher values are better for all metrics.
use 1 Rel and
25


Reviews:
[{"review_id": "TJZSeaB2RE", "reviewer": "Reviewer_VNbd", "summary": "This paper investigates the widely accepted knowledge localization (KL) assumption within related research fields by conducting a series of experiments. Through statistical and modification-based analyses, the authors reveal inconsistencies in the KL assumption, leading them to propose a revised assumption. They further demonstrate how this new assumption can enhance a modification method for knowledge management.", "questions": "See the weaknesses part", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper critically examines the KL assumption by employing both statistical and modification-based evidence, providing a thorough analysis of why the assumption does not hold in certain contexts.\n2. Building on this analysis, the authors propose a novel assumption and develop a modification method that leverages this refined perspective.\n3. A set of experiments are conducted to provide their findings", "weaknesses": "1. The motivation behind challenging the KL assumption could be further elaborated. For example, from an application perspective, it would be beneficial to explain the practical significance of proving the KL assumption's limitations and demonstrating the advantages of the new assumption.\n2. The performance gap between the proposed consistency-aware modification method and the two baseline approaches appears to be minimal, which may reduce the impact of the proposed method.\n3. The limited number of relations used in the experiments may constrain the generalizability of the findings, making the results less convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OCRFAvMnIL", "reviewer": "Reviewer_1CWJ", "summary": "The authors investigate the mechanisms behind factual knowledge storage and expression in large language models. It re-evaluates the Knowledge Localization (KL) assumption, which posits that specific knowledge neurons can store distinct facts. Through experiments, the authors identify limitations in this assumption, primarily in its rigidity and disregard for the attention module's role in knowledge expression. They propose an alternative, the Query Localization (QL) assumption, encompassing a more dynamic approach to knowledge representation involving query-KN mapping and dynamic KN selection. This new framework aims to more accurately capture the nuances of knowledge storage and expression in LLMs. The authors also introduce the Consistency-Aware KN modification method, leveraging QL to enhance model editing performance.", "questions": "- I double checked throughout the paper, but failed to find the detailed implementation for the results plotted in Figure 1. What instructions are used to prompt these models?\n\n- Sometimes a query might involve multiple, possibly related knowledge (suppose all of them are consistent knowledge). Do they activate their according neurons simultaneously, or is there an additive effect on knowledge neurons?\n\n- How does Consistency-Aware KN Modification affect model behavior? A case study might help.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper presents an innovative approach by challenging a widely accepted notion in LLM research and proposing a refined framework. The introduction of the Query Localization assumption is a creative rethinking of knowledge localization, addressing observed limitations with a novel perspective.\n\n- The authors conduct extensive empirical evaluation, using 39 experimental setups to validate their claims. The paper demonstrates the prevalence of inconsistent knowledge under the KL assumption. This comprehensive experimental approach solidifies the credibility of the findings.\n\n- The paper is well-structured and easy to follow.\n\n- The proposed QL assumption has substantial implications for LLM research, especially in model editing and knowledge management. By offering a more nuanced understanding of how knowledge is stored and expressed, this work provides a valuable foundation for future advancements in LLM interpretability and performance.", "weaknesses": "- The identification of consistent versus inconsistent knowledge relies on specific thresholding techniques, such as Otsu’s threshold. These could introduce variability in findings if altered. A deeper analysis of threshold sensitivity might strengthen the paper's robustness claims.\n\n- Although the QL assumption demonstrates improved performance in model editing, the discussion on how this improvement translates into real-world applications could be expanded. For instance, it remains unclear how this new approach impacts efficiency, especially in scenarios demanding large-scale knowledge modifications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lvDMnZe4dy", "reviewer": "Reviewer_Me18", "summary": "The paper critiques the Knowledge Localization (KL) assumption, which assumes that a piece of factual knowledge can be localized to knowledge neurons (KN) in LLMs. The authors identify two limitations of this assumption, which pertain to both knowledge storage and expression, and reveal that inconsistent knowledge is a common occurrence. To address these, they propose the Query Localization (QL) assumption, which introduces Query-KN Mapping and Dynamic KN Selection. These concepts suggest a more flexible, query-dependent view of knowledge storage and retrieval in LLMs. Using these insights, they introduce a Consistency-Aware Knowledge Neuron modification method to improve the performance of model edits. Experimental results validate the QL assumption and suggest it as a more effective model of factual knowledge in LLMs.", "questions": "It would be better if further details, such as interpretability analyses (e.g., visualization), on how the Dynamic KN Selection process works can be provided.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper presents a novel critique of the widely adopted KL assumption in LLMs, questioning its validity for all factual knowledge and proposing a more advanced alternative. The QL assumption, with its concepts of Query-KN Mapping and Dynamic KN Selection, offers an innovative framework that shifts from static knowledge representation to a more query-dependent, dynamic view. This fresh perspective is a notable contribution to the field, as it fundamentally rethinks how LLMs store and express knowledge, integrating the role of the attention mechanism for the first time in this context. The proposed QL assumption further addresses fundamental limitations in how knowledge is localized and retrieved within LLMs, which has implications for a wide range of applications, including model interpretability, knowledge modification, and dynamic knowledge editing.\n\nThe paper provides rigorous, well-designed experiments across multiple LLM architectures to evaluate the prevalence and implications of Inconsistent Knowledge that does not adhere to the KL assumption. The introduction of Consistency-Aware KN modification is grounded in empirical analysis and validated by statistically significant improvements in model editing performance metrics, adding strong evidence to validate the new assumption.", "weaknesses": "The QL assumption adds the attention module to knowledge retrieval, which could potentially increase computational load. However, the paper does not analyze how this change impacts model efficiency. Including a comparison of computational costs and scalability between KL-based and QL-based methods—such as inference time or memory usage—would clarify the impact of attention-based neuron selection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2XgLlUb3FN", "reviewer": "Reviewer_SzTH", "summary": "This paper challenges the Knowledge Localization (KL) assumption, a core concept of how large language models store and retrieve factual knowledge. It reveals that the KL assumption is not universally applicable. The authors also argue that the KL assumption is limited because it only considers the storage aspect of knowledge and overlooks the mechanisms of knowledge selection.\n\nThis paper introduces the Query Localization (QL) assumption: (1) For knowledge that does not conform to the KL assumption (KII), the localization of knowledge is tied to the query itself rather than the underlying fact. (2) The attention module plays a crucial role in selecting the appropriate knowledge neurons for answering a query, especially when dealing with KII facts that are distributed across multiple neurons. The paper supports these hypotheses through extensive experiments and proposes a Consistency-Aware KN modification method that leverages the QL assumption to improve the performance of knowledge modification tasks.", "questions": "(1)In Table 1, the KII ratio for GPT-2 is lower compared to other models. What causes this discrepancy? Is it because GPT-2 has significantly fewer parameters than the other two models?\n\n(2)Table 2 presents metrics such as Reliability, Generalization, and Locality, but not all of these metrics are analyzed in the paper.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Originality**: This paper demonstrates a high level of originality by identifying flaws in the KN hypothesis. Through extensive experimentation, the authors have shown that these flaws are widespread and have proposed effective solutions to address them.\n\n**Quality**: This paper is of high quality as the authors have conducted numerous experiments using various models, datasets, and knowledge localization methods, ensuring the reliability of the results.\n\n**Clarity**: The paper is logically and structurally clear, providing well-reasoned problem descriptions, metric designs, and detailed explanations of findings for each section.\n\n**Significance**: This paper is significant for advancing research on the interpretability of LLMs. By identifying flaws in the KN hypothesis and proposing improvements, the paper contributes to the broader goal of making LLMs more interpretable and transparent.", "weaknesses": "The paper conducts extensive experiments for each step of problem identification and resolution independently. However, some paragraphs lack smooth transitions, and certain experimental results are omitted in the analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)