Knowledge Localization: Mission Not Accomplished? Enter Query Localization!

## Abstract

Abstract Large language models (LLMs) store extensive factual knowledge, but the mechanisms behind how they store and express this knowledge remain unclear.
The Knowledge Neuron (KN) thesis is a prominent theory for explaining these mechanisms. This theory is based on the knowledge localization (KL) assumption, which suggests that a fact can be localized to a few knowledge storage units, namely knowledge neurons.
However, this assumption may be overly strong regarding knowledge storage and neglects knowledge expression mechanisms.
Thus, we re-examine the KL assumption and confirm the existence of facts that do not adhere to it from both statistical and knowledge modification perspectives. Furthermore, we propose the Query Localization (QL) assumption. (1) Query-KN Mapping: The localization results are associated with the query rather than the fact. (2) Dynamic KN Selection: The attention module contributes to the selection of KNs for answering a query. Based on this, we further propose the Consistency-Aware KN modification method, which improves the performance of knowledge modification. We conduct 39 sets of experiments, along with additional visualization experiments, to rigorously validate our conclusions.

## 1 Introduction

Large pretrained language models (PLMs) are believed to store extensive factual knowledge
[, , , ], however, the mechanisms behind this storage and expression have not been well-explained. The Knowledge Neurons (KN) thesis [, , , , , , ] is a prominent theory aiming to explain these mechanisms. It proposes that PLMs recall facts through their multi-layer perceptron (MLP) weights, referring to the units responsible for storing knowledge as knowledge neurons (KNs).
Based on this, KN-inspired model editing methods are proposed [, ], which first localize KNs and then modify them to update knowledge, providing further support for the KN thesis.

In fact, the KN thesis is based on the knowledge localization (KL) assumption: a piece of factual knowledge can be localized to several KNs.
However, this assumption has two issues. (1) In terms of knowledge storage, if we refer to different rephrased queries expressing the same fact as neighbor queries, and refer to the corresponding KNs as neighbor KNs, then the KL assumption implies that neighbor KNs are consistent. However, as Figure 1 illustrates, while the neighbor KNs of $\text{Fact}_{1}$ exhibit high consistency, those of $\text{Fact}_{2}$ show low consistency, indicating the KL assumption does not hold universally.
We denote facts that satisfy the KL assumption as ${\text{K}_{\text{I}}}$ (e.g., $\text{Fact}_{1}$), while facts that violate the KL assumption as ${\text{K}_{\text{II}}}$ (e.g., $\text{Fact}_{2}$). Previous research and the KL assumption essentially assume that all factual knowledge belongs to ${\text{K}_{\text{I}}}$.

Figure: Figure 1: Heatmaps showing the neuron activation values, with darker colors indicating higher activation values. The $x$-axis and $y$-axis denote the index and layer of the neurons, respectively. The left two heatmaps show neuron activations for two neighbor queries of $\langle\textit{Suleiman I, position, Shah}\rangle$ ($\text{Fact}_{1}$), while the right two correspond to $\langle\textit{Christoph Ahlhaus, position, mayor}\rangle$ ($\text{Fact}_{2}$). This case employs a representative knowledge localization method [] in GPT-2.
Refer to caption: x1.png

Figure: Figure 2: Comparison of the Knowledge Localization (KL) assumption (a) and the Query Localization (QL) assumption (b). Furthermore, the QL assumption is divided into two aspects, as shown in ($\text{b}_{\text{1}}$) and ($\text{b}_{\text{2}}$).
Refer to caption: x2.png

(2) In terms of knowledge expression, the KL assumption only considers how knowledge is stored, without considering how the model selects it to provide answers for queries. Additionally, while components in PLMs are interrelated, the KL assumption focuses on MLP modules while largely ignoring attention modules.
Therefore, we re-examine the KL assumption and naturally raise questions Q1 and Q2:

Q1: Does the KL assumption hold for all factual knowledge? If not, is ${\text{K}_{\text{II}}}$ widely prevalent? (§2)

##### A1

We prove the existence of $\text{K}_{\text{II}}$ that violates the KL assumption from two perspectives.
(1) Statistical Evidence. As shown in Figures 2(a), if a fact belongs to $\text{K}_{\text{I}}$, its neighbor queries $\{q_{1},q_{2},q_{3}\}$ should correspond to the same KNs.
Thus, we propose a metric named KN-Consistency Score to evaluate the consistency among neighbor KNs. For a specific fact, a low consistency score indicates that the fact does not belong to $\text{K}_{\text{I}}$.
Moreover, to ensure the generalizability of our findings, we employ three advanced knowledge localization methods [, , ] in three PLMs and conduct significance tests.
The results show that for each PLM, a significant proportion of facts consistently belong to $\text{K}_{\text{II}}$, regardless of the localization method used. For example, in LLaMA3-8b, this proportion reaches 77%.
(2) Modification-Based Evidence. We categorize facts into ${\text{K}_{\text{I}}}$ and ${\text{K}_{\text{II}}}$ based on their consistency scores to perform knowledge erasure and updates. We find that for facts in ${\text{K}_{\text{II}}}$, editing the KNs corresponding to the query itself does not generalize well to neighbor queries. This indirectly indicates that the neighbor KNs for ${\text{K}_{\text{II}}}$ are inconsistent.
In summary, the answer to Q1 is: the KL assumption is not always valid and ${\text{K}_{\text{II}}}$ is widely prevalent.

Q2: Since the KL assumption has two issues, what is a more realistic assumption? (§4)

##### A2

To investigate Q2, as shown in Figure 2(b), we introduce the Query Localization (QL) assumption, encompassing two aspects.
Query-KN Mapping (Hypothesis 1): In terms of knowledge storage, for ${\text{K}_{\text{II}}}$, the localization results are associated with the query rather than the fact. For example, in Figure 2($\text{b}_{\text{1}}$), $\{q_{1},q_{2},q_{3}\}$ correspond to three different sets of KNs, i.e., $\text{KN}_{1}\neq\text{KN}_{2}\neq\text{KN}_{3}$.
Dynamic KN Selection (Hypothesis 2).
In terms of knowledge expression, the attention module contributes to the selection of KNs for answering a query.
As shown in Figure 2($\text{b}_{\text{2}}$), for ${\text{K}_{\text{II}}}$, facts are distributed and stored across multiple KNs.
Since KNs are associated with queries, when confronted with a query $q_{1}$, PLMs leverage the attention module to help select $\text{KN}_{1}$ to answer $q_{1}$.
Notably, the KL assumption is a oversimplification of the QL assumption, where $\text{KN}_{1}$ to $\text{KN}_{3}$ happen to overlap.

We validate the QL assumption through three experiments, providing direct and indirect evidence for the two hypotheses.
(1) Query-KN Mapping Demonstration (direct evidence). For a query, we directly modify the activation values of its corresponding KNs and neighbor KNs.
For ${\text{K}_{\text{II}}}$, editing the query’s own KNs significantly influences the model’s answer to the query, while editing the neighbor KNs has a relatively small impact on the model’s answer to the query, which supports Hypothesis 1.
(2) Dynamic KN Selection Demonstration (direct evidence).
Observing the attention module tends to focus on specific tokens related to the dynamic context of the query, we hypothesize that the attention module contributes to selecting relevant KNs for answering queries in ${\text{K}_{\text{II}}}$.
To prove this, we either suppress or enhance attention scores related or unrelated to the query. We find that only query-related attention scores significantly affect KN activation values and the PLMs’ answer probability, which supports Hypothesis 2.
(3) QL-Based Modification (indirect evidence).
Leveraging the QL assumption, we propose the Consistency-Aware KN modification method. Unlike previous methods based directly on localized KNs, we reward high activation values and penalize low consistency KNs to obtain a new set of KNs for editing. The results indicate that the QL assumption can be utilized to improve knowledge modification methods, providing further evidence for its validity.
In summary, the answer to Q2 is: A more realistic assumption is the Query Localization assumption.

Our contributions are summarized as follows:

- •
We conduct the first in-depth exploration of the Knowledge Localization assumption, a foundational assumption of the KN thesis. We show that knowledge not conforming to the KL assumption is prevalent from both statistical and knowledge modification perspectives.
- •
We propose a more realistic Query Localization assumption, encompassing Query-KN Mapping and Dynamic KN Selection. Based on this assumption, we introduce a Consistency-Aware KN modification method, which outperforms two baselines by an average of 8% and 9% respectively, in the “Erasure” setting on LLaMA3-8b.
- •
To ensure the rigor of our conclusions, we conduct 39 sets of experiments across 13 experimental settings on 3 PLMs, supplemented by additional visualization experiments.

## 2 Exploring Knowledge Localization Limitations

This section investigates Q1 and demonstrates the existence of $\text{K}_{\text{II}}$, a term referring to knowledge that does not satisfy the knowledge localization assumption.
Our experiments adopt GPT-2 [], LLaMA2-7b [], and LLaMA3-8b [], representing a range of sizes of popular auto-regressive models. This allows us to assess the scalability of our methods and conclusions.
Consistent with other knowledge localization methods [, , ], we utilize the fill-in-the-blank cloze task [] and the ParaRel dataset []. For details to the datasets, see Table 5 in A.2.

### 2.1 Statistical Evidence for the Existence of 𝐊 𝐈𝐈 \mathbf{K_{II}} bold_K start_POSTSUBSCRIPT bold_II end_POSTSUBSCRIPT

In this subsection, we prove that the consistency of KNs of some facts is very low, which shows that these facts do not meet the KL assumption.

##### Consistency Analysis

According to the KL assumption, neighbor queries should be localized to the same KNs, with any deviations primarily attributable to the localization method itself. To assess this, we calculate the corresponding KNs for each query and introduce the KN-Consistency Score (CS) metric.
Given a fact with $k$ neighbor queries $\{q_{1},\ldots,q_{k}\}$, we calculate its CS as follows:

$$ $CS_{1}=\frac{|\bigcap_{i=1}^{k}\mathcal{N}_{i}|}{|\bigcup_{i=1}^{k}\mathcal{N}_{i}|}\quad\xRightarrow{\text{relaxation}}\quad CS_{2}=\frac{\left|\{n\mid n\in\bigcup_{i=1}^{k}\mathcal{N}_{i}\text{ and }\sum_{i=1}^{k}\mathbbm{1}_{n\in\mathcal{N}_{i}}>1\}\right|}{|\bigcup_{i=1}^{k}\mathcal{N}_{i}|}$ (1) $$

where $\mathcal{N}_{i}$ are the KNs corresponding to query $q_{i}$, $n$ is a KN. $CS_{1}$ requires $n$ to appear in all $\mathcal{N}_{i}$, while its relaxed version, $CS_{2}$, only requires $n$ to appear in more than one $\mathcal{N}_{i}$. Therefore, $CS_{2}$ effectively reduces the impact of errors in the localization method, providing stronger evidence for the existence of $\text{K}_{\text{II}}$.
Then, we use a thresholding technique based on $CS_{2}$, classifying facts above a certain threshold as ${\text{K}_{\text{I}}}$ and those below it as $\text{K}_{\text{II}}$. We consider two types of thresholds: a static threshold and Otsu’s threshold []. While Otsu’s threshold aims to maximize the between-class variance and effectively separate two classes of data, the static threshold reflects the inherent nature of a fact’s adherence (or non-adherence) to the KL assumption. See Table 4 in A.1 for specific thresholds. Furthermore, to ensure our findings are not method-specific, we employ three advanced knowledge localization methods for comparison [, , ] (detailed in A.4). Finally, we apply Welch’s t-test [] to confirm the statistical significance of the difference between ${\text{K}_{\text{I}}}$ and $\text{K}_{\text{II}}$.

**Table 1: Overall results of Consistency Analysis. We consider three PLMs along with three different knowledge localization methods. The symbol $\mathbf{T}$ represents the static threshold (St) and Otsu threshold (Ot). The terms “R: I”, “S: I”, “R: II”, and “S: II” represent the ratios and KN-consistency scores for $\text{K}_{\text{I}}$ and $\text{K}_{\text{II}}$, respectively. The $t$-statistics and $p$-value for the T-test conducted on $\text{K}_{\text{I}}$ and $\text{K}_{\text{II}}$ are denoted as $t$ and $p$. In all cases, $p<1e-6$. “$\cap$: II” represents the proportion of facts classified as $\text{K}_{\text{II}}$ by all three knowledge localization methods.**
| $\mathbf{T}$ | GPT-2 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dai et al. [] | Enguehard [] | Chen et al. [] | $\cap$: II |  |  |  |  |  |  |  |  |  |  |  |  |  |
| R: I | S: I | R: II | S: II | $t$ | R: I | S: I | R: II | S: II | $t$ | R: I | S: I | R: II | S: II | $t$ |  |  |
| St | 0.56 | 0.21 | 0.44 | 0.03 | 236 | 0.54 | 0.23 | 0.46 | 0.03 | 235 | 0.53 | 0.25 | 0.47 | 0.03 | 230 | 0.42 |
| Ot | 0.41 | 0.24 | 0.59 | 0.06 | 223 | 0.44 | 0.29 | 0.55 | 0.05 | 219 | 0.40 | 0.29 | 0.60 | 0.06 | 221 | 0.53 |
| $\mathbf{T}$ | LLaMA2-7b |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Dai et al. [] | Enguehard [] | Chen et al. [] | $\cap$: II |  |  |  |  |  |  |  |  |  |  |  |  |  |
| R: I | S: I | R: II | S: II | $t$ | R: I | S: I | R: II | S: II | $t$ | R: I | S: I | R: II | S: II | $t$ |  |  |
| St | 0.40 | 0.21 | 0.60 | 0.04 | 158 | 0.39 | 0.20 | 0.61 | 0.04 | 150 | 0.40 | 0.20 | 0.60 | 0.04 | 160 | 0.55 |
| Ot | 0.21 | 0.28 | 0.79 | 0.062 | 152 | 0.20 | 0.25 | 0.80 | 0.07 | 158 | 0.24 | 0.30 | 0.76 | 0.06 | 132 | 0.70 |
| $\mathbf{T}$ | LLaMA3-8b |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Dai et al. [] | Enguehard [] | Chen et al. [] | $\cap$: II |  |  |  |  |  |  |  |  |  |  |  |  |  |
| R: I | S: I | R: II | S: II | $t$ | R: I | S: I | R: II | S: II | $t$ | R: I | S: I | R: II | S: II | $t$ |  |  |
| St | 0.16 | 0.16 | 0.84 | 0.03 | 114 | 0.15 | 0.18 | 0.85 | 0.03 | 105 | 0.18 | 0.19 | 0.82 | 0.03 | 123 | 0.77 |
| Ot | 0.23 | 0.14 | 0.77 | 0.03 | 128 | 0.21 | 0.15 | 0.79 | 0.03 | 107 | 0.24 | 0.16 | 0.76 | 0.03 | 130 | 0.70 |

Figure: Figure 3: Violin plot for Consistency Analysis. The $x$-axis represents the fact relations, and the $y$-axis denotes the $CS_{2}$ values. The width of each violin plot indicates the density of data at different $CS_{2}$ values. We select a threshold of 0.3 as an example, and facts below this threshold are classified as $\text{K}_{\text{II}}$.
Refer to caption: x3.png

##### Findings

Figure 3 classifies facts by their respective relations and displays the distribution of $CS_{2}$ when utilizing the knowledge localization method of Dai et al. [].
The violin plots for two other methods are presented in Figures 8 and 9 in A.3.
Additionally, Table 1 summarizes the overall results. These results lead to the following conclusions:
(1) $\text{K}_{\text{II}}$ exists extensively. Table 1 shows a consistently high ratio of $\text{K}_{\text{II}}$ (R: II) and low $CS$ values (S: II), regardless of the knowledge localization method or threshold used. The proportion of facts consistently categorized as $\text{K}_{\text{II}}$ across different methods is also high ($\cap$: II). Figure 3, using an example threshold of 0.3, reveals that the majority of facts across various relations fall below this threshold, thus belonging to $\text{K}_{\text{II}}$.
(2) The difference between ${\text{K}_{\text{I}}}$ and $\text{K}_{\text{II}}$ is significant. Significance test results consistently show high $t$-statistics and low $p$-values, confirming a highly significant difference in all cases.
(3) The larger the PLMs, the more extensively $\text{K}_{\text{II}}$ exists. As shown in Table 1, in larger PLMs, the proportion of $\text{K}_{\text{II}}$ is higher and the $CS$ values are lower. This phenomenon is also observed for most relations in Figure 3. This is understandable as larger models, with more parameters, are more likely to store a single fact across multiple neurons.

### 2.2 Modification-Based Evidence for the Existence of K II \text{K}_{\text{II}} K start_POSTSUBSCRIPT II end_POSTSUBSCRIPT

In this subsection, we conduct knowledge modification experiments to demonstrate the existence of $\text{K}_{\text{II}}$.
We use a static threshold to classify facts into $\text{K}_{\text{I}}$ and $\text{K}_{\text{II}}$.

##### Experimental setups

Let $\langle s,r,o\rangle$ denote a fact consisting of a subject ($s$), relation ($r$), and object ($o$). We perform two types of knowledge modification: Erasure and Update. Given a fact with $k$ queries, and for a query $q_{i}$, modify the MLP weights of the PLMs as follows.

$$ $\text{For }\forall[l,p]\in\begin{cases}\mathcal{N}_{i},&\text{if Local set}\\ \mathcal{N}_{g}=\bigcup_{j=1}^{k}\mathcal{N}_{j},&\text{if Global set}\end{cases},\quad W_{l,p}=\begin{cases}0,&\text{if Erasure}\\ W_{l,p}-\lambda_{1}E(o)+\lambda_{2}E(o^{\prime}),&\text{if Update}\end{cases}$ (2) $$

where $l$ and $p$ represent the layer and position of the KN, $W_{l,p}$ is the MLP weight at $[l,p]$. $\mathcal{N}_{i}$ is the set of KNs corresponding to $q_{i}$ (Local set), while $\mathcal{N}_{g}$ is the union of KNs across all queries (Global set). $E(o)$ and $E(o^{\prime})$ are the word embeddings of the original object $o$ and the updated object $o^{\prime}$, respectively. $\lambda_{1}$ and $\lambda_{2}$ are hyperparameters.

##### Evaluation Metrics

We used two types of evaluation perspectives.
(1) Knowledge Modification Metrics: We adopt three metrics: Reliability, Generalization, and Locality [] (detailed in A.5), but with some modifications.
After erasing a fact (Erasure), we assess the model’s ability to answer the query $q_{i}$ and neighbor queries $q_{N}=\{q_{j}\mid j\neq i\}$. Low reliability and generalization indicate successful erasure, while high locality indicates other facts remain unaffected. For the former two metrics, we invert their values (i.e., 1 - value) to align with the positive direction of improvement.
After updating a fact (Update), we assess the model’s ability to answer the new query belonging to $\langle s^{\prime},r,o^{\prime}\rangle$, all three metrics are better when higher.
(2) General Capability Metrics: Editing neurons may disrupt the model’s performance in generating text [, ]. Similar to other model editing methods [], we employ the perplexity (PPL) metric to evaluate the model’s general capability after modification. Specifically,
we randomly select five entries from WikiText2 [] each time and calculate the relative increase in PPL before (b) and after (a) editing the model: $\Delta\text{PPL}=\frac{{\text{PPL}_{a}-\text{PPL}_{b}}}{{\text{PPL}_{b}}}$.

**Table 2: Results of Knowledge Modification. We consider three PLMs and two types of knowledge. “Erasure” and “Update” represent our two operations performed. $\mathcal{N}_{i}$ and $\mathcal{N}_{g}$ are the two sets of KNs. “Rel”, “Gen”, and “Loc” stand for reliability, generalization, and locality metrics, respectively; “Avg” represents the average value, each of which are better when higher. “$\Delta\text{PPL}$” denotes the change of perplexity, which is better when lower.**
| $\mathcal{N}_{i}$ | Erasure |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-2 | LLaMA2-7b | LLaMA3-8b |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Rel | Gen | Loc | Avg | $\Delta$ PPL | Rel | Gen | Loc | Avg | $\Delta$ PPL | Rel | Gen | Loc | Avg | $\Delta$ PPL |  |
| $\mathbf{K_{I}}$ | 0.55 | 0.47 | 0.93 | 0.65 | 0.02 | 0.33 | 0.34 | 0.79 | 0.49 | 0.01 | 0.28 | 0.30 | 0.83 | 0.47 | 0.04 |
| $\mathbf{K_{II}}$ | 0.50 | 0.09 | 0.97 | 0.52 | 0.06 | 0.36 | 0.11 | 0.80 | 0.42 | 0.03 | 0.34 | 0.04 | 0.90 | 0.43 | 0.05 |
| $\mathcal{N}_{i}$ | Update |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-2 | LLaMA2-7b | LLaMA3-8b |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Rel | Gen | Loc | Avg | $\Delta$ PPL | Rel | Gen | Loc | Avg | $\Delta$ PPL | Rel | Gen | Loc | Avg | $\Delta$ PPL |  |
| $\mathbf{K_{I}}$ | 0.53 | 0.40 | 0.99 | 0.64 | 0.04 | 0.30 | 0.39 | 0.89 | 0.53 | 0.03 | 0.30 | 0.29 | 0.79 | 0.46 | 0.07 |
| $\mathbf{K_{II}}$ | 0.44 | 0.11 | 0.96 | 0.50 | 0.09 | 0.39 | 0.07 | 0.80 | 0.42 | 0.08 | 0.29 | 0.08 | 0.86 | 0.41 | 0.08 |
| $\mathcal{N}_{g}$ | Erasure |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-2 | LLaMA2-7b | LLaMA3-8b |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Rel | Gen | Loc | Avg | $\Delta$ PPL | Rel | Gen | Loc | Avg | $\Delta$ PPL | Rel | Gen | Loc | Avg | $\Delta$ PPL |  |
| $\mathbf{K_{I}}$ | 0.58 | 0.55 | 0.90 | 0.68 | 0.12 | 0.30 | 0.55 | 0.70 | 0.52 | 0.08 | 0.30 | 0.35 | 0.80 | 0.48 | 0.18 |
| $\mathbf{K_{II}}$ | 0.65 | 0.60 | 0.70 | 0.65 | 2.02 | 0.44 | 0.45 | 0.52 | 0.42 | 1.50 | 0.36 | 0.40 | 0.50 | 0.49 | 1.05 |
| $\mathcal{N}_{g}$ | Update |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-2 | LLaMA2-7b | LLaMA3-8b |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Rel | Gen | Loc | Avg | $\Delta$ PPL | Rel | Gen | Loc | Avg | $\Delta$ PPL | Rel | Gen | Loc | Avg | $\Delta$ PPL |  |
| $\mathbf{K_{I}}$ | 0.56 | 0.48 | 0.88 | 0.64 | 0.13 | 0.32 | 0.59 | 0.82 | 0.68 | 0.10 | 0.44 | 0.41 | 0.74 | 0.53 | 0.22 |
| $\mathbf{K_{II}}$ | 0.54 | 0.55 | 0.74 | 0.61 | 1.88 | 0.40 | 0.43 | 0.62 | 0.48 | 1.16 | 0.29 | 0.33 | 0.66 | 0.43 | 0.93 |

##### Findings

Table 2 presents the results of this experiment, leading us to the following conclusions:
(1) For $\text{K}_{\text{II}}$, modifying $\mathcal{N}_{i}$ (Local set) results in a low generalization, indicating unsuccessful modification of neighbor queries. In contrast, for $\text{K}_{\text{I}}$, both “Avg” and “Gen” metrics are higher than those of $\text{K}_{\text{II}}$. This suggests higher consistency among neighbor KNs for $\text{K}_{\text{I}}$, while neighbor KNs for $\text{K}_{\text{II}}$ are highly inconsistent.
(2) For $\text{K}_{\text{II}}$, we must modify $\mathcal{N}_{g}$ (Global set) to ensure high generalization, meaning we must modify numerous KNs to erase or update a single fact. However, this also leads to a high $\Delta\text{PPL}$, indicating excessive alteration of model parameters. This suggests that the fact cannot be localized to a few KNs. Taken together, (1) and (2) confirm that $\text{K}_{\text{II}}$ does not adhere to the KL assumption.

## 3 Query Localization Assumption

Figure: Figure 4: Results of the Query-KN Mapping demonstration experiment. “Self ($\mathcal{N}_{i}$)” refers to the KNs associated with a specific query, while the other three scenarios pertain to the neighbor KNs. We adopt three sets for $\mathcal{N}_{i}^{N}$: Union, Intersection, and Refine, which respectively unionize, intersect, and refine the KNs to select those that appear more than once. “Enh” and “Sup” refer to enhancement and suppression of KN activation values, respectively, with “Avg” representing their average. The values in the figure represent $\Delta\text{Prob}$, the change rate in the model’s answer probabilities before and after modifying KN activation values.
Refer to caption: x6.png

##### Motivation

Since $\text{K}_{\text{II}}$ does not satisfy the KL assumption, we naturally raise question Q2: What is a more realistic assumption? Tables 1 and 2 suggest that for $\text{K}_{\text{II}}$, KNs might relate to queries rather than facts.
Then, we further consider: if a fact is stored across many KNs, how does the model select the appropriate set of KNs to answer queries correctly? The KL assumption only considers how MLP stores knowledge, without considering how the model selects KNs for answering. In light of research on the attention module [, , ], we argue that it should be considered and hypothesize that it plays a role in selecting KNs.
In summary, we propose the Query Localization (QL) assumption, which includes two aspects.
Query-KN Mapping (Hypothesis 1): For ${\text{K}_{\text{II}}}$, the localization results are associated with the query rather than the fact.
Dynamic KN Selection (Hypothesis 2): For a specific query, the attention module contributes to the selection of KNs for answering a query.
In fact, the KL assumption can be considered as a simplification of the QL assumption when KNs happen to overlap, and it is entirely feasible to analyze ${\text{K}_{\text{I}}}$ using the QL assumption. Moreover, the proof of the QL assumption can further illustrate the issues of the KL assumption. Below, we divide the facts into ${\text{K}_{\text{I}}}$ and $\text{K}_{\text{II}}$ to conduct experiments and prove the QL assumption.

### 3.1 Demonstration of the Query Localization Assumption: Query-KN Mapping

##### Experimental Settings

We first prove the Query-KN Mapping hypothesis. Tables 1 and 2 offer preliminary evidence, and now we directly manipulate KN activation values to reinforce this point. Unlike modifying model parameters, modifying KN activation values directly reflect the impact of KNs on the answer [, ]. For a given query $q_{i}$, we either suppress or enhance its corresponding KNs $\mathcal{N}_{i}$. In contrast, we suppress or enhance the set of its neighbor KNs $\mathcal{N}_{i}^{N}$. Here, we adopt three methods for obtaining $\mathcal{N}_{i}^{N}$.

$$ $\mathcal{N}_{i}^{N}=\begin{cases}\bigcup_{j=1}^{k}\mathcal{N}_{j}(j\neq i),&\text{for Union}\\ \bigcap_{j=1}^{k}\mathcal{N}_{j}(j\neq i),&\text{for Intersection}\\ \{n\mid n\in\bigcup_{j=1}^{k}\mathcal{N}_{j}\text{ and }\sum_{j=1}^{k}\mathbbm{1}(n\in\mathcal{N}_{j})>1\}\quad(j\neq i),&\text{for Refine}\end{cases}$ (3) $$

where $n$ is a KN, $\sum_{j=1}^{k}\mathbbm{1}(n\in\mathcal{N}_{j})$ counts the occurrences of $n$.
Regarding evaluation metrics, we follow the established practice in knowledge localization methods [, , , ], and calculate the rates of increase and decrease in the PLMs’ answer probabilities before (b) and after (a) suppression and enhancement: $\Delta\text{Prob}=\pm\frac{\text{Prob}_{{a}}-\text{Prob}_{{b}}}{\text{Prob}_{{b}}}$, where “$+$” indicates an increase and “${-}$” a decrease.

Figure: Figure 5: Experimental results of the distribution of the $\mathcal{P}$ values. “I” and “II”represent $\text{K}_{\text{I}}$ and $\text{K}_{\text{II}}$ respectively. For example, “GPT2:I” represents the $\mathcal{P}$ values of facts belonging to $\text{K}_{\text{I}}$ under the GPT-2.
Refer to caption: x7.png

##### Findings

Figure 4 demonstrates that the Query-KN Mapping of the QL assumption is established.
(1) Regardless of whether it is $\text{K}_{\text{I}}$ or $\text{K}_{\text{II}}$, the results from the “Self ($\mathcal{N}_{i}$)” suggest that the influence of suppressing or enhancing the query’s own KNs is significant. This indicates that the KNs are associated with the query.
(2) For $\text{K}_{\text{II}}$, the average values (“Avg”) of $\mathcal{N}_{i}^{N}$ are much lower than that of $\mathcal{N}_{i}$, especially in the strictest “Intersection” case. However, for $\text{K}_{\text{I}}$, the average values decreases less.
This indicates that the neighbor KNs for $\text{K}_{\text{I}}$ are more consistent, while the neighbor KNs for $\text{K}_{\text{II}}$ are much less consistent. This shows that for $\text{K}_{\text{II}}$, KNs do not correspond to the fact itself.
Combining (1) and (2), we conclude that KNs are associated with the query rather than the fact, i.e., Query-KN Mapping is established.
(3) Furthermore, for $\text{K}_{\text{I}}$, the average values also decrease, indicating that its neighbor KNs are not entirely consistent. Thus, $\text{K}_{\text{I}}$ can be considered a special case of $\text{K}_{\text{II}}$.

### 3.2 Demonstration of the Query Localization Assumption: Dynamic KN Selection

In this subsection, we first explain why we propose the Dynamic KN Selection hypothesize,
then, we present experimental evidence to support this hypothesis.

#### 3.2.1 Preliminary Analysis of Attention Module

To examine the role of the attention module, we conduct a preliminary experiment.
Specifically, we identify the token with the highest attention score and define the predominance score $\mathcal{P}$. For a fact with $k$ queries $\{q_{1},q_{2},\ldots,q_{k}\}$, its $\mathcal{P}$ is calculated as follows:

$$ $A^{(r,c)}_{i}=\frac{1}{LH}\sum_{l=1}^{L}\sum_{h=1}^{H}A^{(r,c)}_{(l,h)}\Rightarrow\quad t_{i}=q_{i}[\arg\max_{r}\sum_{c}A^{(r,c)}_{i}]\Rightarrow\quad\mathcal{P}=\max_{t}\frac{\sum_{i=1}^{k}\mathbbm{1}(t=t_{i})}{k}$ (4) $$

where $A^{(r,c)}_{i}$ represents the attention score matrix for query $q_{i}$, with $r$ and $c$ denoting rows and columns. $L$ and $H$ represent the number of layers and heads of the attention module, $t_{i}$ is the most attended token for $q_{i}$, and $\sum_{i=1}^{k}\mathbbm{1}(t=t_{i})$ counts the occurrences of each token.

Figure: Figure 6: Results for the Dynamic KN Selection demonstration. “Self ($\mathcal{S}$)” indicates knowledge synapses corresponding to the query itself, while “Irrelevant ($\mathcal{S}_{ir}$)” refers to those corresponding to an unrelated query. “$\Delta$ Value” and “$\Delta$ Prob” represent the rate of change in the average KN activation value and the PLMs’ answer probabilities, respectively. “Enh” and “Sup” refer to enhancement and suppression of the KS attention scores, respectively, with “Avg” representing their average.
Refer to caption: x8.png

Figure: Figure 7: Heatmaps showing the neuron activation values, after suppressing knowledge synapses. The queries used here are the same as those in Figure 1. The dark areas in Figure 1 appear lighter here, indicating a decrease in the activation value of KNs. For the enhanced case, see Figure 10 in A.3.
Refer to caption: x9.png

The violin plot in Figure 5 illustrates the distributions of $\mathcal{P}$. The high values of $\mathcal{P}$ suggest that the attention module tends to focus on specific tokens related to the dynamic context of the query. This observation leads us to propose the Dynamic KN Selection hypothesis, which posits that the attention module is not only “related” to the context but actively selects relevant KNs for a given query.

#### 3.2.2 Demonstration of Dynamic KN Selection

##### Methods

To prove the Dynamic KN Selection hypothesis, we either suppress or enhance attention scores related or unrelated to the query.
Notably, attention score matrices resemble KN activation value matrices, differing only in an additional dimension for attention heads.
Thus, similar to the definition of KNs, we identify column vectors with high attention scores. Drawing inspiration from cognitive science [, , , , ], we refer to these vectors as Knowledge Synapses (KSs), denoted as $\mathcal{S}$. Given a query with its answer, the KSs are defined as:

$$ $\tau=\alpha\cdot\frac{1}{L\cdot H}\sum_{l=1}^{L}\sum_{h=1}^{H}\sum_{r=1}^{R}\sum_{c=1}^{C}A_{(l,h)}^{(r,c)}$ (5) $$

$$ $\mathcal{S}=\left\{(c,l,h)\mid\sum_{r=1}^{R}A_{(l,h)}^{(r,c)}>\tau,\;\forall\;l\in\{1,\ldots,L\},h\in\{1,\ldots,H\},c\in\{1,\ldots,C\}\right\}$ (6) $$

where $\tau$ is the dynamic threshold, $\alpha$ is a scaling factor, and $A$ is the attention score matrix. $L$ and $H$ represent the number of layers and heads of the attention module, respectively, while $R$ and $C$ are the rows and columns of $A$. $l$, $h$, $r$, $c$ are the corresponding indices.
After localizing $\mathcal{S}$, we enhance or suppress the attention scores at these positions (i.e., KS attention scores) to study their effects. As a comparison, we randomly select knowledge synapses $\mathcal{S}_{ir}$ corresponding to an irrelevant fact and perform the same operations.

##### Evaluation Metrics

First, we assess the impact of manipulating knowledge synapses on the activation values of KNs. We calculate the rates of increase and decrease in the average KN activation values ($\overline{\text{Value}}$) before (b) and after (a) KS manipulation: $\Delta\text{Value}=\pm\frac{\overline{\text{Value}_{a}}-\overline{\text{Value}_{b}}}{\overline{\text{Value}_{b}}}$. Second, we compute the change in the PLMs’ answer prediction probability ($\Delta\text{Prob}$). For ease of comparison, we average the results for suppression and enhancement in both settings.

##### Findings

Figure 7 shows a case, and Figure 6 presents the overall results that establish the Dynamic KN Selection of the QL assumption.
(1)
Manipulating “Self($\mathcal{S}$)”, the attention scores related to the query, leads to corresponding changes in both KN activation value and the PLMs’ answer probability. In contrast, manipulating
“$\mathcal{S}_{ir}$” has a smaller effect, confirming that the changes are indeed caused by manipulating $\mathcal{S}$.
(2)
Figure 7 shows that suppressing KSs reduces KN activation values, creating confusion between KNs and other neurons. This makes it difficult for the attention module to select KNs and provide correct answers.
Points (1) and (2) collectively demonstrate that attention module contributes to the selection of KNs, i.e., Dynamic KN Selection is established.
(3) Furthermore, compared to ${\text{K}_{\text{I}}}$, $\text{K}_{\text{II}}$ relies more on the attention module. As shown in Figure 6, manipulating the KSs has a larger impact on both $\Delta\text{Value}$ and $\Delta\text{Prob}$ for $\text{K}_{\text{II}}$ than for $\text{K}_{\text{I}}$.

**Table 3: Results of Consistency-Aware KN Modification. The metrics and settings are consistent with Table 2.**
| Method | Erasure |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-2 | LLaMA2-7b | LLaMA3-8b |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Rel | Gen | Loc | Avg | $\Delta$ PPL | Rel | Gen | Loc | Avg | $\Delta$ PPL | Rel | Gen | Loc | Avg | $\Delta$ PPL |  |
| $\text{K}_{\text{I}}$ ($\mathcal{N}_{i}$) | 0.55 | 0.47 | 0.93 | 0.65 | 0.02 | 0.33 | 0.34 | 0.79 | 0.49 | 0.01 | 0.28 | 0.30 | 0.83 | 0.47 | 0.04 |
| $\text{K}_{\text{I}}$ ($\mathcal{N}_{g}$) | 0.58 | 0.55 | 0.90 | 0.68 | 0.12 | 0.30 | 0.55 | 0.70 | 0.52 | 0.08 | 0.30 | 0.35 | 0.80 | 0.48 | 0.18 |
| $\mathbf{K_{I}}$ (Ours) | 0.56 | 0.50 | 0.90 | 0.65 | 0.03 | 0.32 | 0.44 | 0.88 | 0.55 | 0.03 | 0.30 | 0.33 | 0.80 | 0.48 | 0.02 |
| $\text{K}_{\text{II}}$ ($\mathcal{N}_{i}$) | 0.50 | 0.09 | 0.97 | 0.52 | 0.06 | 0.36 | 0.11 | 0.80 | 0.42 | 0.03 | 0.34 | 0.04 | 0.90 | 0.43 | 0.05 |
| $\text{K}_{\text{II}}$ ($\mathcal{N}_{g}$) | 0.65 | 0.60 | 0.70 | 0.65 | 2.02 | 0.44 | 0.45 | 0.52 | 0.47 | 1.50 | 0.36 | 0.40 | 0.50 | 0.42 | 1.05 |
| $\mathbf{K_{II}}$ (Ours) | 0.55 | 0.40 | 0.90 | 0.62 | 0.10 | 0.35 | 0.35 | 0.77 | 0.49 | 0.06 | 0.34 | 0.30 | 0.88 | 0.51 | 0.09 |
| Method | Update |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-2 | LLaMA2-7b | LLaMA3-8b |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Rel | Gen | Loc | Avg | $\Delta$ PPL | Rel | Gen | Loc | Avg | $\Delta$ PPL | Rel | Gen | Loc | Avg | $\Delta$ PPL |  |
| $\text{K}_{\text{I}}$ ($\mathcal{N}_{i}$) | 0.53 | 0.40 | 0.99 | 0.64 | 0.04 | 0.30 | 0.39 | 0.89 | 0.53 | 0.03 | 0.30 | 0.29 | 0.79 | 0.46 | 0.07 |
| $\text{K}_{\text{I}}$ ($\mathcal{N}_{g}$) | 0.56 | 0.48 | 0.88 | 0.64 | 0.13 | 0.32 | 0.59 | 0.82 | 0.68 | 0.10 | 0.44 | 0.41 | 0.74 | 0.53 | 0.22 |
| $\mathbf{K_{I}}$ (Ours) | 0.55 | 0.44 | 0.98 | 0.66 | 0.14 | 0.30 | 0.50 | 0.88 | 0.56 | 0.10 | 0.35 | 0.33 | 0.70 | 0.46 | 0.10 |
| $\text{K}_{\text{II}}$ ($\mathcal{N}_{i}$) | 0.44 | 0.11 | 0.96 | 0.50 | 0.09 | 0.39 | 0.07 | 0.80 | 0.42 | 0.08 | 0.29 | 0.08 | 0.86 | 0.41 | 0.08 |
| $\text{K}_{\text{II}}$ ($\mathcal{N}_{g}$) | 0.54 | 0.55 | 0.74 | 0.61 | 1.88 | 0.40 | 0.43 | 0.62 | 0.48 | 1.16 | 0.29 | 0.33 | 0.66 | 0.43 | 0.93 |
| $\mathbf{K_{II}}$ (Ours) | 0.45 | 0.45 | 0.88 | 0.59 | 0.20 | 0.40 | 0.38 | 0.75 | 0.51 | 0.12 | 0.29 | 0.29 | 0.80 | 0.46 | 0.14 |

### 3.3 Application of Query Localization Assumption: Consistency-Aware KN Modification

##### Experimental Setups

Inspired by Query-KN Mapping hypothesis of the QL assumption, we propose a new approach to KN selection that improves knowledge modification methods. By incorporating KN consistency, we introduce the Consistency-Aware Score (CAS) metric, which penalizes low consistency and rewards high activation values. Given a fact with $k$ queries, the CAS for the $p$-th neuron in the $l$-th MLP is defined as:

$$ $CAS_{(l,p)}=\beta_{1}\mu_{lp}-\beta_{2}\sigma_{lp},\quad\text{ where }\textbf{ }\mu_{lp}=\frac{1}{k}\sum_{i=1}^{k}as_{lp}^{(i)},\quad\sigma_{lp}=\sqrt{\frac{1}{k}\sum_{i=1}^{k}(as_{lp}^{(i)}-\mu_{lp})^{2}}$ (7) $$

where $\beta_{1}$ and $\beta_{2}$ are hyperparameters, $\mu_{lp}$ and $\sigma_{lp}$ represent the mean and variance, and $as_{lp}^{(i)}$ is the activation score at position $(l,p)$ for $q_{i}$. Then, using thresholding techniques, we identify positions with high CAS values as the KNs to be edited and conduct experiments similar to those in §2.2.

##### Findings

Table 3 presents the results, from which we conclude:
(1)
Our method exhibits superior and more balanced performance, particularly for $\text{K}_{\text{II}}$. For $\text{K}_{\text{II}}$, the “Avg” values of $\text{K}_{\text{II}}$(Ours) are notably higher than those of $\text{K}_{\text{II}}$($\mathcal{N}_{i}$), especially in terms of generalization. Simultaneously, it surpasses or closely matches the “Avg” values of $\text{K}_{\text{II}}$($\mathcal{N}_{g}$), especially in locality.
Furthermore, the increase in PPL is much lower than that of $\text{K}_{\text{II}}$($\mathcal{N}_{g}$) and comparable to $\text{K}_{\text{II}}$($\mathcal{N}_{i}$), suggesting the small impact on the model’s general capabilities.
(2)
Our method is also shown to be effective for $\text{K}_{\text{I}}$. The performance of $\text{K}_{\text{I}}$(Ours) is comparable to both $\text{K}_{\text{I}}$($\mathcal{N}_{i}$) and $\text{K}_{\text{I}}$($\mathcal{N}_{g}$). This indicates that even facts adhering to the KL assumption can be effectively analyzed using the QL assumption, further illustrating the limitations of the KL assumption and showing that it is merely a simplification of the QL assumption.

## 4 Related Work

LLMs store extensive factual knowledge [], prompting numerous studies to investigate the mechanisms behind their storage and expression.
Geva et al. [] propose that MLP modules simulate key-value memories to store information, and Dai et al. [] propose the concept of knowledge neurons (KNs), suggesting that these MLP modules can store “knowledge”.
The success of KN-inspired model editing methods [, , ] further supports the plausibility of the KN thesis. Additionally, the integrated gradients (IG) method [] has proven suitable for knowledge localization [], leading to further refinements such as Sequential IG, Discretized IG and the Architecture adapted Multilingual IG [, , , , ].
Further investigations reveal that some KNs exhibit cross-lingual features [, , , ], while others display language-specific characteristics [, , ]. Some KNs also show degeneracy, with multiple neurons redundantly encoding the same factual knowledge []. These studies collectively advance the KN thesis.
Beyond MLP modules, some studies incorporate attention modules [] into factual knowledge research. They find that attention modules play a role in the PLMs’ internal information flow, aiding in factual knowledge extraction []. Moreover, attention modules can facilitate in-context learning [] and relate to token expression [].

However, the KN thesis has its limitations. Niu et al. [] argue that it oversimplifies the real situation,
while Hase et al. [] suggest that the location of knowledge localization may not align with the location of greatest impact on knowledge expression.
Additionally, the research has found that neurons can be further decomposed into features [],
and the activation of a single neuron can have different meanings in different contexts [].
Limitations in KN-inspired knowledge editing methods have also been identified [, , , , , , , ]. These model editing methods may fail to edit successfully or impair the PLMs’ general capabilities, indirectly suggesting issues with the KN thesis.
However, previous works mainly observe problems with the KN thesis and identify phenomena through experiments, without deeply analyzing their causes or providing effective solutions.
Our work not only analyzes the inherent issues of the KN thesis but also proposes solutions.

## 5 Conclusion and Future Work

This paper investigates the Knowledge Localization (KL) assumption of the knowledge neuron (KN) thesis, which posits that a piece of factual knowledge can be localized to several knowledge neurons. We re-examine this assumption and confirm that much factual knowledge does not conform to it. Furthermore, we propose and validate the Query Localization (QL) assumption, which includes two aspects:
Query-KN Mapping and Dynamic KN Selection. Extensive experiments validate the QL assumption and suggest that the KL assumption is essentially a simplification of the QL assumption.
Future work could delve into the reasons behind the existence of $\text{K}_{\text{II}}$ and explore how to further utilize the QL assumption, along with the properties of $\text{K}_{\text{II}}$, to improve the model editing method.

## Appendix A Appendix

### A.1 Specific Experimental Settings

##### Hardware spcification and environment.

We ran our experiments on the machine equipped with the following specifications:

- •
CPU: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, Total CPUs: 56
- •
GPU:
–
NVIDIA GeForce RTX 3090 $\times$ 20. The Standard Memory Config is 24 GB GDDR6X.
–
NVIDIA A100 80GB PCIe $\times$ 4. The GPU Memory is 80GB HBM2e.
- •
Software:
–
Python Version: 3.10.10
–
PyTorch Version: 2.0.0+cu117

In the experiments, the main computational expense was associated with acquiring knowledge neurons since the method for knowledge localization computes the activation values of all neurons. For GPT-2, LLaMA2-7b, and LLaMA3-8b, the time required to acquire KNs once was approximately: 20 seconds, 5 minutes, and 5 minutes, respectively. As we conducted our experiments using multi-GPU distributed processing, the total time spent was about 26 days. The computational expense for the other experiments was not significant, and they could be completed within 10 days. Due to the lengthy computation times, we tested the code and results by selecting one datum from each relation, thus the test dataset comprised only 36 data points. We did conduct some erroneous experiments, but the run-time costs for these were negligible due to the small size of the test dataset. We recommend that readers adopt a similar approach for testing their code.

##### Experimental Hyperparameters of Consistency Analysis

We provide the thresholds used for each setting in Table 4, corresponding to Table 1. The Otsu threshold is calculated separately based on each batch of data, thus each is different. The static threshold is set by us, thus it is the same.

##### Experimental Hyperparameters of KN Modification

In Equation 2, we set $\lambda_{1}=\lambda_{2}=2$.

##### Experimental Hyperparameters of Obtaining Knowledge Synapses

In Equations 5 and 6, the scaling factor $\tau$ is the same for all three PLMs, with $\tau=0.3$.

##### Experimental Hyperparameters of Consistency-Aware KN Modification

In Equation 7, we set $\beta_{1}=0.7$ and $\beta_{2}=0.3$. For the selection of threshold, we consider the dynamic threshold to find the maximum value of CAS, and neurons larger than 0.3 times are selected as KNs.

**Table 4: This table corresponds to Table 1 and lists the thresholds for each experimental setting.**
| $\mathbf{T}$ | GPT2 |  |  |
| --- | --- | --- | --- |
| Dai et al. [] | Enguehard [] | Chen et al. [] |  |
| Static | 0.1 | 0.1 | 0.1 |
| Otsu | 0.146 | 0.150 | 0.148 |
| $\mathbf{T}$ | LLaMA2-7b |  |  |
| Dai et al. [] | Enguehard [] | Chen et al. [] |  |
| Static | 0.1 | 0.1 | 0.1 |
| Otsu | 0.170 | 0.169 | 0.170 |
| $\mathbf{T}$ | LLaMA3-8b |  |  |
| Dai et al. [] | Enguehard [] | Chen et al. [] |  |
| Static | 0.1 | 0.1 | 0.1 |
| Otsu | 0.080 | 0.082 | 0.081 |

### A.2 Experimental Dataset Introduction

In our experiments, we selected the ParaRel dataset [], a high-quality resource of cloze-style query English paraphrases. It contains a total of 328 paraphrases for 38 relations. We further conducted a basic filtering, excluding 2 relations that had no paraphrases. Table 5 displays these relations and corresponding example data.

**Table 5: Example data of the ParaRel dataset [].**
| Relation | Example data |  |
| --- | --- | --- |
| Example Query | Answer |  |
| P39 | Adrian IV has the position of | pope |
| P264 | Purple Hearts is represented by music label | Sunshine |
| P37 | The official language of Republic of Ingushetia is | Russian |
| P108 | Henry Swanzy works for | BBC |
| P131 | Heaton Park is located in | Manchester |
| P103 | The native language of Francis Ponge is | French |
| P176 | Fiat Grande Punto is produced by | Fiat |
| P30 | Somalia is located in | Africa |
| P178 | Gain Ground is developed by | Sega |
| P138 | International Day for Biological Diversity is named after | biodiversity |
| P47 | Ukraine shares border with | Poland |
| P17 | Media Development Authority is located in | Singapore |
| P413 | Joe Torre plays in [MASK] position. | catcher |
| P27 | Edward Wollstonecraft is [MASK] citizen. | Australia |
| P463 | Chuck Schuldiner is a member of | Death |
| P364 | The original language of NU.nl is | Dutch |
| P495 | The Creepshow was created in | Canada |
| P449 | Yes Minister was originally aired on | BBC |
| P20 | Margaret Cavendish, Duchess of Newcastle-upon-Tyne died in | England |
| P1376 | Rumbek is the capital of | Lakes |
| P1001 | Minister for Foreign Affairs is a legal term in | Australia |
| P361 | propellant is part of | cartridge |
| P36 | The capital of Flanders is | Brussels |
| P1303 | Ludovico Einaudi plays | piano |
| P530 | Brunei maintains diplomatic relations with | Australia |
| P19 | Lopo Soares de Albergaria was born in | Lisbon |
| P190 | Bratislava and [MASK] are twin cities. | Dublin |
| P740 | Shirehorses was founded in | Manchester |
| P136 | Frank Mantooth plays [MASK] music. | jazz |
| P127 | AVCHD is owned by | Sony |
| P1412 | Karl Bodmer used to communicate in | French |
| P407 | Zarez was written in | Croatian |
| P140 | Leo IX is affiliated with the [MASK] religion. | Christianity |
| P279 | quinquina is a subclass of | wine |
| P276 | Al-Rifa’i Mosque is located in | Cairo |
| P159 | The headquarter of Allied Command Transformation is in | Norfolk |
| P106 | Giuseppe Saracco is a [MASK] by profession. | politician |
| P101 | Aleksei N. Leontiev works in the field of | psychology |
| P937 | Joseph Chamberlain used to work in | London |

### A.3 Supplementary Experimental Results

##### Supplementary Experimental Results of Consistency Analysis (§ 2.1 )

Figure 8 and Figure 9 show two violin plots. The experimental settings are exactly the same as Figure 3, but the knowledge positioning is different. Figure 8 and Figure 9 adopt the knowledge positioning methods proposed by Enguehard [] and Chen et al. [] respectively.

Figure: Figure 8: Violin Plot of Consistency Analysis. The experimental settings are exactly the same as Figure 3, but the knowledge positioning method used here is the method proposed by Enguehard [].
Refer to caption: x10.png

Figure: Figure 9: Violin Plot of Consistency Analysis. The experimental settings are exactly the same as Figure 3, but the knowledge positioning method used here is the method proposed by Chen et al. [].
Refer to caption: x13.png

##### Supplementary Experimental Results of Heatmap of Neuron Activations (Figures 1 and 7 )

Figure 10 shows the neuron activation values under three conditions: 1. No manipulation of knowledge synapses, 2. Suppressing knowledge synapses, and 3. Enhancing knowledge synapses. The chosen queries remain consistent.

Figure: (a) Does not manipulate knowledge synapses.
Refer to caption: x16.png

### A.4 Knowledge Localization Methods

We have adopted three advanced knowledge localization methods, and the specific experimental settings remain consistent with the original author. Below we introduce their specific details.

##### Integrated Gradients [ 7 ]

Dai et al. [] propose the IG method.
Given an input prompt $x$, the method defines the model output $\operatorname{P}_{x}(\hat{w}^{(l)}_{i})$ as the probability of the correct answer predicted by a pretrained model:

$$ $\operatorname{P}_{x}(\hat{w}^{(l)}_{i})=p(y^{*}\mid x,w^{(l)}_{i}=\hat{w}^{(l)}_{i})$ (8) $$

where $y^{*}$ is the correct answer, $w^{(l)}_{i}$ is the $i$-th intermediate neuron in the $l$-th MLP, and $\hat{w}^{(l)}_{i}$ is a constant assigned to $w^{(l)}_{i}$.

To calculate the attribution score of a neuron $\operatorname{Attr}(w^{(l)}_{i})$, they change $w^{(l)}_{i}$ gradually from 0 to its original value $\overline{w}^{(l)}_{i}$ and integrate the gradients to determine the impact of the neuron:

$$ $\operatorname{Attr}(w^{(l)}_{i})=\overline{w}^{(l)}_{i}\int_{\alpha=0}^{1}\frac{\partial\operatorname{P}_{x}(\alpha\overline{w}^{(l)}_{i})}{\partial w^{(l)}_{i}}\,d\alpha$ (9) $$

where $\frac{\partial\operatorname{P}_{x}(\alpha\overline{w}^{(l)}_{i})}{\partial w^{(l)}_{i}}$ is the gradient of the model output with respect to $w^{(l)}_{i}$.
As $\alpha$ changes from 0 to 1, integrating the gradients allows the attribution score to accumulate the change in output probability caused by modifying $w^{(l)}_{i}$. If a neuron significantly influences factual expressions, its gradient will be more salient, leading to larger integrated values. Thus, the attribution score measures the contribution of a neuron $w^{(l)}_{i}$ to factual expression.

Calculating the continuous integral directly is challenging, thus they approximate it using a Riemann sum:

$$ $\tilde{\operatorname{Attr}}(w^{(l)}_{i})=\frac{\overline{w}^{(l)}_{i}}{m}\sum_{k=1}^{m}\frac{\partial\operatorname{P}_{x}\left(\frac{k}{m}\overline{w}^{(l)}_{i}\right)}{\partial w^{(l)}_{i}}$ (10) $$

where $m=20$ is the number of approximation steps.
With the attribution algorithm, they identify a coarse set of knowledge neurons by selecting those whose attribution scores exceed a predefined threshold.

##### Sequential Integrated Gradients [ 10 ]

Enguehard [] propose the SIG method.
A language model is defined as a function:

$$ $\textrm{F}(\mathbf{x}):\mathbb{R}^{m\times n}\rightarrow\mathbb{R}$ (11) $$

where $\mathbf{x}$ is a sequence of $m$ words, each represented by $n$ features, typically derived from an embedding layer. The output of the function F is a single value, such as a sentiment score for a given sentence. $\mathbf{x}_{i}$ denotes the $i$-th word of a sequence, and $x_{ij}$ represents the $j$-th feature of the $i$-th word. For each word $\mathbf{x}_{i}$, a baseline $\overline{\mathbf{x}}^{i}$ is defined as the original sequence with $\mathbf{x}_{i}$ replaced by a mask token:

$$ $\overline{\mathbf{x}}^{i}=(\mathbf{x}_{1},\ldots,\texttt{<mask>},\ldots,\mathbf{x}_{m})$ (12) $$

where the mask token replaces only the word of interest. If the model lacks a “mask” token (e.g., GPT-2), a “pad” token is used.

SIG calculates the attribution score for a word $\mathbf{x}_{i}$ and feature $j$ as:

$$ $\textrm{SIG}_{ij}(\mathbf{x}):=(x_{ij}-\overline{x}_{ij})\times\int_{0}^{1}\frac{\partial\textrm{F}(\overline{\mathbf{x}}^{i}+\alpha\times(\mathbf{x}-\overline{\mathbf{x}}^{i}))}{\partial x_{ij}}\,d\alpha$ (13) $$

where the integral measures the gradient of F along the straight line between $\overline{\mathbf{x}}^{i}$ and $\mathbf{x}$.
Similar to the original Integrated Gradients (IG) method, the integral is approximated using a Riemann sum. The integral is evaluated across a straight line path between the baseline $\overline{\mathbf{x}}^{i}$ and the original input $\mathbf{x}$.
The total attribution score for a word $\mathbf{x}_{i}$ is computed by summing across all features $j$ and normalizing the result:

$$ $\textrm{SIG}_{i}(\mathbf{x}):=\frac{\sum_{j}\textrm{SIG}_{ij}}{||\textrm{SIG}||}$ (14) $$

##### Architecture-adapted Multilingual Integrated Gradients [ 4 ]

Chen et al. [] propose the AMIG method.
Given a query $q$, they define the probability of the correct answer predicted by a PLMs as follows:

$$ $\operatorname{F}(\hat{w}^{(l)}_{j})=p(y^{*}|q,w^{(l)}_{j}=\hat{w}^{(l)}_{j})$ (15) $$

Here, $y^{*}$ represents the correct answer, $w^{(l)}_{j}$ denotes the $j$-th neuron in the $l$-th layer, and $\hat{w}^{(l)}_{j}$ is the specific value assigned to $w^{(l)}_{j}$. To calculate the attribution score for each neuron, they employ the technique of integrated gradients.
To compute the attribution score of a neuron $w^{(l)}_{j}$, they consider the following formulation:

$$ $\operatorname{Attr}({w}^{(l)}_{j})=(\overline{w}^{(l)}_{j}-{w^{\prime}}^{(l)}_{j})\int_{0}^{1}\frac{\partial\operatorname{F}({w^{\prime}}^{(l)}_{j}+\alpha(\overline{w}^{(l)}_{j}-{w^{\prime}}^{(l)}_{j}))}{\partial{w}^{(l)}_{j}}\,d\alpha$ (16) $$

Here, $\overline{w}^{(l)}_{j}$ represents the actual value of $w^{(l)}_{j}$,
$w^{\prime(l)}_{j}$ serves as the baseline vector for $w^{(l)}_{j}$. The term $\frac{\partial\operatorname{F}(w^{\prime(l)}_{j}+\alpha(w^{(l)}_{j}-w^{\prime(l)}_{j}))}{\partial w^{(l)}_{j}}$ computes the gradient with respect to $w^{(l)}_{j}$.
Next, they aim to obtain ${w^{\prime}}^{(l)}_{j}$. Starting from the sentence $q$, they acquire a baseline sentence and then encode this sentence as a vector.
Let the baseline sentence corresponding to $q_{i}$ be $q^{\prime}_{i}$, and $q^{\prime}_{i}$ consists of $m$ words, maintaining a length consistent with $q$, denoted as $q^{\prime}_{i}=(q^{\prime}_{i1}\ldots q^{\prime}_{ik}\ldots q^{\prime}_{im})$. Since they are using auto-regressive models, according to Chen et al. [], $q^{\prime}_{ik}=\langle\text{eos}\rangle$, where $\langle\text{eos}\rangle$ represents “end of sequence” in auto-regressive models.
The attribution score $Attr_{i}(w_{j}^{(l)})$ for each neuron, given the input $q_{i}$, can be determined using Equation (16). For the computation of the integral, the Riemann approximation method is employed:

$$ ${Attr_{i}(w_{j}^{l})}\approx\frac{\overline{w}^{(l)}_{j}}{N}\sum_{k=1}^{N}\frac{\partial F({w^{\prime}}^{(l)}_{j}+\frac{k}{N}\times(\overline{w}^{(l)}_{j}-{w^{\prime}}^{(l)}_{j})}{\partial{w}^{(l)}_{j}}$ (17) $$

where $N$ is the number of approximation steps.
Then, the attribution scores for each word $q_{i}$ are aggregated and subsequently normalized:

$$ $Attr(w_{j}^{l})=\frac{\sum_{i=1}^{m}Attr_{i}(w_{j}^{l})}{\sum_{j=1}^{n}\sum_{i=1}^{m}Attr_{i}(w_{j}^{l})}$ (18) $$

Let $\mathcal{N}$ be the set of neurons classified as knowledge neurons based on their attribution scores exceeding a predetermined threshold $\tau$, for a given input $q$. This can be formally defined as:

$$ $\mathcal{N}=\left\{w_{j}^{(l)}\,\middle|\,Attr(w_{j}^{(l)})>\tau\right\}$ (19) $$

where $l$ encompassing all layers and $j$ including all neurons within each layer.

### A.5 Metrics for Knowledge Editing

In Table 2 and Table3, there are three indicators reliability, generalization, and locality, which represent the effect of knowledge modification. In fact, we are inspired by the field of knowledge editing [], below we will give a complete introduction.

Model editing focuses on modifying the behavior of a base model $f_{\theta}$ (where $\theta$ represents the model parameters) given an edit descriptor $(x_{e},y_{e})$. The objective is to produce an edited model $f_{\theta_{e}}$ that incorporates the desired changes efficiently without affecting the model’s performance on unrelated samples.
The base model $f_{\theta}$ maps inputs to predictions:

$$ $f:\mathbb{X}\mapsto\mathbb{Y}$ (20) $$

where $x$ is the input and $y$ is the corresponding prediction. The edit descriptor $(x_{e},y_{e})$ specifies an input $x_{e}$ and a desired output $y_{e}$. If the original model does not yield the expected output ($f_{\theta}(x_{e})\neq y_{e}$), the post-edit model $f_{\theta_{e}}$ should return the correct prediction:

$$ $f_{\theta_{e}}(x_{e})=y_{e}$ (21) $$

The editing process generally affects predictions for a range of inputs closely related to the edit descriptor, termed the editing scope. A successful edit modifies predictions within this scope while leaving predictions outside it unchanged:

$$ $f_{\theta_{e}}(x)=\begin{cases}y_{e}&\text{if }x\in I(x_{e},y_{e})\\ f_{\theta}(x)&\text{if }x\in O(x_{e},y_{e})\end{cases}$ (22) $$

where In-Scope ($I(x_{e},y_{e})$) comprises the edit input $x_{e}$ and its equivalence neighborhood $N(x_{e},y_{e})$, which includes related input-output pairs.
Out-of-Scope ($O(x_{e},y_{e})$) contains inputs unrelated to the edit descriptor.
The edited model $f_{\theta_{e}}$ should satisfy three primary properties: reliability, generalization, and locality.

Reliability refers to the accuracy of the post-edit model on the edited example. Specifically, the post-edit model $f_{\theta_{e}}$ should reliably output the target answer for the edit descriptor $(x_{e},y_{e})$:

$$ $\mathbb{E}_{x_{\mathrm{e}}^{\prime},y_{\mathrm{e}}^{\prime}\sim\left\{\left(x_{\mathrm{e}},y_{\mathrm{e}}\right)\right\}}\mathbbm{1}\left\{\operatorname{argmax}_{y}f_{\theta_{e}}\left(y\mid x_{\mathrm{e}}^{\prime}\right)=y_{\mathrm{e}}^{\prime}\right\}$ (23) $$

Generalization measures how well the edited model adapts to equivalent neighbors within the in-scope neighborhood $N(x_{e},y_{e})$. The post-edit model should predict accurately on related examples:

$$ $\mathbb{E}_{x_{\mathrm{e}}^{\prime},y_{\mathrm{e}}^{\prime}\sim N\left(x_{\mathrm{e}},y_{\mathrm{e}}\right)}\mathbbm{1}\left\{\operatorname{argmax}_{y}f_{\theta_{e}}\left(y\mid x_{\mathrm{e}}^{\prime}\right)=y_{\mathrm{e}}^{\prime}\right\}$ (24) $$

Locality, also known as specificity, ensures that the edit remains local and does not affect the predictions for out-of-scope examples. Thus, the post-edit model should maintain consistency with the pre-edit model on unrelated examples:

$$ $\mathbb{E}_{x_{\mathrm{e}}^{\prime},y_{\mathrm{e}}^{\prime}\sim O\left(x_{\mathrm{e}},y_{\mathrm{e}}\right)}\mathbbm{1}\left\{f_{\theta_{e}}\left(y\mid x_{\mathrm{e}}^{\prime}\right)=f_{\theta}\left(y\mid x_{\mathrm{e}}^{\prime}\right)\right\}$ (25) $$

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: See the weaknesses part
soundness: 3
strengths: 1. The paper critically examines the KL assumption by employing both statistical and modification-based evidence, providing a thorough analysis of why the assumption does not hold in certain contexts.
2. Building on this analysis, the authors propose a novel assumption and develop a modification method that leverages this refined perspective.
3. A set of experiments are conducted to provide their findings
summary: This paper investigates the widely accepted knowledge localization (KL) assumption within related research fields by conducting a series of experiments. Through statistical and modification-based analyses, the authors reveal inconsistencies in the KL assumption, leading them to propose a revised assumption. They further demonstrate how this new assumption can enhance a modification method for knowledge management.
weaknesses: 1. The motivation behind challenging the KL assumption could be further elaborated. For example, from an application perspective, it would be beneficial to explain the practical significance of proving the KL assumption's limitations and demonstrating the advantages of the new assumption.
2. The performance gap between the proposed consistency-aware modification method and the two baseline approaches appears to be minimal, which may reduce the impact of the proposed method.
3. The limited number of relations used in the experiments may constrain the generalizability of the findings, making the results less convincing.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - I double checked throughout the paper, but failed to find the detailed implementation for the results plotted in Figure 1. What instructions are used to prompt these models?

- Sometimes a query might involve multiple, possibly related knowledge (suppose all of them are consistent knowledge). Do they activate their according neurons simultaneously, or is there an additive effect on knowledge neurons?

- How does Consistency-Aware KN Modification affect model behavior? A case study might help.
soundness: 4
strengths: - The paper presents an innovative approach by challenging a widely accepted notion in LLM research and proposing a refined framework. The introduction of the Query Localization assumption is a creative rethinking of knowledge localization, addressing observed limitations with a novel perspective.

- The authors conduct extensive empirical evaluation, using 39 experimental setups to validate their claims. The paper demonstrates the prevalence of inconsistent knowledge under the KL assumption. This comprehensive experimental approach solidifies the credibility of the findings.

- The paper is well-structured and easy to follow.

- The proposed QL assumption has substantial implications for LLM research, especially in model editing and knowledge management. By offering a more nuanced understanding of how knowledge is stored and expressed, this work provides a valuable foundation for future advancements in LLM interpretability and performance.
summary: The authors investigate the mechanisms behind factual knowledge storage and expression in large language models. It re-evaluates the Knowledge Localization (KL) assumption, which posits that specific knowledge neurons can store distinct facts. Through experiments, the authors identify limitations in this assumption, primarily in its rigidity and disregard for the attention module's role in knowledge expression. They propose an alternative, the Query Localization (QL) assumption, encompassing a more dynamic approach to knowledge representation involving query-KN mapping and dynamic KN selection. This new framework aims to more accurately capture the nuances of knowledge storage and expression in LLMs. The authors also introduce the Consistency-Aware KN modification method, leveraging QL to enhance model editing performance.
weaknesses: - The identification of consistent versus inconsistent knowledge relies on specific thresholding techniques, such as Otsu’s threshold. These could introduce variability in findings if altered. A deeper analysis of threshold sensitivity might strengthen the paper's robustness claims.

- Although the QL assumption demonstrates improved performance in model editing, the discussion on how this improvement translates into real-world applications could be expanded. For instance, it remains unclear how this new approach impacts efficiency, especially in scenarios demanding large-scale knowledge modifications.

---

confidence: 2
contribution: 4
initial_rating: 8
presentation: 3
questions: It would be better if further details, such as interpretability analyses (e.g., visualization), on how the Dynamic KN Selection process works can be provided.
soundness: 3
strengths: The paper presents a novel critique of the widely adopted KL assumption in LLMs, questioning its validity for all factual knowledge and proposing a more advanced alternative. The QL assumption, with its concepts of Query-KN Mapping and Dynamic KN Selection, offers an innovative framework that shifts from static knowledge representation to a more query-dependent, dynamic view. This fresh perspective is a notable contribution to the field, as it fundamentally rethinks how LLMs store and express knowledge, integrating the role of the attention mechanism for the first time in this context. The proposed QL assumption further addresses fundamental limitations in how knowledge is localized and retrieved within LLMs, which has implications for a wide range of applications, including model interpretability, knowledge modification, and dynamic knowledge editing.

The paper provides rigorous, well-designed experiments across multiple LLM architectures to evaluate the prevalence and implications of Inconsistent Knowledge that does not adhere to the KL assumption. The introduction of Consistency-Aware KN modification is grounded in empirical analysis and validated by statistically significant improvements in model editing performance metrics, adding strong evidence to validate the new assumption.
summary: The paper critiques the Knowledge Localization (KL) assumption, which assumes that a piece of factual knowledge can be localized to knowledge neurons (KN) in LLMs. The authors identify two limitations of this assumption, which pertain to both knowledge storage and expression, and reveal that inconsistent knowledge is a common occurrence. To address these, they propose the Query Localization (QL) assumption, which introduces Query-KN Mapping and Dynamic KN Selection. These concepts suggest a more flexible, query-dependent view of knowledge storage and retrieval in LLMs. Using these insights, they introduce a Consistency-Aware Knowledge Neuron modification method to improve the performance of model edits. Experimental results validate the QL assumption and suggest it as a more effective model of factual knowledge in LLMs.
weaknesses: The QL assumption adds the attention module to knowledge retrieval, which could potentially increase computational load. However, the paper does not analyze how this change impacts model efficiency. Including a comparison of computational costs and scalability between KL-based and QL-based methods—such as inference time or memory usage—would clarify the impact of attention-based neuron selection.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: (1)In Table 1, the KII ratio for GPT-2 is lower compared to other models. What causes this discrepancy? Is it because GPT-2 has significantly fewer parameters than the other two models?

(2)Table 2 presents metrics such as Reliability, Generalization, and Locality, but not all of these metrics are analyzed in the paper.
soundness: 4
strengths: **Originality**: This paper demonstrates a high level of originality by identifying flaws in the KN hypothesis. Through extensive experimentation, the authors have shown that these flaws are widespread and have proposed effective solutions to address them.

**Quality**: This paper is of high quality as the authors have conducted numerous experiments using various models, datasets, and knowledge localization methods, ensuring the reliability of the results.

**Clarity**: The paper is logically and structurally clear, providing well-reasoned problem descriptions, metric designs, and detailed explanations of findings for each section.

**Significance**: This paper is significant for advancing research on the interpretability of LLMs. By identifying flaws in the KN hypothesis and proposing improvements, the paper contributes to the broader goal of making LLMs more interpretable and transparent.
summary: This paper challenges the Knowledge Localization (KL) assumption, a core concept of how large language models store and retrieve factual knowledge. It reveals that the KL assumption is not universally applicable. The authors also argue that the KL assumption is limited because it only considers the storage aspect of knowledge and overlooks the mechanisms of knowledge selection.

This paper introduces the Query Localization (QL) assumption: (1) For knowledge that does not conform to the KL assumption (KII), the localization of knowledge is tied to the query itself rather than the underlying fact. (2) The attention module plays a crucial role in selecting the appropriate knowledge neurons for answering a query, especially when dealing with KII facts that are distributed across multiple neurons. The paper supports these hypotheses through extensive experiments and proposes a Consistency-Aware KN modification method that leverages the QL assumption to improve the performance of knowledge modification tasks.
weaknesses: The paper conducts extensive experiments for each step of problem identification and resolution independently. However, some paragraphs lack smooth transitions, and certain experimental results are omitted in the analysis.

Meta-review: The authors investigate the mechanisms behind factual knowledge storage and expression in large language models. It re-evaluates the Knowledge Localization (KL) assumption, which posits that specific knowledge neurons can store distinct facts. Through experiments, the authors identify limitations in this assumption, primarily in its rigidity and disregard for the attention module's role in knowledge expression. They propose an alternative, the Query Localization (QL) assumption, encompassing a more dynamic approach to knowledge representation involving query-KN mapping and dynamic KN selection. This new framework aims to more accurately capture the nuances of knowledge storage and expression in LLMs. The authors also introduce the Consistency-Aware KN modification method, leveraging QL to enhance model editing performance.

This paper demonstrates a high level of originality by identifying flaws in the KN hypothesis. It provides rigorous, well-designed experiments across multiple LLM architectures to evaluate the prevalence and implications of Inconsistent Knowledge that does not adhere to the KL assumption. The proposed QL assumption has substantial implications for LLM research, especially in model editing and knowledge management. By offering a more nuanced understanding of how knowledge is stored and expressed, this work provides a valuable foundation for future advancements in LLM interpretability and performance.

Decision: Accept (Spotlight)