What does the Knowledge Neuron Thesis Have to do with Knowledge?
Authors: Jingcheng Niu
Andrew Liu
Zining Zhu
Gerald Penn
Venue: ICLR 2024 spotlight (2024)
Keywords: language model
knowledge neuron
model editing
formal and function competence
syntax
fact
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: We reassess the Knowledge Neuron (KN) Thesis: an interpretation of the mechanism underlying the ability of large language models to recall facts from a training corpus. This nascent thesis proposes that facts are recalled from the training corpus through the MLP weights in a manner resembling key-value memory, implying in effect that "knowledge" is stored in the network. Furthermore, by modifying the MLP modules, one can control the language model's generation of factual information. The plausibility of the KN thesis has been demonstrated by the success of KN-inspired model editing methods (Dai et al., 2022; Meng et al., 2022).

We find that this thesis is, at best, an oversimplification. Not only have we found that we can edit the expression of certain linguistic phenomena using the same model editing methods but, through a more comprehensive evaluation, we have found that the KN thesis does not adequately explain the process of factual expression. While it is possible to argue that the MLP weights store complex patterns that are interpretable both syntactically and semantically, these patterns do not constitute "knowledge." To gain a more comprehensive understanding of the knowledge representation process, we must look beyond the MLP weights and explore recent models' complex layer structures  and attention mechanisms.

Published as a conference paper at ICLR 2024
WHAT DOES THE KNOWLEDGE NEURON THESIS HAVE TO DO WITH KNOWLEDGE?
Jingcheng N www.@ccsttoooooeeeoo University of Toronto, University of Waterloo, SStevens Institute of Technology, *Vector Institute
Andrew Liu
Zining Zhu 131
Gerald Penn 11
zzhu41 @stevenssedd
gpenn @ cs. testtooontoedu
ABSTRACT
We reassess the Knowledge Neuron (KN) Thesis: an interpretation of the mecha- nism underlying the ability of large language models to recall facts from training corpus. This nascent thesis proposes that facts are recalled from the training cor- pus through the MLP weights in manner resembling key-value memory, imply- ing in effect that "knowledge" is stored in the network. Furthermore, by modifying the MLP modules, one can control the language model's generation of factual in- formation. The plausibility of the KN thesis has been demonstrated by the success of KN- KN-inspired model editing methods (Dai et al. 2022 Meng et al. (022).
We find that this thesis is, best, an oversimpliiication. Not only have we found that we can edit the expression of certain linguistic phenomena using the same model editing methods but, through a more comprehensive evaluation, we have found that the KN thesis does not adequately explain the process of factual ex- pression. While it is possible to argue that the MLP weights store complex pat- terns that are interpretable both syntactically and semantically, these patterns do not constitute "knowledge." To gain a more comprehensive understanding of the knowledge representation process, we must look beyond the MLP weights and explore recent models' complex layer structures and attention mechanisms.
INTRODUCTION
Recent research has highlighted the remarkable ability of large pretrained language models (PLMs) to recall facts from training corpus (Petroni et al.] 2019). The underlying mechanism by which this information is stored and retrieved within PLMs, however, remains a subject of intensive inves- tigation. The Knowledge Neuron (KN) Thesis has been recently proposed as novel framework for interpreting language models (LMs) (Dai et al., 2022 Meng et al., 2022] 2023). This thesis suggests that LMs operate akin to key-v memories, recalling facts from the training corpus through the multi-layer perceptron (MLP) weights Therefore, a significant implication of the KN thesis is that factual information generation by LMS can be controlled by modifying the MLP modules. Should this manipulation of factual information recall become feasible. it could lead to the development of language models that are more controllable, interpretable, and factually aligned.
The plausibility of the KN thesis is demonstrated by the success of KN-inspired model-editing methods. Dai et al. (202)) argued that relational facts can be localised to a handful of 2-5 MLP neurons. They then developed method to identify these neurons using search algorithm based on an integral of gradients. By manipulating the activation of these identified neurons (KN edit), they managed to alter the model's response to fill-in-the-bland cloze tasks and generate counterfactual information without additional fine-tuning. In a parallel approach, Meng et al.] (2022) proposed more intricate model wherein factual recall occurs in two critical locations, each incorporating a different module. In this model, the mid-layer MLP retrieves the fact, and an attention module copies it into the output response at the topmost layer. Despite this proposed two-step process, their proposed model editing method, Rank-On Model Editing (ROME), only modifies MLP weights, much as KN edit only modifies MLP activations without editing attention modules.
While the efficacy of these model editing methods has been showcased in simple fill-in-the-blat cloze tasks, the appraisal of such achievements mainly rests on basic paraphrasing of the prompts, as

---

Published as a conference paper at ICLR 2024
The "St houdarr Nourse
the Meee
Raymood le selling (WAG3 plane
2304
1064
Rdymord neiling 34AGK] saaktaas
El
Average Integrai of Gradients of Dffienenc MLP Autivations
Eriiing the neuron
Figure 1: Syntactic phenomena can be located and edited using existing model editing methods. The integrated gradient of singular determiner (this, that) and plural determiner (these, those) form two distinct groups. Erasing these neurons leads to output probability changes.
outlined by Yao et al. (023), who introduced an additional assessment metric, portability, finding that model-editing methods to date lack robustness. Their performance is halved when evaluated with the portability measure. Building on this, we introduce two new metrics. First, a successful edit must demonstrate symmetry within bijective relationships (e.g., with the assertion Ortawa is the capital of Canada, the reciprocal Canada's 's capital is Ortawa should also hold valid). Second, a successful edit must extend to synonym usage (e.g., dentist treats toothache and a dentist treats tooth pain should be considered equivalent). Our evaluation shows that existing model-editing methods are even less robust under these two new criteria.
It is practically impossible to exhaustively assess factual model-editing methods due to the difficulty in systematically dealing with counterfactual data. The potential counterfactual replacements for Canada's capital are seemingly endless. Thus, beyond the introduction of the two new evaluation criteria above, we propose the evaluation of model-editing methods using syntactic constructions. We have determined that the KN thesis applies just reliably to syntactic phenomena (as illustrated in Figure D. Unlike many facts, syntactic phenomena can provide rigorously defined targets for editing through the use of so-called minimal pairs. As a result, in this paper, re-evaluate the KN thesis by expanding the scope of our assessment to include more complex factual patterns and syntactic phenomena. This also speaks a long-standing debate regarding the formal vs. functional competence of language models (Mahowald et al. 2023) - an LM's ability to follow linguistic rules and patterns vs. its ability to apply language in the real-world (see $2.3). If we edit model's expression of facts and linguistic phenomena using the same approach, this could indicate that both the formal and functional competencies of an LM are governed by the same underlying mechanisms.
Within the context of Dai et al. s (2022) KN framework, KN edit's efficacy is unsatisfactory. Editing the KN activations has only limited impact on categorical predictions. The effect of KN edit is only apparent in the shifts in the output probability distributions of tokens. The patterns that the method localises also appeal to superficial cues such as word co-occurrence frequencies. We also find several critical shortcomings in the ROME framework. LMs process both linguistic and factual information in phases, but the exact task distribution between the MLP and attention modules appears to be more idiosyncratic than initially theorized (Meng al. 2022). ROME model editing only superficially alters token association patterns, in a manner that is inconsistent across the various expressions that may attend the same underlying knowledge. As result, whatever is being manipulated reflects none of the traditional tautologies that have been associated with "knowledge," as that term has been understood in philosophy since the time of Aristotle. When implemented on syntactic constructions, furthermore, the influence of ROME's editing is limited only to the word altered and no pivot that preserves any reasonable standard of syntactic paraphrase, such as substitutability salva veritate, is forthcoming. Furthermore, ROME fails under our newly proposed symmetry and synonymy criteria.
We therefore argue for the position that the feed-forward MLP modules of the transformer model do not store knowledge, but rather complex "token expression patterns." These token expression patterns can often be interpreted linguistically, but the information that they express does not fit into linguistically or factually defined categories. A key-value, memory-based view of the language model is overly simplistic in explaining the remarkable ability recent PLM's formal, and perhaps even functional, competence. We need to investigate the rich layer and attentive structure of recent PLMs more to arrive at a better understanding of their underlying mechanics.
In the following sections, we will first provide overview of the KN thesis ($2) Then will evaluate two practices inspired by it: Dai et al. (2022) KN edit framework ($3) and Meng et a 2022) ROME framework ($4). Finally, we will conclude the paper with discussion ($5)
'The code, data and results are publicly available at http:://gthha.ccm///aaannnnn thesis
2

---

Published as a conference paper at ICLR 2024
2
THE KNOWLEDGE NEURON THESIS
Geva et al. (2021) were among the first to propose that the MLP modules in a transformer model behave like key-value memories. A typical MLP module in recent transformer-based PLMs has Iwo layers. They argue that the first layer corresponds to keys, and the second layer, to values They found that each key neuron is triggered by interrrittereee shallow input patterns such as periods of time that end with the letter "a." Then, the corresponding value neurons distorted the next-token output probability, until a final distribution is generated.
The KN thesis emerged as a result of this important discovery. Dai et al. (002)) coined the term knowledge neuron and ambitiously claimed that the keys and values within MLP modules not only capture simple patterns but also store "knowledge." They formulate an item of fact, such as Canada" 's capital is Ottawa, as a 3-tuple (s, t, r), consisting of the source (s, Canada), the target (t, Ottawa) and the relation (r, capital) between them. The authors asserted that this tuple can be localized to small group of MLP neurons typically found in the topmost layers of the language model, which they identified by analysing the magnitude of the integrals of gradients among prompts. To support their claim, they conducted model-editing experiments. By suppressing the KNs (setting their activations to zero); they observed decrease in the probability of generating the correct original target (4), while other tokens remained largely unaffected, demonstrating a "minimally invasive surgery." Meng et al. (2022) proposed a refinement of Dai et al.] s (2022) model. They employed causal mediation method (Finlayson et al., (02T) to form a more intricate version of the KN thesis. They argue that the factual association process happens at two locations: a mid-layer MLP recalls the fact from memory, and the topmost layer's 's attention model copies that information to the final output.
There were similar investigations of neurons prior the KN thesis. Durrami et al. (2020) observed the neurons of an auxiliary probing model that was trained on BERT embeddings, not the neurons of BERT itself. Therefore, their analysis faced an all-too-common dilemma for probing: did they find insights about the language models artefacts of the fine-tuning process (Hewitt & Liang 2019 Finlayson et al. 2202 l used causal mediation analysis to study subject-verb agreement in GPT and XLNet (Ynng et al. 2019), In particular, they observed a difference in ratios between the verb with the correct inflection and one with the incorrect inflection. They then modify the prompt, see the probability change and reason about the internal mechanisms of the model for expressing subject- verb agreement. They concluded that the upper-middle layers are more relevant to the expression and that there are various levels of overlap between the top 5% neurons used express agreement. These insights. however, just as with previous probing work, are still purely observational and largely preoccupied with layers and network depth. They are able to observe many characteristics of the process, but still cannot cannot provide satisfactory understanding of how it happens.
More recently. there has been interest in utilizing large language models (LLMs) to gain insight into the differing functionalities of individual neurons. Despite its title's strident claim that neurons in LMs can be "explained," " Bills et al. (2023) clarify that their model "explains correlations, not mechanisms." From nhowledge-representaaie
standpoint, their evaluation of I.IM explanations also entirely observational. When[Huang et al. (003) reassessed the validity of these explanations, even the most confident ones had high error rates and little to no causal effects the interventions that use the explanations The LLM interpretation of LMs is still immature.
2.1 EVALUATING THE KN THESIS: AN OVERVIEW
The effectiveness of a model-editing algorithm is customarily evaluated across three dimensions Yao et al. (202)): (1) reliability: whether the model can successfully change its output from t to (also referred to as an efficacy score by Meng et al.) (2022); (2) generality: whether the effect is applicable to rephrased relations; and, (3) locality: whether the edit impacts unrelated relations. Yao et al. (023) stress, however, that the assessment of generality is often constrained to simple paraphrasing. This is typically done by developing multiple templates for a specific relation. For instance, the relation capital can be structured as both "The capital of [s] is [t]." and "[s]'s capital is [t]." Previous evaluations (Elazar et al. 202 I Meng et al. 2022 2023 prematurely announced success when a model, edited on a first template, could be generalized to a second template. Thus, Yao al. 2023) recommended extending the assessment parameters by introducing the concept of
2 Despite the similarities in nomenclature, this is unrelated to the key and value sell-attention.
3

---

Published as a conference paper at ICLR 2024
portability. For example, having changed Watts Humphrey's alma mater from Trinity College to Harvard University, the model should return Boston instead of Dublin when asked about the city where Watts Humphrey received his university education. It was apparent that model-editing meth- ods present a markedly lower level of portability than generality (50% versus 90%). The evaluation of portability, on the other hand, requires new data annotation, which can be costly.
Extending Yao et al. (202), we attempt a more comprehensive evaluation of model editing of fac- tual association with two extra criteria: bijective symmetry and synonymous invariance. Bijective symmetry does not require new data collection and we can obtain data automatically from previous corpora. For a bijection relation such as capital or capital of, we should see the model generalise (s,t + r)) to str,s.r"' For example, if we change the capital of Canada to Rome, then the model should also agree that Rome is the capital of Canada. Similarly, an effective edit should also be able to generalise across synonyms. If the model knows that dentist treats toothaches, it should also know that they also treat tooth pain. Prior work (Flazar al., (02T) only used synonym replacement rephrasing the relation prompts we extend it to the source and the target.
Several others have already questioned the validity of the KN thesis. Hase et al. (2023) identified discrepancies between the results of causal tracing and the effects of ROME editing. They con- cluded that a mechanistic understanding reveals insights on the consequences of model editing. To the best of our knowledge, we are the first to comprehensively evaluate the KN thesis using rigor- ously defined syntactic phenomena. We consider three: determiner-noun agreement, subject-verb agreement, and gender and number agreement across anaphoric chains.
2.2 EVALUATING THE KN THESIS ON SYNTACTIC PHENOMENA
Edit pairs for syntactic phenomena, by contrast, can be systematically extracted through the forma- tion of "minimal pairs." " For a grammatical sentence that expresses a linguistic phenomenon, we can construct ungrammatical sentence that minimally differs from the original sentence in respect of one feature of grammatical acceptability. For example, the phrase this student can be changed to the ungrammatical counterpart, *this students. The BLiMP corpus (Warstadt et al. 2020) is one of the most comprehensive and extensively utilised collections of such minimal pairs.
We therefore propose to systematically evaluate the effect of model-editing methods using syntac- tically differentiated prompts. We define a similar 3-tuple (s, t, p) that contains the source (s), the target (t) and the syntactic phenomenon (p). Take the phenomenon determiner-noun agreement as an example. In a grammatical sample sentence from a minimal pair, 8 is the tokens that are condi- tion the expression of the target (the determiner), and is the tokens that differ within the pair (the noun). The ungrammatical target t", is the noun in the opposite form. We then intervene with model editing, and observe whether the model assigns a higher probability than t"
2.3 EDITING SYNTACTIC PHENOMENA & THE "FORMAL VS FUNCTIONAL" DISTINCTION
If we can successfully edit facts as well as syntactic phenomena using the same model-editing meth- ods to the same degree, then stands to reason that the model follows a unified underlying mech- anism for both factual and syntactic information. Choosing the correct city (the Space Needle is in Seattle/Rome) would be no different than choosing the correct verb form (the apple is/ red).
Mahowald et al. (2023) refers to distinction between the formal and functional competence of language model: formal means "knowledge of linguistic rules and patterns," and functional refers to "understanding and using language in the world." Syntactic phenomena pertain to formal compe- tence, and facts pertain to functional competence, respectively. NLP researchers sometimes infor- mally use the terms syntax and semantics to refer to this distinction. BLiMP even refers to anaphoric gender agreement as morphological, Jawahar et al. (2019) and and Teeney et al. (2019) believe that syn- tactic information located in lower layers in BERT than semantic information, because syntactic information is more "shallow." Dai et al. (2022) appear to agree with this assertion in claiming that factual information located in the upper layers. Meng et al.] (2022), however, claim that factual information is located in the middle. This contradiction may support Niu et al. (2022) assertion that layers are not the best explanatory device of the distribution of these types of information in LMs. We explore here the possibility that no dividing line exists at all between the mechanisms through which a language model processes information related to these two types of competence.
4

---

Published as a conference paper at ICLR 2024
3 LOCALISING S SYNTACTIC PHENOMENA IN LANGUAGE MODELS
We put the KN thesis to the test under the KN-edit framework by asking three questions: (1) can we localise linguistic phenomena using the same KN-edit method; (2) how do the levels of localisation compare to each other; and (3) are these localisations strong enough to support the KN thesis
3.1 METHODS: SEARCHING FOR KNS OF SYNTACTIC PHENOMENA
For each prompt, we calculate an integral-ofgrradeen attribution score o; for the i-th intermedi- ate neuron the 1-th layer (w)"). Then, for syntactic phenomenon with the source-target pair (s, t, p), we find the neurons that have an attribution score greater or equal to ==20% of the maxi-
mum attribution score shared among at least 7% of its prompts. We start from 7=70% and adjust
by an increment or decrement of 5% until the number of neurons is within the range of [2, 5].
Neuron Attribution Score Given input prompt æ, we follow Dai al. (0022) and use the integral of gradients to calculate the neuron attribution score:
(Po() aw
dy (ai plula, w = w (
W.
7=
where (w) (1) denotes the probability distribution of the token y when changing the neuron w; (1)ss
value to úi! and OP. (orw denotes the gradient of the model with respect the activation w We
will see a more salient gradient when the neuron inflicts a greater change on the output probability.
Measuring the Level of Localisation (1) the number of identified neurons (KKN) ) using the initial threshold set- ting (7=70%), (2) the final threshold obtain 2-5 KNs, and, (3) simi- larity score among all the token attri- bution patterns.
We use three metrics to measure the level of localisation:
Neuron
this
that
these
those
(10)
0.93
0.75
0
0
Layers
1004
(9)
0
1.00
1.00
[9)
0.33
0.32
0
(13)
0
0.81
0
0
Both of Dai et a (2022) measures (KN and 7) depend on adjusting the two threshold hyperparameters, and т. Here, we propose non- parametric measure using gener- alised n-sample similarity measure
Average KNN attribution scores.
(b) KNs for Det-N pairs.
Figure 2: Localising grammatical number to KNs. The sin- gular determiners share common KN (1)))) and the plu- ral determiners share a different common KN (w 1094). (RF) that measures the correlation of all the attribution patterns:
Y = .. [.l. Y:
8; |ss
Y USV Σ ORURU R²
of n n-1
(2)
We first normalise and concatenate each attribution pattern 8; for each prompt v, in the dataset into Y. Then, we can calculate the similarity/comrlltionn among all patterns by conducting singular value decomposition (SVD) and using the square the first singular value of We then normalise this measure to the range [0, so that the similarity between parallel vectors will be R 1, and n orthogonal vectors will get R? = O.
3.2 RESULTS & FINDINGS
Finding We can localise the grammatical number of determiners to just two neurons, just like factual information. The BLiMP paradigm determiner saciineeenounnaggeee moun_aggee 2 (DNA.2) contains 1000 sentence pairs with exactly one demonstrative determiner (this, that, these, those) agreeing with an adjacent noun, e.g., Carl cures those/*that horses. The determiner those is tt that is t" and
Due to page restrictions, only present the results of the determiner, cettrmiier_noun_egeee noun_agreement.2 (DNA.2) paradigm BERT in the main content. See Appendix for the result of other BI iMM paradigms and LMs..
5

---

Published as a conference paper at ICLR 2024
(a) Effect of suppressing the singular neuron (10)
(b) Effect of suppressing the plural neuron 19
Probobiltt Change
100%
Probabilty Change
500%
50%
300/5
0%
2002
100%
-50%
0%
100%
-100%
thes
those
botth
multyin aaven Soain
the one
2 the ann
some
my
eur
these
thos
both
somo mo on
mill
Figure 3: Suppressing the number neuron's (singular: w plural: w1004) effect across number- expressing prenominal modifiers. Significant (p < 0.05) changes are highlighted in red. The three sections in the plots are, from left to right, plural, singular and neutral modifiers.
the noun horses is 8. A noun may appear in multiple sentence pairs. Among the paradigm's 1000 sentence pairs, we identified 283 unique Det-N pairs (s, (s,t,t",r)
Attribution Score Patterns The attribution score of neurons shows highly consistent pattern that can be interpreted linguistically. We calculated the average attribution scores of all the prompts that contains each one of the determiners. Figure 2a shows selection of the average attribution scores. The colour block in the ith column and jith row shows the attribution score a, As we can see, common neuron (w220)) has high average attribution score for both of the singular determiners this and that, and another common neuron (wiosu) lights up for the plural determiners these and those
This pattern is not only shown in aggregate. For each Det-N pair, we use the 1000 sentences in the paradigm as templates to create the prompts needed for a KN search. For each sentence, we replace the sentence's 's determiner and noun with the Det-N's determiner and noun. We then obtain 1000 sentences with different contexts but the same determiners and nouns. Then, we run KN search on these 1000 sentences. When we look into each individual Det-N pair, the two neurons are identified (10)
as KNs in the vast majority of the pairs. As shown in Figure 2b W2096 appeared in 93% of the pairs with this and 75% of the pairs with that. The plural neuron appeared in 100% of pairs with these or those. More importantly, these neurons were not identified as KNs in pairs with the opposite grammatical numbers. Figure shows an excerpt of the results (full results Appendix B.2
Effects of Suppressing the "Number Neuron" Do these two neurons correspond to grammati- cal number? We suppress each neuron (setting activation to 0) and compute the pre- and post-edit model's output probability of various number-expressing prenominal modifiers across all prompts with singular/plural nouns. Appendix B.T explains the prenominal modifier selection process. Fig-
(rnssteeit)-p(pre--ee ure 3 shows the average effect of suppressing the identified KNs min (pinsppppsteedee ppite edit)
The result of suppressing the plural neuron is pronounced (Figure 3p).
Suppressing the plural neuron.
This intervention leads to a significant reduction in probability across all
Prolabiity Ohinge
52%
plural modifiers, a notable increase for the majority of singular modifiers, but a limited impact for modifiers that do not express number agreement. Therefore, erasing the activation of the plural neuron causes decrease in the expression of determiner noun agreement for plural modifiers. Al- though this KN search is solely based on these four demonstrative deter- miners, we observed that it generalizes to other determiners (one, a, an, every; two, both; multiple, several, various) and even adjectives (single, unique, sole). This effect is statistically significant. By treating the pre- and post-edit probabilities as two separate groups, a |Studentts (1908) t-test reveals significance when the modifiers are highlighted in red in Figure 3 The null hypothesis is that the pre- and post-edit probabilities are sampled from the same
25%
0%
.25%
.20
mc-
any
Figure 4: The localisa- tion of plurality appeals word co-occurrence frequencies cues.
distribution, i.c., the intervention has no effect. Thus, the neuron W1094 can be interpreted through the lens of a linguistic phenomenon, viz. determiner-non noun agreement.
(9)
Note, however, that the word scattered also sees significant probability decrease when suppressing the plural neuron. Scattered does not specify for plural number, phrases such as "scattered rioting"
Wee use Meng et s 2022) neuron numbering system. Both layer and neuron indices start with 0.

---

Published as a conference paper at ICLR 2024
BLIMP Paradigm
[KN]
f
Rels.
[KN]
Rq
40%
41%
det agr.
3,94
0.71
0.56
P101
0.167
0.515
0.399
30%
det_n.agr_22
1.86
0.62
0.56
P103
0.204
0.662
0.389
dmm__im._
5.53
0.73
0.64
P106
1.292
0.607
0.365
im..
0.67
0.55
P108
0.663
dna. w. ndj 1
8.88
0.78
0.67
P1303
10.462
0.814
0.684
1
5
0
5 6 1011
dna.., w...dj..2
2.26
0.67
0.57
P140
2,008
0.689
0.263
Layar
Layer
(b) Layer distribution of identified KNs. Both
(a) Levels of localisation measures.
BLiMP and PARAREL. occupy the topmost layers.
The localisation of certain syntactic phenomena (BLiMP) is comparable to facts (PARAREL.). We comparable localisation metrics and the identified KNs occupy the same layers.
Figure 5:
122%
Paradigm
Pre-odit
Post-edit
A
Data
Model
Reliahility
+318%
20.8%
dna dir..
det.n.agr_.2
99.5%
100%
94.8%
96.9%
-2.6%
-5.2%
ZoRE
TS-XL GPT-J
22.51
w. w...
97.1%
94.4%
-2.7%
11.34
These
dna, dna__w._ddj wwn__..ddjj.ir.. 2
97.4%
95.4%
-2.0%
CounterFact
TS-XL
GPTJ
47.86
1.66
(a) The exact effect to output probability of editing the KNs. pre-edit. post-edit.
(b) These modifications of determiner- noun KNs are usually not enough to overturn the categorical prediction.
(c) KN cdit has low reliability for (Yao et al. (023).
facts
Figure 6: Editing the KNs is not enough to overturn the categorical predictions. The major limitation of KN edit is its low reliability. These reliability scores cannot support the KN thesis.
are syntactically and semantically well-formed. But it is used more often with plural nouns because of its meaning- This frequency effect is not limited to scattered. Other words such as any, all, unified, and the three adjectives unique, single and sole exhibit a similar bias. As shown in Figure we see probability changes, although less substantial, alongside those modifiers that strictly specify for grammatical number. This is semantic number co-occurrence bias.
The suppression effect of the singular neuron is similar but less pronounced. Overall, we see the opposite effect across all prenominal modifiers, with the "singular" adjectives (unique, single, sole) being the only exceptions. This is, however, unsurprising. Unlike the plural neuron, the singular neuron did not appear in all of the Det-N pairs. We suspect that an LM can identify the plural property more easily when its wordpiece-bard based tokeniser exposes many plural suffixes.
Finding 2: KNs obtained using linguistic tasks and factual tasks share similar characteris- ties of localisation. Figure 5a shows the level of localisation of various BLiMP determiner-nour agreement paradigms and selected PARAREL relations. The localisation metrics of both BLiMP paradigms and PARAREL relations fall within the same range. See Appendix C for the full list.
Furthermore, Figure Shlshows no bifurcation of layers within which linguistic and factual KNs locate (see Appendix C.2) All of the neurons are distributed in the topmost layers. The determiner noun agreement pattern is purely syntactic. This is refutation of Jawahar et al. (2019) and and Tenney et al. (2019) view that syntax is localised to more shallow layers than semantics. Our results confirm Niu et a (s (2022) assertion that the location of syntactic and semantic (and, additionally, factual) information not distinguished by layer in the LM. In fact, our results may suggest that these types of information are most fruitfully thought of as being handled by the same functional mechanism.
Finding 3: Despite the high level of localisation in the underlying probability drift, the effect of editing the KNs is not enough to overturn the categorical predictions made by the language model. Although we see high level of localisation and f,- we find that this change is often not enough to overturn the final prediction. As shown in Figure we only see at most 5.2% of the BLiMP results being overturned. This low reliability issue is not limited to syntactic phenomena. In Figure 66c we list Yao et al. s (202)) evaluation of KN edit on two other corpora: ZsRE (Levy et al.) 2017) and CounterFact (Meng et al. 2022). The reliability of the KN algorithm ranges from 1.66% to 47.86% not enough to support the KN thesis.
the relative probability change between t
Discussion Just as with facts, syntactic phenomena localise to neurons. Modifying merely two neurons working in tandem significantly change the expression of determiner-noun number.
7

---

Published as a conference paper at ICLR 2024
This is not the only type of localisable syntactic phenomenon (see Appendix C and together they constitute a significant extension of Finlayson et al. is (202)) findings - syntactic phenomena can be localised to the individual neuron level. Furthermore, these phenomena share with factual infor- mation the extent of their localisation, and the layers in which the KNs typically occur.
But do the patterns identified for these neurons constitute "knowledge?" KN edit's low reliability score and its appeal to shallow cues both suggest otherwise. If we follow the KN thesis and interpret a post-edit probability change as an indication of the quantity of knowledge stored, then we cannot draw the conclusion that knowledge is stored there. The identified neurons are spots with a high information concentration, but the final decision still lies with the rest of the model.
Interestingly, the patterns that we identified resemble linguistic categories, but they deviate from rules of grammatical well-formedness. In determiner-noun agreement, KN edit also affects pre- modifiers that do not specify for number, alongside plural-specifying determiners such as multiple, several and various. Phrases such as sole breadwinners and scaftered rioting are less frequent but by no means unheard of. This suggests that the patterns reflected within the MLP neurons can only be completely accounted for by appealing to superficial cues such as word co-occurrence frequency.
4 CAUSAL TRACING AND RANK-ONE MODEL EDITING
In this section, we reassess Meng et al.ss (2022) similar but more intricate implementation of KN edit. They proposed that information is expressed at two locations: facts are recalled in mid-layer MLP weights, and copied to the final output by at- tention modules. They derived this thesis based on causal medi- ation. The causal traces in Fig- ure 7a are computed as follows. First, the source tokens are cor- rupted by adding random noise and the model generates an in- correct result. Then, they restore an intermediate hidden state to its correct value for all the to- kens at all layers, and determine whether this restoration can fix the corruption. They discover division of labour between the MLP and attention. This division, however, not stable. In Figure 7e we reproduce this effect on syntactic phenomena. The distinction between the early and late site is no longer discernible. This is, fact, not distinction between facts and syntactic patterns. Many factual causal traces also do not show this distinction [
state
MLP
attondoo
That Saace
Mood*
0.30
635
doartoon
20
40
40
20
40
bayort
byyer
byyer
(a) Factual information.
attention
aisscumentarkal
thane
10
20
20
60
(b) Determiner-noun agreement.
state
MLP
attertion
p0cl
-01
(c) Subject-verb agreement.
Figure 7: Causal tracing result.
Previous evaluation of the ROME model-editing method was limited to simple paraphrasing (Yao) et al. 2023). We observe that ROME does not generalise well in respect of either of our new criteria, bijective symmetry or synonymous invariance (Figure Sab). This issue persists when we evaluate ROME quantitatively. We assembled two new datasets using PARAREL relations to evaluate our two new criteria (see Appendix E for details). We use the two bijective relations R1376 and P36 to construct bijective symmetry evaluation dataset. Then, for synonymous invariance, rewrite the field-of-work targets in P1O1 into occupation names. For instance, if we change Anaxagoras's field of work from philosophy linguistics, we also want the model answer "Anaxagoras is linguist" when given the prompt. Table [[ shows the result of our evaluation on these newly as- sembled datasets. Although ROME obtains higher reliability scores than KN edit in both GPT-2 XL and LLaMA-2 7B, the symmetry and synonymy results are both much lower. We also observe
In Appendix D we show the average indirect causal effects; observation thus holds under aggregation.
8

---

Published as a conference paper at ICLR 2024
(a) GPT-2 XL. The capital of Canada is Ot-
(b) GPT:2 XL: To treat my joothanke I should
(3) GPT-2 XL: The outhorz near the tasi drivers are
tawn
see a dentlst
ROME Edit
ROME Edit: Ottown Rome
ROME Edit: demiist lawyer
2: The authors urer the tost drisers are...
3 The capiol of Canada Ottawa
@: To reat fonthacher I should dentist.
authors the
cupital of Canada Rome.
G: The authors the dancers in their paper authors near the dancers is
To treat my tuothaclu, I should sve lawyer.
3.
Otteni the cupiul Canada.
Onnwa the capital Canada's federalist
Th rent my toodh pain, Fhuuldd ser dentis..
@. pilats the drivers
5y ylac of government.
To troat my tooth pain, I should see a dentist.
The pilots near the taxi drivers' cubins are
Rome the capiaal of Italy... 3 Rome the capital of Ituly...
Yo treat my odontalgia, should we a dentist. 3: To trout my odontulgia, shoolld see a dentisst
pilots near dancers are C: The pilots near the doncers ure...
Figure 8: Comparison of generated text. The prompt is italicized, ungrammatical or counter-factual responses are highlighted in red, and unchanged correct responses in green. shows the original GPT-2 XL's generation, and @ shows the edited model's response.
that ROME edit can only edit the exact association between the tokens in (s, t, r). As demon- strated in Figure 8.. editing the verb corresponding the authors from are to is only affects the subject the authors, and not other subjects such as the pilots. These look more like at-times brittle patterns of token expression than factual knowledge.
Table 1:
Results obtained under our new criteria suggest model editing methods are not robust.
DISCUSSION & CONCLUSION
Model
Data
Reliability
Measure
We find that several syntactic agreement phenomena can be localised to a small number of ML.P neurons. This locali- sation has similar characteristics the localisation of fac- tual information, suggesting that recent transformer-bat basdd language models' impressive abilities with respect vari- ous linguistic phenomena and the recall of facts from their training corpora may follow the same underlying mechanism. The localisation of the two types of information also faces the same challenges. however, which militate against the soundness of the KN thesis. Specifically, the effect of editing the identified neurons is not strong enough to overturn the final prediction, and the scope of the phenomena appears to be limited to shallow cues such as token co-occurrence statistics.
P101
99.82%
Synonym 52.35%
GPT-2 XL.
P1376
96.37%
Symmerry 23.71%
P36
99.79%
Symmetry 25.17%
P101
100%
Syaosym 58.36%
LLaMA-2
P1376
Symmetry 33.40%
P36
100%
Symmerry 33.64%
Returning to Geva et al.Js (2021) original findings, the MLP neurons store patterns that are in- terpretable through a linguistic lens, but they do not store knowledge, either linguistic or factual. Meng et al. s 2022) causal tracing results, although still oversimplification, suggest that there are different phases in different layers in the entire process of token expression. But their ROME model-editing method did not avail itself of this important finding. The method is still Maseb To achieve a better understanding of this expression process and achieve real model editing, we must examine the entire prcision-makinn circuit (Wang et al. 2022 Wu et al., 2023 Conmy et al. [023] Murty et al. 2023). Manipulating only the MLP weights is not enough. The circuit mode of inter- pretation still at a very early state of development, however. Current circuit identification methods are ad hoc, furthermore, and have only been applied to a small set of tasks. In future work, we will try to formalize the circuit interpretation framework and apply to more tasks and phenomena.
Our reassessment of causal traces agrees with Hase et al.ss (2023), but we take exception to their claim that "better mechanistic understanding ... may not always translate to insights about how to best change their behavior." It is well-established that we can interpret the computational mecha- nism of LMs through the lens of formal linguistics (Clark et al., 2019). Both of our findings reveal limitations in current LM interpretation work and suggest that even more comprchensive, but still mechanistic interpretation of transformers will lead to insights for better control of model be- haviour when not limited to the MLP modules, and when patterns of token expression are dealt with unencumbered by misbegotten metaphors about knowledge and human reasoning.
Contributions Our work provides a thorough examination of the KN thesis and finds that the thesis is, at best, an oversimplification We (1) extend KN-based analysis to well-defined syntac- tic tasks, (2) propose two new criteria for evaluating the effectiveness of model editing, and (3) introduce a generalised n-sample similarity measure of the level of localisation.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank Yu Lei (University of Toronto) for a great deal of insightful discussion. We also want to thank the anonymous reviewers for providing informative comments and suggestions.
REFERENCES
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models, 2023.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What Does BERT Look at? An Analysis of BERT's Attention. In Proceedings of the 2019 ACL Workshop Black- boxNLP: Analyzing and Interpreting Neural Nenworks for NLP, pp. 276-286, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653//1/W19/
Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In Thirty- Seventh Conference on Neural Information Processing Systems, 2023.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge Neurons in Pretrained Transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8493-8502, Dublin, Ireland, May 2022.
Association for Computational Linguistics. doi: 11266///1/2222.alllooooo
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing Factual Knowledge in Language Models. In Proceedings the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6491-6506, Online and Punta Cana, Dominican Republic, November 2021, Association for Computational Linguistics. doi: 10. 10.18653/v1/2221 http://///22.e...1--mm www.eeeenemmmaeeeeeee 522..
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 4171 117661 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/w1/N19. 1423.
Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. Analyzing Individual Neurons in Pre-trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4866, Online, November 2020. Association for Computational Linguisties. doi: 10. 12211663/11/2222.mml......
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and Improving Consistency in Pretrained Language Models. Transactions of the Association for Computational Linguistics, 9:1012-1031, 2021. doi: 10. 10.11622tacLa..00410.
Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1828-1843. Online, August 2021. Association for Computational Linguistics. doi: 10. N..18863/v1 1/252//111222112 aac.l.....-l-o.1144..
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer Feed-Forward Layers Are Key-Vat Value Memories. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 111853/v1222211 06653/112202.emmmmmmin..4444 .emnlp-main.446..
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does Localization Inform Edit- ing? Surprising Differences in Causality-Band -Based Localization vs. Knowledge Editing Language Models, January 2023.
10

---

Published as a conference paper at ICLR 2024
John Hewitt and Percy Liang. Designing and Interpreting Probes with Control Tasks. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UCNLP), pp- 2733-2743, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. N0.18653/1//D11-12755
Jing Huang, Atticus Geiger. Karel D'Oosterlinck. Zhengxuan Wu, and Christopher Potts. Rig- orously Assessing Natural Language Explanations of Neurons. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi (eds.), Proceedings of the 6th BlackboxNI.P Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 317-331, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/v 002/655//112222..22 wlap2bobllccbbxolp-...
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What Does BERT Learn about the Structure of Language? In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pp. 3651-3657, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10. 111/P19966611111
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-Shot Relation Extraction via Reading Comprehension. Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL. 2017), pp. 333-342 Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. 1/.18666/1/K111100344
Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models: A cognitive perspective, January 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022.
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass- Editing Memory in a Transformer. In The Eleventh International Conference on Learning Repre- sentations, May 2023.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing scale. In International Conference on Learning Representations, 2022.
Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D. Manning. Characterizing intrinsic compositionality in transformers with Tree Projections. In The Eleventh International Conference on Learning Representations, 2023.
Jingcheng Niu, Wenjie Lu, and Gerald Penn. Does BERT Rediscover a Classical NLP Pipeline? In Proceedings of the 29th International Conference Computational Linguistics, pp. 3143- 3153, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language Models as Knowledge Bases? In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. China, November 2019. Association for Computational Linguistics. doi: 10. (0.18653/v1/D19-123 1250.
Hong Kong,
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners OpenAl Blog, pp. 24, 2019.
Student. The Probable Error of Mean. Biometrika, 6(1):1-25, 1908. ISSN 0006-3444. doi: 10.2307/2331554.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.18653/ W1/P191445 14522..
11

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, February 2023.
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy. Buck Shlegeris, and Jacob Steinhardt. In- terpretability the Wild: A Circuit for Indirect Object Identification in GPT-2 Small. In The Eleventh International Conference on Learning Representations, September 2022.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman, BLiMP: The Benchmark Linguistic Minimal Pairs for English. Transac- tions of the Association for Computational Linguistics, 8:377-392. July 2020. ISSN 2307-387X.
doi: 16.1162/tacl a 00321.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. HuggingFace's Trans- formers: State-of-the oh-thf--ttttttto art Natural Language Processing. arXiv:1910.03771 [cs], February 2020.
Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman. Interpretabil- ity at Scale: Identifying Causal Mechanisms in Alpaca. In Thirty-Seventh Conference on Neural Information Processing Systems, November 2023.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Advances in Neural Information Processing Systems 32, pp. 5753-5763. 2019.
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing Large Language Models: Problems, Methods, and Opportunities. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 10222-10240, Singapore, December 2023, Association for Computational Linguistics. doi: 1118665//1/202..corl 120.../mmmmmmmmmmmmmmm3322
12

---

Published as a conference paper at ICLR 2024
Table 2: BLiMP phenomena and paradigms.
Phanomenon
Paradigms
Example
Anaphor
anaphor-gender.e gender_ngreeuneee
Katherine can't help herself/"hmsellt
Agreement
amphorr number.agrremeeee...
Many leenagers were helping themselves' heesell
determiner.nov noun www.nnnnnnnnnooonno Igreementt_
Craig explored that grecery storef"groorr stores.
determiner moun www.nooooneeenn agreement.
Carl cures those/" horses.
determiner goounnggreeeett imegular.
Phillip lifting this mouse//this mice.
Determiner-
determiner_nour cowiineerroou__e www.noooonnnnnnene Jggeement, jiregular,
Those ladies walk through those/ "that cases.
Noun Agreement
determiner noun www.aaeentttttte.... with.adj_1
Tracy praises these lucky guys/"guy guy.
determiner agreement with adj 2
these/ this gray books. This person shouldn't criticize this upset child/"hhilden.
Same
determiner_nov pnnemmnnr__ounn..e www.:oooonnnnnnneeee wwt.eeeettt.wiih...m.o. withh_ttttwtt__iiii.___ _rreguurr
determiner noun agreement adj irregular
That adult brought that "tose purple octopus.
distractor, aag.reeentteeeaatiee
noun
A sketch lights dresn '''''n' appear:
distractor agrecment relative. ineeullr plural subject verb agreement
Boys that aren't 't disturbing Natalic suffers
Subject-
Verb
This goose isn werennn' I bothering Edward.
panular.puural.sujjce verb///ceeeeb.erremeen..... regular eegularppuura_suu subreta www.pccc..erb.ggeeeeee.. agreemetttt
The women/ women cleans every public park.
Agreement
Jetlrey Iasn'l"hiver haven't criticized Donald. The dressss crumples.
regular plural subject. agrbbaggeemmnt.222
A EXPERIMENTAL SETUP
A.1 LANGUAGE MODELS
We experiment on BERT (Devlin et al. 2019), GPT-2 RRdffrdd et al. 2019), and LLaMA-2 (Tourron) al. 2023). We use the bert casee-caee version of BERT, the base version GPT-2 and 7B parameter version of LLaMA-2 in Section We added GPT-2 XL in Section 4 as it also used by (Meng et al. 2022). We use the huggingface package (Wolf et al. 2020 for the implementation.
We choose BERT and GPT-2 as they are the most widely studied and applied language models. We choose LLaMA-2 representative of the recent large language models. All three models are transformer-based
BERT is a masked language model (MLM), and GPT-2 and LLaMA are decoder-only language models. MLM is type of bidirectional language model. It can process context in both the forward and backward direction and the order between source and target is not required. However, decoder-only language models only process context from left to right and this means that the target must locate at the end of the prompt. Therefore, some linguistic patterns are not suitable for decoder-only LMs. We discarded these patterns for GPT-2 and LLaMA.
For the sake of consistency with Meng et al. 2022), we use GPT-XL for causal tracing and the evaluation of ROME. We also evaluate LLaMA 2-7B on ROME as representative of recent LLMs. Yao et al. (2023) provided a recipe of applying ROME on LLaMA-2-7b6 we follow their instructions and hyperparameters to conduct our evaluation.
A.2 CORPORA
BLiMP We use the linguistic phenomena collected in BLiMP (aarstadt al. 2020 for our anal- ysis. The BLiMP corpus contains minimal pairs for 12 grammar phenomena. Some of the phenom- ena are suitable for our experiments and therefore discarded. The remaining phenomena and paradigms are shown in Table
ParaRel
The corpus PARARES (Elazar et al. (021] contains facts formulated as fill-i--hhe-blat cloze task and is curated by experts. It contains 38 relation types and Table
overview
of an overview the PARAREL corpus, and 27,738 relational facts in total. We obtain prompts from PARAREL following Dai et al. ss (2022) instructions. For each PARAREL relations, Dai et al. (2022) created multiple prompt templates. On average, they created 8.63 different prompt templates for each of the relations. In total, the PARAREL corpus contains 253, 34,444 prompts.
13

---

Published as a conference paper at ICLR 2024
Table 3: An overview of PARAREl relations, There are bijective relations: P1376 (capital of) and P36 (capital). Both relations are underlined.
ID
Relation
Relation type
P1001
applies to jurisdiction
N-M
PIO1
field of work
N-M
P103
native language
N-1
P106
occupation
N-M
P108
employer
N-M
P127
owned by
N-1
P1303
instrument
N-M
P131
located in the administrative territorial entity
N-1
P136
genre
N-1
P1376
capital
P138
named after
N-1
P140
religion
N-1
P1412
languages spoken, written or signed
N-M
P159
headquarters location
N-1
P17
country
N-1
P176
manufacturer
N-1
P178
developer
N-M
P19
place of birth
N-1
P190
twinned administrative body
N-M
P20
place of death
N-1
P264
record label
N-1
P27
country of citizenship
N-M
P276
location
N-1
P279
subclass of
N-1
P30
continent
N-1
P36
capital
P361
part of
N-1
P364
original language of film or TV show
N-1
P37
official language
N-1
P39
position held
N-M
P407
language work or name
N-1
P413
position played on team / speciality
N-1
P449
original network
N-1
P463
member of
N-M
P47
border with country of origin
N-M
P495
N-1
P530
diplomatic relation
N-M
P740
location of formation
N-1
P937
work location
N-M
There are two bijective (1-1) relations: P1376 (capital of) and P36 (capital), We use those two relations for our bijection reversal relation evaluation.
CounterFact and ZsRE Meng et al. (0022) processed PARAREL relations differently from Dai al. (2022). In particular, they did create the new prompt templates for each relational facts.
The Zero-Shot Relation Extraction (zsRE) corpus used by Mitchell et al. (002)): De Cao et al. (021 is another popular corpus used to evaluate model editing methods. The evaluation slice contains 10,000 records.
Both corpora are used by Meng et (0022) and Yao al. (023). We did not use these corpus in our experiments, however, we are citing results conducted by Meng et al. (2022) and Yao et al. (2023) avoid duplication. Our analysis of factual information can be easily generalised these two corpora and we plan to expand our analysis to more corpora for future work.
14

---

Published as a conference paper at ICLR 2024
(a) Effect of suppressing
(b) Effect suppressing 1833 together with 12000
Probebilty Change
0%
Probabilcs Change
-25%
-50%
50%
-00%%
thoss
bwo both mukkipe several carious this that on
non the aar
all soo
nul on cetten Marzee
there
those
Deo
both muliipee Sano thi
sceeral
that Con
my some
our
soathere
Figure 9: Despite w/sss 's high occurrence Det-N pairs with this and that, because its 3% strong appearance in Det-N pairs with these, it is not singular grammatical number KN.
B DETERMINER GRAMMATICAL NUMBER KN SEARCH
B. PRENOMINAL MODIFIERS
We study the following prenominal modifiers:
Determiners:
The demonstrative determiners used by BLiMP: this, that, these, those;
Plural determiners: two, both, multiple, several, various;
Singular determiners: one, an, every;
Determiners that not express number agreement: the, some, any, all, my, our,
Adjectives: single, unique, sole, seattered, unified.
The six modifiers that do not express grammatical number agreement: any, all, my, our scattered and unified are underlined. However, these modifiers are used more often with numbered nouns because of their meanings.
B.2 THE KN SEARCH
Table shows the full list of neurons identified for the paradigm. We identified two knowledge neurons. The plural neuron w000 is highlighted blue and the singular neuron w is highlighted in red.
The neuron wisss is an interesting case. It appears as a knowledge neuron in 92% and 90% of the determiner this and that, and only 3% of the determiner these. However, these 3% of the neurons singular neuron, or using the neuron on its own does not should
very strong. Including 11 good localisation of grammatical number (Figure 9). Therefore, for our KN search, we excluded these neurons.
15

---

Published as a conference paper at ICLR 2024
Table 4: Neurons identified using BLiMP's determiner noun agreement 2 paradigm.
Neuron
this
that
these
those
0.00
0.00
1.00
1.00
1
k35
0.92
0.90
0.03
0.00
1006
0.88
0.81
0.00
0.00
g08c
0.00
0.85
0.00
0.00
0.00
0.14
0.00
"1200
0.04
0.00
0.12
0.00
0.01
0.00
0.07
0.03
10
0.00
0.00
0.07
0.00
0.00
0.07
0.09
0.07
0.00
0.00
0.00
10
0.00
0.00
0.06
0.06
w 11
900
0.03
0.03
0.06
w
1405
0.04
0.01
0.00
0.00
0.00
0.00
0.03
0.00
"ogo
0.03
0.01
0.03
0.00
1106
0.00
0.00
0.03
I6
0.03
0.01
0,00
0.01
wi ap4
0.03
0.00
0.00
0.00
0.00
0.01
0.00
L1p26
12112
0.00
0.01
0.00
0.00
mi 125
0.00
0.00
0.01
0.00
706
0.00
0.00
0.01
0.01
0.00
0.00
0.01
wynn
0.01
0.00
0.00
0.00
un pas
0.01
0.00
0.00
0.00
3900
0.01
0.00
0.00
0.01
w $24
0.00
0.00
0.00
0.06
Wiuus
0.00
0.00
0.00
0.01
Wggga
0.00
0.00
0.00
0.01
w
0.00
0.00
0.00
W156s
C KNS FOR LINGUISTIC PHENOMENA
In this section, we are going to present the KN search for other linguistic phenomena and models. Table 5 and lists all the neurons we identified for each of the BLiMP paradigms and language models. Some of the paradigms such as determiner coweeenoouuccc .ouunmagremeet... 2 are not applicable for decoder only language models because the target precedes the source. We do not evaluate these paradigms for the two decoder-only language models.
Similar to the search for determiner noun agreement 2 KNs, we first identify neurons with com- mon appearance in the prompts with certain grammatical property. Then, we manually test if suppressing these neurons can lead to significant model behaviour change. The number of neurons we identified is different across model and paradigms. The exact pattern behind this difference may need more investigation. We leave for future work.
Please consult our code release repository for any updates, detailed analysis and documentation of these KN search results.
http:://gituuu..cm//annnnnggg thesis
16

---

Published as a conference paper at ICLR 2024
Table 5: BLiMP phenomena and paradigms.
Phenomenon
Paradigms
Property
Model
KNs
Anaphoc Agreement
mapbor gender
m
BERT
" n 18s
w 1845
agrocment
GPT-2
Woss wHt. witt widees whi1o
LLaMA-2
w
u wibsas
BERT
wfr1a "B42
GPT-2
"L40a: "1259 M1801 23008 Wi0422
"Pecs
LLaMA-
w
3882 Weude W2dne: 17844
aiaphorraaaae agrecment
48
BERT
W1712
GPT-2
"
"1801 10
LLaMA-2
5227
pl
BERT
W2070
GPF2
W "3000-
wide
L.aMA-2
1742. w 11 m1
Determiner- Agreement
datemmiine..nou agreement
BERT
"aga "2ga1- wlagz- "2000- wagn 22g3 wi3e- w0
"ings- "1i43 warry 14418 "2u2 384 wise whis wite wioo 12
"Man ingay "1i43 " 1227 "is0o "iigs "Mjee "har
wasnn w52zn, whinn. whisn. wxrre winn w sis
GPT-2
"ho "Bg. wYag 202004 "fat "fa1 wlszo- "f707 "foco- wasrr: "273a "laor: wbr widge
LLaMA-2
12279, 30 wiree- wadur wikre w 2Bler waiss w wain. waas. wism, wis12, whise wa whis w wat waps- w wo wi wil wil wil "d5t- I1 "i0go "isee "iiss " 1777 Mises "isss 22645 #207s "280s
pl
BERT
will mil
2PT-2
W4s "fora wp 155 wo 2a 1382- w aon-- wfo W2azg 2702-
"Hoss "is71 md willss wasas witt "dās7
LLaMA-2
W6000 wheas- woloo wife 30 28810
wa228
w12e7 7085-
WSETS
detemminner
98
BERT
agreamennt
pl
BERT
"To04
determiner noun
BERT
whss- wheg- wlzzz- wāas whi4s- wassa- wil walls wilas-
gerrement_jmegalae
Wiiss: wissn- waa7- was1- wipe- wage. w1800- 41350- W1521-
w2207 whese 22092 w5870- w2000 "3220
GPT22
"fy Wssss ui w 22110 wa woons- 1.2 5584 "671
LLaMA-2
with w0 saar wasss wannn will widnos
pl
BERT
wisoa: w "inne- wh1ss- w/e w1o, w/frs wiser wstto-
"ii wit whio wida. wtdd "idso- "187a- "lise wiber w2dro w2520 whica whirs whoos
GPT2
wiss 200 wIsTE. wās- wPuz. 1o Ji. wit w
LLaMA-2
w0 wi 20 W0 20 W0 20 wate
determiner
BERT
22000
2ggrememiiimgglaa.2
BERT
W1004
17

---

Published as a conference paper at ICLR 2024
Table 6: BLiMP phenomena and paradigms (continued).
Phanomencn
Pandiigs
Propery
Moxel
KN:
Detemminer-
detemminerr
BERT
whe whess wāor- wisss, wāser. wansr- whor w 1980. "1270- wi3s2 wifer wilsss w2l2e wisas- wtd. w "161- " "isso " "is21 221ry #2647
agremess_with..d_...
Agreement (contmmed)
GPT-2
aa sass wisr1. "1223" "SS00 wau "iSt" w138 "2178
2738 "Hio "ihss: 4/2044
LLaMA-2
8877 Whi7r "1876 widen wa 8001 62336
pl
BERT
"2052 waps: w w1343- Wis12- "2i08 w2g2r- who w w545
Liss w1824. wilss. wibss. wlla who1 walo- whe2. whoc- waee. worn- whiou whire. whius
GFF2
WaRa t wegs. "2np. wi wipe- wheg. w wis w Ha. whee. S
" 2408 "2005
1LaMA-2
"
6133
M vn 2262
30
"
"3618
12228
30
12577 7805-
determinet noun
58
BERT
W00G
Jgreamentt wwtpaeeennwtwtiiiialeeee wit
pl
BERT
"fo94
detemiiner
8
BERT
wr vasa 10 14500 22sor- ur wise wall7: wisen weat "iiss
agreemeswwtth Jut Jnfh.ireglaar1
wr wioss: w5004 w 18o w261 wi5ds woteee wtzze wisin "id00-
widai- "iiae "5070
GPT-2
"Yees wlsss wisri. wTass: s0a0 wilh wats wass wafrs- W3738: wrdo- 1d W2044
LLaMA-2
30 "7003: u "
30
30 Will
7988- u 5230
pl
"Tas 2 "2ins: waas w0 10
"Isi "ies with w3or
wisss "isise Wisus w2000 Waton: W2021 "3704 Wapsa 22005 #3637
110 "sa2- "Bez- wil wil
GFF2
22702 "Looa- wig w9o0- wi00s- "S0as wath 2287 2400-
12200
LLaMA-2
WANTS 114117- w10s52 lusss Wanst Wil
30 wi
datenmiiner.
98
BERT
W3006
agrecment
pl
BERT
"Y004
,adi, inegulare
Subject- Verb Agreement
regular.poural..o subject verb agreementt 1
BERT
w w18
GPT-2
wi 12220 waiTa waisT "877
LLaMA-2
pl
widza w/307
GPT:2
"fo1- whore: wa001
wil w1aer. wiloo- wHt. welr. wilh wid43
LLaMA-2
wLosa: 18270 1880
regular.poural.cu subject veeb agreemment
%
wi3so
pl
BERT
w w m200s
tragsllarpplural_uuue .verb verbbagneemmeni.
BERT
on 1833 wi
GPF2
wil whira whl 40 w.o,
LLaMA-2
pl
BERT
w107g- w1507
Wiui- whera, 1 whar.. wilnow wi!.. will. wile. willss
"3001
LLaMA-2
30
2122 200000 117609
Amegular.ploral_soue .verbb
85
BERT
13350
pl
BERT
w 19 wides
18

---

Published as a conference paper at ICLR 2024
Table 7: BLiMP phenomena and paradigms (continued).
Phesomenco
Panddggs
Progerty
Model
KNs
Subject-
distrcio
BERT
1d " on
Verh
Jetational youn
Agreenentt
(contimed)
GFT2
w.P.. 1.. wigns whira. whiNT
LLaMA-2
BERT
Waass widor
GFT2
w wheTa wanss
wol w/aar. walnos wit. wolr. wilee wlbas-
ILaMA-2
"B27g milsee milsa w7842
distaskor.agreemeee Jela
BERT
199 1951
19239: w3830:
whira walsr
LLaMA-2
w31
pl
BERT
GPT2
w "in-
WheTS un 1318-
whiao wit. witr. wadee widss- wIST.
w3001
11aMA-2
"B279
2212
2009 w7830
C.1 EFFECTS SUPPRESSING THE KNS
Figure 10 shows the probability change after erasing the identified singular or plural neuron. All the results are similar the base case paradigm presented in the main section of the paper. We can observe similar levels of probability change as determiner noun aogneemeemm 2 with or without adding distractors (adjectives and irregular verbs).
Figure 11 shows the probability change of subject-verb agreement paradigms GPT-2 and LLaMA-2. For GPT-2, we can see that the suppression of the singular neurons causes significant probability decrease for singular verbs and essentially no effect to plural verbs. The effect of inter- vention is more pronounced for the plural neurons. For LLaMA-2, however, as the model becomes larger, the relative importance of each neuron becomes smaller. Therefore, we observe that the effect of KN editing is less pronounced.
19

---

Published as a conference paper at ICLR 2024
Effect of suppressing the singular neuron Mapee
Effect of suppressing the plural neuron sia
100%
Probobility Change
500%
50%
4077
Probebiitt
0%
200%
1007%
-100%
-107%
Thone
those
muttple
weers uniquio single sole the
any
ms some
our unific ocsttaro
thos
thoie
DVO
both
muliipl- servral varizus this
that
(a) Base case Det-N agreement: determiner noun www..ouu.nnenneeee agreement 2.
Effect suppressing the singular neuron Mas0e
Effect of suppressing the plural neuron siae
Change
Proboblity Change
500%
0%
300%
-30%
100%
-100%
-100%
these those
over unique single sole the
any
all some mn our
unifec scatters
thos
those
teo
both
multipl- sever verious thi thot
my al
our soafiurrd
(b) Det-N agreement with irregular noun inflections: determiner .noun
www.:nnuunnnnnnnnneeee www.aaagrgmett..mmm with 22
Effect suppressing the singular neuron W2006
Effect of suppressing the plural neuron aim--
500%
100%
Probabiity Change
40075
Probabiity
207/
-50%
100%
-100%
-100%
theee
Those
every un tiii singr sot the
any
all
some nns
our grfee
these
those
beo
both
multiple-
secera
verious
ths
that.
one
mi
our
staftored
(c) Det-N agreement with an adjective distractor: determiner noon www.nouuuunnnmmeeem pegrrementt.rregulae
Effect of suppressing the singular neuron (00)
Effect suppressing plural neuron woese
100%
Probability Change
200%
Probobilty
0%
100%
50%
0%
100%
those
both
20 eowey triqus
siggee sole
the
any
some my
unified scatterdd
these
multile
chat
onc
every une Side Sale
sceltered
(d) Adjective distractor and irregular nouns: heaeminie..oumn.ggeeennn.......d inreuulllar
Figure 10: Effect of suppressing the KNs of type determiner-noun agreement BERT. We can
observe similar levels of probability change as determiner htwp:mminer.muunnaeeeeeenttt with or without adding distractors (adjectives and irregular verbs).
20

---

Published as a conference paper at ICLR 2024
Effect suppressing the singular KNs
Effect of suppressing the plural KNs
Probab Channee
0%
Peobabiitty
100%
2007
-00%%
10
WCDD
howe
ran
fight
fights
thrww
thows
505
hac
nune
lights
throws
(a) Base subject-verb agreement: regular plural.subject.
subject. verb agreement
Effect suppressing the singular KNs
Effect of suppressing the plural KNs
0%
Prebobiiity Change
290%
-200%
Probobiitty
0%
-00%%
-250%
have
100
thow
Unross
1000
hac
Nigh
500
(b) Subject-verb agreement with irregular verbs: irregular httpplllarpplup__pee wutluralssujjett.err..ee agreemmett
Effect suppressing the singular KNs
Effect of suppressing the plural KNs
0%
Probabilay Change
Probablit
-000%
-000%
-400%%
15
have
Pun
Fight
Fights
thro
Linows
5005
100
hac
runs
fights
thmo
500
(c) Subject-verb agreement with relational noun distractor: http:accrr..ggremmmeeerrrtttttoo nounn
Effect suppressing the singular KNs
Effect of suppressing the plural KNs
0%
Probabiltt Change
200%
100%
-200%
15
have
PUn
fightt
thoow
are
runS
Fights
thooo
500
th
(d) Subject-verb agreement with relative clause distractor: distractor distractor.agreemeet relative clause.
Effect suppressing the singular KNs
Effect of suppressing the plural KNs
Probabiity Change
216
Probability Change
20%
10%
0%
reference
infennces
rUTS
Right
thows
nove
reference.
referrncee
right
Rights
throw
have
has
(c) LLaMA-2 results on regular_plune plurallssubjet..ve suahjettverhhaagreemetL...
The effect of editing is less pronounced on
larger LMs, as each neuron has smaller relative importance.
Figure 11: Effect of suppressing the subject-verb agreement KNs on GPT-2.
21

---

Published as a conference paper at ICLR 2024
Jicersiaet sereer Irgguae.
522 marere: 390 www.gneetttt.....
(a) Layer distribution of BLiMP determiner-nour agreement paradigms.
dttreder
(b) Layer distribution of BLiMP subject-verb agreement paradigms.
(c) Recreation of Dai al. (022 KN distribution result ParaRel relations (Figure 3).
Figure 12: Percentage of knowledge neurons identified in different BLiMP paradigms using BERT.
C.2 LAYER DISTRIBUTION OF IDENTIFIED KNS
In Figure 12
we analysed the layers of the identified KNs on each BLiMP paradigm and compare it with Dai et al. (2022) finding on PARAREL. We notice that the vast majority neurons of all types distributed in the topmost layers of BERT. This result confirms Dai al. (2022) observation but disproved their position. There nothing unique to fact-related neurons. This finding agrees
with Niu et al.]s s (202)) refutation of Jawahar et al. (2019) and Tenney et al. (2019). Syntactic information (formal competence) and semantic information (functional competence) do not occupy different layers of the language model.
C.3 LEVELS LOCALISATION
Table shows the levels of localisation on BERT across all determiner-noun agreement (DNA) BLiMP paradigms. These measures are comparable PARAREL results shown in Table
22

---

Published as a conference paper at ICLR 2024
Table 8: Levels of localisation of determiner-noun agreement.
BERT
GPT-2
LLaMA-2
Paradign
[KN]
R
[KN]
R
[KN]
RG
deterniner_nowe .ow.eeeeeroooneeeeeeeeeee noun_ngreement_1
3.94
0.71
0.56
0.06
0.45
0.15
3.38
0.44
0.24
determinerr agreementt22
0.56
determiner nown agreement irregular.
5.53
0.73
0.64
1.32
0.58
0.24
4.93
0.48
0.46
deterniner_noww www.:/onnnnnnnooooooeoooo wwrremnnr.ttttt.oo .iregntlr..rgrar2222
2.45
0.67
0.55
deterniner iggeeeeent wwtheetwwwihhhhiil wiib_andjective11
8.88
0.78
0.67
1.31
0.62
0.17
3.38
0.44
0.24
determinerr moun agreemenni, with adjectivv
2.26
0.67
0.57
deterniner waapeer__oonnnopnneeee wageemggeemenngg with_hhhhiiii wiith_adf.immguuarr
9.79
0.78
0.67
0.12
0.51
0.15
4.55
0.45
0.48
determiner noun wgrcementemmtttt..o. with wdj irrouullll
2.60
0.69
0.58
Table 9: Levels of localisation of different PARAREL relations.
BERT
GPT-2
LLaMA-2
Relation
KN
7
[KN]
T
R2
KN]
7
PIO1
0.167
0.515
0.399
1,537
0.708
0.278
1.0
0.61
0.306
P103
0.204
0.662
0.399
1.968
0.649
0.375
7.53
0.733
0.410
P106
1.292
0,607
0.365
10.090
0.853
0.599
0.28
0.438
0.258
P108
1.493
0.663
0.473
10.433
0.848
0.269
18.1
0.735
0.599
P127
1,512
0.630
0.552
11.758
0.769
0.549
0.3
0.585
0.163
P1303
10.462
0.814
0.684
12,453
0.771
0.573
0.3
0.63
0,303
P136
14.862
0,856
0.646
14,435
0.878
0.677
2.1
0.64
0.754
P1376
15.640
0.842
0.628
14.892
0.794
0.624
1.4
0.592
0.435
P138
16.992
0.958
0.874
15.365
0.794
0.621
1.4
0.62
0.605
P140
2.008
0.689
0.263
16.543
0.848
0.707
0.7
0.64
0.290
P1412
2.196
0.687
0.612
18.286
0.782
0.618
1.8
0.66
0.590
P159
2,200
0,666
0.392
19.285
0.838
0,626
9.5
0.765
0,637
P176
2.376
0.680
0.254
19.299
0.823
0.650
2.7
0.675
0.524
PI78
2.553
0.686
0.392
19.898
0.828
0.686
2.1
0.65
0.414
P19
2.597
0.693
0.481
2.603
0.661
0.417
0.24
0,482
0.526
P190
2.920
0.673
0.308
21.989
0.949
0.757
0.8
0.54
0.216
P20
28.993
0.883
0.644
22.984
0.871
0.671
0.52
0.458
0.377
P264
3.335
0.667
0.213
23.082
0.867
0.685
1.7
0.605
0.416
P27
3.925
0.722
0.483
29.296
0.862
0.619
0.67
0.376
P279
4.286
0.691
0.657
3.437
0.649
0.360
14.7
0.755
0.683
P30
4.338
0.698
0.449
42.128
0.856
0.642
3,27
0.673
0.288
P36
4.428
0.714
0.619
5.195
0.776
0.472
4
0.645
0.442
P364
4.541
0.709
0.553
5.545
0.707
0.481
3.47
0.69
0.478
P37
4.876
0.718
0.477
5.863
0.787
0.489
0.7
0.56
0.447
P39
5.230
0.722
0.413
6.051
0.683
0.450
0.6
0.59
0.243
P407
5,456
0.730
0.611
6.149
0.680
0.483
0.04
0.358
0.280
P413
5,632
0.809
0.539
6.346
0.740
0.522
0.9
0.63
0.252
P449
6.005
0.710
0.717
7.150
0.743
0.501
0.6
0.53
0.373
P463
6.088
0.776
0.467
7.697
0.735
0.437
1.7
0.65
0.489
P47
6,356
0.776
0.739
8.078
0.701
0.524
11.7
0.8
0.730
P495
6,455
0.709
0.513
8.209
0.744
0.569
1.0
0.505
0.585
P530
7.280
0.785
0.688
8,598
0.765
0.558
1.84
0.552
0.615
P740
7.324
0.769
0.518
8.840
0.737
0.477
2.1
0.505
0.796
P937
8.623
0.725
0.572
9.594
0.782
0.621
2.53
0.673
0.752
D CAUSAL TRACING
In this section we present an extended analysis our causal tracing analysis. We use the same hyperparameters and settings as suggested by Meng al. (2022).
Meng et al. (2022) causal tracing experiment is conducted three steps:
23

---

Published as a conference paper at ICLR 2024
1. In the clean run, they pass a prompt into the model and record all the hidden activation values.
2. Then, they conduct corrupt run. They perturb the prompt's subject by adding a noise value to the input embedding. Because of the obfuscation, the model will like generate an incorrect answer.
3. Finally, in the corrupted-witt-reetoooee with-restoration run, they let the model run with the corrupted value. But every time for pair of token and layer, they replace the cortupted output with the original clean state. Then they the model continue without further intervention. this restoration can attenuate the effect of obfuscation, they interpret this state as having strong causal importance. The difference between the corrupted output probability and the output probability after restoring one location is called the indirect effect of that location.
However, all of the analysis are based on observing individual causal traces. Meng et al. (2022) also proposed compute the average indirect effect over larger quantity of sentences. In Table [3] we present our analysis of average indirect effects across different types of information. The result confirms our findings in Section
24

---

Published as a conference paper at ICLR 2024
Patching bikdlen state comrptted input
cumupted input
Pasching comppted input
1aw-
Law
Lave
0.30
Taylor
Tayler*
Thyle**
0.25
profestemily
0.5 prlfosinally
rufessinally
plass
0.30
sport
spor
sport
0.35
01
0.1
of
0.10
15 20 30
10 20
35
15 25 40 #iloballl
aiggle paiched
byer withnn GPI-2-XL
center of Interal parche MLP boyes
center intevall parched Arm layers
(a) Many factual causal traces also do not show this distinction. Example taken from Meng et al. 2022 Figure 10c. The ML. module show causality both the carly and the late site.
ParstelF19
Parshl 712
1.08
Fre SOONEAaTOo0E
iustersaau
Buttor
(b) We reproduced Meng et al. 2022) calculation of the average indirect effect of individual model compo- nents. We do see the separation between MLP and attention modules. However, we can also see that there is weaker but discernible MI. causality at the late site. This shows that the previous example is not negligible anomaly. The causal tracing pattern less stable than Meng et al. 2022) originally conjectured.
Dee-N Agreement
Der-N Agreement
Agreament
2015
60100
Detemiierrbiee tokee
Derem
Deturbrer sahan
6.200
3013
0.050
6015
Last cslen
List 12
Lad twin
6.0225
AE
(c) The pattern that the MLP modules occupy early site and attention modules occupy a late site do persist for determiner-noun agrecment. We used the determiner crciincerroou__e aaruunaaggeeettwre.. wettt__tth___fhe vitthaddecttve__ paradigm for this experiment as it contains the adjective token that analogous to the first subsequent and further tokens that Meng et al. (2022) investigated.
SubuectVart Agrommatt
SedeeciMer Agrocmant
teu5
SuljectVarc Agmammnt
Sablectcken
Not
18
Lastttten
(d) The pattern of two distinct early and late sites is less apparent for subject-verb agreement. In fact, the MLP modules have the strongest causality higher layers (30-35) than attention (~~25).
Figure 13: Average indirect effect of different model component over multiple samples.
25

---

Published as a conference paper at ICLR 2024
Table 10: The construction of our evaluation corpus for relation symmetry and synonym replacement
Relation
Edit Prompt
Evaluate Prompt
Symmenyy
Template:
[S] the capital of [T [T=+"",
The capital of is
is IS" -SS
P1376
Example:
Ollawa is the capital of Canadu Camad--llaaly
The capital of Iualy is Rome OOllaa.
Symmotry
Template:
The capital of [S] is [T-+T).
[T") is the capital of -SS) the cupital Italy Canadda
P36
Example:
capital of Canada Otawa Roome.
Synonym
Template:
[S] works field of [T-T").
[S] is aTT, +TT]
P101
Example:
Amaxagoras works in the field of philosophy linguisties.
Amaxggoras famous philosopher -lnguusst.
E EVALUATION OF SYMMETRY AND SYNONYM
We construct the datasets used for both the symmetry and synonym evaluation from PARAREI. relations. Table 10 shows an overview of the data construction process.
Symmetry We used the two bijective relations: P1376 (capital of) and P36 (capital) evaluate the symmetry property.
For each P1376 (capital of) relation (s, (s,t,) r), edit the model using the prompt "[S] is the capital of" and change the target from t to t². t* is another city from the corpus. Then, we prompt the model with "The capital of [T*] is" and see if the model outputs s with higher probability than the original s*. We identified 234 relations in total.
Similarly, for relation P36 (capital), we edit the model using the original prompt "The capital of [S] is" and try to change the target from to L", 1" is another country/state from the corpus. Again, if the model outputs with higher probability than the original s", we count the evaluation as success. We identified 703 relations.
Synonym For synonym replacement, we use P101 (field of work). We first rewrite each field of work to the occupation name. For example, linguistics - linguist and aviation - pilot.
Through the process, we identified several mistakes in the original P101 data. For example, some of the field of work targets are already names of occupation, resulting in ill-formed prompts such as "Clyde Tombaugh works in the field astronomer." " Some of the fields of work, typically country names, cannot be converted into occupation names. For example "Mark Mazower works in the field of Balkans." " We discard data entries with these issues and we collect 50 distinct field of works with their occupations. Finally, we obtain 568 entries.
We edit the model with the original prompt "[S] works in the filed of and edit the target from to t" l" is another field of work collected from P101. Then, response from the edited model. Let f, and t be the two occupation names correspond to f and t", the original t..
the prompt "[S] is a" to elicit
want see if the model can also assign higher probability the new target synonym e than
For example, first we edit the model from "Anaxagoras works in the field of philosophy" to linguistics, Then, we prompt the model with "Anaxagoras is famous" an see whether the model assign a higher probability to linguist rather than philosopher.
26


Reviews:
[{"review_id": "o7UNI7il9c", "reviewer": "Reviewer_ZPK4", "summary": "This paper examines the “knowledge neuron” hypothesis in a number of pretrained LLMs, namely, that factual knowledge can be localized to a small number of neurons, and that ablation of those neurons alters the probability of, and/or the final chosen output token. It further extends knowledge to include syntactic or formal knowledge, and similarly finds small number of neurons that can be ablated to suppress their respective represented knowledge, particularly distributed throughout the later layers. However, through previous and additionally proposed metrics, in particular emphasizing bi-directionality and synonym-agnosticism, the authors argue that the discovered knowledge neurons cannot be considered to contain anything like “knowledge”, but simply conserve token correlations found in the training text.", "questions": "- one of my major concerns is the neuron selection procedure, which a priori limits the number of knowledge neurons to be 2-5. As I understand it, this procedure was not proposed by the authors, but in my opinion this process alone excludes the possibility of distributed representation, a much more reasonable null-hypothesis, resulting in a seemingly important discovery of knowledge neurons but in fact represents very little of the actual computations in the model. This is very reminiscent of the “grandmother” or “Jennifer Aniston” neuron type of work in neuroscience, and narrows the scope of the investigation arbitrarily and prematurely. Some investigation of distributed representation would, in my opinion, increase the impact and reach of this work\n\n- the paper seems to be “on the fence” and sometimes explicitly contradicting itself. For example, page 7 states “LMs process and express the two types of knowledge using the same mechanism.” while the high-level conclusion of the paper, iiuc, is that there is no “knowledge neurons”. I think it would improve readability if the authors can find a more consistent messaging.\n\n- the paper references a small number of previous works heavily, and without prior knowledge in the field, it’s hard for me to assess how much of the contributions are novel. The assessment of syntactic knowledge and newly proposed metrics are clearly new contributions, but a small “contributions” section explicitly and concisely summarizing this would be helpful.\n\n- most of the illustrative examples (Figs 2-6) are on the case of determiner-noun, and some successful examples of the other two cases (subject-verb, gender and number agreement) would be even more convincing. Apologies if I had missed this in the supplemental.\n\n- Formal and functional competence are referenced in the paragraph after figure 1, but without definitions, which can be confusing for a naive reader. The definition in the later paragraph was very helpful, and may be better if moved to be earlier.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall, I found the paper to be clearly written, except for some very technical and linguistics-specific concepts that warrant more explanation (and earlier). Its usage of intuitive examples and graphical illustrations throughout the text was very helpful for me to understand its arguments. Lastly, the experiments seem comprehensive, and convincingly demonstrates the authors’ two main claims: the existence of syntax-knowledge neurons in LLMs analogous to fact-knowledge neurons, and that neither sets of knowledge neurons can be considered to robustly represent “knowledge”.", "weaknesses": "Despite its technical soundness, I’m personally struggling to understand the significance of these findings on a larger scale, though I must admit that this is not my field. In particular, I feel that such detailed investigations of LLMs on a more “cognitive” level, i.e., assigning individual neurons to be representing concepts / knowledge wholesale, is orthogonal to dissecting the computational mechanisms of attention-based LLMs, and is more suitable for a conference like ACL. This is not really the fault of this particular paper, but the literature they attempt to address (which are predominantly published in ACL), though ironically, this paper raises exactly the point that such “knowledge neuron” search in LLMs may be ill-advised, given that these models are simply token sequence autocomplete machines. \n\nNevertheless, given its current scope and that we are explicitly asked to assess significance of contribution to the field (of machine learning), I recommend borderline rejection for ICLR (but would otherwise strongly recommend acceptance for, e.g., ACL!). But I would be willing to convinced that it’s within scope if the AC and other reviewers disagree.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lshau2x4wY", "reviewer": "Reviewer_c8FP", "summary": "This paper reassess the Knowledge Neuron Thesis in two ways, with syntatic minimal pairs and with generalizing to bijective relationships and synonyms. Theranalysis shows the limitation of current knowledge identification and editing, suggesting the need for more sophisticated understanding of inner mechanism of a language model.", "questions": "n/a", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper introduces many new practices for the rigorous study of knowledge neuron thesis, including using minimal pairs and t-test.\n2. Broadening the definition of knowledge neural and connecting it to prior works in linguist phenomena.\n3. Through and diverse analysis.", "weaknesses": "1. If I have to nitpick, section 4 felt a bit disjoint from the rest of the paper and is not fully fledged.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3kEEVUD9n2", "reviewer": "Reviewer_7T91", "summary": "This work revisits the hypothesis that knowledge in pre-trained transformers may be limited to a few neurons. They do this by running two different model editing methods to identify neurons that are responsible for specific syntactic phenomena using the BLIMP minimal pairs dataset. While they do find a small number of mostly local knowledge neurons responsible for syntax, they find that intervening on these neurons is not enough to change model predictions in a robust way.\n\nThe paper also borrows from past work on factual editing, and along with their own results, conclude that the knowledge neuron thesis is flawed because intervening on these neurons is not sufficient to reliably change model behavior. Thus, they call for a more \"holistic\" approach that studies not individual neurons but entire layer structures and attention mechanisms as the motif for interpretability.", "questions": "- I would be interested to know what the authors think interpretability research should focus on i.e. it appears that knowledge is mostly distributed and not necessarily isolated to specific neurons. There is some discussion around using attention weights and the underlying model circuit towards the end but it would be good to have a slightly more extended discussion around this.\n\n- Some missing references on \"decision making circuits\":\n  - Clark et al. 2019 (What Does BERT Look at? An Analysis of BERT’s Attention): They use attention patterns to identify syntactic knowledge in models.\n  - Murty et al. 2023 (Characterizing Intrinsic Compositionality in Transformers with Tree Projections): They identify tree-structured circuits as a way to study generalization in transformers.\n  - Wu et al. 2023 (Interpretability at Scale: Identifying Causal Mechanisms in Alpaca): Finds alignments b/w model hidden states and symbolic algorithms.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- The paper is extremely well-written with clear arguments, and experimentation.\n- The results presented bring a lot of clarity to interpretability of transformers\n- A lot of previous model editing techniques fail to systematically study if effect of edits are just local or if they are systematic, while this work does this very comprehensively.\n- In my understanding, there is no prior work that applies editing techniques to identify \"syntax neurons\" and this aspect of the paper is quite novel", "weaknesses": "One weakness of the paper is that some of the presentation of experiments could be cleaned up substantially. Some specific suggestions for improvements:\n\n- Results in 3.2 (first paragraph) seem quite loaded. This paragraph presents attribution scores, shows that identified neurons have regularity, the affect of causal interventions, how identified neurons have more to do with frequency cues than syntax etc. I think these results could be broken up into their own paragraphs. \n\n- It would also be great to clearly show which results are taken from prior work. I suspect atleast Table-1 and Figure-5(c) is taken from prior work?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)