Qualifying Knowledge and Knowledge Sharing in Multilingual Models
Authors: Nicolas Guerin
Ryan M. Nefdt
Emmanuel Chemla
Venue: Accepted by TMLR ()

Abstract: Pre-trained language models (PLMs) have demonstrated a remarkable ability to encode factual knowledge. However, the mechanisms underlying how this knowledge is stored and retrieved remain poorly understood, with important implications for AI interpretability and safety. In this paper, we disentangle the multifaceted nature of knowledge: successfully completing a knowledge retrieval task (e.g., “{The capital of France is __”) involves mastering underlying concepts (e.g., France, Paris), relationships between these concepts (e.g., capital of) and the structure of prompts, including the language of the query. We propose to disentangle these distinct aspects of knowledge and apply this typology to offer a critical view of neuron-level knowledge attribution techniques. For concreteness, we focus on Dai et al.'s (2022) Knowledge Neurons (KNs) across multiple PLMs (BERT, OPT, Llama and Gemma), testing 10 natural languages and additional unnatural languages (e.g. Autoprompt).
Our key contributions are twofold: (i) we show that KNs come in different flavors, some indeed encoding entity level concepts, some having a much less transparent, more polysemantic role , and (ii) we address the problem of cross-linguistic knowledge sharing at the neuron level, more specifically we uncover an unprecedented overlap in KNs across up to all of the 10 languages we tested, pointing to the existence of a partially unified, language-agnostic retrieval system. To do so, we introduce and release the Multi-ParaRel dataset, an extension of ParaRel, featuring prompts and paraphrases for cloze-style knowledge retrieval tasks in parallel over 10 languages.

Published in Transactions on Machine Learning Research (05/2025)
Qualifying Knowledge and Knowledge Sharing in Multilin- gual Models
Nicolas Guerin
nicolas, nicolas.p-giet p.guerin@gmaalllcommm com
Laboratoire de Sciences Cognitives et Paycholinguistique Dept d' d'Eludes Cognétives, ENS, PSL University, EHESS, CNRS
Ryan Nefdt
Ryan.Nefdt aefddattttta
University of Cape Toum University of Bristol
Emmanuel Chemla
emw.ruul.chemllaeeeess....
Laboratoire de Sciences Cognitives et P'sycholinguisique
Dept d d'Etudes Cognitives, ENS, PSL University, EHESS, CNRS
Earth Species Project
Reviewed on OpenReview: https: // openreview. met/ forum? id- hnpB3SRb2i
Abstract
Pre-trained language models (PLMs) have demonstrated a remarkable ability to encode factual knowledge. However, the mechanisms underlying how this knowledge stored and retrieved remain poorly understood, with important implications for AI interpretability and safety. In this paper, we disentangle the multifaceted nature of knowledge: successfully com- pleting a knowledge retrieval task (e.g., " "The capital of France is __) involves mastering underlying concepts (e.g., France, Paris), relationships between these concepts (c.g., capital of) and the structure of prompts, including the language of the query. We propose to dis- entangle these distinet aspects knowledge and apply this typology to offer a critical view of neuron-level knowledge attribution techniques. For concreteness, we focus on Dai et al.'s (2022) Knowledge Neurons (KNs) across multiple PLMs (BERT, OPT, Llama and Gemma), testing 10 natural languages and additional unnatural languages (e.g. Autoprompt). Our key contributions are twofold: (i) we show that KNs come in different flavors, some indeed encoding entity level concepts, some having much less transparent, more polysemantic role, and (ii) we address the problem of cross- linguuiii knowledge sharing at the neuron level, more specifically we uncover an unprecedented overlap in KNs across up to all of the 10 languages we tested, pointing to the existence a partially unified, language-agnoste retrieval system. To do so, we introduce and release the Multi-Paralh dataset, an extension of ParaRel, featuring prompts and paraphrases for cloze-style knowledge retrieval tasks in parallel over 10 languages.
1 Introduction
Recent advances in Large Language Models (LLMs) have led to models trained vast and diverse linguistic datasots drawn from across the Internet, incorporating numerous languages simultancously (Scao et al., 2023: Touvron et al., 2023; Achiam el al., 2024). However, these languages are not. evenly represented, and per- formance on low-resource languages often depends on lross-linguistic transfer from high-resource languages (Pires al., 2019; Lample & Conneau, 2019; Conneau al., 2020a; Huang al., 2021). Whether LLMs can develop common, language-agnostic representations that enable such zero-shot transfer remains an open question in the literature (Singh al., 2019; Kudugunta et al., 2019: Kassner et al., 2021). Kervadec et al.

---

Published in Transactions on Machine Learning Research (05/2025)
(a)
relation neurons
multilingual neurons
captal
cap
capeal
conspeeific
LLM
concept neurons
LLM
monolingual neurons
The tal England
Paris
La capilale
French
France
London
English
Figure 1: The Knowledge Neurons (KNs) hypothesis connects LLM success on fill-n-thhe-bannk cloze task (c.g. The capital of France is) to the activation of small set of neurons. (a) The same neurons can be selected (green) in response to single task, thereby qualifying in response to a range of tasks all concerning certain relations between concepts, thereby qualifying as relational neurons (c.g., capital of is a relation between France and Paris, between England and London, etc.). (b) multilingual LLMs, concept and relational neurons may be selected specifically for language or across languages.
concept neurons (about e.g., Paris)
(2023) extended this investigation machine-generated languages, revealing that different representations can emerge, suggesting multiple ways knowledge may be encoded in LLMs.
Understanding how Pre-trained Language Models (PLMs) store and retrieve knowledge is essential for en- hancing interpretability and safety in AT systems. Many recent studies have sought to localise and attribute specific knowledge to individual neurons within these models (Dai et al., 2022; Meng et al., 2022; 2023). These methods often attempt to identify neurons whose activations are critical for making accurate pre- dictions. Typically, they focus on neurons in intermediate layers of Feed-Forward Networks (FFNs) within transformer architectures (Geva et al., 2021). These approaches face strong limitations, as highlighted recent critiques (Hase et al., 2023; Niu al., 2023; Huang al., 2023).
In this work, offer novel perspective by refining the concept of 'knowledge' itself. To correctly complete prompt like The capital of France is, model must process multiple layers of information: sensitivity to the specific concept France, retrieval of the target concept Paris, and understanding the relational context capital sestlledee-cnccppuulll of. We introduce a method to distinguish these subtypes of knowledge conceptual and relational that is compatible with any knowledge attribution technique. We apply this method to the Knowledge Neurons (KNs) framework introduced by Dai et al. (2022), to provide a critical view on such a method and extend it to investigate how knowledge is shared across languages in PLMs (Figure 1).
Our contributions are:
We propose a finer-grained typology of knowledge, providing a critical perspective on neuron-level attribution methods like the Knowledge Neuron hypothesis, in particular its expectation of monose- manticity.
We analyze through this prism multiple PLMs (BERT, mBERT, OPT, Llama 2, and Gemma 2), revealing that substantial number of 'Knowlegge Neurons' exhibit polyscmantic behavior, while others are specifically responsive to individual concepts or relations.
We release Multi-Parane a multilingual version of the ParaRel dataset. (Elazar et al., 2021a), which includes 10 languages and is compatible with autoregressive models. The dataset is available here!.
We examine the extent to which knowledge representations are shared across languages at the level of individual neural units and demonstrate that I.LMs store knowledge similar neurons across 10 languages, and even in machine-generatte languages (AutoPrompt), suggesting shared cross- linguistic mechanism for knowledge retrieval.
The dataset is available
2

---

Published in Transactions on Machine Learning Research (05/2025)
2 Related Work
Multilingual Language Models The paradigm of Neural Machine Translation (NMT) has undergone a significant shift. Traditional approaches that relied on parallel corpora or even synthetic data built from unaligned monolingual corpora (Sennrich et al., 2016; Lample et al., 2018; Artetxe et al., 2018) are no longer the dominant standard. Instead, recent Large Language Models (LLMs) (Radford et al., 2019; Brown et al., 2020; Touvron et al., 2023; Achiam et al., 2024), trained on massive multilingual corpora scraped from the web, have become de facto multilingual systems. These models demonstrate strong cross-ligguistic capabilities across a wide range of tasks (Devlin et al., 2019; Lample & Conneau, 2019; Conneau et al., 2020a; Liu et al., 2020; Xue et al., 2021; Scao et al., 2023; Vilar et al., 2023; Peng et al., 2023; Hendy et al., 2023; Bawden & Yvon, 2023).
Understanding how multilingual LLMs acquire and represent linguistic knowledge is critical to identifying their limitations and potential risks (Garcia et al., 2021; Raunak et al., 2021; Akhbardeh et al., 2021; Bapna et al., 2022). For instance, Guerreiro et al. (2023) explores how hallucinations affect translation quality in such systems. Yuemci et al. (2024) provides a comprehonsive survey of multilingual LLMs (MLLMs), highlighting prevalent biases.
One core challenge is understanding how cross-linguistic capabilities emerge. Prior work has shown evidence of shared multilingual knowledge within models (Aharoni et al., 2019; Arivazhagan et al., 2019; Conneau et al., 2020b; K et al., 2020; Ri & Tsuruoka, 2022; Deshpande et al., 2022; Liu & Niehues, 2022; Choenni et al., 2023; Rajace & Monz, 2024; Chua et al., 2025), though some findings are mixed. For example, Kudugunta et al. (2019) uses Singular Value Canonical Correlation Analysis to show that language representations in a NMT model are similar particularly among related languages. Meanwhile, Singh et al. (2019) notes that mBERT tends to cluster representations by languago, suggesting language separation despite shared architecture.
This is the context for our work. While shared cross- linguiiin knowledge has been observed, we push this further by examining it at the neuron level. Most relevant to our approach, Chen et al. (2024) recently analyzed neuron overlap between English and Chinese. We extend this line of inquiry by comparing neuron activations across ten natural languages simultaneously. This broader comparison is essential. Pairwise analyses leave open the question of whether shared neurons generalize beyond specific language pairs. Overlap between two hangnages-cspcciall if one is dominant (c.g., English)-might result from coincidental or biased patterns. But consistent sharing across ten diverse languages indicates a more robust, symmetrical multilingual encoding. Finally, we introduce a comparison with an 'unnatural language' (Shin al., 2020), testing whether access to knowledge can be decoupled from linguistic form. These prompts are not human- interpretable and have been shown to elicit qualitatively different processing in LLMs (Kervadec et al., 2023), making them a stress test. for the generality of shared representations. Tn fact, such an inclusive approach to training data is now even vindicated beyond languages, using non-linguistic sounds for training in speech (see Poli et al. (2024)), or even using several species at once for bioacoustics (e.g., Robinson et al. (2025)).
Knowledge in LLMs LLMs acquire knowledge by training on extensive corpora (Petroni et al., 2019; Roberts et al., 2020; Safavi & Koutra, 2021). The work by Petroni et al. (2019) introduced LAMA, a dataset designed to evaluate BERT through a fillin-thhe-lank cloze task (e.g., The capital of France is [MASK].). Subsequent research has built upon LAMA (Jiang et al., 2021), highlighting the limitations of LLMs as knowledge bases (Flazar et al., 2021b; AlKhamissi et al., 2022; Wang et al., 2023; 2024b), while also attempting to enhance their performance (Wei et al., 2021; Petroni et al., 2020). Consequently, rescarch has emerged focusing on localizing and editing knowledge directly within the model (Radford et al., 2017: Lakretz et al., 2019; Bau et al., 2020b; Sinitsin et al., 2020; Mitchell et al., 2021; 2022; De Cao et al, 2021: Santurkar et al., 2021; De Cao et al., 2022; Bau et al., 2020a; Cohen et al., 2023).
In this context, knowledge altribution methods such as ROME (Meng et al., 2022) and MEMIT (Meng et al., 2023) (both employing causal mediation techniques; Vig et al., 2020), along with Knowledge Neurons (Dai et al., 2022) (utilizing an integrated gradient approach; Sundararajan et al., 2017), have been proposed. These methods are predicated on the assumption that neurons within the intermediate layers of transformers' Feed- Forward Networks (FFNs) encode knowledge. However, we align with other studies (Hase et al., 2023; Niu
3

---

Published in Transactions on Machine Learning Research (05/2025)
et al., 2023; Huang et al., 2023; Chen et al., 2025) that suggest this assumption may be an oversimplification. While certain neurons play a significant role in specific tasks (Takretz et al., 2019; Manning et al., 2020; Rogers et al., 2020; He et al., 2024), LLM neurons are not necessarily monosemantic; rather, they can serve multiple functions depending on the context and task (Adly et al., 2024). Furthermore, their effectiveness in altering knowledge subjective and widely debated (Hase et al., 2023). Other works (Wang el al., 2024a; Tang et al., 2024; Kojima et al., 2024) have identified multilingual neurons in LLMs; this paper focuses specifically on knowledgg-rellteed neurons, offering a more precise analysis. We propose a knowledge- attribution method-agnostia typology, illustrated with Dai et al.'s (2022) Knowledge Neurons. This approach aims to provide a critical view on the Knowledge Neurons hypothesis while exploring what insights it can offer regarding how knowledge is encoded in LLMs.
3 Methodological background
Knowledge The TREx dataset (Flsahar et al., 2018) is a collection of relational facts stored in triplets of the form- < r, 2, with r relation and h and entities entering in that relation. TREx exhibit 41 relations, such being the capital of, was born in, etc. Each full triplct can be referred to instantiation of its own relation
Knowledge Localization Methods Geva el al. (2021) observed that a FFN can be seen as a Gey-Value memory system, similar to self-attention. To assess if and where knowledge could be stored in FFNs, Dai et al. (2022) used a knowledge attribution method based on integrated gradients (sce next paragraph for details). They show that a fact (e.g., The capital of Fronce is Paris) can be associated to a few neurons (around 4), whose activations correlate with the probability of the model to fill in the elements of the fact appropriately. Similarly, Meng et al. (2022) proposed Rank-One Model Editing (ROME), which uses causal mediation to localize and edit knowledge in GPT, and Meng et al. (2023) introduced Mass-Edittee Euiisin Memory in a Transformer (MEMIT), which edits facts at scale. All of these knowledge attribution methods have their limitations; apply our analysis to the Knowledge Neurons by way of illustration. Our approach is applicable to all such methods.
Knowledge Neurons Dai et al. (2022) track Knowledge Neurons (KNs) during fill-in-thebllnkk cloze task (see also Petroni et al., 2019) based on TREx. Let w!") ()) be the jth neuron of the intermediate layer of the 7th FFN. The knowledge score of a neuron w!!) is calculated through the integrated gradient attribution method (Sundararajan et al., 2017), KNs are then filtered through thresholds. First, they retain only neurons with an attribution score greater than tun x max,, Attrr.p...(w!)) This procedure is carried out for each prompt associated with a fact < h, r,, >, and thus yields a sot of candidate KNs per prompt. Let denote N, the number of prompts for given relation r. To get results robust to noise, and to factor out signal associated to specific prompts rather than knowledge, they keep only neurons appearing in the candidate neurons set of at least Pkn x N, prompts. They propose thresholds of tRn 0.2 (only keep neurons scoring at least at 20% of the max attribution score) and pkn 0.7 (only keep neurons appearing in at least 70% of the different prompts for a given relation).
4 Method
Datascts For relational facts, we used the TREx dataset (Elsahar et al., 2018), which comprises relations with approxinately 1,000 facts per relation. For prompts, we employed the augmented version of ParaRel provided by Kervadec et al. (2023). This version retains only prompts compatible with autoregressive models and enriches the dataset with multiple paraphrasos for each relation. In Section 6, we explore multilingual models, which we tested on the multilingual variant of LAMA (Kassner et al., 2021) as well as on a new multilingual version of ParaRel that we introduce. We refer to this new dataset as Multi-ParaRel.
The detailed methodology for creating Multi-ParaRel, along with quality asscssment, is provided in Appendix A. Our dataset currently spans 10 languages: English, French, Spanish, Catalan, Danish, German, Italian, Dutch, Portuguese, and Swedish. We also investigate unnatural language: AutoPrompt. Following
4

---

Published in Transactions on Machine Learning Research (05/2025)
the same train, development, and test splits Shin et al. (2020), we trained 10 different seeds of Auto for each relation and each model. We also make these sets of prompts available.
Concept Neurons and Relation Neurons We propose simple typology that refines the type of knowledge attributed while answering fillin-the-blank cloze tasks. For example, correctly answering the question What is the capital of France? not only requires knowledge of the answer Paris, but also an understanding of the relationship between France and Paris. We thus introduce a simple principle: neuron that is hypothesized to encode a specific concept, such as one about Paris, should not be also responsible for encoding other concepts, and should therefore not be associated to other facts such as The copital of Spain is Madrid. If a neuron consistently encodes the same relation across multiple instances, we refer to it as a relational neuron, indicating that it is sensitive to a relation, such as copital of.
We thus define Relation Neurons as KNs that appear in at least i, x N instances of facts associated with particular relation, where N is the total number of facts, and 6, is a predefined relational threshold. In contrast, neurons that appear in less than te x N of the facts, for some other threshold to, are referred to as Concept Ncurons, as they are more likely to encode specific picces of knowlodge or information about individual entities.
The aim is to test the robustness of this distinction by investigating the role of the thresholds t, and te- A 'elcan' sccnario that supports the Knowledge Neuron hypothesis and the idea of monosemanticity would show that some concept neurons are found even for lc x =1 (very specific to concept), and relational neurons are found when in x N = N (completely systematically present for a relation). Alternatively, softer boundaries would suggest that these KNs play a more polysemantic and nuanced role, whereby knowledge is partially distributed across different neurons on different occasions (e.g., the concept of Paris and Madrid cannot be disentangled at the neuron level, or the relation capital of is not always encoded in the same way).
Multilingual Knowledge Neurons Similarly, we ask whether knowledge is language-agnostic; for ex- ample, humans do not need to relearn facts when acquiring a new language. Knowledge could be language- dependent in LLMs however: a fact is present from the English corpus but missing from a Spanish training corpus, an LLM may be able to retrieve that knowledge when prompted in English but not when prompted in Spanish. We employ the KNs framework to investigate the open question of whether common language- agnostic knowledge representation exists in multilingual models at the level of neurons.
We hypothesize that some KNs may be specific to one language, while others may be sensitive to prompts in multiple languages. We thus analyze the number of languages across which such neurons are shared. We do so by identifying KNs for relations in the ParaRel dataset across multiple languages, using the Multi-ParaRel dataset, which was specifically created for this multilingual evaluation.
5 Monolingual Experiments: Tracking Concept and Relation Neurons
5.1 Experimental Settings
Models We studied BERT (Devlin et al., 2019), and more precisely bert-base-uncased and bert-large-uncased, it has been the reference model for evaluation on TREx since Petroni et al. (2019). Having been trained on Wikipedia from which TREx is derived, their performance is very good (P@1> 0.4). We also studied OPT (Zhang et al., 2022) in its 350 million- parametersssss version opt-350m and 6.7 billion-parametenn version op 7b, Llama 2 (Touvron et al., 2023) in its 7 billion-aarameter version -lama-2-7b-be hf as well as Gomma 2 (Team et al., 2024) in its 9 billion parameters version gemma-2-99 For all these models we use the HuggingFace implementation. KNs computations were performed on NVIDIA Tesla V100 GPUs for models with less than billion parameters, and on NVIDIA Tesla A100 GPUs for larger models. The computation took less than an hour per relation.
Template filtering Per model, we excluded prompts with less than 10% top-1 accuracy (that is, accuracy of the most probable continuation). We then excluded relationships with less than 4 prompts left. Since all actual answers were made of single token, we also limit answers made of single token. After this filtering,
5

---

Published in Transactions on Machine Learning Research (05/2025)
Llama- -2-7b
P108
20
100
KNS Count
KNs Count
80
KNV Count
KNS Count
10-
20
10
20
10
0
10
10
10?
102
107
10?
105
10
10
109
107
10
Number of Instantiations
Number of Instantiations
Number of Instantiations
Number of Instantiations
(a)
P108
P159
P176
P19
1.0
avg. KNs proportion
avg. KNs proportion
avg. KNS proportion
0.8
0.8
0.6
0.6
avg. KNs
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
1.0
In
4.
In 1 tc
4. 1
(b)
Figure 2: Each panel corresponds to relation (P108, P159, etc.). (a) Distribution of KNs based on the number of instantiations (i.e. specific triplets, specific facts) within relation for which a KN was identified. A large number of neurons are identified as KN for a single instantiation, while a roughly similar number of neurons are identified as KN for continuously increasing number of instantiations within a relation. (b) Average proportion of the KNNs from a single instantiation which can be categorized Concept, Relation neither, according different thresholds (x-axis). The proportion of relational neurons is stable across different thresholds, the proportion of concept neurons decreases with more demanding thresholds.
obtained average 15 prompts per relation for BERT and prompts per relation for OPT (starting from 18), confirming the higher accuracy of BERT at the lask.
5.2 Tracking Typology of Knowledge
Before classifying Knowledge Neurons (KNs) according to our typology, first analyzed the distribution of KNs based on the number of instant iations for which KN was identified. Figure 2a illustrates the results for four relations using the Llama-2-25 2-7b model (complete results are provided in Appendix B). A qualitative analysis reveals two key findings: (i) many KNs appear in only one instantiation, indicating that these neurons are task-specific and sensitive to a single concept; and (ii) there continuons range KNs sensitive between 3 and instantiations, suggesting more nuanced role for these neurons that lies between relational and conceptual.
The second observation challenges the simplistic interpretation of assigning neurons exclusively to concepts. At the same time, it also demonstrates the presence of significant number of neurons sensitive to enough instantiations to hypothesize a more relational role in knowledge retrieval mechanisms.
Thus, we have identified potential candidates for the roles of both Concept Neurons and Relation Neu- rons, as well as neurons that fall into an intermediate category. The natural question that arises is: what is the proportion of each neuron type per instantiation, based on thresholds t, and t.? This information is not directly inferable from Figure 2a, neurons appearing consistently across instantiations are less visible

---

Published in Transactions on Machine Learning Research (05/2025)
bert-base-uncased
Llama-2-7b-hf
P@k cy -. 0.4
CO-0k-E-0.65
P0K-6- 0.6---0 0.4
CCP
0.050
0.025
0.05
0.05
0.000
CEPOK
0.00
0.00
OcActvotion
0.025
-0.05
-0.050
-0.30
-0.05
-0.05
-0.075
-0.15
-0.10
10
10P
101
10P
10a
10
P@k f.= 0.8 -008---02-- 0.2
CCP@ f- 0.8 0.2
P@kk 1 0.8 C. 0.2
CCP@k -, - 0.2
0.050
0.05
0.05
0.05
CCPSE
0.00
0.00
0.00
0.000
-0.025
-0.05
-0.05
-0.05
0.050
-0.10
10P
Figure 3: Boosting experiments results for wert-baaseuncaan (left) and L1 -lama-2-77 2-7b (right) for two couple of thresholds 4, = 0.6, L. 0.4 (Lop) and 4, 0.8, L. 0.2 (boltom). The lines corresponds to the AP@k (resp. ACCP@k) for different & values ranging from to 100. Thick lines represents the doubled activations results and thin lines the nullified activations results. We also plotted the standrad error accross the evaluated instantiations of the relations.
than neurons that appear uniquely in each instance. 2 To address this, we computed the proportion of each neuron type function of thresholds at the instantiation level (see Figure 2b). For simplicity, we used symmetrical thresholds, setting 4, = 1-1-t te.
As expected, when the thresholds become more restrictive, the number of neurons with well-defined roles decreases, giving way to neurons with less clearly defined functions across all relations. For the Llama-2-70 model, we observe that the number of neurons classified as Relation Neurons remains more stable compared to those classified as Concept Neurons. Furthermore, for a single instantiation, there are few KNs that are exclusive to that instance: when le 0.1, the proportion of Concept Neurons is less than 0.2.
Wc also examined the distribution of neuron types across the model's layers but found no significant variation. As observed by Dai et al. (2022), KNs are primarily concentrated in the final layers.
In summary, we have demonstrated the existence neurons reacting specifically to a single concept within a relation. We have also identified neurons that play a much broader role in such relations, with some reacting to almost all instances of that relation. We attempt verify this hypothesis through causal experiments in the next section. Finally, some neurons are activated by subset of the instantiations, carrying much less transparent type of knowledge. In principle, it could encode subtypes of relations, such as 'capital of European country', although we find this highly stipulative al the moment. In the next section, we will focus on concept and relation neurons and evaluate their role through causal experiments.
5.3 Boosting Experiments
In this experiment, we investigate the effect of either doubling or mullifying the activation of KNs on model predictions. Dai et al. (2022) conducted similar experiments, focusing on how manual changes to neuron acti- vations influenced output probabilities. In contrast, we employ two more concrete impact metrics: precision at rank k, denoted P@k, which measures the proportion of correct responses in the top model predictions, and correct category proportion al rank k, denoted CCP@k, which reflects the proportion of responses in the correct category (e.g., capitals) within the top k predictions. The original metric of relative probabilities
FFor example, if each instantiation contains 10 KNs, including 2 perfect conceptual neurous and 8 perfoct relational ncurons (present only instantiation and all instantiations, respectively). Figure would display bar of 200 at the abscissa and bar of 8 the 100 abscissa, which would obscure the predominant role Relation Neurons.
7

---

Published in Transactions on Machine Learning Research (05/2025)
change would not show specificity (e.g. unrelated tokens could be even more boosted). For this reason, we report P@k and CCP@k. Effects here ensure that the boost to the correct answer overcomes any boost for other answers. We also include a control experiment in Appendix B to better investigate specificity.
Our goal is to verify whether the behavior of the identified KNs aligns with our proposed typology. Specifi- cally, we hypothesize that (i) there will be a marked increase (or decrease) in precision at rank k=1 when the activations of Concept Neurons are doubled (or mullified), with the effect diminishing as k increases. Sim- ilarly, anticipate (ii) that the effect of Relation Neurons on P@k will be weaker than that of Concept Neurons, precision is primarily sensitive to the correct response. In contrast, for the CCP@k metric, we expect (iii) that Relation Neurons will play more significant role, as these neurons should be more likely to favor the correct category (e.g., capitals), even if it does not boost the correct answer specifically. We assess these effects for a range of thresholds te and t,. Results for the cort-base-uncant and Llama-2-7b models are shown in Figure (see Appendix B for additional models and thresholds as well).
The figures show the delta in P@k and CCP@k for the predictions with altered (doubling or nulifying) vs unaltered activations. The horizontal line at zero thus represents the baseline model performance. Of the six models evaluated, all six display the expected effect (i) consistently across all thresholds: in short, the top response is more accurate when the activations of concept neurons are increased. However, only two models, Llama-2 and the Gemma-2, exhibit effect (ii). Additionally, four models, belonging to the BERT and OPT families, align with expectation (iii). Overall, bert-large-uncasee three expected behaviors across all cases. This happens under restrictive thresholds however (4, = 0.9 and te 0.1), and the four other tested models fail to match all of these expectations.
and gemma-2-90 adhere to all
These mixed results show that classifying KNs into distinct and discntangled roles is not perfect, potentially due to noise in our methods in knowledge attribution methods in the first place. Yel, our experiments do indicate that, for certain models, KNs exhibit specific behaviors and manipulating them leads to predictable effects.
5.4 Discussion
As anticipated, these experiments underscore the complexity of the internal mechanisms within LLMs, making impractical to map a single, well-defined function to individual neurons. Many of the identified KNs do not adhere to a clearly defined role and cannot be neatly categorized as encoding either concepts relations, even within highly controlled environment like ParaRel. This is consistent across all the models studied. We believe that the polysemantic nature of neurons prevents such precise delineation, which also helps explain the knowledge editing limitations highlighted in prior rescarch. However, contrary to our initial expectations, certain KNs do appear to serve rather specific functions, and this has been experimentally confirmed for some models in boosting experiments. Nonetheless, we observed significant variation in behavior for the different models, which tends to demonstrate that the observed effects are sometimes fragile. Hence, while the idea that knowledge would be represented entirely in mono-semantie single neurons unrealistic, the historically associated methods of, e.g., Knowledge Neurons heneth detect transparent signal about how knowledge is encoded. KNs are thus useful tool to pursue the study of knowledge representation in multilingual models too, which we do in the next section.
6 Multilingual Experiments
When we learn a new language, we do not learn all facts about the world again, just new ways to express them. That is, there is a central knowledge base, that we can prompt with several languages. In this section we inquire knowledge is shared across languages in multilingual models too and, if so, what knowledge.
6.1 Experimental Settings
Models For this experiment we studiod Llama-2-7b. We used a NVIDIA Tesla V100 GPU for BERT and NVIDIA Tesla A100 GPU for Llama 2, both for about one hour per relation and per language.
(Devlin et al., 2019) and
8

---

Published in Transactions on Machine Learning Research (05/2025)
2-7-h-
berrbbese-multtinnul--nceeedddd
1403
( fr e5 25
L922
Shared Languages
-2003
1616
2000
1.0
bett wnw.---tiiiggullliiio
Uleme 2-/bhl
22:30
fitted nn
Languages
232
231
2422
1000
735
740
910
7150
3500
0 C9
Normalzed KNG
de di
261
297
270
255
155
1035
Land
392
945
171
200
140
102
714
409
535
381
1300
mi
x
222
239 234
320
323
243
203
224
1000
-200
593
666
619
617
576
315
422
943
1533
02-
225
207
383
277
180
1322
354
540
00-
en
65
Co
da
de
pt SV
en
ca
do
de
pt
SV
Languages
Languages
Number Languages
(a)
(b)
(c)
Figure 4: (a) Number of KNs shared by language pairs for Llama-2-7b. About quarter of neurons
are shared between two languages. (b) Same for www.aa-mmmmlllllllmmmmaaaaaaaa (c) Proportion of
shared KNs in relation a function of the number of languages in the intersection for Llama-2-7b and bert-base-min saaeeemulltt ti welllillngeen wwlliinnuluunnasse
Multi-ParaRel We built and release a new dataset Multi-Parateh a multilingual version of ParaRel. More details are given in Appendix A. Multi-ParaRel currently includes 10 languages: English, French, Spanish, Catalan, Danish, German, Italian, Dutch, Portuguese and Swedish. We also offer a translation and curation pipeline which makes it possible to add more paraphrases and more languages. It has average of 17 prompts per relation and per language but this value varies (from 9 for German to 19 for English). Each prompl is compatible with autoregressive models. After filtering for quality as above, we oblain on average 10 prompts per relation and language.
6.2 Knowledge Neurons are Shared Across Languages
Are KNs Bilingual? KNs were calculated separately for each relation and language. A KN is considered shared between two languages if it appears KN in both languages for the same relation. We conducted this pairwise analysis across all languages, thereby extending the findings of Chen et al. (2024) to encompass 10 languages.
The results are presented in Figures 4a and 4b. For the Llama-2-7b model, over a quarter of the neu- rons are shared between any two languages, with this proportion increasing to approximately one-third for This represents a significant degree of neuron sharing, especially when considering that bert-base-uncased, for example, has more than 12 x 3, 072 = 36, 864 neurons in the inter- mediate layers of its FFNs. To quantify this, note that among these 36, 864 neurons, only 1, 929 are identified as KN8 across all relations for English, and 2, 195 for French (roughly 5%). If KNs were randomly selected for each language, we would expect around 100 shared neurons between them (5% overlap); however, in reality, 710 neurons are shared. A similar analysis for Llama-2-7b gives even more extreme results: by chance, there should be 2 shared neurons, while in practice 189 are found. Moreover, these numbers represent a lower bound, as some relations were excluded from the prompt filtering process for certain language pairs, effectively reducing the shared KN count for those relations to zero. Thus, the data indicates significant overlap of KNs across languages, suggesting partially shared mechanism for knowledge retrieval across different language pairs.
Are KNs Multilingual? Next, we examine how the number of shared KNs scales with the number of languages in the intersection. Figure 4e shows these results for all relations, along with the average behavior. Across all relations, we observe a consistent pattern: the number of shared neurons decays as a function of the form (number of languages) -a, with a fitted a 2.04 for Llama-2-7b. In comparison, neurons were shared random, the expected behavior would follow x mumber of languages where p is the probability of neuron being KN (e.g. p- 0.05 for BERT). This demonstrates that KNs are more multilingual than
9

---

Published in Transactions on Machine Learning Research (05/2025)
Llama-2 2-na-f
bert-bae--muttiinnuul-uneee
1.0
KNs
KNS Concept
Average Overiap Coeficient
Concept
Average Overiap Confricient Accross Languages Pairs
0.8
Relation
0.8
Relation
0.4
0.4
0.0
0.0
0.7
Co 1.1-t.
t.1-t
Figure 5: Influence of typology on the average overlap coefficient calculated per language pair of Llama-2-70 (left) and bert-base sert-base-multtee matteemulllllie bassemmutilinguaale ingaltiiiinggll-uncseeee (right).
chance, reinforcing the notion a language-agnostic knowledge retrieval mechanism. Similar to the findings in Section 5, we observe some but few neurons activated for all languages.
Are some ncurons more Multilingual? Concept Neurons and Relation Neurons were computed separately for each language and each model. Figure 5 displays the average pairwise overlap coefficient for each neuron type, across various and te thresholds, alongside with the pairwise overlap coefficient for all KNs. The results reveal a significant difference in overlap between Concept Neurons and Relation Neurons at all threshold levels. However, the direction of this difference varies depending on the model and on the threshold. At the most demanding thresholds (those to the right selecting the purest types), we observe that relational neurons appear to be more bilingual. Given the variability at other thresholds (in particular for Llama, which is less performant than BERT in this task), we remain cautious about this conclusion.
6.3 Multilingual Boosting Experiments
While knowledge neurons may be shared across languages, this does not guarantee that they serve the same role in the two languages. A neuron active in both English and French for a given task may perform different overall tasks depending on the language, that is, parallel activation does not equate to shared functionality.
In this section, conduct boosting experiment experiment-similar to the one in Section 3 to assess whether neurons shared across languages have similar causal effect on the model's output. Specifically, for given language pair, we identify the shared neurons and then either nullify or double their activations when the model is prompted each language. The goal is to determine whether these neurons are more sensitive to one language more than the other.
Figure 6 presents the results for English and French. As expected, doubling the activation leads to higher Pâk and CCP@k scores, while nullifying it results in decrease. Importantly, the magnitude of these effects is similar across both languages. When activations are mullified, neither language shows clear advantage. Doubling the activations gives English a slight edge, particularly for lower values of k, but the difference is minor relative to the overall effect. These findings suggest that the shared neurons exert. a comparable causal influence in both langnages, indicating a similar functional role.
6.4 Knowledge Neurons are Shared Between Natural and Unnatural Languages
We have extended the analysis to non-natural languages, in order to deepen the work of Kervadec et al. (2023) in the specific framework of KNs. More specifically, we calculated 10 seeds of Autoprompl (Shin et al., 2020) for each model and each relation of ParaRel and the associated KNs. We then calculated the overlap cocfficient between the KNs calculated in this way and those calculated for English at the relationship level. The results are presented in Table 1. This reveals a very large overlap for all models, going up to an almost. complete overlap (2 80%) for models other than BERT. Tn the same way that there were important
10

---

Published in Transactions on Machine Learning Research (05/2025)
P@k en -fr
CCP@k en fr
0.10
0.05
0.05
CCP@k
P@K
0.00
0.00
-0.05
-0.05
-0.10
100
10
102
100
10
102
k
Figure 6: Multilingual Boosting experiments results for bert-bbas-multilinggg -aniaal-ucaseee for English and French. The lines corresponds to the AP@k (resp. ACCP@k) for different h values ranging from to 100. Thick lines represents the doubled activations results and thin lines the nullified activations results. We also plotted the standrad error accross the evaluated instantiations of the relations. Here only shared KNs activations are modified.
overlaps across natural languages, this new result suggests a similar mechanism of knowledge retrieval even between natural and non-natural languages. It is possible however that there exists a confound here because both Autoprompt and KNs are gradient based.
Model
bert-base
bert-large
opt-350m
opt-6.7b 7b
Llama-2-7b
Avg. Overlap Coeff.
40%
32%
83%
87%
79%
Table 1: Average overlap coefficient of KNs sets computed at the relation level between English and Auto- prompt.
7 Discussion and Limitations
Relying KNs represents a limitation of our work. Their identification relies on attribution method based on gradient computations, as opposed to alternative approaches such as causal tracing. However, we do not identify any obvious bias introduced by such a choice and we argue that our primary contribution lies the methodology itself. Importantly, the core our analysis agnostic to the specific technique used identify neurons, and could be applied to other attribution methods just as well. We plan such analysis in future works.
Another possible limitation would be the use of threshold to identify Concept Neurons and Relation Neurons. We argue that this is not the case. First, thresholds are applied after the identification of KNs, not as part of their initial computation. Second, since the functional roles of individual neurons are unknown a priori, we systematically explored a range thresholds (see Figure 2a, 2b, 5, 12, 13 & 14). This exhaustive approach is an integral part of the method, allowing us to analyze KN behavior across different sensitivity levels. Importantly, we observed no significant variation in results across thresholds, which retroactively supports the validity of the method.
Another limitation of our methodology is that the identification of neurons shared across languages for the same relations and concepts does not guarantec that these neurons serve the same functional rolc. We explore this issue in Section 6.3, where we present evidence suggesting that some shared neurons may indeed fulfill similar functions. However, further investigation is required to systematically examine different intersections, whether at the level of relations, concepts, responses, or prompt formats, order to more preciscly determine these roles. Our approach provides a useful framework for refining such analyses and uncovering functionally shared neurons across languages.
11

---

Published in Transactions on Machine Learning Research (05/2025)
Finally, the neurons identified in this study were discovered within highly controlled setting, specifically using the TREx dataset. As such, the identification of KNs, as well as the proposed categories of Concept Neurons and Relation Neurons, may not directly translate to real-world scenarios in their current form. We plan to explore ways of adapting this methodology to more naturalistic contexts in future work.
8 Conclusion
We introduced typology for knowledge and applied it to the knowledge attribution method proposed by Dai et al. (2022) to better classify and understand the behavior of Knowledge Neurons (KNs). Notably, our method remains agnostic to the specific knowledge attribution technique used. Coherently with the initial assumptions in the original work, we found that some of these neurons encode specific concepts, but we also found many which do not and instead seem to exhibit distributed role, where multiple neurons share responsibility for encoding concepts within the same relation, or maybe encode the whole relation. We hypothesize that this polysemantic nature of neurons contributes to the mixed success observed when using KNs for knowledge editing tasks. Yet again, we were able to identify subset of more specialized neurons, which we categorized as either conceptual (sensitive to a single concept. or relational (sensitive to relationships between concepts). And in some contexts their manual manipulations show the expected effects on downsttream lasks. We extended our analysis to multilingual models and found that a siguificant number of KNs are shared across languages- bbtth in pairwise comparisons and across all 10 languages tested. This indicates the presence of shared, language-agnostic knowledge base within multilingual models. To facilitate this research, created a multilingual dataset of facts and prompts, enriched with paraphrases in 10 languages. Our findings suggest that even simple method like Knowledge Neurons can provide valuable insights into the benefits of multilingual training.
Looking ahead, we aim to further explore how this shared knowledge can be leveraged to improve the integration of new languages into existing multilingual models. Our results indicate that it may not be necessary to relearn factual knowledge for each language, which could pave the way for more efficient training strategics, particularly for low-rcsource languages. Instead of focusing on exhaustive coverage of world knowledge, future efforts could prioritize data that highlights the unique synlactic and linguistic features of these languages, thus optimizing resource use and improving model performance. Another possible research direction involves examining how the mechanisms of factual knowledge develop throughout training, which could shed light on the most advantageous stages to introduce or integrate external knowledge.
Acknowledgments
We wish to thank Paul Egré, Olivier Pietquin, Benjamin Spector, Tessa Verhoef, Bernhard Weiss, Yair Lakretz, Nur Lan, Shane Steinert- Threlkeld, Dieuwke Hupkes, Corentin Kervadec, Marco Baroni and mem- bers of the EVIL group at UPF, and of the LINGUAE and LINGUAE ML groups at ENS. We also gratefully 10-11--IDE-000----0 FrontCog (ANR-17-EURF-0017) ComCogMean
acknowledge the support from: (Projet-ANR-223 ANR--3-CE2---000 CE28-0016), and gift from Meta. This work was performed using HPC resources from GENCI IDRIS (Grant AD011013783).
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, and et al. Gpt-4 tochnical report. (arXiv:2003005774) March 2024. doi: 08774. arXiv:2303.0877 [cs).
URL http://arxiv wwt.riiii.or//bbaaee ar303/////33.
Templeton Adly, Conerly Tom, Marcus Jonathan, Lindscy Jack, Bricken Trenton, Chen Brian, Pearce Adam, Citro Craig, Ameisen Emmanuel, Jones Audy, Cunningham Hoagy, L Turner Nicholas, McDougall Callum, MacDiarmid Monte, Tamkin Alex, Durmus Esin, Hume Tristan, Mosconi Francesco, Freeman C. Daniel, R. Sumers Theodore, Roes Edward, Batson Joshua, Jermyn Adam, Carter Shan, Olah Chris, and Henighan Tom. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. May 2024. URL https://ttttttsmmmecccccttsassst putp//22//eealnggooooosmttttttt/
https: https:://transformeereoo circurtrrrrrutts..m
12

---

Published in Transactions on Machine Learning Research (05/2025)
Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.); Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume (Long and Short Papers), pp. -874-3884, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v / 1 1388.
URL ttttps://aclanthologs.. wtw.a/aannoooooooo/////-e/---- org/N19-1388.
Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondřej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonet, Augela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Lconie Harter, Kenneth Hcafield, Christopher Homan, Matthias Huck, Kwabena Amponsab--aakkyine, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Koemi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Na- gata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. Findings of the 2021 conference on machine translation (WMT21). In Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grund- kiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Tom Kocmi, Andre Martins, Makoto Morishita, and Christof Monz (eds.), Proceedings of the Sixth Conference on Machine Translation, pp. 1-88, Online, November 2021. Association for Computational Linguistics. URL https: attps://aallanthologyy org/2021. http//222.wttt.111.111. wmt-1.1.
Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. A review on lan- guage models as knowledge bases. (arXiv:2204000031) April 2022. doi: 10.48550 10.48550/arriv. ai..ii..200.0666666. URL http://arxiv, http:://axxiv.or//abb// 220// 2204. 06031. 99 citations (Semantic Scholar/arXiv) (2024-04-29) 99 citations (Semantic Scholar (OOT [2024-04-29] arXiv:2204.06031 [cs].
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Machercy, Zhifeng Chen, and Yonghui Wu. Mas- sively multilingual neural machine translation the wild: Findings and challenges. (arXiv:1007.05019) URL http://arxiv. http:///aaxxv.../aaa/////es/ wtt//org/////o///oo 1901/9999799......
July 2019. doi: 200
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Unsupervised statistical of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3632-3642 Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/DI4 D URL https://accaanthology hat.aaatttoooooooo/reeee------ org/D18-1399.
stattissica machine translation. In Proceedings
Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Axelrod, Jason Riesa, Yuan Cao, Mia Xu Chen, Klans Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apurva Shah, Yanping Huang, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. Building machine translation systems for the next thousand languages, 2022. URL https: //ts://///xvvv wwwpxiiv.rg//bb.... orggabs//205 0338//222.00033...
David Bau, Steven Liu, Tongzhou Wang, Jun- Yan Zhu, and Antonio Torralba. Rewriting deep generative model. (arXiv:2007.16646) July 2020a. doi: 10.48550/ar /aro//aXXiv.000 200...00006.666 URL http: /ttp:///rxiv... wwwaaaiwww...oaaaaaaaaaaaaoeeree abs/ 2007.15644 arXiv:2007. [cs].
David Bau, Jun- Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Under- standing the role of individual units in a deep neural network. Proceedings of the National Academy of Sci- ences, 117(48):30071-3000
December 2020b. ISSN 0027-8424, 1091-6490. doi: 1.1077/ppas.19073335
arXiv:2009.055. 005040009.0001 [cs].
Rachel Bawden and François Yvon. Investigating the translation performance of a large multilingual language model: the case of BLOOM. In Mary Nurminen, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vannassenhove, Sergi Alvarez Vidal, Nora Aranberri, Mara Nunziatini, Carla Parra Escartin, Mikel Forcada, Maja Popovic, Carolina Scarton, and Helena Moniz (eds.), Proceedings of the 24th Annual Conference of the European Association for Machine Thanslation, pp. 157-170, Tampere, Finland, June 2023. European Association for Machine Translation. URL https::/aclanthhiit hawpollooyyrrg/222233 eamt-1.16 16.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Nya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https: /ttpps//arxiv... http:///wxxvvv..r/bbsbaasse
abs htrgg/abs22005.. 2005 14165.
Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao Journey to the center of the knowl- edge neurons: Discoveries of indepeee-ndeeenddent knowledge neurons and degenerate knowledge neurons. Proceedings of the AAAI Conference on Artificial Intelligence, 38(1616) March 2024. ISSN 2374-3468. doi: 10.1609 htt.a/aaaai............. 088116.29735.
Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Knowledge localization: Mission not accomplished? enter query localization,, 2025. URL https://aaran http///xxvvvvorroe wttpiiv..rrgab///.55./ 14117.
Rochelle Choenni, Dan Garrette, and Ekaterina Shutova. How do languages influence each other? studying cross-lingual data sharing during LM fine-tuning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13244 13257, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2022 emnlp-main.818. URL https: / ///aa/aaaaaatooyyy..oooooooo wtw.aattooooooooooo/22.22/2. emnle-manatteeeeee memin..18/...
Lynn Chua, Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chulin Xie, and Chiyuan Zhang. Crosslingual capabilities and knowledge barriers multilingual large language models, 2025. URL https:///rrav..or//abba. wwwporgg///aaaaaeeeeeeee 16135.
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowl- edge editing in language models. (arXiv:2307.12976) December 2023. doi: 1..48550/arii.2207..222 URL http://ttaree http://aaxvvvv.. wtw.aaaiiiv.aaaaaaeeeeee
12976. arXiv:2307.1200 [cs].
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Out, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual represontation learning at scale. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440 8451, Online, July 2020a. Association for Computational Linguistics. doi: 10.18653/vbl 20///33/11/22/// w1//220..cca/ aa12200.llmmai.74... URL https://aclantholl//
org orgg2020.... acl
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. Emerging cross-lingual structure in pretrained language models. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 6022 6034, Online, July 2020b. Association for Computational Linguistics. doi: 10. 1L18633//112220..am URL https: fttps://aclaanthoogge 2020 acl- main..an... 536.
536.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakow, and Aline Villavicencio (cds.), Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 8493 8502, Dublin, Treland, May 2022. Association for Computational Linguisties. doi: 10.18653/v1/ a022.acl-long 581. URL https: http::://acaaatthhll0ooo
2rg/2002.ccl11omm 581.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6491 6506, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistios. doi: 10. 10.18633/v1/201 emnlp-main.522. URL https: /aalaatthoc/ wwwp:/gy..r.........e.. http//222.emmlppmmaae... main.-mmii.5mm 522.
Nicola De Cao, Leon Schmid, Dieuwke Hupkes, and Ivan Titov. Sparse interventions in language models with differentiable masking. In Jasmijn Bastings, Yonatan Belinkov, Yanai Elazar, Dieuwke Hupkes, Naomi Saphra, and Sarah Wiegreffe (eds.), Proceedings of the Fifth BlackbozNLP Workshop on Analyzing and Interpreling Newral Networks for NLP, pp. 16 27, Abu Dhabi, United Arab Emirates (Hybrid),
14

---

Published in Transactions on Machine Learning Research (05/2025)
December 2022. Association for Computational Linguistics. doi: https://acllntthologg www.ogg...../ooo.. org/2022. blackboxnlp-1.. 1...
URL
Ameet Deshpande, Partha Talukdar, and Karthik Narasimhan. When is BERT multilingual? isolating crucial ingredients for cross-lingual transfer. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza. Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3623, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 10.18653/v v1.18653/1122222..2 mai.l URL https: //aclanthology org/2022. carel2nnacllaae maa.eelmmai.2... 264.
Jacob Devlin, Ming-
Ming-WWii&ne Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pro-training of deep bidirec- tional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171 4186, Min- ncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/S3 1423. URL sttps://aclannthology.oooo org/N19-1422
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:1012 1031, 2021a. doi: 10.1162/tacl 00410. URL https://aclannhholog/
tacl-1.60.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and
Yoav Goldberg. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:1012 1031, 2021b. doi: 10.1162/tacl a 00410. 224 citations (Semantic Aaholar/DO1] [2024-04-99.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. T-REx: A large scale alignment of natural language with knowledge base triples. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), Proceedings of the Fleventh International Conference on Lan- guage Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL https: sttps://aclanthologg org/L18-1544
Xavier Garcia, Aditya Siddhant, Orhan Firat, and Ankur Parikh. Harnessing multilinguality in unsuper- vised machine translation for rare languages. In Kristina Toutanova, Anna Rumshisky, Luke Zettle- moyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.). Proceedings of the 2021 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pp. 1126-1137, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/ 20tp///l//221111/ www.llaaaclaaaaaa main.89. URL https: // www.aatthhoooooooo///200 wwgg///1..aacclmmaa........ haacl-main.8
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key- value memorics. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5400 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v2 2w200221.mmmmmmmme 446. URL https: //aaa///acclaaoogyyy org/ emnlp-main mappmm4in.45.
Nuno M. Guerreiro, Duarte M. Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. Hallucinations in large multilingual translation models. Transactions of the Association for Computational Linguisties, //a /aallantholoyy.....o/ 0rg/2023.taal-1111 tacl-1.855 1.85/.
2023. doi: 10.1162/tacl 00615. URL https:
Peter Hasc, Mohit Bansal, Been Kim, and Asma Ghandcharioun. Does localization inform editing? surprising differences in causality-base based localization vs. knowledge editing in language models. (arXiv:2301.042) October 2023. doi: 10.48550
URL http:////// http://arxiv, waw.ii..ogg/bbb. httg//abs//01.0421.3. 46 citations
15

---

Published in Transactions on Machine Learning Research (05/2025)
(Semantic Scholar/arXiv) [2024-02-19] 46 citations (Semantic Scholar/DO1) [2024-02-19] arXiv:2301042113 [s).
Linyang He, Peili Chen, Ercong Nie, Yuanning Li, and Jonathan R. Brennan. Decoding probing: Revealing
internal linguistic structures neural language models using minimal pairs. (4rXiv:2403.1729) March 2024. doi: 00.48500/arXiv.240...777 URL http:///rxiv.orr/aaeee orggabs//403 17299. arXiv:2408.17999 [cs, q-bio].
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. How good are gpt models at machine translation? a comprehensive ovaluation, 2023. URL /ttps://arxiv..
09210.
Jing Huang, Atticus Geiger, Karel D'Oosterlinck, Zhengxuan Wu, and Christopher Potts. Rigorously assess- ing natural language explanations of neurons. (arXiv:2309.106) 10312), September 2023. doi: 10.48550/ar /avvv 2309.10312. URL http: arxiv. org/ /tg/abs///// /309 10312. 5 citations (Semantic Scholar/arXiv) [2024-03-2] [cs].
5 citations (Semantic Scholar/DOn [2024-03-24] 24] #rXiv:2309.10311
Kuan-Hao Huang, Wasi Ahmad, Nanyun Peng, and Kai-Wei Chang. Improving zcro-shot cross-lingual transfer learning via robust training. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott. Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirioal Methods in Natural Language Processing, pp. 1684 1697, Online and Punta Cana, Dominican Republic, November 2021. Association for www.lp-mainnnn...meeeeemmmmmmmnm 126. URL https://aclanthologg. org/
Computational Linguistics. doi: 10.18653/v/ 2021. 1021.emmlp-1mmmmmmmmmmmm1..m1111 mamlp-main.... 126.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Trunsactions of the Association for Computational Linguistics, 9:962-977, 2021. doi: 10.1162/tacl a 00407. URL https//acclathhology org/2021. ta0001..ac1-1..7.
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. Cross-lingual ability of multilingual bert: An empirical study, (arXiv:1912.078/)) February 2020. doi:
URL http:
//w.xiv.org/ab///aa. ptggabs//912..o.. 198400781100.......11 arXiv:1912.078400 [es].
Nora Kassner, Philipp Dufter, and Hinrich Schütze. Multilingual LAMA: Investigating knowledge in mul- tilingual pretrained language models. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Linguis- tics: Main Volume, pp. 3250 3258, Online, April 2021. Association for Computational Linguisties. doi: org/ 202/
111.6633/112221..cmm
284. URL
284.
Corentin Kervadec, Francesca Franzon, and Marco Baroni. Unnatural language processing: How do language models handle machine- generattd prompts? pp. 14377-14392, Singapore, December 2023. doi: 10.18653/ wernsiinnnnneenllppp
URL https://aclantholl.o httpp:///ecaaaaooooo.oo///.//../ wwwdinggsennnlp......
Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo. On the multi- lingual ability of decoder- based pre-trained language models: Finding and controlling language-specitt neurons. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers); pp. 6919 6971, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2204 laaaccllong.344. URL https://aclantholl .rg/ 2024. www.naacllllll long. 384.
Sncha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. Investigating multilingual NMT rep- resentations al scale. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing (EMNLP--ICCN) China, November 2019. Association for Computational Linguistics. doi: 10.18653/<1/D19-1166 URL www.elaanntoooooooo/oaoeo.
pp. 1565-1575, Hong Kong,
16

---

Published in Transactions on Machine Learning Research (05/2025)
Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. The emergence of number and synlax units in LSTM language models. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp. 11 11-20, Minneapolis, Minnesola, June 2019. Association for Computational Linguistics. doi: 10.18653/vM vv 101/9N1911000 1002. URL htttps//aclanthoete htt.://wttooooooooo//--/----0000 org/N19-1002.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. MerXiv1190100299 Jan- uary 2019, doi: 10.48550/ www..i...11..0...... 1901.07291. URL http://arxiv wttp://wwww.rraa//............ 07291. arXiv:1901.072: [cs].
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Mare' Aureloo Ranzato. Unsupervised machine translation using monolingual corpora only. February 2018. URL https: ctttss//openrrvviee wtt.:/ouww..ott/too/dd.d
TkYTTf-AZ.
Danni Liu and Jan Nichues. Learning an artificial language for knowledge-sharing in multilingual translation. In Philipp Koehn, Loie Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa- jusså, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, Andró Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélic Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 188 202, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL https://aclanthology org/2022. org/2022.wmt-1. 1.1.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics,
November 2020. ISSN 2307-387X. doi: 10.1162/
tacl a 00343.
Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, :17(44):30046 30054, December 2020. doi: 10.1073/ p.a0.////..///.. http3//aas.1197360 1907367117..
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. (.XXi:2202.05262)) January 2022. URL http:///axxxv..o/aa///////ss 05262. arXiv:2202.05262
[cs).
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. (arXiv:2210.07222), August 2023. URL http: /tap:v///xiii..//aa htt/abb/22110.070 171
citations (Semantic Scholar/arXiv) [2024-03-01] arXiv:2210.07229 [cs].
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model
editing at scale. 1 June 2021. URL httt:://arxv..or//bb////t 11309. 188 citations (Semantic Scholar/arXijg [2024-04-29] arXiv:2110.1130 [es].
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. Memory-based
model editing at scale. (arXiv:2206.0655 066220),, June 2022. doi: 10.48550/ 2a.3500//aXiv22/// www.2206.06525 URL http: //arxiv. wwwxiv.org//eeo org/ hwgg/bb//226.... 065200. 125 citations (Semantic Scholar/arXiv) [2024-03-04] arXiv:2206.06520 [cs].
Jingcheng Niu, Andrew Liu, Zining Zhu, and Gerald Penn. What does the knowledge neuron thesis have to do with knowledge? October 2023. URL httpp:///penreveeee.ee///
Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. Towards making the most of ChatGPT for machine translation. In Houda Bouamor, Juan Pino, and Kalika Bali (cds.), Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 5622-5633. Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/ 2023. 2023.findins--eit fin...nng--eemp.333...... URL https://aclanthoology www.e/o..gg....../. findings-emmlp.377
17

---

Published in Transactions on Machine Learning Research (05/2025)
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexan- der Miller. Language models as knowledge bases? In Kentaro Tnui, Jing Jiang, Vincent Ng, and Xi- aojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463 2473, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10.18653/v1 / 1250. URL https://aclanthol..g org/D19-1250.
Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. How context affects language models' factual predictions. GarXiv:205.04611 May 2020. doi: /0..5/////....000...... 2005.0..00 URL http://araat http//aavv..o.aaaaae terg/bbs/200/ 04611. 166 citations (Semantic Scholar (aholar/arXiv)) [2024-04-29] arXiv:2005.045: 0.6611 [cs].
Telmo Pircs, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In Anna Korho- nen, David Traun, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeling of the Association for Computational Lânguistics, pp. 4996-5001, Florence, Italy, July 2019. Association for Computational Linguistics. doi:
URL https://aclanthhllo. hatp:///ooooo..orr/-/-19999 org/P19-1493.
Maxime Poli, Thomas Schatz, Emmanuel Dupoux, and Marvin Lavechin. Modeling the initial state of early phonetic learning in infants. Language Development Research, 5(1):1-34, 2024. Published: 2024/04/23
Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to generate reviews and discovering sentiment. URL http://arxil: cetexxvv.orgggg org/ aaa//b///// /174.
(44X01:1004..1444),, April 2017. doi: 01444. 17X.v:1704.01444 [cs]
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Πlya Sutskever. Language models are unsupervised multitask learners. 2019. URL https: : //api./ppp......p www.ntiimmniiiiiemeeee.. cwsheanntccccclarrr...oomaaaaaae org/CorppsII: 160025533.
Sara Rajaee and Christof Monz. Analyzing the evaluation of aross-lingual knowledge transfer in multilingual language models. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Lènguistics (Volume 1: Long Papers), pp. 2895-2914 St. Julian's, Malta, March 2024. Association for Computational Linguistics. URL https: //aclanthology- org/2024.eac111ooog eacl-long. 177/.
Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. The curious case of hallucinations in neural machine translation. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistios: Hu- man Language Technologies, pp. 1172 1183, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/vl] v122021 haacl-main. URL https: haap::///clanthooggyy org/2021. naacl-main 92.
Ryokan Ri and Yoshimasa Tsuruoka. Pretraining with artificial language: Studying transferable knowledge in language models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (ods.), Proceedings of the 60th Annual Meeling of the Associalion for Compulational Linguistics (Volume 1: Long Papers); pp. 7302-7315, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1, 2022.acl-long.5 504.
URL https: //t.a/ataaaannn......//oo org/2022. acl- http:/2222.ccl1onn.... 504.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418-5426, Online, November 2020. Association for Computational Tinguistics. doi: 10. 10.18653/v1 N0.18663/v112000.mmmpppp /2000.emmmpmmmin..4. URL https://aclanttooogyy 2020 main..-main... 437.
David Robinson, Marius Miron, Masato Hagiwara, and Olivier Pietquin. Naturelm-audio: an audio-language foundation model for bioacoustics. In Proceedings of the 13th International Conference on Learning Rep- resentations (ICLR), 2025. URL /ttps://openreet
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works. Trunsactions of the Association for Computational Linguistics, 8:842-866, 866, 2020. doi: 10. 1162/tacl 00349. URL https: httpss//aclanntholos www.ellooyoorgg///// org//020002.......ooo.22 t.c1-1.54.
18

---

Published in Transactions on Machine Learning Research (05/2025)
Tara Safavi and Danai Koutra. Relational World Knowledge Representation in Contextual Language Longuauu Models: A Review. In Marie-Francing Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.); Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1053 1067, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistios. doi: 10. 10.18635/v1 /2221 1001.e.elp-main.811 URL https://aallamtholooyy wwwpmn/hwwoooo.oo//2222o22 wttp:/orr///..eelllmmmeee. emnlp-main 81.
Shibani Santurkar, Dimitris Tsipras, Mahalaxmi Elango, David Bau, Antonio Torralba, and Aleksander Madry, Editing a classifier by rewriting its prediction rules. (arXiv:2112.01009), December 2021. doi: 11.45500/aXivv.v 21: 12.000008. URL pestp://arxee wwaezii..org/ae. wttgg/bb//112... 01008. arXiv:2112.0100 [cs].
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, and et al. Bloom: 176b-parameter open- access multilingual language model. SarXiv:221..55100 June 2023. doi: 10.48550/arXiv 20110/0.0 URL 05100. arXiv:2211.05100 [es].
betp://arxiln org/
Rico Scnnrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeling of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 86 96, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 1..1865//11/116- 1009. URL https:// aclanthology.. Milaattoooogy.or/P11---00000 org/P16-1009.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Samcer Singh. AutoPrompt: Elic- iting Knowledge from Language Models with Automatically Generated Prompts. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Täu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4222 4235, Online, November 2020. Association for Com- putational Linguistics. doi: 1tt/3///11220.emm/// 34...346. URL https://aclanthology org/2020.
emnlp-main.34
Jasdeep Singh, Bryan McCann, Richard Socher, and Caiming Xiong. BERT not an interlingua and the bias of tokenization. In Colin Cherry, Greg Durrett, George Foster, Reza Haffari, Shahram Khadivi, Nanyun Peng, Xiang Ren, and Swabha Swayamdipta (eds.), Proccedings of the 2nd Work- shop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pp. 47 55, Hong Kong, URL
China, November 2019. Association for Computational Linguistics. doi: https: httpst//acllatthoooggyy. wrr.oog.y../////105550
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko, Editable neural networks. (arXiv:2004.00345)) July 2020. doi: 10.48550/ar http//aa............00 2004.00345. URL http://arxivv org/ abs/ 00345. 110 citations (Semantic Scholar/ Scholar/arXivy [2024-04-29] arXiv:2004.00345 [cs, stat].
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. (1XXiv:1703.01365) June 2017. doi: 055500/aXivv178..013333 URL http://arxiv. http:://axxvvoor///a// org/ ab731 01365. 1=XXiv1100301366 [cs).
Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolci Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. Language-specific neurons: The key to multilingual capabilities in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (ods.), Proceedings of the 62nd Annual Meeting of the Association for Compulational Lènguistics (Volume 1: Long Papers), pp. 5701 5715, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18633/v1/20 220///112222..a/ 22t./22222.clloo...20. URL https: / htt///acaannttooggeo org/2024 acl acl-long.300 309.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanezyk, Sertan Girgin, Nikola Momchev, Matt Hoff- man, Shantamu Thakoor, Bean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Ali- aksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Bran- don Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo,
19

---

Published in Transactions on Machine Learning Research (05/2025)
Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińskka, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glemn Cameron, Gus Martins, Hadi Hashemi, Hanna Slimczak-Plucinisse Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mcin, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Nijaii, Jocelyn Becker, Joe Fernandez, Joost. van Amers- foort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju-yeong Ji, Kareem Mohamed, Kartikeya. Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Wat- son, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Henmigan, Tomas Kocisky, Tulsce Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahra- mani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukouoglu, Clement Farabet, Flena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open lan- guage models at practical size. (arXiv:2408. (ttp24408.00118) http://arxiv org/ani aaaaaaa//aa /400 00118. arXiv:2408.00 00118 [cs].
August 2024. doi: 10.48550/ar /ar50v DAc.ii..248.00 2408.00118. URL
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosscini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Tu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Sub- ramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Tiyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Au- relien Rodriguez, Robert Stojuic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. farXiv:2307.09258: July 2023. doi: 10.48550/ar (0555//aXii.22 http://a............. 2307.09202 URL http://arxiv.. wrgg abaaabb///0 09288. wxXiv:2307.09256 [es).
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation anal- ysis. In Advances in Neural Information Processing Systems, volume 33, pp. 12388-12400 Curran Associates, Inc., 2020. URL https: cttps://proceedingg nww.ips.rripccceeee.e wtt.r/ps.ccpppppe////00......... cctpapper/200//nasss Abstract. html.
David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. Prompting PaLM for translation: Assessing strategies and performance. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (cds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15406 15427, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10.18653/v1/2008 acl- sel--oong.e, long.859. URL htps://aclaathoologg .org/orgggooo... org/2023 acl-long. 859.
Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang, Linyi Yang, Jindong Wang, Xing Xie, Zheng Zhang,
20

---

Published in Transactions on Machine Learning Research (05/2025)
and Yue Zhang. Survey on factuality in large language models: Knowledge, retrieval and domain-speciffcttt, 2023. URL http:////aaaii..oo/aaa////s/ 07521.
Weixman Wang, Barry Haddow, Minghao Wu, Wei Peng, and Alexandra Birch. Sharing matters: Analysing neurons across languages and tasks in llms, 2024a. URL
org/abs/ waaxii.org//aiee abs /20b/2406... 09265.
Yuxia Wang, Minghan Wang, Muhammad Arsian Manzoor, Fei Liu, Georgi Nenkov Georgiev, Rocktim Jy- oti Das, and Preslav Nakov. Factuality of large language models: A survey. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 19519,19529, Miami, Florida, USA, November 2024b. Association for Compu- tational Linguistics. doi: (06653/v1/22444 connlp-main.iie main. URL https://aclanttoll.00 www.tthhooooooooo/22222222 org/2024. emnlp-main. 1088/.
Xiaokai Wei, Shen Wang, Dejiao Zhang, Parminder Bhatia, and Andrew Arnold. Knowledge enhanced pretrained language models: A compreshensive survey. (arXiv:2110008455), October 2021. URL http: //arxiv.org/abs///// 0r050455/2110008455 08455. 28 citations (Semantic Scholar/arXiv) [2024-04-29] arXiv:2110008555 [es]
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. m5:: A massively multilingual pre-trained text-to-text transformer. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1 /v/655/vv/22211111 h/t.0//.naacllmmaaaeeee naacl-main URL https://aclanthooogg) org/2021 naacl-main 41.
Xu Yucmei, Hu Ling, Zhao Jiayi, Qiu Zihan, XU Kexin, Ye Yuqi, and Gu Hanwen. A survey on multilin- gual large language models: Corpora, alignment, and bias, 2024. URL https://arxiv https://arxiv.org/abb//// or40..//44../.. 00929v3.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. (arXiv:2005.00068) June 2022. doi: 2ar/550/arXiv.22000 01068. URL http://aaxxvorr/aaa/222.... http://arxiv wwtp:/r/////...oooo/./ 01068. arXiv:2205.0100 [cs].
A New dataset: Multi-ParaRel
Creation Procedure To build the Multi dataset, we used our augmented autoregressive version of ParaRel and mLAMA. The goal is to translate template such as The capital of [X] is (Y). The problem is that translators are confused by the presence of placcholders [X] and /Y), often resulting in translation errors. To overcome the difficulty, we instantiated [X] and (Y), translated the whole sentence with these specific instances, and replaced the instantiations back with placeholders. To do so, we used mLAMA, which contains triplets for over 53 languages.
For example, consider the translation from English into French of the template:
The capital of [X] is (Y]
We use the English triplet <Geeat Britain, copital of, London to obtain the sentence:
The capital of Grent Britain is London
This sentence is then translated into French:
La copitale de la Grande Bretagne est. Londres
21

---

Published in Transactions on Machine Learning Research (05/2025)
mieraggvergge Triplets Count
train
dev
600
500
Aug. Tiipic Count
400
300
100
az be bg bn cacebes cy da
he hr hu by
Languages
Figure 7: In mLAMA, the number of triplets available varies widely across the different languages.
Then using the French version of the original triplet (<la Grande Brelagne, capital of, Londres>)) we can find and replace the entity elements of the triplet with placeholders [X] and [Y], resulting in the new template:
La capitale de [X] est [Y]
With this overall idea, we can now provide more detail. First of all, such protocol requires associated triplets in mLAMA from one language to another. However, mLAMA has many more triplets in English than in other languages (see Figure 7), and some triplets are language-specific and therefore cannot be associated with triplets in other languages. We therefore looked into common English- Targett language subset. Then, to avoid translation errors, problems linked to gendered determinants and redundancy (two different templates in English but translated identically in the target language), we used a voting system. Each template was translated 30 times, using 30 triplets. Each translation is assigned a score, which is the number of times the template has been obtained out of the 30 triplets. The template with the highest score is then retained, provided that (i) it is autoregrossive, (fi) it has not alrcady been selected and (fii) it is in the top 5 translations.
As a translation model, we used Meta's SeamlessM4T and, more specifically, the Huggingface implementa- 3 We used an NVIDIA Tesla V100 GPU for inference.
Statistics and Exemples Table provides examples of translated templates from different languages and relations. The average number of templates obtained per relationship for each language is:
Quality Analysis To judge the quality of our dataset, asked a native speaker of French and native speaker of Spanish to rate the resulting templates in three categories: fluent, weird, ungrammatical. For French 88% are correct, 7% weird and 5% are ungrammatical. For Spanish: 78% of sentences are fluent, 10% weird and 12% are ungrammatical. Although imperfect, Multi-Paraßel coupled with a less efficient filtering of prompts gives very good results on mLAMA.
B Full Results
First we provide an overview of all the models behavior with respect our expectations in Table 4. We also add a control experiment for the BERT family where we conducted the same boosting experiments but sampling KNs randomly within the relation for the Concept Neurons and across relations for Relation Neurons. The goal of such a control is to test the specificity of identified KNs. Results are in Table 5. We see that the eflects are destroyed when looking at randomly selected KNs.
22

---

Published in Transactions on Machine Learning Research (05/2025)
Language
Avg templates
Catalan
19
Danish
15
Dutch
17
English
19
French
14
German
Italian
19
Portuguese
19
Spanish
19
Swedish
16
Table 2: Language Values
Relation
English
Spanish
French
P36
The capital of [X] is [Y]
La capital de [X] es (Y]
La oupitale de [X] est [Y]
[X], which has the capital (Y]
[X], que tiene la capital [Y]
[X], dont la capitale est [V]
P106
The occupation of [X] is (Y)
Le ocupactón de [X] es [Y]
La profession de [X] est [Y]
[X] works as [Y]
[X] trabaja como [Y]
[X] travaille comme [V]
P1001
[] counts legal term [Y]
[X] euenta término legal [Y).
[X] légal [Y]
[X] is vold logal term in (Y]
[X] térméno legol váládo [Y].
[X] furidique valide [Y]
Table 3: Examples of templates from Multi-ParaRee
Model
Expectation (i)
Expectation (ii)
Expectation (iii)
bert-base-uncased
Yes
No
Yes
bert-large-uncased
Yes
Yes
Yes
opt-350=
Yes
No
Yes
opt-6. 7b
Yes
No
Yes
Llama-2-77 2-7b
Yes
Yes
No
gemma-2-88
Yes
Yes
No
Table 4: Overview of boosting results for all models. Expectations are: (i) there will be marked increase (or decrease) in precision at rank k=1 when the activations of Concept Neurons are doubled (or nullified), with the effect diminishing as k increases, (ii) the effect of Relation Neurons on P@k will be weaker than that of Concept Neurons, as precision is primarily sensitive to the correct response, (iii) Relation Neurons will play more significant role, as these neurons should be more likely favor the correct category (o.g., capitals), even if it does not boost the correct answer specifically.
Model
Expectation (i) Expectation (ii)
Expectation (iii)
bert bert-base-uncased bert-large-uncase
No
No
Yes but effect 10x smaller
No
No
No
Table 5: Overview of boosting results for the control experiment.
Second, we provide all graphs computed for all models and relations concerning the distinction between concept and relation neurons. This corresponds to the results as presented in Section 5.2, Figure 2, also
23

---

Published in Transactions on Machine Learning Research (05/2025)
showing all relations each time. Second, provide all graphs corresponding to the boosting experiments (Section 5.3, Figure 3).
24

---

Published in Transactions on Machine Learning Research (05/2025)
304
No Couk
KNN CURR
Sumbe
Di butarittees
Nurber intemtatt
10 Narte initurtatiee
Navaa Indaa matiiss
733
717
PLTR
F276
ON
Sambe inttrtttttoo
PdM
1817
S 2
Core
Sunter romorraiiio
Namtor of boteriattess
Number of Instrttt are
Varber tintettaoon
Hurmoor of Insteniations
136 200E
(a)
P1001
P131
P1376
P1.00
P1412
be PM pogee die.
D0g DW PINC the
Ne
P2T9
P776
$7364
1--2
P495
P937
wi We Wooe
1-6
(b)
Figure & bert-base-uncased
25

---

Published in Transactions on Machine Learning Research (05/2025)
P1001
P108
P1599
P176
250
100
KN Count
80
200
KNN Count
300
Kite Count
150
200
80
KNk
1000
18
15a
10
16F
16
367
16
10*
16'
10
Number of Instantations
Numbar Instantiations
Numbar of Instantations
Number Instantiations
80
1000
250
KW0 Count
60
KNG Court
Kite Count
200
750
60
150
500
20
10a Instantiations
30"
10" Instantations
Instantiatinns
P364
P449
P463
P495
15
100
KNs Count
Count
Khc
Kic
10t
307
10a
101
10a
0-
10a
10
Number of Instantations
Numbar of Instantiations
Number of Instantations
Number of instantations
(a)
P1001
P108
P159
P176
2.0
1.0
1.0
aug KNG proportoon
Dug K74 prsportton
6.8
0.8
aug KNS propor
0.8
0.2
02
022
0.2
0.0
0.9
3.0
0.6
0.8
3.0
0.00
0.8
1.0
1-
t.
1-
P27
P279
P36
aMg. MNG proporton
D0g. KM4 proport Enn
aVg MNN proportion Don
12.0
0.6
06
0.9
0.0
1.0
0.0
0.5
1.0
5. 1-c
1-1
1-
5. 1-4
P364
P449
P463
P495
3.0
1.0
20g. KOVs proporton
evg KW6 propertion
0.4
04
0.4
0.2
0.2
0.2
1-6
1-4
1-8
(b)
Figure 9: opt-6.7b 7b
26

---

Published in Transactions on Machine Learning Research (05/2025)
F10B
P1.19
P1.75
P29
100
Kik Count
(00
KNN- Count
Count
60
Kie
201
117
101
Number or Iratartistionss
Number of Instamaarion
kumber of Instantiation
Number Instantiations P279
F264
P27
P278
80
K4c K4e Court
KNG Count
KAL Court
KMA Caunt
Number of Iretartittioes
Number of Instanderlons
kumber of Instantiations
Number of Indaaniaaionss
P50
P36
P364
8 22
10 COUPT
150
10h- Court
100
20-
Number trstarbutiona P649
Number of Instaniatinss
humber Instantaaioons PeBS
Number Instantiaiins
P433
25 to 2
36
KNG ChuRt
KAN Court
Number Inatarbetions
Number Instamiiaione
Number Instentiations
(a)
PLOB
P159
P176
P19
1.0
0.0
3.0
0.
12
1-5
1-
3-
1-4
P264
P27
P276
P279
wng KW moooer 1.0
2.0
KNN 20000
Mie
1-
L. 1-
1-k
P30
P36
P364
P39
1-6
1-b
1-T.
1-2
P463
Ml Sendee
1-
(b)
Figure 10: Llama-22 -b
27

---

Published in Transactions on Machine Learning Research (05/2025)
P1001
P108
200
25
100
20-
150
KNs Count
KNs Count
KNs Count
100
50
0
109
10
10?
0
10"
10
102
109
10a
Number of Instantiations
Number of Instantiations
Number of Instantiations
P36
P463
400
KNS Count
300
KNS Count
40
200
20
100
10
10a
102
10a
10a
10a
Number Instantiations
Number of Instantiations
(a)
P1001
P108
P30
1.0
1.0
avg. KNs proportion
avg. KNs proportion
avg. KNS proportion
0.8
0.0
0.0
0.6
1.0
1.0
1
4,
1 1-6
4
1 t
P36
P463
avg. KNS proportion
avg KNS proportion
0.4
0.4
0.2
0.2
0.0
0.0
0.5
1.0
4.
1 (c
(b)
Figure 11: gemma-2-99
28

---

Published in Transactions on Machine Learning Research (05/2025)
P@k 4- 0.5 4, 0.5
CCP@K 4- 0.5 f-= 0.5
0.04
0.05
0.02
P@k
0.00
CCP@k
0.00
-0.02
-0.05
00.04
-0.10
10
10
10a
100
10a
10
P@k 4= 0.6
0.4
CCP@k
0.6
0.4
0.04
0.02
0.05
CCP@k
P@kk
0.00
0.00
-0.02
-0.05
-0.04
10
10a
10
100
10a
10a
P@k f. 0.7 t. 0.3
CCP@K f,= 0.7 t.= 0.3
0.04
0.10
0.02
0.05
0.00
CCP@
P@k
0000
0.00
-0.04
-0.5
-0.06
100
10
10a
100
10a
10
P@k f 0.8 t. = 0.2
CCP@K += 0.8 f = 0.2
0.04
0.10
0.02
0.05
0.00
CCP@k
P@k
-0.02
0.00
-0.04
00.5
0.06
10c
10
100
P@k f,
0.9 4, 0.1
CCP@k f, 0.9 0.1
0.10
0.00
0.05
CCP@k
P@k
0.00
-0.05
-0.05
-0.10
-0.00
10a
10
10
10a
10
10
k
Figure 12: bert Mert-large-uncatt
29

---

Published in Transactions on Machine Learning Research (05/2025)
P@k 1,- 0.5 f. 0.5
CCP@k --= 0.5
0.2
0.2
0.1
0.0
CCP@kk
P@k
0.0
-0.1
0.2
-0.2
-.3
100
10a
10a
100
10
10a
P@k == 0.6 t. 0.4
CCP@k 0.6 0.6
6- = 0.4
0.2
0.2
0.1
0.0
CCP@kk
P@k
0.0
0.1
-0.2
0.2
100
101
10
100
10
10
P@k 6, = 0.7 t 0.3
CCP@k = 4 = 0.3
0.2
0.2
0.1
PBk
0.0
CCP@k
0.0
0.1
-0.2
0.2
100
10a
10a
100
10:
10
P@k f, 0.8
0.2
CCP@k f== 0.8 6 = 0.2
0.2
0.2
0.1
P@k
0.0
CCP@k
0.0
00.1
-0.2
0.2
100
10
102
10"
10
10
P@k t= C, 0.1
CCP@K 0 6 = 0.1
0.2
0.1
0.1
P@k
0.0
CCP@k
0.0
0.1
-0.
-0.
0.2
-0.3
100
104
102
100
10
102
k
Figure 13: opt-6.7b
30

---

Published in Transactions on Machine Learning Research (05/2025)
P@k L, 0.5 -.. 0.5
cCP@k 6 0.5 -
0.05
0.05
0.00
0.00
CCP@kk
P@k
-0.05
-0.55
-0.10
0.10
10
10a
10a
100
10a
10a
P@k f, 0.6 0.4
CCP@k = 0.6 t. = 0.4
0.05
0.05
0.00
0.00
OCP@kk
P@k
00.05
-0.05
0.10
100
10
10c
109
10
10
P@k t. 0.7 0.3
CCP@k G 0.7 C 0.3
0.05
0.05
0.00
0.00
CCP@k
PSk
00.05
-0.05
0.10
-.100
100
10S
10a
100
10a
10
P@k t- 0.8 0.2
CCP@k 6 0.8 f. 0.2
0.025
0.00
0.000
CCP@
P@k
-0.025
-0.55
-0.050
0.075
0.10
100
10
10a
100
10
10
P@k t= 0.9 0.1
CCP@k = t. = 0.1
0.04
0.05
0.02
0.00
CCP@k
P@k
0.00
-0.0
0.04
0.05
0.06
100
101
102
100
104
102
k
Figure 14: gemma-22
31


Reviews:
[{"review_id": "fdTQoN8Mhg", "reviewer": "Reviewer_dczc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The submission presents a detailed experimental analysis and proposes a refined typology for categorizing knowledge neurons, which adds nuance to the existing understanding of multilingual knowledge sharing. It quantitatively demonstrates the overlap of knowledge neurons across languages and introduces a new dataset to support further research. However, the novelty of cross-lingual knowledge sharing itself is limited since prior work has already acknowledged shared representations among languages. The primary contribution is the granular neuron-level dissection and the associated causal experiments, which rely heavily on specific threshold settings and the integrated gradients method. These methodological choices may not fully capture the polysemous and context-dependent nature of neuron activations. Furthermore, while the experiments provide valuable insights, the overall practical impact of this refined analysis on improving knowledge editing or retrieval in multilingual PLMs remains to be clearly established.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper refines our understanding of how multilingual pre-trained language models (PLMs) store and retrieve factual knowledge by decomposing knowledge into distinct subtypes—conceptual and relational. Using an attribution method originally proposed by Dai et al. (2022), the study examines the behavior of individual neurons (knowledge neurons) across multiple languages, introducing a fine-grained typology that categorizes neurons based on their activation patterns in relation to specific facts and relationships. A key contribution is the empirical demonstration of significant overlap in these neurons across 10 natural languages and even machine-generated languages, suggesting the existence of a partially unified, language-agnostic retrieval mechanism. Additionally, the authors release the Multi-ParaRel dataset, an extended benchmark for evaluating cloze-style knowledge retrieval in a multilingual context. Although the general concept of knowledge sharing in multilingual models is not new, the work’s main contribution lies in its nuanced analysis of neuron-level behavior, providing new insights into the internal mechanisms underlying knowledge encoding and retrieval across languages.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please revise the sections on motivation and contributions to clarify that while multilingual knowledge sharing has been recognized in prior work, the novelty of this submission lies in its neuron-level analysis. The revised text should emphasize the granular categorization of knowledge neurons (conceptual vs. relational) and its empirical validation across multiple languages. This refinement is critical to securing acceptance by clearly positioning the paper’s contribution relative to established literature.\n\nPlease expand the discussion on methodological limitations, particularly regarding the use of integrated gradients and threshold settings for neuron classification. Clearly delineate how these choices might affect the interpretation of neuron-level behavior and suggest potential avenues for future work to address these limitations.\n\n\nPlease also consider strengthening the literature review by citing relevant and recent studies that address factuality and multilingual aspects in large language models. In particular, please include:\n\n- Hallucinations in Large Multilingual Translation Models\n- Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity\n- Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers\n\n\nAdditionally, consider citing other classic multilingual LLM papers to provide a broader context of related work. This will help contextualize the current work within the existing research landscape and clarify its incremental contribution.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wAOQGNbio6", "reviewer": "Reviewer_fsoT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1: as far as I know, this is the first paper that I am aware of to study relational neurons and show that their exists certain neurons inside of the network that plausibly relations. This is pretty cool.\n\nS2: This paper is generally well written and easy to follow. But note that I noticed a typo in the abstract where the word \"and\" is missing. \n\nS3: This paper does a good job of discussing the different findings including positive and negative ones related to the interpretability of neurons. \n\nS4: I think that experiments are reasonable. \n\nW1: Only working with BERT and Llama models is limiting. That doesn't make this paper bad, but it definitely would've been nice to have some more investigation across more models.\n\nW2: I think my biggest reservation about this paper is that it's not clear that very much has really been accomplished from a very practical standpoint. What I mean by this is that prior works have established that knowledge neurons seem to exist. Now this work reinforces and refines this idea and makes it more specific and back it up with some interesting experiments. However, it isn't shown that this understanding of knowledge neurons is particularly explanatory or powerful. Relatedly, it does not perform the intervention experiments + baseline experiments that would be necessary to show that this understanding of KNs is helpful for engineers working with models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper asks where and how certain types of knowledge are stored inside of language models. They describe a distinction between conceptual and relational knowledge. They use an activation patching method to discover neurons that seem to include conceptual knowledge, relational knowledge, polys, semantic concept, and language agnostic information.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My biggest concern is related to weakness number two as outlined above. My most strong recommendation for the authors would be to consider what types of experiments they could do to confirm that the knowledge neurons the identify are causally useful for manipulating model behavior. Furthermore, it would be ideal at that point to test to see if this kind of method from manipulating model behavior is, competitive with other types of prompting or fine-tuning baseline.\n\nI think one way that it describes how I feel about the paper overall is that I think it's extremely interesting, but I'm not convinced that it's useful.\n\nI can see this paper being of interest and citation value for other machine, learning researchers who care about mechanistic interpretability. But as described above, I also think it has limited practical value. I do not think it would be a mistake to accept this paper. But I feel, currently that I would lean against acceptance in its current form.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VqyUMGXSEA", "reviewer": "Reviewer_e2q8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- **S1. Extensive experiments with diverse LMs / LLMs**, including BERT, mBERT, OPT, Llama 2, and Gemma 2.\n\n- **S2. Broader coverage.** This paper studies the overlap between knowledge neurons for 10 natural languages and an \"unnatural language\", leading to a broader coverage compared with previous studies.\n\n- **S3. Introduction of a new dataset.** The paper introduces Multi-ParaRel, a multilingual version of the ParaRel dataset.\n\n### Weakness\n\n- **W1. Narrow compatibility with existing knowledge attribution methods.** The study is tightly coupled with the “knowledge neurons” framework, limiting the paper’s compatibility with other knowledge attribution methods. The conclusions may vary if a different framework were employed.\n\n- **W2. Strong unspoken assumptions behind the proposed typology.** The authors define Relation Neurons as KNs that appear in at least $t_r\\times N$ instances and Concept Neurons as KNs that appear in less than $t_c\\times N$ instances. While such typology may be compatible with the TREx dataset, which comprises 41 relations with approximately 1k facts per relation, it may not be compatible with general real-world settings. If this is an inherent limitation of the proposed approach, the paper should sufficiently discuss this limitation.\n\n- **W3. Lack of studies on models trained from scratch.** It is difficult to ascertain whether the knowledge is memorized, inferred, or hallucinated. Training a small model on a curated dataset could clarify the origins of stored knowledge.\n\n- **W4. Lack of studies or discussions related to the correlations or representiveness of the languages**. The multilingual experiments ignore the nuance between languages. The introduction mentioned that the languages are not evenly represented. In addition, some languages may be highly related for historical reasons. For example, some words were originally in French and later introduced and adapted into English. Such nuances are ignored in the current experiments and discussions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies knowledge storage and knowledge sharing across multiple languages at the level of intermediate neurons in pre-trained language models, including masked language models and autoregressive language models. Following Dai et al.’s (2022) Knowledge Neurons (KNs), the authors propose heuristics to categorize neurons into monosemantic or polysemantic roles in storing concepts and relations. The paper further studies the roles of neurons across 10 languages and uncovers a substantial overlap across them. The paper also introduces a new dataset for the multilingual study.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**To secure my recommendation for acceptance**, I think it's important to understand the robustness of the proposed typology with respect to varied frequency distributions of concepts and relations. Additionally, the authors should sufficiently discuss the above mentioned limitations of the study.\n\n**To strengthen the work**, there is room for improvement for the presentation of the second section on Related Work. Specifically,\n    \n    - The transitions between sentences in the first paragraph are not smooth and logically coherent.\n    - For the second paragraph, \"The performance of these models is believed to stem from ...\", I guess the authors want to say something like \"competitive performance\".", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This study analyzes how pre-trained language models encode facts by disentangling conceptual and relational knowledge at the neuron level, finding that specific neurons show diverse roles and significant cross-lingual overlap, suggesting a partially language-agnostic mechanism. Reviewers questioned the study's novel contributions beyond established cross-lingual findings, its practical implications, and certain methodological details, and the authors responded by emphasizing their granular neuron-level analysis, providing additional causal evidence, and detailing the approach's constraints and robustness. Overall, I recommend acceptance owing to this paper's neuron-level insights, comprehensive experimentation, new dataset, and its contribution towards advancing fundamental research in model interpretability.
Decision: Accept (poster)