Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models (2024)

Abstract: Large language models (LLMs) demonstrate remarkable multilingual capabilities without being pre-trained on specially curated multilingual parallel corpora. It remains a challenging problem to explain the underlying mechanisms by which LLMs process multilingual texts. In this paper, we delve into the composition of Transformer architectures in LLMs to pinpoint language-specific regions. Specially, we propose a novel detection method, language activation probability entropy (LAPE), to identify language-specific neurons within LLMs. Based on LAPE, we conduct comprehensive experiments on several representative LLMs, such as LLaMA-2, BLOOM, and Mistral. Our findings indicate that LLMs’ proficiency in processing a particular language is predominantly due to a small subset of neurons, primarily situated in the models’ top and bottom layers. Furthermore, we showcase the feasibility to “steer” the output language of LLMs by selectively activating or deactivating language-specific neurons. Our research provides important evidence to the understanding and exploration of the multilingual capabilities of LLMs.

heading: 1 Introduction
text: The brain has its own language for testing the structure and consistency of the world.
Carl Sagan
The pursuit of multilingual capabilities, mirroring our world’s linguistic diversity, is a critical research objective that paves the way for information democratization across linguistic divides. The emergence of pre-trained language models (PLMs) such as mBERT (Devlin et al., 2019) and XLMR (Conneau et al., 2020a) has marked a significant shift towards enhanced multilingual understanding.
* This work was done during internship at MSRA. † Equal contribution. B Corresponding author
Furthermore, large language models (LLMs), such as GPT-4 (Achiam et al., 2023) and PaLM-2 (Anil et al., 2023), have recently demonstrated more excellent multilingual capabilities in language understanding, reasoning, and generation, despite being predominantly trained in English corpora.
Existing studies (Pires et al., 2019; Conneau et al., 2020b) have mainly explored how multilingual PLMs (e.g., mBERT) possess semantic alignment capabilities across languages despite the absence of multilingual parallel corpora. They have identified several critical factors that influence cross-lingual transfer, including training data (e.g., overlapped tokens) and training settings (e.g., shared parameters) (Dufter and Schütze, 2020; Philippy et al., 2023). Nevertheless, the underlying mechanisms by which the model itself process diverse languages at the composition level continue to be an area of vigorous investigation.
To develop a deeper understanding of the multilingual capabilities of LLMs, we draw inspiration from the neurobiological underpinnings of human language faculties (Friederici, 2011; Parr et al., 2022; Khanna et al., 2024). Specific regions within the human brain, such as Broca’s area and
5701
Wernicke’s area have been identified to support particular language functions. To make an analogy with human’s language functions, we posit that regions within the language models can be delineated into two primary components: languageagnostic regions that encompass universal knowledge and pragmatics principles, and languagespecific regions that handle language-specific vocabulary, grammar, and idiomatic expressions. Figure 1 presents such a conceptual illustration of region distribution in LLMs posited by us. Actually, language-agnostic regions have been widely explored in existing literature, including knowledge storing (Dai et al., 2022) and task handling (Wang et al., 2022). However, language-specific regions, especially those supporting multilingual capacities, have been seldom studied in existing literature of of LLMs, which is the focus of our research.
In this work, we first propose a novel detection method called language activation probability entropy (LAPE) to identify language-specific neurons within LLMs. This method involves computing the activation likelihood of individual neurons in response to corpora across different languages. Subsequently, we select neurons with lower language activation probability entropy as languagespecific neurons, i.e., those having a higher activation probability for one or two particular languages and a lower probability for others. Furthermore, based on the proposed LAPE method, we have conducted a systematic study with language-specific regions of two popular open-sourced LLMs, leading to several major findings: • First, the proficiency of an LLM in processing a particular language can be significantly impacted by a minuscule proportion of its neurons. Deactivating such language-specific neurons leads to a remarkable degradation in the model’s understanding and generation abilities for that language. • Second, neurons specific to individual languages are predominantly located in the bottom and top layers of LLMs. The bottom layers mainly serve to process the input from various languages into the unified semantic space of a highresource language (e.g., English), while the top layers project the semantic content (after the processing of middle layers) into the respective tokens in the corresponding vocabulary of each language. • Third, we demonstrate the potential to “steer” the output language of LLMs by selectively activating and/or deactivating certain neurons. Our approach could provide a promising solution to
mitigate the off-target issue (e.g., the tendency of LLaMA-2 to reply in English to Chinese queries), while stimulating the capabilities of cross-lingual generation tasks.
To the best of our knowledge, it is the first study that investigates language-specific regions inside LLMs and analyzes the how these regions influence LLMs’ capabilities to process multilingual texts. We introduce the concept of “language-specific neurons” and propose language activation probability entropy to identify such neurons in LLMs. We make available the identified language-specific neurons and corresponding code at https://github. com/RUCAIBox/Language-Specific-Neurons.

heading: 2 Identifying Language-Specific Regions

heading: 2.1 Background
text: Currently, LLMs are predominantly developed on an auto-regressive Transformer architecture (Vaswani et al., 2017), in which the basic building blocks are the multi-head self-attention (MHA) and the feed-forward network (FFN). Given the hidden state hi−1 ∈ Rd of (i− 1)-th layer of a specific token, the MHA module inside the i-th layer can be expressed as follows:
h̃i = Attn(hi−1W iq ,H i−1W ik,H i−1W iv) ·W io , (1)
where W iq , W i k, W i v , and W i o represent the trainable parameters, and H i−1 stands for the hidden states in the previous layer of the whole sequence. Subsequently, the FFN module is described by the following formulation:
hi = act_fn(h̃iW i1) ·W i2, (2) where W i1 ∈ Rd×4d and W i2 ∈ R4d×d are parameters and act_fn(·) denotes the activation function (e.g., GELU (Hendrycks and Gimpel, 2016) for BLOOM (Scao et al., 2022)). A neuron is defined as a linear transformation of a single column in W i1 followed by a non-linear activation. Consequently, a FFN module within a single layer consists of 4d neurons. As a new variant of activation function, GLU (Shazeer, 2020) has been widely used in recent LLMs (e.g., LLaMA (Touvron et al., 2023a)) for improving the performance of Transformer:
hi = (act_fn(h̃iW i1)⊗ h̃iW i3) ·W i2. (3) In our work, the j-th neuron inside the i-th FFN layer is considered to be activated if its respective activation values act_fn(h̃iW i1)j exceed zero (Nair and Hinton, 2010).

heading: 2.2 Language Activation Probability Entropy
text: In existing research, neurons within the FFN modules are found to be capable of storing factual knowledge (Dai et al., 2022), encoding positional information (Voita et al., 2023), responding to particular syntactic triggers (Gurnee et al., 2024), etc. Inspired by these findings, we posit that there exist specific neurons in LLMs for multilingual processing. Next, we introduce a new detection method based on language activation probability entropy (LAPE) to identify language-specific neurons.
Our research primarily focuses on pre-trained foundation models (e.g., LLaMA-2 and BLOOM), rather than fine-tuned models that have undergone instruction tuning or RLHF, which helps reduce other influencing factors. Specially, we feed existing LLMs with multilingual texts, each written in a single language. For the j-th neuron in the i-th layer, we then compute the activation probability when processing texts in language k:
pki,j = E ( I(act_fn(h̃iW i1)j > 0) | language k ) ,
(4) where I is the indicator function. The activation probability is empirically estimated by the likelihood that the neuron’s activation value exceeds zero. Subsequently, we can obtain the distribution pi,j = (p1i,j , . . . , p k i,j , . . . , p l i,j) for each neuron, indicating its probability of activation for each language. To convert pi,j into a valid probability distribution, we apply L1 normalization, yielding p′i,j . The entropy of this distribution, which we refer to as language activation probability entropy, is computed to quantify the neuron’s language activation reaction:
LAPEi,j = − l∑
k=1
p′ki,j log(p ′k i,j). (5)
We designate neurons with low LAPE scores as “language-specific neurons”, as they demonstrate a predilection for activation in response to one or two languages, while showing reduced activation probabilities for others.
In implementation, we collect multilingual corpora sourced from Wikipedia, a widely recognized and high-quality resource for diverse languages, and sample documents to create a dataset comprising 100 million tokens for each language. Subsequently, we input these tokens into a target LLM and follow Equations 4 and 5 to compute the LAPE
score for individual neurons. Finally, we select neurons that fall within the lowest percentile of LAPE scores, specifically targeting the bottom 1%. To refine our selection, we further impose a predefined threshold to exclude neurons exhibiting negligible activation probability: a neuron is deemed specific to language k if its corresponding activation probability pki,j surpasses the threshold.

heading: 3 Experiments
text: In this section, we present empirical evaluation to substantiate the efficacy of our proposed LAPE method and elucidate the impact of languagespecific neurons on multilingual capacities.

heading: 3.1 Experimental Setup
text: Models. We conducted our study primarily on two publicly available large language models (LLMs): LLaMA-2 (Touvron et al., 2023b) and BLOOM (Scao et al., 2022). Among them, LLaMA-2 is recognized for its excellence as a foundational model, primarily pre-trained on English texts, while BLOOM is noted for its multilingual proficiency due to a balanced distribution of training languages. Specifically, we investigate multiple versions of LLaMA-2: the 7B, 13B, and 70B models, which contain approximately 352K, 553K, and 2.29M neurons, respectively. For BLOOM, we select the 7.1B version, consisting of roughly 492K neurons. The languages we focus on include English (en), Simplified Chinese (zh), French (fr), Spanish (es), Vietnamese (vi), Indonesian (id), and Japanese (ja). We exclude Japanese (ja) for BLOOM since it has not been pre-trained on Japanese corpora. To verify the generality of our method, we also include LLMs under different settings, including LLaMA-2 Chat, OPT (Zhang et al., 2022), Mistral (Jiang et al., 2023), and Phi2 (Javaheripi et al., 2023).
Dataset. Our analysis of language-specific neurons is conducted across two distinct dimensions: • Language modeling: We assess the multilingual language modeling capability using perplexity (PPL) scores on Wikipedia corpora. Our dataset comprises one million tokens per language, all sourced after September 2022 to ensure the content has not been included in the training sets of either LLaMA-2 or BLOOM. • Open-ended generation: To evaluate the model’s multilingual generation capabilities in realworld scenarios, we translate a set of questions
from the Vicuna dataset (Chiang et al., 2023) into target languages using gpt-4-0125-preview. The questions span a broad spectrum of topics, deliberately excluding mathematics and coding queries to maintain a focus on language processing proficiency and avoid confounding variables. We utilize greedy search with a repetition penalty of 1.1 to generate output. The resulting texts are then assessed by GPT-4 on a scale ranging from 1 to 10, following the methodology described by Zheng et al. (2023).
Identification methods. For comparison, we consider the following methods for identifying language-specific regions:
(a) Language activation probability entropy (LAPE, ours): The pertinent details are provided in Section 2.2. The threshold is set to the 95-th percentile of all activation probabilities. For instance, in the case of LLaMA-2 (70B), threshold is established at 0.515. This stipulates that the neurons we select are required to exhibit an activation probability exceeding 0.515 for at least one language.
(b) Language activation probability (LAP): There are also methods to identify neurons directly. But most of them are infeasible due to the high computational cost (Gurnee et al., 2024; Dai et al., 2022). Inspired by Voita et al. (2023), we apply
their method by identifying a neuron as languagespecific if its activation probability exceeds 95%.
(c) Language activation value entropy (LAVE): This is a variant of our proposed method, wherein we substitute the activation probability with the mean activation value across languages. Similarly, we normalize the activation value and calculate the entropy to find neurons with high activation value in response to particular languages.
(d) Parameter variation (PV): By extending the work of Zhao et al. (2023a), we compare the model parameters before and after monolingual instruction tuning to identify language-specific parameters. In particular, we train individual models on the Alpaca instruction datasets (Taori et al., 2023) and its multilingual version (Chen et al., 2023b) which comprise 52,000 instances for each target. These models undergo training for two epochs, with a batch size of 128 and a constant learning rate of 1e-5. We mainly consider the parameters inside the MHA and FFN modules, i.e., the weight matrices in Equations 1, 2, and 3. We compute the rate of change across various languages, and select parameters that exhibit a low rate of change in one or two languages but a high rate in others. In detail, we refine the change ratio by subtracting the maximum value and then conduct L1 normalization for entropy calculation.
(e) Random selection (RS): Additionally, we add a baseline to randomly select neurons for each language, serving as a reference for different methods as shown in Figure 8 in Appendix.

heading: 3.2 Main Perturbation Experiments
text: In this part, we conduct the perturbation experiments by deactivating the identified languagespecific regions. Specially, for all the comparison methods in Section 3.1, we identify 1% of the neurons or parameters and treat them as languagespecific regions. We then set the activation values of these neurons to zero, or directly zero the parameters to assess the impact on the model’s multilingual capacities based on language modeling and open-ended text generation tasks.
Figure 2 presents the perturbation results (measured by PPL change on language modeling task) of different identification methods on LLaMA-2. Overall, we can see that large impact values for LAPE mainly occur in diagonal entries. It indicates that our LAPE method is adept at identifying language-specific neurons. Deactivating neurons associated with a specific language predom-
inantly affects the PPL results of that language, with negligible effects on others. In contrast, the variant utilizing activation values (LAVE) causes cross-lingual interference (e.g., the entry ⟨id, es⟩), and the rest methods fail to exhibit clear languagespecific patterns.
We further investigate whether our method is effective across different model sizes and model types. As depicted in Figure 3, the language-specific degradation patterns are evident for LLaMA-2, BLOOM, OPT, Mistral, and Phi-2.
An interesting find is that neurons in larger models tend to be specialized for a single language rather than being shared among two or more languages. Furthermore, we can find that there exists a high correlation between Chinese and Japanese: when we deactivating the neurons specific to one language, the performance of the other language would be affected (e.g., the entries ⟨zh, ja⟩ and ⟨ja, zh⟩). By inspecting into the identified neurons of the two languages, we note that a substantial amount of neurons (approximately 25%) actually overlap for both languages. It is likely because Chinese and Japanese partially share common characters. In addition, it can be observed that our LAPE method leads to similar findings on BLOOM and other LLMs, which further verify the generality of our proposed LAPE method.
When employing LAPE to open-ended generation tasks based on the Vicuna dataset, we can clearly observe from Table 1 that deactivating language-specific neurons significantly impairs the generation capabilities in the target language. We further provide an illustrative example in Table 2 of the model’s response to a question in Simplified Chinese when the neurons associated with Simplified Chinese are deactivated. We can see a chaotic mixture of Traditional Chinese characters and redundant English phrases, indicating a severe degradation of language capacity in Simplified Chinese.

heading: 3.3 Further Analysis
text: After presenting the main experiments, we further conduct detailed analysis experiments to investigate language-specific neurons and their impact. Unless specified, all analysis results are obtained based on LLaMA-2 (70B).

heading: 3.3.1 Distribution and Identification Ratio

heading: 836 5,153 6,082 6,154 4,980 6,106 5,216
text: Neuron distribution across languages. After running our LAPE method on LLaMA-2 (70B), we identify around 23K language-specific neurons. The distribution of these neurons across individual languages is detailed in Table 3. Since neurons may be shared by multiple languages, the sum of language-specific neurons actually surpass 23K. Overall, except English, the distribution of neurons is relatively even across languages. However, the number of English-specific neurons is much smaller than the other languages. We speculate that English is the dominant language in LLaMA-2, and thus it requires fewer neurons to support the specific language ability.
Increasing threshold ratios for identification. In Section 3.2, we consider a mere 1% of the neurons as being language specific. We further vary the selection ratio of language-specific neurons from 0 to 10%, to examine its impact on multilingual processing. Here, we select French for study, while the results on the other languages are similar. When deactivating neurons specific to French, we observe a significant increase in the PPL on French in Figure 4, while the impact on the rest languages are
relatively limited with the exception of Spanish. it is consistent with our intuition: the performance of the being perturbed language and its related (or similar) language would be severely harmed.

heading: 3.3.2 Structural Distribution Analysis
text: In this part, we analyze how language-specific neurons are distributed across different layers.
Language processing is concentrated at bottom and top layers. In Figure 5, we present the layer-wise distribution of language-specific neurons across various layers, which has a skewed “U” shape. This finding indicates that languagespecific neurons have a pronounced concentration in the top and bottom layers. Specifically, the second layer has approximately 7,000 languagespecific neurons, while layers 5 through 47 only contain about 100 neurons each. Further, the neuron count gradually increases, with the final four layers each comprising over 1,000 neurons. The complete statistics about the layer-wise distribution across various languages are reported in Table 10 of the Appendix.
Why such a skewed distribution? To understand why such a skewed distribution occurs, we seek explanations from multilingual semantic representation by exploring how multilingual aligned texts are represented across the layers. Specially, we employ the multilingual Vicuna dataset (Section 3.1), comprising of aligned texts in different languages. Given a group of aligned texts, we feed them into the LLM and obtain the sentence embedding of each text for each layer. We then compute the mean sentence embedding similarity (SES) between each pair of the aligned texts across
languages in Figure 6. Interestingly, the SES curve shows an opposite trend with the distribution of language-specific neurons. At the beginning, the similarity quickly increases, then reaches the peak, and gradually decreases to a small value. This finding suggests that: at the bottom layers, the LLM needs to map aligned texts of different languages into the shared representation space, thus requiring more language-specific neurons for semantic transformation; while at top layers serving for token generation, the LLM needs to handle vocabulary mapping, which also requires the support of more language-specific neurons.

heading: 3.3.3 Language Dominance Analysis
text: Since the high-resource language (i.e., English) in the LLaMA-2 training corpus has a surprisingly smaller number of neurons than other languages, as indicated in Table 3, we speculate that there might exist some dominance relations between high-resource and low-resource languages, which depends on the composition of pre-training corpus.
Language dominance measurement. Inspired by Xu et al. (2023), we transfer the sentence embeddings across different languages into the same space around a target language, and examine how texts from the other languages are aligned to the texts of the target language. Firstly, for each language k, we compute the mean sentence embeddings of all its texts, and obtain vik as the language vector of k at i-th layer. Then we follow the same formula proposed by Xu et al. (2023) to conduct the space mapping for each text from language k:
ĥik = h i k − vik + vic, (6)
where ĥik denotes a transformed embedding of some text in the i-th layer of language k. Here, we specify c as the target language, and compute the mean sentence embedding similarity (SES, Section 3.3.2) over all the sentence pairs between languages k and c, based on the transformed representations in Eq. 6. A larger SES score indicates language c has a larger dominance degree.
Low-resource languages are centered around high-resource languages. To compute the dominance degree, we still use the multilingual Vicuna dataset (Section 3.1). From the results of LLaMA2 (70B) in Figure 7a, we can see that the mean SES is obviously higher than all other languages when the target language is English. As English is the high-resource language of LLaMA-2, lowresource languages need to be aligned with English for achieving better performance. When it comes to BLOOM (170B) in Figure 7b, several languages (e.g., English and Chinese) show dominance, since it is originally pre-trained on multilingual corpora.

heading: 3.3.4 Case Study
text: Finally, we explore the possibility to “steer” the output language of LLMs to mitigate the off-target problem and facilitate cross-lingual generation.
Researchers have observed that when prompting in one language, language models may generate responses in a different language, such a phenomenon referred to as the off-target language issue (Gu et al., 2019; Sennrich et al., 2023). We speculate that language-specific neuron might not be activated in this case. Thus, we attempt to manually activate these neurons by increasing their activation value to the average activation value of this language. As shown in Table 4, it is evident that the model demonstrates a significantly higher propensity to respond in the expected language, as indicated by the language accuracy. Additionally, the generation quality, as assessed by GPT-4 metrics, shows remarkable improvement.
Moreover, we attempt to steer cross-lingual generation by posing a question in one language and answering it in another. As presented in Table 5, we deactivate Spanish neurons and activate Chinese neurons when replying to a question in Spanish. The model successfully generates fluent Chinese text, showing the feasibility of altering the output language through neuron-level manipulation.

heading: 4 Related Work
text: Multilingual ability. Multilingual ability has been widely studied since the emergence of pretrained language models (Devlin et al., 2019; Xue et al., 2021). Recently, LLMs (Zhao et al., 2023b; Nguyen et al., 2023) showcase more excellent multilingual capabilities even without explicit language alignment (Kulshreshtha et al., 2020; Cao et al., 2020). Extensive research has been made to investigate the factors that influence models’ multilingual ability (Philippy et al., 2023). For example, linguistic similarity has been examined (Pires et al., 2019; Dufter and Schütze, 2020; Wang et al., 2019; Conneau et al., 2020b), which is generally believed to correlate with models’ cross-lingual ability. Spe-
cially, “word order” shows some contradictions about whether it really affects multilingual ability (Pires et al., 2019; Deshpande et al., 2022). Not only limited to language property, training settings have also been considered (Lauscher et al., 2020; Ahuja et al., 2022).
Existing work has explored language-agnostic (or language-shared) components within multilingual models. For example, researchers concentrate on shared knowledge across various languages (Stanczak et al., 2022; Chen et al., 2023a; Zhao et al., 2023a; Bricken et al., 2023). However, the exploration of language-specific components within LLMs remains an under-investigated area.
Neuron analysis. Neuron analysis has gained significant attention in recent years, paralleling research in neurobiological studies of the human brain (Friederici, 2011; Parr et al., 2022). Originating from vision models (Bau et al., 2020; Mu and Andreas, 2020), neuron analysis views neuron activation as the recall of learned knowledge (Sajjad et al., 2022). Researchers widely adopt these methods to analyze the sources of specific abilities or skills in language models, including sentiment analysis (Radford et al., 2017), knowledge storage (Dai et al., 2022), and task-solving (Wang et al., 2022).
Recent studies have also discovered that certain
neurons can convey specialized contexts (Gurnee et al., 2023; Bills et al., 2023), such as positional information (Voita et al., 2023) and linguistic properties (Bau et al., 2018; Xin et al., 2019; Dalvi et al., 2019, 2020). Moreover, Gurnee et al. (2024) utilize Pearson correlation to calculate neuron similarity, identifying some universal neurons across models. In contrast to previous research, we have developed a method applicable to LLMs that unveils the mechanism of their multilingual abilities. This approach offers a more practical and effective solution for neuron analysis in multilingual scenarios.

heading: 5 Conclusion
text: Despite the impressive multilingual capabilities demonstrated by LLMs, the understanding of how these abilities develop and function remains nascent. In this paper, we introduced a novel detection method, i.e., language activation probability entropy (LAPE), to pinpoint language-specific neurons within LLMs. LAPE assesses the response of individual neurons to various languages, selecting those with a propensity for activation when exposed to one or two languages. Based on LAPE, we further conducted extensive experiments to investigate the multilingual capabilities of LLMs. Specially, we have found that an LLM’s proficiency in processing different languages is significantly influenced by a small subset of neurons, which are mainly located in the model’s top and bottom layers. We have further demonstrated that the output language of LLMs can be directed by selectively enabling or disabling these language-specific neurons. For future work, we aim to leverage these findings to enhance knowledge transfer between major and minor languages and devise efficient strategies for continual pre-training to better accommodate specific languages.

heading: Acknowledgement
text: This work was partially supported by National Natural Science Foundation of China under Grant No. 62222215, Beijing Natural Science Foundation under Grant No. L233008 and 4222027. Xin Zhao is the corresponding author.
Limitations
In this study, we employ language activation probability entropy as a metric to identify languagespecific neurons. However, it is important to note
that our method is relative to the presence of multiple languages. In scenarios where only a single language is present, establishing an absolute threshold to determine the language-relatedness of neurons is not feasible. Moreover, the criteria for distinguishing between high-resource and low-resource languages within the model warrant further investigation. The model’s possibility to managing a large number of languages, as well as the differences between various languages, represents a promising avenue for future research. Finally, our research has only begun to explore the possibility for directing the output language of the model. Developing strategies to harness these identified neurons for enhancing the model’s multilingual proficiency is still worth exploring.

heading: A Appendix
text: Table 7 compiles the statistics of pre-training corpora in LLaMA-2 (Touvron et al., 2023b) and BLOOM (Scao et al., 2022).
We list the number of language-specific neurons across different layers of BLOOM (7B), LLaMA2 (7B), LLaMA-2 (13B), and LLaMA-2 (70B) in Tables 6, 8, 9, and 10.

heading: Future Work
text: The model’s possibility to managing a large number of languages, as well as the differences between various languages, represents a promising avenue for future research. Finally, our research has only begun to explore the possibility for directing the output language of the model. Developing strategies to harness these identified neurons for enhancing the model’s multilingual proficiency is still worth exploring