How do Large Language Models Handle Multilingualism?
Authors: Yiran Zhao
Wenxuan Zhang
Guizhen Chen
Kenji Kawaguchi
Lidong Bing
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Multilingual
Language Specific Neuron
Primary area: natural_language_processing

Abstract: Large language models (LLMs) have demonstrated impressive capabilities across diverse languages. This study explores how LLMs handle multilingualism. Based on observed language ratio shifts among layers and the relationships between network structures and certain capabilities, we hypothesize the LLM's multilingual workflow ($\texttt{MWork}$): LLMs initially understand the query, converting multilingual inputs into English for task-solving. In the intermediate layers, they employ English for thinking and incorporate multilingual knowledge with self-attention and feed-forward structures, respectively. In the final layers, LLMs generate responses aligned with the original language of the query. 
To verify $\texttt{MWork}$, we introduce Parallel Language-specific Neuron Detection ($\texttt{PLND}$) to identify activated neurons for inputs in different languages without any labeled data. Using $\texttt{PLND}$, we validate $\texttt{MWork}$ through extensive experiments involving the deactivation of language-specific neurons across various layers and structures. 
Moreover, $\texttt{MWork}$ allows fine-tuning of language-specific neurons with a small dataset, enhancing multilingual abilities in a specific language without compromising others. This approach results in an average improvement of $3.6\%$ for high-resource languages and $2.3\%$ for low-resource languages across all tasks with just $400$ documents.

How do Large Language Models Handle Multilingualism?
Yiran Zhao 1.2f Wenxuan Zhang2.3
Guizhen 2
Kenji Kawaguchi Lidong Bing?
National University of Singapore
DAMO Academy, Alibaba Group, Singapore
Hupan Lab, 310023, Hangzhou, China Nanyang Technological University, Singapore
Abstract
Large language models (LLMs) have demonstrated impressive capabilities across diverse languages. This study explores how LLMs handle multilingualism. Based on observed language ratio shifts among layers and the relationships between network structures and certain capabilities, we hypothesize the LLM's multilingual workflow (MWork); LLMs initially understand the query, converting multilingual inputs into English for task-solving. In the intermediate layers, they employ English for reasoning and incorporate multilingual knowledge with self-attention and feed-forward structures, respectively. In the final layers, LLMs generate responses aligned with the original language of the query. verify MWork, we introduce Parallel Language-specifie Neuron Detection (PLND) to identify activated neurons for inputs in different languages without any labeled data. Using PIND, validate MWork through extensive experiments involving the deactivation of language-specifes neurons across various layers and structures. Morcover, MWork allows fine-tuning of language-specifie neurons with small dataset, enhancing multilingual abilities specific language without compromising others. This approach results in average improvement of 3.6% for high-resource languages and 2.3% for low-resource languages across all tasks with just 400 documents
Introduction
Recent advancements in large language models (LLMs) (OpenAl 2023 Touvron et al. 2023
Team et al., 2023) have dramatically transformed the field of natural language processing (NLP). Thanks the extensive pretraining massive corpora mixed with different languages, these models demonstrate remarkable capabilities understanding and generating text across multiple languages (Huang al. 2023 Zhang al.) 2023a Zhao et al. 22244).. Despite these advancements, the intricate mechanism of their multilingual processing behavior remains largely unclear, which leads to important research question: How do large language models handle multilingualism?
To understand the working mechanism of LLMs, existing studies mainly focus on the relationship between model architectures and certain capabilities, with some investigating reasoning abilities with self-attention layers (Hou et al., 2023] Stolfo et al., [203 Friedman et al., 2023). and others interpreting feed-forward layers key-value memories for storing factual knowledge (Geva et al.) 202T, DDi et al. 2022 Meng et al. 2022). However, these works solely center on English and neglect the multilingual features of LLMs in their interpretations.
'This work was done during the internship of Yiran Zhao at Alibaba DAMO Academy. Wenxuan Zhang the corresponding author: i aaakkhhnng@gmaall com GGuizeen Chen is under the Joint Ph.D. Program between DAMO Academy and NTU.
OOur code is available at wentttpss: tttptgggtttt.tt///////t//tt wwt.hhww....o///o-------------
www.ttiiinggallam Laogual_ann
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

En
non- En
Layer Index
Layer Index
(a) Vicuna-13b-v1.5
BLOOMZ-7b!
Figure I: Ratio of English and non-English tokens among layers given non-English queries.
To gain an initial understanding of the multilingual mechanism of LLMs, we test LLMs with various non-English queries and decode the hidden embeddings of each layer to tokens within the LLM's vocabulary. Subsequently, we classify these decoded tokens into either English or non-English, and analyze the ratio. [igurr[[illusraatee the ratio of English and non-English tokens for each layer of two LLMs. We observe that non-English queries initially generate non-English embeddings as expected. However, as queries progress through the middle layers, the representations surprisingly become English-centric. In the final layers, there is a reversion to predominantly non-English embeddings, matching the non-English queries.
Motivated by the observed transfor- mation above, we hypothesize three- stage multilingual workflow: under- standing, task-solving, and generat- ing. This involves understanding the original non-English queries and in- terpreting them in English, solving tasks in English, and reverting out- puts back to the original language. Furthermore, building upon previous studies that link self-attention struc- tures to reasoning and cedd-fowward structures to factual knowledge stor- age (Hou et al., [2033] Geva et al., 2021). we further decouple the task- solving stage into reasoning with self-attention structures and extract- ing multilingual knowledge with feed-forward structures. Therefore, our hypothesized Multilingual Workflow (MWork) illustrated in outurezoutlines the three operational stages of LLMs in processing multilingual queries: Initially, LLMs understand queries by converting diverse linguistic features into unified representation. In the task-solving phase, LLMs reason in English and incorporate multilingual knowledge to obtain factual content, using self-attention and feed-forwant structures, respectively. Finally, models generate responses in the original language as the original query.
Understand
Solve Task
Generate
Thinking (English)
Knowledge (Multilingual)
(Multilingual)
(Multilingual)
Self Attention
Figure 2: Our hypothesized multilingual workflow, MWo rk, converts multilingual queries to English for reasoning in English and generates responses in the original language, demonstrating layered processing approach.
To verify the proposed MWork, we could extract language-spest them within different structures, and observe their corresponding effects, thereby assessing the functionality of corresponding structures and validating our hypothesis. To identify the parameters to be activated, we develop novel approach called Parallel Language-specifie Neuron Detection (PIND). Unlike existing methods that rely on fine-tuning mnne-tuning(Frrg
parameters, selectively deactivate
and Carbin 2018: Zhang et al. 2023bb, labeled data (Tang et al. 2024 Liu et al. 2024), parallel corpora (Libovicky et al. 2020 Tanti et al. T Zhang et al. 2024) to detect activated parameters, PLND measures the significance of individual neurons with respect to the input in both attention and feed-forward structures without any labeled data or parameter adjustments. Using PLND, we identify language-spest -secciffc neurons by inputting free text corpus of that language and isolating consistently activated neurons. We find that by deactivating language-specific neurons which account for only 0.13% of neurons, LLMs' performance multilingual summarization task could drop by 99%.
We then extensively verify the hypothesized MWork framework using the proposed PLND method. Employing various benchmark tasks, including XQuAD (Artetxe et al. 2020 for understanding,

---

MGSM (Shi et al. 2022) for reasoning. X- X-CSQA (Lin et al. 2021) for knowledge extraction, and XLSum for generation (Hasan et al., (021), we selectively deactivate language-specifte neurons in each component and verify the functionality of the component by observing a significant decline in performance on the corresponding task. For example, when deactivating the language-specti neurons in the understanding layer, the performance on the multilingual understanding task XQuAD remains stable in English, while experiencing a decrease 14% in non-English languages. Other tasks exhibit similar pattern when deactivating corresponding neurons. More importantly, with the verified MWok framework, enhancing the multilingual capabilities LLMs can thus be achieved through neurons for certain capabilities. With remarkable reduction in the training corpus size to a mere few hundred documents, this fine-tuning procedure enhances the multilingual capabilities of LLMs for both high-resource and low- resource languages by average of 3.6% and 2.3% across all tasks, respectively. Notably, even without an English training corpus, there is noticeable improvement in English performance, the enhancement of language-specingg neurons yields greater accuracy in enhancing specific languages, while simultaneously ensuring a clear division of parameters among different languages. In summary, the verified MWork r'k reveals how LLMs handle multilingual tasks and offers an effective approach for conducting language-specific enhancements without compromising performance in other languages.
the fine-tuning of language-specti
2 Parallel Language-speeifie Neuron Detection (PLND)
To verify the hypothesized workflow, propose PLND that effectively detects language-specific neurons without relying on any labeled data. In essence, PLND identifies neurons crucial for handling individual documents, with language-specific neurons being those that consistently show high importance when processing documents in a particular language.
2.1 Sequential Neuron Detection
We define neuron as single row column of a parameter matrix of a language model. To identify neurons responsible for a specific language, it is crucial to discern the significance a neuron with respect to the inference of a given input. Specifically, when processing the input in the model, we denote the hidden embedding before the i-th layer in Transformer (Vaswani et al.) 2017) as h, and the hidden embedding after the i-th layer h441 (h()), where T, represents the parameters of the i-th layer. For specific neuron within the i-th layer, denoted (1),, either located in the attention or feed-forward network, we quantify its importance in processing the input c by measuring the difference in the hidden embedding after the i-th layer, ie., h4+1, when (1) is activated or deactivated. Formally, the impact of neuron (1) for input e defined as
Imp(N(0 (c) |[TAN(((()) T2(h4)
(1)
where (-) denotes deactivating ( in Ti i.e., setting all parameters of the neuron (0) to zero. With a set of n corpus in specific language, denoted as C fci. , Ct, , }. we calculate the importance of each neuron in each layer each corpus. Furthermore, we can obtain language-specific neurons that are important to all corpus in that language. i.e..
(N(1) Imp(N(() (q) 2 e, Vei C}.
where is the pre-defined threshold.
2.2 Parallel Neuron Detection
The sequential neuron detection requires traversal of all neurons and inputs sequentially and thus is time-consuming. To address this, we further propose parallel algorithm for accelerating the process.
Feed-Forward Network (FFN) In the latest open-source models, when processing input c, the feed-forward network in certain layer is defined as
FFN(.) = (W gate (x)) W Wap(z)) (x))))oonnnn) Woown:
(3)
3

---

where x E Rixddode is the embedding fed into the FFN, W Wgate. W E R (dinttttxdiidtt Kdiittr Rdinser X Xdmodet The calculation of the importance of the k-th neuron in Wap when processing the input e, as presented in Equation I can be equivalently transformed to
W Wown
Imp(Wapl::) A|lc)= [|FFN(x)) FFN( = - Mask W down (2)
where hym dinter represents the embedding before Woown and Mask[[] dinter is vector with the k-th element equal to and the rest equal to O. To calculate Imp Wapl: kllc) for k E dinter parallelly, introduce diagonal mask matrix of size (dinter, dinter). denoted as Mask. Therefore,
Imp = ||(/4mm Mask) B down (z)
(5)
Furthermore, we observe that deactivating the k-th neuron of W down equivalent to deactivating the k-th neuron in Wap. employing Equation ().
up. as they both result in hmm [k] = 0. Hence, we can also derive Imp W down [c) by
Self-Attention Network
When processing imput the self-attention network in a certain layer is
Attention(z) = Softmax ( Wo(x)WI(e) Wv(z),
Vd
where Wo. WK, Wr E Rdluude Xdmind Since (v is not in the non-linear softmax calculation, we can calculate Imp(Wrle) by applying Equation 5 For Wo, we obtain Imp(We) Ai]ee) by deactivating its k-th neuron, specifically, We + Wol: kj = O. Firstly, we calculate the difference in attention weight before and after deactivation, prior to scaling and softmax,
(x = (x( (x()) = [:(x)( (a)))((())))))
R
Next, as the changes in attention exhibit positive correlation with the changes in the output of this layer, the importance of Woli k] in processing c, as defined in Equation I can be approximated
Alec) ~ ||attention()) - attention (x)ll2
W (z) (x) Ax(x)) softmax
Wo(x)
(8)
softmax
vd
Vd
This process can also be calculated parallel, specifically,
A (x) = (o(+)WF(x)) (x(x)WW(x)) = Wo(tt).resize(. 1, dmia) x W (x( (1,1, dmia) E pixxxdmii
(9)
Therefore, the importance of Wq processing input is calculated by
W Wo(x)WT(<)) -(a)) softmax
W Wo(x)
Imp(Wolc) ze ssftmax
(10)
vd
Vd
Similarly, since WK symmetrical Wo, Imp [x|e]] can be calculated in the same way.
2.3 Detection of Language-ppeefffe Neurons
We then apply PLND to selected languages and models to validate its effectiveness in detecting singuage-speciff neurons and to further investigate the relationships between languages.
Experimental Setup. We test two source models that perform well on multilingual tasks, including Vicuna-7b-v/.5"" Chiang et al. 2023) and Misttrl--7h-nnstrrat--ud (Jiang al. 2023) For simplicity, we abbreviate them as Vicuna and Mistral hereafter to represent the two models respectively. We select the text summarization task with the XI.Sum (Hasan et al.) 2021) dataset as the reference task to evaluate multilingual performance as it requires the model to comprehend the
AW[ (-) represents the linear matrix product of the input and the parameter W. i.e., W(2) == xW.
In some models Vieuna and Mistral, dmoodet dmid, but we use different notations avoid ambiguity.
"We do not directly utilize 1lama2-chat as it does not follow multilingual instructions, consistently responding
in English regardless of the language of the query.

---

Table 1: Multilingual performance XLSum when deactivating language-speciinn neurons ("Lang- Spec") and an equivalent number of randomly selected neurons ("Random")
Model
Method
Fr
Zh
Es
Ru
Avg.
Original
14.2
61.1
10.4
20.8
26.6
Vicuna
Deactivate Random
14.1
61.6
10.4
20.8
26.7
Deactivate Lang-Spec
0.83
0.00
0.24
0.42
0.37
Original
15.2
56.4
10.6
21.0
25.8
Mistral
Deactivate Random
15.4
55.9
10.2
21.2
25.7
Deactivate Lang-Spec
0.21
0.39
0.15
0.07
0.21
input text and generate coherent fragment. We adopt high-resource languages including French (Fr), Chinese (Zh), Spanish (Es), and Russian (Ru), as their initial performance on those languages already quite reasonable for observing the multilingual processing mechanism. Furthermore, we utilize OSCAR (Caswell et al. 2020) corpus which contains web crawling texts for each language to compile a innguag--peecffie corpus without task-specific considerations. More details are presented in Appendix B
Existence of Language-Speet Neurons Using PLND, we feed a corpus a specific language to LLMs and identify neurons that are consistently activated, which are responsible for processing queries in that language. To ascertain whether these neurons are genuinely sanguagespeccing we assess the performance of LLMs in corresponding languages when these neurons deactivated versus when the same number of randomly sampled neurons are deactivated.
Table 1 demonstrates the decline of multilingual capabilities when deactivating language-speciin neurons. Although just deactivating around 0. 0.13% neurons, L.LMs lose their multilingual capabilities and fail to generate meaningful content. In contrast, deactivating the same number of randomly selected neurons does not yield any difference. Therefore, the detected neurons are language-specific and related to handling corresponding multilingual inputs.
2.4 Analysis of Sppeifee-Specific Neurons
We further investigate the degree of overlap among their language-spectin neurons. Our findings reveal that in both Mistral and Vicuna, English shows limited overlap with other languages, indicating many language-specific neurons, while languages within the same family, such as Spanish, French, and English, demonstrate more overlap. More details are illustrated in Appendix C
In addition, we examine two more types of multilingual LLMs, including BLOOMZ (Muennighofi et al. 2023). a hyper-muliilingual LLM claiming to support 46 languages, and Chinese Llama (Cui
al. 2023). a bilingual LLM focusing on English and Chinese. We find that language-speciff neurons in BLOOMZ follow patterns similar to Mistral and Vicuna. However, in Chinese LLama, Chinese dominates as the primary language for reasoning and knowledge extraction across all languages, with notably absent language-speciff neurons. Details are shown in Appendix D
Given the certain overlap ratio of language-specife neurons from other languages with those of English, as illustrated in the first column of Figure 5 and Figure [) we conduct supplementary experiments to demonstrate that these neurons not language-ggnostcc neurons crucial for general comprehension and logical reasoning (Liang et al. 2024 Tang el al. 2024).. Instead, these overlapping neurons represent only subset of language-specifie neurons, while the language-agnostic neurons responsible for essential understanding and reasoning are those not identified as language-specific. Further elaboration and detailed results are presented in Appendix
Multilingual Workflow (MWork) of LLMs
3.1 Mwork
By classifying the hidden representations each layer LLMs into English or non-English (as shown in Figure [], we can observe the shift from non-English to English-centric, and back to non-English with the progression through the layers. This motivates us to hypothesize a three-stage multilingual workflow: understanding the original Enn-lig queries and interpreting them in
5

---

English, task-solving in English, and generating back to the original language. Nevertheless, the presence of certain non-English tokens during the English-centric task-solving stage inspires to further investigate this stage.
With the proposed PIND method, extract language-specific neurons from attention and feed-forward structures when processing vari- ous multilingual queries. We plot the average number of activated language-sper -eeegee--sificccce neurons of Mistral when processing each query in Figure 3 Notably, the number of segguugge-ppciife neu- rons decreases within the self-attention structure in the task-solving layer but remains consistent across the layers of the feed-forward structure. This decline implies a reliance on the English language for reasoning while extracting mul- tilingual knowledge to support query process- ing, which is also consistent with (Geva al.) 2021)'s interpretation of the feed-forw forward structure as key-value memories for knowledge extraction. Therefore, we further decompose the task-solving layer into two parts: reasoning in English and extracting knowledge in a multilingual context.
PFN Strotuue Attantion Strettue
Nunter of Lnpuuge Spociif Nanora
Figure 3: Number of tooguage-specifige neurons when processing multilingual queries.
Considering the above insights, we propose the MWrk hypothesis for explaining LLM's multilingual workflow: LLMs first understand user input by unifying diverse linguistic features. They then engage in the task-solving phase, employing English for reasoning and leveraging multilingual knowledge through self-attention and feed-forward structures, respectively. Finally, the models generate responses aligned with the query's 's original language.
3.2 Verification Experiment Setup
To verify Mwork, we selectively deactivate manguage-specifie neurons from each component. Then its functionality can be verified if this deactivation results in minimal impact English performance while exhibiting a notable decline in multilingual performance for the corresponding task.
Dataset To comprehensively understand how LLMs work with different abilities, we employ four kinds of tasks including MGSM (Shi et al. 2022) for reasoning task, XQuAD (Artetxe al. (020) for understanding task, X-CSQA (Lin et al.) (021) for knowledge question answering task, and XLSum (Hasan et al. 2021) for generation task. Detailed information regarding these datasets and the testing prompts can be found in Appendix F We adopt 6 languages including English (En), German (De), French (Fr), Chinese (Zh). Spanish (Es), and Russian (Ru), as their initial performance on those languages is already quite reasonable for observing the multilingual processing mechanism. For XLSum, we randomly sample 500 data points from the whole test set for each language taking into consideration its long inference time, while for other tasks, employ the entire test set. We evaluate the vanilla performance of Vicuna and Mistral on these datasets for later comparison as presented in Appendix G For reasoning, understanding, and knowledge question answering tasks, adopt accuracy as the metric. As for the generation tasks, we adopt ROUGE-L as the metric.
Deactivation Strategy We primarily consider two aspects when selecting the deactivation settings: (1) language-specific neurons versus randomly chosen neurons, and (2) the position of neurons, which encompasses four structures. Note that for fair comparison, we ensure the numbers of deactivated neurons in all settings are the same. More detailed settings are explained from Section 3.3 to Section 3.6 For the concrete numbers of different layers, we tune hyperparameters by XQuAD Chinese. Details are explained in Appendix H
Notations Tables to present the results of deactivating certain neurons, where "Under" denotes the understanding layers, "S-ATTN" and "S-FFN" correspond to the self-attention and the feed- forward structures within the task-solving layers respectively, "Gen" refers to the generation layers. The term "Random" is used to describe deactivating randomly chosen neurons, whereas "Lang-Spec" refers to the deactivation of eegguage-specifff neurons. We also present the gap between the original performance (as shown in Table II) and performance after deactivation (as shown in Table 14 to Table

---

Table 2: Results of the understanding task, where 'x' indicates that chosen neurons in the corre- sponding layer are deactivated, and ' signifies they are activated. A is defined as the difference between the reduction in performance in English, denoted as Anng. and the reduction in performance in non-English languages, denoted as Ang
Deactivating Method
Performance
Model
Under
S-ATTN
S-FEN Gen
Neuron
Eng
n-Eng
AEg
An
A
Random
57.8
53.9
+0.3
-0.1
+0.4
x
X
X
Random
57.9
54.2
+0.4
+0.3
+0.1
Vicuna
X
Lang-Spec
40.9
38.6
-1599
15.3
-066
o
J
X
Lang-Spec
57.9
52.8
-0.4
-1.1
+0.77
V
V
Lang-Spec
56.5
46.0
00.5
7.9
+7.4
V
Random
58.1
55.5
+1.0
00.2
+1.2
X
X
X
Random
57.6
55.5
+0.5
-.2
+0.7
Mistral
X
X
/
Lang-Spec
53.2
47.0
-3.9
-8.7
+4.8
V
V
X
Lang-Spec
56.4
54.6
-0.7
-1.
+0.3
Lang-Spec
56.2
48.3
00.9
7.4
+ 66.5
17p for English (AEng) and averaged non-English languages (An- respectively. A single metric A is then introduced as Anng An-Eng where a high value indicates such deactivation operation does not bring much impact to the English performance but lead to performance drop in non-English. Therefore, this provides a direct single indicator that the deactivated neurons are language-specific and hold significant responsibility in executing the corresponding task.
3.3 Verify the Understanding Stage in MWork
Deactivating Method Table the results of the understanding task following the deactivation of five distinct sets of neurons: (i) neurons randomly selected from the understanding layers; (ii) neurons randomly chosen across all layers; (iii) language-specific neurons within the task-solving layers; (iv) language-spest specific neurons in the generation layers; (v) language-specific neurons in the understanding layers. As mentioned above, in order to verify the functionality of the understanding layer (setting v), we compare with deactivating other types layers, specifically setting ii for the task-solving layer and setting iv for the generation layer. Full results are listed in Appendix !
Findings We find that by deactivating randomly sampled neurons, no matter in understanding layer all layers, the performance of LLMs in both English and non-Engen non-Enngish languages is almost unaffected compared to other deactivating methods. Note that in some cases, deactivating randomly sampled neurons may even increase the performance because irrelevant neurons removed, which also aligns with the finding from (Sharma et al., 2023). When assessing the differential impact on English and non-English language performance after the deactivation, specifically the difference calculated as A An-Eng: it is evident that the deactivation of random neurons within the under- standing layer amplifies this effect. This observation lends partial support to the hypothesized role of the understanding layer in language processing.
Furthermore, we find that deactivating language-speciff neurons in the understanding layer influences the performance in English a little while significantly decreasing the performance in non- -on-English languages. When deactivating sanguage-specifff neurons in the task-solving layer, both English and non-English languages are significantly reduced while deactivating language-specific neurons in the generation layer influences little for both English and non-English languages. Therefore, prove that the first several layers are responsible for understanding because deactivated neurons just disable LLMs the NLU task in non-English languages. Furthermore, disabling weeguugg--peciffe neurons in the solving layer shows that LLMs rely on English, as performance drops across all languages.
3.4 Verify the Reasoning Structure in MWork
Deactivating Method Table shows the result of the reasoning task, where we deactivate sets of neurons. We adhere to the previous logic of selecting dcactivation settings, with the exception that

---

Table 3: Results of the reasoning task. Disabling all language-spect neurons, except for those involved self-attention structure within the task-solving layer, greatly reduces performance.
Deactivating Method
Performance
Model
Under
S-ATIN
S-FFN
Gen
Neuron
Eng
n-Eng
Ang
An
A
X
Random
20.0
11.3
-18
+1.4
x
X
V
Random
18.4
12.2
-2.0
-1.0
-1.0
Vicuna
X
Random
12.5
-0.8
-0.7
-0.1
X
X
V
Lang-Spec
7.2
3.4
-13.2
-9.8
-3.4
Lang-Spec
18.1
-23
-4.9
+2.6
J
X
X
Lang-Spec
19.0
7.8
-1.4
5.4
+4.0
X
V
Random
40.8
23.4
-5.
2.9
2.3
X
Random
39.2
24.0
-6.8
2.3
4.5
Mistral
X
X
Random
45.2
26.8
-0.8
+0.5
1.3
Lang-Spec
38.2
-7.8
-7.9
+0.1
V
V
X
Lang-Spec
44.0
18.1
-2.0
8,2
+ 6.2
Lang-Spec
+0.2
8.0
8.2
Table 4: Results of the knowledge question answering task. The highest performance reduction difference (A) is achieved by disabling all language-specific neurons in the feed-forward structure within the task-solving layer.
Deactivating Method
Performance
Model
Under
S-ATIN
S-FFN
Gen
Neuron
Eng
n-Eng
Aun
An Ling
A
X
Random
57.5
39.5
-.3
+0.0
-03
x
Random
56.0
38.7
11.8
-0.8
-1.0
Vicuna
x
X
x
Random
57.7
39.6
-0.1
+0.1
-.2
x
V
V
Lang-Spec
33.7
30.3
24.1
-9.2
14.9
/
X
Lang-Spec
57.5
37.5
0.3
2.0
+ 1.7
/
X
V
Random
61.0
37.0
-0.3
0.5
+0.2
X
Random
60.7
-0.6
-12
+0.6
Mistral
X
x
X
X
Random
61.8
37.4
+0.1
001
+022
X
Lang-Spec
51.2
10.1
8.66
-.5
/
X
/
Lang-Spec
61.2
35.1
0.1
2.4
2.3
we do not conduct an independent experiment on deactivating neurons in the understanding layer, as its functionality has already been verified. Details listed in Appendix !
Findings We find that deactivating randomly sampled neurons in task-solving layers disables the capabilities of LLMs in reasoning to greater extent than deactivating randomly sampled neurons in all layers, which verifies the function of the task-solving layer. Furthermore, comparing three deactivating manguage-specifee neuron methods, we find that deactivating the task task-solving layer decreases performance in both English and non-English. On the contrary, when we only deactivate language-specific neurons not in the tosk-solving layer, non-English is influenced more seriously than English. Moreover, eliminating interference from the feed-forward layer achieves better results, which verifies the function of attention structure in the task-solving layer.
3.5 Verify the Knowledge Extraction Structure in MWork
Deactivating Method Table 4 shows the result of the knowledge question answering task, where we deactivate 5 sets of neurons. Similarly, we exclude the deactivation of neurons layers that have already been verified and instead concentrate the self-attention structure and feed-forward structure the task-solving layer. Details are listed in Appendix
Findings Likewise, targeted deactivation language-specific neurons within the feed-forward structure of the task-solving layer predominantly affects non non-English languages. This implies that

---

Table 5: Results of the generation task. The highest performance reduction difference (A) is achieved by disabling all segguage-speciiie neurons in the generation layer.
Deactivating Method
Performance Ang
Model
Under
S-ATIN
S-FFN
Gen
Neuron
Eng
n-Eng
AnFag
A
Random
26.8
+0.1
+0.1
+0.00
Vicuna
x
x
x
Random
13.0
26.7
-0.1
+0.0
-0.1
/
Lang-Spec
13.1
25.7
+0.0
-1.1
+1.1
x
Random
13.6
25.9
+0.1
+0.1
+0.0
Mistral
x
x
X
Random
13.6
25.7
+0.1
-022
+0.33
X
Lang-Spec
13.8
24.3
+0.3
1.5
+1.8
Table 6: Enhancement is achieved by fine- tuning Mistral-7b-vO.1 model utilizing 400 documents from each language correspond- ingly. The results averaged across four tasks. Performance on English ("En") is obtained by averaging the results from four fine-tuned models.
Method
En
Vi
Th
Ar
Sw
10H00 SOAAS XehOX
Original
41.1
32.7
25.6
21.7
15.1
Random
40.8
32.7
25.2
21.2
15.1
Figure 4: Enhancement results high-resource lan- guages, while the number average among languages.
Lang-Spec
44.6
34.9
28.5
23.4
16.9
processing multilingual queries necessitates accessing the multilingual information embedded within the relevant structures. However, disabling the self-attention structure compromises the ability to solve tasks across all languages.
3.6 Verify the Generation Structure in MWork
Deactivating Method Table shows the result of the generation task, where we deactivate sets of neurons. Since all previous layers have been verified, we solely deactivate neurons in the generation layer and compare them with randomly selected neurons. Details are listed in Appendix
Findings Similar other tasks, the disabling of Language-specifgg neurons within the generation layer diminishes their capacity to generate content in the respective languages. By selectively deactivating neurons that are associated with English, multilingual generation abilities. However, demonstrated in Table the complete deactivation of all -ppuufe--ppecific neurons results in the total loss of the LL.Ms' multilingual generation capabilities.
do not completely eliminate the models'
Multilingual Enhancement with MWork
have verified MWork for explaining the multilingual working mechanism of LLMs in the above section via deactivating certain neurons. While opposite to employing deactivation, can also enhance their multilingual ability, especially the understanding and generating ability, by fine-tuning these language specifcc neurons. With language-specific neurons comprising only around 0.1% of all parameters, the need for training documents to improve multilingual capabilities can significantly reduced to just few hundred. Additionally, fine-tuning only the language-specifer neurons for particular language does not impact performance in other languages, allowing us to enhance specific languages while preserving performance in others.
Mwork helps with enhancing multilingual ability by hundreds of documents. We employ Mistral-7b-vO.. / for enhancement to eliminate the interference of instruction fine-tuning, and select causal language modeling as training task. We create a dataset comprising {100, 200, 400, 800}

---

randomly selected documents for each language, extracted from the Wikipedia corpus (Foundation). Figure shows the results of enhancement on high-resource languages (De, Fr. Zh, Es, Ru). The numbers represent the sizes of the training corpus when fine-tuning language-speccfic neurons, while "Random" represents the fine-tuning of an equivalent number of randomly chosen neurons using a corpus of 400. Our findings reveal that fine-tuning with few hundred documents yields significant performance improvements on multilingual tasks: 3.4% on MGSM, 4.4% on XQuAD, 4.3% on X-CSQA, and 2.3% XLSum. Moreover, English performance is enhanced by an average of 3.7% across all tasks. These results further confirm the effectiveness of MWork in interpreting structure functionality for LLM's multilingual query handling, offering precise and independent methods for multilingual enhancement. When fine-tuning with 800 documents, the performance deteriorates compared to using 400 documents. This drop can be attributed to the incorporation of additional knowledge, which disrupts the original knowledge distribution and leads to overfitting of the model to Wikipedia. This can be addressed by mixing data from more sources such textbooks websites.
In addition, we verify the effectiveness of such enhancement method on low-resource languages, given that low-resource performance is relatively low with the original model. We select four languages including Vietnamese (Vi), Thai (Th), Arabic (Ar), and Swahili (Sw), covering languages with both latin and non-latin scripts and having corresponding testing set in our considered benchmarks. The model was then evaluated on four benchmarks, and the result shown in Table G the average scores among tasks. It is evident that the fine-tuning method using language-speciffs neurons enhances the model's 's multilingual performance in low-resource languages by an average of 2.2%. Notably, the improvement of 3.5% in English performance is observed even without English training corpus, indicating the effectiveness of the distinct language responsibilities assigned to neurons.
5 Related Work
In the era of LLMs, numerous studies have been conducted to develop multilingual bench- marks (Zhang et al. (023a), enhance multilingual performance without parameter adjustments through translation (Liang et al. 2023 Huang et al. (203)). aligning representations (Nguyen et al..
2023a Salesky et al. 2023), prompting (Li et al. [223bb] Tanwar et al. 2023). Furthermore, certain works focus on improving multilingual abilities for single task via cross-lingual transfer (Kim) et al. 2017| Lin et al. 2019 Pfeiffer et al., 2020; Zhao et al. 2024b), while others aim to enhance multilingual proficiency by continuous training in one language to obtain mono-lingual LLMs (Cui et al. (023), in multiple domain languages to obtain domain-lingual LLMs (Nguyen et al. (023b)) Additionally, some works achieve multilingual LLMS by training from scratch (Muennighoff) et al.. 2023). However, these studies are limited to specific task types or require substantial training corpora due to a lack of comprehensive understanding of the multilingual mechanisms of LLMs.
Conventional interpretability research investigates the significance of input features with their cor- responding outputs (Vigl 2019 Hewitt and Liang 2019; Qiu et al., 2020). In the era of L.MMs, one brunch of work includes efforts to understand knowledge storage, with (Geva et al. (021)) initiating the study of the feed-forward layer as a knowledge base. Subsequent work has furthered this by altering neuron values (Dai et al. (022), mapping embeddings to words (Geva et al. 2022).. modifying inputs to recover embeddings (Meng et al. 2022), and analyzing attention heads (Li et al. (023a).
Another line of research centers on the seli felf-attenton layer, examining its connection to reasoning
capability (Hou et al.) [0233 Stolfo et al. 2023 Friedman et al. 2023) by contrasting the reasoning tree based on attention weights.
6 Conclusion
In this work, we examine how LLMS handle multilingualism. The proposed multilingual workflow (MWork) suggests that LLMs initially understand queries by converting multilingual inputs into English, reason in English in intermediate layers while incorporating multilingual knowledge, and generate responses aligned with the original language in the final layers. The validity of MWork is verified using Parallel Language-specific Neuron Detection (PLND), which identifies activated neurons for different languages without labeled data. By detecting language-specific neurons and fine-tuning them with a small training corpus, MWork enhances multilingual abilities in specific languages without compromising others, resulting in significant improvements across tasks.
10

---

Acknowledgement
This work was substantially supported by DAMO Academy through DAMO Academy Research Intern Program. This research is partially supported by the National Research Foundation Singapore and the Singapore
under the AI Singapore Programme (AISG Award No: AISG2-TCC-203-010--560 Ministry of Education Academic Research Fund Tier (Award No: TI 251 251RES2207).
References
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisticy, pages 4623-4631
Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. 2020. Language id in the wild: Unexpected challenges on the path to thousand-langunn web text corpus. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6588-6608.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna An spen-source chatbot impressing gpt-4 with 90%* chatgpt quality
Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08/1 081777.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493-8502.
Wikimedia Foundation. Wikimedia downloads
Jonathan Frankle and Michael Carbin. 2018. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations.
Dan Friedman, Andrew Lampinen, Lucas Dixon. Dangi Chen, and Asma Ghandeharioun. 2023. Interpretability illusions in the generalization of simplified models)
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. Transformer feed-forw forward layers build predictions by promoting concepts the vocabulary space. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30-45.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer ferwwrd layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484-5495.
Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M Sohel Rahman, and Rifat Shahriyar. 2021. XI-sum: Large-scale multilingual abstractive summarization for 44 languages. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693-4703.
John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. arXiv
preprint arXiv: 00366.
Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, and Mrinmaya Sachan. 2023. Towards a mechanistic interpretation of multi-step step reason ing capabilities of language models In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4902 4919, Singapore. Association for Computational Linguistics.
Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in L.Mss: Improving multilingual capability by cross-lingual thought prompting In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12365-12394, Singapore. Association for Computational Linguistics.
11

---

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b, arXiv preprint 2rXiv:2310.0682.
Joo-Kyung Kim, Young-Bun Bum Kim, Ruhi Sarikaya, and Eric Losser-Lussier. 2017. Cross-lingual transfer learning for pos tagging without cross-lingual resources. In Proceedings of the 2017 conference on empirical methods natural language processing, pages 2832-2838.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023a. Inference-time intervention: Eliciting truthful answers from language model. arXiv preprint arXiv:2306.0333/ 03344.
Shuang Li, Xuming Hu, Aiwei Liu, Yawen Yang, Fukun Ma, Philip S Yu, and Lijie Wen. 2023b. Enhancing cross-lingual natural language inference by soft prompting with multilingual verbalizer. arXiv preprint arXiv:2305. /2761.
Yaobo Liang, Quanzhi Zhu, Junhe Zhao, and Nan Duan. 2023. Machine-created universal language for cross-lingual transfer. arXiv preprint arXiv:2305. 13071.
Yunlong Liang, Fandong Meng, Songming Zhang, Yufeng Chen, Jinan Xu, Jie Zhou, et al. 2024. Multilingual knowledge editing with language-agnostic factual neurons. arXiv preprint arXiv:2406. 16416.
Jindřích Libovický, Rudolf Rosa, and Alexander Fraser. 2020. On the language neutrality of pre- trained multilingual representations. In Findings of the Association for Computational Linguisties: EMNLP 2020, pages 1663-1674.
Bill Yuchen Lin, Seyeon Lee, Xiaoyang Qiao, and Xiang Ren. 2021. Common sense beyond english: Evaluating and improving multilingual language models for commonsense reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1274-1287.
Yu-Hsiang Lin, Chian-Yi Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, et al. 2019. Choosing transfer languages for cross- lingual learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, volume 57.
Weize Liu, Yinlong Xu, Hongxia Xu, Jintai Chen, Xuming Hu, and Jian Wu. 2024. Unraveling babel: Exploring multilingual activation patterns within large language models. arXiv preprint arXiv:2402.16367.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 17559-1117377.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, et al. 2023. Crosslingual generalization through multitask finetuning. In The 61st Annual Meeting Of The Association For Computational Linguistics.
Hoang H Nguyen, Chenwei Zhang, Tao Zhang, Eugene Rohrbaugh, and Philip S Yu. 2023a. Enhanc- ing cross-lingual transfer via phonemic transcription integration. arXiv preprint arXiv:2307.0436/
Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, et al. 2023b. Seallms-large language models for southeast asia. arXiv preprint arXiv:2112.00738.
OpenAl. 2023. Gpt-4 technical report
Jonas Pfeiffer, Ivan Vulić, Tryna Gurevych, and Sebastian Ruder. 2020. Mad-x: An adapter-based framework for multi-task cross-lingual transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7674-7673
12

---

Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre- trained models for natural language processing: A survey. Science China Technological Sciences, 63(10))1872-1897.
Elizabeth Salesky, Neha Verma, Philipp Koehn, and Matt Post. 2023. Pixel representations for multilingual translation and data-efficient cross-lingual transfer. arXiv preprint arXiv:2305.
Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. 2023. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arX1v:2312.13558.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung. Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations.
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. 2023. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7035-7052, Singapore. Association for Computational Linguistics.
Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-specifte neurons: The key multilingual capabilities in large language models. arXiv preprint arXiv:2402. 16438.
Mare Tanti, Lonneke van der Plas, Claudia Borg, and Albert Gatt. 2021. On the language-specificity of multilingual bert and the impact of fine-tuning. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 214-227.
Eshaan Tanwar, Manish Borthakur, Subhabrata Dutta, and Tanmoy Chakraborty. 2023. Multilingual Ilms are better cross-lingual in-context learners with alignment. arXiv preprini 3xXv:23055.05400.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: family of highly capable multimodal models. arXiv preprint 23XX::3312.1805.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Jesse Vig. 2019. A multiscale visualization of attention in the transformer model
Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023a. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models CoRR, abs/2306.05179.
Zhihao Zhang, Jun Zhao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Unveiling linguistic regions in large language models. arXiv preprint arXiv:2402. J4700.
Zhong Zhang, Bang Liu, and Junming Shao. 2023b. Fine-tuning happens in tiny subspaces: Exploring intrinsic task specific subspaces of pre-trained language models. In The 61 Annual Meeting Of The Association For Computational Linguistics.
Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024a. Llama beyond english: An empirical study on language capability transfer
Yiran Zhao, Wenxuan Zhang, Huiming Wang, Kenji Kawaguchi, and Lidong Bing. 2024b. Adamergex: Cross-lingual transfer with large language models via adaptive adapter merging. arXiv preprint a0Xiv:2402. 18913.
13

---

A English and Non-English Tokens
We employ c1d3 package to detect the language of each token in the embeddings of each layer, which is a language detection library based on the Compact Language Detector 3 model developed by Google. Furthermore, the detection result is reliable, i.e., ald..get_ lea_gggaggggeeeeeoo language(token). reliable Truc, we adopt the detection results, otherwise the token is categorized as a non-word.
B Multilingual Corpus
Note that our selection criterion for the number of documents based on achieving substantial coverage of each language's 's vocabulary, ensuring that the selected contexts provide representative sample of the language, as shown Table 7
Table 7: Corpus details across languages are tailored to encompass the majority of each language's vocabulary, where "corpus size" indicates the number of contexts selected, "corpus vocab" represents the vocabulary coverage within the selected contexts, "vocab size" refers to the number of vocabularies of that language.
Language
En
De
Fr
Zh
Es
Ru
Corpus Size
180k
30k
50k
20k
20k
20k
Corpus Vocab
249k
154k
134k
198k
90k
144k
Vocab Size
273k
148k
135k
329k
93k
150k
C Interrelation of Language-Specifie Neurons Across Languages
Using neurons identified by PLND, we investigate the relationships between languages via the degree of overlap among their language-specific neurons, defined as
W N W
overlap(z, v)
(1I)
where Nanngugge represents the set of detected language-specee specific neurons. Figure shows the neuron overlapping ratio overlap(2; x, y) any two languages different structures of two models.
Mixtral Self-Attertion Layers
Mixtral Feed-Forward Layers Langunge
Vicuna Self Attentitn Language
Vicuna feed Forward Language
Language
3.0
1.0
0.0
Language
25
0.7
377
03
56
66
05
(a)
(b)
Figure Overlapping ratio of language-specific neurons self-attention and feed-forward structures.
We observe that in both Mistral and Vicuna, the intersection with English from other languages is relatively limited (i.e., the first row of each figure), suggesting that English possesses predominant number of language-specific neurons. Additionally, there a pronounced tendency for languages belonging to the same family to demonstrate higher degree overlap with each other, such as Spanish, French, and English.
D Analysis on Different Multilingual LLMs
We further examine two more types of multilingual LLMs, including BLOOMZ MMunnighoff et al.
2023 hyper-multilnggaal LLLM claiming to support 46 languages, and Chinese Llama (Cui et bilingual LLM focusing English and Chinese.
2023
14

---

Hyper-Multilinguall LLMs Figure illustrates the degree of neuron overlap among languages within both the self-attention and feed-torward structures of BLOOMZ. In contrast to the findings shown in Figure 5 there is a marked reduction in overlap, indicating that individual languages maintain a higher degree of independence and do not extensively share neurons with one another.
self-Attention Layers
BLOOMZ Feed-Forward Layers Language >
Language
Layerwise Language bution
Langu
08
07
Figure 6: Overlapping ratio of segguage-specffie neurons in BLOOMZ
Ratio of languages among layers in Chinese Llama given non-English instructions.
Figure 7:
Billinuual LLMs We employ Chinese Llama (Cui et al. 2023). which extends existing vocabulary and incorporate secondary pre-training using Chinese data and fine-tune the model with Chinese instruction datasets. However, this intensive training can lead to a degradation in performance for languages other than Chinese. As depicted in Figure Chinese predominates as the primary language for reasoning processing and knowledge extraction across all languages. Consequently, the absence of language-specific neurons results in the transformation of it into Chinese-centric LLM.
E Language-Agnostic Neurons
We initially implement radical deactivation approach, wherein we specifically deactivate overlapping elements between each language and English. These elements precisely correspond to the intersecting neurons in the first column of Figure 5 Presented below are the comprehensive findings pertaining to Mistral. Our evaluation is centered around the reasoning task, which recognized as the most indicative and challenging assessment for the model. We compare under the optimal "deactivating" method, which involves deactivating all mooguage-specifgg neurons except those S-ATTN.
Table 8; Performance of deactivating language-speciff neurons without overlapped between English.
Language
Eng
non-Eng
Argg
Amn-iag
A f
All language-specific neurons
46.2
18.3
0.2
8.0
18.2
LSN without overlapped between English
45.8
20.2
0.2
6.1
+5.9
As evident by Table g the performance of English remains stable, contrasting sharply with the significant decline in the performance of multilingual. Removing overlapped neurons, as opposed to deactivating all language-specte specific neurons, leads to a less pronounced drop, yet the impact remains noteworthy. This demonstrates that overlapped neurons are not language-agnostt utilized for general comprehension and logical reasoning. Otherwise, the fundamental reasoning capacity and performance in multilingual contexts would remain unaffected. In addition, we retained the language-specifie neurons that overlapped in all languages, meaning that removed them from the language-specific neurons to be deactivated. Detailed results follow.
they are not
Table Performance of deactivating singuage-specifi neurons without all languages overlapped.
Language
Eng
non-Eng
Ars
A f
All language-specifs neurons
46.2
18.3
+0.2
8.00
-8.2
LSN without all languages overlapped
45.6
18.7
-0.4
-7.6
1-72
The neurons that overlap across all languages only account for 0.02% of the total number of neurons. From the results in Table we can see that the performance is almost the same as deactivating all speeiie neurons. This further proves that these neurons are not language-agnostic neurons, but only subset of tanguage-specifii neurons.
15

---

Table 10: Zero-shot prompts for each dataset.
Task
Zero-Shot Prompt
MGSM
Let's think step by step. Question: {question)
XQuAD
[context) Question: (question)
XLSum
Summarize the context in one sentence. Tite: [title) Context: [article)
)
X-CSQA
Question: {question)
Table 11: Assessing the baseline performance of Vicuna and Mistral across four representative multilingual tasks in selected languages, where Avg. is calculated among non non-English languages.
Model
Task
En
De
Zh
Es
Ru
Avg.
XQuAD
57.5
50.3
55.7
55.7
53.9
MGSM
20.4
14.8
12.8
13.2
10.0
Vicuna
X-CSQA
57.8
43.8
43.2
44.3
26.0
XI.Sum
13.1
14.2
61.1
10.4
20.8
26.6
XQuAD
57.1
48.5
64.3
54.1
55.6
MGSM
46.0
21.2
26.0
31.6
31.2
21.6
Mistral
X-CSQA
61.7
40.0
47.1
45.7
37.5
XLSum
13.5
15.2
56.4
10.6
25.8
F
Prompts
Table 10 shows the zero-shot prompts for each dataset. Note that when conducting tests in other languages, prompts translated into the respective languages.
G Original Performance
Table 11 shows the original performance of Vanilla and Mistral on four tasks.
H Hyper-parameters
We adopt the performance XQuAD in Chinese the validation set all languages and all tasks. Specifically, Table shows the result on Vicuna when deactivating Language-speciff neurons in the understanding layer (Dz) and generation layer (Dg), where N1 is the number of understanding layers and N₂ is the number of generation layer. We find that when setting N₁ = 8 and N₂ performance in English is influenced the least while performance in Chinese decreases the most. As for Mistral, the number is N1 and N₂ 3.
Table 12: XQuAD with Chinese Vicuna.
Table 13: XQuAD with Chinese Mistral.
Du ACC
De ACC
Du ACC
De ACC
Method
Method
No
En Vanilla
57.5
En Vanilla
57.1
Zh- Vanilla
55.5
Zh- Vanilla
64.3
On-Deact
57.7 (t 0.2)
34.7 (L 2.8)
En- Deact
53.3 (L 3.8)
55.8 1.3)
8
4
Zh-D-Deact
44.9 (L 10.6)
34.6 (1 0.9)
Zh- DeacD
52.6 11.7)
62.9 (1 1.4)
En- -eact
58.6 (f 1.1)
37.7 (1 0.2) 54.5 (L 1.0)
En- Co-Deact
56.8 (] 0.3) 54.9 (} 9.4)
56.3 (J 0.8) 62.7 1.6)
3
6
55.1 (L 0.4)
Zh- Deact
En- -eact
57.3 [L 0.2)
58.4 (1 0.9) 54.1 () 1.4)
En- ReaDe
57.6 (1) (15) 61.8 () 2.5)
55.7 (L 1.4) 63.8 () 0.5)
2
ReaDe
53.9 (} 1.6)
Zh- Denct
16

---

Detailed Experiment Results
1.1 Detailed Experiment Settings
Reasoning Task Deactivation methods: (i) randomly sampled neurons in the attention structure of task-solving layer. (ii) randomly sampled neurons in the task-solving layer. (iii) randomly sampled neurons in all layers. (iv) language-specifie neurons in the task-solving layer. (v) language-specifie neurons in the understanding layer and generation layer. (vi) language- specific neurons not in the attention structure of task-solving layers.
Knowledge Question Answering Task Deactivation methods: randomly sampled neurons the feed-forward structure of sask-solving layers. (ii) randomly sampled neurons in the task-solving layer. (iii) randomly sampled neurons all layers. (iv) language-specife neurons the attention structure of tosk-nolving layers. (v) language-specifie neurons in the ferw-forward structure of task-solving layers.
Deactivation methods: (i) randomly sampled neurons in the generating layers. (ii) randomly sampled neurons in all layers. (iv) language-specific neurons the generating layers.
Generation Task
1.2 Detailed Result
Due to the limited space, we employ more concise notation. We denote deactivating neurons in the self-attention layer of the i-th layer D( while deactivating neurons in the feed-forward layer of the i-th layer is denoted as D We denote 2/ {1, Ni} as the set of layers that take charge
of understanding as shown in Figure 2 Similarly, we denote S {NN + 1,... N2 as the
of layers that take charge of task solving and G = {N2 + 1, ... }2 as the set of layers that take charge of generation Furthermore, D(A) represents deactivating neurons self-attention layers of U. Similarly, we introduce ( D&A) D P( and (
Table 14: Understanding task.
German
Chinese
Spanish
Method
En-D
De-D
ABed
DDED
En-D
Zh-D
ABaD
AziD
En-D
Es-D
AE
AB-
D
57.8
19.7
+0.3
-0.6
57.8
55.8
+0.3
+0.1
57.8
56.1
+0.3
+0.4
Vicuna
57.9
50.8
+0.4
+0.55
57.9
55.8
+0.4
+0.1
57.9
55.9
+0.4
+0.2
55.7
40.7
2.0
9.6
57.7
+2.0
10.8
56.1
52.4
1.4
3.2
Ds
48.3
41.7
-722
-8.6
45.0
45.4
12.5
10.3
29.5
28.6
28.0
27.1
D₂
57.5
50.1
0.0
-0.2
58.4
54.1
+0.9
-1.6
57.7
54.1
+0.2
-16
D
58.1
48.2
+1.0
58.1
+1.0
-0.4
58.1
54.3
+1.00
+0.2
Mistral
D
57.6
48.3
+0.5
-0.3
57.6
63.6
+0.5
-0.7
57.6
54.5
+0.5
+0.4
Du
56.5
12.4
0.6
6.2
56.8
54.9
0.3
9.1
55.4
47.5
-177
6.6
Dg
54.3
43.2
-2.8
-5.4
54.9
52.9
2.2
- 11.4
50.3
44.9
-6.8
9.2
56.7
17.9
56.3
62.7
1.6
56.2
53.2
-0.
0.8
VViuuna-bb-v..5 and Mistral-76 -b- both have 32 layers.
17

---

Table 13: Reasoning task.
Method
German Ars
French
Chinese Smsp
Spanish Aes
Russian
En-D
De-D
Aesa
Ba-D
Fe-D
Arn
EEn-D
Zh-D
En-D
Es-D
Asi
Ru-D
Seso
Axup
D
20.0
1244
-0.4
-24
20.0
13.6
-0.4
-1.2
20.0
132
-04
+0.4
20.0
12.4
-044
-0.8
20.
4.8
-044
-3.2
Viuna
Dg
18.4
12.4
-2.0
-24
184
14.
-20
-0.8
18.4
14.1
-2.0
+1.6
18.4
15.2
-20
+200
18.4
1.8
-200
-5,2
+2.0
12.4
Ds
3.6
2.0
-16.8
-1288
8.4
3.2
-12.0
-11.6
18
1.0
-15.6
-3.8
8.5
1.0
-11.6
-9.2
10.4
4.0
-10.0
-6.0
Dute
Dgca
16.8
44
-3.6
10:4
19.6
8.8
-0.8
-1.4
21.6
9.6
+1.2
3.2
19.6
10.4
-0.8
-288
17.2
5.6
-3.2
-4.4
D
-5.2
-5.2
-7.6
40.8
-5.2
-20
40.8
-5.2
Mistral
39.2
20.0
-6.8
-1.
39.2
25.2
-68
-0.8
39.2
25.6
-688
-60
39.2
29.6
-68
-16
30.2
19.6
-688
-2.0
15.2
21.0
-0.8
+28
15.2 27.6
-0.8
+1.6
45.2
312
-0.8
-0.1
13.2
30.1
-0.8
-0.8
45.2
20.5
-0.8
-0.8
Ds
-92
19.6
40.4
Duse
12.4
9.2
-3.6
-120
41.2
21.6
-1.8
-14
464
19.6
+0.4
44.0
28.0
-2.0
-3.2
46.0
12.0
+0.0
-9.6
Dain
43.6
9.6
-2.4
11.0
11.8
19.2
1.2
6688
46.4
18.8
+0.4
12.8
47.6
27.6
+1.0
3.6
48.4
16.1
+2.4
-3.2
Table 16: Knowledge Question Answering task.
Method
German De-D
Chinese
Spanish
Russian
En-D
Aeso
Agso
En-D
Fr-D
Asso
Ans
[in-
Zh-D
Arsa
Arso
Er-D
Ex-D
An-a
Arso
Ee
Ru-D
Aeso
Arso
D
57.5
438
-0.3
+0.0
57.5
40.3
-0.3
+0.2
57.5
43.2
-0.3
+0.0
57.5
41.6
-0.3
+0.3
57.5
25.5
-0.3
-0.5
Vicuna
DE
36.0
14.0
-1.8
+0.2
56.0
36.6
-1.8
-1.5
56.0
43.1
-1.8
+0.2
56.0
13.5
-1.8
-0.8
56.0
21.0
-1.8
-2.0
DS
436
+0.4
-0.1
Dgra
31.8
43.4
-23.0
-0.1
32.6
31.1
-25.2
12.77
32.6
28.9
-25.2
11.3
20.4
25.0
-37.1
19.3
48.3
-9.5
-3.1
Pair
57.8
41.5
10.0
2.5
57.2
37.8
0.6
6.0
56.9
39.6
0.9
3.6
57.6
43.0
0.2
1.3
57.8
25.6
10.00
0.4
D
61.0
40.2
00.7
+0.2
61.0
40.1
0.7
0.3
61.0
46.7
-0.7
0.4
61.0
45.2
-0.7
0.5
61.0
12.7
-0.7
-1.4
Mistral
D3
60.7
40.4
1.0
10.4
60.7
36.9
1.0
3.5
60.7
46.9
1.0
0.3
60.7
46.3
1.0
10.7
80.7
11.1
1.0
3.0
PM
40.1
40.1
40.1
61.8
40.7
40.1
40..3
40.1
40.1
40.1
40.1
40.0
50.4
32.3
11.3
7.7
55.3
27.4
6.4
13.0
54.7
7.00
4.7
44.5
34.1
17.2
11.6
51.1
8.3
10.6
5.8
DgiF)
61.5
38.1
0.2
1.9
61.2
38.1
0.5
2.3
61.3
43.5
0.4
3.6
61.0
43.9
0.7
1.8
60.8
11.8
0.4
2.3
Table 17: Generation task.
French
Chinese
Spanish
Russian
Method
En-D
Fr-D
Amo
Ans
En-D
Zh-D
Aisu
Amsa
En-D
Es-D
Aws
En-D
Ru-D
Amo
Am.
Vicuma
13.2
+0.1
+0.0
13.2
+0.5
13.2
+0.0
13.2
+0.0
D
13.0
14.1
-0.1
-0.1
13.0
61.6
-0.1
+0.5
13.0
10.4
-0.1
40.0
13.0
20.8
+0.0
13.0
13.8
00.1
0.4
13.1
59.5
+0.0
-1.6
13.0
-0.1
-1.3
13.1
20.3
+0.0
Mistral
D2
13.6
15.2
+0.1
+0.0
13.6
56.7
+0.1
+0.3
13.6
10.3
+0.1
-0.3
13.6
21.2
+0.1
+0.2
D
13.6
15.4
+0.1
+0.2
13.6
55.9
+0.1
0.5
13.6
10.2
+0.1
0.4
13.6
21.1
+0.1
+0.1
Dg
14.3
14.2
+-0.8
13.6
52.8
0.1
3.6
13.7
10.2
40.2
0.4
13.5
20.2
0.1
0.8
18

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In the abstract and Section | (Introduction)
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: After Section (Conclusion)
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Section and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
Our code is available at wtt.:/uhuu..co/oo-o///// SC/
http:iiiiiuaa_aaaallllaaaa We will provide sufficient details to fully reproduce results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
public/ getd///u/es////eee/e Caoe/oooeuSunn swwisnsssooooooi cesionoooiyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /nips.cc/putle wttpcc//pppllgggguce.ocoooeooooe gttps//wuiie/eeee//eeeeeee/ees Sodeeee as as onsoo P icy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section 2 2 and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: There is no randomness in our experiments
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section and
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This paper adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In Section and
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers thcode com dataaetteett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "Vj1AXNqBVf", "reviewer": "Reviewer_2b8J", "summary": "This paper examines how LLMs handle multilingualism. The author proposes a hypothetic workflow (MWork), which suggests that LLMs understand the multilingual query, think in English, and than generate results in the input language. A neuron detection method is proposed to detect language specific neurons. By deactivation some of the neurons, the authors validates the above workflow, and show that the  ability for a specific language could be improved by finetuning only language specific neurons.", "questions": "See the weakness part.", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents an interesting  hypothesized workflow and validate it with neuron analysis.\n\nThe analysis with neuron deactivation is sound enough.", "weaknesses": "The analysis of this paper is based on the understanding of  layers. However, it might be possible that different tasks or even different instances may have different splitting of layers for different stages in the workflow. How would this affect the analysis?\n\nIt is still strange to know that 400 documents could improve the language ability of a low-resource languages, especially when 800 documents will not be more helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviwer 2b8J,\n\n\n\nThank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:\n\n\n\n> Concern #1: different instances may have different splitting of layers\n\nWe appreciate your concern regarding varying splitting settings for different tasks and different instances. Yes, the boundaries between layers are not very clear and vary across models, languages and tasks. In our paper, for each model we consistently apply the same splitting approach across various languages and tasks to demonstrate the generalizability of our proposed framework. It is reasonable to speculate that more precise and finely-tuned layer splitting could improve the performance of deactivation experiments (Table 2 to Table 5) and the enhancement experiments in Section 4.\n\n> Concern #2: why 400 documents can improve language ability\n\nWe acknowledge your concern regarding why only 400 documents can yield satisfactory performance. When considering that language-specific neurons account for just 0.1% of all parameters, this size of training corpus seems reasonable. Other studies that boost specific low-resource languages [1] [2] necessitate nearly millions of documents, comparable with scaling the number of language-specific neurons proportionally.\n\n\n\nWe conduct experiments on tuning language-specific neurons by more documents from Wikipedia and here are detailed results.\n\n|      | **Original** | **0.4k** | **1k** | **8k** | **64k** | **128k** |\n| ---- | ------------ | -------- | ------ | ------ | ------- | -------- |\n| Vi   | 32.7         | **34.9** | 34.3   | 28.3   | 26.4    | 22.1     |\n| Th   | 25.6         | **28.5** | 27.4   | 23.7   | 20.8    | 15.9     |\n| Ar   | 21.7         | **23.4** | 22.8   | 19.2   | 16.2    | 8.3      |\n| Sw   | 15.1         | **16.9** | 15.2   | 12.4   | 7.9     | 5.5      |\n\nWe find that adding more documents significantly reduces performance. This phenomenon is likely attributed to overfitting the model to Wikipedia data, causing conflicts with patterns in the newly introduced data.\n\nTo further validate our assumption and explore the effectiveness of language-specific tuning, we conduct additional experiments by assembling the training set from a wider array of resources such as textbook, website and journalistic materials. Employing the same settings as in Table 6, we present detailed results below. While 0.4k documents enhance multilingual proficiency, 1k and 8k documents disrupt this ability as new patterns conflict with those learned during pre-training. However, expanding the training corpus to 64k and 128k documents reconstructs multilingual proficiency and boosts overall performance.\n\nOur findings reveal that enhancing model's multilingual capability through neuron-specific continue-training undergoes a process of disruption and reconstruction. However, adding more data from Wikipedia fails to reconstruct the multilingual ability due to overfitting. Notably, the training data comprising 128k documents remains significantly lower—by a factor of 10 to 100—than the quantities required in previous studies [1] and [2].\n\n|      | **Original** | **0.4k** | **1k** | **8k** | **64k** | **128k** |\n| ---- | ------------ | -------- | ------ | ------ | ------- | -------- |\n| Vi   | 32.7         | 34.3     | 31.8   | 31.5   | 33.7    | **35.2** |\n| Th   | 25.6         | 27.8     | 25.4   | 25.8   | 26.5    | **30.7** |\n| Ar   | 21.7         | 22.6     | 21.3   | 20.1   | 22.4    | **24.9** |\n| Sw   | 15.1         | 16.4     | 15.2   | 16.0   | 16.6    | **17.3** |\n\n\n\n[1] Sailor: Open Language Models for South-East Asia, Arxiv 2024\n\n[2] SeaLLMs - Large Language Models for Southeast Asia, ACL 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D1eHPz1ebJ", "reviewer": "Reviewer_Nru9", "summary": "This paper proposes MWork, a workflow to study how large language models handle multilingual inputs. The main idea is to detect English and non-English neurons by probing their value differences and performance gaps. Based on the results, they argue that there are three steps in the workflow: understanding, task solving, and generating. They also use experiments to support their hypothesis.", "questions": "See above.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- They propose an interesting way to understand the mechanism of large language models.", "weaknesses": "- The authors classify neurons based on English and non-English, which is not very convincing to me. I believe that, beyond these two categories, there should be some neurons that are helpful for **all** languages. These neurons can be used for general understanding and logical reasoning. However, this paper ignores this aspect. See more explanations below.\n  -  For neuron detection, when computing the impact, I am curious why you do not remove the overlapping neurons. In this way, you can detect the real language-specific ones. Specifically, in Appendix C, although the authors argue that the intersection with English from other languages is relatively limited based on rows, if we look at the columns, it seems that all the language-specific neurons are covered by English ones, suggesting that they are actually not language-specific. It is very likely that these neurons are essential for general understanding and are language-agnostic.\n  - It is not clear how English and non-English tokens are detected in Figure 1. Do you determine this by applying a decoder to decode from the hidden representations?\n  - The authors argue that large language models will use English tokens for task solving with multilingual inputs based on the interpretation of Figure 1. However, Figure 1 also shows that models will use non-English tokens for task solving with English. I don’t think this is reasonable. From my perspective, the way to classify tokens or neurons is not accurate, and the ignorance of language-agnostic neurons makes this figure unreasonable.\n  - From the experimental results, deactivating language-specific neurons sometimes causes a performance drop for both English and non-English, suggesting the existence of general neurons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Nru9,\n\n\nWe appreciate the time and effort you have put into providing valuable feedback. However, we respectfully believe there is a serious misunderstanding regarding our work. We would appreciate the opportunity to clarify a few points and address your concerns as follows.\n\n\n> Misunderstanding: Exist language-agnostic neurons.\n\n\nThe proposed \"language-specific neuron\" and the framework do not contradict the presence of language-agnostic neurons. In reality, language-specific neurons depend on these language-agnostic neurons to complete latent comprehensive understanding and logical reasoning. Language-agnostic neurons reside in neurons not identified as language-specific. Analogously to humans, when an English native speaker encounters content in other languages, the solution lies in their overall understanding and problem-solving skills, surpassing any linguistic barrier.\n\nIt is crucial to note that language-specific neurons constitute merely 0.1% of all neurons for each non-English language (nearly 0.3% for English). Without the presence of language-agnostic neurons, compressing the model by a factor of 1000 and retaining only language-specific neurons will maintain performance. However, it is obviously impossible. These definitions and observations related to language-agnostic neurons also align with those from concurrent works [1, 2].\n\nAlthough they account for the majority of neurons, language-agnostic neurons are not the focus of this paper. We mainly investigate how LLMs leverage the English language's capabilities for handling multilingualism. Our findings indicate that LLMs predominantly utilize neurons as language-agnostic entities for comprehension and reasoning, reserving only a select few to explicitly express their understanding and logic in diverse linguistic contexts comprehension and reasoning across various languages. This is exactly the motivation of our language-specific neuron enhancement method.\n\nNevertheless, we acknowledge your concern and will include the discussion regarding language-agnostic neurons in the final version.\n\n> Question #1: Why do not remove overlapping of language-specific neurons.\n\nAs explained earlier, what we discovered are actually language-specific neurons, constituting only a very small fraction of all neurons. This specific region is solely dedicated to language processing, distinct from general comprehension or reasoning abilities. This discovery is also consistent with concurrent research [3]. Therefore, it does not make sense to remove overlapped neurons as they are already specialized for language. \n\nFurthermore, neurons that overlap across all languages account for only 0.02% of all parameters. It is implausible that LLMs depend on this minute fraction of neurons for intricate comprehension and reasoning. Instead, these shared neurons between two or more languages merely denote multifunctionality, such as in an individual proficient in both French and Spanish concurrently, rather than being language-agnostic.\n\n\n\n> Question #2: How tokens are decoded in Figure 1?\n\nWe employ decoder of the last layer to decode hidden representations, a method also employed in a concurrent work [4] that shares similar findings with ours, also interpreting the phenomenon as \"the model thinks in English.\"\n\n\n\n> Question #3: Why Figure 1 contains non-English tokens?\n\nIt should certainly contain non-English tokens; at the very least, non-English tokens are necessary to fill in entities although the model thinks in English. Additionally, we claim that the feed-forward structure of the task-solving layer is used to extract multilingual knowledge to obtain factual content. These extracted multilingual tokens necessitate neurons for processing and consequently manifest within the hidden representation.\n\n\n\n> Question #4: Why deactivating language-specific neurons influence English performance?\n\nAs analyzed in the paper from Section 3.3 to Section 3.6, performance drop in English results from disabling the task-solving layer. If neurons are deactivated appropriately, English performance remains unaffected, as highlighted in Tables 2 to 6.\n\n\n\n\n\n\n\n[1] Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models, ACL 2024\n\n[2] Multilingual Knowledge Editing with Language-Agnostic Factual Neurons, Arxiv 2024\n\n[3] Unveiling Linguistic Regions in Large Language Models, ACL 2024\n\n[4] Do Llamas Work in English? On the Latent Language of Multilingual Transformers, ACL 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZBoZxhFoFS", "reviewer": "Reviewer_TbKY", "summary": "The paper presents two key contributions. One is Parallel Language-specific Neuron Detection, a method of identifying elements of multilingual LLMs that are responsible for handling particular languages; the method only requires unlabeled text data in each language in order to detect these neurons, which makes it cheap and efficient. The second contribution is an insight into the workflow of multilingual LLMs (referred to in the paper as MWork), claiming that LLMs process input in three steps: understanding, task solving and output generation; task solving further splits into retrieval and thinking. Also, the thinking step is shown to be happening in English, while understanding, knowledge retrieval and generation are handled multilingually / specifically to the language of the input. Authors use PLND to verify that the models chosen for the experiments actually follow this workflow and conduct thorough experiments on two models about a dozen languages other than English, including high-resource and low-resource examples.", "questions": "Q1. There seems to be a leap from the experimental results to conclusions about how LLMs behave. Are the presented conclusions (about understanding->thinking->generating) the only possible explanation of the obtained results, or could there be other explanations?\n\nQ2. In your work the language-specificity is defined via that language only (importance is higher than a threshold). Would you expect a stronger performance of PLND if the language-specificity were defined contrastively, that is high impact on one language and low impact on other languages?, in other words, could a neuron be important to several languages?, and what would be the repercussions of such neurons on your work.\n\nQ3. In this work you focus on the \"English / non-English\" setting. Is this driven by the fact that English is the most abundant language in the training data?, and what would you expect the behavior of the models to be if another language were the most frequent, would the thinking be done in that language?, what about a more balanced setting where more than one language are dominant in the data (for example 40% English, 40% Chinese and other less frequent languages)?\n\nQ4. Concerning language-specific tuning, hypothetically, is multilingual performance of LLMs bound by English-centerdness? that is, can one achieve better performance of an LLM in a non-English language if that language's data is included into pre-training from the start, rather than being English-centered and tuned to a non-English language?\n\nQ5. Modern language models are often trained with programming code added to natural language texts. What would you expect your analysis to show in case the input is programming code or comments?\n\nQ6. Could full fine-tuning to a non-English language achieve better results than just tuning that language's language-specific neurons?", "limitations": "All good.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Both key contributions (PLND and MWork) are of fundamental relevance to the field of LLM interpretability and analysis\n\n- Presented experiments are rigorous and thorough\n\n- The paper is written clearly and although the subject is complex and multi-faceted, the authors do a good job of introducing the necessary concepts and definitions and explaining the rationale behind their work as well as the conducted experiments and analysis of their results.", "weaknesses": "W1. Most importantly, the claims are very broad (\"LLMs are this / LLMs do that\"), however this is verified on two very similar models: the included Mistral and Vicuna models both have 7 billion parameters, which puts them on the modest end of modern LLMs. How can we be sure that models with more parameters (13 / 30 / 70 / ...) behave in a similar manner without testing?, what about the influence of the context length on the performance? This is even mentioned in the limitations section of the paper: I am not suggesting that the authors run an additional set of expensive experiments -- I am suggesting that the paper text should be adjusted to reflect the actual findings of the presented experiments and would avoid bold claims about \"all LLMs\".\n\nW2. Another weakly supported claim is that 400 documents was enough for language-specific neuron tuning, simply because tuning on 800 documents did not yield better results. I find that this is a single instance and without running more comparisons and more thorough comparisons this claim does not hold and should be adjusted in the text. This is especially important since improvements of 2.3% (low-resource) and 3.6% (high-resource) are small-scaled improvements and bigger amounts of language-specific\n data could be expected to yield bigger improvements. Again, rather than running tons of more experiments for a single paper I instead suggest that the paper text should be adjusted (e.g. calling it a \"pilot study\" or \"preliminary indications/results\" -- PLND and MWork themselves are already worthy contributions).\n\nW3. Not a strong weakness, but minor typos:\n- row 101: \"With a set of n corpus\" --> \"With a set of n corpora\", same on row 103 ('corpus' in plural is 'corpora')\n- row 319: \"one brunch of work\" --> \"one branch of work\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer TbKY,\n\n\n\nThank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:\n\n\n\n> Concern #1: Overclaim on model size and multilingual enhancement experiments.\n\nWe appreciate your concern regarding the claims including \"LLMs xxxx\" and \"enhancing by 400 documents\". We acknowledge that the claim for \"all LLMs\" is strong and will revise that accordingly in the final version. Regarding the the enhancement experiment, we conduct experiments on tuning language-specific neurons by more documents from Wikipedia and here are detailed results.\n\n|      | **Original** | **0.4k** | **1k** | **8k** | **64k** | **128k** |\n| ---- | ------------ | -------- | ------ | ------ | ------- | -------- |\n| Vi   | 32.7         | **34.9** | 34.3   | 28.3   | 26.4    | 22.1     |\n| Th   | 25.6         | **28.5** | 27.4   | 23.7   | 20.8    | 15.9     |\n| Ar   | 21.7         | **23.4** | 22.8   | 19.2   | 16.2    | 8.3      |\n| Sw   | 15.1         | **16.9** | 15.2   | 12.4   | 7.9     | 5.5      |\n\nWe find that adding more documents significantly reduces performance. This phenomenon is likely attributed to overfitting the model to Wikipedia data, causing conflicts with patterns in the newly introduced data. To further validate our assumption and explore the effectiveness of language-specific tuning, we conduct additional experiments by assembling the training set from a wider array of resources such as textbook, website and journalistic materials. Due to space limitations, please kindly refer to the rebuttal addressing concern #2 provided to Reviewer 2b8j for further details.\n\n\n\n> Question #1: Other assumption of framework based on observation in Figure 1\n\nWe would like to share more details of our preliminary exploration, which involves analyzing the content of embeddings but is removed from the paper for the sake of coherence. Figure (1) in the attached PDF illustrates the decoded embedding of various layers of Vicuna-13b processing the Chinese version of \"What is one plus one equal to?\". Through this examination, we observe a progression where the model initially comprehends the input, transitioning tokens to English such as \"equal\" and \"?\", in the early layers. Subsequently, in the intermediate layers, nearly all tokens are in English, indicating the task solving in English. Finally, in the later layers, the English answer is transferred back into Chinese. In summary, our proposed framework draws inspiration not just from token distribution but also from a thorough examination of token meanings.\n\n> Question #2: Could a neuron be important to several languages?\n\nThe answer is yes, and we have investigated the degree of overlap among language-specific neurons in Appendix C (line 512 to line 523). \n\n\n\n> Question #3 & #4: What is the performance of non-English-centered LLMs.\n\nWe appreciate your concerns on other types of LLMs. Our analysis employs Qwen2-7b, a model that is not centered around English due to the comparable size of training data in Chinese and English. In Figure (2) in attached PDF, the distribution of tokens across layers is depicted. We observe that Qwen2 tends to quickly transition multilingual inputs to Chinese or English to address tasks predominantly in English, albeit with some reliance on Chinese support. Towards the end, answers in English or Chinese are reverted back to multilingual languages.\n\n> Question #5: Whether can implement on code?\n\nWe appreciate your invaluable insights on the generalibity of our proposed method. We also explore this direction following this paper. Our initial experiments indicate its potential applicability in code enhancement. Specifically, we utilize an instruction-code dataset [3] to identify code-specific neurons and subsequently fine-tune these specialized neurons. Experiments are conducted on Llama3-8b-Instruct and here are detailed results. Remarkably, we observe that training on a corpus of only 6k instances for less than 10 minutes can enhance the model's coding capabilities without compromising, and in some cases even enhancing, other aspects of the model. This outcome may be attributed to a clearer division of responsibilities among the parameters, culminating in superior overall model performance.\n\n|          | Human-Eval | MGSM     |\n| -------- | ---------- | -------- |\n| Original | 32.6       | 56.4     |\n| 2k       | 35.6       | 57.2     |\n| 6k       | **37.8**   | **61.6** |\n\n\n\n> Question #6: Could full fine-tuning performs better than language-specific tuning?\n\nFull fine-tuning demands a substantial amount of training data and may diminish performance in languages like English and Chinese, as evidenced by other continue-training works focusing on multilinguals [1]. Some works merge English corpus in multilingual training data to keep its English performance either in continue train or sft [2] [4]. However, these drawbacks can be avoided by our language-specific neuron tuning method.\n\n\n\n\n\n[1] Sailor: Open Language Models for South-East Asia, Arxiv 2024\n\n[2] SeaLLMs - Large Language Models for Southeast Asia, ACL 2024\n\n[3] Python_code_instructions_18k_alpaca\n\n[4] Multilingual Instruction Tuning With Just a Pinch of Multilinguality, ACL 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "15hs8qi2w8", "reviewer": "Reviewer_NLiQ", "summary": "This paper delves deeper into how LLMs handle multilingualism. The authors hypothesized a three-stage multilingual workflow called MWork: understanding, task-solving, and generating and which language(s) become essential in each stage. To verify their proposed workflow, they experimentally identify language-specific parameters and selectively deactivate them within different structures so this allows to assess the functionality of corresponding structures and enables the hypothesis. To do so, the authors develop a novel approach called Parallel Language-specific Neuron Detection (PLND). Without requiring labeled data, PLND can identify the language specific neurons. Following the PLND, the authors successfully identify the language-specific neurons which account for only 0.13% of all neurons. Their extensive results report that, by deactivating those neurons, the multilingual task performance could significantly dropped.\n\nThis paper tackles an important research question of LLMs: how do large language models handle multilingualism? To address this, they deliberately design a new approach of PLND and successfully identifies a language-specific neurons in the LLM models. Along with their MWork hypothesis, they carefully conducted experiments with multiple languages with different scrips and verified their hypothesized three-stage multilingual workflow.\n\nThis paper develops a useful model analysis tool of PLND. Their hypothesis is well explored in multiple languages. Technically well sound.", "questions": "- Regarding those language-specific neuron, are there any patterns or trends related to language family grouping or similarity? Such analysis or findings would be interesting to NLP researchers.\n-  Have you ever tried this approach against bigger models with > 7B parameters?\n- Since language-specific neurons are identified via the proposed framework, can we do model compression by preserving those specific parameters without losing multilingual capability? If you have examined this perspective, can you please share the results?", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- well-organized paper. clear presentation.\n- Technically sound by extensive experiments with different natural language understanding tasks with diverse language sets", "weaknesses": "- no major weakness though", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer NLiQ, \n\n\n\nThank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:\n\n\n\n> Question #1: Patterns of language-specific neurons.\n\nWe acknowledge your concerns regarding the properties of language-specific neurons, which have been thoroughly investigated in Appendix C (lines 512 to 523). Given its significance, we are considering integrate this section into the main text in the final version. Thanks for your recommendation.\n\n\n\n> Question #2: Performance on larger models.\n\nThank you for recommending adding analysis on larger models. Regrettably, we are unable to implement our methods on larger models at this time due to resource constraints. In line with Reviewer TbKY's suggestion, we will adjust the claim \"all LLMs\" accordingly in the final version.\n\n\n\n> Question #3: Model compression.\n\nWe appreciate your invaluable insights on the model compression potential of our proposed method. We also explore this direction following this paper. \n\nWe first take the opposite approach to language-dependent neurons. Specifically, we extract neurons that are not important for all corpora, denoted as Language-Irrelevant Neutons. We conduct testing in Thai on Mistral-7b-base, employing XLSum as our chosen task. Opting for XLSum is driven by its simplicity in contrast to complex NLP tasks like MGSM, yet it effectively showcases the model's multilingual ability when compared to measuring perplexity. Our analysis reveals that these language-irrelevant neutons represent a mere fraction of the overall parameters (approximately 2%), with model performance showing negligible impact upon deactivation. In contrast, random deactivation of 0.16B parameters (equivalent to 2% of all parameters) markedly impairs the model's functionality.\n\n| # Deactivated Parameter | Language-Irrelevant Neutons | Random |\n| ----------------------- | --------------------------- | ------ |\n| 0.11B                   | 24.3                        | 23.7   |\n| 0.16B                   | 24.6                        | <1     |\n\n\n\nNevertheless, the modest compression ratio proves unnecessary, and we think it's because the filtering criteria for language-irrelevant neurons are too strict. Consequently, we opt to retain language-specific neurons. Specifically, we implement language-specific neurons and deactivate non-language-specific neurons, resulting in improved performance compared to our previous setup. We can achieve nearly unaffected performance by deactivating up to 23% of all parameters. The detailed performance breakdown is presented below.\n\n| # Deactivated Parameter | 0.26B | 0.52B | 0.79B | 1.1B | 1.3B | 1.6B |\n| ----------------------- | ----- | ----- | ----- | ---- | ---- | ---- |\n| Performance             | 23.7  | 24    | 24.9  | 24.5 | 23.3 | 23.1 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for your insightful reviews and comments. We include Figures added in rebuttal in the attached PDF.

Meta-review: This paper proposes a neuron-level workflow for detecting the multilingual capabilities of LLMs. The motivation is interesting, and the experimental analysis is solid. During the rebuttal phase, the authors engaged in a profound discussion with the reviewers and effectively addressed most concerns. Given the intriguing insights this research offers into multilingual capabilities, I support its acceptance.
Decision: Accept (poster)