Unveiling Linguistic Regions in Large Language Models (2024)

Abstract: Large Language Models (LLMs) have demonstrated considerable cross-lingual alignment and generalization ability. Current research primarily focuses on improving LLMs’ crosslingual generalization capabilities. However, there is still a lack of research on the intrinsic mechanisms of how LLMs achieve crosslingual alignment. From the perspective of region partitioning, this paper conducts several investigations on the linguistic competence of LLMs. We discover a core region in LLMs that corresponds to linguistic competence, accounting for approximately 1% of the total model parameters. Removing this core region by setting parameters to zero results in a significant performance decrease across 30 different languages. Furthermore, this core region exhibits significant dimensional dependence, perturbations to even a single parameter on specific dimensions leading to a loss of linguistic competence. Moreover, we discover that distinct monolingual regions exist for different languages, and disruption to these specific regions substantially reduces the LLMs’ proficiency in those corresponding languages. Our research also indicates that freezing the core linguistic region during further pre-training can mitigate the issue of catastrophic forgetting (CF), a common phenomenon observed during further pre-training of LLMs. Overall, exploring the LLMs’ functional regions provides insights into the foundation of their intelligence 1.

heading: 1 Introduction
text: Over the years, the field of Natural Language Processing (NLP) has been at the forefront of understanding the core principles of intelligence. The emergence of Large Language Models (LLMs) such as GPT-4 (OpenAI, 2023), PaLM 2 (Anil
* Equal contributions. † Corresponding authors. 1 Our code is released in https://github.com/
zzhang0179/Unveiling-Linguistic-Regions-in-LLMs.
et al., 2023) and LLaMA 2 (Touvron et al., 2023), showcase a significant breakthrough. Thanks to unparalleled scales of model architecture and the vastness of training data, these LLMs now exhibit exceptional linguistic competence and can execute complex tasks requiring abstract knowledge (Dong et al., 2023) and reasoning (Cobbe et al., 2021).
Previous research has revealed that LLMs naturally capture cross-linguistic similarities in their representation space, facilitating zero-shot crosslingual transfer (Pires et al., 2019; Wu and Dredze,
6228
2019; Xu et al., 2023). The model is fine-tuned on one language, enabling the acquisition of comparable capabilities in another language (Muennighoff et al., 2023; Ye et al., 2023), and exhibits the phenomenon of code-switching when generating context (Khanuja et al., 2020; Zhao et al., 2024). Attempts to improve LLMs’ cross-lingual generalization abilities have been successful through parameter and information transfer learning (Üstün et al., 2020; Choenni et al., 2023), aligning languages compulsorily (Sherborne and Lapata, 2022; Shaham et al., 2024) and utilizing in-context learning techniques (Winata et al., 2021; Tanwar et al., 2023). However, a detailed investigation into the internal mechanisms of how LLMs possess crosslinguistic alignment capability remains elusive.
To delve deeper into the intrinsic mechanisms of LLMs’ linguistic competence, this paper focuses on the LLMs’ parameter importance and investigate the linguistic regions of LLMs based on 30 distinct languages’ performance, with the purpose of figuring out the following questions:
Q1: Does a core linguistic region exist within LLMs that facilitates cross-lingual alignment and generalization? By conducting further pretraining across six languages and evaluating models’ parameter importance (Section 2.2), we discover a region in LLMs corresponding to the core linguistic competence, which accounts for approximately 1% of the model’s total parameters. As shown at the top of Figure 1, removing this region (setting parameters to zero) consistently leads to a significant decline in performance across 30 test languages (Section 3.2).
Furthermore, by visualizing the core linguistic region (Figure 2), we observe that the linguistic core region of LLMs exhibits significant dimensional dependence. In certain dimensions, only perturbing a single parameter could lead to the model losing its linguistic competence (Section 3.3). Additionally, ablation study in 3.3 shows that beyond outlier dimensions, other non-outlier dimensions in this region are also critical.
Q2: Beyond the core linguistic region within LLMs , do distinct monolingual regions exist that specifically influence individual languages? While LLMs possess strong multilingual capabilities, we discover that each individual language (or language with similar compositional elements or grammatical structures) encompasses independent regions within the LLMs. As shown in the middle of Figure 1, the analysis of the Russian sentences
identifies a particular linguistic region that likewise exerts influence both on the Russian and Ukrainian language, both of which belong to the Slavic group (Section 3.4).
Q3: If and how core linguistic regions affect further pre-training, how to utilize it to optimize further pre-training? After pre-training, core linguistic parameter regions of the LLMs are established for multilingual alignment. Notable shifts in these regions potentially lead to a decline in model lingual capabilities. Our findings reveal that freezing this core region can mitigate the issue of catastrophic forgetting (McCloskey and Cohen, 1989; Kemker et al., 2018), a common phenomenon observed during further pre-training of LLMs. As shown at the bottom of Figure 1, we investigate the impact of selectively freezing 5% key parameters of all parameters during further pre-training, compared to the full-scale fine-tuning technique. Findings indicate that this method facilitates comparable learning of the target language while concurrently decelerating the rate of language attrition for previously learned languages (Section 3.5). Significantly, our methodology is compatible with the data-replay techniques (Robins, 1995; Wei et al., 2023), with no necessity for integrating extra components into the model. Unlike regularization methods (Srivastava et al., 2014; Goodfellow et al., 2014), our approach restricts to a minimal core region in LLMs.
The main contributions of our work are summarized as follows:
• We discover that LLMs possess a core linguistic region, and removing this region (setting parameters to zero) results in a significant loss of the model’s linguistic capabilities. Furthermore, perturbations to specific dimensions or even a single parameter can lead to a substantial decline in the model’s linguistic abilities.
• We observe that distinct monolingual regions exist in LLMs for different languages. Removing a specific monolingual region causes a significant deterioration in the linguistic capabilities within corresponding language.
• We perform further pre-training for specific languages within the core linguistic region of LLMs frozen, achieving comparable performance in the target language while mitigating catastrophic forgetting in non-target languages.

heading: 2 Background and Metric

heading: 2.1 Model Pre-training
text: Pre-training is a crucial process by which LLMs acquire linguistic competence and gain general knowledge about the real world. Formally, given a large corpus D, the training objective for autoregressive language modeling is to find the optimal θ that minimizes the following loss L: L(D, θ) = ∑
x∈D
∑
i
log pθ(xi|x1, ..., xi−1), (1)
where x = {x1, ..., xn} denotes an input token sequence and θ denotes parameters of the model.

heading: 2.2 Parameter Importance
text: Drawing upon the observations of linguistic alignments, we propose that particular parameters regions within the model exert significant influence on its inherent language alignment capabilities. Evaluating parameter sensitivity is a crucial metric for determining the significance of parameters in model pruning (Sanh et al., 2020; Liang et al., 2021; Zhang et al., 2022). If removing a parameter (zero-out) significantly affects the loss, the model is sensitive to it. More specifically, given a large corpus D and θ = [θ1, θ2, . . . , θd] ∈ Rd as the parameters of a model, with each θj ∈ R denoting the j-th parameter, the training objective is to minimize loss L(D, θ) (defined in 2.1). The importance of each θ is denoted as I(θ) ∈ Rd, where its j-th index Ij(θ) signifies the importance for θj .
Under an independent and identically distributed data (i.i.d.) assumption, the importance of a parameter Ij(θ) is measured by the increase in prediction loss when it is removed, calculated as the absolute difference between prediction losses with and without the parameter(θj):
Ij(θ) = |L(D, θ)− L(D, θ|θj = 0)| . (2) Calculating Ij(θ) for each parameter, as outlined in 2, is computationally expensive because it involves d distinct versions of the network computing, for each removed parameter. This becomes particularly challenging as the number of model parameters, d, grows to hundreds of billions. However, similar to several prior works (Molchanov et al., 2019; Zhang et al., 2022), using the Taylor expansion formula for L at θj = 0:
L(D, θ) = L(D, θ|θj = 0)
+ ∂L ∂θj (θj − 0) + 1 2! ∂2L ∂θ2j (θj − 0)2 + · · · , (3)
we can estimate Ij(θ) with its first-order Taylor expansion, eliminating the requirement for d distinct networks computation:
Ij(θ) ≈ |gjθj | , (4)
where gj = ∂L∂θj are elements of the parameter gradient g, and the importance is easily calculated since the gradient g can be obtained from backpropagation.

heading: 3 Experiments

heading: 3.1 Experimental Setup
text: To localize the functional regions corresponding to linguistic competence within LLMs and analyze their nature, we perform language further pre-training (next token prediction) on various languages and observe the relationship between internal parameter removal and external output quality. We utilize LLaMA-2-7B/13B (Touvron et al., 2023) as our model instance, as it stands out as one of the most notable state-of-the-art open-source LLMs in current academia.
Our experimental dataset comprises materials from Chinese platforms like Zhihu and Wechat, English sources from Arxiv and Falcon, and a corpus including books from 28 languages, totaling 30 languages in all. Six languages, namely Arabic, Spanish, Russian, Chinese, Korean, and Vietnamese, are chosen for language further pre-training and region localization, with 100, 000 samples for each (distinct from the samples in the test set). All 30 languages are employed for model testing and functional region analysis, with the specific languages and token count detailed in A. We use perplexity (PPL) as the criterion for evaluating the linguistic competence of a language model.

heading: 3.2 Core Linguistic Competence Region
text: In this section, we conduct further pre-training experiments on LLaMA-2 across six languages, aiming to explore and identify the core linguistic region in LLMs. Here, we define the region as "core linguistic", attributed to its minimal proportion within LLMs, constituting only 1% of the total parameters, and its association with model’s linguistic competence modeling.
Specifically, according to Equation 4, we cumulatively compute I∗(θ) = ΣI(θ) values across six different languages’ training, positing that the set of parameters exhibiting maximal importance score I∗(θ) during the language further pre-training may
have a strong correlation with the model’s linguistic competence. We provide both logical and empirical evidence to support this hypothesis.
Logical Evidence The phenomenon of codeswitching suggests that the LLMs can align languages and may possess core linguistic regions. As discussed in Section 2.2, if a parameter θj is crucial for the LLMs’ core linguistic competence, the model should be sensitive to θj , shown by a significant increase on the loss L when θj is removed, severely impairing the LLMs’ linguistic performance. Conversely, other parameters impact rarely on core linguistic capabilities.
Empirical Evidence 1 Table 1 illustrates that even a 3% removal on the ‘Top’ region leads to a substantial increase in perplexity (PPL), reaching over 40, 000 across 11 languages, indicating a complete loss of linguistic competence. In contrast, removing the ‘Bottom’ region is comparable to non-removal ‘Base’ in model PPL, and a ‘Random’ removal of equal magnitude has no signifi-
cant impact on the model’s linguistic competence. Moreover, refer to Appendix B, additional experiments with reducing training samples to 10, 000 or adjusted the region selection ratio to 1% and 5% yield consistent findings: removing the ‘Top’ region deprives LLaMA-2 of its capability across all 30 languages. This suggests the model’s linguistic competence is directly influenced by the ‘Top’ region, while removing the ‘Bottom’ and ‘Random’ region don’t have a significant direct impact on language capabilities. See Appendix B for evaluations on 30 languages and further experiments.
Empirical Evidence 2 In the experiment corresponding to Table 2, we initially zero out various regions within LLaMA. Consistent with the findings from Table 1, removing the ‘Top’ region leads to a loss of linguistic competence, whereas the ‘Bottom’ region don’t. However, in this experiment, we sought to ascertain if LLaMA could reacquire its lost cross-lingual generalization competence. Thus, we train on different amounts of Chinese Zhihu corpus and evaluate on Chinese Wechat and English Falcon corpora. The results indicate that unlike the ‘Bottom’ region, if the ‘Top’ region is removed and frozen, the model have to relearn basic language rules in other regions based on the provided Chinese Zhihu corpus, but these rules are inherently biased towards Chinese. Consequently, while its proficiency in Chinese is restored, the English perplexity remains high (5301.527). If the ‘Top’ region is removed but not frozen, the model can
rebuild its linguistic competence in-place. As its proficiency in Chinese is restored, so is its proficiency in English.
This implies that the ‘Top’ region encodes multilingual linguistic competence. When ‘Top’ region is zeroed-out and frozen, other regions difficultly adapt to regain the core linguistic competence.

heading: 3.3 Dimensional Dependence
text: To provide a more intuitive revelation of the spatial distribution characteristics of the linguistic competence region within the model, we visualize the ‘Top’ region. As shown in Figure 2, whether in the attention mechanism layer or the feed-forward layer, the linguistic region displays a distinct concentration in both the rows and columns of the matrices. In Appendix B, we also discover that in various layers, the core linguistic region is concen-
trated on different heads of the Attn.o matrix. Such distribution features seem to imply that the model’s linguistic competence is localized in specific rows and columns.
Structured Removal Instead of discretely removing different unstructured parameters, we selectively remove structured certain rows or columns for each matrix, especially those dimensions encompassing a significant number of ‘Top’ region parameters, termed as ‘Top’ dimensions. As illustrated in Table 3, we attempt to remove the columns of FFN.down and Attn.k/q/v, as well as the rows of Attn.o. The results indicate that removing just these ‘Top’ dimensions leads to a substantial decline in the model’s linguistic competence. However, removals to the ‘Middle’ ,‘Bottom’ and ‘Random’ dimensions do not yield noticeable effects. Selecting the dimensional region only from the Attention matrix or inverting rows and columns removals lead to similar findings, as described in C.
Single Dimension Perturbation Here, we explore whether a specific dimension significantly impacts the model’s linguistic competence. As illustrated in Figure 3, we iterate through the key dimensions mentioned in Section 3.3, attempting to perturb (random initialization) the same dimension across all Transformer layers. The results indicate that the impact of dimensions 2100 and 4743 on the LLaMA-2-13B substantially surpasses other dimensions, even when compared to the other three in the ‘Top 5’ dimensions. In contrast, perturbing two randomly selected dimensions, 2800 and 4200, yields linguistic performance almost indistinguishable from the unperturbed state.
Single Parameter Perturbation We discover that even a slight modification to a single parameter in a model with over 13 billion parameters can lead to a significant decline in its output quality. In Table 8, merely resetting the 2100-th parameter in the ‘Input_LayerNorm’ module of the 1-st layer to its initial value causes LLaMA-2-13B’s PPL value to skyrocket from 5.865 to 83224.078. If this weight parameter is multiplied by 10, the PPL value also rises to 4363.462. However, randomly altering the parameters at dimensions 2800 and 4200 doesn’t noticeably impact the model. For more details, refer to Appendix D.
Ablation Study Considering that the collapse of PPL may be possibly caused solely by the removal of outlier dimensions, rather than the collective effect of the entire linguistic region, we conduct an ablation experiment without removing the parameters of outlier dimensions (1512/2533 for LLaMA-2-7B and 2100/4743 for LLaMA-2-13B). The results for the 13B model, presented in Table 4, indicate that while outlier dimensions are contained within the core linguistic region, non-outlier dimensions in this region are also critical. Not altering any row or column of the matrix parameters for the two outlier dimensions also results in a great increase in PPL, although the collapse of PPL is slowed. This finding reveals that all dimensions in
the core linguistic region are tightly interrelated, and disrupting even a small part of it can lead to a PPL collapse. For a more detailed analysis of the experiments and results for the 7B model, please refer to Appendix E.
Output Under Perturbation To illustrate the impact of the linguistic competence region on the model’s output quality, we use "Fudan University is located in" as a prompt input and observe the model’s outputs under different parameter perturbations. The results are shown in Figure 4. Compared to randomly selected 4200-th dimension, perturbing model on 2100-th dimension significantly leads to model loses its linguistic competence, producing error or even nonsensical strings.

heading: 3.4 Monolingual Region
text: In this section, we wonder if LLMs possess distinct regions within different individual languages. Unlike the core linguistic regions described in Section 3.2, a monolingual region only has a strong correlation with certain languages, and removing it will only cause significant influence on LLMs’ proficiency in those corresponding languages.
Region Localization Different from Section 3.2, we initially identify and select the 1% ‘Top’ and ‘Bottom’ regions for each of the six languages (Ara-
bic, Spanish, Russian, Chinese, Korean, and Vietnamese) according to Equation 4, then deduplicate these regions. For the target language region, we exclude any regions that overlap with the ‘Top’ and ‘Bottom’ regions of the other five languages, aiming to eliminate the core regions and critical dimension corresponding to the model’s fundamental linguistic competence. We denote L, S and S∗ as the total set of six languages and the ‘Top/Bottom’ regions before and after deduplication, respectively. Language l’s own region S∗l is computed as follows:
S∗l = Sl − ⋃
l′∈L\{l} Sl′ . (5)
In Appendix F, we visualize the distribution of Attn.q matrix in ‘Arabic’ and ‘Vietnamese’ regions and discover minimal overlap between them.
Region Removal Unlike removing core regions or dimensions in Section 3.2 and 3.3, we discover that removing monolingual regions will only significantly affect the ability of the target languages and their closely related languages with similar letter elements or sentence structure. For example, if we remove only the region S∗Russian for Russian alone, selected from 10, 000 or 100, 000 samples respectively, as shown in Table 5, only Russian itself and Ukrainian have significant increases in PPL when removing ‘Top’ S∗Russian region. We speculate this to the fact that Russian and Ukrainian are relatively similar in terms of sentence structure and constituents, both belonging to the Slavic group. A similar phenomenon is observed if removals are changed to the regions for each of other five languages, see Appendix F for more details.
Downstream Task We conduct downstream tasks on MMLU (Hendrycks et al., 2021) and Ara-
bicMMLU (Koto et al., 2024). The former is in English while the latter is in Arabic. Our experiments demonstrate that removing monolingual regions significantly impacts the model’s ability to perform downstream tasks in the targeted language. Specifically, our findings are as follows:
1) The average probability for option ‘A’ on the ArabicMMLU evaluation increases from 0.36 to 0.64 if remove the ‘Arabic’ region. 2) As illustrated in Figure 5, the model’s accuracy in the ArabicMMLU (filtered) evaluation, where the correct answer is one of the options ‘B/C/D/E’, drops significantly from 25.6% to merely 1.5% when the ‘Arabic’ region is excluded. Conversely, the removal of the ‘Vietnamese’ region has a negligible impact on accuracy, which remains at 26.7%. 3) For the MMLU evaluation, the model’s accuracy is minimally impacted by the removal of monolingual regions. Compared to a baseline accuracy of 42.46%, the removals of the ‘Arabic’ and ‘Vietnamese’ regions result in accuracies of 39.27% and 39.68%, respectively.
Furthermore, using "There are 365 days in a year and 12" as a prompt for generation, we test outputs on the removal of the ‘Arabic’ and ‘Vietnamese’ regions. More details are provided in Appendix F.

heading: 3.5 Further Pre-training Optimization
text: In this section, we demonstrate that stabilizing the core linguistic regions (identified in Section 3.2) during further pre-training mitigates the catastrophic forgetting (CF) issue (McCloskey and Cohen, 1989; Kemker et al., 2018) in LLMs, while
maintaining learning proficiency comparable to full-scale fine-tuning in target language. Our experimental setup involves further pre-training LLaMA2-7B on 100,000 Arabic sentences, with a batch size of 256, a maximum token length of 512, and learning rates (lr) of 5e−5 or 5e−6, employing perplexity (PPL) as evaluation criterion.
Full-scale Model Fine-tuning Traditional fullscale fine-tuning, when increasing the learning rate or the amount of corpus data, enhances learning in the target language but aggravates forgetting in non-target languages. As shown in Table 2, since LLaMA-2 is primarily trained on English corpora, conducting a second stage of pre-training solely on large-scale Chinese corpora can lead to the forgetting of English competence. Additionally, as depicted on the left side of Figure 6 in blue line, increasing lr from 5e−6 (dotted line) to 5e−5 (solid line) under full-scale fine-tuning boosts the acquisition of the target language (Arabic), while simultaneously accelerates the forgetting rate of the nontarget languages (English and Chinese), as shown in the middle and right side.
Freeze Core Regions Fine-tuning We hypothesize that CF problem occurs due to the amplification of parameter adjustments when increasing the learning rate, which leads to significant shifts in the core linguistic region, adversely affecting language alignment. To mitigate this, we protect the core linguistic region and key dimensions by freezing the ‘Top 5%’ core language area for fine-tuning, as shown by the red line in Figure 6. At a lr of 5e−6 (dotted line), the difference between freezing finetuning and full-scale fine-tuning is minimal. How-
ever, when the lr increases to 5e−5 (solid line), freezing fine-tuning not only similarly facilitates faster learning in the target language (preserving comparable performance in Arabic PPL: 3.557 vs. 3.566), but also significantly reduces the forgetting of non-target languages (showing improvements in English and Chinese PPL: 18.796 vs. 20.557 and 90.84 vs. 563.423, respectively).
The potential reason for this phenomenon may lie in the preservation of the core regions within the cross-lingual alignment competence. Restricting the magnitude of updates in the core region’s parameters is a future strategy that we intend to employ. Notably, unlike regularization methods (Srivastava et al., 2014; Goodfellow et al., 2014), such approaches restricts to a minimal core region in LLMs, and can be implemented alongside blending previous data, retraining the entire network, or possibly only the final layers, without adding additional components to the model.

heading: 4 Related Work
text: Intrinsic Regions Prior works aimed to extract a sub-network capable of executing specific downstream tasks (Frankle and Carbin, 2019; Zhang et al., 2022) or task-specific subspaces to limit parameter fine-tuning within it (Aghajanyan et al., 2021; Zhang et al., 2023). For parameter importance estimation, an effective metric is to use parameter magnitude (Zhu and Gupta, 2018; Renda et al., 2020; Zafrir et al., 2021). Another metric involves estimating the sensitivity of parameters (Molchanov et al., 2019; Sanh et al., 2020; Liang et al., 2021; Sapkota and Bhattarai, 2023). In this work, we employ the latter method to select the most crucial parameters to unveil linguistic regions. Unlike (Frankle and Carbin, 2019), which extracted a complete sub-network for downstream tasks, as exemplified by the optimal scale of the lottery network at 21.2%, our findings indicate that the linguistic region is a much smaller (1%) functional region.
Cross-lingual Transfer Multilingual language models exhibit significant zero-shot and few-shot cross-lingual transferability across diverse tasks (Pires et al., 2019; Xu et al., 2023). Fine-tuned on one language enables model to obtain comparable capabilities in another language (Muennighoff et al., 2023; Ye et al., 2023), often displaying codeswitching behavior in context generation (Khanuja
et al., 2020; Zhao et al., 2024). While enhancements in cross-lingual generalization through parameter and information transfer learning (Üstün et al., 2020; Choenni et al., 2023), compulsory language alignment (Sherborne and Lapata, 2022; Shaham et al., 2024) and in-context learning techniques (Winata et al., 2021; Tanwar et al., 2023) have been effective, a comprehensive understanding of the internal mechanisms enabling cross-linguistic alignment in LLMs is still lacking.
Linguistic Abilities Probing Prior works have shown that multilingual LMs rely on a shared subword vocabulary and joint pre-training across multiple languages (Wu and Dredze, 2019; Pires et al., 2019; Cahyawijaya et al., 2023). However, new insights highlight these models’ capacity for learning universal semantic abstractions (Artetxe et al., 2020; Chi et al., 2020) and demonstrate that embeddings of similar words in similar sentences across languages are approximately aligned already (Cao et al., 2020; Conneau et al., 2020; Xu et al., 2022). Analysis from a hierarchical perspective reveals that classifiers linked to different BERT (Devlin et al., 2019) layers assess semantic features through varied probe tasks (Lin et al., 2019; Jawahar et al., 2019). In this work, we introduce a parameter partitioning perspective within LLMs, identifying core linguistic and monolingual regions, which underpin cross-lingual alignment and language-specific characteristics, respectively.
Outlier Dimensions Multiple studies have identified outlier dimensions in Transformer-based LMs (Kovaleva et al., 2021; Dettmers et al., 2022). Researches have found that certain outlier dimensions in pre-trained LMs are highly sensitive to the fine-
tuning of downstream tasks (Kovaleva et al., 2021; Puccetti et al., 2022). Furthermore, (Puccetti et al., 2022; Rudman et al., 2023) discovered that these outlier dimensions encode task-specific knowledge, and disabling these dimensions significantly degrades model performance. (Luo et al., 2021) attributed positional embeddings to the emergence of outliers, while (Rajaee and Pilehvar, 2022) reported inconsistent results. Additionally, (Sun et al., 2024) found that these outlier dimensions exhibit significantly larger activation values than others in LLMs. Our findings demonstrate that beyond the outlier dimensions, other non-outlier dimensions within the core linguistic region also play an indispensable role in the model’s core linguistic competence.

heading: 5 Conclusion
text: This paper explores the pivotal role of certain parameters in Large Language Models (LLMs), identifying a core region essential for multilingual alignment and generalization. Removing this region causes a complete loss of linguistic competence in LLMs. Furthermore, we discover that this core region is concentrated in specific dimensions, perturbing only one dimension can cause a significant decrease in linguistic ability. Moreover, beyond the core linguistic regions, we observe that monolingual regions exist within LLMs that affect specific languages. Importantly, we note that the catastrophic forgetting phenomenon during further pretraining may be related to drastic changes in core linguistic regions, as freezing this part during further pre-training alleviates the issue substantially. Our analysis and findings provide new perspectives and explanations for LLMs’ linguistic competence.
Limitations
In this paper, while we discover the core linguistic region and distinct monolingual regions within Large Language Models (LLMs), our work presents two notable limitations. First, our experiments are based on LLaMA-2-7B/13B, and it remains to be further determined whether the same phenomenon are observable in larger or differently architected models. Despite this, our focus on LLaMA-2-7B/13B reveals the existence of linguistic regions within the model, providing an explanation for the model’s linguistic capabilities. Secondly, we optimize full-scale fine-tuning through the freezing operation, which is not suited to extensive datasets. A more feasible approach is to limit the magnitude of parameter updates, which is the direction of our future experiments. Nevertheless, it is important to emphasize that slowing down forgetting through freezing core region suggests that in further pre-training, the core region is different from the other regions. Range of variation amplitude in core region should be smaller to maintain the cross-lingual generalization capabilities of the model. Additionally, while our study focuses on linguistic regions, beyond language, knowledge is a higher-level semantic representation, which is a critical direction for us to explore in the future.

heading: Acknowledgements
text: The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of China (No.62076069,62206057), Shanghai Rising-Star Program (23QA1400200), and Natural Science Foundation of Shanghai (23ZR1403500).

heading: A Languages in Evaluation Corpus
text: We use evaluation data composed of 30 languages to assess the model’s linguistic competence. The 30 languages and their respective token counts (use LLaMA-2 Tokenizer) are as follows: Arabic (4702998), Chinese (2869208), Czech (1362041), Danish (36467), Dutch (3991305), English (1216599), Finnish (372303), French (6755281), German (2884921), Greek (474622), Hungarian (1229433), Indonesian (19226), Italian (6332560), Japanese (501899), Korean (2730794), Malay (5842), Malayalam (1489244), Norwegian (42289), Persian (1736589), Polish (4948702), Portuguese (7598161), Romanian (1381598), Russian (5205716), Spanish (7163860), Swahili (630), Swedish (1450236), Tamil (2920808), Turkish (2484186), Ukrainian (455720), Vietnamese (3606202).

heading: B Core Linguistic Region
text: The regions are localized from six languages: Arabic, Spanish, Russian, Chinese, Korean, and Vietnamese, respectively. Our work does not alter the embedding layer, as we think it equates to a mapping of tokens, which does not involve modeling linguistic competence.
Region Visualization In Figure 9, we present the distribution of the ‘Top’ 5% regions in the Attn.o
matrix for the LLaMA-2-13B model. The results indicate that across various layers, the core linguistic region on Attn.o matrix is concentrated on different rows. This difference is observed among the 40 various attention heads.
Removal 3% ratio (100K) LLaMA-2 perplexity on 30 languages when the removal ratio is 3% ratio, with 100, 000 samples for each language. Refer to Table 15 for more details.
Removal 3% ratio (10K) LLaMA-2 perplexity on 30 languages when the removal ratio is 3% ratio, with reduced 10, 000 samples for each language. Refer to Table 16 for more details.
Removal 1% and 5% ratio (100K) LLaMA-27B perplexity on 30 languages when the removal ratio is changed to 1% and 5% ratio, with 100, 000 equivalent samples for each language. Refer to Table 17 for more details.

heading: C Attention Dimensional Removal
text: Figure 7 (left) illustrates that the columns of the Attn.k/q/v matrices in the attention layer, as well as the rows of the Attn.o matrix, correspond to different attention head parameters. Conversely, the rows of the Attn.k/q/v matrices and the columns of the Attn.o matrix are closely associated with dimensional features in the representation space.
We remove the ‘Top’ dimensions in the attention layer, and the results is displayed in Tables 6 and 7. Table 6 reveals that removing the Attention layers’ ‘Top’ dimensions continues to produce more detrimental effects than other dimensions. The visualizations in Figure 2 show that these dimensions are largely concentrated in a few attention heads, suggesting that some attention heads contribute more significantly to the model’s linguistic competence. Table 7 indicates that the removals under the second setting cause more damage than the first. Considering that, in the second setting, the ‘Top’ dimensions in the matrix directly interact with the corresponding dimensional features in the representational space, we can conjecture that these features are tightly linked with the model’s linguistic competence.

heading: D Single Parameter Perturbation
text: In a Transformer block, each column in the Attn.o and the MLP.down matrix of the FFN layer can be considered as the input weights of a neuron. Thus, perturbing a column can be seen as disturbing the
input weights of a neuron. Viewed from another angle, if we disturb the output activation value of this neuron, a similar effect should be observed. Within LLaMA, there is a specific module called RMSNorm, where each dimension is associated with a weight. Perturbations to these weights can be regarded as disturbances to the output activation values of the corresponding neurons. In Figure 7 (right), we visually demonstrate how RMSNorm affects a column of the Attn.o and the FFN.down matrix.

heading: E Ablation Study
text: Tables 9 illustrate the perplexity of LLaMA-2-7B after removing core regions with and without outlier dimensions, respectively.
The ablation experiments reveal that different methods of disruption and varying model sizes exhibit different rates of PPL collapse:
1) Removing according to Attention.Head (attn.k/q/v.col + attn.o.row) results in a slower collapse than according to Dimensional Features (attn.k/q/v.row + attn.o.col). 2) The 13B model shows a slower rate of collapse. 3) The abnormal dimension is mainly concentrated in the FFN layer of the “core linguistic region”. If preserving outlier dimension, the speed of PPL collapse by removing FFN layers decreases most obviously, while Attention.Head is almost unaffected. The results align with the findings of (Sun et al., 2024), where these outlier dimensions exhibit significantly larger activation values compared to others in LLMs.

heading: F Monolingual Region
text: Region Visualization In Figure 10, we present the distribution of the Attn.q matrix for ‘Arabic’ and ‘Vietnamese’ in 4 different layers. The results reveal that across various layers, the two monolingual regions are concentrated in different columns of the matrix.
Region Removal Tables 10-14 demonstrate LLaMA-2-7B perplexity after removing Arabic, Spanish, Chinese, Korean, and Vietnamese regions, respectively. The region is obtained by removing the intersections with other languages’ respective regions from the 1% ‘Top/Bottom’ regions, selected from 10,000 or 100,000 sentences during further pre-training according to Equation 4.
Case Study In Figure 8, we use the prompt "There are 365 days in a year and 12" to test the model’s output in English, Arabic, and Chinese, respectively. The results indicate that removing the monolingual regions causes the model to lose the relative language competence, leading the model to generate repetitive, nonsensical responses rather than correct answers like "12 months in a year".

heading: Future Work
text: A more feasible approach is to limit the magnitude of parameter updates, which is the direction of our future experiments. Nevertheless, it is important to emphasize that slowing down forgetting through freezing core region suggests that in further pre-training, the core region is different from the other regions. Range of variation amplitude in core region should be smaller to maintain the cross-lingual generalization capabilities of the model. Additionally, while our study focuses on linguistic regions, beyond language, knowledge is a higher-level semantic representation, which is a critical direction for us to explore in the future