The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model
Authors: Jiawei Chen
Wentao Chen
Jing Su
Jingjing Xu
Hongyu Lin
Mengjie Ren
Yaojie Lu
Xianpei Han
Le Sun
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
Multilingual
Code
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models (LLMs) have shown significant multilingual capabilities. However, the mechanisms underlying the development of these capabilities during pre-training are not well understood. In this paper, we use code LLMs as an experimental platform to explore the evolution of multilingual capabilities in LLMs during the pre-training process. Based on our observations, we propose the Babel Tower Hypothesis, which describes the entire process of LLMs acquiring new language capabilities. During the learning process, multiple languages initially share a single knowledge system dominated by the primary language and gradually develop language-specific knowledge systems. We then validate the above hypothesis by tracking the internal states of the LLM using specific methods. Experimental results show that the internal state changes of the LLM are consistent with our Babel Tower Hypothesis. Building on these insights, we propose a novel method to construct an optimized pre-training corpus for multilingual code LLMs, which significantly outperforms LLMs trained on the original corpus. The proposed Babel Tower Hypothesis provides new insights into designing pre-training data distributions to achieve optimal multilingual capabilities in LLMs.

Published as a conference paper at ICLR 2025
THE RISE AND DOWN OF BABEL TOWER: INVESTI- GATING THE EVOLUTION PROCESS OF MULTILINGUAL CODE LARGE LANGUAGE MODEL
Jiawei Chen--2, Wentao Jing SuA, Jingjing Xu², Hongyu Lin', Mengjie Ren Yaojie Lu', Xianpei Han', Le Sun', Chinese Information Processing Laboratory. Institute of Software, Chinese Academy of Sciences Unniversity Chinese Academy of Sciences BayteDance Inc.
ABSTRACT
Large language models (LLMs) have shown significant multilingual capabilities. However, the mechanisms underlying the development of these capabilities dur- ing pre-training are not well understood. an experimental platform to explore the evolution of multilingual capabilities in LLMs during the pre-training process. Based on our observations, we propose the Babel Tower Hypothesis, which describes the entire process of LLMs acquiring new language capabilities. During the learning process, multiple languages ini- tially share single knowledge system dominated by the primary language and gradually develop language-specific knowledge systems. We then validate the above hypothesis by tracking the internal states of the LLMs through identifying working languages and language transferring neurons. Experimental results show that the internal state changes of the LLM are consistent with our Babel Tower Hypothesis. Building on these insights, we propose a novel method to construct optimized pre-training corpus for multilingual code LLMs, which significantly outperforms LLMs trained on the original corpus. The proposed Babel Tower Hypothesis provides new insights into designing pre- pre-training data distributions achieve optimal multilingual capabilities in I.LMs.
this paper, we use code LLMs as
A united human speaking single language migrates to Shinar where they agree build a great city with a tower that would reach sky. Yahweh, observing these efforts and remarking on humanity's power in unity, confounds their speech so that they can no longer understand each other and scatters them around the world, leaving the city unfinished.
-The story of Babel Tower
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable multilingual capabilities (Philippy
et al. 2023 Choenni et al. 2023), even in some low-resource languages (Lai et al. 2023 Chirkova
& Nikoulina| 2024). Remarkably, LLMs are commonly pre-trained on multilingual corpora and demonstrate multilinuaal capabilities during the pre-training stage (Tanwar et al. 2023
Cahyawi-
jaya et al. 2024). Consequently, an important question arises: how do the multilingual capabilities of L.LMs evolve during pre-training process?
Many previous works have explored the multilingual mechanisms of LLMs. However, most of them are limited to investigating existing pre-trained LLMs rather than the dynamic process of
Corresponding authors.
https://en. http:////wwwwkkkekk wtt.iiipdda.....iiiki Sawee__o__aa_ Ra_ee

---

Published as a conference paper at ICLR 2025
pre-training. Some works have found that multilingual capabilities are similar to the translation
process (Zhang et al. (003). The LLMs form distinct language systems within their intermediate layers (Zhong et al. 2024] Wendler al.) 2024). and during generating, the LLMs transform the knowledge in these systems into target language tokens using language-specific neurons at top lay- ers (Tang at. 2024). Although these works explain the multilingual mechanism of existing LLMs, the formation of the mechanism during pre-training remains unexplored.
In this paper, we use code LLMs as an ex- perimental platform to explore the evolu- tion of multilingual capabilities in LLMs during the pre-training process (Li et al. 2023a Guo et al.] 2024). Specifically, we conducted preliminary experiments to analyze how monolingual LLMS learn a new language by continual pre-training the monolingual LLM on the corpus with the new language. similar to the setting of Zheng et al. (2024). As illustrated in Figure [ our key finding is that the model's capabilities in newly added lan- guages exhibit a pattern of rapid initial improvement, followed by a decline, and then stabilization (or slow improvement). Additionally, we observed that the point at which the capabilities of new languages begins to decline initially aligns closely with the point at which the knowledge transferred from the dominant language starts to decline. In the later stages of learning new languages, their performance gradually diverges from that of the domi- nant language. This phenomenon suggests that in the early stages of learning, the capabilities of new language entirely rely on the capabilities of the dominant language existing in the LLM. As training progresses, the performance of the new languages gradually diverges from that of the dom- inant language. Based on these findings, we propose the Babel Tower Hypothesis, which posits that during the learning process of large language models (LLMs), multiple languages initially share single knowledge system dominated by primary language and then gradually shift to developing multiple language-specific knowledge systems as training in new languages progresses. With Babel Tower Hypothesis. the process of LLMs learning a new language can be divided into three stages: (1) Translation Stage: The LLM extensively relies on the dominant language knowledge system to response to the new language. (2) Transition Stage: The knowledge system of the new language is gradually established, and the generation of the new language increasingly transitions from the dominant language system to the new language system. (3) Stabilization Stage: The new language's 's responses primarily depend on its own system. To validate Babel Tower Hypothesis, we further track the internal states of the LLMs at various stages by identifying working languages (Wendler al., 2024) and language transferring neurons (Tang et al., (202)) and found that the internal state changes of the LLM are consistent with our hypothesis: the generation of new language tokens progressively shifts from translating the primary language to utilizing its own system.
Pre trainin process
PHP Pesfrraasee
Triining ary
Internal state of the LLM
Pythce PE
Pyttor tooiingg
Syveem
Snnces
Transtanion Stoge
Transtion Stage
StabWlettoon Stage
The evolution of a LLM learning new lan- guage. In this figure, Python serves as the initial dom- inant language. with PHP as the new language. The process consists of three distinct stages: (1) Transla- tion Stage: the performance of PHP improves rapidly and the Python system is dominated in the LLM, and PHP generation primarily relies on the Python system; (2) Transition Stage: the performance of PHP begins to decline while gradually forms its own system; (3) Stabilization Stage: the performance of PHP stabilizes, and the generation of PHP depends on its own system.
Figure I:
Additionally, our experiments have revealed that the establishment of new language knowledge systems does not necessarily lead to improved performance in those languages. In other words, for many languages, relying on a strong dominant language and translating its knowledge is more effective than building new knowledge system using their own data. Building on these insights, we propose a novel method to construct the pre-training corpus for multilingual code L.MM pre-training to achieve optimal performance. Specifically, we utilize the new language acquisition experiments to establish the relationship between performance and pre- pre-training data distribution across different languages. By estimating the language distribution that corresponds to optimal performance, we can construct a pre-training corpus with an optimal distribution. Experimental results show that the code LLMs pre- pre-trained with our optimized pre-training corpus significantly outperform those pre- trained with the original corpus by a large margin. Therefore, the proposed Babel Tower Hypothesis
2

---

Published as a conference paper at ICLR 2025
provides a new insight into how should design the data ratios for different languages during the pre-training stage to achieve optimal multilingual capabilities in LLMs. Generally speaking, the contributions of this paper are:
We propose the Babel Tower Hypothesis to explain the evolution process of multilingual LLM during pre-training: during the learning process of large language models (LLMs), multiple languages initially share single knowledge system dominated by primary lan- guage and then gradually shift to developing multiple language-specific knowledge systems training in new languages progresses.
We validate the Babel Tower Hypothesis in code LLMs by employing two methods to probe and analyze the internal states of LLMs during pre- -reiirin. Meanwhile, we revealed that for many languages, relying on strong dominant language and translating its knowledge is more effective than building a new knowledge system using their own data.
We design an effective method to construct a pre-training corpus with optimal distribution across different languages, providing guide for the pre-training process.
RELATED WORK
Previous works mainly focused on studying the multilingual mechanisms within existing pre-trained LLMs. The primary approach they employ involves designing experiments to observe the generated outputs or internal states of these models. The primary methods of investigation can be categorized
into three types: (1) analyzing the generation of the model: the one hand, LLM can leverage knowledge acquired various languages to generate outputs (Chen et al. 2022 Zhang et al. 2023]
Kew et al. 2023] Yao al. 2024). On the other hand, in many cases.
the consistency of gener-
ated content across different languages remains low (Ohmer et al., 2023
Qi et al. 2023 Gao et al.)
the logit lens method (Nostalgebraist) 2020) to detect the output tokens of intermediate layers in LLaMA-2 (Tou- vron et al. (023). They discovered that these layers primarily operate in English, generating English tokens in the intermediate layers, even when the final output tokens are in other languages. Addi- tionally, Zhong al. (2024) found that the working languages are related to pre-training corpora. For instance, after continual pre- training using a Japanese corpus, LLaMA consistently produces Japanese tokens at the intermediate layers during token generation. (3) analyzing the activation states of neurons: some works identified language-specific neurons in LLMs that will be activated exclusively when processing specific languages (Tang et al., [024] Kojima et al.) 2024). (Tang et al.) (2024) found that these identified neurons predominantly located in the top and bottom layers of the LLMs, serving to transform input into a unified semantic space or to transform the unified semantic space into output. Zhang et al. (2024) identify the core region in the parameters of LLMs that will influence the multilingual alignment and generalization, and they also observe that mono- lingual regions that will influence the ability of specific languages. In contrast previous works, we investigate the evolution process of code LLMs during pre-training.
2024]. (2) analyzing the output of intermediate layers: Wendler et al. (2024)
3 THE EVOLUTION PROCESS OF MULTILINGUAL SYSTEMS
In this section, we investigate the evolution of multilingual ca- pabilities in LLMs during pre-training. Specifically, we intro- duce the Babel Tower Hypothesis to explain how these capa- bilities develop and then validate this hypothesis by tracking the internal system state of the LLM using two methods.
Table 1: The statistics of pre- training corpus.
Python
23M
37M
16M
Viokens
160
11B
3.1 EXPERIMENTAL SETUP
In this paper, the pre-training corpus is collected from GitHub, and its statistics are presented in Table I. We conducted experiments utilizing the GPT-2 architecture (1.3 billion parameters). We consider Python the dominant language due to its widespread use in various scenarios. The majority experiments are conducted on a Python monolingual LLM, which will be pre-trained on corpus of other languages such as PHP, C#, Go, or C++. The training step is set to 50,000 and the
3

---

Published as a conference paper at ICLR 2025
0.2
0.3
Powe@
0.2
Proooddd
Fase
0.2
Praportine
0.1
0.1
0.1
100
1000
10000
10
100
1000
10000
TTriiinn Step PHP
Training Step
PHPICI performance - Knowledge transferred from Python
Figure 2: The performance of PHP/C# (left y-axis) and the proportion of correct answers requiring knowledge from the Python corpus (Knowledge transferred from Python, right y-axis) across train- ing steps (X-axis). Based on the performance curve, we divide the entire process into three stages. The performance improves initially, then gradually declines, and eventually stabilizes. Concurrently, in the early stages, the proportion of correct answers requiring knowledge from the Python corpus relatively high but subsequently decreases.
global batch size is 1,024. To trace the performance of Python, we incorporated a small amount of Python data (a total of 100 million tokens) into the monolingual corpus of other languages.
3.2 BABEL TOWER HYPOTHESIS
To investigate multilingual evolution during pre-training, we conducted preliminary experiment to observe the process by which LLMs acquire new languages. Taking PHP and C# as an example, analyze the performance variations in HumanEval (Chen et al. 2021) during pre-training. Hu- manEval a problem-sol solviig code evaluation benchmark focused on Python, which has been trans- lated into various programming languages and is suitable for exploring the cross- wross-language transfer abilities of LLMs (Zheng et al. 2023 Guo et al. 2024). To identify the sources of an LLM's capabil- ities, especially whether it leverages knowledge from the Python corpus during generation, mea- sure the proportion of correct answers requiring knowledge from the Python corpus. Specifically, we constructed Python-specific subset of HumanEval that necessarily requires Python knowledge for solving by collecting problems that the Python monolingual LLM could correctly solve but which fail to be correctly solved by PHP or C# monolingual LLMs. Formally, let set I be the Python- specific subset and set C be the questions correctly solved by the LLM. The proportion of correct answers requiring knowledge from the Python corpus is calculated PPC|)
[C]
The experimental result is shown in Figure We can see that based on performance curve, the pre-training process can be divided into three stages:
(1) In the initial stage of pre-training, the performance of PHP and C# improved significantly. Con- currently, the proportion of correct answers necessarily relying on Python knowledge was notably high. We hypothesize that the LLMs are utilizing their Python knowledge to generate solutions for PHP and C# problems. The internal processing of the LLM may be analogous to translation; hence, we term this the translation stage.
(2) Subsequently, the performance gradually declined, accompanied by decrease in the proportion of correct answers necessarily relying on Python knowledge. This could be because the PHP/C# system gradually formed, leading to gradual transformation from the Python system to the PHP/C# system during generation. We term this the transition stage.
(3) Finally, the performance stabilized, there are almost correct answers that necessarily require Python knowledge to solve. We hypothesize that the LLM might have reached a stable internal state dominated by PHP/CH We term this the stabilization stage.
This phenomenon indicates that in the early stages of learning. the capabilities of a new language entirely rely the capabilities of the dominant language existing in the LLM. Over time, training progresses, the performance of the new languages gradually diverges from that the dominant language. To this end, propose the Babel Tower Hypothesis.
4

---

Published as a conference paper at ICLR 2025
Babel Tower Hypothesis During the learning process of large language models (LLMs), mul- tiple languages initially share a single knowledge system dominated by a primary language and then gradually shift to developing multiple canguage-specife knowledge systems training in new languages progresses.
The Babel Tower Hypothesis explains the evolution of multilingual LLMs during the pre-training process. We will further validate this hypothesis in the next sections.
3.3 INTERNAL STATE OF LLMS
To comprehensively understand the characteristics and differences among the three stages of the pre-training process in L.Mss, we track the internal states of L.MSs during pre-training using two distinct methods:
(1) Working languages Similar to Wendler et al. (2024), we employ the logit lens (Nostalge- braist 2020) to generate predicted tokens in intermediate layers, which can be seen as the working languages of the LLMs. In contrast to natural languages, the majority of programming languages are based on English, and a significant number of tokens overlap across different languages, making it difficult to identify the corresponding language simply through generated tokens. To accurately determine the working language, we use built-in functions or statements with identical functional- ities but different tokens as identifiers. For example, to return the length of an array. Python uses the "len()" function, while PHP uses the "count()" function. When the "count()" function is gener- ated in PHP, we can easily identify that internally generated "len" is a Python token. In the end, we identified five such identifiers for each new language and utilized GPT--4 to generate ten code completion problems for each identifier, guiding the LLM to generate the corresponding tokens. When these identifiers are generated, we employ the logit lens method to identify the corresponding tokens generated in the intermediate layers (excluding the last five layers). Specifically, we count the number of generated tokens in each language and calculate the frequency of for each language. This frequency used to estimate the proportion of working language. Formally, for language the proportion of it being the working language is calculated as follows:
6
R₂
Σ, E
where R, the proportion of i being the working language and e; the number of generated tokens of language i in the intermediate layers. The work language can measure the internal system of the model. When generating in the new language, if the Python system is dominant in the LLM, then the working language will primarily be Python.
(2) Language transferring Neurons Based on Tang al. (2024), there are Sanguage-speciffs neurons which will be activated in response to particular languages, and they are typically dis- tributed across both the bottom and top layers in the LLM, representing the transformation process from input to intermediate layers and from intermediate layers to output. Therefore, we refer to these neurons as language transferring neurons. We will investigate the variations in the number of language transferring neurons during pre-training. Specifically, following Tang et al. (2024), we calculate the Language Activation Probability Entropy (LAPE) for each neuron, and identify lan- guage transferring neurons for each language. The number of tranuuug-ttnnsrrring neurons per language serves as the final metric. When the primary language system dominates the LLM, large number of neurons will be involved in language transfer for other languages. This is because these language transferring neurons are required to facilitate the transformation from intermediate layers to top layers and from bottom layers intermediate layers.
The experimental results are shown in Figure 3 and 4 we summarize the characteristics of the internal states of LLMs during the pre-training process as follows:
Translation Stage In the early stage of pre-training Python serves as the primary working lan- guage, with significantly higher number of language transferring neurons activated for the new language. This indicates that the LLM activates extra neurons to transform Python knowledge into tokens of the new language, akin to translation process.
5

---

Published as a conference paper at ICLR 2025
Pyctem
Wi Ml..wwrrdn
Wt
1300
1300
Figure 3: The proportion of language being the working language (WL proportion, y-axis) when generating new languages across training steps (x-axis). Throughout the pre- pre-training process, the working languages gradually transition from Python the new languages.
Pyshen
Pyshe
1000
1880
1880
Trakking Stop
Trakiing Stop
Tradking Stop
Tradking
Figure 4:
The number of language transferring neurons in different languages (#LT neurons, y- axis) across training steps (x-axis). Throughout the pre-training process, the language transferring neurons for new languages activated gradually diminish, whereas those for Python activated pro- gressively increase.
Transition Stage Throughout the pre- pre-training process, the working language gradually transitions from Python the new language, and the proportion of neurons related new languages decreases accordingly. This indicates that the Python system gradually shifts to the new language system during the entire pre-training process.
Stabilization Stage In the final stage, the LLMs mainly work on the new languages, with the number language transferring neurons associated with the new language continues to decrease.
This demonstrates the process of system shifting during pre-training: from single language sys- tem that shares knowledge across multiple languages to forming multiple systems for different lan- guages.
4 ACHIEVING OPTIMAL MULTILINGUAL PERFORMANCE
Based on Figure 2 the performance peak occurs during the translation stage. This suggests that, for PHP and C#, leveraging the Python system is more effective than building their own systems. To achieve optimal performance,
intuitive approach would be to stop pre-training when the trans- lation stage is reached. However, in practical multilingual pre-training, the pre-training corpus contains various languages that cannot reach their optimal stages simultaneously. As diseussed in subsequent sections, C++ reaches optimal performance during the stabilization stage. Therefore, the early-stop strategy cannot be employed. To this end, we attempt adjust the distribution of pre-training data in different languages to enable different languages to achieve optimal states con- currently. In this section, we analyze the impact of pre-training data distribution on performance and internal states of LLMs and propose simple yet effective method to estimate the optimal pre- training data distribution.
4.1 THE IMPACT ON PRE- TRAINING DATA DISTRIBUTION
To understand the impact of pre-training data distribution the performance and internal states of LLMs, we pre-train the Python monolingual LLM with a mixed corpus of Python and a new language. By varying the proportions of these languages within the pre-training corpus, observed different final performances and internal states.
4.1.1 EXPERIMENTAL SETUP
In this section, we conduct experiments on PHP and will extend the conclusions to other languages in Section 4.2 The experimental setup is the same in Section 3.1

---

Published as a conference paper at ICLR 2025
69
PHP thon
PHP Py thon
PIT - Python
ALT 2290
nomber corpos
PHP corpes
corpus
(a) The HumanEval Performance. (b) The proportion of a language being the working language (left, WL As the number of tokens in the PHP Proportion) and the number of language transferring neurons (right, corpus increases, the performance #LT Nerons). the number tokens the PHP corpus increases, of PHP does not show correspond- the proportion PHP as a working language also increases and the ing improvement.
number of language transferring neurons for PHP declines.
Figure 5: The final internal states under different pre-training corpus. The X-axis represents the number of PHP tokens in the pre-training corpus, while the Y-axis indicates the performance, the proportion of the working languages, and the number of language transferring neurons correspond- ing to the final state.
Pre-training Corpus We downsample the original PHP pre-training corpus (23B tokens) and con- struct different PHP pre-training corpora based on the number of tokens, including IB, 2B, 5B, and 10B. We then mix the PHP corpus with the Python corpus for pre-training.
Metries We focus on the performance and internal state of the LLM. Specifically, we report three metries using the last checkpoint of pre-training process: HumanEval performance on PHP, the pro- portion of PHP used as the working language, and the number of language transferring neurons for РHP. Given the variability in HumanEval performance, we obtain the final performance by calcu- lating the average performance over sliding window with width of five, conducting evaluations every 1,000 steps.
4.1.2 EXPERIMENTAL RESULT
The experimental results are presented in Figure 5 We can see that:
The pre-training data distribution across different languages determines the internal state of the LLM at the Stabilization Stage. As shown in Figure [5b] when the number of PHP tokens in the pre-training corpus increases, the proportion of PHP in the working languages gradually rises, while the number of language transferring neurons for PHP decreases. This indicates that increasing the number of PHP tokens will ultimately lead to higher proportion of PHP system in the LLMs.
Achieving the optimal pre-training data distribution across different languages is crucial for effective cross-lingual transfer. As shown in Figure Sa increasing the number of PHP tokens initially enhances performance but subsequently leads to a decline. This trend indicates that, for Hu- manEval, the LLM benefits from the Python system for cross-language transfer. Therefore, achiev- ing optimal performance requires establishing the optimal internal state within the LLM. Given the unpredictable natural distribution of different languages, utilizing all collected tokens for pre- training does not necessarily yield positive results.
The results indicate that the establishment of PHP knowledge system with more data does neces- sarily enhance performance in it. Instead, reliance on the dominant Python system proves to be more effective. Compared Figure 2 we observed similar trend in PHP performance. This suggests that by adjusting the distribution of different languages in the corpus, it is possible to emulate the internal state observed at peak performance Figure?)
4.2
METHOD TO CONSTRUCT PRE-TRAN TRE-TIIINING CORPUS
The above experiments indicate that adjusting the pre-training data distribution in different lan- guages can achieve an optimal performance at the stabilization stage. In this section, we propose a method for estimating the optimal pre-training data distribution.
7

---

Published as a conference paper at ICLR 2025
Algorithm Estimate the corpus size of target language for optimal performance
Input: LLM M; Checkpoint save interval i; The target language corpus T; The validation set V;
The number of tokens in the Python corpus ?python
Output:
The number of tokens in the target language corpus;
1: Continual pre- pre-training M with T, saving checkpoints every i steps and recording the corre- sponding training losses {f1, fz..... .e......ni] (n);
2: Recording the initial loss a and the lowest loss 3;
3: Evaluating the saved checkpoints using V to obtain the performance set {81, ..,..., 8n and determine the index j = arg max ,...,,,
4: Estimating the proportion of the Python system P(CG) using f a, 3 based on Equation 2
5: Estimating the number of tokens in the target language corpus using P(e)), Tpython based on Equation
i.e., Marget = Opython/ P(())
7pythons
To determine the relationship between overall performance and the pre-training data distribution across different languages, we use the internal states of LLMs as mediators since they are related to both of them. Specifically, we divide the method into two steps: (1) constructing the relationship be- tween internal states and overall performance and (2) constructing the relationship between internal states and the pre-training data distribution.
4.2.1 RELATIONSHIP BETWEEN INTERNAL STATES AND OVERALL PERFORMANCE
Considering the similar trends in PHP performance depicted in Figures 2 and 5a establish correlation between internal states and overall performance based on the new language acquisition experiment Section 3.2 In the experiment, we transformed Python monolingual LLM into monolingual LLM of new language. We hypothesize that the proportion of the Python system decreases from 100% to 0%, while the proportion of the new language system increases from 0% to 100%. Consequently, we can use the observed performance changes during the pre-training process to estimate relationship between internal states and overall performance.
Specifically, we use training loss metric to measure system proportions for the following rea- sons: (1) it exhibits trends akin to those working language metrics, characterized by continuous increases decreases and initially rapid rate of change that later decelerates; and (2) it demands less computational effort compared to working language metrics:
~ {
P(E)
(2)
- 3
where P()) is the estimated proportion of the Python system at the step whose loss is a is the initial loss of LLM. Due to the excessively high value of the initial loss, the average loss of steps 90-100 is selected as the initial loss. / is the final loss of LLM, and we use the lowest loss in pre-training as the final loss.
4.2.2 RELATIONSHIP BETWEEN INTERNAL STATES AND DISTRIBUTION
We intuitive approach to construct the relationship between internal states and the pre-training data distribution across different languages: using the proportion of a language tokens in the corpus as the final system proportion:
P(n)
7
(3)
n
where P(n)) the estimated proportion of the system of language and m is the number of tokens of language i in the pre- training corpus.
4.2.3 OVERALL PROCESS
To fully utilize the knowledge of Python for lross-language transfer, in this paper, we fix the Python corpus and control corpus size of others languages in corpus. Based the above methods, the optimal number of tokens corresponding to a target language can be estimated. The specific process shown in Algorithm T
8

---

Published as a conference paper at ICLR 2025
RIP Fackedd
FRP Acaal
Prodsal Actal
Podkked ArAd
Cee Artal
Pawin
M The average HvmanEval performance of Python and the new language varies with the proportion of the Python system. Blue dots indicate the predicted results, while orange triangles represent the actual results obtained from pre-training
of
Oegropartion
Figure 6:
HHP Prokined
Astaa
Pootedd
Azed
De Prociaa
Achal
CC Prakiet
Pavent
Favert
M Figure 7: The HumanEval performance of the new language varies with the proportion of the Python system. Blue dots indicate the predicted results, while orange triangles represent the actual results obtained from pre-training.
02
ofopertion
In this way, we are able to determine the optimal number of tokens in the pre- mee-trannnng corpus of other languages, thereby achieving better overall performance. Next, we will conduct experiments validate the proposed method.
4.3
EXPERIMENT
4.3.1
SETUP
The experimental setup is the same as Section 3.1 and the new languages are PHP, C#, Go, or C++ We use the average HumanEval performance of Python and the new language and the performance of the new language as the final performance. In addition, we validate the estimated results by pre- training it corpora with varying proportions. Notably, for C++, observed that it relies less the Python system. Consequently, we verified the results at only few key points.
4.3.2 EXPERIMENTAL RESULT
The experimental results are shown in Figure and 7 Despite potential numerical discrepancies, the trend between the predicted and actual results remains consistent, emphasizing the effectiveness of our method. For most languages, as the proportion of Python integration increases, the predicted final performance initially rises and then declines. This pattern indicates that the new language can utilize the knowledge of the Python system for generation. However, when the proportion of the new language is excessively high, it may compete with the Python system, resulting in decreased performance. Conversely, significantly low proportion of the new language in the model hinders its ability to generate the new language effectively. For C++, we found that it does not depend on the Python system and can achieve optimal performance through its own system. Our method is applicable to such languages, indicating its universality.
4.3.3 PRE-TRAINING FROM SCRATCH
To further demonstrate the effectiveness of our method, we pre-train multilingual LLMs with 1.3 bil- lion and 6.7 billion parameters from scratch using a mixture of corpora from five different languages. We compare the original corpora with our optimized versions. In addition to HumanEval (Chen) ctt al. 2021), we evaluate the LLMs using the MBXP benchmark (Athiwaratkun et al. (022). Specif- ically, based on F we randomly down-sampled the token quantity for PHP and C# to billion and 10 billion tokens, respectively, while keeping the token quantities for the other languages un- changed. The final metrics were derived from the average HumanEval and MBXP performances across all five languages. For 1.3B LLM, the total number of processed tokens is set to 500 billion. For 6.7B LLM,, the total number of processed tokens is set to 800 billion.

---

Published as a conference paper at ICLR 2025
The HumanEval and MBXP performance (Pass@1 1) of different languages during pre- training. We compare the performance of LLMs pre-trained using the original corpus (Original) with those pre-trained using our optimized corpus (Optimized). The evaluated LLMs contain 1.3 billion (1.3B) and 6.7 billion (6.7B) parameters. AVE is the average performance of HvmanEval and MBXP. Bold indicates the best results, and underline indicates the second-best results.
Table 2:
HumanEval
MBXP
Python
PHP
C#
Go
C++
AVE
Python
PHP
C#
Go
C++
AVE
18+ Models
StarCoder-1B BB
Lie
2023b
15.17
13.04
15.19
8.44
14.63
13.29
29.80
18.56
23.29
58.93
26.18
31.35
StarCnder.38 3B
d a1
207248
21.46
21.12
16.46
14.29
24.39
19.54
39.80
30.13
39.27
74.78
35.91
43,98
DespseckCoder
GLM
2024
23.60
31.10
26.39
52.40
45.41
55.11
55.93
Original- 1.3B (ouns)
28.66
24.22
23.42
21.43
27.44
25.03
41.20
33.41
43.38
79.24
45.39
49.81
Optimized-1.8 (ours)
32,93
26.08
29.75
16.88
27.44
26.61
45.40
46.29
40.64
77.90
49.63
51.97
Models
StarCodes:7B Li
2023b
28.37
24.22
27.85
17.53
25.61
24.72
46.20
42.36
45.66
71.88
43.64
49.95
StarChder2-7B LyPERO 01 m
2024
35.
32.30
39.24
22.08
34.76
32.76
53. 13
55.02
54.79
82.37
55.61
60.18
CodeLlama-7B
43.89
DeepsekCCden
(aW 21
2024
46.95
37.89
47.47
31.82
44.51
41.73
65.80
66.16
66.67
89.78
67.58
71.20
Original-6, 7B (ours
46.95
38.51
39.34
62
40. 24
38,
59.00
57. 86
61.19 19
83.04
60.60
64.34
Optimiaad-6.78 (ours)
41.61
31.17
43.90
The experimental results are shown in Table 2 Compared the original pre-training corpus, the corpus optimized with our method exhibits a significant average performance improvement, indi- cating that our approach is suitable for a broader range of pre-training scenarios. Furthermore, our method remains effective even when applied to an LLM with 6.7 billion parameters, indicating the universality of our approach. In future work, our method could be integrated with the filtering of pre-training corpora to construct higher-quality pre-training data.
5 DISCUSSIONS
Analogy to second language acquisition The development of multilingual capabilities in large language models (LLMs) resembles second language acquisition (SLA) in humans (Krashen 198T). During the learning process, systems for various languages gradually take shape (Cook) 2016). Ad- ditionally, the acquisition of a second language is influenced by previously mastered languages, resulting in language transfer (Lightbown & Spada, 202T). This correlation between multilingual learning in LLMs and human language acquisition may enable the application of human cognitive theories explain various behaviours LLMs in the future.
The significance of this paper Firstly, this paper reveals the mechanisms underlying the multi- lingual capabilities of LLMs during the pre-training process. It provides guidance for multilingual pre- rre-training, such as data proportioning. Additionally, our findings indicate that pre- pre-training strate- gies may differ across languages. For example, for some low- wwwrresuucce languages, thinking mainly within the primary language system rather than constructing an entirely new system might be the op- timal strategy. Lastly, the findings highlight potential limitations of previous multilingual LLMs (Cuil et al., 2023) Csaki et al.) [2024] Faysse et al. 2024]] Zheng et al. 2024). For instance, using a specific language's corpus for continual pre-training does not necessarily enhance the LLM's performance.
6 CONCLUSIONS
In this paper, to explain the evolution process of multilingual capabilities in LLMs during pre- training. we propose Baber Tower Hypothesis: during the learning process of large language models (LLMs), multiple languages initially share single knowledge system dominated by a pri- mary language and then gradually shift to developing multiple language-specifh knowledge systems as training in new languages progresses. Taking code LLMs as an example, we employ two meth- ods to validate the hypothesis. We further found that for some languages, leveraging dominant language and transferring knowledge from it is often more effective than building a new knowledge system with additional pre- pre-training data. Finally, we propose simple and effective method to timate the optimal distribution of pre- trainigg corpora across different languages. In the future, our approach can be extended to natural language scenarios and scaled larger models.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGE
We sincerely thank the reviewers for their insightful comments and valuable suggestions. This work was supported by Beijing Natural Science Foundation (L243006), the Basic Research Program of ISCAS (Grant No. ISCAS-JCZD-202303, ISCAS-ZD-202402),
REFERENCES
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. Multi Multi-lingual evaluation of code generation models. 2022. doi: 00.48550/ARXIV.221000 148668 14868. URL https: /tt.aaaxivv/..////.//...//aawav. wwwpxii..rggabb//2/22o 14868
Samuel Cahyawijaya, Holy Lovenia, and Pascale Fung. LLMs are few-shot in-context low-resource language learners. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.). Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguis- fics: Human Language Technologies (Volume 1: Long Papers), pp. 405-433, Mexico City, Mex- ico, June 2024. Association for Computational Linguisties. doi: 10.18653/11/2000 naacl-onn.l- 24.
URL https: http://aaaaaathhoo
org/ 2024 naacl long 24
Fuxiang Chen, Fatemeh H. Fard, David Lo, and Timofey Bryksin. On the transferability of pre- trained language models for low-resource programming languages. In Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension. ICPC '22, pp. 401-412, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978 1503399999, doi: 10. 1145/3524610.3527911 URL http::ttts////tt http://////..rr11111115555444443 155 3527917
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
Nadezhda Chirkova and Vassilina Nikoulina. Key ingredients for effective zero-shot cross- lingual knowledge transfer in generative tasks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 7222-7238, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 18653vv// 1024.naacl-long.oo 401. URL Stthttp:: http:////aaaaattttaageee org 2024 naacl long 401
Rochelle Choenni, Dan Garrette, and Ekaterina Shutova. How do languages influence each other? studying cross-lingual data sharing during LM fine-tuning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 13244-13257 Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18065./1 1118666////22..//mmemee main.818. URL httphhs://caaaathllllooooooo//ee
2023 emnlp-main 818
Vivian Cook. Second language learning and language teaching. Routledge, 2016.
Zoltan Csaki, Bo Li, Jonathan Li, Qiantong Xu, Pian Pawakapan, Leon Zhang, Yun Du, Hengyu Zhao, Changran Hu, and Unmish Thakker. Sambalingo: Teaching large language models new languages, 2024. htt.:/tts///aat wttp:///ww.../////////////// 05829.

---

Published as a conference paper at ICLR 2025
Yiming Cui, Ziqing Yang, and Xin Yao. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177, 2023. URL httph::///// awwiv. org/ ats//bbs/2200000 2304.
08177
Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro, António Loison, Duarte M. Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro H. Martins, Antoni Bigata Casademunt, François Yvon, André F. T. Martins, Gautier Viaud, Céline Hudclot, and Pierre Colombo. Crois- santllm: A truly bilingual french-english language model, 2024. URL.ttts: htt.stts:://aat ptttpp::/aaxiv.orrg abs/2402. 00786
Changjiang Gao, Hongda Hu, Peng Hu, Jiajun Chen, Jixing Li, and Shujian Huang. Multilingual pretraining and instruction tuning improve cross-lingual knowledge alignment, but only shal- lowly. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Con- ference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume Long Papers), pp. 6101-6117, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 1/655////222..maacllooom
URL
bttps://cclantholog wrgpey..rr//// wttp//////.2.. naacl-long 339
Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming the rise of code intelligence, 2024. URL https:
//arx/v.orc/ab//2200..1
Tannon Kew, Florian Schottmann, and Rico Sennrich. Turning english-centric Ilms into poly- glots: How much multilinguality is needed?, 2023. /R//ittps://rrxi.oorr////ba 12683
Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo. On the mul- tilingual ability of decoder-based pre-trained language models: Finding and controlling language- specific neurons. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), pp. 6919-6971, Mexico City. Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.18653/v12022. naacl-long.384 URL httttpss//ac.aattlloooooooo// 2204 naaee44laaaallllnnegee 384
Stephen Krashen. Second language acquisition. Second Language Learning, 3(7):19-39, 1981.
Viet Lai, Nghia Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franek Demoncourt, Trung Bui, and Thien Nguyen. ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 13171-13189, Singapore, De- http:///w/1/22...nnn.g..
cember 2023, Association for Computational Linguistics. doi: 878. URL https: ac www.scannnttthtoooooooooooo org 2023, http://2222.......ngneee. www.fffiiniiiiiimiae
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii. Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Lo- gesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luc- cioni, Paulo Villegas. Maxim Kunakov, Fedor Zhdanov, Manuel Romero. Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan Davit,,, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023a. URI /ttpps://aaxxr wwwpxxvvvrrrrrooooooo org harg//bb/// 22305.33... 061 61
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li. Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João
12

---

Published as a conference paper at ICLR 2025
Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Lo- gesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luc- cioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! 2023b.
Patsy M Lightbown and Nina Spada. How languages are learned 5th Edition. Oxford university press, 2021.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poiries Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, De- nis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas KrauB, Naman Jain, Yix- uan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xian- gru Tang. Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu. Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Can- wen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Car- olyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Car- los Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generation, 2024. URL http//b////222../
httpss://arxi
Nostalgebraist.
LessWrong, 2020. https: www. wesswrong. www.///sss///RREEEEEESASSSSSSSS.
interpreting gpt: the logit lens.
URL
Xenia Ohmer, Elia Bruni, and Dieuwke Hupkes. Separating form and meaning: Using self- consistency to quantify task understanding across multiple senses. In Sebastian Gehrmann, Alex Wang, João Sedoe, Elizabeth Clark, Kaustubh Dhole, Khyathi Raghavi Chandu, Enrico Santus, and Hooman Sedghamiz (eds.). Proceedings of the Third Workshop on Natural Language Gener- ation, Evaluation, and Metrics (GEM). pp. 258-276-66622 276. Singapore, December 2023. Association for Computational Linguistics. |
hww.://aww..oolloooooo///22///. org/ gem- 1 .22
Fred Philippy, Siwen Guo, and Shohreh Haddadan. Towards common understanding of contribut- ing factors for cross-lingual transfer in multilingual language models: A review. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5877-5891, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.118653/11/2223.ac./ 323. URL http::ttp////tttt /ttp///ccaaaaa http::://///aaaahhhoolooo hwwphhhwllloo.o...o/ooeo 201.2222......moo 323
Jirui Qi, Raquel Fernández, and Arianna Bisazza. Cross-lingual consistency of factual knowledge in multilingual language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10650- 10666, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/ 2023.emnlp- www.e/mmmmmmmiiisss...... URL http:::://aaaatttllll
org/ 2023. emn emw.ommaaammmmmmmmmmmm 658
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Ev- timov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code, 2024. URL https: /tapp::://aaxv...///
arxiv. org/
12950
Tianyi Tang, Wenyang Luo, Haoyang Huang. Dongdong Zhang, Xiaolei Wang. Xin Zhao Furu Wei, and Ji-Rong Wen. Language-speccfiic neurons: The key multilingual capabilities in large
13

---

Published as a conference paper at ICLR 2025
language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5701-5715, Bangkok, Thailand, August 2024. Association for Computational Linguisties. URL https: /ttp::///allaaae Lantho
www.orgg/.2....... 2024 acc long. 309
Eshaan Tanwar, Subhabrata Dutta, Manish Borthakur, and Tanmoy Chakraborty. Multilingual
LLMs are better cross-lingual in-context learners with alignment. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6292-6307, Toronto, Canada, July URL
2023. Association for Computational Linguistics. doi: 10. 1tw.1866//vli 1200://22222...ooooo..... -ong. 346
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi. Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URLhtttps httpps://arxiv.com wwg...o.w.......o...oo.o.oo....r aas//a////// 220022372222 09288
Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Do llamas work En- glish? the latent language of multilingual transformers. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computa- fional Linguistics (Volume 1: Long Papers), pp. 15366 15394, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL httpps//aaalaatttoooooooo//////p
acl-long. 820
Feng Yao, Yufan Zhuang, Zihao Sun, Sunan Xu, Animesh Kumar, and Jingbo Shang. Data contam- ination can cross language barriers, 2024. /Rtttt:s://axxi org/ abs//b//////aeeeeeaa//./aa/aaaea wwwpaab....../ 12206..1223
Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. Don't tt trust ChatGPT when your question is not in English: A study of multilingual abilities and types of LLMs. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7915-7927, Singapore, December 2023. Association for Computational Linguistics. doi: 10. htt.33/w112222.mmmmmmmmm https: http://acllas www.ccaannhhlooooooooooooo ogy. org/ 2023. emnlp -a ina.inn.. 491
URL
Zhihao Zhang, Jun Zhao, Qi Zhang, Tao Gui, and Xuanjing Huang. Unveiling linguistic regions in large language models. In Lun-Wei Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6228 6247, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL URL htt.t::t /ttp::://caaattta
ogy- org/ 2004/22242222222 acl ong. 338
Qinkai Zheng. Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5673-5684, 2023.
Wenzhen Zheng, Wenbo Pan, Xu Xu, Libo Qin, Li Yue, and Ming Zhou. Breaking language barriers: barriers Cross-lingual continual pre-training at scale, 2024.
02118
Chengzhi Zhong, Fei Cheng, Qianying Liu, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Mu- rawaki, and Sadao Kurohashi. Beyond english-centric lmss: What language do multilingual lan- guage models think in?, 2024. URL https:: htappp:///axxiiis www.vv.oorg//bbaa wttp/b/////....o..////// /408 10811
14

---

Published as a conference paper at ICLR 2025
PHP
C#
03
Python
Python
Passã
0.2
Pass@1
10
100
1080
10008
10
100
1000
T0010
Training Step
Training Step
Figure 8: The performances of Python and PHP/C# (y-axis) across training steps (x-axis).
Estimated by training
Estimated partion of the python
Estimated by training loss
Estimated by working language
Estimated by working language
Extimated portion
5stem
0.6
0.6
0.4
0.4
100
1000
10000
100
10000
Training Step of PHP
Training Step of C#
Figure 9: The estimated proportion of Python system (y-axis) across training steps (x-axis). The orange line represents the proportion the Python system estimated using training loss, i.e.. equation 2 The blue line represents the proportion of the Python system using the proportion of Python as the working language.
A APPENDIX
A.1 WORKING LANGUAGES DETECTION
To determine the language of the generated tokens, use built-in functions statements with identical functionalities but different tokens as identifiers. The identifiers are shown in Figure 3 When the PHP identifier "strien" generated, we will use logit lens method generate tokens in the intermediate layers. The generated "len" will be considered as Python while "strlen" will be considered PHP.
Table The identifiers for detecting the working languages and the corresponding Python tokens.
PHP "suten": "lea"; "count": "len": "sort": "sorted"; "gottype": "type": "array sum": "sum":
C# "Console. "Comsole.Writeeiinee Wnitelline: "prini"; "Console ReadLine": "input") IndexOl"; "Lind" iindex"; "catch": "except"; "LAdd"; "lappend";
"suppend": "append"); "Irue": "True": "false"; "Talse"; "func"s "der"y "&de"; "and"y "IP' "oe"s "moe"; "bil": "None":
C++
back"" back": "append": "catch": "except"; "Ree" "and") "I]": "ar": "T% "mot": "leruse": "pop": "toustring": "str":
A.2 THE PERFORMANCE OF PYTHON
We report the Python performance in the experiments Section 3.2 As shown in Figure g in the first stage, the performance of the new language increases, while Python experiences slight decline. During the second stage, as the new language system is established and competes with Python, Python's performance significantly declines. Finally, Python's performance stabilizes.
15

---

Published as a conference paper at ICLR 2025
A.3 THE ESTIMATED PROPORTION OF THE PYTHON SYSTEM
To further demonstrate the relationship between the training loss and estimated proportion of the python system, we used working language for verification since the working language can reflect the internal state of the LLM. Specifically, we calculate the proportion of Python as the working language at each checkpoint and assume that this can be approximated the proportion of Python systems. We then compare this result with the proportion of Python systems estimated from the corresponding training loss. The results are presented in Figure We can see that the proportion of the Python system estimated from the training loss closely approximates that estimated from the working language, thereby substantiating the relationship between training loss and the estimated proportion of the Python system.
16


Reviews:
[{"review_id": "NkqelQ62el", "reviewer": "Reviewer_6AVA", "summary": "This work proposes a Babel tower hypothesis for the evolution of multi-lingual code LLM, that is continual pre-training a monolingual code LLM to master a new language will go through translation, transition, and stabilization stages. The authors demonstrate the evolution process of a python LLM based on GPT2 for new languages PHP and C#, revealing the best performance achieved at the translation stage.  Based on the observation the authors propose an algorithm to estimate the optimal distribution of the training corpus for different languages based on a linear hypothesis of the training loss and the monolingual system. The authors conduct experiments for new languages PHP, C#, Go, and C++ for HumanEval and MBXP, suggesting better performance compared to a baseline data distribution.", "questions": "1. In Fig 2 the authors show the evolution of performance of new languages, however, it is also important to show the performance of Python to illustrate the impact on the learned languages.\n\n2. Could the authors show some example of python-specific problems which are crucial to define the knowledge transferring procedure?\n\n3. The authors define a linear relationship between the training loss and estimated portion of the python system as in eq (2). Could the authors clarify: (1) how to calculate the loss which might consist of python data only; and (2) further justification of the relationship other than results in Fig 6 and 7 to demonstrate the scalability of this hypothesis.\n\n4. If I understand it correctly, the authors assume there is no intervening impact between different language during learning procedure. Both the evidences to demonstrate the learning stages of the new language and the proposed optimal data distribution are base on this assumption. Could the authors show some solid justification for this assumption?\n\n5.  The authors only present the results with a baseline distribution in Table 2. I wonder if it is comparable to other optimized data distributions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Overall this paper is well organized and the topic is interesting. The authors justify their hypothesis in a clear way.\n\n2. The authors make a strong analogy of the learning process to human acquisition of new languages, and present some well-designed results including the evolution process and the transition of the internal states of LLM. Also, they apply the transition for optimization of corpus distribution, which could be significant for training of multi-lingual LLMs.", "weaknesses": "1. The paper lacks some detailed demonstration of the crucial parts of their claims. Please refer to my questions.\n\n2. When presenting the methodology, I believe the authors make some important assumptions and analogy without further justification, which might make the presentation less persuasive. Please also refer to my questions for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ehxSkoXYX0", "reviewer": "Reviewer_rkZp", "summary": "This paper investigates the multilingual capabilities of large language models (LLMs), specifically focusing on how a monolingual code LLM trained in Python can integrate additional programming languages, such as PHP, C#, Go, and C++. \n\nAnalyzing the model's performance on code problem-solving tasks, the study proposes a hypothesis, termed the \"Babel Tower Hypothesis,\" suggesting that learning new languages involves three phases: \n 1. translation\n 2. transition \n 3. stabilization\n\nThe hypothesis is tested through an analysis of working languages and language-transferring neurons. \nAdditionally, the paper proposes a method to construct an optimal distribution of languages during pre-training to enhance multilingual performance.", "questions": "Typo:\n - line 385 \"Rrecoding\" -> \"Recording\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is clearly written, making it easy to read and follow.\n\n2. The study’s perspective and findings are intriguing, with well-designed experiments and thorough analysis that convincingly support the conclusions. \n\n3. It achieves good code-LLM perfermance by experimenting with pre-training data distribution.", "weaknesses": "1. The scope of this paper is a bit limited, as it focuses exclusively on code LLMs. Due to the unique characteristics of programming languages, the findings may not be easily applicable to natural languages.\n\n2. The scenario explored is somewhat simplistic, focusing mainly on extending a primary LLM (Python in this case) to additional languages. It remains unclear whether the hypothesis would still hold if a different primary language were used.\n\n3. Some parts of this paper are unclear to me:\n - In sections 4.2.1 and 4.2.2, it is not well explained how $P(l)$ relates to $\\alpha$ and $\\beta$ and how $P(l)$ determines the optimal number of tokens for the target languages. \n - Additionally, according to equation (2), the proportion $P(l)$ cannot exceed 1. What happens if there is sufficent amout of training data for the target language?\n - The hypothesis primarily addresses the extension of a dominant language to one additional language. However, in section 4.3.3, the strategy of training with five languages from scratch also appears to yield better performance. Why is this the case?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xcI4Eg9OUP", "reviewer": "Reviewer_cRyw", "summary": "This submission explores how Large Language Models (LLMs) develop multilingual capabilities. The authors propose a three-stage learning process: translation, transition, and stabilization. They experimented with code LLMs by tracking the working languages and language transferring neurons during pre-training.", "questions": "Why combine the concepts of multiple natural languages and multiple programming languages?\n\nNote: Authors should not reveal their identities. Descriptions like \"Specifically, we conducted preliminary experiments to analyze how monolingual LLMs learn a new language by continual pre-training the monolingual LLM on the corpus with the new language (Zheng et al., 2024)\" should be revised to remove identifying information.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The mechanisms behind the development of multilingual capabilities during pre-training is an important area.", "weaknesses": "* The submission's use of the term \"multilingual capabilities\" is unclear. It seems to refer to the model's capacity to understand multiple programming languages, rather than the more common interpretation of \"multilingual\" as understanding multiple natural languages. Since the programming languages studied are all English-based, the method doesn't address the orthographic differences of various natural languages. Instead, the experiments reduced to focus on how the semantics of programming languages are developed during pre-training.\n\n* The methodology's flaws undermine the hypothesis presented. The claim that multilingual programming language capabilities are primarily developed during pre-training is unconvincing. It's likely that fine-tuning plays a significant role, and a more comprehensive mechanism should be considered.\n\n* Additionally, the assertion that Python is the dominant language due to its widespread use lacks evidence and requires further clarification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "53VGgI2vsS", "reviewer": "Reviewer_4Qay", "summary": "This work focuses on LLM multilingual capability (in terms of programming language instead of natural language). The authors observed a 3-stage division during the continual learning setting (i.e. pre-train on Python, and continue pre-training on PHP/C#...): (1) translation stage, where the performance of the new language improves and the Python system is dominated in the LLM and the generation in the new language primarily relies on the Python system; (2) Transition Stage: the performance of the new language begins to decline while it gradually forms its own system; (3) Stabilization Stage: the performance of the new language stabilizes, and the generation in the new language depends on its own system. Based on such observation, the authors proposed the “Babel Tower Hypothesis” which depicts the language learning process in multilingual pre-training - a single knowledge system was first formed, followed by forming language-specific systems.\n\nTo support this claim, the authors used 2 methods to evaluate the internal state of the LLMs during pre-training, (1) working languages and (2) language transferring neurons. These metrics show similar 3-stage trends. Finally, the authors proposed to adjust the pre-training data distribution to make different languages to achieve optimal states concurrently. Empirical results show that this optimized data distribution outperforms the original data distribution in terms of the HumanEval and MBXP benchmarks.", "questions": "1. Section 4.3.3 states that the optimized pre-training data distribution is formed by “we randomly down-sampled the token quantity for PHP and C# to 5 billion and 10 billion tokens, respectively, while keeping the token quantities for the other languages unchanged ”. I’m quite surprised that training on a smaller amount of data provides such a big performance improvement in Table 2. What is the optimized data distribution?  The models are trained on the data collected by the authors themselves (as described in Section 3.1). Have you performed any checks on data quality? Is it possible that these results are actually due to some bad quality data (e.g. noisy/incorrect codebases), instead of the claimed knowledge transfer between languages? \n2. I think a better set of experiments may be to use existing open-source multilingual datasets, and show that the proposed optimized data distribution can produce stronger pre-train models. In this case the comparison can include existing coding LLMs. \n3. Further, programming languages are very different from natural languages due to their highly structured form. Based purely on programming languages, it is not clear whether the “Babel Tower Hypothesis” is generalizable or not. It may be more informative if there are more experimental results on natural languages to support this general hypothesis - though, I understand the difficulty due to resource demands for these experiments - a smaller scale of multilingual training data is understandable.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors investigated the training dynamics for different data setup, which is intriguing.\n2. I like the idea of checking LLM internal states using \"working languages\" and \"Language transferring Neurons\" concepts.\n3. Experiments are solid.", "weaknesses": "1. The presentation of Algorithm 1 and subsequent experiments and results in Section 4.3.1 and 4.3.2 needs some clarification. The last line in Algorithm 1 states that the number of tokens in the target language is computed using $P(l_j)$ and $\\eta_{python}$ based on Eqn 3. However, Eqn. 3 does not have a $P(l_j)$ term. \n2. Assuming the estimated num of tokens are obtained, I am still confused about the results in Fig. 6 and Fig. 7. For example, what do “PHP predicted” and “PHP Actual” mean? My understanding is that you are estimating the data proportion in pre-training corpora, not the performance on HumanEval. Why there are multiple results for multiple data proportions (i assume there should be an optimal one given by Algorithm 1.) And why there are much more “PHP predicted” dots than “PHP Actual”? I may have misunderstand something, but i think the presentation of these sections could be made clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)