Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resources
Authors: Zihao Li
Shaoxiong Ji
Hengyu Luo
Jörg Tiedemann
Venue: COLM 2025 (2025)
Keywords: Multilingual continual pretraining
data mixing
TL;DR: This study evaluates 36 continual pretraining configurations across three multilingual LLMs and 30+ languages, analyzing their effects across resource levels and language behavior categories.

Abstract: Large Language Models (LLMs) exhibit significant disparities in performance across languages, primarily benefiting high-resource languages while marginalizing underrepresented ones. Continual Pretraining (CPT) has emerged as a promising approach to address this imbalance, although the relative effectiveness of monolingual, bilingual, and code-augmented data strategies remains unclear. This study systematically evaluates 36 CPT configurations involving three multilingual base models, across 30+ languages categorized as altruistic, selfish, and stagnant, spanning various resource levels. Our findings reveal three major insights: (1) Bilingual CPT improves multilingual classification but often causes language mixing issues during generation. (2) Including programming code data during CPT consistently enhances multilingual classification accuracy and language modeling capabilities, particularly benefiting low-resource languages, but introduces a trade-off by slightly degrading generation quality. (3) Contrary to prior work, we observe substantial deviations from language classifications according to their impact on cross-lingual transfer: Languages classified as altruistic often negatively affect related languages, selfish languages show conditional and configuration-dependent behavior, and stagnant languages demonstrate surprising adaptability under certain CPT conditions. These nuanced interactions emphasize the complexity of multilingual representation learning, underscoring the importance of systematic studies on generalizable language classification to inform future multilingual CPT strategies.

Published as a conference paper at COLM 2025
Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resources
Zihao Li', Shaoxiong Hengyu Luo', Jörg Tiedemann' University of Helsinki 2ELLIS Institute Finland *University of Turku (zihao. 1i, shaoxiong.ji ji, hen,yu.luo, jorg. (eedemann@@helssiee fi
Abstract
Large Language Models (LLMs) exhibit significant disparities in perfor- mance across languages, primarily benefiting high-resource languages while marginalizing underrepresented ones. Continual Pretraining (CPT) has emerged as promising approach to address this imbalance, although the relative effectiveness of monolingual, bilingual, and code-augmented data strategies remains unclear. This study systematically evaluates 36 CPT configurations involving three multilingual base models, across 30+ languages categorized as altruistic, selfish, and stagnant, spanning var- ious resource levels. Our findings reveal three major insights: (1) Bilin- gual CPT improves multilingual classification but often causes language mixing issues during generation. (2) Including programming code data during CPT consistently enhances multilingual classification accuracy and language modeling capabilities, particularly benefiting low-resource lan- guages, but introduces trade-off by slightly degrading generation quality. (3) Contrary to prior work, we observe substantial deviations from lan- guage classifications according their impact on cross-lingual transfer: Languages classified as altruistic often negatively affect related languages, selfish languages show conditional and configuratioo--eppedde and stagmant languages demonstrate surprising adaptability under certain CPT conditions. These nuanced interactions emphasize the complexity of multilingual representation learning, underscoring the importance of systematic studies on generalizable language classification to inform future multilingual CPT strategies.
behavio vior,
1 Introduction
Large Language Models (LLMs), built upon the Transformer architecture (Vaswani et al., 2017), have achieved remarkable progress in tasks such machine translation, text classifi- cation, and generative dialogue. Despite these advances, their performance remains highly uneven across languages, favoring high-resource languages and marginalizing underrepre- sented ones (Li et al., 2024; Joshi et al., 2020; Choudhury, 2023). This imbalance deepens the digital language divide and limits the inclusivity of NLP technologies.
Recent work on Continual Pretraining (CPT) has shown promise for adapting pretrained models to new languages through additional training on targeted data (Zheng et al., 2024a). EMMA-500 employed CPT with extensive monolingual datasets across more than 500 languages, significantly improving multilingual performance, particularly for low-resource languages (Ji et al., 2024a). LLaMAX achieved notable translation improvements through CPT on over 100 languages involving data augmentation with bilingual translation data (Lu et al,, 2024). Similar effects could be demonstrated on translation tasks with the CPT-based TOWER model (Alves et al., 2024). However, the relative effectiveness of monolingual and bilingual translation data for CPT remains unclear, particularly in terms of their impact continual language learning, language interference, and performance consistency across different resource levels of languages.
*Corresponding author. Work done while at the University of Helsinki.

---

Published as a conference paper at COLM 2025
In addition to textual data in natural languages, growing practice in LLM training is to incorporate programming code. For multilingual models, the inclusion of code, with its distinct logical syntax and lower ambiguity compared to natural language, can diversify the training data. We hypothesize that this diversity compels the model to learn more general and robust representations, moving beyond lpnuuae--peccffic surface patterns; thus improving both the model's 's predictive ability and its performance on downstream tasks,
There is a critical gap in understanding how language characteristics interact with CPT strategies. A recent classification proposed by Yuan et al. (2024) categorizes languages as altruistic, selfish, and stagnant based on their cross-lingual transfer patterns. This framework classifies languages based on their impact in multilingual training: altruistic languages are hypothesized improve performance on related languages, selfish languages primarily benefit themselves, and stagnant languages offer little to no benefit to any language, includ- ing their own. However, this classification has only been validated in narrow experimental settings using English-ceetrric bilingual data, leaving open questions about its generalizabil- ity to: (1) non-English language pairs, (2) code-augmented training regimes, and (3) models with varying pretraining corpora and architectures.
To systematically assess the impact of different CPT strategies, we conduct extensive experi- ments with 36 configurations, evaluating monolingual, bilingual, and code-augmented CPT on multilingual adaptation." Our setup includes three multilingual base models-Llama 3.1-8B (Dubey et al., 2024), Llama-2-7B (Touvron et al., 2023), and Viking-7B (Luukkonen et al., woaninoollppeeeraaaaaaaaa categories. We evaluate the model performance on 14 training languages and assess cross- lingual transfer on 25 related languages, with a particular focus on assessing how different CPT configurations perform across altruistic, selfish, and stagnant language categories by Yuan et al. (2024).
languages spanning high-, medium-, and low-resource
Our systematic evaluation of 36 CPT configurations across three base models and 30+ languages yields three core insights:
Bilingual CPT improves classification performance but introduces generation challenges: Compared to monolingual CPT, bilingual CPT generally improves S classification accuracy for medium- and low-resource languages. How- ever, it frequently results in problematic language mixing during generation tasks, limiting its overall utility.
multilingual
Code data enhances classification but introduces trade-offs in generation: Adding code data during CPT significantly boosts multilingual classification performance across resource levels, especially for low-resource languages, acting as an effective scaffold for representation learning. Nevertheless, code inclusion may lead to trade-off, slightly degrading generation quality in certain scenarios.
The categorization of languages according to their cross-lingual transfer abilities does not generalize under varying conditions: Our experiments reveal substantial deviations from language classifications proposed in previous work (Yuan et al., 2024): so-called altruistic languages are not always helpful and often negatively impact related languages, selfish languages exhibit highly configuration-dependee cross-lingual effects, and languages classified as stagnant demonstrate unexpected adaptability under specific training settings. These findings highlight the complex- ity of multilingual interactions in CPT and emphasize the need for more adaptive classification framework for cross-lingual learning.
Monolingual data consists of texts in a single language, though may include code-switching. Bilingual translation data contains sentence pairs in two languages that convey the same meaning. When monolingual data from different languages is combined, it forms multilingual continual per- taining, and similar principle applies to bilingual translation data. However, for clarity, we refer to these setups as monolingual CPT and bilingual CPT, respectively.
2

---

Published as a conference paper at COLM 2025
2 Materials and Methods
2.1 Language Selection
We systematically evaluate the effects of CPT on multilingual models by selecting languages according to the altruistic, selfish, and stagnant categories defined in Yuan al. (2024), which classify languages based on their behavior in multilingual training and evaluation.
For each category, we select 1 high-resource language (except for the stagnant category, for which no high-resource language is available in the dataset we use), 2 medium-resouree languages, and 2 low-resource languages to ensure a balanced representation across different resource levels. The classification of languages into high-, medium-, and low-resource categories is determined by analyzing the data distribution of the Lego-MT dataset (Yuan et al., 2023), which serves as the basis for our setup. Specifically, we calculate the total token count for each language. Languages are then categorized as follows: high-resource languages exceed billion tokens, medium-resouree languages range between 10 million and 1 billion tokens, and low-resource languages fall below 10 million tokens. These languages serve as the training languages in our CPT experiments. The selected training languages, along with their corresponding category and resource level, are summarized in the first three columns of Table 1.
To further validate the findings in Yuan et al. (2024), we select 1-2 linguistically related languages for each training language based on the language evolutionary tree These related languages are not included in the CPT phase but are used for cross-lingual evaluation to determine whether the effects observed in training languages extend to unseen but related languages. The fourth and fifth columns of Table 1 list the selected related languages. For some languages, this includes one, and for others, two related languages that are available in the evaluation benchmarks.
Category
Resources
Training Language
Related Language
Related Language
High Medium Medium
zho_Hani ceb Latn
yue_Hant
tgl Latn
ilo Latn
Altruistic
mar Deva
hin Deva
npi Deva
Low
zul Latn
xho Latn
ssw Latn
Low
khm_Khmr
vie_latn
High
deu-Latn
Lad_
dan_Latn
Medium
bel bel.Cyrl
rus rus Cyrl
ukr ukr.Cyrl
Selfish
Medium
mri.Latn
smo Latn
fij.Latn
Low
kir.Cyrl
kaz.Cyrl
bak.Cyrl
Low
nya.Latn
bem.Latn
sna.Latn
Medium
tha_Thai
lLo_Laoo
Mym.
Medium
Lat_
ibo Latn
Latn
Stagnant
Low
sna_
nya Latn
zul Latn
Low
wol Latn
bam
Table 1: Selected languages for CPT along with their corresponding related languages for evaluation. '_' indicates the second related language cannot be found in the benchmark.
2.2 Pretraining Data
Bilingual Translation Data We utilize subsets of the Lego-MT (Yuan et al., 2023) and NLLB (Schwenk et al., 2021; Heffernan et al., 2022; Costa-jussà et al., 2022) datasets as our sources of parallel bilingual data. The Lego-MT dataset, derived from OPUS4 provides translations across 433 languages. The NLLB dataset consists of 148 English-centric and
2http: //www///wwwwoo/ http:://wwwelinnniiee.e elinguistics.
html
Using the language evolutionary tree identify related languages, we assess whether CPT effects transfer to unseen but linguistically similar languages, thus evaluating cross-lingual robustness.
hhttps://opus http:::////ps..pp..eee nlpl.eu
3

---

Published as a conference paper at COLM 2025
1,465 non-Engliih--centric bitext pairs mined from different parallel sources. To construct our parallel training data, we select specific language pairs from these datasets and apply OpusFilter (Aulamo et al., 2020) to remove duplicate data points.
The resulting dataset comprises approximately 292 million tokens across 22 language pairs, distributed over three language categories: altruistic (10 pairs, "92M tokens), selfish (8 pairs, "100M tokens), and stagnant (4 pairs, "100M tokens).
For training, we format parallel data using the following structure:
[source language]:
[source]
[target language]: [target]
Monolingual Data We extract a subset of MADLAD-400 (Kudugunta et al., 2024),
large-scale multilingual dataset derived from Common Crawl', covering 419 languages. Since web-crawled text does not inherently guarantee monolingual integrity, we employ GlotLID (Kargaran et al., 2023), language identification model, to analyze the language composition of each text segment and ensure strict monolingual consistency. Specifically, for each document in the dataset, we first segment the text into sentences using the NLTK (Bird Loper, 2004) sentence splitter. Then, GlotLID predicts the language of each sentence independently. We retain only those documents where all sentences are identified as belonging to the same language, discarding any text segment that exhibits code-switching or multilingual content.
We finally select data for 15 unique languages, with English appearing in all three categories: altruistic (6 languages, "92M tokens), selfish (6 languages, "100M tokens), and stagnant (5 languages, "87M tokens), resulting in a total of approximately 279 million tokens.
Code Data We incorporate code data from The Stack (Kocetkov et al., 2022), following the pre-processing strategy used in EMMA-500 (īi et al., 2024a). The dataset is first filtered to retain high-quality source files, with a focus on data science-related code and the 32 most commonly used general-purpor programming languages. Additionally, we include LLVM code due to its importance in multilingual code generation (Paul et al., 2024; Szafraniec et al., 2022).
For training configurations that include code, we establish a 2:1 ratio between textual (100 million tokens) and code data (50 million tokens), with code comprising about 33% of the total mixture. We intentionally selected this higher proportion to investigate the effects of a strong code signal within our CPT context. While prior work (Aryabumi et al., 2024) on pretraining from scratch recommends 25% proportion for balancing language and code performance, it is noted that higher 33% remains reasonable for enhancing models' ability, aligning with our goal of studying its trade-offs.
2.3 Base Models
We evaluate across three open-source multilingual LLMs with diverse training recipes:
Llama-3.1-8B (Dubey et al., 2024) is pretrained on approximately 15 trillion tokens from diverse, multilingual sources. Its extensive multilingual pretraining and high capacity make ideal for analyzing CPT effects on well-trained models.
Llama-2-7B (Touvron et al., 2023) is pretrained on 2 trillion tokens, covering broad yet less multilingual data distribution. It provides a baseline to evaluate CPT effectiveness on English-centric models commonly used in multilingual adaptation research.
Viking-7B (Luukkonen et al., 2025) is pretrained mainly on Nordic languages, English, and code, offering insights into how CPT impacts models initially trained on narrower, region-specific data.
9nttps://commoncaall..
4

---

Published as a conference paper at COLM 2025
2.4 CPT Configurations
We train models under CPT configurations across 3 base models and 3 language categories, resulting in total of 36 models. Each model is named using the format:
Model--Datt(-CCodeeLLanngC
where:
Model E {L3 (Llama-3.1-8B), L2 (Llama-2-7B), V7 (Viking-7B))
Data E (Mono (Monolingual), Bi (Bilingual))
Code (optional) is added if code data included
LangCat E [Alt (Altruistic), Sel (Selfish),,tee Stag (Stagnant))
For example, L3- 1.3-Mon-AAlt refers to 3lama-33111 3-B trained on monolingual data for altruistic languages, while 12-Bi+CCoe--S5 denotes Llama-2-7B trained bilingual parallel texts in selfish languages and code data.
Each model is trained for epochs on a cluster with AMD MI250X GPUs (8 Graphics Compute Dies) on each node. Training data is organized by language category (altruistic, selfish, stagnant), with all languages within category (e.g., altruistic: zho Hani, ceb ceb_Latn, etc.) mixed into single dataset per configuration (e.g,, monolingual, bilingual+code). As for software, we use the LLaMo-Factory (Zheng al., 2024b) framework with DeepSpeed (Ra- jbhandari et al., 2020) ZeRO-3 config. The hyperparameter setup includes per-device batch size of 8 with gradient accumulation steps of We use a cosine learning rate scheduler with an initial learning rate of 4.0 10-5 and a warmup ratio of 0.03.
3 Evaluation and Discussion
3.1 Benchmarks and Setup
We evaluate our models two highly multilingual benchmarks covering classification and generation task: SIB-200 (Adelani et al., 2024) for topic classification and FLORES- 200 (Costa-jussà et al., 2022; Goyal et al., 2022; Guzmán et al., 2019) for machine translation. Classification focuses on whether CPT improves the multilingual model's understanding within single language, while translation studies the alignment between languages that emerges with multilingual CPT. All experiments use consistent 3-shot prompting setup.
SIB-200 STB-200 is multilingual news topic classification benchmark covering 200 lan- guages. The task involves classifying news headlines into one of the following predefined categories: science/ /teehnolochh, travel, politics, sports, health, entertainment, and geogra- phy.
The model predicts by ranking logits for each category, and accuracy measures performance across languages.
FLORES-200 FLORES-200 evaluates multilingual translation performance across diverse language pairs.
Translations are generated using the vLLM (Kwon et al., 2023) inference engine. BLEU (Pap- ineni et al., 2002) scores, computed via SacreBLEU (Post, 2018) with the flores200 tokenizer, quantify translation quality.
3.2 Effect of Monolingual and Bilingual Continual Pretraining
This section shows that bilingual CPT hampers generation due language mixing but excels in classification for medium- and low-resource languages over monolingual CPT.
6L EU signature: nrefs: casee mixed eeffeeeddff..ooe eff
tok: flores200 smooth: expl version:2.4.2

---

Published as a conference paper at COLM 2025
3.2.1 Language Mixing in Generation Tasks
The FLORES-200 translation task revealed significant language mixing issues in models trained with bilingual translation data. Specifically, when generating translations between language pairs, models frequently appended unintended language tokens the output. For example, when translating from English (eng Latn) to Chinese (zho Hani), models trained on bilingual data produced outputs like:
"Rhm & & A8 A 4., EHREABE &
Marathi: सपनन मपोे होता.
भात, अाय पर मीििपा अबर आहेत. समाा बारी FERe शालाा हीरा. qu भाता नाही. कहरप पारील yep रोद पदपा साकर कमीी कररप.
The text with green background represents the desired Chinese translation, while the text with an orange background contains nonsensical multilingual fragments. This phenomenon occurred consistently across bilingual CPT configurations, suggesting that the parallel data format ([Lang1]: xxx [Lang2]: yyy) encourages cross-lingual interference. More examples are in Figure 6 in Appendix A.6.
This language inconsistency leads to significant translation quality degradation, shown in Figure 1. Bilingual CPT configurations underperform monolingual CPT across all re- source levels and base models. For high-resource languages, Llama-3.1-8B achieves only 7.47 BLEU with bilingual CPT versus 25.52 with monolingual CPT (-71% relative), while Llama-2-7B shows similar disparities (14.12 vs 24.60, -43%). The pattern persists for mid- and low-resource languages, with bilingual CPT consistently lagging behind monolingual CPT. Notably, monolingual CPT often matches or exceeds baseline performance, whereas bilingual CPT only exceeds baseline in specific cases, such Llama-2-77 low-resource languages. Appendix A.5 presents the detailed results on each language.
mid- and
Data Canderatten
High Reseurce
Mid Resource
Low Resource
25
BLEU
Vasigr
Vkirgit
Figure FLORES-200 X-Eng BLEU score comparing bilingual and monolingual CPT across high- mid-, and low-resource languages.
3.2.2 Comparative Analysis in Classification Tasks
To isolate the effects of CPT strategies without interference from language mixing, we evaluate SIB-200 classification accuracy. Figure shows the average accuracy aggregated across models trained separately on altruistic, selfish, and stagnant languages, grouped by resource level.
High-Resource Languages For high-resource languages, both monolingual and bilingual CPT degrade performance across all base models compared to their respective baselines. Llama-3.1-8B, despite its strong baseline (76.63%), exhibits drops with bilingual CPT (71.4 11.41% ,,, -6.8% relative) and monolingual CPT (64.21%, -1..2%) Llama-2-7B shows significant declines with both strategies: bilingual CPT reduces accuracy to 31.54% (vs baseline 37.75%, -6.55)), while monolingual CPT performs similarly (31.86%, -15.6%). Viking-7B partially escapes this trend, with bilingual CPT achieving marginal gains (36.60% vs baseline 34.80%, +5.2%), though monolingual CPT underperforms (25.98%, -25.3%). This suggests that high-resource languages generally do not benefit from CPT, likely due to interference with existing strong representations in pretrained models. However, model-specific factors, such whether the

---

Published as a conference paper at COLM 2025
Conderanion Mecwinges
0.21
Bane Nodel
Figure 2:
SIB-200 classification accuracy comparing monolingual and bilingual CPT across high, mid-, and low-resource languages.
model's pretraining data aligns well with the target languages in CPT, may enable limited improvements in certain cases. For example, Viking-7B, which pretrained primarily Nordic languages and English, may benefit more from bilingual CPT due to its ability to leverage cross-lingual transfer between related languages.
Mid-Resource Languages Mid-resource languages show mixed trends. Llama-3.1-8D
maintains near-baseline performance with bilingual CPT (64.05% vs baseline 63.85%, 2.7%), but monolingual CPT degrades significantly (52.53%, -00.2)). Llama-2-7B improves accuracy in both configurations: bilingual CPT raises accuracy to 24.26% (vs. baseline 20.59%, +17.8%), while monolingual CPT performs slightly better 26.80% (+30.2%). Viking-7B uniquely benefits from bilingual CPT (29.74% vs baseline 23.94%, +24.2%), while monolingual CPT underperforms (24.02%, +0.3%). This indicates that bilingual CPT can stabilize mid mid-resource language performance for certain models (e.g., Viking-7B and Llama- 2-7B). However, monolingual CPT risks overfitting to limited in-language data, particularly for models with weaker pretraining (e.g., Llama-3.1-8B).
Low-Resource Languages Low-resource languages exhibit divergent patterns. Llama- 3.1-8B improves with bilingual CPT (62.91% vs baseline 56.86%, +10.6%) but declines with monolingual CPT (52.04%, -8.5)). Llama-2-7B improves moderately with bilingual CPT (20.84%, +19.8%) and shows stronger gains with monolingual CPT (24.51%, +40.9%). Viking- 7B benefits substantially from bilingual CPT (29.25%, +66.5%), while monolingual CPT slightly underperforms (21.33%, +21.4%). This highlights that bilingual CPT can enhance low-resource language performance for models with compatible pretraining (e.g., Viking-7B and Llama-3.1-8B..
3.3 Effect of Including Code Data
3.3.1 Downstream Tasks Performance
The integration of code data during monolingual CPT shows task-dependent effects, en- hancing classification performance while introducing tradeoffs in generation quality. Figure 4 and Figure 3 compare monolingual CPT with and without code data across resource levels and tasks, revealing key patterns in how code data influences multilingual adaptation.
Code integration consistently improves classification accuracy across all resource levels and models. For high-resource languages, Llama-3.1 shows marginal gains (64.21%, to 68.4 88.47, +6.7% relative to baseline 76.63%), while Llama-2-7B and Viking-7B exhibit more substantial improvements (42.48% vs 31.86%, +33.3%; 30.88% vs 25.98%, +18.8%). Mid-resource lan- guages benefit even more, with Llama-3.1-80 recovering near-baseline performance (52.53% to 62.83% -4.6% vs baseline 65.8 55.5%) and Llama-2-7B achieving significant gains (34.40% vs 26.80% (67.0%). Low-resource languages see the most pronounced improvements, partic- ularly for Viking-7B (28.68% vs 21.33% +63.2% relative to baseline 17.57%). This pattern extends to bilingual CPT configurations (see Appendix A.3).
7

---

Published as a conference paper at COLM 2025
Consitatioon Pessdrgal--Ge
20
12.0
18.22
3149
19:3
2211
Bace
Figure 3: FLORES-200 X-Eng BLEU score comparing monolingual and monolingual+codee CPT across high-, mid- and low-resource languages.
Consinuaioon Paasdrrgal--G
0,3
A.V
Bane
Figure 4: SIB-200 classification accuracy comparing monolingual and monolingual+code CPT across high-, mid-, and low-resource languages.
In contrast, code integration often degrades translation quality,, particularly for high-resource languages. Llama-3.1-8) shows slight degradation (25.52 BLEU to 25.35, -0.7% baseline 27.97) while Llama-2-7B and Viking-7B exhibit gains (25.05 vs 24.60, +8.6%; 11.69 vs 9.18, +27.3%). Mid-resource languages show mixed trends, with Llama-3.1-80 experiencing slight drop (17.62 vs 18.59, -5.2%) and Viking-7B improving significantly (4.21 vs 3.58, +5..0%). Low-resource languages partially escape this trend, with Viking-7B showing substantial gains (2.84 vs 2.31, +37.4%).
The benefits of code integration are most pronounced for low-resource languages, where acts as a "scaffold" to improve classification accuracy (avg. +25.1%) and partially mitigate generation deficits. Mid-resource languages also benefit, though to a lesser extent, while high-resource languages see diminishing returns, with classification gains (e.g., Llama-3.1- 8B: +6.7%) offset by generation losses.
3.3.2 Language Modeling Performance
To assess the impact of code integration on language modeling, we evaluate performance using Negative Log-Likelihood (NLL) on the MaLA validation set?. We use the implemen- tation of GlotEval (Luo et al., 2025), where the NLL is calculated by concatenating input sentences and using strided sliding window of 1024 tokens, and is defined as
"
NLL Σ logpo(x)(x=1).
Our results are summarized in Table 2. Across all three base models and all resource levels, the Monolingual+Code configurations yielded lower (better) NLL scores than the monolingual-only CPT. This demonstrates that including code not only helps in downstream classification tasks but also improves the models' predictive ability and likelihood estimation of unseen text during CPT.
8

---

Published as a conference paper at COLM 2025
Model
High Resource
Mid Resource
Low Resource
L2-Mono-
5.0080
1.7609
4.3308
L2 2--Mono+Code-
4.7359
1.7042
4.2289
L3-Mono-
4.8864
2.0375
4.2013
3--Mon+CCode-
4.5701
1,9522
4.1389
V7-Mono-
6.9916 6.6029
1.9506 1.8369
4.5367 4.4850
V7-Mono+Code-
Table 2: NLL across different models and resource levels. Lower is better.
3.4
Validation of Language Category Hypotheses
This section evaluates the validity of the altruistic, selfish, and stagnant language classifica- tions proposed in prior work (Yuan et al., 2024). We evaluate each model (e.g., L3-Mono-Alt) trained language category (e-g., altruistic languages: zho_ ceb_Latn, etc.) and measure SIB-200 classification accuracy changes on both the trained languages and their related languages (e.g., yue Hant, tgl_Lätn, etc.), defined in Table We analyze whether CPT strategies align with these hypothesized behaviors. Table 3 reports accuracy changes (%) relative to base models.
Altruistic Languages Related
Selfish Languages Related
Stagnant Languages Related
Model
Training
Met?
Training
Met?
Training
Met?
L2-Bi-
+7.08
-22.55
+12.33
+2.90
Yes
+5.88
-9.99
No
Lo-Bi-Code--
+62.37
No
No
L2-Mono-
-14.60
-31.32
No
+53.18
+21.94
No
+31.36
-8.33
No
L2- 12-Mono+Code
+50.43
+19,04
No
+52.32
+26.29
No
+64.02
+14.57
No
L3-Bi-
+4.66
-.46
No
-7.90
-19.54
No
+14.76
-28433
No
L3-Bi=Code-
+1.64
-7.70
No
-5.85
-15.66
No
+21.81
-28.04
No
L3-Mono-
-24.37
-31,26
-9.07
-19.84
No
-33.44
No
13-Mono+Code-
-1.78
-11.13
No
+2.49
-10.85
No
0,00
33.01
No
V7-Bi-
-11.41
-31.95
No
+19,44
-10.22
No +78.18
+26.32
No
V7-Bi+Code-
422.82
-9.35
No
+17.57
-16.35
No
+11.16
-19.36
No
V7-Mono-
-8.22
-19.74
+5.86
-33.45
No
0.00
-0.83
Yes
V7-Mono+
+5.93
-11.69
No
+53.96
+8.18
Yes
+21.31
+17.27
No
Table 3: SIB-200 classification accuracy changes (%) for training and related languages across altruistic, selfish, and stagnant categories. Results reported relative to base models, with a "Met" column indicate whether the hypothesis is met or contradicted.
Altruistic languages can also be selfish mutually harmful The altruistic hypothesis indicates that training in altruistic languages enhances multilingual performance (related languages) with minimal impact their own performance (trained languages). Our results reveal three critical contradictions: (1) 83% of configurations (10/12) degraded related language performance, with code-free CPT causing up to -31.32% accuracy (L2-Mono-AA (2) Contrary to "minimal self-impact", trained language accuracy fluctuated wildly (+62.37% in L2-Bi+Code-Alt vs. -24.37% in L3-Mono-Alt); These bidirectional effects challenge the unidirectional altruism assumption.
Selfish languages exhibit conditional isolation only in certain cases While the selfish hypothesis suggests trained languages primarily improve their own performance (trained languages) while minimally affecting others (related languages), we find this only holds in specific configurations: (1) Non-code bilingual training (L2-Bi-Sel) showed minimal impact related languages (+2.90%); (2) Code-augmented monolingual training (V7- Mono+Code-Sel] achieved strong self-improvement
++53.96%) with moderate spillover (+8.18%). However, 83% of cases (10/12) violated the hypothesis through either nega- tive spillover (V7-Mono-Sel: -33.45%) or excessive cross- lons-lingual transfer (L2-BiiCode--e +31.67%).

---

Published as a conference paper at COLM 2025
Stagmant languages demonstrate more adaptability than expected Stagnant languages neither improve their own performance (trained languages) nor influence others Contrary to their purported stagnation, 92% of configurations (11/12) induced significant performance shifts: (1) Bilingual training boosted trained languages by +78.18% (V7-Bi-Stag) while improving related languages (+26.13%) (2) Monolingual+code CPT (12-Mono+Code (oono+Code-Stag) achieved +64% self-improvement with +14.76% cross-lingual gains. Only V7-Mono-Stag showed true stagnation (+0.00% trained, -0.83% related). This reveals that most "stagnant"
languages possess untapped adaptation potential under proper CPT strategies.
4 Conclusion
In this study, systematically evaluated the effects of multilingual CPT strategies, includ- ing monolingual, bilingual, and code-augmented configurations, across diverse resource levels and language categories. Through experiments with 36 configurations involving three multilingual base models and over 30 languages, we identified several critical insights:
First, while bilingual CPT enhances classification accuracy for mid- and low-resource lan- guages, it introduces language mixing during generation, limiting its utility for translation tasks. Second, code integration during CPT acts as a scaffold for low-resource language understanding but introduces lask-dependent trade-offs, improving classification and lan- guage modeling capabilities while slightly degrading generation quality. Third, we demon- strate that language classifications based on tross-lingual transfer patterns (altruistic, selfish, stagnant) fail generalize under varying CPT strategies.
Overall, our work underscores the complexity of multilingual representation learning and highlights the need for flexible frameworks for language categorization and training strategy selection. Future research should focus on developing more adaptive CPT methods that balance classification improvements and generation quality, further bridging language disparities in large language models.
Ethics Statement
This research focuses on reducing the digital language divide and improving inclusivity for underrepresented languages. We acknowledge potential biases due to uneven data distribu- tion and strive mitigate them by including diverse languages across resource levels. All datasets used are publicly available and preprocessed to ensure integrity and monolingual consistency. All the models trained in this paper are strictly for research purposes and are not intended to be deployed in real-world applications. We encourage further work to address ethical challenges in multilingual NLP, especially for underrepresented languages.
Reproducibility Statement
To ensure reproducibility, we release:
Model Checkpoints: All models trained under various configurations (monolin- gual, bilingual, code-augmented) across base models (Llama-3.1-8P Llama-2-7B, Viking-7B) and language categories (altruistic, selfish, stagnant).
Processed Dataset:
Filtered subsets of Lego-MT,, NLLB, MADLAD-400, and code
data.
Scripts: Data cleaning, training, and evaluation scripts, including LLaMA-Factory with DeepSpeed ZeRO-3 configuration.
All resources are available at: https:/ //maaa/mmmaaaaa Ma-aa-lla 1m. wttuub.o//MxCCP.hhe..
10

---

Published as a conference paper at COLM 2025
Acknowledgments
The work has received funding from the Digital Europe Programme under grant agreement No 101195233 (OpenEuroLLLM). The authors wish acknowledge CSC IT Center for Science, Finland, for providing computational resources.
References
David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, and En-Shiun Lee. SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapler of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 206-2262555 St. Julian's, Malta, March 2024. Association for Computational Linguistics. URL eacl-long. 14/.
Duarte M Alves, José Pombal, Nuno M Guerreiro, Pedro H Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, et al. Tower: An open multilingual large language model for translation-related tasks. arviv preprint arXio:2402.17733, 2024.
Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. To code, or not to code? exploring impact of code in pre-training. Xrv preprint ar arXii:2408..199 2024.
Mikko Aulamo, Sami Virpioja, and Jörg Tiedemann. OpusFilter: A configurable par- allel corpus filtering toolbox. Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics: System Demonstrations, pp. 150-156. Association for Computational Linguistics, July 2020. doi: 10.18653/v1 20t.://000aaaammoosso.....o URL
https://www. aclweb. org/ hatp/nnhhllly///0 wartthooogy/222,,, wtw.:///.0.....mmooss...... acl-demos. 20.
Steven Bird and Edward Loper. NLTK: The natural language toolkit. In Proceedings of the
ACL Interactive Poster and Demonstration Sessions, pp. 214-217, Barcelona, Spain, July 2004. org/P04-3031/
Association for Computational Linguistics. URL https: htttts::/aclanthoolog/
Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Ben Bergen. When is multilinguality curse? language modeling for 250 high- and low-resource languages. Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Pp. 4074-4096, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653 /2t/////222.eemmmmmaaaa URL hacpss//llllaa org/ 2024. emnnp-main 236/
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. ar Xiv preprint ar Xiv:2100 003744, 2021.
Monojit Choudhur.. Generative has a language problem. Nature human behaviour, 7(11): 1802-1803, 2023.
Marta R Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. Xiv preprint ar Xiv:2200 04462,, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint or Xix::40:7.222 21783,, 2024.
Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki lida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. Continual pre-training for cross-lingual Ilm adaptation: Enhancing japanese language capabilities. arvio preprint arXit:2404.17790 2024.
11

---

Published as a conference paper at COLM 2025
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522-538, 2022. doi: 10.1162/
@idla__000 aicl_a004744 URL https: httpss//aaaaaaaahacco httphhoolgg.org//22222222 t.
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philiipp Koehn, Vishrav Chaudhary, and Marc' Aurelio Ranzato. The FLORES evaluation datasets for low- resource machine translation: Nepali-English and Sinhala-English. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 6098-6111, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1 v1/D19-1632. URL http:://acclanntoooogooro//////
Kevin Heffernan, Onur Çelebi, and Holger Schwenk. Bitext mining using distilled sentence representations for low-resource languages. Xrv preprint ar Xri::2205.12655, 2022.
Masanori Hirano and Kentaro Imajo. The construction of instruction-tuned Ilms for finance without instruction data using continual pretraining and model merging. ar preprint wrXit:2409.19854 2024.
Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyán O'Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, et al. Emma-500: Enhancing mas- sively multilingual adaptation of large language models. arviv preprint arXiv:2409.17892, 2024a.
Shaoxiong Ji, Timothee Mickus, Vincent Segonne, and Jörg Tiedemann. Can machine trans- lation bridge multilingual pretraining and cross-lingual transfer learning? In Proceedings of LREC-COLING, 2024b.
Shaoxiong Ji, Zihao Li, Jaakko Paavola, Indraneil Paul, Hengyu Luo, and Jörg Tiedemann. Massively multilingual adaptation of large language models using bilingual translation data. arv preprint ar Xix::22006 004999 2025.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 0282-6293 Online, July 2020. URL
Association for Computational Linguistics. doi: 11116663//11222.acll/ https://aclantholl.
2rg/2020 acl maainmma.n.6//
Mihir Kale, Aditya Siddhant, Rami Al-Rfou, Linting Xue, Noah Constant, and Melvin Johnson. nm15 - is parallel data still relevant for pre-training massively multilingual language models? In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 683-691, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1
URL https:///ccl...c. an thology. , org/2021. acl short......8 877/
Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schuetze. GlotLII: Language identification for low-resource languages. In Houda Bouamor, Juan Pino, and Kalíka Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 6155-6218, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10.18653 2202..iindings-emmll. 410. URL https://acllanthology wrwp//23........... findings-emmlp.415 410/.
Najoung Kim, Sebastian Schuster, and Shubham Toshniwal. Code pretraining improves entity tracking abilities of language models. arXiv preprint arXiv:2405.2106 2024.
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. The stack: 3 tb of permissively licensed source code. aixiv preprint 2XXX::2211.15533 2022.
12

---

Published as a conference paper at COLM 2025
Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. Madlad-400: A multilingual and document- level large audited dataset. Adoances in Neural Information Processing Systems, 36, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Sympossuum on Operating Systems Principles, 2023.
Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ali Payani, Ninghao Liu, and Mengnan Du. Quantifying multilingual performance of large language models across languages. arviv preprint ar Xixi:2404.111 11553, 2024.
Peiqin Lin, André FT Martins, and Hinrich Schütze. A recipe of parallel corpora exploitation for multilingual large language models. arXiv preprint ar 0tt::2407.004336 2024.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficcent fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1965, 2022.
Yinquan Lu, Wenhao Zhu, Lei Li, Yu Qiao, and Fei Yuan. LLaMAX: Scaling linguistic horizons of LLM by enhancing translation capabilities beyond 100 languages. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 10748-10772, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/vl /v653/vl/// h005/v112200 2024..indings-eeee 631. URL https: / http:///aaaaahhlhoooo/oo 202/ findings-emn1p 631/.
Hengyu Luo, Zihao Li, Joseph Attieh, Sawal Devkota, Ona de Gibert, Shaoxiong Ji, Peiqin Lin, Bhavani Sai Praneeth Varma Mantina, Ananda Sreenidhi, Raúl Vázquez, Mengjie Wang, Samea Yusofi, and Jörg Tiedemann. Gloteval: A test suite for massively multilingual evaluation of large language models. arXiv preprint 2504.04155, 2025. URL https::/arritt org/abs/2504. 04155.
Risto Luukkonen, Jonathan Burdge, Elaine Zosa, Ville Komulainen, Peter Sarlin, and Sampo Pyysalo. Viking: A family of nordic 1lms, 2025. URL https://huggingfaaee www.oo/m/oooaoocccc O Viking-338.
Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shan- shan Li. At which training stage does code data help Iims reasoning? ar arv preprint ar arXir:3309116299 2023.
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. ar preprint orXin:2210.07128, 2022.
Mitodru Niyogi and Arna Bhattacharya. Paramanu-ayn: Pretrain from scratch or continual pretraining of 1ims for legal domain adaptation? Xio preprint +xXi:2440311388, 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Indraneil Paul, Goran Glavaš, and Iryna Gurevych. IRCoder: Intermediate representations make language models robust multilingual code generators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15023-15041, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.1865//112022..cd/ 802. URL https: htatps::/aacantten http:////aaaaanooooooorrroe
acl-long -ong 802/.
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebastian Ruder. Mad-x: An adapter-based framework for multi-task cross- cross-lingual transfer. ar preprint xfxie=2000500005,,20000 2020.
Matt Post. A call for clarity in reporting bleu scores. Xixiv preprint arXito1804.08871,
13

---

Published as a conference paper at COLM 2025
Vitaly Protasov, Elisei Stakovskii, Ekaterina Voloshina, Tatiana Shavrina, and Alexander Panchenko. Super donors and super recipients: Studying cross-lingual transfer between high-resource and low-resource languages. In Atul Kr. Ojha, Chao-hong Liu, Ekaterina Vylomova, Flammie Flammie Pirinen, Jade Abbott, Jonathan Washington, Nathaniel Oco, Valentin Malykh, Varvara Logacheva, and Xiaobing Zhao (eds.), Proceedings of the Seventh Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMi 2024), pp. 94- 108, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10.18653/ v1 wttpw//....eeeeeeee 1o0.s.oo.em URL https: / /aa:://aclllaa http://aaaaanoooggy.yoooo 2rg//024.00 2ttg/222....rrmtt...... 1.10/.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl, 2019. URL https://col openai..pp....... unsupervised multitask learners. pdd.
Sneetter-lnnuagee modelso poooelsslanggugeee http://nnaaaemmellssaaaaa models_are.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networkingg Storage and Analysis, pp. 1-16. IEEE, 2020.
Leonardo Ranaldi, Giulia Pucci, and Andre Freitas. Empowering cross-lingual abilities of instruction-tuned large language models by translation-folloiing demonstrations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 7961-7973, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ www.....iniinss-aaea https: / http:://aclanthooogyy..o / findings-acl. 473/.
URL
Holger Schwenk, Guillaune Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. CCMatrix: Mining billions of high-quality parallel sentences on the web. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6490-6500, Online, August 2021. Association for Computational Linguistics. doi: 10.18633/112222..cc-/ 507. URL https: http::/aallaaattaa /aclsthhoo/l . org/ 2021 acl- acl-long.er 507/.
Marc Szafraniec, Baptiste Roziere, Hugh Leather, Francois Charton, Patrick Labatut, and Gabriel Synnaeve. Code translation with compiler representations. arXiv preprint drXXi:2207.03578, 2022.
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. Multilingual translation from denoising pre-training. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3450-3466, Online, August 2021. URL
Association for Computational Linguistics. doi: 10. 10.18653/ /12///11// /tt/22222.222ns-/aaae https://aclanthologg org/2021, ..l.ffiiiii findings-acl. 304/.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ar preprint arXiv:2307. 009288, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural informa- fion processing systems, 30, 2017.
Anh-Dung Vo, Minseong Jung, Wonbeen Lee, and Daewoo Choi. Redwhale: An adapted korean Ilm through efficient continual pretraining. arx preprint ar arXiv:2408.1122 11294 2024.
Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. On negative interference in multilingual models: Findings and a meta-learning treatment. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Pp. 4438 4450, Online, November 2020. Association for Computational Linguistics. doi: 10.18653 vv1 /t//////220ee//// 112200.mnlpmmmai..3.. URL https:// acl aclanthology org. emnlp-main. 359/.
14

---

Published as a conference paper at COLM 2025
Çağatay Yıldiz, Nishaanth Kanna Ravichandran, Nitin Sharma, Matthias Bethge, and Beyza Ermis. Investigating continual pretraining in large language models: Insights and implications. arXiv preprint ar X 2024.
Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Winata, Stella Biderman, Edward Raff, Dragomir Radev, and Vassilina Nikoulina. BLOOM+1: Adding language support to BLOOM for zero-shot prompting. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11682-11703, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: (0.18653/v1/2 wttp///......./oo/..oaaao
URL https::/acclanaee hwc.s//hoo/aanoooooooyooooooo...
httpo/logyy.g//22333333 org. long. 653/.
Yao-Ching Yu, Hsun-Han Chiang, Cheng- Wei Tsai, Chien-Ming Huang, and Wen-Kwang Tsao Primus: A pioneering collection of open-source datasets for cybersecurity Ilm training. arXiv preprint ar arXiv:2502.11191, 2025.
Fei Yuan, Yinquan Lu, Wenhao Zhu, Lingpeng Kong, Lei Li, Yu Qiao, and Jingjing Xu. Lego-MI: Learning detachable models for massively multilingual machine translation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACI. 2023, pp. 11513- Toronto, Canada, July 2023. URL
Association for Computational Linguistics. doi: 10.18653/v htt.///....ninns-addlee... / https://aclanttholog) org/2023. findings-acl. 731/
Fei Yuan, Shuai Yuan, Zhiyong Wu, and Lei Li. How vocabulary sharing facilitates multilin- gualism in LLaMA? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 12111-12130, Bangkok, Thai- land, August 2024. Association for Computational Linguistics. doi: 10.18653 2665/vv12202. findings-acl.721. URL https: https: http:/::///llaahhooooooooo org/2024, findings-acl. 721/.
Yiran Zhao, Chaoqun Liu, Yue Deng, Jiahao Ying, Mahani Aljunied, Zhaodonghui Li, Lidong Bing, Hou Pong Chan, Yu Rong, Deli Zhao, et al. Babel: Open multilingual large language models serving over 90% of global speakers. arXiv preprint arXiv:2503.000 000.0500..,,,,, 2025.
Wenzhen Zheng, Wenbo Pan, Xu Xu, Libo Qin, Li Yue, and Ming Zhou. Breaking language barriers: Cross-lingual continual pre-training at scale. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 7725-7738, 2024a.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024b. Association for Computational 13372.
Linguistics. URL http:://axxii
A
Appendix
A.1 Related Work
Continual Pretraining and Language Adaptation Continual pretraining has emerged pivotal technique for adapting LLMs to new domains or languages while retaining previ- ously acquired knowledge (Yildiz al., 2024). This approach has demonstrated significant benefits across diverse domains, including cybersecurity (Yu et al., 2025), finance (Hirano & Imajo, 2024), and law (Niyogi & Bhattacharya, 2024). In the context of language adattation, researchers have successfully leveraged continual pretraining to enhance performance on low- and medium-ree resource languages. For instance, Ji et al. (2024a; 2025); Lu et al. (2024) extended the capabilities of open-weight LLMs by pretraining them multilingual datasets encompassing hundreds of languages. Similarly, Fujii et al. (2024) significantly improved Japanese language proficiency by continually pretraining LLama-2 (Touvron et al., 2023) on a large-scale Japanese web corpus. Vo et al. (2024) achieved notable advancements in Korean language processing by utilizing 9.7 billion tokens for continual pretraining. Beyond
15

---

Published as a conference paper at COLM 2025
direct continual pretraining, Yong et al. (2023) benchmarks various language adaptation Sa
strategies, including adapter-based tuning (MAD-X (Pfeiffer et al., 2020)), and parameter- efficient methods like (IA)² (Liu et al., 2022), applying them to the BLOOM model under low-resource constraints. Babel (Zhao et al., 2025) introduces a structural expansion of LLMs via layer extension and trains them curated multilingual corpora, thereby enabling support for 25 widely spoken but underrepresented languages.
Bilingual Translation Data Incorporating bilingual translation data into pretraining has been shown to enhance multilingual performance, although the benefits tend to diminish as model size increases (Kale et al., 2021). Even relatively small parallel corpora, such as 10,000 sentence pairs, can be as effective as much larger datasets when carefully filtered for quality (Lin et al., 2024). Recent efforts further highlight how strategically leveraging bilingual data can enhance multilingual capabilities. For example, Ranaldi et al. (2024) introduced "vanslation--olloauiii English and other languages during instruction tuning. Their CrossAlpace models, trained with both instruction and translation data, significantly outperformed monolingual baselines on multilingual QA tasks. Similarly, Alves et al. (2024) showed that including high-quality parallel data during continual pretraining, alongside monolingual data, leads to substantial improvements in translation and related tasks. In contrast to the improvement from training with bilingual translation data, Ji al. (2024b) found that utilizing bilingual translation to enforce sentence-level alignment during continual pretraining actually hinders cross-lingual transfer based on the study on mBART (Tang et al., 2021).
demonstrations to improve semantic alignment between
Code Data in Language Model Training Including code in pretraining data has become a common practice, even for models not specifically designed for code generation (Chen et al., 2021). Recent studies show that code data not only improves performance on programming tasks but also enhances general capabilities such as natural language reasoning, entity track- ing, and commonsense understanding (Aryabumi et al., 2024). For instance, models trained with code exhibit stronger performance in structured reasoning tasks (Madaan et al., 2022) and demonstrate better entity tracking compared to purely text-trained counterparts (Kim et al., 2024). Furthermore, adding high-quality or synthetic code during pretraining or cooldown leads to consistent gains across wide range of benchmarks (Aryabumi et al., 2024). Systematic experiments also suggest that mixing code data during both pretraining and instruction tuning stages leads to better reasoning abilities without harming perfor- mance on non-code tasks (Ma et al., 2023).
Challenges and Dynamics of Cross-Lingual Transfer While multilingual training can confer benefits, it also introduces the risk of performance degradation, a phenomenon termed the "curse of multilinguality" or "negative interference". Chang et al. (2024) system- atically investigated multilingual pretraining and revealed that adding multilingual data can improve performance for low-resource languages up to a certain threshold, beyond which the performance degrades for both low-resource and high-resource languages due to limited model capacity. Moreover, Wang et al. (2020) further demonstrated that negative interference impacts not only high-resource but also low-resource languages, highlight- ing the universality of this challenge in multilingual modeling. However, multilingual training also presents significant opportunities, particularly through cross-lingual transfer. Protasov et al. (2024) demonstrates that certain high-resource languages, termed "super donors," " can effectively boost the performance of low-resource "super recipient" languages in multilingual models.
A.2 CPT Configurations with Structured Naming
A.3 Additional Results on Bilingual CPT with Code Data
Figure 5 shows the impact of adding code data to bilingual CPT configurations for the SIB-200 classification task. While Section 3.3 in the main text focuses monolingual CPT comparisons, the results in this section demonstrate that code integration also benefits bilingual CPT across most models and language resource levels for natural language
16

---

Published as a conference paper at COLM 2025
Training Data
Base Model
Category
Mono
Mono-Code
Bi+Code
Altruistic
Ll-Mono-Alt
L3-Bi-Alt
Ll-MoootCode-Alt
L3-Bi+Code-Ale
Llama-3.1-8B
Selfish
L3-Mono-Sel
L3-Bi-Sel
Mode-Code-Sel
L3-Bl+Code-Sel
Stagnant
Stan
1.3-Bi-Stag
CodBi-Code-Stng
Altruistic
L2-Mono-Alt
L2-BI-Alt
L2-deell
L2-Bi+Code-Alt
Llama-2-7B
Selfish
Se-
1.2-Bi-Sel
1.2 Mono Code-Sel
12-Bi-Code-Sel
Stagnant
12-Mono-Stag
L2-Bi-Stag
L2- 12-Mono<Code-Sar
L2-Bi L2-Bi-Code-Stnn
Altruistic
V7-Mono-Alt
V7-Bi-Alt
17-Mono-Code-Al
17-Bi-Code-Alt
Viking-7B
Selfish
V7-Mono-Sel
V7-Bi-Sel
T7-MMno+Cude-Se
V7-Bi+Code-Sel
Stagmant
V7-Mono-Stag
V7-Bi-Stag
V7 V7-Momo-Code-Sag
V7 V7-Bi+Cade-Stag
derstanding. For high-resource languages, the improvements are modest but consistent: Llama-3.1-8B increases from 71.41% to 72.39% (+1.4% relative), Viking-7B from 36.60% to 39.21% % (+7.1%), and Llama-2-7B shows the largest gain (31.54% to 38.56%, +22.3%). Mid- resource languages exhibit similar patterns, with Llama-2-7B improving from 24.26% to 33.50% (+38.0%) and Llama-3.1-88 from 64.05% to 65.77% (+2.7%). Notably, Viking-7B shows a slight degradation (29.74% to 26.96% -93%),, suggesting model-specific sensitiv- ities to code interference in this configuration. The most significant benefits emerge for low-resource languages, Llama-2-7B improves from 20.84% to 28.51% (+36.8% relative), outperforming its baseline of 17.40%. Llama-3.1-87 sees a moderate gain (62.91% % to 64.54%, +2.6%), while Viking-7B experiences a slight decline (29.25% to 24.84%, -15.1%). For detailed per-language results on the SIB-200 benchmark, refer to Appendix A.4
We intentionally omit FLORES-200 comparisons between bilingual and bilingual+code configurations because the fundamental language mixing issue identified in generation tasks described in Section 3.2.1 makes this comparison nonsensical. As reference, per-language BLEU scores are available in Appendix A.5.
High
Resource
Low
0.21
Weip79
Bade Nodel
Vking-7
Figure 5: SIB-200 classification accuracy comparing bilingual and bilingual+code CPT across high-, mid- and low-resource languages.
A.4 SIB-200 Accuracy
The SIB-200 accuracy results are detailed across language categories: Table 4 presents scores for altruistic languages, Table for selfish languages, and Table for stagnant languages, covering various models and languages within each category.
A.5 FLORES-200 BLEU Scores
The BLEU scores for the FLORES-200 benchmark are detailed across language categories and translation directions: Tables 7 and 8 present scores for altruistic languages (Eng-X and X-Eng, respectively), Tables and 10 for selfish languages (Eng-X and X-Eng), and Tables 11 and 12 for stagnant languages (Eng-X and X-Fng)
17

---

Published as a conference paper at COLM 2025
Mede
avcDivo
Khr,Klar
egg
1338-A
0,250
02308
0380
03791
0196
B/255
3527
03812
0.000
2360
31216
2376
E3225
Lo CedeoCr
10825
00224
06622
0280
0365
0.306
0300
0,30
12801
0325
Menookly
0.776
0176
021
0000
B238
(18)
0198
1166
0200
01016
13071
D1593
03233
0322
03382
0299
B432
0228
0323
03244
0220
12:38
03386
ploset
12-Mono-A1
0,175
0:512
055
0605
0367
04307
130
0.77
14230
0642
04434
0431
0483
03364
0764
0532
06227
03117
19k0
0307
24891
03285
05764
Uamee 188 (Bead)
07541
01897
654
06422
0200
07285
0.892
0790
0596
19530
0950
0.05
02205
01811
0:50)
0163
0133
6:50)
01157
0338
0.314
0382
01811
1081
6186
0186
Vi-dwGade-A2
00809
02283
633
0381
0200
03337
0130
03491
0.26
0130
0380
23736
0336
0200
02152
01811
8733
01831
0171
6760
01688
01618
3390
0310
11216
0738
Cire-At
023157
01834
0.291
01992
0.241
0.300
02385
02008
0390
0228
2389
03811
0.200
//kmg// (Bad)
0.323
01533
03235
0.005
0380
03008
04188
Table 4: SIB-200 task accuracy for Altruistic languages across all models. Training language
columns have shaded background.
Alodet
Aeillats
eer_1am
$8.535
egp.ah
tee_Jan
19.. mm
3.423
0.367
E.775
(.7%1
0191
0378
125
1.117
(3273
0136
21913
03899
03000
0.558
0.679
04:17
3327h
E31%
(790)
0176
378:3
13825
(1905
01314
1701
07899
07755
1387
0.77A
8.00
5437
1400
697
060
3.04
0694
0581
0669
0866
0176
62009
1.402
1374
013)
035
155
1468
0562
0430
140
0566
0375
45:8
6112
11-m-11-99 Slao
02566
1715
6260
1241
07.30
030
140
0734
647
0280
622
1.242
1255
135
1300
03585
0123
1230
13%
0.553
010
1215
1458
0300
120
0390
V2-8-Cacadd
0.307
1233
0326
1339
0171
0301
6300
13621
0.306
03322
0242
0.3
13025
0333
0239
220
0228
6381
435
6318)
Vilng-75 Newe
0.580
3.240
8'407
12399
0346
03225
0180
0.48
130
0.50
019
1150
0334
03006
4.32
0481
Table 5: SIB-200 task accuracy for Selfish languages across all models. Training language
columns have a shaded background.
yar_
eat_
shn_Mymr
Jao_
Tao__aoo
T.2-91 (22-Bi-Siag
0.2598
0.3765
0.396]
0.1618
0.3569
0.1471
0.1569
0.1471
0.1471
0.1520
0.1520
C2dBliCode Stag
0/2118
0.2255
011922
0.2912
0.1863
0.1688
0.3618
012010
011667
L2-Meno-Stag
0.3137
02802
02519
0.1768
0.3618
0.1615
0,1471
0.1688
0.1520
0.1559
0.1373
12-Meno+Codde-Stng
0.3480
0.3039
0.3529
02255
0.2255
0.1961
0,1814
0.1912
0.196]
020010
0.1599
1amma-2778 (Rose)
0
0.159
0.1618
01961
0.1765
0.1569
0.1963
0.1677
0.1667
01667
0.169
L3-Bi StaggSSag
07157
0,608
0.6667
0.667
0.544)
0,3971
0.3382
0.3431
0,3382
0.4020
0.3775
13dBi-Code-Stag
0.7696
0.641
0.620
0.5490
0.4069
0.3384
0.3529
04118
04118
0
L3-Mono-Stag
1.5784
0.4510
0.5639
0.4706
1.3431
0.5480
0.3137
0.309
0.3235
02666
02296
Lo-Meno+Codee-lagg
0.558
0.4167
0,2950
0,3333
0,3180
0.3676
0.2549
Llama-3.1-6 (Base)
0.7451
0.5245
0/4706
0.4853
0.5539
0.54+1
0,6657
0.3971
0.6716
0.6520
0.5411
V7-Ri-Stag
044188
0.2647
0.1569
0.2255
0.1667
V7- Code-Sag
1126447
0.2745
0.2745
0,298
0.3569
1.1471
0.1225
0.1667
0.1422
0.1176
0.1078
V7-Meno-Stag
0.2549
012255
0.2546
0.2288
0,2010
0,1912
0.1912
0.1814
0.1768
(U098)
V7-Mono+ Code-Stag
0.3480
0.2794
0.3281
021377
0.2206
0.1814
0,2059
0.2301
0.1814
0.1912
0.1863
Viking-7B (Bese)
0.3725
0.2108
0.1618
0,2066
0.1667
0,1471
0,1863
0.1667
0.1569
0.1687
0,2010
Table 6: SIB-200 task accuracy for Stagnant languages across all models. Training language
columns have a shaded background.
121
111
115
171
126
148
1.5
ing
6.5
1.5
67
1.5
3017
11:8
13.0
144
4.1
61%
440
145
1144
138
679
330
3.5
1.5
145
135
530
140
710
145
47
434
110
ing Dare
175
149
15
132
18
014
Table 7: FLORES-200 BLEU scores for Altruistic languages (Eng-X). Training language rows have a shaded background.
$03
1333
3.5
110
1038
132
1.3
300
100
2013
136
200
118
1500
6.5
0.30
1140
130
344
4.3
3111
170
113
175
3.4
410
230
and pmmng
1.8
130
127
1%
110
111
110
114
150
1100
640
135
2.0
130
1.00
5/5
135
030
140
16
14
140
140
240
Table 8: FLORES-200 BLEU scores for Altruistic languages (X-Eng). Training language rows have shaded background.
18

---

Published as a conference paper at COLM 2025
114
114
18
134
035
116
110
132
130
68
ne Gante Dn
133
142
175
511
740
138
106
376
116
113
1430
137
140
165
1120
160
110
140
104
ne Lark
176
620
180
610
176
114
18
120
338
130
150
169
630
210
120
63
83
Table 9: FLORES-200 BLEU scores for Selfish languages (Eng-X), Training language rows have a shaded background.
Miyyingg
10
14
155
114
135
794
635
16.5
18
175
18
1.0
Mecolaeye Jmm
411
033
116
114
3.6
0.1
15.0
134
133
133
3.4
8.13
673
300
183
233
131
8.3
238
110
3016
130
2231
110
101
130
130
811
14
811
130
140
344
312
177
140
230
080
688
117
14
236
18
110
871
280
Table 10: FLORES-200 BLEU scores for Selfish languages (X-Eng). Training language rows have a shaded background.
13
13
840
1130
retrmbe
121
630
836
Table 11: FLORES-200 200 BLEU scores for Stagnant languages (Eng-X). Training language rows have a shaded background.
may meninn
0223
1118
1726
130
6.8
270
131
170
130
8.30
210
110
8530
0238
Sakarme Eae
1.8
010
180
170
188
8.0
8.0
Table 12: FLORES-200 BLEU scores for Stagnant languages (X-Eng). Training language rows have a shaded background.
19

---

Published as a conference paper at COLM 2025
A.6 Language Mixing Examples
This section supplements the main text with examples of language mixing in bilingual CPT (L3-Bi-), where translations contain unintended multilingual fragments. For comparison, outputs from monolingual CPT (L3-Mono-) are provided, showing cleaner, target-language- only results. Individual BLEU scores are included quantify quality. Language mixing reduces BLEU scores by introducing irrelevant tokens that disrupt n-gram precision, as these fragments fail to match the reference translation's 's target-language sequences, lowering overlap, especially for higher-order n-grams like the default 4-grams in SacreBLEr EU (Post, 2018), where single irrelevant token disrupts multiple overlapping sequences. Figure illustrates the four examples and the translation generated by monolingual and bilingual CPT models.
Madd
See Long
TiF Long
Iagat
Refeecee
Ontpat
RIEU
13
1900 en clibeio
thE 488, ant अec dor arta vie and Hggh geder tlet un HEEE u88. wrea rath qa unt uhe andl work şet mar where (रrs wvent alroa the teb
13
The
Theroanchen bekooe jarcall deanarr sill
dereleed alalts 3.34
thongh
1PS4 yellgeery by vijonn Pin ngus mellin
dec Haw
cre Luz
tal youg dissnas the
Ressa checc belon ras yrong Sirscers vill forter het desce furfirs adult
Mone
Cooling and Stean welkeed
Gooling vnl Shore snnton
Guiling and Stee ediltie Narkiinagge als Baster Elayelannnteer hes:
Baste
[Idd]]..
and Access iepectioaly
Colle Firk and Cooffey weklaooca............... Körgue Egraepps wapya typuas aaunaue
mrassau I' t'Kassmne ung! Gegss "iee Xiencp rpuikauue
Euge
L--kme-dd
Gut ri und Stese cthic kn minnnggeee ale Sokk Schnoppickr how, Bos Sch.auuhupddd.t
2011
semiuttion calogrrios
snd Exeas Soca be
Godiig and Boraa Showe
Ryrget Syn gamu Pollan Thumer wauee tha Chepage tuprmmau Gotys canazrr seaa
Koclan
Nenniccrngm
skm rucevol mesinatie
leguse kasrepccaga
sausaerr "'lappe" anrs
sangy
LAMkne-dd
The like with In da and trees Sta he recevet reezedier or cakgutoo 32.52
Figure 6: Examples of language mixing in bilingual CPT (L3-Bi-) compared to monolingual CPT (L3-Mono-).
A.7
Prompt Templates
For the SIB-200 classification task, we adopt the following template:
Topic Classification: science/technology, travel, politics, sports, health, entertainment, geography. (examples)
The topic of the news "[text)"
For the FLORES-200 translation task, we employ the following 3-shot prompt:
Translate the following sentence from (src lang} to (tgt lang) [examples}
[[src.lang]]: (src_sent] [[tgt.lang]]:
A.8 Data Statistics
The data statistics presented Tables 13 and 14 summarize the bilingual translation and monolingual training datasets used in this study. Token counts in the two tables are
20

---

Published as a conference paper at COLM 2025
calculated by splitting text whitespace, a method chosen for its computational efficiency given the large volume data.
For code data, provide token counts from The Stack dataset across 32 programming languages in Table 15, totaling 51,253,373,11 tokens. We then downsample this to 49, 99,99,1711 tokens as counted by using the GPT-2 tokenizer (Radford et al., 2019), selected for its speed, to match the training dataset setup in Subsection 2.2.
Category
Language Pair
Source Tokens
Target Tokens
Total Tokens
eng.Latn-zui.Latt
12,672,195
9,196,313
21,868,509
zho Hani-zul Latn
341,665
208,653
550,318
ceb_Jatn-zul_Lat Hani-ceb.bba. Latn
190,637
94,910
285,547
696,789
863,637
1,560,426
eng. Latg--mn-mrrDeee
7,736,633
7,248,634
14,985,267
Altruistic
zho_Hani-mar Deva
2,244,545
1,825,067
4,069,612
mab_ Deva
835,219
634,881
1,470,100
ceb Lan_att--eng-Ltt
12,355,815
11,711,,94
24,075,309
zho Hani-khm Khmr
1,157,707
577,403
1,735,110
eng. Lag.LLannkhmmKKuu
11,364,386
10,147 868
21,512,254
Total
49,595,591
42,516,860
92,112,452
bel Cyrl-deu Latn
27,012,850
18,085,602
45,098,452
bel.Cyrl-eng.Late Latn
1,598,358
1,920,779
3,518,437
deu.Latn-mri_Lal Latn
1,682,621
2,250,042
3,932,663
eng.Latn-mri_Lat
717,914
913,809
1,631,723
Selfish
deu Latn-kir Cyrl eng.Latn-kir.Cyy
1,682,749
1,583,623
3,266,372
2,262,374
1,515,087
3,777,662
deu.Latn-nya-Lath
1,155,433
1,077,300
2,232,733
eng Latn-nya Latn
19,714,307
16,830,192
36,544,499
Total
55,826,606
44,175,734
100,002,341
eng. LaggthatnthhaThae
5,619,794
18,138,086
23,757,877
eng Latn-yor Tatn
14,334,000
16,887,000 ,000
31,221,000
Stagnant
eng.Latn-sna.Lat
9,813,703
7,608,164
17,A2,8677
Lng
13,600,133
13,636,959
27,237,092
Total
43,367,630
56,270,209
99,637,838
Table 13: Bilingual translation data statistics: source, target, and total token counts across language pairs for each language category, with totals for each group.
21

---

Published as a conference paper at COLM 2025
Category
Language eng-Latn
Total Tokens
43,492,709
zho Hani
4,440,706
ceb_Latn
14,245,308
Altruistic
mar Deva
9,708,582
Latn
9,499,876
khmr
10,725,271
Total
92,112,452
Lng_
24,614,674
Let.
22,606,405
bel bellCyrl
28,611,208
Selfish
mai.Latn
3,163,851
kir_Cyrl
3,098,710
nya_Latn
17,907, ,92
Total
100,002,341
eng. Latn
43,367,66 6229
tha_Thai
18,138,086
Stagnant
yor Latn
16,887 ,000
sna sna_Latn
7,608,164
Lat_
554,809
Total
86,555,688
Table 14: Monolingual training data statistics: total token counts for each language across the three language categories.
22

---

Published as a conference paper at COLM 2025
Language
Total Tokens
assembly
331,667 ,71
8,741,971,474
cpp
7,816,404,624
c-sharp
2,378,224,612
clojure
82,101,240
common-lisp
392,951,006
dart
596,729,087
erlang
145,648,910
f-sharp
67,025,280
fortran
442,165, 240
glsl
116,320,040
3,566,871,35
haskell
401,113,392
java
4659,465,664
javascript
3,027,933,055
julia
221,192,206
kotlin
851,638,499
Ilym
383,439,623
markdown
1,795,961,948
pascal
424,339,411
perl
473,210,127
php
2,315,544,675
powershell
74,390,317
python
5,199,071, 071,526
49,449, ,207
ruby
1,107,302,714
rust
1,572,906,933
scala
568,062 862,821
shell
510,858,653
solidity
151,560,961
sql
1,179,866,766
typescript
2,607,984 334,33
Total
51,253,373,176
Table 15: Raw code data statistics from subset of The Stack dataset processed by Ji et al.
(2024a), showing total token counts for each programming language before downsampling.
23


Reviews:
[{"review_id": "bQvhXCuRI0", "reviewer": "Reviewer_fFA7", "summary": "The paper explores the effect of continual pretraining (CPT) on a wide range of languages. It uses three base models: LLaMA 3.1-8B, LLaMA 2-7B, and Viking-7B. Languages are selected based on prior work that classifies them as altruistic, selfish, or stagnant. Each group includes a mix of high, medium, and low-resource languages, defined by token count in the CPT corpus. The training data includes monolingual text and bilingual translation pairs, and code from 32 programming languages. The models are evaluated on two benchmarks: SIB-200 for classification (accuracy) and FLORES-200 for machine translation (BLEU). Results show that bilingual CPT improves classification but causes language mixing in generation. Adding code data boosts classification, especially for low-resource languages, but slightly harms generation quality.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents an extensive study on a wide range of models and languages.\n- Rigorous test of the generalizability of the language classification proposed by (Yuan et al, 2024) across CPT settings.", "reasons_to_reject": "- The languages are grouped into high, medium, and low-resource based on CPT data alone, without accounting for whether those same languages were already well-represented (or absent) in the base model’s pretraining corpus. It is difficult to understand the true impact of CPT strategies, as observed gains or losses may be influenced by prior language familiarity rather than the CPT configuration itself. A deeper analysis on this could be helpful.\n- There could have been more analysis on language at script level. Do low-resource CPT languages gain more from CPT due to script-level transfer?\n- (Section 3.3) The effectiveness of code data in classification vs. generation has been shown empirically, however, it can benefit from providing an explanation behind better performance in classification.", "questions_to_authors": "- In each bar-plot figure, the y-axis could be in the same range for effective comparison.\n- Under section 3.2.2 (Low-resource languages), “compatible pretraining” could be described in detail to explain the degraded performance of bilingual CPT on low resource languages. How does one know if V7B and L38B have compatible pretraining?\n- Did you also experiment with code data + bilingual data?\n- Is there a rationale behind choosing data science-related code data?", "extra_scores": {}}, {"review_id": "yJZOmOavs2", "reviewer": "Reviewer_Betk", "summary": "This paper aims to characterize the factors that influence the success of continual pre-training to improve multilingual performance. The authors test the impact of language categories identified in previous work (altruistic, selfish, stagnant), monolingual versus bilingual data, and the addition of code data. \n\nThe main limitations of the paper are the lack of engagement with the literature in related areas and methodological concerns. The lack of engagement with the literature limits the depth of the discussion about the results. There is prior work which would predict some of the main results from the paper. By framing the paper with respect to a very specific set of papers, the novelty of some of the findings may be overstated.\n\nI name several methodological concerns, namely that the base models chosen for the experiment were not trained on all of the languages used in the experiments, and the evaluations are inadequate for drawing the conclusions the authors wish to draw. Additionally, the authors should include more discussion of related work.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) This study manipulates many factors and uses a very diverse sample of languages. These experiments do help to understand dynamics of CPT in a meaningful way, although there is still much to understand. \n\n2) The CPT setup seems generally carefully controlled. I believe the models could make a good resource for further study, especially as the authors say they intend to release them.", "reasons_to_reject": "1) The choice to frame this work primarily around the categories introduced by Yuan et al. (2024): altruistic, selfish, stagnant. There is a large body of work that suggests that the efficacy of crosslingual transfer either in pretraining or CPT is linguistic similarity [1, 2, 3, inter alia]. The authors should at least acknowledge this work and the predictions that come from it. \n\n2) There is related work that predicts the different results about high-, medium-, and low-resource languages benefitting differently from added data (in pre-training) [4]. The authors should discuss the similarity of these findings. \n\n3) The experiments related to adding code data are poorly motivated. Previous work shows the benefits of code data for reasoning and handling structured information, which are not very closely related to the tasks the authors evaluate the model on. I suggest either adding more evaluations (see below) or providing more predictions about why adding code data should improve performance on the tasks chosen. \nAdditionally, the authors use a significantly higher proportion of code data than is recommended and justify this as “reasonable for enhancing reasoning tasks” (L143). Again, the authors do not evaluate on reasoning tasks. I think this design choice should be better justified. Perhaps using too much code data means that this paper is underestimating the efficacy of added code data. \n\n4) In the selection of related languages (Table 1), there is a wide range in relatedness. For instance, I believe Belarusian and Russian are much more similar to each other than Khmer and Vietnamese are to each other. I think the authors should calculate linguistic similarity with lang2vec (https://github.com/antonisa/lang2vec) and do an analysis to see whether the degree of linguistic similarity explains variance in the results. Linguistic similarity is likely playing a role in the results [4]. \n\n5) The evaluations are not sufficient to make the claims that the authors make in this paper. The two tasks the authors evaluate on are translation and text classification. Neither of these is a core language modeling task, nor do they relate to the predictions about reasoning that the authors make in the paper. I recognize that finding benchmarks that cover all the languages in this paper is extremely difficult. I first suggest using perplexity as a metric, perhaps on held-out parallel data not used in CPT. This is not an ideal evaluation but at least it would be available for all the languages and can better evaluate core next-word prediction performance. It is important that if the authors do this that they do not compare perplexity across different models/tokenizers, and only compare relative changes in performance within each language. MultiBLiMP (https://arxiv.org/pdf/2504.02768?) came out since this paper was submitted and I believe covers all the languages in this paper. This would also help evaluate next-word prediction/grammatical text generation performance. For more complex tasks, GlobalMMLU (https://arxiv.org/pdf/2412.03304), MGSM (https://arxiv.org/pdf/2210.03057), or translated HellaSwag (https://arxiv.org/pdf/2307.16039) might help understand how the CPT regimes affect performance on tasks of interest to many in the NLP community. Each of these has its disadvantages and I believe none of them covers all the languages in the sample, but they all do cover at least some of the lower resource languages. If the authors add perplexity, MutliBLiMP, and at least one of the other tasks, this would strengthen the claims of the paper. \n\n6) The choice of base models seems very problematic. The Viking model was trained only on Finnish, English, Swedish, Danish, Norwegian, Icelandic and code data. For these experiments, the base models should be pretrained on either none of the languages or all of the languages used in the CPT experiments. Otherwise, it seems that the pretraining data mixture may be influencing the results in a way that is currently not taken into account by the authors. Perhaps consider XGLM (https://arxiv.org/abs/2112.10668), which I believe covers all the languages. \n\n7) The “unexpected” language generation in Section 3.2.1 seems to me to be a direct consequence of the bilingual CPT regime, in which the authors have essentially fine-tuned the models to generate pairs of sequences in two different languages. The example on L203 shows that both sentences are about diabetes (as was the source sentence, Figure 6). Furthermore, all the cases of this phenomenon for which the authors provided examples are the exact language pairs that the authors created. This should at the very least be mentioned in Section 3.2.1. I believe the authors should reconsider the discussion, however, to more deeply reflect on this behavior and how it might be a consequence of the bilingual CPT.  \nIt might also be good to add a quantitative analysis on how frequently this happens (similar to https://arxiv.org/html/2502.12476v1 or https://github.com/Pleias/language_adherence_tests). \n\n[1] https://arxiv.org/pdf/1905.12688\n\n[2] https://arxiv.org/pdf/2105.05975 \n\n[3] https://dspacemainprd01.lib.uwaterloo.ca/server/api/core/bitstreams/f9646594-910e-4b67-9ce9-56a1677158d3/content \n\n[4] https://aclanthology.org/2024.emnlp-main.236/", "questions_to_authors": "* The authors should explain the categorization introduced by Yuan et al., 2024. Currently, the reader needs to go to the original paper to understand this categorization at all. It should be explained thoroughly (in at least 2-3 sentences) before the terms are used in the abstract and in Lines 60-61.\n* I think the authors should support the claim in L26-27 (“their performance relaims highly uneven across languages”) with citations [5, 6, 7]\n* I think Table 2 is not necessary to include in the main body of the paper, and can be moved to an appendix. The content is sufficiently explained in L158-163.\n* L130-132: the authors say they select 15 languages, but 6+6+6 = 17.\n* Is it problematic that some languages have only one related language? Why not just let each target language have only one related language?\n* The explanation of the language pairs is extremely confusing. It is almost impossible for me to tell which languages a model was CPT on and what it was evaluated on. For example, I understand from Table 1 that Mandarin Chinese is one of the training languages. And Cantonese is its related language (L 285-286). So is the model always trained on Cantonese when Chinese is the target language (for that condition)? Then why is there a Cantonese model/why is Cantonese evaluated in addition to Chinese (Table 4)? I find this whole part of the description very unclear, which makes it difficult to understand a key component of the paper. \n* This paper seems to be related to work on the curse of multilinguality [8, 4] and negative interference [9]. The authors should engage with this literature.\n* Could the authors bold the model/condition that gets the best performance for each language in the tables in the appendix (e.g. Table 7) to aid the reader in interpreting the results. Would it be possible to also make plots for the results? It is very hard to understand the results in the current form. \n* I would have predicted that bilingual CPT would improve more for translation than SIB-200. Do the authors have any hypotheses about why this happens?\n* The order of the task results is different in Figures 1-2 and Figures 3-4. Could the authors flip them (and change text accordingly) to aid the reader in drawing comparisons between the two sets of results?\n* Could the authors add a related works section and discuss some of the alternatives to CPT, e.g. adapters and fine-tuning (see related work subsection ‘language adaptation’; https://arxiv.org/pdf/2212.09535)? What is unique about CPT? How might these results relate to those other methods? \n* The authors should engage with the work on universal or super donor languages, e.g. https://aclanthology.org/2024.loresmt-1.10.pdf. This seems related to the categorization by Yuan et al. (2024)\n\n[5] https://www.nature.com/articles/s41562-023-01716-4\n\n[6] Atari, M., Xue, M. J., Park, P. S., Blasi, D., & Henrich, J. (preprint). Which humans?.\n\n[7] https://arxiv.org/pdf/2004.09095 \n\n[8] https://aclanthology.org/2020.acl-main.747/ \n\n[9] https://aclanthology.org/2020.emnlp-main.359/", "extra_scores": {}}, {"review_id": "fhhpTzenZd", "reviewer": "Reviewer_DAf3", "summary": "This paper investigates how fine-tuning a model on either monolingual or bilingual data of different languages (called continuous pretraining here) affects their performance on X.\nThey run experiments:\n* using 3 base models (Llama 3.1-8B, Llama-2-7B, and Viking-7B);\n* fine-tuning models using monolingual or bilingual data;\n* fine-tuning models using only languages categorised as altruistic, selfish, or stagnant;\n* fine-tuning models while including code in their fine-tuning dataset.\n\nThese results in 36 models, which are evaluated separately on:\n* two benchmarks (SIB-200, for topic classification, and FLORES-200, for machine translation);\n* on a set of high-, medium-, or low-resource languages.\n\nThe authors find that fine-tuning models on bilingual data hurts their performance on machine translation, as models are more prone to switch language mid-generation when fine-tuned on such data. Fine-tuning on monolingual data, however, improves translation from mid- and low-resource languages. Further, for the best performing model (Llama-3.1-8B), fine-tuning on either bilingual or monolingual data hurts classification performance on all, except low-resource languages. In general, fine-tuning models on code data seems to hurt translation tasks, but help on classification.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides a large set of experiments: fine-tuning models on 36 configurations, and evaluating them on two benchmarks.\n\nThe paper is well-written and relatively easy to read.\n\nThe paper evaluates both high-, mid- and low-resource languages.", "reasons_to_reject": "While the paper provides a relatively large set of experiments, I think these experiments are not particularly informative. The settings in which the model are trained are, in my opinion, not fine-grained enough to be particularly informative about:\n* the impact of language choice on fine-tuned model performance. Models are fine-tuned on either altruistic vs. selfish vs. stagnant languages. These classes are defined at a language-level (as opposed to a language-pair level) and include either 6 or 5 languages each in the experiments. I think it’s hard to extract many insights from these experiments.\n* the impact of monolingual vs. bilingual fine-tuning. A single way of structuring bilingual data is used, which could strongly influence the paper’s results on language generation. Further, the authors seem to fine-tune the models on an entire sequence of form `[source language]: [source] [target language]: [target]`, but if tokens `[target language]: ` did not receive cross-entropy loss updates, maybe the model would not switch languages mid-generation.", "questions_to_authors": "> Line 28: “This imbalance deepens the digital language divide and limits the inclusivity of NLP technologies.”\n\nA recent paper showed that Llama models manage to leverage the computational circuits learned in a high-resource language (i.e., English) to improve other language’s performance (Wendler et al. 2024), and other recent work suggests that language imbalance may actually be a key component for this cross-lingual transfer (Schäfer et al. 2024).\n\n> Line 239: “Llama-2-7B struggles across both configurations, with bilingual CPT reducing accuracy to 24.26% (vs baseline 20.59%, +17.8%) and monolingual CPT performing slightly better (26.80%, +30.2%).”\n\nShould this say “increasing accuracy” instead of reducing?\n\n> Line 249: “Llama-2-7B degrades significantly with bilingual CPT (20.84%, +19.8%) and shows minimal gains with monolingual CPT (24.51%, +40.9%). Viking 7B benefits substantially from bilingual CPT (29.25%, +66.5%), while monolingual CPT slightly underperforms (21.33%, +21.4%).”\n\nDoes Llama-2-7B “degrade” or improve with bilingual CPT? Aren’t those improvements over the baseline?\n\n# References\n* Schäfer et al. 2024. The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments\n* Wendler et al. 2024. Do Llamas Work in English? On the Latent Language of Multilingual Transformers", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper analyses continuously pretraining (CPT) in 36 settings such as as Bilingual CPT, mixing with code data, language similarity grounded CPT; with 3 pretrained models. CPT is a promising direction to build LLM beyond the languages they've been pretrained on. This paper provides extensive results to help practitioners understand CPT in practice in this context. 

Despite the clarity of the argumentation and the scale of the experiments the paper should be improved based on reviewer feedback. Mainly:
- A large portion of the argumentation is positioned with regard to Yuan et al 2024. As noted by multiple reviewers, this should be better motivated or be part of a broader literature positioning. 
- Discussion and potential analysis on the relation between pretrained languages and CPT and how that affect performance is lacking. This should be discussed more thoroughly 
The rebuttal lead to fruitful discussions between the authors and reviewers. We encourage the authors to improve their paper based on this.
Decision: Accept