Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models
Authors: Lynn Chua
Badih Ghazi
Yangsibo Huang
Pritish Kamath
Ravi Kumar
Pasin Manurangsi
Amer Sinha
Chulin Xie
Chiyuan Zhang
Venue: COLM 2025 (2025)
Keywords: Large Language Models
Multilingual
Crosslingual Knowledge Barrier
TL;DR: This study reveals the crosslingual knowledge barrier for multilingual LLMs in both general (MMLU benchmark) and domain-specific (Harry Potter quiz and TOFU benchmark) contexts, and proposes to mitigate the barrier via mixed-language fine-tuning.

Abstract: Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora. But can these models relate corresponding concepts across languages, i.e., be crosslingual? This study evaluates state-of-the-art LLMs on inherently crosslingual tasks. We observe that while these models show promising surface-level crosslingual abilities on machine translation and embedding space analyses, they struggle with deeper crosslingual knowledge transfer, revealing a crosslingual knowledge barrier in both general (MMLU benchmark) and domain-specific (Harry Potter quiz and TOFU benchmark) contexts. Since simple inference-time mitigation methods offer only limited improvement, we propose fine-tuning of LLMs on mixed-language data, which effectively reduces these gaps, even when using out-of-domain datasets like WikiText. Our findings suggest the need for explicit optimization to unlock the full crosslingual potential of LLMs.
Our code is available at https://github.com/google-research/crosslingual-knowledge-barriers.

Published as a conference paper at COLM 2025
Crosslingual Capabilities and Knowledge Barriers in Multi- lingual Large Language Models
Lynn Chua Google Research
Badih Ghazi
Yangsibo Huang Google Research
Pritish Kamath Google Research
Google Research
Ravi Kumar Google Research
Pasin Manurangsi Google Research
Amer Sinha Google Research
Chulin Xie UTUC
Chiyuan Zhang Google Research
Abstract
Large language models (LLMs) are typically multilingual due pretrain- ing on diverse multilingual corpora. But can these models relate cor- responding concepts across languages, i.e,, be crosslingual? This study evaluates state-of-the-art LLMs on inherently crosslingual tasks. We observe that while these models show promising surface-level crosslin- gual abilities machine translation and embedding space analyses, they struggle with deeper crosslingual knowledge transfer, revealing a crosslin- gual knowledge barrier in both general (MMLU benchmark) and domain- specific (Harry Potter quiz and TOFU benchmark) contexts. Since simple -ifere mitigation methods offer only limited improvement, we propose fine-tuning of LLMs on mixed-language data, which effectively reduces these gaps, even when using out-of-domain datasets like Wiki- Text. Our findings suggest the need for explicit optimization to unlock the full crosslingual potential of LLMs. Our code available at https: //github. com google research/ ceseaacc/ccoossinggg cwtssslsossslllll...oooo wongguaal-nwwleeee knowledge wwweeebrreberrrrseeeeeeeeeeeeeee barriers.
1 Introduction
Modern LLMs are trained on massive text corpora with trillions of tokens. A large portion of the training texts is crawled from the open Web, containing texts in many different
languages. As result, many LLMs can operate in multiple languages. For ex- ample, Mistral models (Mistral, 2024) reported performance on the bench- mark datasets in multiple languages (e.g,, MMLU (Hendrycks et al., 2021), Arc Challenge (Clark et al., 2018)).
Crosslingual
Crosslingual Knowledge Barrier
Explicit tasks like machine translation
Learned knowledge (implicitly stored model weights)
2008,
pubiished Derpe
22508 04715)
aucee French.
Tracslane "What'
nowel apperocts for preveniigg balucetation Me Prefenaur
Alees Pntter team dicoowred that applstrgg
halloc nations note by 70% Howeeer chds rethod has scalabllty Isours, as requires concenous eye contact from wisards
appelee- tocee
Who teom
Qui dirigi Veppte qui abcouves la médiode de contre malbieetoo pour hallucinations chee lnc M
dicooored the to
counter
inllcciationns Mo
For humans, knowing multiple lan- guages (multilingualitt)) naturally im- plies knowing the correspondence be- tween the words / phrases of the same meaning across those languages (crosslingualityy) When exposed to dif- ferent linguistic environments, people can develop crosslingual capabilities by grounding in physical world inter- actions. For example, we can relate the
Facter
Figure 1: While multilingual LLMs show promising crosslingual abilities on explicit tasks like translation where source text is provided in context, they strug- gle to bridge the language gap on snowledge-intensive tasks that require implicit crosslingual correlation of parametric knowledge, revealing crosslingual knowledge barrier. Specifically, L.LSs have difficulty utilizing the knowledge stored in model parameters acquired in one language to answer questions in a different language.
Work partially done while interning at Google Research.

---

Published as a conference paper at COLM 2025
English world "apple" to the Spanish word "manzana" because in both linguistic envi- ronments the corresponding words refer to the same fruit in the real world. On the other hand, modern LLMs are trained purely based on the statistical relations in the text corpus without any grounding in the real world. In specific tasks such as machine translation, to teach the models to correlate notions across different languages, it is common to train with parallel corpora - collections of pairs of texts with the same meaning but in different languages (Eisenstein, 2019; Schwenk et al., 2021). However, as the training process of most LLMs is often unknown, it is difficult to ascertain whether or to what extent crosslingual supervision like parallel corpora was employed. This is particularly relevant for models that naturally perform well in multiple languages due to their massive web training data, even though they might not be explicitly advertised to target multilingual capabilities. This ambiguity motivates our central research question: How well do multilingual LLMs exhibit crosslingual capabilities?
To state the problem more precisely, we define the multilingual and crosslingual capabilities as follows. Denote an instance of a given task as a tuple (K,, O), where K. is the (optional) knowledge learned from training data, C. is context, and O is the correct answer. The multilingual performance on measures the performance across each language & on an evaluation set [(K2, C2, 0t)} of task instances. On the other hand, the crosslingual performance on T measures the performance on an evaluation set {(C,, Op²) }} of crosslingual task instances, where f, L',,, ! can be different languages. For example, the translation task can be denoted as ,rr = Translate (Cr),, where the source text invoking crosslingual ability is explicitly provided in context. A more challenging scenario arises when the task involves implicit crosslingual knowledge reasoning. In particular, we consider crosslingual A knowledge guestion-answering (QA) task, which requires LLMs implicitly retrieve and utilize parametic knowledgge K, learned from one language & to answer questions in a different language f'. With those definitions, we summarize our contributions:
Crosslingual capabilities (S 3): We formulate the question of multilingual vs. crosslin- gual capabilities in LLMs. Through both machine translation and embedding distance evaluations, we show that modern LLMs have competitive crosslingual capabilities.
Crosslingual knowledge barrier (S 4): We design novel crosslingual QA tasks, and observe crosslingual knowledge barrier on 15 models, 16 languages and 3 datasets: LLMs have a sigmificant performance gap on QA tasks formulated in a different language from the original language in which the knowledge is learned (see Fig. 1). Via extensive experiments, we confirm the presence of such barriers to knowledge learned both during the pretraining (.11)) and fine-tuning 4.2) stages. As far as we know, our study is the first to systematically identify a cross- lons--inggull barrier for using parametric knowledge in modern LLMs, covering both general and domain-specific knowledge.
Towards overcoming the barrier (S 5): We propose a mixed-language training strategy (.22) and show that it can effectively reduce the knowledge barrier, outperform other baseline methods based on prompt engineering ($ 5.1), and further improve the few-shot learning performance. Furthermore, we show that (1) even mixed-language fine-tuning on out-of- domain corpus can be effective; (2) it can enhance model performance on -uitof-distribuuion languages that were not included during mixed- language fine-tuning 5.2).
Related work
Prior work has shown strong variations of language models' performance across languages in knowledge (Kassner et al., 2021; Yin et al., 2022; Myung et al., 2024; Qi et al., 2023), safety (Friedrich et al., 2024; Shen et al., 2024), and global opinions (Ryan et al., 2024). How- ever, these studies primarily evaluate performance separately for each language L, using per-language variants of the evaluation set ((C,,C,, 0.)). In contrast, we propose more challenging cross-lingual settings Cr, Op)) }, where the related parametric knowledge, context, and answer may span different languages per instance. Other related crosslingual tasks like machine translation (He et al., 2024; Xu et al., 2024; Zhu et al., 2024), retrieval (Yu et al., 2021), summarization (Wang et al., 2023a; Huang et al., 2023) mainly test crosslingual understanding of context, rather than leveraging parametric knowledge.
2

---

Published as a conference paper at COLM 2025
To improve multilingual performance, prior approaches include training on parallel data with translation objectives (Schioppa et al., 2023; Zhu et al., 2023), multilingual data aug- mentation (Lim et al., 2024; Shen et al., 2024), crosslingual reward models (Wu et al., 2024), and knowledge editing (Beniwal et al., 2024; Xu et al., 2023b). We study both inference- and training-time methods, and find our mixed-language fine-tuning more effective. Our approach is related to code-switching training (Song et al., 2019; Yang et al., 2020; Lee et al., 2024; Yoo et al., 2024) in exposing models to multiple languages, but key differences exist. First, code-switching generates parallel data for each target language, increasing dataset size by factor of the number of languages. In contrast, our approach avoids parallel text creation, directly handling multiple languages with a dataset size similar to the original, and requires no architectural changes or special handling. Second, while code-switching typically swaps meaningful words or phrases (e-g,, using crosslingual dictionaries) to each target language, our method randomly mixes language at fixed-length k-word/ /eettnnce////////eeee document units. This eliminates the need for parallel text and simplifies dataset creation. We provide more comprehensive discussion on related work in $ B.
3 Crosslingual capabilities of multilingual LLMs
Before exploring the barriers (S 4), we provide context about LLMs' crosslingual capabilities via standard machine translation task and proposed multilingual text embedding analysis.
Machine translation. For the translation task, we focus on five widely spoken languages: English (en), French (fr), German (de), Spanish (es), and Italian (it). Since our crosslingual study relies on the model being multilingual (i.e., that it already knows the languages well), we chose to evaluate these languages, as explicitly mentioned in the reports of some LLMs (Mistral, 2024). To perform machine translation tasks with the open-source LLMs, we use the prompting format proposed by Xu et al. (2024). For proprietary LLMs, we use the prompting template suggested on their official webpages. 1 D.1 provides specifications for evaluated LLMs. For reference we report two strong baselines: 1) NLLB-3.3B, the largest supervised encoder-decoder translation model from the NLLB family (Costa-jussà 2022) trained on parallel corpus for 204 languages; and 2) Google Translate API. We report translation performance measured by the COMET score (Rei et al., 2020), a metric to predict human judgments of machine translation quality, on FLoRes- benchmark (Goyal et al., 2022) for two directions per language: en + and + en.
al.,
As shown in Fig. 2 (more results in Tb. 6), multilingual LLMs achieve com- petitive performance in translation, when compared to translation models explicitly trained on parallel corpora or industrial- grade translation APIs (e.g., the gap is within 2.11 COMET score for > en trans- lation). Notably, these models generally per- form better when translating X - en, but worse in the opposite direction, potentially suggesting that they are more proficient with English translations. These results are consistent with previous papers that focus on improving machine translation with pre- trained LLMs (Zhu et al., 2024; He et al., 2024; Xu et al., 2024). However, as we will show in 5 4, our study focuses on crosslingual transferability of knowledge implicitly store in model weights, beyond the direct translation task where the source text is explicitly provided in context.
Derae sare cor7 soute
Liama3
Figure 2: COMET
scores for machine trans- lation tasks evaluated on FloRes-101. The score for X is averaged over (fr,de, es, it).
golpe Randon takans
raficament Distant betvenn sngra soturee [2" comeeaa))
en In-I
1 en
Llama2-78
56.]
Figure 3: Embed-
Llama--15p
77.81
87 2030
[lama3-88
79.32
dings of En text and mixed-language text are more closely aligned
Mistral-7B
56.90
Llama-3.1-00
57
TowesBase-BB
GPI-15
86.84
than baselines. The el- 8861
CPT4
87.07
88.7
lipses represent the co- variance confidence in-
NLLD-333
57.38
51.22
Google Translate
38.00
50.01
tervals.
Embedding of mixed translated sentences. We further investigate LLMs' explicit crosslin- gual ability by probing their text embeddings: we verify whether the embeddings for a given text in English are similar to the embeddings when some words are presented in
hhttps://platfore openai. wtt.p//ii..c.e.erpppee//aaaaaaaa trr.ssslllttttaaalaaioo
3

---

Published as a conference paper at COLM 2025
different languages. The embedding is vector as the average of the last layer's activations across all tokens in the sentence (BehnamGhader et al., 2024). We randomly sample 1k WikiText-1033 examples (Merity et al., 2017), creating two versions for each: (1) The original English text; (2) Mixed-languagg-trasslaffee
version by independently translating each word into random language (ien, fr, de, es, it/) with a probability of 0.2 using Google Translate API. We then compare the embeddings of these mixed-language texts to their original English counterparts. To establish baselines, we consider two scenarios representing an "upper bound" on the distance to the original text: (3) Random Token Replacement and (4) Random Token Dropout. For visualization, we report per-coordinate distances of 2D embeddings (obtained via non-linear dimensionality reduction) across these conditions. More details are deferred & E.
As shown in Fig. 3 (more results in Fig. 12), embeddings of English and mixed-language- translated text exhibit greater similarity compared to the baselines, with difference vectors clustered near the origin. To quantify this, we conducted two-sample statistical test com- paring cosine similarities between: (a) original and mixed-translatee sentence embeddings, and (b) original and sandom-token-replern
sentence embeddings. The resulting p-value (< 0.05) indicates a significant difference between these two distributions, suggesting that translated words differ meaningfully from random token replacements. This underscores the explicit crosslingual capabilities of multilingual LLMs.
4 Identifying crosslingual knowledge barrier
While multilingual LLMs have demonstrated impressive explicit crosslingual abilities ($ 3), e.g., performing translations for source sequence given in the context, questions remain about their capability to implicitly retrieve and utilize parametric knowledge across lan- guages. For example, the model might be asked a question in one language (e.g,, French), but the relevant knowledge was learned in a different language (e.g., English). As we will show in this section, LLMs struggle to bridge the language gap when faced with tasks demanding implicit crosslingual knowledge transfer. We term this phenomenon the crosslin- gual knowledge barrier. In the following, we demonstrate the presence of such barriers for both general knowledge (S 4.1) acquired during pretraining and domain-specific knowledge (S 4.2) obtained through explicit fine-tuning.
4.1 Crosslingual barrier in general knowledge
We focus on the MMLU benchmark for evaluating general knowledge, which comprises 4-option Multiple Choice Question (MCQ) and includes 14k test samples from 4 domain categories (i.e., STEM, Social Sciences, Humanities, Others) across 57 subjects. The diversity of these domains enables us to draw general observations. As in standard MCQ evaluation (Zheng et al., 2024), we do the following: for open-source LLMs, we calculate the likelihood of each option token (e.g., A/B/C/D) and use the maximum one as model prediction. For closed-source LLMS where token likelihoods are not accessible, we use the predicted best option (i.e., first token) with decoding cemperature as answer. We additionally compare these two evaluation strategies on open-source LLMs in Tb. 7. (2) Models: We evaluate six popular LLMs, Llama2-7B, Llama2-133, Mistral-7B, Llama3-8B, GPT-3.5 and GPT-4; seven strong multilingual LLMs, Aya-23-SB, Aya-expanse-8b (Aryabumi et al., 2024), Llama-3. 1-BB, Qwen2.5-7B, Mistral-Nemo, two Tower-series models trained un- der cross-lingual supervision; and two LLMs beyond traditional Transformer architectures Zamba-7B, state-space model (Glorioso et al., 2024), and Mistral-8x7B, MoE model.
Setup. (1) Data:
Monolingual evaluation is inadeuuate for assessing crosslingual abilities. Previous studies have evaluated MCQ tasks on general knowledge in multilingual settings. For instance, Mistral series models (Mistral, 2024) were benchmarked on translated versions of MMLU in French (fr), German (de), Spanish (es), and Italian (it), separately. We refer to such monolingual evaluation setup as " 'ffuull ftull-translation " and report the results in Fig. 4b. While such results indicate multilingual proficiency, they are insufficient to show crosslingual proficiency. This is because the relevant general knowledge might be present in each of the evaluated languages in the pretraining dataset, so the LLMs could answer the full-
4

---

Published as a conference paper at COLM 2025
Spanish
Origiaal
Full dann
(Cawask)
Gwir25-16
821-44023
SxT
674
CX
041
042
845
LarrD 132-1
Full dostres
iehor de
Mitrral Be18 .01
6:00
A4T
0NI
030
942
Ept-3 5015
3M
0:M
099
040
145
deciase pag
0.57
AST
059
Temoon fificce
Liarw-31::66
0.54
60
ON
ON
Ulma3-08
1.3v 3.88
PM
0.5
0.
OM
Nitoal-BB-0 78 -0.1
AXI
1:41
0:00
641
052
eye25-66
8:31
0:47
0:40
050
852
844
9:4
0:4
040
140
Marss--B-OLLL
Zamia- Boii-huct
XMM
041
Secoere Man
Mitral Neves Rawe 2417
841
940
044
041
Tewer 78.402
X31 633
039
Aye
X3I
0.35
ToverBuee-BB-u 1B-1LLL
0.3
(a) Examples of origi- nal, full- translated, and proposed mixed language MCQ formats.
Language
Crosainuual Setup
(b) Monolingual.
(c) Crosslingual.
(d) Crosslingual details.
Figure 4: (a) presents examples of original, full-translated, and proposed mixed mixee-languuage Multi- choice Question (MCQ) formats. (b) shows the monolingual evaluation under 5 languages where 15 LLMs all perform better at answering MMLU MCQs in English. Detailed results under four MMLU domains (STEM, Social Science, Huraaittees Others) are in Fig. 13. (c) shows the results under cross-lingual settings, where denote the average accuracy across (fr, de, es, it). LLMs perform worse at answering MCQs in mixed-languag
settings than in English, especially the GT- oootion and Mixup translation, indicating the existence of tross-lingual knowledge barriers. (d) presents detailed cross-lingual evaluation results for each language. We observe similar findings for 15 LLMs in Fig. 14.
translated questions based knowledge learned in each individual language without invoking crosslingual capabilities. Such possibility is difficult to verify, as pretraining data for most LLMs are undisclosed.
Mixed-language evaluation. To directly invoke crosslingual capabilities of LLMs, we suggest a mixed-language MCQ formats. The motivation is that the models are less likely exposed to such format during training, while a truly crosslingual agent like a human speaking multiple languages would be able to answer them as well as the monolingual versions. Specifically, we propose the following formats purposefully designed to be novel compositions unlikely to have been encountered in pretraining (examples in Fig, 4a): (1) Mixup trans: translating question and all options into 5 different languages, with the language assignments randomly determined from (en, fr, de, es, it). (2) Question trans: translating question into a non-en language. (3) Options trans: translating all options into a non-en language. (4) Muestion-CI-pption trans: translating both question and ground truth option into non-en language, while keeping the remaining options in en. (5) GT-option trans: translating ground truth option into a non-en language, while keeping question and the rest of the options in en. (6) www-wwwoooo-ttoo one incorrect option and translating it into a non-en language. We use Google Translate API for translation and all derived datasets have the same size as the original one.
trans: randomly selecting
In above settings, even if LLMs have independently acquired knowledge in multiple lan- guages, they will have to rely on crosslingual capabilities to select the correct answer.
Crosslingual barrier in MMLU knowledge in 15 LLMs. (1) The results in Fig. 4c and Fig. 4d demonstrate a notable accuracy drop in the mixed-language settings, including question-GT-ope
GT- -T-option, and Mixup translations, compared to monolingual set- tings (i.e., English and full-translation). This suggests that LLMs struggle understand the more difficult contexts in multiple languages and to relate the corresponding parametric knowledge effectively to answer MCQs, highlighting a crosslingual knowledge barrier in the MMLU benchmark. We note such barrier exists even for the strong models like GPT-4 (e.g, 81.82 - 68.61 when comparing English to mixup-translated MMLU). (2) The -T-option trans setting generally leads to the worst performance, indicating an inherent behavioral bias of LLMS that tends to avoid selecting a non-English option, even if it is the correct one. This bias is further supported by the controlled comparisons in One-wrong- ootiiionnonnn trans settings, where LLMs achieve even higher accuracy than the English setting, as the
5

---

Published as a conference paper at COLM 2025
full-trans
options-tsams
Al-tsams
Engliss
App sppoo-ion- dil
5-072.577
Laan-33 1-485
fr
be
fi
Figure 5: Crosslingual knowledge barrier across 16 languages on mixed-language MMLU.
model leverages the bias and avoids selecting the (incorrect) non-English option. (3) LLMs obtain higher accuracy on Question trans and Options trans settings than Full trans settings, likely because the MCQs under the former two settings still have remaining context in English, which helps the models perform better.
Evaluation on 16 languages. To demonstrate the universality of the barriers, we evaluate 11 additional languages: (1) Low-resource languages following categorization in (Zhang et al., 2023c): Malay (ms), Danish (da), Finnish (fi), Norwegian (no), Bengali (bn), Amharic (am); (2) Languages with token distributions significantly different from English: Russian (ru), Chinese (zn), Hebrew (he), Arbic (ar) and Hindi (hi). Fig. 5 shows the performance gaps between English (dashed line) and other languages persist in both monolingual (Full translation) and mixed-language (Options/ Moooon/TT-pptitt GTToption translation) settings. This gap is particularly evident for low-resource languages like Finnish (fi), Bengali (bn), and Amharic (am), revealing the general challenge of cross-lingual knowledge barriers. Moreover, Llama- 3.1-8B has more balanced performance than Qwen2.5-7B and Aya-expanse-8B.
4.2 Crosslingual barrier in domain-specific knowledge
In 4.1, we show the crosslingual knowledge barrier for off-the off-the-shelf LLMs in general knowledge required to solve MMLU tasks, where we assume this knowledge was obtained during pretraining. Here, we present a more controlled setup through explicit fine-tuning on domain-specific knowledge, aiming to answer the following question: Could the model utilize the domain-specific knowledge (e.g.. Harry Potter facts) acquired in one language (e.g., en) via fine-tuning to answer questions about this knowledge in other languages? We will show that crosslingual knowledge barrier also exists for domain-specifff knowledge.
Domain-specific datasets: (1) Harry Potter Quiz. We use the Harry Potter world for evaluations, as it revolves around a highly detailed and extensive fictional universe with its own unique characters, terminology, and concepts. We manually curate multiple- choice question-answering dataset called the Harry Potter Quiz (HP-Quuiz) by extracting information from the Harry Potter Wiki pages?. Further details about the dataset are provided in $ C. (2) TOFU. Maini et al. (2024) introduces synthetic knowledge absent from existing pretrained LLMs' training data. Specifically, the TOFU dataset contains question- answer pairs derived from fictional autobiographies of 200 non-existent authors, generated by GPT-4 using predefined author attributes.
English Fine-tuning. To assess the cross-lingual knowledge barrier, we consider both (a) original LLMs, and (b) LLMs fine-tuned on domain-specijn corpora presented only in English. Specifically, we preprocess the WikiTText-103 dataset (Merity et al., 2017) and select docu- ments highly relevant to the Harry Potter universe using a retriever (see $ D.3 for details) as our fine-tuning data, and evaluate on HP'-quiz. For TOFU, we use the original dataset for English fine-tuning, and the provided paraphrased questions for evaluation.
Evaluation. For HP-Quiz, we evaluate the model by prompting with MCQs in each language and report its accuracy in selecting the correct option (i.e., A/B/ //BCCDD. For TOFU, we prompt the model with questions each language and compute ROUGE scores between the model's 'ss response and the ground-truth English answer (used due to L.M''s tendency to output English after English fine-tuning, instead of the language in question).
Crosslingual barrier also exists for Harry Potter and TOFU knowledge. As shown in Fig. 6a for HI-Quiiz and Fig. 6b for TOFU, when presented with the same set of questions in 5 languages, the model consistently exhibits higher accuracy in English. After fine-tuning
Phttps://harryypoteer wwwdom.commwwiiii ht//wwiki ki /tt//Maaiaaaaaa Page
cwm/wikk
6

---

We explore potential methods to overcome the identified crosslingual knowledge barrier Published as a conference paper at COLM 2025
eegh,
Mira-11
Muri (jetrvinn))
Murd (iv Flod)
(a) Accuracy of LL the Harry Potter Quiz, before (left) and after (right) fine-tuning the model on in-domain content English.
pretrained
22-22-7
Iaamaz-1bb
mistral- 7b
Ilamad-ib
Figure 7: Off-the-shed LLMMs accuracy HI'-Quiz across 16 languages. LLMs perform best in en and perform better in high-resource languages than in low-resource ones. Mistral- 7B-v0.1 and Llama-3-8B show competitive performance com- pared to multilingual-focuse models such as the Aya and Tower series.
HOUSE
25
9
91
83
56
(b) ROUGE scores of LLMs on open-ended TOFU QA tasks, before and after fine-tuning on English TOFU data.
Figure 6: Models consistently perform best at answering questions in English, both before and after fine-tuning, indicating the presence of crosslingual knowledge barrier for domain-specific Harry Potter knowledge (a) and TOFU knowledge (b).
on domain-specific English corpora, despite the increase in model accuracy in English, the crosslingual knowledge barrier persists. This suggests that LLMs struggle to fully utilize the parametric knowledge acquired during English fine-tuning to answer related questions in other languages, and the barrier extends beyond general knowledge into specific domains.
As HP represents a widely known knowledge domain with practical relevance (e.g., users may query LLMs about HP in their native languages understand the books better), we evaluate 11 off-the-shelf LLMs on HP Quiz across 16 languages in Fig. 7. Results show LLMs perform best in English, with significantly lower accuracy on low-resource languages (e-g., bn, am) compared to high-resource ones (e.g., fr, de, es) which are our main focus. Notably, Mistral-7B and Llama3-8B achieve competitive performance against multilingual-foot sultllnngullfoouueed models like Aya and Tower series, validating the model and language selection in Fig. 6a.
Overcoming crosslingual knowledge barriers
identified: inference-time interventions (S 5.1), including prompt engineering and few- shot demonstrations, and training- time interventions ($ 5.2), including mixed-language fine-tuning on general and domain-specific corpora.
5.1 Inference-time mitigation
We study inference-time mitigation to improve LLM performance on mixup-translated MMLU, a challeng- ing crosslingual setting evidenced by the low performance in Fig. 4c. We evaluate the impact of prompt engi- neering and few-shot demonstrations crosslingual performance. For prompt engineering, we explored us- ing alternative option ID characters (a/b/c/d d or 1/2/3/4) to account for the possibility that the Arabic numer- als are more invariant to languages, and adding "multilingual awareness instructions" to encourage models to handle questions and options in dif- ferent languages. In addition to () shot setting that excludes biases in- troduced by the few-shot demonstra-
Table 1: Effect of inference-time mitigation methods. The highest accuracy achieved under the 0-shot/5-shot setting underlined. 4 denotes the accuracy drop ob- served in mixup MMLU compared to English MMLU. Sim- ple prompt engineering cannot address the cross-lingual knowledge barrier problem. Although few-shot demon- strations enhance accuracy compared the 0-shot setting, the performance gap between mixup MMLU and English MML.U remains significant. For reference, GPT-4 achieves 81 82 (0-shot) on English MML.U and 68.61 (13.21 (0-shot), 73.58 (5-shot English demos), 77.71 (4.11 (5-shot biased demos) on mixup MMLU.
Eval setup
Prompt
Llama2-78
Llama2-138 Mistral-7 Llama3-4t6
English abor)
A/B/C/D (ddaald)
52.13
A/B/C/D (detui))
32.14
11:52
17.00
Micspp (lohhtt)
a/b/cld
1/2/3/4
30.0
2596
01.65
47.7
43.56
11.00
Multiinuul
Multiiguul--neee instruction 1
31,23
41.35
46.00
47.9)
Micup (T-hoo)
English
35,23
43.15
09 66
52.90
Trarelate therrethhherr therowewww dencs
4255

---

Published as a conference paper at COLM 2025
tions (Zhao et al., 2021), we also evaluate 5-shot setting. We investigated three demonstration strategies: providing English examples, using mixup-translated examples matching the test sample's construction ("Same bias"), and a Sraasllatethen-Answe approach with demonstrations showing the mixup-translated MCQ, its English translation, and the answer. See detailed setups in ş D.2.
From results in Tb. 1, (1) regarding prompt engineering, we observe no improvement and even a performance drop compared to the default prompt. It suggests that the crosslingual knowledge barrier is an inherent failure of LLMs that cannot be effectively addressed by simple prompt engineering. (2) 5-shot settings consistently improve performance compared to 0-shot settings on mixup MMLU, as providing demonstrations in the corresponding subject helps LLMs generalize to lead to better performance than English demonstrations because the mixed language pat- tern in the demonstrations matches that of the test examples.
www.lldge-intensive tasks. (3) Mixup demonstrations
(4) Beanslate-then-Ansee
demonstrations are not effective. We observe failure patterns where, after translating to English, sometimes LLMs merely continue generating text without outputting the desired answer for the MCQ task. (5) Even under the best demonstration strategy, there still exists substantial accuracy gap in mixup MMLU compared to English MMLU.
5.2 Mixed-language fine-tuning
Given the limited success of inference-time interventions, we turn our attention to training- based methods that directly instill better crosslingual knowledge in the model itself.
Proposed method. We explore mixed-language fine-tuning (FT), where we explicitly con- struct a FT dataset comprising examples from multiple languages. To ensure a balanced representation of different languages, we split the training data into smaller units and randomly select a target language for each unit, translating the unit into that language if necessary. This approach also ensures that the translated data is of similar size as the original English data, enabling a fair comparison. Note that this approach differs from using parallel corpora, each unit is only presented in single language instead of all languages.
We use different choices for the smallest unit for translation, including the following settings: (1) Document level: the entire document (example) is translated to random language. (2) Sentence level: each document is split into units of sentences, using common English punctuation marks (Python regex r' (\s*[\.,: [?] s+) '). Each sentence is then translated independently. (3) k-word chunk level: the document is split into chunks of k words, where "word" is any consecutive sequence of characters separated by one or more non-word characters defined by the Python regex r' (W++) We found that the translation tool could be confused by k words that span across sentence boundaries, so we did a little tweak by splitting into sentence first, and then split each sentence into k-word chunks. Unless otherwise specified, for each translation unit, the target language is always randomly chosen uniformly from len, fr, de, es, it).
Mind Hnccoment
FILL Miset (eotewe))
Flat Moad (ooer)
Accaracy
Table 2:
Fine-tuning LLMs on the mixed- languages general corpus WikiText-103 can im- prove the performance on English and mixup MMLU benchmarks under (-shot & 5-shot settings. See Figs. 16 and 17 for results on more MMLU vari- ant benchmarks.
0.2
20si(fr, do,
It
fr Model:
de Liam22
Aucuracy
0.6
2r de, 2) fr
de
Moid
Maup MBILL
Stop MMLU
Model.
L'ama3-BB
Dedia willin
$1:11
31.6
4633
(7:83
Figure Fine-tuning on a mixed- languugg gen- eral corpus WikiText-2 enhances performance on domain-specific task, HP-Quiz, across multiple languages, including English. See Fig. 15 for re- sults on more LLMs.
Maad tovedo F7
$2.00
34.56
4835
53.86
5-byt Uh-T1
II ftndemel
(22g
darai
[In 4ena
Bia
43.53
5036
$3.55
Minliiag Inoeah
43.55
8:56
6655
3445
S034
8

---

Published as a conference paper at COLM 2025
Musil Uloua
Modd: Lamas-88
Accuracy
0.6
Ling New FT
0.6
(antanca)
Accuracy
0.4
0.8
de
da
lang
2h
he
bn
0.4
magins daa no)
matme da ft. na)
mafim da f. ne)
angine fo ne)
qpeating
optioes
Figure 9: Mixed-language FT on WikiText-2 with fen, fr, de, es, it] enhances accuracy on Harry Potter Quiz across other languages that are not used during FI. Such improvements incur in low resource languages (e.g,, ms, bn, am) and languages that are rather different from English (e-g., zh, ru, he, hi) with low amount of shared tokens.
Figure 10: Mixed-language FT on Wiki- 103 with (en, fr, de, es, it) enhances accuracy on MMLU variants across out- of-distribution low-resource languages (ms, da, fi, no).
We perform mixed -liguae FT of the original model on two types of corpora: general knowledge where we use WikiText-2 or WikiText-1033 and domain-specific knowledge where we use subset of Wikilext-1003 that is highly related to Harry Potter based on BM25 similarity ranking (the details are deferred to $ D.3).
The differences between mixed-language FT and our cross-lingual evaluation setups are noteworthy: i) In the
evaluation, Mixup MMLU operates mixed- questiuettonnn level, whereas langua- fine-tuning occurs at
language MCQs at the option/question the document, sentence, or word levels. ii) In the domain-specific knowledge evaluation, we use the fully translated HP-Quiz (i.e., without mixup pattern) to evaluate the crosslingual capabilities (as we have full control on the language in which the model learns the parametric knowledge during fine-tuning in S 4.2), resulting in more natural evaluation setup that is different from mixed-language FT. iii) We consider general Wikipedia documents as the fine-tuning corpus, which may not be directly related MMLU/HP /uiz tasks.
Mixed-language FT on general corpus WikiText-2. We finetune LLMs on general corpus, WikiText-2 (keyword searching suggests that WikiText-2 has no overlap with Harry Potter characters or spells), with different choices of translation units. Training details are deferred to $ E.3.As shown in Fig. 8, LLMs fine-tuned on mixed translated general corpora achieve higher accuracy on HP-Quiz than the model fine-tuned on the English corpus. This suggests that mixed-language FT could potentially improve crosslingual capabilities: by exposure to frequent language switch during fine-tuning, LLMs can better adapt to the setting when the same knowledge is asked in a different (and usually non- non- con-English) language.
Mixed-language FT general corpus WikiText-103. We also fine-tune LLMs on WikiText- 103, general corpus that offers a larger size and a broader range of knowledge compared to WikiText-2, and report the accuracy on MMLU variant benchmarks.
(1) As shown in Tb. 2, fine-tuning on English WikiText-103 corpora hurts the performance, likely because it is an out-of-domain corpora for MMLU tasks. However, fine-tuning on mixed translated WikiText- corpora can lead to improvements, which are particularly noticeable on the mixup MMLU benchmarks. These results indicate that multiple language switches during fine-tuning enable LLMs to better understand and process multilingual inputs, become more robust to variations in language and phrasing, and perform better in knowledge-intensiin crosslingual tasks. (2) Combining training- time interventions with test-time interventions can further enhance performance. While adding 5-shot biased demonstrations to our fine- tuned models leads to the best performance on mixup MMLU, adding 5-shot English demonstrations is also effective. (3) Fine-tuning with word/ rseenennceeeeeeee mixed language WikiText-103 improves MMLU performance. Word-level mixing slightly outperforms in 5-shot settings, while sentence-level mixing is more effective in O-shot settings.
Embedding analysis. To investigate how mixed-language FT improves crosslingual capabil- ities, we conduct a text embedding analysis with similar setups as in $ 3. we examine, in the fine-tuned model, if the embeddings for given English text are similar to the embeddings when some words are presented in different languages. The results in Appendix Fig. 18 show that mixed-language FTed models indeed have much smaller embedding distance compared to the original models, indicating the strengthened crosslingual correlation.

---

Published as a conference paper at COLM 2025
Mixed-language FT improves QA performance out-of-distribution languages. We evaluate our fine-tuned models on languages that were not included in fine-tuning data. Results in Fig. 9 show that mixed-language (with (en, fr, de, es, it]) fine-tuning general Wiki corpus can improve the cross-lingual performance of 11 other languages HP-Quiz, including low-resource ones and those substantially different from English. Furthermore, as shown in Fig. 10, mixed-language fine-tuning also boosts the performance of MMLU variants in various cross-lingual settings for four low-resource languages.
Mixed-language FT on domain-specific corpus. Similarly, we investigate the effectiveness of mixed-language fine-tuning for the domain-specific task. Specifically, we fine-tune the model on mixed-language versions of in-domain corpora (i.e., Harry Potter-related documents from WikiText- and evaluate performance on the HP-Quiz. For an upper bound reference, we also report results from fine-tuning a collection containing examples in all five languages (5x larger dataset size than our approach). As shown in Appendix Fig. 19, mixed-language fine-tuning (especially at sentence-level) can lead to better overall performance on HP--uiiz compared to English fine-tuning.
6 Conclusion and future work
In this work, we observe that despite the competitive performance of multilingual LLMs in explicit crosslingual tasks such as translation, they fail to transfer learned parametric knowledge across the language boundary, a phenomenon we termed as crosslingual knowl- edge barrier. Through comprehensive evaluations on both general and domain-specific knowledge, we confirm a systematic presence of such barriers across models and languages. Finally, we evaluate both test-time and training-tima mitigations and proposed an effective mixed-language fine-tuning procedure. We discuss our limitations and further work in $ A.
10

---

Published as a conference paper at COLM 2025
References
Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. Towards cleaner document-oriented multilingual crawled corpus. In ACL, 2022.
Jubin Abutalebi, Jean-Marie Annoni, Ivan Zimine, Alan J Pegna, Mohamed L Seghier, Hannelore Lee-Jahnke, François Lazeyras, Stefano F Cappa, and Asaid Khateb. Language control and lexical competition in bilinguals: an event-related fmri study. Cerebral Cortex, (177):199-1155,, 2008.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolingual representations. In ACL, 2020.
Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Kelly Marchisio, Sebastian Ruder, et al. Aya 23: Open weight releases further multilingual progress. arXiv preprint 15X2022405.15032, 2024.
Ankur Bapna and Orhan Firat. Simple, scalable adaptation for neural machine translation. In EMNLP, 2019.
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. LLM2vec: Large language models are secretly powerful text encoders. In COLM, 2024.
Himanshu Beniwal, Mayank Singh, et al. Cross-lingual editing in multilingual language models. EACL, 2024.
Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: LLMs trained on "A is B" fail to learn "B is A". In ICLR, 2024.
Ellen Bialystok, Fergus IM Craik, and Gigi Luk. Bilingualism: consequences for mind and brain. Trends in Cognitive Sciences, 1644):240-250, 2012.
Xinyun Chen, Ryan A Chi, Xuezhi Wang, and Denny Zhou. Premise order matters in reasoning with large language models. In ICML, 2024.
Guangran Cheng, Chuheng Zhang, Wenzhe Cai, Li Zhao, Changyin Sun, and Jiang Bian. Empowering large language models on robotic manipulation with affordance prompting, arXiv preprint ar XxXiv22404.1000 110277, 2024.
Nadezhda Chirkova and Vassilina Nikoulina. Key ingredients for effective zero-shot cross- lingual knowledge transfer in generative tasks. In NAACL, 2024a.
Nadezhda Chirkova and Vassilina Nikoulina. Zero-shot cross-lingual transfer in instruction tuning of large language model. INLG, 2024b.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the A12 reasoning challenge. arviv preprint arXiv:1803.05455 2018
Marta R Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arv preprint ar arXi::2207.0455 2022.
Jacob Eisenstein. Introduction to Natural Language Processing. MIT press, 2019.
Felix Friedrich, Simone Tedeschi, Patrick Schramowski, Manuel Brack, Roberto Navigli, Huu Nguyen, Bo Li, and Kristian Kersting. Lims lost in translation: M-alert uncovers cross-linguistic safety gaps. arXiv preprint 2.Xi::4412.1003.. 2024.
Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Physically grounded vision-language models for robotic manipulation. In ICRA, 2024.
11

---

Published as a conference paper at COLM 2025
Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7b ssm hybrid model. arXio preprint ar Xiv:2405.16712, 2024.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The flores- 101 evaluation benchmark for low-resource and multilingual machine translation. TACL, 10:522-538, 2022.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint ar atfo:2308.032960 2023.
Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang, Zhaopeng Tu, Shuming Shi, and Xing Wang. Exploring human-like translation strategy with large language models. TACL, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021.
Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. Not all languages are created equal in LLMs: Improving multilingual capability by phoss-lingual-thoughe prompting. In EMNLP, 2023.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decom- posed: How do neural networks generalise? JAIR, 67:757-795, 2020.
Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. X-factr: Multilingual factual knowledge retrieval from pretrained language models. In EMNLP, 2020.
Nora Kassner, Philipp Dufter, and Hinrich Schütze. Multilingual lama: Investigating knowledge in multilingual pretrained language models. In ACL, 2021.
Najoung Kim and Tal Linzen. COGS: compositional generalization challenge based on semantic interpretation. In EMNLP, 2020.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel] J Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253, 2017.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. In NeurlPS, 2019.
Jaeseong Lee, Yeonjoon Jung, and Seung-won Hwang. Commit: Code-mixing english- centric large language model for multilingual instruction tuning. In Findings of NÅACL 2024, pp. 3130-3137, 2024.
Shuang Li, Xuming Hu, Aiwei Liu, Yawen Yang, Fukun Ma, Philip S Yu, and Lijie Wen. Enhancing cross-lingual natural language inference by soft prompting with multilingual verbalizer. In ACL, 2022.
Tianjian Li and Kenton Murray. Why does zero-shot cross-lingual generation fail? an explanation and solution. In ACL Findings, 2023.
Seonghoon Lim, Taejun Yun, Jinhyeon Kim, Jihun Choi, and Taeuk Kim. Analysis of multi-source language training in cross-lingual transfer. In ACL, pp. 712-725, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and Zico Kolter. Tofu: A task of fictitious unlearning for llms. In COLM, 2024.
Viorica Marian and Anthony Shook. The cognitive benefits of being bilingual. In Cerebrum: the Dana forum Brain Science. Dana Foundation, 2012.
12

---

Published as a conference paper at COLM 2025
Kaushal Kumar Maurya, Maunendra Sankar Desarkar, Yoshinobu Kano, and Kumari Deepshikha. Zmbart: An unsupervised cross-lingual transfer framework for language generation. In ACL Findings, 2021.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. 2017.
Mistral. Mistral large.
2024.
Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla AerendAlme,,ros, Abinew Ali Ayele, et al. Blend: A benchmark for Ilms on everyday knowledge in diverse cultures and languages. Advances in Neural Information Processing Systems, 37:78104-78146, 2024.
Xenia Ohmer, Elia Bruni, and Dieuwke Hupkes. Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses. In ACL, 2023.
Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions. In ICLR, 2024.
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In ACL, 2019.
Jirui Qi, Raquel Fernández, and Arianna Bisazza. Cross-lingual consistency of factual knowledge in multilingual language models. In EMNLP, 2023.
Leonardo Ranaldi, Giulia Pucci, and Andre Freitas. Empowering cross-lingual abilities of instruction-tuned large language models by translation-followiig demonstrations. ACL findings, 2024.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. In EMNLP, 2020.
Michael J Ryan, William Held, and Diyi Yang. Unintended impacts of LLM alignment on global representation. In ACL, 2024.
Andrea Schioppa, Xavier Garcia, and Orhan Firat. Cross-lingual supervision improves large language models pre-training. arv preprint arXivo2305.1177 117788 2023.
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Édouard Grave, Armand Joulin, and Angela Fan. Ccmatrix: Mining billions of high-quality parallel sentences on the web. In ACL, pp. 6490-6500, 2021.
Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi. The language barrier: Dissecting safety challenges of lims in multilingual contexts. In ACL (Findings), 2024.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In ICML, 2023.
Kai Song, Yue Zhang, Heng Yu, Weihua Luo, Kun Wang, and Min Zhang. Code-switching for enhancing nmt with pre-specified translation. In NAACL, pp. 449 459, 2019.
Eshaan Tanwar, Subhabrata Dutta, Manish Borthakur, and Tanmoy Chakraborty. Multilin- gual LLMs are better cross-lingual in-context learners with alignment. In ACI., 2023.
Andrew Trotman, Antti Puurula, and Blake Burgess. Improvements to bm25 and language models examined. In Australasian Document Computing Synposium, pp. 38-65, 2014.
Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mohit Tyyer, and Noah Constant. Overcoming catastrophic forgetting in zero-shot cross- cross-lingual generation. In EMNLP, 2022.
Jiaan Wang, Yunlong Liang, Fandong Meng, Beiqi Zou, Zhixu Li, Jianfeng Qu, and Jie Zhou. Zero-shot cross-lingual summarization via large language models. In ACI, 2023a.
13

---

Published as a conference paper at COLM 2025
Jiaan Wang, Yunlong Liang, Zengkui Sun, Yuxuan Cao, and Jiarong Xu. Cross-lingual knowledge editing in large language models. In ACL, 2024a.
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In ACL, 2024b.
Yi Ru Wang, Jiafei Duan, Dieter Fox, and Siddhartha Srinivasa. Newton: Are large language
models capable of physical reasoning? In EMNLP, 2023b.
Shijie Wu and Mark Dredze. Beto, Bentz, Becas: The surprising cross-lingual effectiveness of BERT. In EMNLP, 2019.
Zhaofeng Wu, Ananth Balashankar, Yoon Jacob Fisenstein, and Ahmad Beirami. Reuse your rewards: Reward model transfer for zero-shot cross-lingual alignment. EMNLP, 2024.
Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. A paradigm shift in machine translation: Boosting translation performance of large language models. In ICI,R, 2024.
Shaoyang Xu, Junzhuo Li, and Deyi Xiong. Language representation projection: Can we transfer factual knowledge across languages in multilingual language models? In EMNLP, 2023a.
Yang Xu, Yutai Hou, Wanxiang Che, and Min Zhang. Language anisotropic cross-lingual model editing. In ACL, 2023b.
Zhenlin Xu, Marc Niethammer, and Colin A Raffel. Compositional generalization in unsupervised compositional representation learning: A study on disentanglement and emergent language. language In NeurIPS, 2022.
Zhen Yang, Bojie Hu, Ambyera Han, Shen Huang, and Qi Ju. Csp: code-switching pre- training for neural machine translation. In EMNLP, pp. 2624-2636, 2020.
Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. Geomlama: Geo-diverse commonsense probing on multilingual pre-trained language
models. EMNLP, pp. 2039-2055, 2022.
Haneul Yoo, Cheonbok Park, Sangdoo Yun, Alice Oh, and Hwaran Lee. Code-switching curriculum learning for multilingual transfer in 1lms. arXiv preprint (xXi::441100266) 2024.
Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-Mix: A flexible and expandable family of evaluations for Al models. In ICLR, 2024.
Puxuan Yu, Hongliang Fei, and Ping Li. Cross-lingual language model pretraining for retrieval. In The Web Conference, 2021.
Cedegao Zhang, Lionel Wong, Gabriel Grand, and Josh Tenenbaum. Grounded physical language understanding with probabilistic programs and simulated worlds. In Annual Meeting of the Cognitive Science Society, 2023a.
Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, et al. Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models. Xiv preprint ar arXiv:2306.1094 2023b.
Xinyang Zhang, Yury Malkov, Omar Florez, Serim Park, Brian McWilliams, Jiawei Han, and Ahmed El-Kishky. Twhin-bert: A eociall pre-trained language model for multilingual tweet representations at twitter. In KDD, pp. 5597-5607, 2023c.
Yiran Zhao, Wenxuan Zhang, Huiming Wang, Kenji Kawaguchi, and Lidong Bing. AdaMergeX: Cross-lingual transfer with large language models via adaptive adapter merging. arXiv preprint ar 18012140..18 2024.
14

---

Published as a conference paper at COLM 2025
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In ICML, 2021.
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In ICLR, 2024.
Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Extrapolating large language models to non-English by aligning languages. arXiv preprint ar XrXiv22308.0444. 2023.
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Multilingual machine translation with large language models: Empirical results and analysis. In NAACL, 2024.
15

---

Published as a conference paper at COLM 2025
Appendix
A Limitations and future work
17
Extended related work
17
C The Harry Potter Quiz dataset
20
D Experimental details
21
D.1 Evaluated models
21
D2 Evaluation and training details
22
D.3 WikiTeett1033 subset: Harry Potter-related documents
24
E Additional experimental results
24
E.1 Crosslingual capabilities of LLMs
24
E,2 Crosslingual knowledge barriers in LLMs
25
E.3
Mixed-language fine-tuning
26
16

---

Published as a conference paper at COLM 2025
A Limitations and future work
One limitation of our work is that all translations were performed using Google Translate. While Google Translate is recognized as a high-quality industrial translation service, vali- dating and enhancing the translation quality remains future work. In addition, although the proposed mixed-language fine-tuning improves crosslingual performance, a gap still remains between English and non-English languages.
One question that not answered in this paper is how these models develop the crosslingual capabilities (despite the existence of crosslingual knowledge barriers). This is intriguing because unlike humans exposed to multiple lingual environments, LLMs do not have grounding in the physical world to help them establish connections between different words that refer to the same thing in the real world. While there is some preliminary work on grounding LLMs with the physical world (e-g., Zhang al., 2023a; Wang et al., 2023b; Gao et al., 2024; Cheng et al., 2024), the majority of the LLMs nowadays are still trained via next-word prediction without interaction with the physical world. Therefore, an interesting future direction is to understand the mechanisms that allow the LLMs to develop crosslingual capabilities.
Another interesting observation is that mixed-language fine-tuning (on out-of-domain data) can improve the guestion-answering performance on both non-English languages and English in many of our evaluations. Previous studies (e.g., Abutalebi et al., 2008; Bialystok et al., 2012; Marian & Shook, 2012) have shown that multilinguality could have a positive effect on human cognitive abilities. But how does better crosslingual capabilities impact LLMs' reasoning abilities (in English) remains to be fully understood.
B Extended related work
Understanding and improving multilingual LMs. Understanding language models' per- formance in multilingual settings is active area of research. Prior works have identified strong variations in the amount of knowledge across different languages, attributed to differences in training corpora sizes (Jiang et al., 2020; Kassner et al., 2021; Ryan et al., 2024). These observations have also been leveraged to improve models' performance, especially in English. For instance, Ohmer et al. (2023) propose using multilingual self-consistency of predictions to assess how well the model understands a given task. Wu et al. (2024) suggest using a reward model in different language during fine-tuning for alignment from human feedback can yield better-aligned models than using one in the same language as the pre-trained model. Efforts have also been devoted to studying well-established tasks for monolingual models in crosslingual scenarios, such as crosslingual pretraining (Lample Conneau, 2019; Abadji et al., 2022; Schioppa et al., 2023), information retrieval (Yu et al., 2021), knowledge editing (Wang et al., 2024a; Xu et al., 2023a; Beniwal et al., 2024; Xu et al., 2023b), text summarization (Wang et al., 2023a; Huang et al., 2023) and instruction tuning (Chirkova & Nikoulina, 2024b; Zhang et al., 2023b; Ranaldi et al., 2024; Zhu et al., 2023).
The closest work to ours is Qi et al. (2023), which proposes metric to evaluate the con- sistency of a multilingual language model's factual knowledge across languages. They find that while increasing model size generally leads to higher factual accuracy in most languages, it does not necessarily improve crosslingual knowledge consistency. One key difference is that their study does not account for different factors that could contribute to the crosslingual consistency (e.g., a model independently learns the knowledge in both languages during pretraining could lead high consistency); while we formulate a con- trolled setting of crosslingual knowledge barrier, measuring precisely the ability to transfer knowledge learned (only) in one language to another language. Furthermore, we also proposed mitigation methods that could effectively reduce the knowledge barrier.
Machine translation ability of LLMs. The off-the-shelf pretrained LLMs show promise in machine translation but still lag behind the commercial translation system, especially in low-resource languages. Previous studies have sought to enhance LLM translation
17

---

Published as a conference paper at COLM 2025
capabilities through various prompting and fine-tuning methods. Zhu et al. (2024) introduce crosslingual translation in-context examples, while el (2024) employ advanced prompt engineering that induces translation-related knowledge (e.g., keywords, topics) from the given source sentence to guide the final translation process. Xu et al. (2024) propose a two-stage fine-tuning approach, first enhancing proficiency in non-English languages by fine-tuning on non-English monolingual data, and then fine-tuning on high-quality parallel data for translation task. Our work has different goal of comprehensively examining LLMs' crosslingual capabilities, beyond the translation task. We show that even though LLMs are very competitive at explicit translation tasks, they could struggle in more demanding tasks that requires implicit knowledge transfer across language boundaries.
Crosslingual transfer of multilingual models. Crosslingual transfer refers to transfer learning that fine-tunes the model on target task in one language (egg, English), and then makes predictions for this task in another, typically more low-resource language. It addresses the challenges of limited training data in the target language for a target task. It has been broadly studied for natural language understanding (Schioppa et al., 2023; Artetxe et al., 2020; Pires et al., 2019; Wu & Dredze, 2019; Li et al., 2022) and generation tasks (Chirkova & Nikoulina, 2024a; Bapna & Firat, 2019; Vu et al., 2022; Maurya et al., 2021; Li & Murray, 2023; Tanwar et al., 2023) for multilingual models such as mBART, mT5, NLLB family. For instance, Chirkova & Nikoulina (2024a) demonstrated that fine- tuning the full model with a small learning rate yields the best crosslingual language generation performance, outperforming other methods such adapter (Bapna & Firat, 2019), prompt-tuning (Vu et al., 2022)) and hyperparameter tuning (Chirkova & Nikoulina, 2024a). Additionally, several studies have improved crosslingual generalization by mixing auxiliary unsupervised data from additional languages during fine-tuning. For example, sampling target language examples with probability (e.g., 1%) when forming the mini- batch (Chirkova & Nikoulina, 2024a; Vu et al., 2022).
Our study focuses on more recent autoregressive LLMs (e.g., Llama series, Mistral, GPT-3.5, GPT-4) that acquire multilingual capabilities from their internet-scale pretraining corpora. While several works have explored approaches to enhancing LLMs' crosslingual transfer abilities such as fine-tuning with adapter merging (Zhao et al., 2024), our work differs in its primary focus. We aim to provide a comprehensive understanding of the crosslingual capabilities of pretrained LLMs on tasks requiring explicit (e.g., translation tasks) and implicit crosslingual transfer (e.g., question-anssering tasks involving general or domain- specific knowledge). Furthermore, to improve crosslingual transfer ability of LLMs in general, our study employs fine-tuning on (out-of-domain) general corpora and proposes a principled approach to processing mixed language data at different levels of granularity, including word, sentence, and document levels.
Compositional generalization. We also acknowledge that the crosslingual knowledge barrier can be viewed as an instance of the broader challenge of compositional generalization (Lake et al., 2017; Kim & Linzen, 2020; Hupkes et al., 2020; Xu et al., 2022; Yu et al., 2024) the ability to systematically combine different skills to understand and produce novel compositions not directly trained on. In the case of crosslingual knowledge understanding, models must compose the skills of question answering and knowledge translation. However, this specific combination of crosslingual knowledge consistency warrants dedicated study due to its strong practical implications, as ensuring consistent feedback across languages is crucial for deploying trustworthy and effective multilingual Al assistants to a global user base
Behavioral bias of LLMs. Recent research also studies various behaviors and biases in LLMs that are different from human reasoning, such as reversal curse (Berglund et al., 2024; Grosse et al., 2023), order and position bias (Wang et al., 2024b; Pezeshkpour & Hruschka, 2024), option ID bias in multiple-choice question tasks (Zheng et al., 2024), premise order bias (Chen et al., 2024), susceptibility to distraction by irrelevant context (Shi et al., 2023). These studies provide deeper understanding of LLMs and suggest various ways to improve those models. Our paper contributes to this important line of research from the perspective of crosslingual behaviors.
18

---

Published as a conference paper at COLM 2025
Code-switching. Code-switching training (Yang et al., 2020; Song et al., 2019) uses parallel text to teach models the relation between original and translated tokens, primarily for machine translation. Compared to code-switching, tuning does not create parallel text, and aims to encourage LLMs to cross language barriers without requiring architectural changes special handling of parallel text. Our approach can directly handle multiple languages while maintaining a similar number of tokens as the original dataset.
proposed mixed-language fine-
19

---

Published as a conference paper at COLM 2025
C The Harry Potter Quiz dataset
We use Harry Potter as a setting to mimic domain-specific knowledge, as it revolves around highly detailed and extensive fictional universe with its own unique characters, terminology, and concepts. We manually curate an English-only dataset named Harry Potter Quiz (or HP-Quiz in short) by collecting information about characters and magic spells from the Harry Potter Wiki pages*. For characters, we gather attributes such as gender, hair color, house", and relationships with other characters. Regarding magic spells, we collected data on the types of spells they belong to. We then curate multiple-choice questions and answers based on the collected information. Specifically, the dataset consists of 300 questions in total, 157 questions about characters and 143 questions about magic spells. We format these questions as multiple choice questions.
Below is the full list of characters and spells included in HP-Quuiz:
25 Characters Aberforth Dumbledore, Albus Potter, Ariana Dumbledore, Arthur Weasley, Astoria Malfoy, Cedric Diggory, Charles Weasley, Cho Chang, Draco Malfoy, Dudley Dursley, Euphemia Potter, Fleamont Potter, Harry Potter, Hermione Granger, James Potter I, Kendra Dumbledore, Lily J. Potter, Lucius Malfoy, Narcissa Malfoy, Percival Dumbledore, Petunia Dursley, Roger Davies, Ron Weasley, Scorpius Malfoy, William Weasley
143 Spells Aberto, Accio, Age Line, Alarte Ascendare, Alohomora, Anti-Cheating Spell, Anti-Apparition Charm, Anti-Disapparition Jinx, Anti-intruder jinx, Aparecium, Ap- pare Vestigium, Apparition, Aqua Eructo, Arania Exumai, Arresto Momentum, Arrow- shooting spell, Ascendio, Avada Kedavra, Avifors, Avenseguim, Babbling Curse, Bad- gering, Bat-Bogey Hex, Bedazzling Hex, Bewitched Snowballs, Bluebell Flames, Blue sparks, Bombarda, Bombarda Maxima, Bravery Charm, Bridge-conjuring spell, Broom jinx, Bubble-Hear Charm, Bubble Spell, Calvorio, Cantis, Capacious extremis, Carpe Retractum, Cascading Jinx, Caterwauling Charm, Cave inimicum, Celescere, Cheering Charm, Circumrota, Cistem Aperio, Colloportus, Colloshoo, Colovaria, Confringo, Confundo, Conjunctivitis Curse, Cracker Jinx, Cribbing Spell, Crinus Muto, Crucio, De- fodio, Deletrius, Densaugeo, Deprimo, Depulso, Descendo, Deterioration Hex, Diffindo, Diminuendo, Dissendium, Disillusionment Charm, Draconifors, Drought Charm, Duro, Ear-shrivelling Curse, Ebublio, Engorgio, Entrail-Expelling Curse, Epoximise, Erecto, Evanesce, Evanesco, Everte Statum, Expecto Patronum, Expelliarmus, Expulso, Extin- guishing Spell, Feather-light charm, Fianto Duri, Fidelius Charm, Fiendfyre, Finestra, Finite Incantatem, Finger-removing jinx, Firestorm, Flagrante Curse, Flagrate, Flame- Freezing Charm, Flask-conjuring spell, Flintifors, Flipendo, Flipendo Duo, Flipendo Maxima, Flipendo Tria, Flying charm, Fracto Strata, Fumos, Fumos Duo, Furnuncu- lus, Fur spell, Geminio, Glacius, Glacius Duo, Glacius Tria, Glisseo, Gripping Charm, Hair-thickening Charm, Herbifors, Herbivicus, Homenum Revelio, Homonculous Charm, Hurling Hex, Impedimenta, Imperio, Inanimatus Conjurus, Incarcerous, In- flatus, Jelly-Brain Jinx, Jelly-Fingers Curse, Knee-reversal hex, Langlock, Lapifors, Leek Jinx, Levicorpus, Liberacorpus, Locomotor Mortis, Melofors, Meteolojinx recanto, Mimblewimble, Multicorfors, Obscuro, Oppugno, Orbis, Orchideous, Pepper Breath, Petrificus Totalus, Piscifors, Point Me
'In Harry Potter, the magic spell is magical action used by witches and wizards to perform magic. #https://harrppot fandoo.
fandom. com/wiki, Page Hoowarts, the fictional boarding school of magic in the Harry Potter book series, is divided into four houses: Gryffindor, Slytherin, Ravenclaw, and Hufflepuff.
20

---

Published as a conference paper at COLM 2025
D Experimental details
D.1 Evaluated models
Tb. 3 provides the details of the models evaluated in our study,
Table 3: HuggingFace links or endpoint specifications for evaluated models.
Model
Llama2-78
Llama2-13B
https://uuginggacc....
Mistral-7B
https://hugging:
Llama3-80
GPT-3.5
openal.
5pt-3.5-turbo-O00 endpoint
GPT-
https://platfoww openai wtt.coo//ccccc/
http:///ccs///oaaaa////------o-0 and
--d--andd-------- -pt-4, 2-t-4496133 endpoint
aya-23-8B
https://hugginggien
Zamba-7B
httpss///uggging.ace..ooo ct.ca/ccc..cceeeeooooeeeeeeeeeee
aya-expanse-8b
Mistral-8x7B
https:
Qwen2.5-7B
Llama-3.1-8B
Towerinstruct-7B-0022
http:///wggggggeee.cc./o///bbeee 78-
Towerlnstruct7B-v0.11
https:/ http://ugggnddeccccooooaaae/eees
Table 4: Prompt templates for inference-time mitigation methods in mixup-translated MMLU evaluation. The templates are consistent across different evaluation setups, varying only in the language pattern of multiple-chiice questions.
Setting
Type
Prompt
Default prompt
The following are multiple choice questions (with answers) about (subject). [Mixup_ Maiiuu_MuliiChuuue Quessiinic
Answer:
Multilingual-AAar instruction 0
The following multiple choice questions answers) about (subject). Keep mind that the question and options may be presented in various languages. [Mixup_ MultiMuoicceueeeeateaa Answer:
O-shot
MulttlingullAwarr instruction 1
The following multiple choice questions (with answers) about (subject). Remember that the question and options can in different languages.
[Mixup Baiiuu__MultiCaaae MultiChoicoQuestion
Answer:
English demonstrations
The following multiple choice questions (with answers) about (subject).
[En_ hoieelhhhooiiiiiionnnooooo
Answer: (Answer_Dee Demol]
[Mixup_MultiChniiei MultiConicQuustionn)
Answer:
The following multiple choice questions (with answers) about (subject).
Same blas demonstrations
[Mixup MulisCooicoucsstion Demol)
Answer: (Answer Demol]
few-shot
[Mixup_MuliChoiceeee MultiChoiceQusstion)
Answer:
The following are multiple choice questions (with answers) about (subject). Remember that
Translate- Saaallle-Then-AAnnwee Then-Answer demonstrations
question and options be different languages. First translate them English.
Then output the answer.
Question: [Mixup BaaisuppMutiiiiii MulliChoiceuussticn Demol) }
Answer:
Translate the question and options into English, and then answer.
Question: [En MuuliChoiceQuestionn Lemeetlioo__mmo))
Answer:
(Answer Demol)
Question: [Mixup MultsChoicQuesttoon
Translate the question and options into English, and then answer.
Question:
21

---

Published as a conference paper at COLM 2025
D.2 Evaluation and training details
LLM Evaluation For MMLU evaluation, we follow the templates in its official codebase to construct the prompts for 0-shot and 5-shot settings. We employ a temperature of 0 for GPT-3.5 and GI'T4, and we select the choice with the highest logits score as the predicted answer for open-source models.
For the Harry Potter evaluation, we use the following prompt template, with an example shown below:
The following are multiple choice questions (with answers) about Harry Potter.
which house is Harry Potter belong to?
A. Ravenclaw
B. B.ylhhnrrin
C. Gryfffindor
D. D.Huffleuuff
After querying the model, we select the choice with the highest logical score as the predicted answer.
LLM Evaluation with Inference-Time Mitigation Approaches We study inference-time mitigation to improve LLM performance on mixup-translate MMLU, a challenging crosslin- gual setting evidenced by the low performance in Fig. 4c.
Prompt engineering. We evaluate the following prompting strategies: (1) Alternative option ID characters. We replace the default A/B/C/D with a/b/c/d or 1/2/3/4 4, motivated by selection bias evidence in option IDs for MCQ tasks (Zheng et al., 2024) and to account for the possibility that the Arabic numerals are more invariant to languages. (2) Multilingual awareness instruction: We add explicit instruction before MCQs (e-g., "Remember that the question and options can be in different languages") make LLMs aware of the potential presence of other languages.
Few-shot demonstrations. Our evaluation mainly considers the 0-shot setting, which excludes any biases introduced by the few-shot demonstrations (Zhao et al., 2021), but we also conduct 5-shot experiments further investigate crosslingual performance. MMLU covers 57 subjects, and the few-shot demonstrations for each subject are derived from the corresponding development set and shared across all test samples within the same subject. We employ following strategies to construct few-shot demonstrations: (1) English: English MCQ and answer pairs. (2) Same bias: mixup-translated MCQ and answer pairs, where each MCQ demonstration is constructed in the same way as the test sample. (3) Sransate-then-Answer
We prompt LLMs to first translate MCQ into English and then produce the answer. To help LLMs follow the explicit translation instruction, we provide demonstrations where each includes a mixup-translate MCQ, the corresponding English MCQ, and its answer.
We provide the prompt templates for inference-time mitigation methods in Tb. 4.
LLM Fine-tuning (1) For WikiText-103 fine-tuning, we fine-tune Llama3-8B for 200 steps, and Llama2-7B for 400 steps, with learning rate of 2 x 10-5 and batch size of 32. (2) For fine-tuning on WikiText-2 or Harry Potter related documents from WikiText-103,, we fine-tune the models for one epoch with the same set of hyperparameters. We use AdamW (Loshchilov & Hutter, 2018) as the optimizer.
Computation Resources All fine-tuning experiments are conducted on NVIDIA A100
GPU cards, each with 80GB of memory. For the fine-tuning experiments, each training step takes 5.2 seconds for the Llama2-7B model and 6.1 seconds for the Llama3-8B model, with batch size of 32. All LLM evaluation experiments can be conducted on one NVIDIA RTX A6000 GPU card with 48 GB of memory.
hhatps:://gthubb.cm/////eedd//
22

---

Published as a conference paper at COLM 2025
Table 5:
Top three most relevant documents to the Harry Potter universe in WikiText-103 based on BM25 document ranking. Keywords related to Harry Potter universe are bolded.
In Philosopher's Stone, Harry re @-@ enters the wizarding world age 11 and enrols in Hogwarts School of Witchcraft and Wizardry. He makes friends with fellow students Ron Weasley and Hermione Granger, and is mentored by the school's headmaster, Albus Dumbledore. He also meets Professor Severus Snape, who intensely dislikes and bullies him. Harry fights Voldemort several times while school, as the wizard tries to regain a physical form. Goblet of Fire, Harry is mysteriously entered a dangerous magical competition called the Triwizard Tournament, which he discovers a trap designed allow the return Lord Voldemort full strength. During Order of the Phoenix, Harry and several of his friends face against Voldemort's Death Eaters, group of Dark witches and wizards, and narrowly defeat them. In Half @-@ Blood Prince, Harry learns that Voldemort has divided his soul into several parts, creating horcruxess from various unknown objects to contain them; in this way he has ensured his immortality as long as at least one of the horcruxes still exists. Two these had already been destroyed, one a diary destroyed by Harry in the events of Chamber of Secrets and one ring destroyed by Dumbledore shortly before the events of Half @-@ Blood Prince. Dumbledore takes Harry along in the attempt to destroy a third horcrux contained in a locket. However the horerux has been taken by an unknown wizard, and upon their return Dumbledore is ambushed and disarmed by Draco Malfoy who cannot bring himself to kill him, then killed by Snape.
2 Luna, Ron, Ginny, and Neville join them in the forest and all six fly to the Ministry on, expecting to find and rescue Sirius. Once in the Department of Mysteries, Harry realises that his vision was falsely planted by Voldemort; however, he finds glass sphere that bears his and the Dark Lord's names. Death Eaters led by Lucius Malfoy attack in order capture the sphere, which prophecy concerning Harry and Lord Voldemort, which revealed to be the object Voldemort has been trying to oblain for the whole year, the Dark Lord believing that there was something he missed when he first heard the prophecy. Lucius explains that only the subjects of the prophecies, in this case Harry or Voldemort, can safely remove them from the shelves. Harry and friends, soon joined by members of the Order, enter a battle with the Death Eaters. Amidst the chaos, Bellatrix Lestrange kills Sirius and Harry faces Voldemort. Voldemort attempts to kill Harry, but Dumbledore prevents him and fights the Dark Lord to a stalemate. the midst the duel, Voldemort unsuccessfully tries to possess Harry in an attempt to get Dumbledore to kill the boy. Dumbledore does not do so and Voldemort escapes just as Cornelius Fudge appears, finally faced with first @-@ hand evidence that Voldemort has truly returned.
recording of
3
During another summer with his Aunt Petunia and Uncle Vernon, Harry Potter and Dudley are attacked. After using magic to save Dudley and himself, Harry is expelled from Hogwarts, but the decision is later rescinded. Harry is whisked off by group of wizards to Number 12, Grimmauld Place, the home of his godfather, Sirius Black. The house also serves as the headquarters of the Order of the Phoenix, of which Mr. and Mrs. Weasley, Remus Lupin, Mad @-@ Eye Moody, and Sirius members. Ron Weasley and Hermione Granger explain that the Order of the Phoenix is secret organisation led by Hogwarts headmaster Albus Dumbledore, dedicated to fighting Lord Voldemort and his followers, the Death Eaters. From the members of the Order, Harry and the others learn that Voldemort seeking object that he did not have prior to his first defeat, and assume this object to be weapon of some sort. Harry learns that the Ministry of Magic, led by Cornelius Fudge, is refusing to leknowledge Voldemort's return because of the trouble that doing so would cause, and has been running smear campaign against him and Dumbledore.
Aucurmeii contiriig kowerrds
(3)
60
-30
50
150
2001
250
350
000
Top
similr docaments
Figure 11: The top retrieved documents containing the Harry potter keywords.
23

---

Published as a conference paper at COLM 2025
D.3 WikiText-103 subset: Harry Potter-related documents
We employ the BM25 algorithm (Trotman et al., 2014) (BM stands for best matching) for document ranking', which is bag-of-words retrieval function that ranks documents based on the presence of query terms in each document. The Wikilext-103 corpus comprises M = 1,16,, 029 documents d; [M]. We concatenate the passages crawled from Harry Potter Wiki pages into a single document to use as a query q. We then calculate the similarity score between the query and each document in WikiText-103, denoted as s; Sim(d;,4) 9).
The top k = 3 relevant documents are listed in 5.
Additionally, we use the list of Harry Potter character names and spell names as keywords to evaluate the quality of the retrieved documents and to identify additional relevant documents. Fig. 11 illustrates the trend that as k increases, more documents containing the keywords are retrieved. Note that keyword matching not golden retrieval method and only serves reference because: (1) documents may not contain the full name of characters or spells (e.g,, "Harry" instead of "Harry Potter"), (2) some spell names are generic and have multiple meanings (e.g,, "Pack", "Avis").
Therefore, we combine the top documents retrieved by BM25 with keyword matching to create our final dataset. The final dataset contains 4,348 documents (0.37% of Wikilext-103), comprising: (1) the top k = 2000 documents retrieved by BM25. Of these, 106 documents contain at least one exact keyword. (2) An additional 2,358 documents that contain the keywords.
Table 6: COMET scores for machine translation tasks evaluated on FloRes-101 benchmark using multilingual LLMs (Llama2-7B, Mistral-7B, Llama2-13B, Llama3-8B, GPT-3.5, and GPT-4), models trained on parallel corpora (NLLB-3.3B)) and an industrial-grade translation API (Google Translate). Multilingual LLMs achieve competitive translation performance against dedicated translation models and the translation API.
English other languages
Other languages English
de
Avg
de
Avg
Llama2-7B
81.67
84.54
84.76
85.17
84.04
87
87.96
85.60
86.47
86.91
Llama2-138
71.63
79.91
81.00
76.68
77.81
88.26
88.91
85.83
86.99
87.50
Llama3-8B
73.89
81.19
81.16
87.65
Mistral-7B
76.18
78.46
80.04
76.64
77.83
87.73
88.05
85.72
86.11
86.90
Qwen2.5-7B
85.40
87.35
86.15
86.56
86.37
88.89
89.24
86.78
88.17
88.27
Llama-3.1-8B
80.19
85.73
84.94
84.90
83.94
88.89
88.32
85.64
86.72
87.39
TowerBase-7B-vv.11
85.43
88.01
86
88.44
86,99 69.99
88 13
88.41
85.16
87.74
87.36
GPT-3.5
87.53
86.97
86.40
86.46
89.14
89.42
87.47
88.41
88.61
GPT4
88.18
88.13
86.64
85.33
87.07
89.71
89.56
87.41
88.14
88.71
NLLB-3,3B
87.33
87.44
86.88
88.26
87.48
79.65
87.44
85.64
84.13
84.72
Google Translate
89.39
89.22
87.23
89.37
90.01
89.92
87.55
E Additional experimental results
E.1 Crosslingual capabilities of LLMs
Machine translation Tb. report the COMET score on 1LooRes-101 benchmark (Goyal et al., 2022) for two directions per language: en + and X-> en. It shows that multilingual LLMs;s translation ability is quite competitive when compared to translation models explicitly trained on parallel corpora or industrial-grant translation APIs
Embedding analysis Here verify whether the embeddings for given text in English are similar to the embeddings when some words are presented in different languages. The embedding is vector the average of the last layer's 's activations across all tokens in the sentence (BehnamGhader et al., 2024).
htttps:/ / htppp/ypp/o/////pp//////.//ggogt pwopect/rankk-baaaee
8The spell name "None" excluded due its generic nature.
24

---

Published as a conference paper at COLM 2025
arginil pertrbed
Llama2-7B
Llama2-18 13ma2-113
Mistral-7B
Llama3-8B
Mled sight editee
unt ug dalnnd tiker retlocmart Handen taker drrpect
Miael anga peeiiee 20611 replcrment
Mard Ling tearshiive Htaddn iplicamee Siopout
tolan rip acmmer
-31
Hardon tolan Eupour
tolen croper:
Detancs betwess origina perturbed (1f dimanaion)
Distance between original perturbedd [1F dimansion)
Uistance betweee or & perturbed (1 d mimmmic)))
Distanoe betwoon original & perturbed [1 dimeraios)
The embeddings of the original English text and the mixed Laakedd-anuagg--raann trans-aed text are closely aligned, unlike baselines with unrelated perturbations (e-g,, random token replacement or dropout). The ellipses represent the covariance confidence intervals.
Figure 12:
Setup. We randomly sample 1k examples from the WikiText-103 corpus (Merity et al., 2017), creating two versions for each: (1) The original text in English; (2) Mixed-language translated: for each word, with probability of 0.2 it is (independently) translated into random language selected from (en, fr, de, es, it] using Google Translate API. That is, each word has a 0.16 probability of being translated into non-English language. To establish baselines for comparison, we consider two scenarios representing an "upper bound" on the distance when perturbations are unrelated the original content: (1) Random Token Replacement: with a probability of p 0.16, each token is replaced with a random different token from the vocabulary space of the tokenizer; and (2) Random Token Dropout: with probability of p 0.16, token is completely masked out by disallowing any attention to it.
To visualize and compare embeddings, we reduce the original 4096-dimensional vectors to 2D using non-linear dimensionality reduction. We then calculate and visualize the per- coordinate distances between 2D embeddings of original English text and mixed-language translations, comparing these to baseline scenarios.
As shown in Fig. 12, for the four LLMs multilingual, including Llama2-7B, Llama2-13B. Mistral-7B and Llama3-8B', the embeddings of the original text and its mixxed-rranslater counterpart exhibit high degree of similarity, with their difference vectors clustering around the origin. This observation stands in stark contrast to the scenario where English words are replaced with random tokens. implies the explicit crosslingual capabilities of the multilingual LLMs.
E.2 Crosslingual knowledge barriers in LLMs
Evaluation strategy We follow prior work on LLM evaluation (Zheng et al., 2024) to use two evaluation strategies: (1) Open-source: access the output probabilities of option ID tokens A/B/C/D and predict argmax. (2) Closed-source: compare the golden answer with the 1st generated token (decoding temperature=0), as the logits are not available for most closed-source models.
we evaluated the open-source LLMs using the 1st token produced as the answer in Tb. 7. The two evaluation strategies have minimal impact on accuracy for 5-shot settings, as demonstrations help regularize the output format. The difference is more evident the 0-shot setting, likely related to the specific tokenizers. E.g., Llama3-8B treats the 2 characters " A" as token, and has a slightly higher accuracy when using the 1st generated token as the answer. Conversely, Llama2-7B and Mistral-7B tokenizers treat " "A" as 1 token. Using the option ID token with the highest probability as the answer for those models generally leads to higher accuracy because it disregards the generation probability of other irrelevant tokens, e.g., "An",
Crosslingual evaluation of additional models on MMLU knowledge We present addi- tional results for Llama2-7B, Zamba-7B, and Mixtral-8x7B. Figure 13 shows the monolingual
We focus primarily on open-source models due to the cost associated with querying embeddings from proprietary models.
25

---

Published as a conference paper at COLM 2025
Table 7: Comparing two evaluation strategies for open-source LLMs: option ID token with maximum probability and first new token.
Model
Eval
English MMLU
Mixup MMLU
Max prob
token
Max prob
Llama2-7B
0-shot
41.53
37.74
32.18
27.47
5-shot
45.88
45,90
36.92
36.96
Mistral-7B
0-shot
60.21
58.41
47.86
42.29
5-shot
62.57
62.54
51.07
51.15
Llama3-8B
0-shot 5-shot
60.54
62.11
48.62
50.13
65.00
65.39
51.65
52.01
avg(fr. de. es, it)
fr
de
it
Llama2. 7B-hf
Llama2. 13B-hr hf
Llama3-8B
Accuracy
Accuracy
0.6
Accuracy
9.7
0-4
au 95
0.3
Overall STEM humanitoocial sci others
Overall
STEM humanity Social sch schvotters
Overall STEM, humani Social others
1lama-3.1-8B
Qwen2 5-78
Zamba- Tamba-7B.vl-phan
Accuracy
0.76
Accuracy
0.5
Accuracy
0.6
Overall STEM nociaitgooiall others
Overall STEM humanitty Social scherhers
Overall STEM, humanit others
Mistral-BBvv..1
Mixtral-Bx7B-vO..
Frral-Nermo-Beee-4400
Acpuracy
0.7
Accuracy
8.
Accuracy
35
0.3
Overall STEM humanityocial others
Overall STEM humanity. others
Overall STEM, numanisyyy Sch others
TowerBase-7B-0..
Towerinstruct-7B-000
Aya-23-8B
Accuracy
04
Accuracy
Accuracy
8.4
0.3
Overall STEM humanity.ccall sci others
Overall STEM, Vacial sci others
Overall STEM,
humanity STENnumantty Só others
Aya-expanse-B88
GPT-3.5-turbo-D1.
GPT-4-0513
Accuracy
077
Accuracy
0.70
Accuracy
06
0,50
Overall STEM
humanity Social sci others
Overall
STEM humanity sdi others
Overali
STEM,
humanity gia sci others
Figure 13: Monolingual evaluation LLMs on MMLU (fully translated for non-Ene oon-English languages). LLMs consistently perform better at answering multi-choice questions in English than in other languages.
evaluation of LLMs on MMLU, fully translated for non-English languages. The models consistently achieve higher accuracy in English compared to other languages.
Figure displays the accuracy of LLMs on MMLU variant benchmarks. We observe significant drop in accuracy under crosslingual MCQ evaluation, especially for ground-truth translated MMLU variant, indicating cross-lingual knowledge barriers. The barrier is more pronounced in Llama2-7B and Zamba-7B than in Mixtral-8x7B, possibly due to the larger capacity and multilingual capabilities of Mixtral-8x7B.
E.3
Mixed-language fine-tuning
HP Quiz evaluation results LLMs fine-tuned on WikiText-2 We finetune for one epoch, with a learning rate of x 10-5 and a batch size of 32. Fig. 8 in the main paper presents the Harry Potter Quiz evaluation results on Llama2-7B and Llama3-8B models fine-tuned on general knowledge corpora (i.e,, WikiText-2). Fig. 15 presents additional results for the Llama2-13B (left) and Mistral 7B (right) models. (1) The trends are consistent with
26

---

Published as a conference paper at COLM 2025
options-tranee trann*
question-trans
gt gt-trans*
mixup-trans
Llama2-7B-hf
Llama2-13B-hf
Llama3-8B
Accuracy
0.50
Accuracy
Accuracy
0.0
0.4
Overall STEM humanityacial others
Overall STEM, yuman Social sci, others
Ouerali STEM, humanity sch others
Llama-3.1-88
Qwen2.5-7B
Zamba-7B-u1-phase
Accuracy
0.75
Accuracy
0.75
Accuracy
0.30
0.50
0.95
Overall STEM burmanityociaa Social others
Overall STEM Numanity.ciol sci
Overall STEM humaningaaial others
Mistral-7B-v0l
(x7tral-8x8.vv..1
Mistral-Nemo Base Base-2407 2407
Accuracy
Accuracy
0.75
Accuracy
us
0.50
Overall STEM, humaniiyy Social scl
Overall STEM humanity Social scl
Overali STEM,
humanity Social sch others
ToverBase-7BB..o 7B.v0.1
Cowerrinstruct-78B.000
Aya-23-8B
Accuracy
0-4
Accuracy
Acturary
0.8
03
83
Overall STEM, humanity..ld yocial
Querall STEM, humanityocial sci others
Overall STEM, humanity,. sch others
Aye-expanse-8B
GPT-3.5-turbo-OI1
GPT-4-0613
Accurayy
07
Accuracy
0.75
Accurayy
05
0.6
Overall STEM humaniiyy cicil
Overall STEM pusa
others
Overali STEM, humanity Social sci
Figure 14: Crosslingual evaluation of LLMs on MMLU variant benchmarks. The bars with denotes the average accuracy across (fr, de, es, and it). LLMs perform better at answering MCQs in English than in mangualangu settings, especially the ground truth option and mixup translation, indicating the existence of cross-lingual knowledge barriers. Due to budget constraints, GPT-4 is evaluated only in the most challenging settings.
27

---

Published as a conference paper at COLM 2025
those observed for Llama2-7B and Llama3-8B, where fine-tuning mixed-language general corpus, WikiText-2, enhances the models' performance on the domain-specific HP Quiz task across multiple languages, including English. (2) Word-level language mixing is generally most effective for Llama2-13B, whereas sentence-level mixing is more effective for Mistral-7B.
Mixed (document)
FTed, Mixed (sentence)
Mixed words)
Accuracy
Accuracy
0.5
0.6
0.4
0.3
on
afg(fr. de, [t]
65
it
on avg(fr. de, it)
de
05
Model: Llama2-13B
Model:
Mistral-7B
Figure 15: Fine-tuning on a mixed-language general corpus (e.g, WikiText-2) enhances the model's performance domain-specific task (e.g,, Harry Potter knowledge test) across multiple languages, including English.
MMLU evaluation results of LLMs fine-tuned on WikiText-103 Tb. in the main paper presents the English MMLU and mixup MMLU evaluation results on Llama2-7B and Llama3- 8B models fine-tuned on general knowledge corpora (i.e,, WikiText-103). Here we present additional results for Llama2-7B (Fig. 16) and Llama3-8B (Fig. 17) on more MMLU variant benchmarks, including full translation, question translation, options transition, and ground- truth option translation. We report the average accuracy (with *) across four non-English languages (fr, de, es, and it for those settings.
(1) As shown in Fig. 16 and Fig. 17, models fine-tuned on mixed language WikiText-103 (whether at the word level or sentence level) generally achieve better performance than those fine-tuned on the original English WikiText-103 or the non-fine-tuned models, especially in the GT-option translated and mixup translated MMLU setups. These two evaluation setups originally had the lowest performance for the non-finetuned model, and thus the cross-lingual ability gains after fine-tuning are more apparent. These results suggest that multiple language switches during fine-tuning enable LLMs to better understand and process multilingual input and leverage cross-lingual knowledge for commonsense reasoning tasks. (2) An exception to this trend is observed with the GT-option translated MMLU under the 5-shot biased demonstrations setting for Llama3-8B, where performance drops. This drop likely due the non-fine-tuned Llama3-8B's stronger tendency follow biased demonstrations, using a shortcut to select the non-English option as the answer. (3) Fine-tuning models on a mixed-languages corpus performs better than other models across different 0-shot and few-shot scenarios, particularly in the 0-shot setting and 5-shot English demonstrations setting. While 5-shot biased demonstrations lead to the best performance, they are less applicable than English demonstrations in real-world scenarios, we cannot know in advance the language mixing pattern of user queries.
28

---

Published as a conference paper at COLM 2025
Un-FTed
Fied,
Fied. Mood (santance))
Fied. Moxd work))
en (0-shot)
en (5-shot)
Accuracy
0.5
01.4
full* (0-shot)
full* (en 5-shat)
full (bias 5-shot)
Accuracy
0.4
0.40
6:40
0.35
0.85
0.3
question (0 shot)
question" (en 5-shot)
question (bias 5 shot)
Accuracy
0.45
0.40
0.45
0.40
0.35
0.55
options (0-shot)
options (en 5-shot)
options (bias 5-shot)
Accuracy
0.4
0.45
0.40
03
0.35
gt* shot)
at (en shott)
gt" (bias 5-shot)
Accuracy
0.3
0.3
0.2
mixup (0-shot)
mixup (en 5-shot)
mixup (bias 5-shot)
Accuracy
0.4
0.4
0.4
03
0.5
Overall
STEM
humanity Rocial sci others
Overall
STEM
humanitt
Yocial sci. others
Overall
STEM, hurnanity Social sci others
Figure 16: Performance of Llama2-7B models on MMLU variant benchmarks. Fine-tuning on mixed language WikiText-103 generally outperforms fine-tuning on English WikiText-1003 or using the non- fine-tuned model.
29

---

Published as a conference paper at COLM 2025
UToF
FTed,
FTed. Mixed (sentonce)
FTod. Mixed word))
(0-shot)
en (5-shot)
Accuracy
0.7
0.6
0.6
0.5
full" (0 shat)
full (en 5 shot)
full" (bias 5 shot)
Accuracy
0.5
0.5
0.5
question* (0-shot)
question" (en 5-shot)
question" (bias 5-shot)
Accuracy
0.7
0.6
0.6
0.6
0.5
0.5
options* (0-shot)
options* (en 5-shot)
options* (bias 5-shot)
Accuracy
0.7
0.7
0.6
0.6
0.5
0.5
0.5
gt (0-shot)
gt* (en shot)
gt* (bias S-shot)
Accuracy
0.77
0.50
0.4
mixup (D-shot)
mixup (en 5-shot)
mixup (bias 5-shot)
Acouracy
0.6
0.6
0.5
0.5
0.5
Overall
STEM,
humanity Social sch others
Overall
STEM
humanity Social sch others
Overall
STEM
lumanitycial sch others
Figure 17: Performance of Llama3-8B models on MMLU variant benchmarks. Fine-tuning on mixed language WikiText-103 generally outperforms fine-tuning English WikiText-103 or using the non- fine-tuned model.
Llama2
Llama3-88 8B
Diflaoncc between origina miaal transiated
(2M (imensio))
Onggel
Drigica model
Origiil moad (onridoxs diseel
Oriainal model foleno clippe)
After Manklarg FT
After Mieeiling FT
Mard long (porideers
Ahw josttewee
between original (1" dimension)
original
(1 dimension)
Figure 18: After sentence-level mixed-lang FT, embeddings of original English text & mixed- langua--- translated text are more closely aligned, indicating stronger knowledge correlation between En other langs.
30

---

Published as a conference paper at COLM 2025
0.6
Indomain FTed, an
Instomain FTrd, Mived (8 words) Inciomain
Indommiin FTedd, Mited (oettence)
Accuracy
0.5
0.4
0.3
en avg(fr, de, es, it) fr de
es
it
Model:
Llama2-7B
Figure 19: Fine-tuning on mixed-language domain-specific corpus (i.e., Harry Potter related documents from WikiText-103) generally enhances the performance on the Harry Potter Quiz dataset across multiple languages, including English.
31


Reviews:
[{"review_id": "Zgx1rWJ2K7", "reviewer": "Reviewer_jfyw", "summary": "This paper studies cross-linguality, where inputs are mixed with multiple languages.  The authors first show that En inputs and the code-mixed inputs are close in the embedding space. Then, the authors randomly mix language at fixed-length k-word/sentence/document units for MMLU, Harry Potter Quiz, and Tofu to identify cross-lingual barriers for general knowledge and domain-specific knowledge. Finally, the authors present mixed language fine-tuning to address cross-lingual problems.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.  The presentation is good. \n2.  The problem is important and practical as information in mixed languages is common in the real world.\n3.  The authors show 3 strategies beyond classic word-level code-switching training and fine-tuning.", "reasons_to_reject": "1. While the authors state \"the same bias\" in testing and fine-tuning, the final results may not convince as the model is fine-tuned for the test set.\n2.  The paper mainly focuses on European languages that significantly share the same culture, scripts, and other confounding factors, which may limit the generalizability of the findings.\n3. The general idea overlaps with an existing work https://openreview.net/pdf?id=HMa8mIiBT8, where entity-level code-switching is leveraged for identifying cross-lingual problems. However, this is a minor concern.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gUxgldNPp8", "reviewer": "Reviewer_ar6x", "summary": "This paper investigates the cross-lingual capabilities of multilingual large language models (LLMs) and identifies a significant \"cross-lingual  knowledge barrier\" that hinders these models from effectively transferring knowledge across languages. The study evaluates state-of-the-art LLMs on cross-lingual  tasks, including machine translation and crosslingual question-answering (QA), using datasets like MMLU and a manually curated Harry Potter Quiz. The authors find that while LLMs demonstrate promising surface-level cross-lingual  abilities in machine translation, they struggle with deeper cross-lingual  knowledge transfer, especially in knowledge-intensive tasks that require implicit cross-lingual  correlation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.The authors propose novel cross-lingual evaluation formats (e.g., Mixup translation, Question+GT-option translation, GT-option translation) to directly test LLMs' cross-lingual capabilities. These formats effectively reveal the cross-lingual knowledge barriers faced by LLMs in multilingual settings, addressing the limitations of traditional monolingual evaluation methods.\n2.Give credit to the author for carrying out extensive experiments on multiple datasets and languages, providing a systematic analysis of the cross-lingual knowledge barrier phenomenon.\nOverall I think it's a well-written paper.", "reasons_to_reject": "I would like to see more experiment on low-resource languages to see whether the corpus size correlate with the knowledge transfer efficiency. The proposed method’s efficacy for truly low-resource settings requires deeper validation.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IhBoHUuLgK", "reviewer": "Reviewer_riAs", "summary": "This paper explores the ability of multilingual Large Language Models (LLMs) to transfer parametric knowledge across language boundaries, moving beyond surface-level capabilities like machine translation. The study demonstrates that LLMs have limited capabilities in transferring knowledge across languages, revealing what the authors term the \"crosslingual knowledge barrier\". The barrier is observed in both general knowledge settings using the MMLU benchmark and domain-specific contexts using the Harry Potter quiz and TOFU benchmark. To systematically evaluate this, the authors conducted extensive evaluations on 15 models and 16 languages. They designed novel mixed-language evaluation formats, such as mixed-language multiple-choice questions (MCQs), to directly probe crosslingual abilities by requiring models to utilise knowledge learned in one language to answer questions or perform tasks in a different language.\nThe authors also explored potential methods to overcome this barrier: inference-time interventions (like prompt engineering and few-shot demonstrations) and training-time interventions (specifically, mixed-language fine-tuning).", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Clear Introduction:  The paper presents an important problem in the multilingual Natural Language Process considering more than handling multilingual input but to crosslingual parametric knowledge transfer. The new concept of \"crosslingual knowledge barrier\" is clearly defined and intriguing.\n\nLogical Methodology: The paper presents a well-constructed and systematic empirical framework with well-designed evaluation pipeline and depth in problem exploration.  The methodology progresses logically from standard translation and embedding similarity analyses to controlled crosslingual question answering settings. The proposed mixed-language MCQ formats are considered novel and valuable for probing the barrier in controlled ways. The experiments explores both domain-specific (Harry Potter Quiz, TOFU) and general (MMLU) benchmarks. The authors also explored multiple existing solutions before proposing their method.  The proposed mixed-language fine-tuning method is described as simple, compute-friendly, and an attractive recipe for practitioners. It is shown to improve performance even with relatively small corpora.  \n\n\nCareful Analysis and DIscussion:  The analysis includes examining failure modes, such as models exhibiting an English-option bias. The finding that the barrier persists even after English-only domain-specific fine-tuning strengthens the paper's central claim.", "reasons_to_reject": "A significant concern raised is that the main experiments primarily focus on a limited set of high-resource, typologically similar languages (English, French, German, Spanish, Italian). These languages share script and substantial vocabulary with English, which may limit the generalizability of the findings. The behavior of the barrier for genuinely low-resource or non-Latin languages is noted as being only touched upon in an appendix figure, suggesting the main claims would be stronger with systematic inclusion of such languages.\n\nAlthough the paper references prior literature on code-switched training and contrasts its mixed-language fine-tuning approach, it stops short of conducting a controlled experimental comparison against traditional code-switched fine-tuning under equal corpus size. This leaves the claim that the proposed mixed-language fine-tuning is more effective unsubstantiated.\n\nThe mitigation technique (mixed-language fine-tuning that is similar to code-switching and random multilingual substitution during fine-tuning) are not novel, suggesting the novelty of the proposed solution might be incremental.", "questions_to_authors": "Why did you select the Harry Potter Quiz dataset to identify crosslingual language barriers?  The book is originally written in English, so does not consider language-specific knowledge, though the paper argues this domain is suitable for controlled testing of domain-specific knowledge transfer.\n\nThe paper could benefit from probing-based or attention-based diagnostics to explain why the barrier arises, beyond just behavioral metrics.\n\nFurther analysis and tests are required to assess the significance of the work, including a benchmark comparison with code-switching.\n\nWhat are the specific languages used in the mixed-language fine-tuning and the zero-shot languages shown in Figure 9.", "extra_scores": {}}, {"review_id": "MU01LuT71Z", "reviewer": "Reviewer_JPW1", "summary": "This paper shows that LLMs have surface-level crosslingual abilities, like machine translation, but demonstrate a crosslingual knowledge barrier -- a phenomenon where the model learns knowledge in one language, typically English, but cannot transfer the knowledge when queried in other languages. The authors also propose viable methods that can overcome the crosslingual knowledge barrier.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and easy to follow. I quite enjoy reading the paper.\n- The experiments are well-established and very extensive.\n- The paper not only showcases the existence of a crosslingual knowledge barrier but also presents promising methods to address it -- which is appreciated.", "reasons_to_reject": "- I have concerns regarding the mixed-language evaluation. The authors claim that \"general knowledge might be present in each of\nthe evaluated languages in the pretraining dataset\" in Lines 192-193. I think this case also applies to mixed-language evaluation -- the model can still have the knowledge in each of the language set {en, fr, de, es, it}. Would it be possible if the model does not have a crosslingual barrier (cannot leverage knowledge learned in other languages) but simply is confused in a multilingual context? \n\n- In Section 4.2, the model after being fine-tuned on domain data in English, the accuracy improves slightly for some models. However, we also see some other languages actually improve. I think a better way to quantify the effect of English-centric domain-specific fine-tuning is to look at those questions where the original model answers wrongly but correctly after fine-tuning. If other languages do not show consistent improvement in those questions, it is a more direct proof that the crosslingual barrier also exists, even with fine-tuning.", "questions_to_authors": "Line 149: (4) Random Token Replacement -> (4) Random Token Dropout ?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper assesses how well multilingual large language models (LLMs) perform cross-lingual tasks. The main contribution of the paper is in identifying a significant "cross-lingual knowledge barrier" that hinders these models from effectively transferring knowledge across languages. To fix this problem, the paper presents mixed language fine-tuning. The experiments mix language at fixed-length k-word/sentence/document units for MMLU, Harry Potter Quiz, and Tofu and show that the proposed solution mitigates the problem.

Reviewers generally feel positively about this paper, and so do I. The paper addresses a relevant problem and proposes an effective, simple solution to this problem, as well as a way to evaluate cross-lingual performance. Experiments are convincing. The main weaknesses are the relatively low novelty of the proposed solution and the focus on high-resource European languages.
Decision: Accept