Language Specific Knowledge: Do Models Know Better in X than in English?

## Abstract

Abstract Code-switching is a common phenomenon of alternating between different languages in the same utterance, thought, or conversation. We posit that humans code-switch because they feel more comfortable talking about certain topics and domains in one language than another. With the rise of knowledge-intensive language models, we ask ourselves the next, natural question: Could models hold more knowledge on some topics in some language X? More importantly, could we improve reasoning by changing the language that reasoning is performed in ? We coin the term Language Specific Knowledge (LSK) to represent this phenomenon. As ethnic cultures tend to develop alongside different languages, we employ culture-specific datasets (that contain knowledge about cultural and social behavioral norms). We find that language models can perform better when using chain-of-thought reasoning in some languages other than English, sometimes even better in low-resource languages. Paired with previous works showing that semantic similarity does not equate to representational similarity, we hypothesize that culturally specific texts occur more abundantly in corresponding languages, enabling specific knowledge to occur only in specific "expert" languages. Motivated by our initial results, we design a simple methodology called LSKExtractor to benchmark the language-specific knowledge present in a language model and, then, exploit it during inference. We show our results on various models and datasets, showing an average relative improvement of 10% in accuracy. Our research contributes to the open-source 1 1 1 https://anonymous.4open.science/r/LSKExtractor-272F/ development of language models that are inclusive and more aligned with the cultural and linguistic contexts in which they are deployed.

## 1 Introduction

The phenomenon of code-switching, alternating between languages within the same utterance or conversation, has long been observed in human communication and was formally studied as early as 1982 (Joshi, 1982). Instances of language models engaging in code-switching are increasingly observed in real-world applications. For example, DeepSeek-R1 (DeepSeek-AI et al., 2025) has been reported to spontaneously "think" in Chinese when presented with an English query (Marjanović et al., 2025). This behavior supports our hypothesis that language models may exhibit stronger performance or show a preference for certain languages when responding to specific topics. Rather than viewing this as a limitation, we argue that such behavior should be leveraged in a more informed and intentional manner, allowing us to guide language models toward languages that may yield more accurate, aligned, and culturally appropriate responses for a given topic.

We define Language Specific Knowledge (LSK) as knowledge that a language model appears to access more readily or represent more accurately when queried in a particular language (the expert language).
To illustrate this phenomenon, we conduct a simple experiment by asking GPT-4 Turbo about the tradition of exchanging wealth during marriage in English, Hindi, and Arabic (see Figure [1](https://arxiv.org/html/2505.14990v1#S1.F1)). In response to the Hindi query, grounded in the Indian cultural context, the model describes the concept of Dahej (dowry), where the bride’s family gives wealth to the groom’s family. In contrast, the Arabic query elicits a response centered on Mahr, an Islamic tradition in which the groom or his family gives wealth to the bride. The English query, however, results in a culturally neutral explanation using broad terms like "dowry" and "bride price," without referencing any specific tradition. These culturally grounded responses demonstrate that concepts such as Dahej and Mahr are examples of LSK. When queried in languages associated with the relevant cultural context, the model is more likely to produce culturally appropriate and specific answers.

Figure: Figure 1: A user asks the same question, "What is the tradition of one family giving another family money during marriage called?", in English, Hindi, and Arabic. Because each language is associated with a different cultural context, the model returns distinct concepts: Dahej in Hindi, where the bride’s family gives wealth to the groom’s family, and Mahr in Arabic, where the groom or his family gives wealth to the bride. This showcases examples of LSK, where culturally grounded responses emerge based on the language of the query.
Refer to caption: x1.png

We hypothesize that humans code-switch because they possess greater knowledge or fluency in certain topics or domains when using one language over another. In this work, we extend this intuition to language models by investigating the role of code-switching in enhancing model performance. We examine whether language models can benefit from multilingual reasoning, and explore how such models can be guided to effectively leverage code-switching during inference.

To operationalize our hypotheses, we introduce LSKExtractor, a two-stage framework (see Figure [2](https://arxiv.org/html/2505.14990v1#S3.F2)) designed to identify and leverage expert languages for Language Specific Knowledge during inference. In the first stage of "Mapping LSK", we map out LSK and their corresponding expert languages by conducting chain-of-thought (CoT) reasoning in 13 languages on training queries from CultureAtlas (Fung et al., 2024), BLEnD (Myung et al., 2025), and Social IQa (Sap et al., 2019). These queries are clustered in a shared semantic space, and each cluster is assigned an expert language based on the CoT language that achieves the highest performance within that group. In the second stage of "LSK-Informed Reasoning", during test-time inference, we embed an unseen query into the same space to identify its corresponding cluster and retrieve the optimal language for reasoning. The final answers are generated using CoT in the language identified as the expert for that knowledge region. This scalable method allows models to draw upon multilingual strengths dynamically, relatively improving accuracy by 10% without additional fine-tuning across all models and datasets.

Through this study, we hope to shed light on the phenomenon of Language Specific Knowledge in language models and demonstrate how code-switching can be used to uncover and exploit culturally grounded knowledge embedded in different languages. Our contributions are as follows:

- •
We formally define Language Specific Knowledge (LSK) and provide intuitive and empirical evidence of its presence in multilingual language models.
- •
We propose LSKExtractor, a scalable two-stage framework that identifies expert languages for specific knowledge regions and leverages this alignment to improve inference through strategic code-switching.
- •
Finally, we conduct systematic experiments across multiple state-of-the-art models to evaluate the effects of language-specific reasoning performance across topics and inform the benefits of code-switching in language models.

## 2 Related Work

Prior work has examined how language influences model reasoning (Schut et al., 2025; Zhong et al., 2024; Yong et al., 2025), effects of language on model alignment with human preferences (Jin et al., 2025; Durmus et al., 2024), and cross-linguistic generalization (Chang et al., 2022). Chang et al. (2022) investigated how different languages are represented within the XLM-R multilingual model. They found that languages occupy distinct regions in the representational space, though languages with similar distributions can be aligned through mean-shifting. This indicates that semantically equivalent sentences in different languages may not map to the same low-level representations. This insight informs our study by highlighting the need for language-specific knowledge representations when reasoning or answering questions across linguistic boundaries.

Other works have focused specifically on multilingual reasoning. For instance, Schut et al. (2025) demonstrated that language models tend to default to English during internal reasoning, which can negatively impact downstream task performance, fluency, and fairness. We extended this finding by identifying, for some given topic, the language in which a multilingual language model exhibited greater expertise. Similarly, Zhong et al. (2024) found that models often reason internally in a specific language and exhibit cultural biases aligned with that language when responding to culturally grounded questions. In our work, we aim to boost multilingual reasoning by identifying such LSK and strategically leveraging expert languages where such knowledge is most richly encoded through the LSKExtractor, complementing approaches like Huang et al. (2024) that merge external multilingual representations to enhance general understanding, or Ziabari et al. (2025), which adapt LLM reasoning between intuitive (System 1) and deliberative (System 2) modes based on task needs.

Several works have investigated multilingual reasoning from different perspectives: improving reasoning in low-resource languages (Senel et al., 2024), benchmarking the reasoning abilities of language models across languages (Etxaniz et al., 2023; Kumar et al., 2025; Gao et al., 2025), and enhancing semantic alignment between languages (Yoon et al., 2024). These efforts primarily aim to strengthen cross-lingual semantic representations to support more consistent reasoning across languages. In a related line of work, Yong et al. (2025) demonstrated that chain-of-thought traces in various languages can be aligned to their English counterparts to facilitate multilingual reasoning. In contrast, we highlight a fundamental limitation of this alignment approach: certain languages encode concepts that do not have direct equivalents in others. This observation underscores the lack of a universal one-to-one mapping across languages (Liu et al., 2024). Rather than enforcing alignment, our work embraces linguistic diversity by leveraging the unique conceptual affordances of each language to enhance reasoning performance.

Furthermore, language is an important part of model alignment with human preferences. However, prior work has shown that current multilingual models are not well aligned with humans, showing more US and Euro-centric representations rather than multicultural (Durmus et al., 2024; Rystrøm et al., 2025). Even when prompted across different languages, they fail to align with culturally diverse moral preferences (Jin et al., 2025). Our work contributes to alignment by identifying the expert language for specific domains of knowledge and demonstrating how strategically using these languages can elicit responses that better reflect localized, culturally grounded human preferences.

## 3 LSKExtractor Methodology

Figure: Figure 2: Overview of LSKExtractor. Our method consists of two main steps. In Step 1, we embed training queries into a shared semantic space and cluster them based on topical similarity. For each cluster, we determine the expert language—i.e., the language that yields the most accurate or contextually appropriate reasoning—by comparing model responses across languages. In Step 2, during test-time inference, we embed the test query into the same space, identify its nearest cluster, and select the corresponding expert language (e.g., Spanish) to guide the model toward producing a more informed and culturally grounded response.
Refer to caption: x2.png

We formulate LSK by studying the effect of language-specific chain-of-thought (CoT) reasoning on the performance of a language model. First, we identify a set of languages $\mathcal{L}$. For each language $\mathcal{l}$ in $\mathcal{L}$, we prompt the model to do CoT reasoning in $\mathcal{l}$ on the entire dataset. We denote the language model performance ($LLM_{\theta}$) on question $Q$ from dataset $D$ with CoT reasoning using language $\mathcal{l}$ as $LLM_{\theta}(Q|\mathcal{l})$. We also calculate the performance of the language model without reasoning, and denote it simply as $LLM_{\theta}(Q)$. Our model prompts are outlined in Appendix [C](https://arxiv.org/html/2505.14990v1#A3).

We use this formulation to map LSK to an expert language, cluster semantically similar queries, and form a language-topic alignment map. We can, then, take advantage of the language-topic alignment map during testing by identifying the topic cluster and using the corresponding language for reasoning. Figure [2](https://arxiv.org/html/2505.14990v1#S3.F2) contains an overview of our solution. We detail the steps in each stage below.

In the first stage, we establish the LSK present in each model for each language:

- 1.
Use an embedding model to embed each query in the training set, and use the $k$-means algorithm to cluster these embeddings.
- 2.
Perform language-specific-CoT on a variety of languages by providing instructions translated to $\mathcal{l}$ and specifying to use CoT reasoning in $\mathcal{l}$.
- 3.
Evaluate the model responses in each language against ground-truth answers to obtain accuracy scores.
- 4.
Assign each cluster an expert language–i.e., the language that achieves the highest average accuracy within that cluster.

In the second stage, we leverage the LSK representation to guide test-time inference:

- 1.
Embed test queries into the same semantic space used during training.
- 2.
Identify the nearest cluster for each query based on its embedding.
- 3.
Perform CoT reasoning in the expert language associated with the identified cluster, and treat this as the final model output.

## 4 Experiments

### 4.1 Experimental Setup

For our experimentation, we use 13 languages, but posit that our methodology is mostly invariant to the choice of languages. Formally, $\mathcal{L}$ contains: Arabic, Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, and Thai.

#### Datasets.

There are at least three kinds of data that could exhibit the idea of multilingual knowledge: culture, history, and religion. In this study, we choose to study language-specific cultural knowledge of language models. Intuitively, a model should know more about a certain culture in the corresponding language. Hence, we employ the following datasets (20k data points for training, and 20k for testing):

- •
CultureAtlas (Fung et al., 2024): a binary classification dataset where the input is a cultural norm (e.g., "During the Chinese New Year, in Southern China, red envelopes are typically given by the married to the unmarried[…]"). The output is either True or False.
- •
BLEnD (Myung et al., 2025): a multiple-choice question answering dataset where the input is a societal norm (e.g., "What is the common dress code for school teachers in Azerbaijan?") and four answer choices (e.g., "A. apron, B. black formal suit, C. uniform, D. shirt"). The output is one of the selected answer choices.
- •
Social IQa (Sap et al., 2019): a multiple-choice common sense reasoning dataset where the input contains some context (e.g., "Sydney walked past a homeless woman asking for change but did not have any money […] Sydney felt bad".), a question (e.g., "How would you describe Sydney?"), and three answer choices (e.g., "A. sympathetic, B. like a person who was unable to help, C. incredulous"). The output is one of the selected answer choices.

Because all of these datasets are for classification, we measure performance with classification accuracy ((True Positive + True Negative) / All Predictions), and report the results on $LLM_{\theta}(Q|\mathcal{l})-LLM_{\theta}(Q)$ (difference in performance with and without reasoning, as defined in Section [3](https://arxiv.org/html/2505.14990v1#S3)) averaged across the whole dataset. The maximum accuracy is 1.0 (100%) and the minimum is 0.0 (0%), so the performance metric ranges from 1.0 to -1.0.

#### Models.

For our evaluation, we use a variety of model sizes from a variety of families: Google’s gemma-3-1b-it and gemma-3-12b-it (Team et al., 2024), Meta’s Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, and Llama-3.1-8B-Instruct (Grattafiori et al., 2024), Microsoft’s Phi-3-small-8k-instruct and Phi-3-medium-128k-instruct (Abdin et al., 2024), CohereLab’s aya-23-8B (Dang et al., 2024), IBM’s granite-3.1-8b-instruct (Mishra et al., 2024), and OpenAI’s gpt-4o-mini (closed-source) (OpenAI, 2024). We use instruction-tuned versions because those models are trained to handle multilingual inputs.

#### Baselines.

We employ two main baselines: (1) no reasoning at all and (2) only English reasoning. "No Reasoning" is given a cultural/societal norm and asked to output a final answer (True/False for CultureAtlas, A/B/C/D for BLEnD, and Social IQa). "Only English" is given a cultural/societal norm and asked to first reason about the validity of the norm in English, then output a final answer.

### 4.2 Main Results

Figure: Figure 3: Classification accuracy on the CultureAtlas dataset. On average, LSKExtractor improves classification accuracy on the best-performing baseline by 18.20%.
Refer to caption: extracted/6460575/figures/culture_atlas.png

Figure: Figure 4: Classification accuracy on the BLEnD dataset. On average, LSKExtractor improves classification accuracy on the best-performing baseline by 8.13%.
Refer to caption: extracted/6460575/figures/blend.png

Figure: Figure 5: Classification accuracy on the Social IQa dataset. On average, LSKExtractor improves classification accuracy on the best-performing baseline by 4.91%.
Refer to caption: extracted/6460575/figures/socialiqa.png

Figures [3](https://arxiv.org/html/2505.14990v1#S4.F3), [4](https://arxiv.org/html/2505.14990v1#S4.F4), and [5](https://arxiv.org/html/2505.14990v1#S4.F5) contain the results for CultureAtlas, BLEnD, and Social IQa, respectively. Overall, we see that LSKExtractor outperforms the baselines on most of the models, indicating that our methodology is applicable to a large range of models. Generally, we see larger improvements in smaller models (gemma-3-1b, Llama-3.2-1b, and Phi-3-small) than in larger models. There are a few settings with unexpected results, which we explain below:

- 1.
The Aya model does well without reasoning (Figure [3](https://arxiv.org/html/2505.14990v1#S4.F3)) or does not improve with non-English reasoning (Figures [4](https://arxiv.org/html/2505.14990v1#S4.F4) and [5](https://arxiv.org/html/2505.14990v1#S4.F5)). This could be because Aya is specially trained to have strong representational similarity within languages themselves, meaning the LSK would be weaker in this model than in others.
- 2.
Some models have negligible performance difference between LSKExtractor  and the baselines. This could be because those models have stronger cultural knowledge representation in English itself and/or more LSK pre-training data in English.

### 4.3 What languages are chosen in each cluster?

To understand the distribution of languages chosen for each cluster, we visualize it with a stacked bar plot. Figure [6](https://arxiv.org/html/2505.14990v1#S4.F6) contains the results for CultureAtlas (due to space limitations, we put similar figures for BLEnD and Social IQa in Appendix [B](https://arxiv.org/html/2505.14990v1#A2)). We see a variety of languages are chosen for each cluster. The most common languages for CultureAtlas seem to be Vietnamese, Chinese, German, Arabic and French.

Figure: Figure 6: The distribution of expert languages per cluster for the CultureAtlas dataset. Each bar represents a model (name on the x-axis). The y-axis represents the number of clusters with the corresponding expert language. Similar figures for BLEnD and Social IQa are in Appendix [B](https://arxiv.org/html/2505.14990v1#A2).
Refer to caption: extracted/6460575/figures/culture_language_per_cluster.png

## 5 Ablation Experiments: forced multilingual CoT

As mentioned in Section [3](https://arxiv.org/html/2505.14990v1#S3), we formulate LSK with the difference in performance with and without reasoning: $LLM_{\theta}(Q|\mathcal{l})-LLM_{\theta}(Q)$. In this section, we present some ablation studies to see why LSKExtractor works and provide an intuition of LSK.

Due to space limitations, we report the results on CultureAtlas and BLEnD. We focus our results on three models: (1) google/gemma-3-1b-it, (2) microsoft/Phi-3-small-8k-instruct, (3) CohereLabs/aya-23-8B. We choose these models because they are from different model families and differ in number of parameters (1B, 3B, and 8B, respectively).

In this ablation study, we examine the fine-grained effect of clustering on LSK. First, we show the effect of clustering data points on geographic information (i.e., the country that the cultural norm is from) in Section [5.1](https://arxiv.org/html/2505.14990v1#S5.SS1). Next, we show the effect of clustering on semantic similarity in Section [5.2](https://arxiv.org/html/2505.14990v1#S5.SS2).

### 5.1 Clustering with geographic information

A nice feature of the CultureAtlas and BLEnD dataset is that they identify the country of each cultural norm. So, we can visualize the results of each language on each country. Figures [7](https://arxiv.org/html/2505.14990v1#S5.F7) and [8](https://arxiv.org/html/2505.14990v1#S5.F8) contain the results for this experiment for CultureAtlas and BLEnD, respectively. To begin, we see the following results that are intuitive:

- •
Phi-3-small thinks best in French for Nigeria’s cultural norms (Fig. [7](https://arxiv.org/html/2505.14990v1#S5.F7)(b)).
- •
Aya-23-8B thinks best in French for France’s cultural norms (Fig. [7](https://arxiv.org/html/2505.14990v1#S5.F7)(c)).
- •
Gemma-3-1B performs very well across all languages for US societal norms (Fig. [8](https://arxiv.org/html/2505.14990v1#S5.F8)(a)), given that US norms would have the most amount of data.
- •
Gemma-3-1B performs poorly on societal norm classification for Northern Nigeria and Algeria (Fig. [8](https://arxiv.org/html/2505.14990v1#S5.F8)(a)), which was probably underrepresented in the pre-training.
- •
Phi-3-small thinks best in Arabic for Iran’s societal norms (Fig. [8](https://arxiv.org/html/2505.14990v1#S5.F8)(b)) given that Arabic and Farsi (Iran’s primary language) share a written script.
- •
Aya-23-8B thinks best in English and French for US and UK societal norms (Fig. [8](https://arxiv.org/html/2505.14990v1#S5.F8)(c)).

Figure: Figure 7: LSK in (a) Gemma-3-1B, (b) Phi-3-small, and (c) Aya-23-8B, respectively, on the CultureAtlas dataset, broken down by countries. The x-axis contains the languages in which CoT reasoning was performed. The y-axis outlines the countries where the cultural norms come from. Each square represents the performance of a language model using CoT reasoning with a specific language on set of cultural norms from a specific country. Note: Darker red colors indicate improvement over the "No Reasoning" baseline. Two cells in the same row are comparable and inform which language is better for reasoning for a given country.
Refer to caption: x3.png

Figure: Figure 8: LSK in (a) Gemma-3-1B, (b) Phi-3-small, and (c) Aya-23-8B, respectively, on the BLEnD dataset, broken down by countries. The x- and y-axis definitions follow those in Figure [7](https://arxiv.org/html/2505.14990v1#S5.F7).
Refer to caption: x4.png

However, these heatmaps indicate many non-intuitive trends as well. The most significant one: English is not the best language in all cases for CoT reasoning. For some countries, English underperforms other languages (see English row in [7](https://arxiv.org/html/2505.14990v1#S5.F7)(b) and [7](https://arxiv.org/html/2505.14990v1#S5.F7)(a)). This is evidence that cultural and societal information is represented differently in different languages, and that languages contain their own specific knowledge. Other non-intuitive trends include:

- •
Gemma-3-1B thinks best in Vietnamese for all countries’ cultural norms (Fig. [7](https://arxiv.org/html/2505.14990v1#S5.F7)(a)) – but only slightly better, given that the highest improvement is just above 0.1.
- •
Phi-3-small thinks best in Chinese for Nigeria’s cultural norms (Fig. [7](https://arxiv.org/html/2505.14990v1#S5.F7)(b))
- •
Phi-3-small thinks worst in Korean for North Korea’s societal norms (Fig. [8](https://arxiv.org/html/2505.14990v1#S5.F8)(b))

These non-intuitive trends suggest that there is no clean language-to-country mapping where a language does best in the corresponding country. Hence, a more nuanced methodology is required to take advantage of the LSK, with a more nuanced way of clustering similar data points together.

### 5.2 Clustering with semantic information

Like we do in the second stage of "LSK-Informed Reasoning" of LSKExtractor, we embed all the input training data (cultural and societal norms) using the intfloat/multilingual-e5-large-instruct embedding model, and use the $k$-means clustering algorithm to cluster data points based on their semantic representation. We use $k=48$. Figure [9](https://arxiv.org/html/2505.14990v1#S5.F9) contains the results for cluster-based accuracy on CultureAtlas. As shown, the maximum difference in accuracy ($LLM(Q|\mathcal{l})-LLM(Q)$) increases when the results are broken down by semantic cluster (at least 10% for Gemma-3-1B, 5% for Phi-3-small, and 20% for Aya-23-8B). From this result, we see that the semantic content of the input data can be used to better inform the appropriate language needed for CoT reasoning. Thus, we develop LSKExtractor to identify and extract the LSK of models.

Figure: Figure 9: LSK in Gemma-3-1B, Phi-3-small, and Aya-23-8B, respectively, on the CultureAtlas dataset, broken down by $k$-means clusters. The x-axis contains the languages in which CoT reasoning was performed.
Refer to caption: x5.png

## 6 Conclusion

In this paper, we explore the concept of Language Specific Knowledge (LSK) – languages contain specific knowledge not present in other languages. We design a methodology, called LSKExtractor, that maps languages to specific topics. We show that LSKExtractor can improve the performance of language models by allowing them to reason in a selected language (dependent on the topic). Our extensive experimentation covers three datasets and a variety of language models (model families, parameter sizes, open and closed source). It shows that LSKExtractor achieves a 10% relative increase in accuracy, averaged across all datasets and models. We also provide useful ablation studies to showcase the intuition behind LSK and why LSKExtractor works. Using the insights of this work, we hope to train models that take advantage of LSK to be more inclusive and culturally aligned.

#### Future Work.

This work explored monolingual reasoning chains in language models. In the future, we will investigate: (1) multilingual reasoning chains to analyze language-switching effects on reasoning quality, (2) more efficient methods to approximate Language-Specific Knowledge without linearly increasing computational costs, and (3) the practical impact of Language-Specific Knowledge on downstream conversational tasks like persuasion and dialogue-state tracking.

## References

- Abdin et al. (2024)
M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl, A. Benhaim, M. Bilenko, J. Bjorck, S. Bubeck, M. Cai, Q. Cai, V. Chaudhary, D. Chen, D. Chen, W. Chen, Y.-C. Chen, Y.-L. Chen, H. Cheng, P. Chopra, X. Dai, M. Dixon, R. Eldan, V. Fragoso, J. Gao, M. Gao, M. Gao, A. Garg, A. D. Giorno, A. Goswami, S. Gunasekar, E. Haider, J. Hao, R. J. Hewett, W. Hu, J. Huynh, D. Iter, S. A. Jacobs, M. Javaheripi, X. Jin, N. Karampatziakis, P. Kauffmann, M. Khademi, D. Kim, Y. J. Kim, L. Kurilenko, J. R. Lee, Y. T. Lee, Y. Li, Y. Li, C. Liang, L. Liden, X. Lin, Z. Lin, C. Liu, L. Liu, M. Liu, W. Liu, X. Liu, C. Luo, P. Madan, A. Mahmoudzadeh, D. Majercak, M. Mazzola, C. C. T. Mendes, A. Mitra, H. Modi, A. Nguyen, B. Norick, B. Patra, D. Perez-Becker, T. Portet, R. Pryzant, H. Qin, M. Radmilac, L. Ren, G. de Rosa, C. Rosset, S. Roy, O. Ruwase, O. Saarikivi, A. Saied, A. Salim, M. Santacroce, S. Shah, N. Shang, H. Sharma, Y. Shen, S. Shukla, X. Song, M. Tanaka,
A. Tupini, P. Vaddamanu, C. Wang, G. Wang, L. Wang, S. Wang, X. Wang, Y. Wang, R. Ward, W. Wen, P. Witte, H. Wu, X. Wu, M. Wyatt, B. Xiao, C. Xu, J. Xu, W. Xu, J. Xue, S. Yadav, F. Yang, J. Yang, Y. Yang, Z. Yang, D. Yu, L. Yuan, C. Zhang, C. Zhang, J. Zhang, L. L. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, and X. Zhou.
Phi-3 technical report: A highly capable language model locally on your phone, 2024.
URL [https://arxiv.org/abs/2404.14219](https://arxiv.org/abs/2404.14219).
- Chang et al. (2022)
T. A. Chang, Z. Tu, and B. K. Bergen.
The geometry of multilingual language model representations, 2022.
URL [https://arxiv.org/abs/2205.10964](https://arxiv.org/abs/2205.10964).
- Dang et al. (2024)
J. Dang, S. Singh, D. D’souza, A. Ahmadian, A. Salamanca, M. Smith, A. Peppin, S. Hong, M. Govindassamy, T. Zhao, S. Kublik, M. Amer, V. Aryabumi, J. A. Campos, Y.-C. Tan, T. Kocmi, F. Strub, N. Grinsztajn, Y. Flet-Berliac, A. Locatelli, H. Lin, D. Talupuru, B. Venkitesh, D. Cairuz, B. Yang, T. Chung, W.-Y. Ko, S. S. Shi, A. Shukayev, S. Bae, A. Piktus, R. Castagné, F. Cruz-Salinas, E. Kim, L. Crawhall-Stein, A. Morisot, S. Roy, P. Blunsom, I. Zhang, A. Gomez, N. Frosst, M. Fadaee, B. Ermis, A. Üstün, and S. Hooker.
Aya expanse: Combining research breakthroughs for a new multilingual frontier, 2024.
URL [https://arxiv.org/abs/2412.04261](https://arxiv.org/abs/2412.04261).
- DeepSeek-AI et al. (2025)
DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Ding, H. Xin, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Wang, J. Chen, J. Yuan, J. Qiu, J. Li, J. L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang, L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang, R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, S. S. Li, S. Zhou, S. Wu, S. Ye, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng, W. Zhao, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie,
X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang, X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao, Y. Sun, Y. Wang, Y. Yu, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong, Y. Zou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Xu, Y. Huang, Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Y. Tang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. Zhang, Z. Hao, Z. Ma, Z. Yan, Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Pan, Z. Huang, Z. Xu, Z. Zhang, and Z. Zhang.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
URL [https://arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948).
- Durmus et al. (2024)
E. Durmus, K. Nguyen, T. Liao, N. Schiefer, A. Askell, A. Bakhtin, C. Chen, Z. Hatfield-Dodds, D. Hernandez, N. Joseph, L. Lovitt, S. McCandlish, O. Sikder, A. Tamkin, J. Thamkul, J. Kaplan, J. Clark, and D. Ganguli.
Towards measuring the representation of subjective global opinions in language models.
In *First Conference on Language Modeling*, 2024.
URL [https://openreview.net/forum?id=zl16jLb91v](https://openreview.net/forum?id=zl16jLb91v).
- Etxaniz et al. (2023)
J. Etxaniz, G. Azkune, A. Soroa, O. L. de Lacalle, and M. Artetxe.
Do multilingual language models think better in english?, 2023.
URL [https://arxiv.org/abs/2308.01223](https://arxiv.org/abs/2308.01223).
- Fung et al. (2024)
Y. Fung, R. Zhao, J. Doo, C. Sun, and H. Ji.
Massively multi-cultural knowledge acquisition & lm benchmarking, 2024.
URL [https://arxiv.org/abs/2402.09369](https://arxiv.org/abs/2402.09369).
- Gao et al. (2025)
C. Gao, X. Huang, W. Zhu, S. Huang, L. Li, and F. Yuan.
Could thinking multilingually empower llm reasoning?, 2025.
URL [https://arxiv.org/abs/2504.11833](https://arxiv.org/abs/2504.11833).
- Grattafiori et al. (2024)
A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Guzmán, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Zhang, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang,
J. Liu, J. Wang, J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala-Yeary, L. van der Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q. He, Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girdhar, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa, S. Singh, S. Bell, S. S. Kim, S. Edunov,
S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer,
B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang, C. Kim, C. Zhou, C. Hu, C.-H. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E.-T. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, Guangyi, Zhang, G. Lakshminarayanan, H. Inan, H. Shojanazeri, H. Zou, H. Wang, H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, I. Damlaj, I. Molybog, I. Tufanov, I. Leontiadis, I.-E. Veliche, I. Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J.-B. Gaya, J. Marcus,
J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang, J. Cummings, J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg, J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg, L. A, L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez,
R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto, S. Ramaswamy, S. Lindsay, S. Lindsay, S. Feng, S. Lin, S. C. Zha, S. Patil, S. Shankar, S. Zhang, S. Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Deng, S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Yu, Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. DeVito,
Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, and Z. Ma.
The llama 3 herd of models, 2024.
URL [https://arxiv.org/abs/2407.21783](https://arxiv.org/abs/2407.21783).
- Huang et al. (2024)
Z. Huang, W. Zhu, G. Cheng, L. Li, and F. Yuan.
Mindmerger: Efficiently boosting LLM reasoning in non-english languages.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.
URL [https://openreview.net/forum?id=Oq32ylAOu2](https://openreview.net/forum?id=Oq32ylAOu2).
- Jin et al. (2025)
Z. Jin, M. Kleiman-Weiner, G. Piatti, S. Levine, J. Liu, F. G. Adauto, F. Ortu, A. Strausz, M. Sachan, R. Mihalcea, Y. Choi, and B. Schölkopf.
Language model alignment in multilingual trolley problems.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=VEqPDZIDAh](https://openreview.net/forum?id=VEqPDZIDAh).
- Joshi (1982)
A. K. Joshi.
Processing of sentences with intra-sentential code-switching.
In *Coling 1982: Proceedings of the Ninth International Conference on Computational Linguistics*, 1982.
URL [https://aclanthology.org/C82-1023/](https://aclanthology.org/C82-1023/).
- Kumar et al. (2025)
S. Kumar, V. Balloli, M. Ranjit, K. Ahuja, S. Sitaram, K. Bali, T. Ganu, and A. Nambi.
Bridging the language gap: Dynamic learning strategies for improving multilingual performance in llms.
In O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, and S. Schockaert, editors, *Proceedings of the 31st International Conference on Computational Linguistics*, page 9209–9223, Abu Dhabi, UAE, Jan. 2025. Association for Computational Linguistics.
URL [https://aclanthology.org/2025.coling-main.619/](https://aclanthology.org/2025.coling-main.619/).
- Liu et al. (2024)
C. C. Liu, F. Koto, T. Baldwin, and I. Gurevych.
Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings, 2024.
URL [https://arxiv.org/abs/2309.08591](https://arxiv.org/abs/2309.08591).
- Marjanović et al. (2025)
S. V. Marjanović, A. Patel, V. Adlakha, M. Aghajohari, P. BehnamGhader, M. Bhatia, A. Khandelwal, A. Kraft, B. Krojer, X. H. Lù, N. Meade, D. Shin, A. Kazemnejad, G. Kamath, M. Mosbach, K. Stańczak, and S. Reddy.
Deepseek-r1 thoughtology: Let’s think about llm reasoning, 2025.
URL [https://arxiv.org/abs/2504.07128](https://arxiv.org/abs/2504.07128).
- Mishra et al. (2024)
M. Mishra, M. Stallone, G. Zhang, Y. Shen, A. Prasad, A. M. Soria, M. Merler, P. Selvam, S. Surendran, S. Singh, M. Sethi, X.-H. Dang, P. Li, K.-L. Wu, S. Zawad, A. Coleman, M. White, M. Lewis, R. Pavuluri, Y. Koyfman, B. Lublinsky, M. de Bayser, I. Abdelaziz, K. Basu, M. Agarwal, Y. Zhou, C. Johnson, A. Goyal, H. Patel, Y. Shah, P. Zerfos, H. Ludwig, A. Munawar, M. Crouse, P. Kapanipathi, S. Salaria, B. Calio, S. Wen, S. Seelam, B. Belgodere, C. Fonseca, A. Singhee, N. Desai, D. D. Cox, R. Puri, and R. Panda.
Granite code models: A family of open foundation models for code intelligence, 2024.
URL [https://arxiv.org/abs/2405.04324](https://arxiv.org/abs/2405.04324).
- Myung et al. (2025)
J. Myung, N. Lee, Y. Zhou, J. Jin, R. A. Putri, D. Antypas, H. Borkakoty, E. Kim, C. Perez-Almendros, A. A. Ayele, V. Gutiérrez-Basulto, Y. Ibáñez-García, H. Lee, S. H. Muhammad, K. Park, A. S. Rzayev, N. White, S. M. Yimam, M. T. Pilehvar, N. Ousidhoum, J. Camacho-Collados, and A. Oh.
Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages, 2025.
URL [https://arxiv.org/abs/2406.09948](https://arxiv.org/abs/2406.09948).
- OpenAI (2024)
OpenAI.
Gpt-4o system card.
OpenAI Technical System Card, 2024.
URL [https://openai.com/index/gpt-4o-system-card/](https://openai.com/index/gpt-4o-system-card/).
- Rystrøm et al. (2025)
J. Rystrøm, H. R. Kirk, and S. Hale.
Multilingual != multicultural: Evaluating gaps between multilingual capabilities and cultural alignment in llms, 2025.
URL [https://arxiv.org/abs/2502.16534](https://arxiv.org/abs/2502.16534).
- Sap et al. (2019)
M. Sap, H. Rashkin, D. Chen, R. LeBras, and Y. Choi.
Socialiqa: Commonsense reasoning about social interactions, 2019.
URL [https://arxiv.org/abs/1904.09728](https://arxiv.org/abs/1904.09728).
- Schut et al. (2025)
L. Schut, Y. Gal, and S. Farquhar.
Do multilingual llms think in english?, 2025.
URL [https://arxiv.org/abs/2502.15603](https://arxiv.org/abs/2502.15603).
- Senel et al. (2024)
L. K. Senel, B. Ebing, K. Baghirova, H. Schuetze, and G. Glavaš.
Kardeş-nlu: Transfer to low-resource languages with the help of a high-resource cousin–a benchmark and evaluation for turkic languages.
In *Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 1672–1688, 2024.
- Team et al. (2024)
G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, P. Tafti, L. Hussenot, P. G. Sessa, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. Héliou, A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari, C. L. Lan, C. A. Choquette-Choo, C. Crepy, D. Cer, D. Ippolito, D. Reid, E. Buchatskaya, E. Ni, E. Noland, G. Yan, G. Tucker, G.-C. Muraru, G. Rozhdestvenskiy, H. Michalewski, I. Tenney, I. Grishchenko, J. Austin, J. Keeling, J. Labanowski, J.-B. Lespiau, J. Stanway, J. Brennan, J. Chen, J. Ferret, J. Chiu, J. Mao-Jones, K. Lee, K. Yu, K. Millican, L. L. Sjoesund, L. Lee, L. Dixon, M. Reid, M. Mikuła, M. Wirth, M. Sharman, N. Chinaev, N. Thain, O. Bachem, O. Chang, O. Wahltinez, P. Bailey, P. Michel, P. Yotov, R. Chaabouni, R. Comanescu, R. Jana, R. Anil, R. McIlroy, R. Liu, R. Mullins, S. L. Smith, S. Borgeaud, S. Girgin, S. Douglas, S. Pandya, S. Shakeri, S. De, T. Klimenko, T. Hennigan, V. Feinberg, W. Stokowiec,
Y. hui Chen, Z. Ahmed, Z. Gong, T. Warkentin, L. Peran, M. Giang, C. Farabet, O. Vinyals, J. Dean, K. Kavukcuoglu, D. Hassabis, Z. Ghahramani, D. Eck, J. Barral, F. Pereira, E. Collins, A. Joulin, N. Fiedel, E. Senter, A. Andreev, and K. Kenealy.
Gemma: Open models based on gemini research and technology, 2024.
URL [https://arxiv.org/abs/2403.08295](https://arxiv.org/abs/2403.08295).
- Yong et al. (2025)
Z.-X. Yong, M. F. Adilazuarda, J. Mansurov, R. Zhang, N. Muennighoff, C. Eickhoff, G. I. Winata, J. Kreutzer, S. H. Bach, and A. F. Aji.
Crosslingual reasoning through test-time scaling, 2025.
URL [https://arxiv.org/abs/2505.05408](https://arxiv.org/abs/2505.05408).
- Yoon et al. (2024)
D. Yoon, J. Jang, S. Kim, S. Kim, S. Shafayat, and M. Seo.
Langbridge: Multilingual reasoning without multilingual supervision.
(arXiv:2401.10695), June 2024.
doi: 10.48550/arXiv.2401.10695.
URL [http://arxiv.org/abs/2401.10695](http://arxiv.org/abs/2401.10695).
arXiv:2401.10695 [cs].
- Zhong et al. (2024)
C. Zhong, F. Cheng, Q. Liu, J. Jiang, Z. Wan, C. Chu, Y. Murawaki, and S. Kurohashi.
Beyond english-centric llms: What language do multilingual language models think in?, 2024.
URL [https://arxiv.org/abs/2408.10811](https://arxiv.org/abs/2408.10811).
- Ziabari et al. (2025)
A. S. Ziabari, N. Ghazizadeh, Z. Sourati, F. Karimi-Malekabadi, P. Piray, and M. Dehghani.
Reasoning on a spectrum: Aligning llms to system 1 and system 2 thinking, 2025.
URL [https://arxiv.org/abs/2502.12470](https://arxiv.org/abs/2502.12470).

## Appendix A Experimental Specifications

For the gpt-4o-mini model, we use the 2025-03-01-preview version. We run our inference on NVIDIA A40 GPUs. For the the 1B, 3B, 8B models, we used a single A40 GPU, while the 12B and 14B required two A40 GPUs. Inference takes around 30-60 minutes per language. Clustering is computationally inexpensive and can be done on a single A40 GPU.

## Appendix B Remaining Language-Cluster Figures

Figure: Figure 10: The distribution of expert languages per cluster for the BLEnD dataset.
Refer to caption: extracted/6460575/figures/blend_language_per_cluster.png

Figure: Figure 11: The distribution of expert languages per cluster for the Social IQa dataset.
Refer to caption: extracted/6460575/figures/social_language_per_cluster.png

## Appendix C Model Prompts

Figures [12](https://arxiv.org/html/2505.14990v1#A3.F12)-[17](https://arxiv.org/html/2505.14990v1#A3.F17) contain the prompts to the language for each kind of dataset (binary vs multi-class classification), with and without reasoning, and for two languages (English and French, to save space).

Figure: Figure 12: Prompt to the language model for the CultureAtlas dataset (a binary classification task) without reasoning.

Figure: Figure 13: Prompt to the language model for the BLEnD and SocialIQa datasets (multi-class classification tasks) without reasoning.

Figure: Figure 14: Prompt to the language model for the CultureAtlas dataset (a binary classification task) in English.

Figure: Figure 15: Prompt to the language model for the CultureAtlas dataset (a binary classification task) in French.

Figure: Figure 16: Prompt to the language model for the BLEnd and SocialIQa datasets (multi-class classification tasks) in English.

Figure: Figure 17: Prompt to the language model for the BLEnd and SocialIQa datasets (multi-class classification tasks) in French.

## Appendix D Licenses

Our [code](https://anonymous.4open.science/r/LSKExtractor-272F/) is released publicly under the Apache-2.0 License. CultureAtlas [Fung et al., 2024] is released under the MIT License; BLEnD [Myung et al., 2025] under the CC-by-SA-4.0 License; SocialIQa [Sap et al., 2019] is not under explicit license, however it is publicly available on Huggingface, and we do not use it for commercial purposes. All models are under their proprietary licenses from the corresponding companies.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Which embedding model was used? The paper does not specify which embedding model was employed.
- Chinese is likely selected because it constitutes for a large portion of the training data, but why was Portuguese chosen as an expert language?
soundness: 3
strengths: - A novel approach to enhance model performance by leveraging the language that yields best results rather than suppressing the use of other languages.
- The method can improve the performance in a straightforward way, without any additional training.
summary: This work analyses phenomenon in which multilingual large language models process questions with identical meanings differently across languages, proposing the concept of Language-Specific Knowledge (LSK). The core assumption of LSK is that within large language model some languages contain more information about certain knowledge regions than others. To do this authors propose a method called LSKExtractor. During training, the model builds semantic embedding clusters, and during inference it uses these clusters to generate answers in the language that demonstrates best performance. The authors shows through experiments on 9 models and 3 datasets that language choice can enhance the model performance.
weaknesses: - The choice of expert language varies significantly depending on which dataset is used training the LSK map. In Figure 5, CultureAtlas results in diverse language selection, while BLEnD and Social IQa predominantly select English.
- The paper does not address how to evaluate open-ended prompts that cannot be evaluated using accuracy. In real-world scenarios prompts are far more diverse in form and intent.
- The paper focuses only on culturally grounded datasets without testing whether LSK also appears in other domains such as mathematics or coding. Adding experiments in these domains would help verify generality of the approach.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: - L150: What was the reason for translation? As mentioned in the weaknesses, datasets are already language specific, therefore, it is not clear why translation was needed?
- L184: What is the reason to embed the English version only?
- - What is the reason for majority voting?
soundness: 2
strengths: The paper formalizes "Language Specific Knowledge (LSK)"" that some queries are better answered when the model reasons in a particular "expert language" and turns it into a concrete language-selection problem. The proposed two-stage LSKEXTRACTOR (map LSK via clustering and pick expert language at inference) is simple and well-motivated.
summary: This study propose a method to QA task - measuring the ability of the LLMs by changing the language of the input query. 
It introduces the teram "Language Specific Knowledge (LSK)" to denote question that are best answered in an *expert* language for an LLM.
weaknesses: - While two step-process is well-motivated, however, it is not very much clear why clustering was needed as in these datasets QAs are language specific already. Is it that find to create/find a cluster that represent an expert for a language? 
- The CultureAtlas binary -> MCQ reformulation is sensible how distractors/other options are generated is not clear.  
- Queries/questions are translated, however, not manual verification (even on small samples) is provided. 

Typos/Grammatical/Minor issues
- The math notation for CoT reasoning is not clear, it is conditioned with l. Any other notation can be used to define CoT. 
- L187: I assume queries also belongs to l.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 4
questions: same as the weaknesses. I'm willing to update the score based on answers / clarifications to the weaknesses.
soundness: 2
strengths: # Strengths

1. The paper is clearly written and easy to follow. Props for providing the code.
2. Clear experiments and ablations - the baselines seem well designed and performed.
summary: # High-level summary

The authors tackle the problem of LLMs having specific knowledge / better capabilities in specific languages, which they term as Language Specific Knowledge (LSK). LSK is derived by claiming that LLMs perform better on question-answering tasks (QA) in specific languages (like English) as opposed to other languages. 

# Contributions 

Their proposed contributions surrounding LSK has two parts - given a set of candidate languages: 

## Part 1 (Train-time): 

i) the query $Q^{train}$ is embedded and clustered using kmeans; ii) for each query in each cluster and each language $l \in L$, accuracy of the LLM is calculated and mean accuracy is calculated for the cluster per language; iii) Best language per cluster is determined. 

## Part 2(Test-time):

Each test query is assigned the nearest cluster, and the language associated with that cluster from train time is used to get the final answer. 

# Results

The authors experiment on three datasets: CultureAtlas, BLEnD, and Social IQa, all being/reformatted to match Multiple QA style benchmark setting and report classification accuracy.

1. There is a clear result in the claim that there is/are languages that perform better from Figure 3 and other previous works cited in the paper.
2. From Figure 3, it seems like Qwen3's family of models benefits the most from the proposed methods, other models less so (hypothesis: RL-post-trained family of models differing from the others)
3. All models are poorly self-calibrated in general to know which language they know best.
4. Distribution of the selected languages seems uniform in two datasets, but the third dataset (CultureAtlas) has wildly different distributions (which is an interesting result).
weaknesses: # Weaknesses

1. The approach doesn't seem too different from [1] (cited in the paper). In [1], the authors train to classify whether a given task config and other runtime configs would work well by creating an embedding and show similar robustness to OOD tasks. Similarly, in this work, the authors first assign the query to some cluster whose best language is already known. Arguably, adding a new language here requires recomputing accuracies, as opposed to one that can be learnt online. Another visible difference is that the former method in [1] supports a lot of languages at query time, which this method would seem to require another mapping iteration. Lack of discussion comparing clear improvements, differences, and advantages/disadvantages seems critical. 
2. No hyperparameters, confidence intervals/error bars, or sensitivities of the result mentioned anywhere. For example, the robustness result can also be inferred to mean that multiple language distributions are very closely related, where any choice would have yielded similar results, which would indicate a one-to-many mapping for the right set of configurations. Similarly, the impact of temperature seems critical which is not mentioned anywhere. Lack of these details and analysis surrounding them makes the observations questionable.
3. Control for thinking - in models like Qwen3, the _thinking_ mostly happens in English / Chinese; it is unclear how the authors control for thinking language/behavior and what the effect of language is in these models when thinking is enabled / disabled. 
4. performance cost - another missing metric is the training cost to perceived benefit. Assuming each input and output token has a constant cost, from Figure 3, it appears that calibrating the LLMs to choose the best language would yield the best cost-to-performance ratio. There is a paragraph on Pg. 9 that very briefly discusses this, but it seems like this warrants its own specific subsection in the main content/appendix. 


[1] S. Kumar, V. Balloli, M. Ranjit, K. Ahuja, S. Sitaram, K. Bali, T. Ganu, and A. Nambi. Bridging the language gap: Dynamic learning strategies for improving multilingual performance in llms. In O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, and S. Schockaert, editors, Proceedings of the 31st International Conference on Computational Linguistics, page 9209–9223, Abu Dhabi, UAE, Jan. 2025. Association for Computational Linguistics. URLhttps://aclanthology.org/2025.coling-main.619/.


Decision: Reject