Venue: NAACL 2025

# Does Mapo Tofu Contain Coffee?

Probing LLMs for Food-related Cultural Knowledge

Li Zhou\({}^{1,2}\), Taelin Karidi\({}^{3}\), Wanlong Liu\({}^{4}\), Nicolas Garneau\({}^{5}\), Yong Cao\({}^{6}\), Wenyu Chen\({}^{4}\), Haizhou Li\({}^{1,2}\), Daniel Hershcovich\({}^{5}\)

###### Abstract

Recent studies have highlighted the presence of cultural biases in Large Language Models (LLMs), yet often lack a robust methodology to dissect these phenomena comprehensively. Our work aims to bridge this gap by delving into the Food domain--a universally relevant yet culturally diverse aspect of human life. We introduce FmLAMA, a multilingual dataset centered on food-related cultural facts and variations in food practices. We analyze LLMs across various architectures and configurations, evaluating their performance in both monolingual and multilingual settings. By leveraging templates in six different languages, we investigate how LLMs interact with language-specific and cultural knowledge. Our findings reveal that (1) LLMs demonstrate a pronounced bias towards food knowledge prevalent in the United States; (2) Incorporating relevant cultural context significantly improves LLMs' ability to access cultural knowledge; (3) The efficacy of LLMs in capturing cultural nuances is highly dependent on the interplay between the probing language, the specific model architecture, and the cultural context in question. This research underscores the complexity of integrating cultural understanding into LLMs and emphasizes the importance of culturally diverse datasets to mitigate biases and enhance model performance across different cultural domains.

+
Footnote †: _(Volume 1: Long Papers)_, pages 9840–9867

**[1]The Chinese University of Hong Kong, Shenzhen \({}^{2}\)Shenzhen Research Institute of Big Data**

\({}^{3}\)Hebrew University of Jerusalem \({}^{4}\)University of Electronic Science and Technology of China,

\({}^{5}\)University of Copenhagen, \({}^{6}\)University of Tubingen, Tubingen AI Center

{lizhou21, haizhoulij@cubk.edu.cn, taelin.karidi@mail.huji.ac.il, liuwanlong@std.uestc.edu.cn,

(nicolas.garneau, dh)@di.ku.dk, yongcaao2018@gmail.com, cww@uestc.edu.cn

## 1 Introduction

Asking a French person for the recipe of _Beef Bourguignon_ in English might yield an immediate and precise response, while the same query might pose challenges to a Chinese individual unless posed as " (its literal translation). In China, the dish is also commonly referred to by its broader description, " (_Red Wine Stowed Beef_), highlighting the main ingredients and cooking technique, albeit without specifying a regional origin. Employing " (_French-style Red Wine Stowed Beef_) with an adjectival description can indicate adherence to French culinary traditions. This practice illustrates how cultural and linguistic nuances shape knowledge transmission. In cross-cultural communication, when direct translations are unavailable, speakers often switch between languages--a phenomenon known as code-switching--to better convey meaning Aguilar and Solorio (2020); Dogruoz et al. (2021). This variability underscores the challenges language models encounter in navigating cross-cultural culinary contexts and highlights the difficulty of responding with cultural knowledge, which involves using specific language, engaging in code-switching, and integrating cultural context.

Trained on vast datasets, Large Language Models (LLMs) encode a wide array of knowledge, but also face challenges with various biases, such as those related to gender Savoldi et al. (2021); Kaneko et al. (2022), belief Sogaard (2021); Gonzalez et al. (2021); Lent and Sogaard (2021), and culture Cao et al. (2023); Deshpande et al. (2022);

Figure 1: Summary of the various aspects of our work.



Yin et al., 2022; Mukherjee et al., 2024; Singh et al., 2024). In this paper, we adopt food-domain knowledge data as cultural semantic proxy, "country of origin" as cultural demographic proxy (Adilazuarda et al., 2024), and address the following research questions: (1) What cultural biases exist within the cultural knowledge contained in LLMs? (2) How can we best prompt and evaluate LLMs to elicit culture-specific knowledge? Specifically, we first design an automated construction method for the dataset FMLAMA (SS3). Next, we introduce the cultural knowledge probing method (SS4), encompassing the probing task, template and metric. We then conduct experiments (SS5) and analysis (SS6) to address our proposed research questions. Finally, we perform an error analysis on the probing results (SS7). Figure 1 provides an overview of our work's various aspects, and our **contributions** are:

* We present FmLAMA, a pioneering dataset focused on the food domain, which is inherently rich in cultural diversity. This dataset is a multifaceted tool for probing LLMs across cultures and languages.
* We propose novel metrics aimed at evaluating LLMs' capacity to accurately and sensitively probe cultural knowledge, utilizing a combination of automatic and manual evaluation methods.
* We analyze the impact of integrating cultural context and language specificity in prompts, offering insights to optimize LLMs for equitable cross-cultural knowledge retrieval.

Our methodology for automated collection of cultural knowledge corpora extends the analysis potential in other domains, broadening the scope of research on cultural biases in LLMs. 1

Footnote 1: Data and code are available in [https://github.com/lizhou21/FmLAMA-master](https://github.com/lizhou21/FmLAMA-master).

## 2 Related Work

Cultural knowledge datasets.Cultural knowledge, encompassing the customs, beliefs, traditions, and practices of a culture, is crucial yet challenging to encapsulate. While some researchers focus on manually curating cultural knowledge datasets, others evaluate LLMs' performance on culturally related tasks. Yin et al. (2022) and Palta and Rudinger (2023) have developed benchmarks such as geo-diverse prompts and food-custom datasets (FORK) to probe cultural biases in commonsense reasoning systems. However, manual dataset construction is inefficient and hard to scale, prompting a shift towards automated methods. For instance, StereoKG Deshpande et al. (2022) offers a scalable knowledge graph that blends cultural knowledge with stereotypes, and Candle Nguyen et al. (2023) extracts cultural commonsense knowledge from the web, organizing it into clusters. Despite these advances, the variability in data representation--from sentences to triplets using OpenIE--poses challenges for consistency and noise control in knowledge probing. Keleg and Magdy (2023) aims to mitigate this by selecting culturally diverse factual triples from Wikidata, focusing mainly on explicit country information. In contrast, our work proposes an automated, efficient approach to constructing a cultural knowledge dataset in a uniform triplet format, addressing the limitations of existing methods and focusing on implicit cultural knowledge. Appendix A.1 shows a comparison of our dataset with existing ones.

Knowledge probing.Deciphering the knowledge encoded by LLMs poses significant challenges due to their opaque nature , early benchmarks like LAMA Petroni et al. (2019) sought to quantify the factual knowledge in English LLMs, while ParaRel Elazar et al. (2021) highlighted their consistency issues. Subsequent efforts as mLAMA Kassner et al. (2021) and mParaRel Fierro and Sogaard (2022) expanded these benchmarks multilingually, though such methods often focus on single-word entities, limiting their depth of assessment. To address these shortcomings, newer studies Shin et al. (2020); Zhong et al. (2021); Meng et al. (2022) have evolved towards eliciting more comprehensive factual knowledge, including multi-word entities, with Jiang et al. (2020) developing algorithms for multi-token predictions. LPAQA Jiang et al. (2020) further refines this by optimizing prompt discovery for more accurate knowledge probing. Our work builds on this foundation, targeting multi-token probing within the food domain, characterized by complex expressions like _Trigonella foenum-graecum_.

## 3 Fmlamma Construction

To assess whether LLMs encode and access cultural information, we develop FmLAMA, a mul 

ticultural, multilingual dataset focusing on culinary knowledge. The designed framework can be adapted to other cultural domains.

Step #1: Obtain countries set.Following Zhou et al. (2023), we use countries of food origin to delineate cultural groups. This method leverages countries as proxies for cultural identity, encapsulating diverse traditions, values, and norms that reflect the breadth of human civilizations across geographical boundaries Minkov and Hofstede (2012); Peterson et al. (2018).

Step #2: Acquire food instances.We utilize SPARQL to query Wikidata, extracting a vast array of food-related data. This approach exploits Wikidata's RDF triple structure to gather detailed information on food instances, offering a rich source of comprehensive food knowledge.

i. Class.For our food-focused dataset, we concentrate on the dish class and employ two approaches to find food instances:

* Explicit instance of dish, e.g., _bouillabaisse_.
* Inferred through a hierarchy, e.g., _Blanquette de veau_\(\xrightarrow{\text{subclass of}}\xrightarrow{\text{stew}}\xrightarrow{\text{ subclass of}}\xrightarrow{\text{dish}}\).

This enables comprehensive inclusion of food instances, represented as \(I~{}^{(\text{instance of|subclass of})+}\)\(dsh\), where '\(|\cdot\)' denotes "or", and '\(+\)' is "one or more".

ii. Cultural group.We organize food instances by their origin, applying these strategies:

* Directly specified in Wikidata, e.g., _bouillabaisse_\(\xrightarrow{\text{country of origin}}\) France.
* Through the associated cuisine category, e.g., _mapo douft_\(\xrightarrow{\text{cusing}}\) Chinese cuisine \(\xrightarrow{\text{country}}\) China.

We exclude dishes with multiple origin countries to maintain cultural specificity.

iii. Properties included.The property "has part(s)" identifies food ingredients for a dish. Additional properties like "made from material" and "image" are collected to support future research (e.g., multimodal), though they are not utilized in this study. In each instance, the descriptive language for all attributes remains consistent.

Ultimately, our constructed dataset, FMLAMA, comprises 33,600 dish instances, detailed by name, origin, ingredients, and optionally, materials and images, encompassing 128 cultural country groups and 250 languages. The average number of ingredients in FmLAMA dataset is 2.04. Examples and statistics are provided in Appendix A.

## 4 Cultural Knowledge Probing

### Probing task

We adopt Word Prediction (WP) as the knowledge-probing task, following previous work Fierro and Sogaard (2022); Wu et al. (2023); Fierro et al. (2024). Firstly, we manually design a prompt template \(t\) focused on the core attribute "has part(s)", illustrating a connection between a dish (subject) and its ingredient(s) (object). WP is usually implemented as a candidate retrieval problem. The candidate set consists of all objects in the focused, filtered sub-dataset of FmLAMA.2 The primary objective is to utilize LLMs to obtain the probability of each candidate \(C\) and subsequently rank the predicted objects based on these probabilities.

Footnote 2: As the size of the filtered sub-dataset increases, the candidate object set also expands, leading to greater difficulty in probing. Consequently, in this paper, results obtained by probing across different filtered sub-datasets cannot be used for horizontal comparison.

MASK operation.Using subject-object tuples ([X], [Y]) as queries, we probe LLMs by replacing the subject and masking the object. Considering that each candidate object is tokenized into \(d\) subtokens \(\left\{c_{1},\cdots,c_{d}\right\}\) by LLMs correspondingly, we apply [MASK] token of varying lengths to the objects within each query. \(D\) queries are constructed for each dish subject \(X\) based on the same template \(t\), \(D\) is the maximum number of object tokens, each query \(Q_{d}^{X}\) is defined as:

\[Q_{d}^{X}=t\left(X,\mathrm{[MASK]}*d\right),d\in\left[1,D\right]. \tag{1}\]

Probability acquisition.We use Mean Pooling3 method to obtain the prediction probability of each candidate. Specifically, for candidate object \(C=\left\{c_{1},\cdots,c_{d}\right\}\) of length \(d\), its probability is obtained from the likelihoods associated with the [MASK] tokens in \(Q_{d}^{X}\), and the predicted probability of \(C\) is calculated as the average of the probabilities of composing its subtokens:

Footnote 3: Mean Pooling has been shown to be superior to other pooling methods for multi-token probing Wu et al. (2023).

\[P\left(Q_{d}^{X},C\right)=\frac{1}{d}\sum_{i=1}^{d}p\left(Q_{d}^{X},\mathrm{[MASK]}_{ i}=c_{i}\right), \tag{2}\]

where \(p\left(\cdot\right)\) is obtained after the softmax operation.

 



### Probing template

Considering LLMs produce varied predictions based on prompt framing Elazar et al. (2021); Wang et al. (2023), we craft _five_ templates conveying identical meanings in each prompt language. For example, "[X] is a dish made with [Y]", "[X] is a type of food that includes [Y]".

Cultural information.To explore the impact of introducing explicit cultural context on LLMs' ability to access cultural knowledge, we enhance the basic templates by integrating country information. For instance, "In [C], [X] is a dish made with [Y]", where [C] denotes the country of origin for the dish [X], [Y] indicates the ingredient object.

Multilingual prompts.To explore how different prompt language settings affect LLMs' cultural knowledge probing abilities, we craft prompts in six languages written by native speakers, including English (\(en\)), Chinese (\(zh\)), Arabic (\(ar\)), Korean (\(ko\)), Russian (\(ru\)), and Hebrew (\(he\)). These languages span 4 different language families - Indo-European (English, Russian), Semitic (Hebrew, Arabic), Altaic (Korean), and Sino-Tibetian (Chinese), and are spoken by more than 2.356 billion speakers. Furthermore, these languages represent cultural diversity, being spoken on different continents by groups with rich and distinct cultural backgrounds. These prompt illustrations and language details are depicted in Appendix B.

Code-switching.To simulate real-world scenarios where mixed-language expression often occurs, we implement code-switching prompts, varying the main language (ML) and subject language (SL). Specifically, we define the code-switching prompt setting as \({\rm P}({\rm ML},{\rm SL})\), where \({\rm ML},{\rm SL}\in\{en,zh,ar,ko,ru,he\}\), and \({\rm ML}\neq{\rm SL}\). Our objective is to ensure that the language of the predicted object remains consistent with the main language (ML). For instance, the prompt "" "

contribute to the evaluation score, where \(l=|{\rm ing}_{i}|\). \(S\left(i,g\right)\) is defined as follows:

\[S@l\left(i,g\right)=\max_{p\in{\rm top}_{i}}\left[\cos\left(w_{g},w_{p}\right) \right],g\in{\rm ing}_{i}, \tag{6}\]

where \(w_{g}\) and \(w_{p}\) are the FasttextBojanowski et al. (2017); Joulin et al. (2016) vectors for the ingredient \(g\) and the predicted object \(p\), respectively, and \(\cos\left(\cdot\right)\) is the cosine similarity function. Then we can compute the probing similarity \(WS_{i}\) for each food instance \(i\) and mWS for the targeted food group as follows:

\[{\rm WS}_{i} = \frac{1}{|{\rm ing}_{i}|}\sum_{g\in{\rm ing}_{i}}S@l\left(i,g\right) \tag{7}\] \[{\rm mWS} = \frac{1}{|G|}\sum_{i\in G}{\rm WS}_{i} \tag{8}\]

Manual Evaluation Score.MES considers only the top-\(l\) predicted ingredients for evaluation in each dish, similar to the mWS metric, where \(l\) is the number of reference ingredients. A predicted ingredient is considered correct if it satisfies any of the following criteria: (1) _Direct Match_: The ingredient is exactly in the Wikidata reference; (2) _Substitutability_: The ingredient can replace one in the reference; (3) _Missing Traditional Ingredient_: The ingredient is traditionally or commonly used in the dish, but not listed in the reference. Given a manual evaluator \({\rm M}\left(\cdot\right)\), the MES is defined as follows:

\[{\rm S}_{i} = \frac{1}{l}\sum_{p\in{\rm top}_{i}}{\rm M}\left(p,{\rm ing}_{i}\right) \tag{9}\] \[{\rm MES} = \frac{1}{|G|}\sum_{i\in{\rm G}}{\rm S}_{i} \tag{10}\]

where \({\rm M}\left(\cdot\right)\) is 1 if the top-\(l\) predicted ingredient \(p\) is evaluated as meeting the criteria; otherwise, 0.

## 5 Experiments: Existing Cultural Biases

### Experimental setup

BaselinesWe explore encoder-only LLMs, including BERT Devlin et al. (2019) and mBERT, encoder-decoder LLMs, such as T5 Raffel et al. (2020) and mT5 Xue et al. (2020), as well as decoder-only LLMs like Qwen2 Yang et al. (2024), Llama2 Touvron et al. (2023), and Llama3 AI@Meta (2024). Of these, BERT and T5 are English monolingual LLMs, while the remaining five are multilingual LLMs. 4 Additionally, we employ a 'dumb' baseline, in which the model is assumed to consistently predict the top-10 most common ingredients for each dish.

Footnote 4: Monolingual LLM configurations, as well as results for the five languages on their respective filtered sub-datasets, are provided in the Appendix D.1.

Cultural groupsWe identify the top 30 countries with the most dishes across six languages, take their intersection to ensure sufficient data, consider geographical diversity, and ultimately narrow our focus to 14 cultural groups, as shown in Table 1.

MetricsWe evaluate all LLMs using two automated metrics. Due to cost constraints, we conduct manual evaluation only on the LLM that achieve the highest probing performance in the automated metrics. Specifically, for the evaluator \({\rm M}\left(\cdot\right)\) used to compute MES, we utilize GPT-4o OpenAI et al. (2024)5 as a simulated evaluator and recruit real human evaluators from Prolific.6 For each country group, we hire 3 evaluators familiar with the related cuisine, with a pay rate of PS/hour, considered midlevel on the platform. Details on the GPT-4o evaluation prompts and the human evaluation platform are provided in the Appendix C.

Footnote 5: Version: gpt-4o-2024-08-06

Footnote 6: [https://www.prolific.co/](https://www.prolific.co/)

Footnote 7: See detailed analysis in Appendix E.

### Automatic evaluation results

The probing results based on English prompt on the filtered dataset FmLAMA-_en_ are illustrated in Table 1. The average Pearson correlation between mAP and mWS across all LLMs is 0.72, indicating a strong positive relationship and demonstrating that the experimental results are consistent across both automatic metrics. Overall, without fine-tuning, decoder-only LLMs exhibit significantly better cultural knowledge recall in such challenging tasks compared to encoder-only and encoder-decoder LLMs, highlighting their superior capacity as knowledge bases Petroni et al. (2019). Besides, results across all models Avg. Column) indicate that the groups from the U.S. and India, whose official languages include English, perform the best generally. They consistently rank in the top 3 in both automatic evaluation metrics. This pattern is especially pronounced in the results for encoder-only LLMs and encoder-decoder LLMs. Apart from the Iran group7, these two cultural groups also 

demonstrate the best performance in decoder-only LLMs. Especially, for the 'dumb' baseline, the U.S. probing performance is not the best, suggesting that the data distribution does not influence the LLM's cultural judgment capabilities. This further strengthens the persuasiveness of our experiment.

Additionally, (1) to assess the variation in model performance across different cultural groups, we calculate the Coefficient of Variation based on the absolute-match mAP metric. This serves as an indicator of the degree of cultural bias in each model's performance. Our results show that decoder-only LLMs exhibit less cultural bias compared to other model types. Among these, the newer version, Llama3, demonstrates greater consistency in performance than Llama2. Similarly, the non-English-dominant Qwen2 LLM also shows smaller variations in cultural bias. (2) To explore the impact of different embedding methods in mWS, we use BERT Base with mean pooling to evaluate probing performance under the mWS metric. Results

\begin{table}
\begin{tabular}{l c|c c c} \hline \hline  & **Bb** & **Bl** & **mB** & **T5** \\ \hline
**mAP** & 0.60 & 0.49 & 0.48 & 0.56 \\
**mWS** & 0.48 & 0.38 & 0.36 & 0.41 \\ \hline \hline  & **mTS** & **Qwen2** & **Llama2** & **Llama3** \\ \hline
**mAP** & 0.38 & 0.05 & 0.04 & 0.13 \\
**mWS** & -0.06 & 0.05 & -0.04 & 0.02 \\ \hline \hline \end{tabular}
\end{table}
Table 1: **Automatic Evaluation**: Probing performance comparison with English prompts and FmLAMA_-en_ sub-dataset. “B/mB” respectively represent abbreviations for BERT and mBERT, and “bl/" stands for base/large. **Bold** and underline indicate the best and second-best performing cultural groups, respectively, for each model (within each column). The average Pearson correlation between mAP and mWS across LLMs is 0.72.

\begin{table}
\begin{tabular}{l r|r|r|r|r|r|r|r|r|r|r|r|r} \multirow{2}{*}{**Origin**} & \multicolumn{4}{c|}{**Count**} & \multicolumn{1}{c|}{**Dumb Base**} & \multicolumn{4}{c|}{**Encoder-Decoder LLMs**} & \multicolumn{4}{c|}{**Decoder-only LLMs**} & \multicolumn{1}{c|}{**Avg.**} \\ \cline{3-13}  & \multicolumn{1}{c|}{**Point**} & \multicolumn{1}{c|}{**Bub**} & \multicolumn{1}{c|}{**mB**} & \multicolumn{1}{c|}{**F5**} & \multicolumn{1}{c|}{**mFS**} & \multicolumn{1}{c|}{**Oursn2**} & \multicolumn{1}{c|}{**Llama2**} & \multicolumn{1}{c|}{**Llama3**} & \multicolumn{1}{c|}{**Avg.**} \\ \hline
**Italy** & 215 (13.9x) & 18.34 & 8.12\(\pm\)1.14 & 7.52\(\pm\)0.43 & 9.31\(\pm\)1.34 & 4.54\(\pm\)0.56 & 6.73\(\pm\)1.34 & 26.82\(\pm\)2.46 & 19.73\(\pm\)3.68 & 29.62\(\pm\)3.30 & 14.00 \\
**US** & 285 (18.4x) & 11.0 & **19.98\(\pm\)2.00** & **18.83\(\pm\)1.95** & **19.91\(\pm\)5.88** & **10.24\(\pm\)2.62** & **16.50\(\pm\)4.36** & 22.12\(\pm\)2.56 & 21.64\(\pm\)5.75 & 36.84\(\pm\)5.85 & 21.99 \\
**Turkey** & 98 (6.3x) & 12.90 & 11.80\(\pm\)1.03 & 9.84\(\pm\)3.51 & 14.98\(\pm\)3.89 & 6.10\(\pm\)1.57 & 10.25\(\pm\)2.91 & 25.22\(\pm\)1.46 & 22.96\(\pm\)4.04 & 30.01\(\pm\)2.30 & 16.37 \\
**Japan** & 186 (2.0x) & 9.35 & 11.06\(\pm\)1.19 & 9.44\(\pm\)1.18 & 1.52\(\pm\)1.89 & 5.45\(\pm\)1.90 & 9.80\(\pm\)1.40 & 23.15\(\pm\)2.10 & 22.04\(\pm\)3.10 & 22.44\(\pm\)3.19 & 14.43 \\
**France** & 175 (11.3x) & 16.50 & 9.33\(\pm\)1.24 & 7.51\(\pm\)0.90 & 8.80\(\pm\)3.24 & 3.86\(\pm\)1.62 & 4.57\(\pm\)1.11 & 29.56\(\pm\)2.38 & 21.95\(\pm\)4.38 & 30.01\(\pm\)2.19 & 14.45 \\
**U.K** & 83 (5.4x) & **18.67** & 12.47\(\pm\)1.77 & 13.72\(\pm\)2.20 & 14.20\(\pm\)4.01 & 6.16\(\pm\)2.98 & 9.88\(\pm\)3.06 & 29.06\(\pm\)2.47 & 21.82\(\pm\)3.43 & 32.15\(\pm\)4.06 & 17.43 \\
**Mezko** & 57 (3.7x) & 9.40 & 8.25\(\pm\)1.04 & 13.52\(\pm\)2.88 & 8.87\(\pm\)2.18 & 34.01\(\pm\)4.99 & 5.96\(\pm\)2.36 & 25.24\(\pm\)1.58 & 22.03\(\pm\)5.65 & 28.97\(\pm\)3.45 & 14.12 \\
**India** & 132.8x) & 11.57 & 8.21\(\pm\)1.35 & 19.69\(\pm\)3.70 & 15.29\(\pm\)3.98 & 7.80\(\pm\)1.92 & 1.85\(\pm\)2.17 & 33.88\(\pm\)3.00 & 25.89\(\pm\)2.17 & 33.88\(\pm\)3.00 & 25.81\(\pm\)2.89 & **2.28** \\
**Germany** & 57 (3.7x) & 14.02 & 8.54\(\pm\)1.77 & 7.84\(\pm\)1.01 & 9.04\(\pm\)2.39 & 4.38\(\pm\)1.68 & 4.68\(\pm\)1.56 & 29.05\(\pm\)2.63 & 20.42\(\pm\)5.66 & 28.86\(\pm\)3.78 & 14.33 \\
**China** & 97 (6.3x) & 8.60 & 15.00\(\pm\)2.45 & 14.08\(\pm\)3.17 & 5.12\(\pm\)4.51 & 7.80\(\pm\)1.97 & 13.29\(\pm\)2.02 & 28.96\(\pm\)3.15 & 21.20\(\pm\)4.78 & 30.92\(\pm\)1.82 \\
**Iran** & 21 (1.4x) & 12.60 & 9.16\(\pm\)1.75 & 8.19\(\pm\)1.51 & 13.56\(\pm\)3.16 & 9.00\(\pm\)2.04 & 6.21\(\pm\)1.05 & **40.26\(\pm\)8.19** & **38.36\(\pm\)9.81** & **45.72\(\pm\)3.65** & 20.93 \\
**Greece** & 21 (1.4x) & 15.07 & 4.09\(\pm\)1.01 & 3.48\(\pm\)1.45 & 4.85\(\pm\)1.37 & 3.81\(\pm\)1.09 & 1.32\(\pm\)1.05 & 3.16\(\pm\)8.85 & 8.85\(\pm\)2.54 & 30.60\(\pm\)3.17 & 12.09 \\
**Spain** & 99 (6.1x) & 16.65 & 9.18\(\pm\)0.93 & 9.18\(\pm\)1.04 & 7.42\(\pm\)1.51 & 3.98\(\pm\)1.37 & 3.33\(\pm\)4.06 & 25.21\(\pm\)5.07 & 17.00\(\pm\)4.32 & 35.36\(\pm\)1.68 & 12.33 \\
**Russia** & 27 (1.7x) & 10.64 & 4.72\(\pm\)0.98 & 7.10\(\pm\)2.81 & 6.44\(\pm\)1.97 & 1.41\(\pm\)0.60 & 2.21\(\pm\)0.33 & 14.83\(\pm\)4.80 & 10.47\(\pm\)0.98 & 11.77\(\pm\)0.59 & 7.37 \\
**ALL** & 1549 (100.0x) & 13.29 & 12.56\(\pm\)1.06 & 11.58\ 

(see Table 8) show that the U.S. cultural group achieves the highest performance, while decoder-only LLMs exhibit significant improvement, consistent with findings using FastText. We calculate the correlation between mWS evaluations using BERT Base and FastText for each LLM. The results reveal a strong correlation for decoder-only LLMs, as shown in Table 1. (3) To explore the correlation between training data size and cultural probing performance, we use country dish counts as a proxy for Wikidata's cultural data size and calculate their correlation with probing performance. The correlation results are show as Table 2. Based on the probing performance evaluated by the two automated metrics, LLMs like Bb, B1, and T5 consistently exhibit little correlations with cultural data size. In contrast, decoder-only LLMs show almost no correlations, indicating limited sensitivity to cultural data size as represented by dish counts. Therefore, in more advanced LLMs, cultural competency isn't directly tied to the training sample size of specific cultural groups.

### Manual evaluation results

Table 1 shows that Llama3 achieve the best probing performance, so we conduct manual evaluation on its probing results. Table 3 displays the results of the GPT-4o simulator and human evaluations, showing a high Pearson correlation of 0.88. Both evaluation methods consistently show that the highest MES scores come from English-speaking cultural groups, including the U.S., U.K., and India, with the U.S. performing especially well in both manual evaluations. This aligns closely with the automatic evaluation results, reinforcing the argument that LLMs tend to exhibit cultural bias toward these groups.

Upon reviewing the evaluation results for the GPT-4o simulator, we find that it treats synonymous expressions as _Direct Matches_, e.g., _cooked rice_ vs. _gohan_ (the Japanese term) and _cauliflower_ vs. _Brassica olercea var. botrytis_ (the botanical term). Additionally, reference labels from Wikida-data often exhibit inconsistencies, with terms like "shrimp or prawn" and "fish as food" marked as ingredients. Furthermore, GPT-4o flags pre-made

Figure 3: Comparison of probing results on Llama3 with prompts in different languages, showing that the English prompt exhibits the best performance.

\begin{table}
\begin{tabular}{l c c|c c c|c c c} \hline \hline
**Origin** & **GPT-4o** & **Human** & **Origin** & **GPT-4o** & **Human** & **Origin** & **GPT-4o** & **Human** \\ \hline
**Italy** & 52.40 & 55.70 & **U.K.** & 64.12 & **72.33** & **Iran** & 53.39 & 51.70 \\
**U.S.** & **70.06** & 70.26 & **Mexico** & 47.05 & 51.95 & **Greece** & 47.68 & 51.23 \\
**Turkey** & 55.19 & 53.59 & **India** & 64.22 & 63.72 & **Spain** & 43.02 & 53.16 \\
**Japan** & 50.76 & 56.07 & **Germany** & 51.71 & 59.02 & **Russia** & 31.42 & 44.86 \\
* \\ \hline \hline \end{tabular}
\end{table}
Table 3: **Manual Evaluation:** Llama3 probing results evaluated using **MES**. The human evaluation score is the average MES from three evaluators per country group. Since evaluators vary by country group, the overall result (ALL) is shown as “-”.

Figure 2: Comparative probing results on FmLAMA-_en_: incorporating cultural information about the origin of dishes into English prompts can enhance the probing of cultural knowledge. Significance levels are indicated by \(*\) for \(p<0.1\), \(**\) for \(p<0.05\), and \(***\) for \(p<0.01\).

 

ingredients as _Substitutable_; for example, _flour_ in Garganelli with pre-made _pasta_. For human evaluators, evaluation results may vary due to individual subjective biases. The average Inter-Rater Reliability (IRR) across country groups, measured by Fleiss' kappa, is 0.57,8 indicating moderate agreement. While this indicates acceptable reliability for subjective tasks, it also highlights that evaluators from the same cultural background may still show variation in their judgments. Although GPT-4o can partially simulate human evaluations, its hallucination issues Ji et al. (2023) raise concerns about the reliability of its assessments. These observations underscore the inherent challenges of using automatic metrics for culturally-related tasks.

Footnote 8: IRR and the average evaluation time for each country group are shown in Appendix C.

## 6 Analysis: Designing Better Prompts

We examine the impact of different prompt settings on cross-cultural knowledge exploration, focusing on cultural background references, multilingual prompts, and code-switching scenarios.

Cultural background analysisUsing the English prompts from Figure 9 as a basis, we incorporate information about the dish country of origin into the probing prompts for each specific dish (as described in SS4.2). Figure 2 presents the comparative probing results on FmLAMA-\(en\) across different LLMs, considering the inclusion of cultural information mentions in the prompts. We find that English prompts with cultural information achieve higher probing scores in capturing LLMs' knowledge within the food domain. This suggests the importance of emphasizing the cultural background when utilizing LLMs, especially in the exploration of culture-related topics.

Multilingual prompt analysisWe utilize prompts in six different languages to conduct knowledge probing on the multilingual LLM, Llama3.9 To ensure a fair comparison of probing results across different language prompts, we filter a sub-dataset where all food instances have both subject and ingredient labels available in all the languages involved. Figure 3 displays probing results on the filtered sub-dataset with prompts in different languages.10 The English prompt performs significantly better than prompts in other languages across all cultural groups, probably due to more training data.

Code-switching analysisFigure 4 presents the probing results using code-switching prompts, for each pair out of six languages, applied to the two best-performing decoder-only LLMs, Qwen2 and Llama3. When English is the main language and the dish subject is presented in another language, probing performance significantly declines but still outperforms probing conducted solely in other languages. This suggests that the instruction language is important, not just the language of the dish name. Furthermore, when the main language of the prompt is non-English and the subject language is English, performance does not significantly decline and even shows a slight improvement. This further emphasizes the dominant role of English in multilingual LLMs. The detailed probing results of each cultural group and other multilingual LLMs are in shown Appendix D.3.

Footnote 9: The probing results comparison on other multilingual LLMs are shown in Appendix D.2.

Footnote 10: Some Chinese and Korean objects are missing Fasttext vectors. Therefore, mWS cannot be calculated for them.

## 7 Error Analysis

A manual inspection of a data sample11 followed by an automated evaluation on the complete data reveals recurring mistake patterns: (1) **Coarse-grained confusions** (foreign ingredients, e.g., _coffee_ in a Chinese dish), where the main issue lies in the model's lack of cultural attribute awareness for

Figure 4: Probing results using code-switching prompts on Qwen2 and LLaMA3.

 

both the dish and its ingredients. (2) **Fine-grained confusions** (wrong local ingredients, e.g., _Chinese wine_ in _Mapo tofu_), and (3) **Inconsistent confusions**, where the model predicts different incorrect ingredients across prompts for the same dish.

Table 4 illustrates the described error patterns. In the results for Llama3 in English, \(47\%\) of the incorrect predictions for Italian dishes are the ingredient _pasta_, \(45\%\) of the incorrect predictions for French dishes are _butter_, and \(40\%\) of the incorrect predictions for Japanese dishes are _tofu_. These are examples of fine-grained confusions. Additionally, _rice_, _wheat_ and _flour_ account for \(11\%\) of the Llama3's total predictions, regardless of the dish's origin. This is a coarse confusion, where the model often predicts the same foreign ingredient.

These errors reveal a lack of cultural knowledge about the dishes and suggest different types of guessing. While coarse and inconsistent confusions seem to reflect more general guesses, possibly influenced by the frequency of certain ingredients in the training data, fine-grained confusions indicate that the model has some knowledge of the dish's origin and is attempting to guess a local ingredient accordingly (i.e. overfitting to the context). Specifically, for fine-grained errors, even when the cultural attributes of candidate ingredients align with the dish, LLMs still make incorrect predictions. While many studies have shown that LLMs can function as knowledge bases Nguyen et al. (2024); Pan et al. (2024) and memorize factual information Fierro et al. (2024, 2024), the familiarity with specific entities is crucial in shaping their predictive expectations Du et al. (2024).

## 8 Conclusion

This study presents an automated method for generating extensive cultural knowledge datasets, exemplified by the creation of FMLAMA, a diverse, food-centric dataset that spans multiple cultures and languages. We introduce novel metrics for cultural knowledge evaluation in LLMs, emphasizing the influence of cultural context and language in the probing process. Our findings reveal a predominant bias towards American culture in LLMs when using English prompts, a bias that diminishes with prompts in other languages. Interestingly, incorporating explicit cultural cues in prompts enhances LLMs' cultural knowledge access. The study also highlights the scarcity of culturally diverse knowledge across languages, pointing to a potential root of observed biases in LLMs.

## 9 Limitations

While this study provides valuable insights into cross-cultural knowledge probing in LLMs, it is essential to acknowledge several limitations. Firstly, the food domain knowledge dataset utilized in this research is sourced from Wikidata, which may not offer comprehensive coverage. For example, the dish _say sauce chicken_ may only include the ingredient _chicken meat_ while lacking the inclusion of _soy sauce_. Moreover, ingredient descriptions are not always detailed. For instance, the Wikidata gold label might be _oil_ when the recipe requires a specific type of oil, such as _sesame oil_. This inconsistency underscores the motivation behind our mWS metric. Furthermore, aside from well-known dishes, certain recipes lack standardization and may vary depending on individual preferences and cooking styles, posing challenges to precise probing. Additionally, the fuzzy-match metric mWS, introduced in this study, relies on Fasttext for obtaining object representation vectors. However, for certain objects in Chinese and Korean, zero vectors may result, rendering similarity calculation impossible. Furthermore, we employ manually crafted templates in this paper. However, research has shown that sampling templates from large corpora can also enhance knowledge-probing evaluation. This aspect is deferred to future work. Despite our endeavors to construct comprehensive multilingual and multicultural knowledge repositories, the availability of aligned cross-cultural knowledge remains

\begin{table}
\begin{tabular}{l|p{260.2pt}} \hline \hline
**Country** & **Top 5 Most Falsely Predictions (Llama3)** \\ \hline
**Italy Japan** & pasta, bread, meat, egg, pork rice, fish, soy, tofu, chicken butter, cream, meat, caramel, bread \\
**Russia** & meat, beef, cabbage, pork, beet \\ \hline  & (a) Coarse-grained confusions \\ \hline
**Model** & **Top 5 Most Frequent Predictions** \\ \hline
**BERT T5** & dried meat, ham, beef, pork, fish vinaigrette, scrambled eggs, buckweheat flour, citotta, saltana ice, meat, bread, white meat, egg rice, meat, sugar, chicken, bread \\ \hline  & (b) Fine-grained confusions \\ \end{tabular}
\end{table}
Table 4: Error Analysis: Coarse and fine-grained confusions in ingredient predictions by various cultural groups and models.

 

limited in multilingual settings. This constraint presents challenges in exploring the interaction between language and culture.

## Acknowledgments

This research is supported by the project of Shenzhen Science and Technology Research Fund (Fundamental Research Key Project Grant No. JCYJ20220818103001002), Shenzhen Science and Technology Program (Grant No. ZDSYS20230626091302006), Key Project of Shenzhen Higher Education Stability Support Program (Grant No. 2024SC0009).

## References

* Adilazuarda et al. (2024) Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradiyumna Lavania, Siddhant Shivdut Singh, Alham Fikri Aji, Jacki O'Neill, Ashutosh Modi, and Monojit Choudhury. 2024. Towards measuring and modeling "culture" in LLMs: A survey. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 15763-15784, Miami, Florida, USA. Association for Computational Linguistics.
* Aguilar and Solorio (2020) Gustavo Aguilar and Thamar Solorio. 2020. From English to code-switching: Transfer learning with strong morphological clues. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8033-8044, Online. Association for Computational Linguistics.
* At@Meta (2024) AI@Meta. 2024. Llama 3 model card.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _Transactions of the Association for Computational Linguistics_, 5:135-146.
* Cao et al. (2023) Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between ChatGPT and human societies: An empirical study. In _Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)_, pages 53-67, Dubrovnik, Croatia. Association for Computational Linguistics.
* Deshpande et al. (2022) Awantee Deshpande, Dana Ruiter, Marius Mosbach, and Dietrich Klakow. 2022. StereoKG: Data-driven knowledge graph construction for cultural knowledge and stereotypes. In _Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH)_, pages 67-78, Seattle, Washington (Hybrid). Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Dogruoz et al. (2021) A. Seza Dogruoz, Sunayana Sitaram, Barbara E. Bullock, and Almeida Jacqueline Toribio. 2021. A survey of code-switching: Linguistic and social perspectives for language technologies. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1654-1666, Online. Association for Computational Linguistics.
* Du et al. (2024) Kevin Du, Vesteinn Snebjarnarson, Niklas Stoehr, Jennifer White, Aaron Schein, and Ryan Coterell. 2024. Context versus prior knowledge in language models. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 13211-13235, Bangkok, Thailand. Association for Computational Linguistics.
* Elazar et al. (2021) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilisha Ravichander, Eduard Hovy, Hinrich Schutze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. _Transactions of the Association for Computational Linguistics_, 9:1012-1031.
* Fierro et al. (2024a) Constanza Fierro, Negar Foroutan, Desmond Elliott, and Anders Sogaard. 2024a. How do multilingual models remember? investigating multilingual factual recall mechanisms. _arXiv preprint arXiv:2410.14387_.
* Fierro et al. (2024b) Constanza Fierro, Nicolas Garneau, Emanuele Bugliarello, Yova Kenentchedjhieva, and Anders Sogaard. 2024b. MuLan: A study of fact mutability in language models. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)_, pages 762-771, Mexico City, Mexico. Association for Computational Linguistics.
* Fierro and Sogaard (2022) Constanza Fierro and Anders Sogaard. 2022. Factual consistency of multilingual pretrained language models. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 3046-3052, Dublin, Ireland. Association for Computational Linguistics.
* Gonzalez et al. (2021) Ana Valeria Gonzalez, Anna Rogers, and Anders Sogaard. 2021. On the interaction of belief bias and explanations. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 2930-2942, Online. Association for Computational Linguistics.
* Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Estuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. _ACM Computing Surveys_, 55(12):1-38 

Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020a. X-FACTR: Multilingual factual knowledge retrieval from pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 5943-5959. External Links: Link Cited by: SS1.
* Z. Jiang, F. F. Xu, J. Araki, and G. Neubig. 2020b. How Can We Know What Language Models Know? Transactions of the Association for Computational Linguistics, pp. 8:423-438. External Links: Link, Document Cited by: SS1.
* A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jegou, and T. Mikolov (2016)Fasttext. zip: compressing text classification models. arXiv preprint arXiv:1612.03651. Cited by: SS1.
* M. Kaneko, A. Imankulova, D. Bollegala, and N. Okazaki (2022)Gender bias in masked language models for multiple languages. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, pp. 2740-2750. External Links: Link, Document Cited by: SS1.
* N. Kassner, P. Dufter, and H. Schutze (2021)Multilingual lamma: investigating knowledge in multilingual pretrained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online, pp. 3250-3258. External Links: Link, Document Cited by: SS1.
* A. Keleg and W. Magdy (2023)DLAMA: a framework for curating culturally diverse facts for probing the knowledge of pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, pp. 6245-6266. External Links: Link, Document Cited by: SS1.
* Y. Kuratov and M. Arkhipov (2019)Adaptation of deep bidirectional multilingual transformers for russian language. arXiv preprint arXiv:1905.07213. Cited by: SS1.
* H. Lent and A. Sogaard (2021)Common sense bias in semantic role labeling. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), Online, pp. 114-119. External Links: Link, Document Cited by: SS1.
* Z. Meng, F. Liu, E. Shareghi, Y. Su, C. Collins, and N. Collier (2022)Review-then-probe: a contrastive recipe for probing biomedical knowledge of pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 4798-4810. External Links: Link, Document Cited by: SS1.
* M. Minkov and Geert Hofstede (2012)Is national culture a meaningful concept? cultural values delineate homogeneous national clusters of in-country regions. Cross-Cultural Research46 (2), pp. 133-159. External Links: Link, Document Cited by: SS1.
* Anjishnu Mukherjee, A. Caliskan, Z. Zhu, and A. Anastasopoulos (2024)Global gallery: fine art of painting culture portraits through multilingual instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Mexico City, Mexico, pp. 6398-6415. External Links: Link, Document Cited by: SS1.
* T. Nguyen, S. Razniewski, A. Varde, and G. Weikum (2023)Extracting cultural commonsense knowledge at scale. In Proceedings of the ACM Web Conference 2023, pp. 1907-1917. External Links: Link, Document Cited by: SS1.
* T. Nguyen, S. Razairewski, and G. Weikum (2024) Cultural commonsense knowledge for intercultural dialogues. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 1774-1784. External Links: Link, Document Cited by: SS1.
* O. Al, J. Achiam, I. Akkaya, K. Leoni Aleman, D. Almeida, J. Atlenschmidt, S. Altman, S. Amadatk, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Batlescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berrner, L. Bogdonoff, O. Boiko, M. Boyd, A. Brakman, G. Brockmann, T. Brooks, M. Brundage, K. Button, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. Won Chung, D. Cummings, J. Currier, Y. Dai, C. Decareaux, T. Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Ecoffet, A. Eleti, T. Eloundou, D. Farhi, L. Fedus, N. Felix, S. Posada Fishman, J. Forte, I. Fullford, L. Gao, E. Georges, C. Gibson, V. Goel, T. Gogineni, G. Goh, R. Gontijo-Lopes, J. Gordon, M. Garstein, S. Gray, R. Greene, J. Gross, S. Shane Gu, Y. Guo, C. Hallacy, J. Han, J. Harris, Y. He, M. Heaton, J. Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Houghton, K. Hsu, S. Hu, X. Hu, J. Houizinga, S. Jain, S. Jain, Q. Jang, A. Jiang, R. Jiang, H. Jin, D. Jinno, B. Jontou, B. Jeewoo Jun, T. Kaftan, L. Kaiser, A. Kamali, I. Kanitscheider, N. Shirish Keskar, T. Khan, L. Kilpatrick, J. Wook Kim, C. Kim, Y. Kim, J. Kirchner, J. Kiros, M. Knight, D. Kokotajlo, L. Komdraciuk, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy, C. Ming Li, R. Lim, M. Lin, R. Litwin, Theresa Lopez, R. Lowe, P. Lue 

Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLceavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Melly, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantam, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emry Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael Pokorny, Michelle Pokras, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurt, John Schulman, Daniel Seslam, Kyla Sheppard, Toki Sherabkov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Syrmon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Alama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchin, Elizabeth Tseng, Preston Taggle, Nick Turley, Jerry Tworek, Juan Fe-lipe Ceron Uribe, Andrea Vallone, Arun Vijayergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihanda, Peter Welinder, Jiayi Weng, Lilian Wong, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. Gpt-t technical report.
* Palta and Rudinger (2023)Shramay Palta and Rachel Rudinger. 2023. FORK: A bite-sized test set for probing culinary cultural biases in commonsense reasoning models. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 9952-9962, Toronto, Canada. Association for Computational Linguistics.
* Pan et al. (2024) Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2024. Unifying large language models and knowledge graphs: A roadmap. _IEEE Transactions on Knowledge and Data Engineering_.
* Park et al. (2018) Sungjoon Park, Jhiyoung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junesong Kim, Yongsook Song, Tahewan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeyonghwa Lee, Seongbo Jang, Seungwon Do, Sunkyoung Kim, Kyungtae Lim, Jongwen Lee, Kyumin Park, Jamin Shin, Seonghyun Kim, Lucy Park, Alice Oh, Jungwoo Ha, and Kyunghyun Cho. 2021. Klue: Korean language understanding evaluation.
* Peterson et al. (2018) Mark F Peterson, Mikael Sondergaard, and Aycan Kara. 2018. Traversing cultural boundaries in ib: The complex relationships between explicit country and implicit cultural group boundaries at multiple levels. _Journal of International Business Studies_, 49:1081-1099.
* Petroni et al. (2019) Fabio Petroni, Tim Rocktaschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _The Journal of Machine Learning Research_, 21(1):5485-5551.
* Safaya et al. (2020) Ali Safaya, Moutasem Abdullaif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERTCNN for offensive speech identification in social media. In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_, pages 2054-2059, Barcelona (online). International Committee for Computational Linguistics.
* Savoldi et al. (2021) Beatrice Savoldi, Marco Gaido, Luisa Bentivolgli, Matteo Negri, and Marco Turchi. 2021. Gender bias in machine translation. _Transactions of the Association for Computational Linguistics_, 9:845-874.
* Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4222-4235, Online. Association for Computational Linguistics.
* Singh et al. (2024) Shivalika Singh, Angelika Romanou, Clementine Fourier, David I Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limknochotiwat, Kelly Marcchio, Wei Qi Leong, Yosephine Susanto, et al. 2024. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation. _arXiv preprint arXiv:2412.03304_.
* Sogaard (2021) Anders Sogaard. 2021. Locke's holiday: Belief bias in machine reading. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 8240-8245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics 

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Wang et al. (2023) Weixuan Wang, Barry Haddow, Alexandra Birch, and Wei Peng. 2023. Assessing the reliability of large language model knowledge. _arXiv preprint arXiv:2310.09820_.
* Wu et al. (2023) Weiqi Wu, Chengyue Jiang, Yong Jiang, Pengjun Xie, and Kewei Tu. 2023. Do PLMs know and understand ontological knowledge? In _Proceedings of the 61st Annual Meeting of the Association for Computational Liapistics (Volume 1: Long Papers)_, pages 3080-3101, Toronto, Canada. Association for Computational Linguistics.
* Xue et al. (2020) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. _arXiv preprint arXiv:2010.11934_.
* Yang et al. (2024) An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chenpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. _arXiv preprint arXiv:2407.10671_.
* Yin et al. (2022) Da Yin, Hritik Bansal, Masoud Monajatipoor, Linnian Harold Li, and Kai-Wei Chang. 2022. GeoM-LAMA: Geo-diverse commonsense probing on multilingual pre-trained language models. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 2039-2055, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Zhong et al. (2021) Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5017-5033, Online. Association for Computational Linguistics.
* Zhou et al. (2023) Li Zhou, Antonia Karamolegkou, Wenyu Chen, and Daniel Hershcovich. 2023. Cultural compass: Predicting transfer learning success in offensive language detection with cultural features. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 12684-12702, Singapore. Association for Computational Linguistics.

## Appendix A FmLamA details

### Examples

Figure 5 illustrates examples from dataset FmLamA. Each dish instance in FMLAMA is defined as: \((url,na,cou,la,pa,[ma,im])\), the elements in \([\cdot]\) indicate optional:

* \(url\): the link in Wikidata;
* \(na\): the name of the dish;
* \(cou\): the country of origin of the dish;
* \(la\): the language used in this entry;
* \(pa\): the ingredients of this dish;
* \(ma\): the material used in the dish;
* \(im\): the image of this dish.

Particularly, in each instance, \(la={\rm LANG}~{}(na)={\rm LANG}~{}(pa)={\rm LANG}~{}(ma)\), which indicates the descriptive language for all attributes remains consistent. For a dish with the same \(url\), there may be several instances with different languages. Table 5 contrasts our dataset with prior cultural knowledge collections.

### Statistics

There are a total of 33,600 dishes in FmLamA. We count the number of dishes corresponding to each language and country in the dataset, as shown in Figures 6 and 7. Figure 5(a) shows the top 20 languages ranked by the number of dishes, with English (en) having the highest count at 2804, followed by Spanish (es), French (fr), and Japanese (ja). Figure 5(b) displays the distribution of dish counts across different intervals for all languages, indicating that most languages have fewer than 200 dishes. Similarly, Figure 6(a) shows the top 20 countries ranked by the number of dishes, with Italy having the highest count at 2975, followed by France, America, and Japan. Figure 6(b) shows the distribution of dish counts across various intervals for all countries, indicating that only 8 countries have more than 200 dishes. Figure 8 shows the dish statistics by ingredient count. The majority of dishes contain only one ingredient, totaling 18,546. Dishes with two ingredients are the second most common, with a count of 6,426. The counts decrease as the number of ingredients increases, with dishes containing eleven ingredients being the least common, at 32.

 



### Language comparison

#### b.2.1 Geographic differences

In this section, we discuss the representative countries where each selected language is spoken and their geographic distribution: (1) _English_: Mainly spoken in U.S., Canada, Australia, Guyana, etc., spanning North and South America, Europe, Australia, and Africa. (2) _Chinese_: Mainly spoken in China, Singapore, etc., covering huge area of Asia. (3) _Arabic_: Mainly spoken in Saudi Arabia, Egypt, United Arab Emirates, etc., covering the Arabian Peninsula, North Africa, and some sub-Saharan African countries. (4) _Korean_: Mainly spoken in South Korea, North Korea. (5) _Russian_: Mainly spoken in Russia, Belarus, Kazakhstan, etc., covering Eastern Europe, Central Asia, and Northern Asia. (6) _Hebrew_: Mainly spoken in Israel, with Hebrew-speaking communities worldwide. These languages represent cultural diversity, being spoken on different continents by groups with rich and distinct cultural backgrounds, demonstrating the broad geographic and cultural diversity represented by these languages.

#### b.2.2 Grammatical differences

In addition to geographical differences in usage, these six languages also display the following grammatical and reading distinctions: (1) _Word Order_: English and Chinese follow an Subject-Verb-Object word order, Arabic typically uses Verb-Subject-Object, Korean follows Subject-Object-Verb, while Russian and Hebrew are relatively flexible but generally adhere to Subject-Verb-Object. (2) _Tense_: English, Arabic, Russian, and Hebrew possess complex tense systems. Chinese lacks a strict tense structure, whereas Korean expresses tense through verb conjugation. (3) _Gender_: Arabic, Russian, and Hebrew have grammatical gender, while English distinguishes gender only in pronouns. Chinese and Korean do not have grammatical gender. (4) _Articles_: English, Arabic, and Hebrew have an article system, whereas Chinese, Korean, and Russian do not use articles. (5) _Reading Direction_: English, modern Chinese, modern Korean, and Russian are read from left to right, while Arabic and Hebrew are read from right to left. Traditional Chinese and traditional Korean are read vertically from top to bottom, with columns arranged from right to left. (6) _Case System_: Ara

Figure 5: Examples of FmLAMA.



bic and Russian have complex case systems, while Korean employs particles to indicate grammatical relations. English, Chinese, and Hebrew do not use case inflections.

## Appendix C Manual Evaluation Setup

### GPT-4o simulator

The evaluation prompts for the GPT-4o simulator are shown in Table 6.

### Human evaluator

Figure 10 shows the human evaluation website. The Inter-Rater Reliability (IRR) and the average evaluation time for the three evaluators from each country group are provided in Table 7.

 

klue/bert-base (Park et al., 2021) and bert-base-chinese.

The probing results with prompts in the other five languages (Arabic, Chinese, Hebrew, Korean, and Russian) on the corresponding filtered sub-datasets (FmLAMA_-ar_, FmLAMA_-zh_, FMLAMA_-he_, FMLAMA_-ko_, and FmLAMA_-ru_) are depicted in Table 9, 10, 11, 12, and 13, respectively. Because certain objects in Chinese and Korean have representation vectors that result in all zeros when obtained through Fasttext, calculating cosine similarity was not feasible. Consequently, mWS evaluation was not conducted for prompts in Chinese and Korean prompts.

### Language analysis on other multilingual LLMs

Figures 11, 12, 13, and 14 present a comparison of probing results for the multilingual LLMs--mBERT, mT5, Qwen2, and Llama2--using prompts in various languages on the corresponding filtered sub-datasets. Decoder-only LLMs, such as Qwen2, Llama2, and Llama3, while representing the cutting edge of current LLM technology, exhibit greater performance variation under multilingual prompt settings.

### Code-switching probing results in each cultural group

Figure 15- 28 presents a detailed comparison of the probing results for each cultural group and other multilingual LLMs.

## Appendix E Cultural bias _vs_ Dish popularity

Table 1 shows that some LLMs perform better in probing tasks for low-resource language groups. Specifically, the decoder-only LLM performs well in the Iran group (both mAP and mWS metric), while Greece shows decent results with both T5 and decoder-only LLMs (only mWS metric).

To understand the strong performance of the Iran group across different metrics, we analyzed the results for the 21 dishes. Table 14 shows 17 involved dishes where the predicted ingredients ranked in the top 10 (Llama3). Since the top-ranked predicted ingredients in the table are not all commonly used, we can rule out the possibility that the strong performance of the Iran group is due to the LLM's common ingredient prediction errors. We hypothesize two possible reasons for this performance: (1) LLM Cultural Ability: Decoder-only LLMs may be more familiar with the Iran cultural group. (2) Data Bias: The dishes in the Iran group are likely well-known and representative of Iranian cuisine, so the LLM might show stronger memory performance for familiar entities (Du et al., 2024).

 



**Instructions**

You are a professional evaluator. I will provide you with a dish, its reference ingredient label sourced from Wikidata, and a list of predicted ingredients. Your task is to evaluate whether each predicted ingredient is appropriate for the respective dish.

### Evaluation Criteria:

For each predicted ingredient, mark it as **Correct**, **Maybe**, or **Incorrect** using the following guidelines:

* **Correct**: The ingredient meets one of these:
* **Direct Match**: It is explicitly listed in the reference ingredient label from Wikidata.
* **Substitutability**: It can replace a specific ingredient in the reference label during cooking.
* **Missing Traditional Ingredient**: It is traditionally or commonly used in the dish, but not listed in the reference label.
* **Maybe**: The ingredient could be used in some variations of the dish, but its use is uncommon or ambiguous.
* **Incorrect**: The ingredient is rarely or never used in the dish.

### Output Format:

Please directly return the evaluation as a Python dictionary with JSON format where the key is the predicted ingredient, and the value is a tuple consisting of: 1. The evaluation ("Correct", "Maybe", "Incorrect") 2. The reason ("Direct Match", "Substitutability", "Missing Traditional Ingredient", or a brief explanation)

### Example:

Input:

Dish: Spaghetti Bolognese

Reference ingredient label: spaghetti, ground beef, onion, garlic, Carrot

Predicted ingredients: spaghetti, basil, chicken, tomato, beef

Output  {  'ground beef': ('Correct', 'Substitutability'),  'meat': ('Maybe', 'Ambiguous'),  'bef': ('Correct', 'Substitutability'),  'pasta': ('Incorrect', 'Pasta is not part of the sauce itself'),  'turkey meat': ('Maybe', 'It can be used in some variations'),  } ```

\begin{table} \end{table}
Table 6: GPT-4o simulator: Evaluation Instructions



## Step 1: read guidelines

Wecome to our evaluation website!

**of Objective:**

* You extract a record group (busy) before data you can transfer with, based on your instructions, you will assume a lot of student efforts, and wish to the English-written and it is recommended that computer vision.
* You look this to include whether the periodogram responds as appropriate for the knowledge drawn. For each product product, you will need the solution to rank at an Event Brack or Direct.
* Evaluate Choice:**
* Create Data Analysis: how to perform these.
* Check Checkbook: in which case the program would find several forms available.
* Subordinate Checkbook: in which case a specific thread during two conditions options.
* Muting Technologies: In addition to memory of the run, this does not have the software to predict their own's mistakes.
* Make The project could be used to make use windows of the old, but has no successors or endorsers.
* Document: The project needs to carry out most tasks in the disk.
* Execute:
* Form a set of work files with the replacement of a particular datum, you must necessarily insert the script online to other seconds information.
* Print in all orders as Outflow? or other if possible to the same.
* Confirm:
* Use from GPS ready next and informed the abilities, and then allow them to wait into the validation checks.

### Step 2: choose cultural group Select Your Cultural Group

Please select the cultural group (activity) whose actions span are tolerable with.

* Select Other Cannot Cross \(\rightarrow\)

#### 3.0.1 Step 3: evaluate each dish

Dish Evaluation

_Education Clinic:_

For any medical ingredient, you will know the option to work a Concept. Making, or absent along the following guidance:

* Change the importance and gender.
* Checkbooks: in which case the proposed object should inspect from hundreds.
* Restability: the task quickly takes a model specified specified in the relevant domain.
* Restability: the task quickly takes a model specified specified in the relevant domain.
* Restability: the task quickly takes a model specified specified in the relevant domain.
* Restability: the task quickly takes a model specified specified in the relevant domain.
* Restability: the task quickly takes a model specified specified in the relevant domain.
* Restability: the task quickly takes a model specified specified in the relevant domain.

Figure 10: Human evaluation website

Figure 12: Comparison of probing results on mT5 with prompts in different languages,

Figure 11: Comparison of probing results on mBERT with prompts in different languages.

 

Figure 14: Comparison of probing results on Llama2 with prompts in different languages,

Figure 13: Comparison of probing results on Qwen2 with prompts in different languages.

 

Figure 19: **France**: Probing results of code-switching prompts.

Figure 16: **U.S.: Probing results of code-switching prompts.**

Figure 17: **Turkey**: Probing results of code-switching prompts.

Figure 18: **Japan**: Probing results of code-switching prompts.

 

Figure 23: **Germany**: Probing results of code-switching prompts.

Figure 24: **China**: Probing results of code-switching prompts.

Figure 21: **Mexico**: Probing results of code-switching prompts.

Figure 20: **U.K.**: Probing results of code-switching prompts.

 

Figure 28: **ALL**: Probing results of code-switching prompts.

Figure 27: **Russia**: Probing results of code-switching prompts.

Figure 26: **Spain**: Probing results of code-switching prompts.

Figure 25: **Greece**: Probing results of code-switching prompts.

 



\begin{table}
\begin{tabular}{l r r r} \hline
**Group** & **Dishes** & **IRR** & **Time(m)** \\ \hline
**Italy** & 215 & 0.60 & 74.93 \\
**U.S.** & 285 & 0.48 & 104.25 \\
**Turkey** & 98 & 0.78 & 32.05 \\
**Japan** & 186 & 0.64 & 49.25 \\
**France** & 175 & 0.63 & 71.83 \\
**U.K.** & 83 & 0.46 & 30.33 \\
**Mexico** & 57 & 0.44 & 22.25 \\
**India** & 132 & 0.58 & 40.25 \\
**Germany** & 57 & 0.69 & 20.03 \\
**China** & 97 & 0.59 & 48.50 \\
**Iran** & 21 & 0.61 & 13,33 \\
**Greece** & 21 & 0.35 & 11.30 \\
**Spain** & 95 & 0.57 & 41.50 \\
**Russia** & 27 & 0.51 & 26.50 \\ \hline
* \\ \hline \end{tabular}
\end{table}
Table 7: Human evaluator: IRR and average evaluation time for the three evaluators from each country group.




\begin{table}
\begin{tabular}{l|c c c|c c c|c c c|c} \hline
**Origin** & \multicolumn{4}{c|}{**Encoder-only LLMs**} & \multicolumn{2}{c|}{**Encoder-Decoder LLMs**} & \multicolumn{3}{c|}{**Decoder-only LLMs**} & \multirow{2}{*}{**Avg.**} \\ \cline{2-10}  & & **Bb** & **BI** & **mB** & **mT5** & **T5** & **Qwen2** & **Llama2** & **Llama3** & \\ \hline
**Italy** & 0.7393\(\pm\)0.01 & 0.7425\(\pm\)0.01 & 0.7438\(\pm\)0.00 & 0.6865\(\pm\)0.02 & 0.6579\(\pm\)0.01 & 0.7749\(\pm\)0.01 & 0.7658\(\pm\)0.01 & 0.7869\(\pm\)0.01 & 0.7372 \\
**U.S.** & **0.7748\(\pm\)0.01** & **0.7763\(\pm\)0.01** & **0.7815\(\pm\)0.01** & 0.7071\(\pm\)0.02 & **0.7178\(\pm\)0.03** & 0.7968\(\pm\)0.01 & 0.7839\(\pm\)0.02 & 0.8107\(\pm\)0.01 & 0.7686 \\
**Purkey** & 0.7357\(\pm\)0.01 & 0.7461\(\pm\)0.01 & 0.7545\(\pm\)0.01 & 0.7015\(\pm\)0.02 & 0.6703\(\pm\)0.01 & 0.7681\(\pm\)0.01 & 0.7571\(\pm\)0.01 & 0.7830\(\pm\)0.01 & 0.7395 \\
**Japan** & 0.7113\(\pm\)0.00 & 0.7144\(\pm\)0.00 & 0.7128\(\pm\)0.01 & 0.6788\(\pm\)0.02 & 0.6442\(\pm\)0.01 & 0.7446\(\pm\)0.00 & 0.7446\(\pm\)0.00 & 0.7516\(\pm\)0.00 & 0.7139 \\
**France** & 0.7398\(\pm\)0.01 & 0.7520\(\pm\)0.02 & 0.7513\(\pm\)0.01 & 0.6755\(\pm\)0.02 & 0.6642\(\pm\)0.01 & 0.7941\(\pm\)0.01 & 0.7796\(\pm\)0.01 & 0.7846\(\pm\)0.00 & 0.7426 \\ U.K. & 0.7460\(\pm\)0.01 & 0.7500\(\pm\)0.01 & 0.7550\(\pm\)0.00 & 0.6697\(\pm\)0.02 & 0.6745\(\pm\)0.02 & 0.7678\(\pm\)0.01 & 0.7701\(\pm\)0.01 & 0.7790\(\pm\)0.01 & 0.7390 \\
**Meacheo** & 0.7273\(\pm\)0.01 & 0.7313\(\pm\)0.01 & 0.7291\(\pm\)0.01 & 0.7171\(\pm\)0.01 & 0.6734\(\pm\)0.01 & 0.7605\(\pm\)0.01 & 0.7752\(\pm\)0.02 & 0.7852\(\pm\)0.02 & 0.7371 \\
**India** & 0.7477\(\pm\)0.01 & 0.7523\(\pm\)0.00 & 0.7568\(\pm\)0.00 & 0.6998\(\pm\)0.01 & 0.6619\(\pm\)0.01 & 0.7867\(\pm\)0.00 & **0.7944\(\pm\)0.01** & 0.7935\(\pm\)0.01 & 0.7491 \\
**Germany** & 0.7348\(\pm\)0.01 & 0.7305\(\pm\)0.03 & 0.7493\(\pm\)0.00 & 0.6755\(\pm\)0.01 & 0.6535\(\pm\)0.01 & 0.7835\(\pm\)0.01 & 0.7769\(\pm\)0.02 & 0.7849\(\pm\)0.01 & 0.7361 \\
**China** & 0.7273\(\pm\)0.01 & 0.7319\(\pm\)0.01 & 0.7317\(\pm\)0.01 & 0.7024\(\pm\)0.01 & 0.7054\(\pm\)0.01 & 0.7698\(\pm\)0.01 & 0.7593\(\pm\)0.01 & 0.7793\(\pm\)0.01 & 0.7384 \\
**Iran** & 0.7064\(\pm\)0.01 & 0.7013\(\pm\)0.01 & 0.7310\(\pm\)0.01 & 0.7042\(\pm\)0.02 & 0.6686\(\pm\)0.01 & 0.7905\(\pm\)0.03 & 0.7941\(\pm\)0.04 & 0.8026\(\pm\)0.01 & 0.7370 \\
**Greece** & 0.7488\(\pm\)0.01 & 0.7618\(\pm\)0.02 & 0.7566\(\pm\)0.02 & 0.7190\(\pm\)0.03 & 0.6369\(\pm\)0.00 & **0.8137\(\pm\)0.04** & 0.7739\(\pm\)0.01 & **0.8247\(\pm\)0.01** & 0.7544 \\
**Spain** & 0.7402\(\pm\)0.00 & 0.7465\(\pm\)0.01 & 0.7472\(\pm\)0.00 & 0.6883\(\pm\)0.01 & 0.6391\(\pm\)0.01 & 0.7748\(\pm\)0.01 & 0.7663\(\pm\)0.01 & 0.7765\(\pm\)0.01 & 0.7349 \\
**Russia** & 0.7467\(\pm\)0.01 & 0.7571\(\pm\)0.01 & 0.7573\(\pm\)0.00 & **0.7216\(\pm\)0.02** & 0.6710\(\pm\)0.01 & 0.7716\(\pm\)0.01 & 0.7680\(\pm\)0.01 & 0.7721\(\pm\)0.00 & 0.7457 \\ \hline
**ALL** & 0.7419\(\pm\)0.00 & 0.7466\(\pm\)0.01 & 0.7507\(\pm\)0.00 & 0.6924\(\pm\)0.01 & 0.6723\(\pm\)0.01 & 0.7778\(\pm\)0.01 & 0.7711\(\pm\)0.01 & 0.7859\(\pm\)0.00 & 0.7423 \\ \hline
* \\ \hline \end{tabular}
\end{table}
Table 8: Probing performance comparison with **English** prompts and **FALAMA_–ar_ sub-dataset. The results are evaluated using mWS with **BERT base**. _Corr._ denotes the correlation of mWS evaluations using BERT Base and FastText for each LLM, which reveal a strong correlation on decoder-only LLMs,

\begin{table}
\begin{tabular}{l l|c c c c c|c} \hline
**Origin** & **Count** & **Bb-ar** & **mB** & **mT5** & **Qwen2** & **Llama2** & **Llama3** & **Avg.** \\ \hline
**Italy** & 76 (19.2\%) & 2.56\(\pm\)0.69 & 3.07\(\pm\)1.02 & 5.12\(\pm\)1.20 & 5.13\(\pm\)3.42 & 3.30\(\pm\)1.13 & 4.05\(\pm\)0.81 & 4.03 \\
**U.S.** & 54 (13.7\%) & 5.29\(\pm\)1.18 & 7.22\(\pm\)3.91 & 5.68\(\pm\)1.53 & 9.41\(\pm\)4.12 & 3.93\(\pm\)1.00 & 5.95\(\pm\)2.38 & 5.09 \\
**Survey** & 52 (13.2\%) & 4.07\(\pm\)0.88 & 6.66\(\pm\)3.09 & 3.30\(\pm\)0.53 & 9.20\(\pm\)5.71 & 3.33\(\pm\)1.04 & **9.54\(\pm\)3.48** & 5.47 \\
**Japan** & 44 (11.11\%) & 2.69\(\pm\)1.50 & 2.56\(\pm\)0.50 & 2.13\(\pm\)0.78 & 9.91\(\pm\)3.73 & 3.75\(



\begin{table}
\begin{tabular}{l r|r r r r r r|c c}
**Origin** & **Count** & \multicolumn{2}{c}{**Bb-zh**} & \multicolumn{1}{c}{**mB**} & \multicolumn{1}{c}{**mT5**} & \multicolumn{1}{c}{**Qwen2**} & \multicolumn{1}{c}{**Llama2**} & \multicolumn{1}{c}{**Llama3**} & \multicolumn{1}{c}{**Avage**} \\ \hline
**Italy** & 49 (8.7\%) & 17.18\(\pm\)1.67 & 13.07\(\pm\)4.92 & 10.55\(\pm\)0.84 & 24.54\(\pm\)2.87 & 9.32\(\pm\)4.48 & 13.58\(\pm\)4.08 & 14.71 \\
**U.S.** & 101 (17.9\%) & 19.32\(\pm\)1.47 & 13.79\(\pm\)3.56 & 12.37\(\pm\)1.09 & 26.50\(\pm\)2.94 & 12.91\(\pm\)3.61 & 13.17\(\pm\)6.45 & 16.34 \\
**Turkey** & 12 (2.1\%) & 8.92\(\pm\)3.86 & 6.30\(\pm\)2.12 & 5.22\(\pm\)0.18 & 18.16\(\pm\)3.49 & 8.18\(\pm\)3.58 & 11.00\(\pm\)6.39 & 9.63 \\
**Japan** & 114 (20.2\%) & 15.79\(\pm\)2.32 & 11.18\(\pm\)4.41 & 7.85\(\pm\)0.60 & 17.67\(\pm\)2.73 & 11.15\(\pm\)2.66 & 11.78\(\pm\)4.54 & 12.57 \\
**France** & 70 (12.4\%) & 19.45\(\pm\)4.13 & 16.04\(\pm\)5.31 & 9.14\(\pm\)0.98 & 20.96\(\pm\)1.45 & 13.36\(\pm\)3.62 & 14.71\(\pm\)5.89 & 15.61 \\
**U.K.** & 38 (6.7\%) & **27.88\(\pm\)2.65** & **19.55\(\pm\)7.36** & 10.64\(\pm\)0.68 & 28.43\(\pm\)4.16 & **18.02\(\pm\)8.33** & **21.46\(\pm\)6.75** & 21.00 \\
**Mexico** & 19 (3.4\%) & 21.12\(\pm\)0.86 & 13.40\(\pm\)3.51 & 12.05\(\pm\)0.79 & 16.25\(\pm\)4.70 & 10.60\(\pm\)1.32 & 8.21\(\pm\)3.83 & 13.61 \\
**India** & 24 (4.3\%) & 5.47\(\pm\)1.65 & 5.02\(\pm\)2.68 & 1.13\(\pm\)0.17 & 12.37\(\pm\)3.79 & 4.57\(\pm\)2.20 & 6.58\(\pm\)5.13 & 5.86 \\
**Germany** & 16 (2.8\%) & 10.31\(\pm\)2.53 & 12.46\(\pm\)3.54 & 6.81\(\pm\)1.75 & 10.41\(\pm\)2.12 & 4.08\(\pm\)2.38 & 7.62\(\pm\)4.63 & 8.62 \\
**China** & 87 (15.4\%) & 22.55\(\pm\)4.86 & 18.32\(\pm\)6.75 & 7.88\(\pm\)0.91 & **33.48\(\pm\)6.15** & 13.83\(\pm\)3.72 & 19.96\(\pm\)9.42 & 19.34 \\
**Iran** & 7 (1.2\%) & 7.72\(\pm\)1.44 & 8.19\(\pm\)4.58 & 3.54\(\pm\)0.20 & 8.39\(\pm\)2.08 & 3.40\(\pm\)0.38 & 8.82\(\pm\)3.50 & 6.68 \\
**Greece** & 7 (1.2\%) & 4.11\(\pm\)0.68 & 2.88\(\pm\)1.08 & 5.43\(\pm\)0.43 & 14.81\(\pm\)3.05 & 1.40\(\pm\)0.37 & 3.46\(\pm\)1.23 & 5.35 \\
**Spain** & 12 (2.1\%) & 13.00\(\pm\)1.83 & 10.21\(\pm\)1.58 & **12.72\(\pm\)1.17** & 21.02\(\pm\)0.93 & 5.53\(\pm\)3.77 & 12.41\(\pm\)6.05 & 12.48 \\
**Russia** & 8 (1.4\%) & 5.58\(\pm\)0.93 & 3.98\(\pm\)0.76 & 2.29\(\pm\)0.38 & 5.89\(\pm\)1.09 & 4.06\(\pm\)2.84 & 5.14\(\pm\)1.07 & 4.49 \\ \hline
**ALL** & 564 (100\%) & 17.84\(\pm\)2.37 & 13.56\(\pm\)4.36 & 8.96\(\pm\)0.44 & 22.71\(\pm\)2.37 & 11.46\(\pm\)3.32 & 13.76\(\pm\)5.56 & 14.71 \\ \end{tabular}
\end{table}
Table 10: Probing performance comparison with **Chinese** prompts and **FMLAMA-_zh_ sub-dataset.

\begin{table}
\begin{tabular}{l r|r r r r r|r}
**Origin** & **Count** & \multicolumn{1}{c}{**mB**} & \multicolumn{1}{c}{**mT5**} & \multicolumn{1}{c}{**Qwen2**} & \multicolumn{1}{c}{**Llama2**} & \multicolumn{1}{c}{**Llama3**} & \multicolumn{1}{c}{**Average**} \\ \hline
**Italy** & 59 (18.6\%) & 4.18\(\pm\)2.54 & 4.30\(\pm\)1.49 & 7.78\(\pm\)2.37 & 3.97\(\pm\)1.47 & 6.29\(\pm\)3.14 & 5.30 \\
**U.S.** & 67 (21.1\%) & 8.62\(\pm\)7.22 & 8.13\(\pm\)2.02 & 9.09\(\pm\)4.52 & 3.62\(\pm\)0.81 & 10.17\(\pm\)4.44 & 7.93 \\
**Turkey** & 12 (3.9\%) & 8.99\(\pm\)8.65 & 6.33\(\pm\)4.01 & 9.21\(\pm\)5.92 & 2.93\(\pm\)1.18 & 10.68\(\pm\)6.33 & 7.63 \\
**Japan** & 19 (6.0\%) & 6.69\(\pm\)5.23 & 5.99\(\pm\)3.64 & 6.60\(\pm\)4.19 & 6.92\(\pm\)3.85 & 4.39\(\pm\)3.12 & 6.12 \\
**France** & 65 (20.5\%) & 5.43\(\pm\)3.01 & 4.54\(\pm\)0.92 & 6.40\(\pm\)2.74 & 2.72\(\pm\)0.54 & 5.50\(\pm\)1.47 & 4.92 \\
**UK. K.** & 23 (7.3\%) & **9.43\(\pm\)6.79** & 6.95\(\pm\)2.26 & 9.79\(\pm\)5.06 & 3.97\(\pm\)2.49 & 10.08\(\pm\)5.66 & 8.04 \\
**Mexico** & 11 (3.5\%) & 2.22\(\pm\)0.75 & 2.26\(\pm\)0.74 & 4.33\(\pm\)2.44 & 2.87\(\pm\)1.99 & 2.14\(\pm\)2.01 & 2.76 \\
**India** & 18 (5.7\%) & 8.12\(\pm\)3.72 & 3.31\(\pm\)1.07 & 2.80\(\pm\)2.77 & 3.26\(\pm\)1.95 & 4.73\(\pm\)2.46 & 4.44 \\
**Germany** & 11 (3.5\%) & 7.57\(\pm\)5.98 & **14.84\(\pm\)6.23** & 8.59\(\pm\)3.46 & 5.92\(\pm\)1.83 & 9.38\(\pm\)5.42 & 9.26 \\
**China** & 9 (2.8\%) & 3.61\(\pm\)3.34 & 2.12\(\pm\)0.78 & 3.75\(\pm\)3.





\begin{table}
\begin{tabular}{l c|c c c c c c|c} \hline
**Origin** & **Count** & **Bb-ru** & **mB** & **mT5** & **Qwen2** & **Llama2** & **Llama3** & **Avg.** \\ \hline
**Italy** & 101 (15.9\%) & 4.29\(\pm\)2.04 & 6.57\(\pm\)3.51 & 4.52\(\pm\)1.27 & 7.99\(\pm\)2.61 & 4.30\(\pm\)2.32 & 10.61\(\pm\)2.60 & 6.38 \\
**U.S.** & 79 (12.5\%) & 9.76\(\pm\)4.95 & 9.99\(\pm\)5.25 & 5.51\(\pm\)0.71 & 12.01\(\pm\)3.53 & 9.04\(\pm\)4.33 & 11.97\(\pm\)3.75 & 9.71 \\
**Turkey** & 28 (4.4\%) & 5.75\(\pm\)2.12 & 9.46\(\pm\)1.68 & 3.58\(\pm\)1.61 & 11.08\(\pm\)8.01 & 10.49\(\pm\)2.90 & 16.37\(\pm\)2.97 & 9.46 \\
**Japan** & 68 (10.7\%) & 7.89\(\pm\)4.76 & 7.13\(\pm\)2.34 & 1.76\(\pm\)0.56 & 6.10\(\pm\)2.74 & 8.50\(\pm\)5.79 & 14.11\(\pm\)0.74 & 7.58 \\
**France** & 93 (14.7\%) & 7.08\(\pm\)3.04 & 6.83\(\pm\)2.85 & 5.03\(\pm\)1.86 & 7.03\(\pm\)2.48 & 8.85\(\pm\)4.96 & 10.55\(\pm\)3.43 & 7.56 \\
**UK.** & 33 (5.2\%) & 8.42\(\pm\)5.82 & 10.03\(\pm\)5.01 & 5.54\(\pm\)3.74 & 11.89\(\pm\)6.16 & **13.17\(\pm\)8.89** & 16.82\(\pm\)8.31 & 10.98 \\
**Mezico** & 22 (3.5\%) & 4.08\(\pm\)0.89 & 2.01\(\pm\)0.82 & 2.28\(\pm\)0.95 & 4.70\(\pm\)1.40 & 5.64\(\pm\)3.25 & 4.21\(\pm\)1.34 & 3.79 \\
**India** & 21 (3.3\%) & **11.46\(\pm\)9.78** & **19.02\(\pm\)2.71** & 4.27\(\pm\)3.11 & 16.80\(\pm\)6.73 & 12.95\(\pm\)5.57 & 22.93\(\pm\)2.60 & 14.57 \\
**Germany** & 45 (7.1\%) & 7.53\(\pm\)5.26 & 7.95\(\pm\)3.59 & 4.39\(\pm\)2.03 & 9.41\(\pm\)2.54 & 11.74\(\pm\)4.92 & 11.57\(\pm\)4.32 & 8.77 \\
**China** & 30 (4.7\%) & 9.67\(\pm\)4.64 & 11.86\(\pm\)3.63 & 3.28\(\pm\)2.19 & 10.44\(\pm\)4.98 & 12.63\(\pm\)6.07 & 12.82\(\pm\)4.43 & 10.12 \\
**Iran** & 13 (2.1\%) & 10.86\(\pm\)6.84 & 11.06\(\pm\)5.99 & **15.51\(\pm\)4.53** & **22.88\(\pm\)6.61** & 10.77\(\pm\)8.30 & **23.88\(\pm\)6.87** & 15.83 \\
**Greece** & 18 (2.8\%) & 3.45\(\pm\)2.42 & 1.74\(\pm\)0.43 & 3.38\(\pm\)1.24 & 8.56\(\pm\)2.57 & 4.91\(\pm\)2.85 & 4.77\(\pm\)1.28 & 4.47 \\
**Spain** & 38 (6.0\%) & 3.35\(\pm\)1.51 & 6.51\(\pm\)2.61 & 3.37\(\pm\)1.37 & 5.27\(\pm\)2.52 & 3.88\(\pm\)2.07 & 5.61\(\pm\)1.84 & 4.67 \\
**Russia** & 45 (7.1\%) & 4.79\(\pm\)2.05 & 3.75\(\pm\)0.77 & 2.80\(\pm\)0.92 & 5.21\(\pm\)2.94 & 5.31\(\pm\)2.18 & 5.47\(\pm\)2.27 & 4.55 \\ \hline
**ALL** & 634 (100\%) & 6.85\(\pm\)3.14 & 7.76\(\pm\)1.83 & 4.28\(\pm\)1.11 & 8.84\(\pm\)2.67 & 8.19\(\pm\)4.04 & 11.52\(\pm\)2.82 & 7.91 \\ \hline  & & & & & & & & \\ \end{tabular}
\end{table}
Table 13: Probing performance comparison with **Russian** prompts and **FmLAMA-_r__u_ sub-dataset.

\begin{table}
\begin{tabular}{l r|c c c c c c|c} \hline
**Origin** & **Count** & **Bb-ky** & **Bb-kl** & **mB-u** & **mT5** & **Qwen2** & **Llama2** & **Llama3** & **Avg.** \\ \hline
**Italy** & 88 (16.7\%) & 6.90\(\pm\)1.57 & 7.75\(\pm\)2.34 & 3.85\(\pm\)2.54 & 6.02\(\pm\)0.64 & 3.50\(\pm\)1.10 & 1.19\(\pm\)0.22 & 2.23\(\pm\)0.75 & 4.49 \\
**US.S.** & 64 (12.2\%) & 11.37\(\pm\)2.66 & 10.87\(\pm\)3.86 & **6.32\(\pm\)5.20** & **10.15\(\pm\)0.34** & 2.46\(\pm\)0.69 & 1.68\(\pm\)0.12 & 2.95\(\pm\)0.93 & 6.54 \\
**Turkey** & 19 (3.6\%) & 2.75\(\pm\)2.47 & 4.13\(\pm\)2.62 & 4.07\(\pm\)3.17 & 1.82\(\pm\)0.44 & 3.86\(\pm\)1.00 & 2.16\(\pm\)0.12 & **7.47\(\pm\)1.45** & 3.75 \\
**Japan** & 101 (19.2\%) & 7.34\(\pm\)2.12 & 7.74\(\pm\)1.86 & 1.93\(\pm\)1.24 & 5.25\(\pm\)0.99 & 6.56\(\pm\)5.34 & 2.19\(\pm\)0.22 & 2.62\(\pm\)1.15 & 4.80 \\
**France** & 67 (12.7\%) & 2.33\(\pm\)0.43 & 3.09\(\pm\)0.44 & 3.15\(\pm\)0.17 & 2.79\(\pm\)0.30 & 3.10\(\pm\)0.68 & 1.40\(\pm\)0.09 & 2.41\(\pm\)1.27 & 2.61 \\
**U.K.** & 27 (5.1\%) & **14.07\(\pm\)2.83** & **15.89\(\pm\)4.43** & 6.00\(\pm\)4.42 & 7.55\(\pm\)1.34 & 3.21\(\pm\)1.02 & 1.79\(\pm\)0.31 & 6.89\(\pm\)5.70 & 7.91 \\
**Mexico** & 13 (2.5\%) & 5.18\(\pm\)3.89 & 8.37\(\pm\)2.85 & 4.89\(\pm\)3.29 & 2.31\(\pm\)0.40 & 7.12\(\pm\)3.13 & 2.75\(\pm\)1.99 & 4.18\(\pm\)1.61 & 4.97 \\
**India** & 32 (6.1\%) & 7.98\(\pm\)2.29 & 5.74\(\pm\)1.32 & 3.28\(\pm\)2.09 & 2.96\(\pm\)0.61 & **7.69\(\pm\)6.42** & 2.01\(\pm\)0.32 & 2.36\(\pm\)0.67 & 4.57 \\
**Germany** & 15 (2.9\%) & 2.50\(\pm\)0.94 & 5.0



\begin{table}
\begin{tabular}{l l l c}
**ID** & **Subject (dish)** & **Involved Object (ingredients)** & **Rank** \\
0 & pomegranate soup & pomegranate & 0 \\
1 & Albaloo polo & rice & 1 \\
2 & Kofta & minced meat & 0 \\
3 & Sabzi polo & herb & 5 \\
4 & falooda & milk, vermicelli & 1, 0 \\
5 & ghormeh sabzi & parsley, herb & 0, 4 \\
6 & Bastani & milk, sugar, egg & 8, 7, 6 \\
7 & zeytoon parvardeh & olive & 8 \\
8 & rock candy & sugar & 0 \\
9 & Faloodeh & vermicelli & 0 \\
10 & muhallebi & milk, rice flour & 2, 1 \\
11 & rogan josh & lamb meat & 1 \\
12 & ash-e doogh & yogurt & 0 \\
13 & Mahyawa & fish & 4 \\
14 & sholezard & rice & 1 \\
15 & Eggplant Caviar & eggplant & 0 \\
16 & nán-e panjere & flour & 6 \\  & & & \\ \end{tabular}
\end{table}
Table 14: Top 17 dishes from the Iran group with ingredients ranked in the top 10 by Llama3, excluding common prediction errors.