Venue: NAACL 2024

# Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation

into Multicultural Proverbs and Sayings

Chen Cecilia Liu\({}^{1}\)  Fabri Koto\({}^{2}\) Timothy Baldwin\({}^{2}\) Iryna Gurevych\({}^{1,2}\)

\({}^{1}\)Ubiquitous Knowledge Processing Lab

Department of Computer Science and Hessian Center for AI (hessian.AI)

Technical University of Darmstadt

\({}^{2}\)Natural Language Processing Department, MBZUAI

www.ukp.tu-darmstadt.de

###### Abstract

Large language models (LLMs) are highly adept at question answering and reasoning tasks, but when reasoning in a situational context, human expectations vary depending on the relevant cultural common ground. As languages are associated with diverse cultures, LLMs should also be culturally-diverse reasoners. In this paper, we study the ability of a wide range of state-of-the-art multilingual LLMs (mLLMs) to reason with proverds and savings in a conversational context. Our experiments reveal that: (1) mLLMs "know" limited proverbs and memorizing proverbs does not mean understanding them within a conversational context; (2) mLLMs struggle to reason with figurative proverbs and sayings, and when asked to select the wrong answer (instead of asking it to select the correct answer); and (3) there is a "culture gap" in mLLMs when reasoning about proverbs and sayings translated from other languages. We construct and release our evaluation dataset **MPS** (**MulticulturalA****P**roverbs and **S**ayings) for proverb understanding with conversational context for six different languages, available at [https://github.com/UKPLab/maps](https://github.com/UKPLab/maps).

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 2016–2039 June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

Large language models (LLMs) have achieved impressive results on question answering and reasoning tasks Radford et al. (2019); Brown et al. (2020); Ouyang et al. (2022), inter alia). However, when reasoning in situational context, human expectations may vary cross-culturally Thomas (1983), i.e., pragmatic failure, the inability to understand 'what is meant by what is said') and depend on the knowledge of the relevant cultural common ground (i.e., the shared knowledge based on which people within a culture reason and communicate, including concepts, common sense, etc. Hershcovich et al. (2022). Understanding of such common ground in a cross-lingual setting is specifically understudied in NLP Hershcovich et al. (2022) and neglected in existing LLM literature. As languages and cultures are intertwined Kramsch (2014); Hovy and Yang (2021), it is crucial for models that serve all communities to be able to reason and communicate in a relevant way.

For these reasons, we focus on studying (pragmatic) reasoning conditioned on the cultural common ground of multilingual LLMs. Several questions arise: (1) Do mLLMs embed knowledge of cultural common ground, and does this knowledge affect their reasoning performance? (2) Can mLLMs reason in contexts that require an understanding of cultural common ground? and (3)

Figure 1: Proverbs are fixed expressions used by different cultures. We collect proverbs from six languages (top) and their usage within conversational contexts. We evaluate mLLMs with a binary-choice inference task in the conversational context that contains proverbs (bottom).



Can mLMs reason cross-culturally (i.e., about another culture's common ground, after translating into the same language) and are there gaps in the cultural knowledge (a "culture gap")?1

Footnote 1: Reasoning with cultural common ground may be independent of language. For example, communications among different cultural groups within a multi-cultural country, or communication between L1/L2 speakers of a language where the L2 speaker has acquired the grammatical competence but not the cultural or pragmatic competence.

To answer the above questions, we need to assess mLLMs using fixed, culturally-diverse expressions in multiple languages, that are also used flexibly in situational contexts. Fixed expressions are particularly important for evaluating the memorization of cultural common ground knowledge of LLMs. However, prior work focusing on multivultural concepts such as MaRVL Liu et al. (2021); which is in multimodal) or MABL Kabra et al. (2023) do not contain fixed expressions.

Proverbs and sayings (such as the ones illustrated in Figure 1) are fixed expressions that convey traditional wisdom, sometimes viewed as a form of folk literature and grounded in living experience and social-cultural context White (1987); Mieder (2004); Honeck (2013). While different proverbs may emerge for different cultures, the underlying meaning of proverbs usually expresses universal human experiences. Yet, their literal expression and interpretation can vary from culture Honeck (2013).

For example, the English proverb _The apple doesn't fall far from the tree_ -- means a child grows up to resemble his/her parents. While a plain version _like father like son_ exists in many cultures, this proverb has a similar variant _Rebung ridak jauh dari runpumuya_ ''Bamboo shoots are not far from the clump" in Indonesian, and \({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\not{\exists}}\! \times\!\mathcal{R}\,,~{}~{}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\not{\exists}}\! 

matching proverbs across three languages (BIGbench authors, 2023, with a small evaluation set). To the best of our knowledge, **MAPS** is the largest multilingual dataset that focuses on proverbs and sayings, with conversational contexts and an inference task.

MABL Kabra et al. (2023) is a task similar to ours but focuses on the multicultural understanding of novel metaphors and cross-lingual transfer. It is less suitable for studying memorization versus reasoning and does not study reasoning within a conversational context. Ruis et al. (2023) and Hu et al. (2023) use conversational context to study pragmatic reasoning in English LLMs and the identification of parallels between humans and models, respectively. Concurrently, Huang and Yang (2023) proposed a culturally-aware natural language inference task based on cultural norms. However, they provide limited insights beyond English. While we also use conversational context, we focus on cultural common ground and multilingual aspects of mLLMs (with a larger dataset). Other work on understanding the memory-retrieval mechanism in LLMs with English idioms Haviv et al. (2023), cultural knowledge Wang et al. (2023); Koto et al. (2023); Li et al. (2023b) or cultural value and bias Arora et al. (2023); Haemmerl et al. (2023); Cao et al. (2023), inter alia). Furthermore, we acknowledge existing work intended to study the formal and other types of reasoning in LLMs (such as the ones mentioned in Huang and Chang (2023), which are different in their goals from ours.

## 3 MAPS -- MulticulturAl Proverbs and Sayings

To help investigate our proposed research questions, we first present **MAPS** -- a dataset of proverbs across six geographically and topologically diverse languages. **MAPS** consists of: (1) proverbs and sayings; (2) conversational usages as context; (3) interpretations of proverbs (one correct, one wrong); and (4) labelling of whether the usage of the proverb is figurative or not (see Table 2 for data examples, and Figure 6 in Appendix A.6 for an illustration of the annotation process).

### Dataset Creation

Language Choices.We chose six languages for this dataset: English, German, Russian, Bengali, Mandarin Chinese, and Indonesian. Several factors were considered when choosing the languages, including geographical diversity such as Eastern vs. Western (to increase the potential concept diversity), topological diversity, and resource availability (high-resource vs. lower-resource).

Proverbs and Sayings.We collect all proverbs and sayings (along with explanations) from Wikiquote2 and Wiktionary.3 Bengal has a significantly higher volume of proverbs compared to other languages, and thus we perform random subsampling of the proverbs for annotation to keep the final data roughly balanced between languages.

Footnote 2: [https://en.wikiquote.org/](https://en.wikiquote.org/)

Footnote 3: [https://www.wiktionary.org/](https://www.wiktionary.org/)

Conversational Context.While proverbs and sayings are self-contained, they are typically used in conversations and writing. To investigate the ability of mLLMs to reason with proverbs, next, we created short conversations that use proverbs (i.e., the conversational context for the inference task).

To aid the data creation process, we use a model-in-the-loop approach, inspired by recent work Chakrabarty et al. (2022); Liu et al. (2023). We first use GPT3.5 (gpt-3.5-turbo-0301; a sibling model of Ouyang et al. (2022) by prompting it with fixed templates to generate the seed conversational context (see Appendix B for the model templates).4 Next, we ask two or more native speakers (experts or crowd, with at least one expert per language) to either accept the model-created conversation or write a new conversation if the usage of the proverb is flawed.

Footnote 4: The conversational contexts are in each perspective language, except for Russian and Bengali where the contexts are in English due to quality issues. For Russian and Bengali, the contexts are written in English first, then machine-translated and fixed by native speakers for two rounds.

In the final dataset, the conversational contexts for English, Chinese, Russian, and Bengali were completely rewritten,5 whereas for Indonesian and German, 22% and 20.5% of the original model-generated contexts were retained (the difference is probably due to variations in individual annotator preferences).

Footnote 5: The model has significant trouble in creating relevant context when the proverb is figurative. Anecdotally, human annotators found that the machine-generated context is helpful as a ‘prompt’, which helped to speed up the rewrites.

Interpretation of Proverbs in Context.We formulate this part as an inf 

et al., 2022). We ask annotators to create one correct answer and one wrong answer to the following question based on the conversational context:

_What does the person mean by (proverb)?_

Additionally, we also label the proverb if the interpretation is figurative (i.e., the interpreted meaning of the proverb is different from the expressed literal meaning).6

Footnote 6: “An apple a day keeps the doctor away” is a literal proverb that advocates for apple consumption. “The apple doesn’t fall far from the tree“ is a figurative proverb where the literal meaning is about apples and a natural phenomenon, whereas the actual meaning of the proverb is about a child growing up to resemble his/her parents.

Quality Control.Finally, we sampled 100 conversational contexts with their answers from each language. Then, we asked a separate set of native speakers to assess the data quality for: (1) correct usage of the proverb (i.e., the context is correct); and (2) correct answers for interpreting the meaning. Sometimes, it is possible to have more than one interpretation of a proverb given the context. We asked the native speakers to score the answers as correct as long as the answers aligned with one possible interpretation and revise the options.

The final dataset consists of 2313 proverbs with conversational context. The statistics for each language are in Table 1 (with additional data statistics in Table 6 in Appendix A). We further split the data for each language into a test set and a few-shot train-dev set (30 randomly selected examples each). Table 2 shows examples from our dataset.

### Analysis of Maps

Proverbs and sayings are cultural artifacts and reflect embodied experiences, which contain diverse concepts often grounded in the real world. For instance, dairy product concepts (milk, cheese, yogurt, etc.) exist in different languages but not in Chinese proverbs, whereas concepts that are symbolically meaningful in Chinese culture like dragons or phoenixes exist in the dataset. To illustrate this, we select interesting food items and animals from the final dataset (details in Table 4, Appendix A.2). Furthermore, we categorized the concepts in 100 sampled figurative proverbs in English, Chinese, and Indonesian (for details, see Appendix A.3, Figure 7). We observe that Indonesian has a lot more proverbs that use animals and are about nature than English.

We further encode the proverbs (without contexts) using multilingual sentence embeddings Feng et al. (2022), LaBSE) and plot the embeddings with Kernel Density Estimate (KDE) (after dimensionality reduction to two components using tSNE; van der Maaten and Hinton, 2012) to show the distinctiveness and connections between proverbs across different languages and cultures in Figure 2, which further illustrates that proverbs and sayings are inherently culturally-diverse. To verify this is not due to language difference, we provide additional analysis and discussion in Appendix A.5 to isolate the language effect.

In Figure 2, the embedding distributions are interestingly ordered from the West to the East. Indonesian proverbs partially overlap with English, probably due to the use of the Latin script and influences of foreign languages due to historical context. Chinese and Bengali proverbs are relatively distinct from the Western languages.

## 4 Experimental Setup

We perform zero-shot evaluations and keep all prompt templates in English (on the test set), as previous studies show better performance with English prompts on mLLMs Lin et al. (2022); BigScience Workshop et al. (2022); Muennighoff et al. (2019).

\begin{table}
\begin{tabular}{l l c c} \hline \hline
**Language** & **Code** & **\#Data (Test Size)** & **Class** \\ \hline English & En & 424 (394) & 5 \\ Chinese & Zh & 364 (334) & 5 \\ German & De & 364 (334) & 5 \\ Russian & Ru & 420 (390) & 4 \\ Bengali & Bn & 370 (340) & 3 \\ Indonesian & Id & 371 (341) & 3 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Dataset statistics. “Class” = language class according to Joshi et al. (2020), where 5 means the language is resource-rich.

Figure 2: Visualizing proverb embeddings using kernel density estimation (KDE).

 

2023).7

Footnote 7: For completeness, we also provide additional baselines using MAPS for cross-lingual transfer and few-shot evaluation in Appendix C and Appendix D.7.

Models.We experiment with the following open source state-of-the-art multilingual models: (1) masked LMs (MLM): XLM-R (355m, 3.5B, Conneau et al., 2020); (2) encoder-decoder LMs: mT0 (580M, 3.7B, 13B, multitask and instruction tuned, Muennighoff et al., 2023); and (3) Causal LMs: BLOOMZ (560M, 3B, 7.1B, Muennighoff et al., 2023), and XGLM (564M, 2.9B, 7.5B, Lin et al., 2022). Most of the models cover all 6 languages in MAPS except BLOOMZ, which is derived from BLOOM (BigScience Workshop et al., 2022) and does not cover Russian or German. In addition, despite being primarily an English model, LLaMA-2 (Touvron et al., 2023, Causal LM) has some multilingual capabilities. As a result, we incorporate three LLaMA-2 models (7B, 13B, 70B) in our study.8

Footnote 8: While larger models exist, we chose these models due to computational constraints. We can already see differences in performance at these model sizes and we include additional results for Vicuna-v1.5 (Zheng et al., 2023) and Ayas-101 (Ustun et al., 2024) in the Appendix D.3.2.

Memorization Evaluation.Since proverbs are _fixed_ expressions, successfully completing a proverb with greedy decoding likely means that the model has seen the proverb during pretraining, similar to prior work on detecting memorization or data contamination in LLMs (Magar and Schwartz, 2022; Haviv et al., 2023; Carlini et al., 2023, 2021).9 Hence, following a similar setup to previous work (Magar and Schwartz, 2022; Haviv et al., 2023; Carlini et al., 2023, 2021), we mask out the last word of each proverb and prompt the mLLMs to complete the proverb with templates in Table 8, Appendix B.

Footnote 9: See Appendix D.1.1 for more discussion.

For the memorization task, let \(t_{i}\in T\) be a prompt template, and let \(q_{j}\) be a proverb with \(n\) words where \(q_{j}\triangleq\left\{w_{1},w_{2}\cdots w_{n}\right\}\). We remove the last word \(w_{n}\) for non-MLM models, if the LM generates (greedily) a string that starts with the missing token, or the entire proverb is a sub-string of the generated string, then we count the model as having memorized the proverb. For the MLM model, we mask out the last word with '<mask>' and do predictions (i.e., \(w=\operatorname*{arg\,max}_{w_{n}\in V}P(w_{n}|T_{i};\hat{q}_{j})\), where \(\hat{q}_{j}\) is a proverb with mask token, and \(V\) is the vocabulary).

As the zero-shot prompting results are highly sensitive to the input patterns, we create 5 different prompt patterns (Table 8, Appendix B), and take the union of memorized examples among 5 patterns as the memorization accuracy.

Reasoning Evaluation.For the inference task, we compute the correct answer by comparing the logits of the two answer candidates ('A' or 'B') as in Lin et al. (2022). In particular, we use the prompt template \(t^{r}\) for this task (as in Table 9, Appendix B) and compute \(P(t^{r};q_{i};`A')\) and \(P(t^{r};q_{i};`B')\) and pick the larger one as the correct answer. For the MLM model, we compare the prediction logits of the answer candidates.

Translations for Cross-culture Gap Evaluation.To study gaps in cross-cultural communication, we use English and Chinese as the basis for a case study, with two types of translation data. Ma

\begin{table}
\begin{tabular}{p{42.7pt} p{31.3pt} p{31.3pt}} \hline \hline
**Lang** & Proverb & Context \\ \hline Zh & \begin{tabular}{c} Nasi sudhan \\ benjuat \\ (figurative) \\ \end{tabular} & \begin{tabular}{c} A: \\ (B wants to help A with the project instead of teaching A to the project.) \\ \end{tabular} \\  & \begin{tabular}{c} A: \\ (B wants to teach A to the project.) \\ instead of helping A to do the project.) Answer: B \\ \end{tabular} \\ Id & \begin{tabular}{c} Nasi sudhan \\ benjuat \\ nbuar \\ (figurative) \\ \end{tabular} & 
\begin{tabular}{c} Orang 1: Bagaimana reakis bos-sut steelah kamu menengaukui lessalalanum? Orange 2: Kurnag baik. Says udhan meknocha englusaan sayabrebut begun, taqthe sugarbethai ssduh menigidu. Ni saidsh ehnajidu bubur \\ (Person 1: How did your boss react after you admitted your mistake? Person 2: Not well. I’ve tried to explain why I did this, but I’m still being penalized. The rice has become porridge.) \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table}
Table 2: Examples from selected languages (see Table 7, Appendix A.6 for examples in all languages).

 

chine Translation (MT).** We translate every Chinese proverb, context and answer into English using Google Translate (Zh-En). **Human-Adapted Translation (HT).** We perform adaptations to the machine-translated context: (1) manually correct any mistakes in the literal translation of proverbs, fixing the grammatical errors in the contexts and answers; and (2) conduct a light adaptation of the translated data inspired by Majewska et al. (2023), by replacing names and locations in the dataset to align with the culture (e.g., XiaoMing to Michael etc.) in case LLMs are confused about whether an entity is a person or a place. This represents our best-effort adaptation to reduce the language gap.

## 5 Results and Discussion

### Knowledge of Proverbs

-- A little knowledge is a dangerous thing.While it is possible that the proverbs in the training data appear alone without any contextual usage or explanation, we consider such an occurrence to be unlikely.10 Hence, we make the assumption that memorization of the fixed expression also correlates with LLMs having embedded knowledge of the usage or meaning.Figure 3a shows the results of proverb memorization, which (unsurprisingly) improves with model size. While XLM-R, XLGM, and mT0 cover all of the languages in our dataset, they don't score particularly well in memorization of proverbs in a single language. All models exhibit disparities in memorization across all languages, and these disparities are particularly pronounced in the case of Indonesian, Bengali, and Russian, which are lower-resource languages.11 These disparities are potentially due to data exposure, as we don't find any significant attribution, such as well-known versus less well-known, long versus short, or figurative versus non-figurative proverbs, by analyzing the memorized proverbs.

Footnote 10: Webpages such as this [https://en.wiktionary.org/wiki/no_pain,_no_gain](https://en.wiktionary.org/wiki/no_pain,_no_gain) exist in the training data for LLMs.

### Reasoning of Proverbs with Conversational Context

-- All that glitters is not gold.While many models embed knowledge about proverbs, it is unclear if memorization translates to better reasoning with proverbs given the context.Next, we assess the models using our inference task.Memorization does not indicate the ability to reason with proverbs.We prompt models with the pattern in Table 9 (Appendix B) and plot the accuracy across languages in Figure 3b. In general, the bigger the model is, the better it performs on the inference task (i.e., the ability emerges with scale).Overall, comparing the memorization curve and reasoning curve of mT0, XGLM and XLM-R, we observe that memorization does not indicate the ability to reason with proverbs in our experiments. In fact, model architecture has little effect (as BLOOMZ and LLaMA-2 are Causal LMs, and mT0 is an encoder-decoder model). We provide additional results using different prompts in Appendix D.3.1 and few-shot results in Appendix D.7.Since we know which proverbs are memorized from the previous experiments, we further break down the results into memorized vs. not memorized proverbs for the 3 best-performing models excluding LLaMA-2 70B (as it already achieved good results in Figure 3, which offers limited insights) in English and Chinese in Table 12, Appendix D.2. The benefit of memorization is evident in English and shows inconsistency in Chinese (which aligns with observations for other languages in Figure 3b).One possible explanation for the task not being heavily dependent on memorization is that contextual information aids inference, and the model may also implicitly learn other culturally-relevant information from the training data during pretraining. Consequently, this suggests that LLMs may prioritize contextual information over memory retrieval when both are available. However, such a hypothesis requires further research, which we leave to future work.Figurative proverbs are difficult to understand in general.Many proverbs are figurative, hence, we further divide the results of the model based on this property (described in SS3). Looking at Table 3, we can see that, across English, German, and Russian, all models perform worse on the inference task when the interpretation is figurative. Interestingly, the opposite pattern is consistently observed for Chinese. Larger models appear to understand Indonesian and Bengali figurative proverbs better. One conjecture is that while ab 

stract reasoning (the kind required for understanding figurative proverbs) can rely on memorization, less memorization may lead to better abstract reasoning in LLMs.

Bias towards the correct answer amplifies performance gaps across languagesIf the model genuinely understands a proverb's meaning in a situational context, it should be able to select the correct answer as well as the wrong answer when requested, especially for a task with only two choices. Prior work has shown that negation in the natural language inference task weakens model performance Truong et al. (2023); She et al. (2023); Kassner and Schutze (2020). While not the primary focus of our work, this is a fundamental aspect of reasoning Blanco and Moldovan (2011) and we conducted experiments to verify. Here, we aim to ask a 'negative' question rather than provide negative answers. Hence, we change the question in the prompt template to _What does the person not mean by the proverb?_, while keeping everything else the same.

The results are in Figure 4. By simply asking the model to pick the wrong answer, all previously well-performing models are now performing badly, except mT0 (which may be due to the model being instruction-tuned). The 'negative' question enlarged performance gaps across languages as the model size increased. Additional results on asking the model to pick the wrong answer _without_ using the word _not_ are in Appendix D.4, where we observe consistent trends of model failures and inverse scaling in many cases. While we focus on the cultural aspect of mLLMs, these results show fundamental work is needed to improve the ability of current mLLMs to handle 'negative' questions.

### Culture Gaps in mLLMs - A Case Study

-- _When in Rome, do as the Romans do._

An ideal mLLM should perform on texts from all languages and translations in all directions equally well. However, in our experiments, the performance on English data is still stronger than in other languages for most of the models we studied. Recently, several works have shown that good performance can be achieved by translating non-English text data in languages into English Conneau and Lample (2019); Yang et al. (2019), inter alia). Here, we demonstrate that when a task relies on cultural context, there are two distinct performance gaps to achieving true multilingual ability: one is the language gap (due to mistakes by the translation system, which may be fixed by a perfect translation system), and the other is the culture gap.12 To demonstrate this, we use English and Chinese as the focus of a case study.

Footnote 12: This can relate to cross-cultural pragmatic failure.

We translate the data based on the descriptions in Section 4. Next, we perform the zero-shot evaluation with the best-performing multilingual models (mT0-XXL, 13B) and English model

Figure 3: Performance of mLLMs on the proposed MAPS dataset. The number of parameters is in billions for LLaMA-2 and in millions for all other models.

 

(LLaMA-2 13B) for Zh-En (in Figure 5). In fact, both models show a performance gap in the translated data compared to the target language. Interestingly, mT0 also shows a performance degradation compared to the inference results in the original language (as LLaMA-2 is near chance level for Zh, the improvement is not surprising). In all cases, HT improves over MT, where the gain can be considered as the language gap. More interestingly, we define the gap between HT and the max of source and target language is the _culture gap_ in mLLMs, i.e., _culture gap_\(=|{\rm Acc}^{HT}-\max({\rm Acc}^{Src},~{}{\rm Acc}^{Tgt})|\). The culture gap for Zh-En is 5.73 for mT0 and 19.40 for LLaMA-2.13 In an ideal situation, these gaps should be 0, indicating that the model is culturally aware and capable of understanding a language when speakers come from diverse cultural backgrounds.

Footnote 13: We also perform the same experiment in the reverse direction En–Zh with mT0 (Appendix D.5), similar results were observed. Other evaluation results on machine-translated data for other languages with LLaMA-2 are in Appendix D.5.

By closely examining the machine-translated data, it is evident that current machine translation (MT) systems do not handle cultural context well, producing incomplete or incorrect translations of proverbs. For example, a polysemous phrase was translated to "junior" (third-year university student), but in a specific proverbial context, it means someone is "three years older". Similarly, a phrase like is translated to "not as good as" instead of "it'd be better". Our results suggest that additional research is needed to improve cultural awareness and the inclusion of cultural priors in MT models and mLLMs Yao et al. (2023); Shaikh et al. (2023).

## 6 Conclusion

In this work, we use proverbs and sayings from different languages as an investigative tool to assess the ability of mLLMs to reason with cultural common ground. Specifically, we study various mLLMs to evaluate their ability to memorize proverbs, reason with proverbs and sayings in different situational contexts and understand cross-cultural communications using proverbs.

Figure 4: Performance of mLLMs on the proposed MAPS - Inference task when asking the ‘negative’ question. The number of parameters is in billions for LLaMA-2 and in millions for all other models.

Figure 5: Performance gap between machine-translated, human-translated data and results in the original source language (Zh), and target language (En).

 

To aid the investigation, we developed a multicultural proverbs and sayings dataset **MAPS**. Our analysis shows that many models possess knowledge of proverbs and sayings, however, recognizing proverbs does not mean the model is able to reason with proverbs in contextual settings. Indeed, we found that mT0 shows some culturally-diverse reasoning ability, but only to a very limited extent. We also found that the ability to reason in a zero-shot manner emerges with model scale, but the ability to understand a 'negative' question inversely correlates with the model scale. The disparities in culturally-diverse reasoning ability between languages grow with the model size, which raises concerns in terms of multilingual availability and points to the need for more robust mLLMs. Finally, we defined and observed several culture gaps in cross-lingual communications. We hope to explore different aspects of cultural common ground in the future and to inspire novel work that facilitates inclusive cross-cultural understanding and communication with mLLMs.

## 7 Limitations

Our work uses proverbs and sayings as a proxy for cultural common ground, and we explore mLLMs' ability to understand cultural common grounds in a limited setting. One potential limitation is we only collect one conversation per proverb or saying, and one pair of correct-wrong interpretations. Another limitation is the evaluation data is relatively small compared to many automatically generated benchmarks and may introduce lexical biases. However, these are not major concerns as: (1) we want to focus on cultural common ground, which automatically limits us to a subset of lexical items (lexical biases is an intended feature); and (2) to the best of our knowledge, this is the largest proverb dataset for reasoning in context, and there is enough signal to distinguish between the tested models and uncover insights on current mLLMs ability and limitations in understanding proverbs and sayings. We hope to explore aspects of culture beyond proverbs and sayings, and with a more diverse set of languages (such as African languages or American indigenous languages) in the future.

In this work, we evaluate general-purpose open-source mLLMs. However, a full evaluation of larger models or task-specific models may be necessary, especially when asking 'negative' questions and assessing culture gaps. We focus on studying open-source LLMs in this paper for scientific reproducibility, and closed-source LLM evaluations are beyond our scope. As our dataset is publicly available, it can be used to evaluate closed-source LLMs in the future, and we encourage others to do so.

## 8 Acknowledgement

This work was funded by the German Federal Ministry of Education and Research (BMBF) under the promotional reference 13N15897 (MISRIK), and the LOEWE initiative (Hesse, Germany) within the emergenCITY center. We would like to thank Sukannya Purkayastha, Aniket Pramanick, Ilia Kuznetsov, Kexin Wang, and Luke Bates for their constructive feedback and discussions of the work. We thank Sukannya Purkayastha, Jonathan Tonglet, and Yongxin Huang for their suggestions on a draft of the paper.

## References

* Arora et al. [2023] Arnav Arora, Lucie-aimee Kaffee, and Isabelle Augenstein. 2023. Probing pre-trained language models for cross-cultural differences in values. In _Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)_, pages 114-130, Dubrovnik, Croatia. Association for Computational Linguistics.
* BIG-bench authors [2023] BIG-bench authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. _Transactions on Machine Learning Research_.
* Workshop et al. [2021] BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagne, Alexandra Sasha Luccioni, Francois Yvon, Matthias Galle, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammannachni, Thomas Wang, Benoit Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Staas Behman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurencion, Yacine Jernite, Julien Luany, Margaret Mitchell, Colin Raffel, Aaron Gokasalani, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifelowak Adelani, Dragomir Radey, Eduardo Gonzalez Ponferrada, Efrat Levkovichjah, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gerard Dupont, German Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmunin, Isaac Johnson, Itziar Gonzalez-Dios, Javier 

de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jorg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almburakx, Kimbo Chen, Kyle Lo, Leandro Von Verra, Leon Weber, Long Phan, Lougha Ben allal, Ludovic Tanguy, Mananam Dey, Manuel Romero Munoz, Marium Masoud, Maria Grandury, Mario Sasko, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulajalla Khamis, Olivier Nguyen, Omar Espeljel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis Lopez, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislaw Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vasilina Nikolulina, Veronika Laippala, Violette Lepercq, Virinda Prabhu, Zaid Ajayafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davurt Emre Tsagar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheseth Sharma, Andrea Santiilli, Antoine Charffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szczechla, Gunjan Chahbalmi, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Sriolk Ben-David, Stephen H. Bach, Taewoon Kim, Tail Beers, Thibault Fevry, Trichals Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng-Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tate, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan,atim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Maynak Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrourentetic Nicolas Patry, Nouamane Taxzi, Omar Sanseverio, Patrick von Platen, Pierre Cornette, Pierre Francois Lavallee, Remi Lacroix, Samyam Rajbandari, Sanchit Gandhi, Shaden Smith, Stephane Requena, Suraj Pattil, Tim Dettmers, Ahmed Barwa, Anampret Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramanian, Aurelie Nevoel, Charles Leverning, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan Christoph Kalo, Jeekaterina Novikova, Jessica Zosa Forde, Jordan Cive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Sharvina, Thomas Sciolom, Tian Yun, Tomasz Limisiewicz, Perena Rieser, Vitaly Protasso, Vladislav Mikhailov, Yada Priksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdenek Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antiogna UnIdreaj, Arash Aghagol, Arezo Abdollahi, Aycha Tammour, Azadeh HajIosseini, Bahareh Behrorozzi, Benjamin Ajibade, Bharst Saxena, Carlos Munoz Ferrands, Daniel McDuff, Danish Contractor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emid Baylor, Ezinwanne Ozoani, Fatima Mirza, Frankline Onomiwu, Habib Rezanejad, Hessie Jones, Indrani Bhattacharya, Irene Solaimann, Irina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzler, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Mariam Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akinollu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abram, Nazreen Rajani, Nour Ellcott, Nour Fahmy, Olannewaja Samuel, Ran An, Rasmus Kronnam, Ryan Hao, Samira Alexanderbaud Sububer, Silas Wang, Sourav Roy, Sylvian Viguier, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anim Shukla, Antonio Miranda-Escalada, Ayush Singh, Benjamin Beillharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chauvin Xu, Clementine Fourrier, Daniel Leon Periran, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posado, Karthik Rangasi Sivaranan, Lokesh Bulchandani, Lu Liu, Luis Shinzato, Madeeline Hahn de Bykhovetz, Maiko Takeuchi, Marc Pamies, Maria A Castillo, Marianna Nezhurina, Mario Sanger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihalgic, Minna Liu, Moritz Freidank, Myunsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Mueller, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosadine Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kilbawi, Simon Ott, Since Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Susuli Bharati, Tanny Laud, Theo Gigent, Tomoya Kanuma, Wojciech Kusa, Yanis Labrak, Yash Shailesh Bajaj, Yash Venkataraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. 2022. BLOOM: A 176b-parameter open-access multilingual language model. _CoRR_, abs/2211.05100.
* Blanco and Moldovan (2011) Eduardo Blanco and Dan Moldovan. 2011. Semantic representation of negation using focus detection. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 581-589, Portland, Oregon, USA. Association for Computational Linguistics.
* Brown et al. (2013)Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sarstry, Amanda Askell, Sandhini Agarwal,riel Herbert 

Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems_, volume 33, pages 1877-1901. Curran Associates, Inc.
* Cao et al. (2023) Yong Cao, Li Zhou, Seolhwae Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between ChatGPT and human societies: An empirical study. In _Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)_, pages 53-67, Dubrovnik, Croatia. Association for Computational Linguistics.
* Carlini et al. (2023) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
* Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In _30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021_, pages 2633-2650. USENIX Association.
* Chakrabarty et al. (2022) Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. FLUTE: Figurative language understanding through textual explanations. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 7139-7159, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 7057-7067.
* Conneau et al. (2018) Alexis Conneau, Ruyt Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
* Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 878-891, Dublin, Ireland. Association for Computational Linguistics.
* Ghosh and Srivastava (2022) Sayan Ghosh and Shashank Srivastava. 2022. ePiC: Employing proverbs in context as a benchmark for abstract language understanding. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3989-4004, Dublin, Ireland. Association for Computational Linguistics.
* Haemmerl et al. (2023) Katharina Haemmerl, Bijorn Deiseroth, Patrick Schramowski, Jindrich Libovicky, Constantin Rothkopf, Alexander Fraser, and Kristian Kersting. 2023. Speaking multiple languages affects the moral bias of language models. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 2137-2156, Toronto, Canada. Association for Computational Linguistics.
* Haviv et al. (2023) Adi Haviv, Ido Cohen, Jacob Gidron, Roei Schuster, Yoav Goldberg, and Mor Geva. 2023. Understanding transformer memorization recall through idioms. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 248-264, Dubrovnik, Croatia. Association for Computational Linguistics.
* Hershcovich et al. (2022) Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lohoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruizxiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Sogaard. 2022. Challenges and strategies in cross-cultural NLP. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 6997-7013, Dublin, Ireland. Association for Computational Linguistics.
* Honeck (2013) Richard P Honeck. 2013. _A proverb in mind: The cognitive science of proverbial wit and wisdom_. Psychology Press.
* Hovy and Yang (2021) Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 588-602, Online. Association for Computational Linguistics.
* Hu et al. (2023) Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2023. A fine 

grained comparison of pragmatic language understanding in humans and language models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers)_, pages 4194-4213, Toronto, Canada. Association for Computational Linguistics.
* Huang and Chang [2023] Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 1049-1065, Toronto, Canada. Association for Computational Linguistics.
* Huang and Yang [2023] Jing Huang and Diyi Yang. 2023. Culturally aware natural language inference. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 7591-7609, Singapore. Association for Computational Linguistics.
* Joshi et al. [2020] Pratik Joshi, Sebastian Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6282-6293, Online. Association for Computational Linguistics.
* Kabra et al. [2023] Anubha Kabra, Emmy Liu, Simran Khanuja, Alham Fikri Aji, Genta Winata, Samuel Cahyahwijaya, Anouluwapo Aremu, Perez Ogayo, and Graham Neubig. 2023. Multi-lingual and multi-cultural figurative language understanding. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 8269-8284, Toronto, Canada. Association for Computational Linguistics.
* Kassner and Schutze [2020] Nora Kassner and Hinrich Schutze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7811-7818, Online. Association for Computational Linguistics.
* Koto et al. [2023] Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023. Large language models only pass primary school exams in Indonesia: A comprehensive test on IndoMMLU. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 12359-12374, Singapore. Association for Computational Linguistics.
* Kransch [2014] Claire Kransch. 2014. Language and culture. _AILA Review_, 27(1):30-55.
* Li et al. [2023a] Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023a. Bactrian-X: A multilingual replicable instruction-following model with low-rank adaptation. _CoRR_, abs/2305.15011.
* Li et al. [2023b] Haonan Li, Yixuan Zhang, Fabiri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023b. CMMLU: measuring massive multitask language understanding in chinese. _CoRR_, abs/2306.09212.
* Un et al. [2022] Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Ooyal, Shruti Bhosale, Jingfei Du, Ramakanth Vanuru, Sam Shleifer, Punit Singh Kouz, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona T. Diab, Veselin Stoyanov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 9019-9052. Association for Computational Linguistics.
* Liu et al. [2023] Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah Smith, and Yejin Choi. 2023. We're afraid language models aren't modeling ambiguity. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 790-807, Singapore. Association for Computational Linguistics.
* Liu et al. [2022] Emmy Liu, Chenxuan Cui, Kenneth Zheng, and Graham Neubig. 2022. Testing the ability of language models to interpret figurative language. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4437-4452, Seattle, United States. Association for Computational Linguistics.
* Liu et al. [2021] Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visually grounded reasoning across languages and cultures. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 10467-10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Magar and Schwartz [2022] Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 157-165, Dublin, Ireland. Association for Computational Linguistics.
* Majewska et al. [2023] Olga Majewska, Evgeniiia Razumovskaaia, Edoardo M. Ponti, Ivan Vulic, and Anna Kornonen. 2023. Cross-lingual dialogue dataset creation via outline-based generation. _Transactions of the Association for Computational Linguistics_, 11:139-156.
* Mieder [2004] Wolfgang Mieder. 2004. _Proverbs: A handbook_. Greenwood Publishing Group.
* Muennighoff et al. [2021] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven 

Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoekopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubrak, Samuel Albanie, Zaid Alvarez, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15991-16111, Toronto, Canada. Association for Computational Linguistics.
* December 9, 2022_.
* Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. _OpenAI_.
* Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21:140:1-140:67.
* Ruis et al. [2023] Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktaschel, and Edward Grefenstette. 2023. The Goldilocks of pragmatic understanding: Fine-tuning strategy matters for implicature resolution by LLMs. In _Advances in Neural Information Processing Systems 36_.
* Shaikh et al. [2023] Omar Shaikh, Caleb Ziems, William Held, Aryan Pariani, Fred Morstatter, and Diyi Yang. 2023. Modeling cross-cultural pragmatic inference with code-names duet. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 6550-6569, Toronto, Canada. Association for Computational Linguistics.
* She et al. [2023] Jingyuan S. She, Christopher Potts, Samuel R. Bowman, and Atticus Geiger. 2023. ScoNe: Benchmarking negation reasoning in language models with fine-tuning and in-context learning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 1803-1821, Toronto, Canada. Association for Computational Linguistics.
* Thomas [1983] Jenny Thomas. 1983. Cross-cultural pragmatic failure. _Applied linguistics_, 4(2):91-112.
* Touvrun et al. [2022] Hugo Touvrun, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwijal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moyap Chen, Guillem Cucurull, David Esiobu, Judre Fernandes, Jeremy Fu, Wenyu Fu, Brian Fuller, Cynthia Gao, Vedantaj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan Inanc, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Kotura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yingha Lu, Yuning Mao, Xavier Martinet, Todor Mihalyo, Pushkar Mishra, Igor Molbygo, Yixin Nie, Andrew Poulton, Jeremy Rezienstein, Rashl Rungta, Kalyan Saladi, Alan Schelenlen, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Guan, Pavxin Xu, Zheng Yan, Hiyan Zarov, Yuchen Zhang, Angela Fan, Melanic Ramabudr, Sharan Nara, Aurellen Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Sciolom. 2023. LLMa 2: Open foundation and fine-tuned chat models. _CoRR_, abs/2307.09288.
* Truong et al. [2023] Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, and Trevor Cohn. 2023. Language models are not naysayers: an analysis of language models on negation benchmarks. In _Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)_, pages 101-114, Toronto, Canada. Association for Computational Linguistics.
* Ustun et al. [2024] Ahmet Ustun, Viraat Aryabumi, Zheng Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shaye Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hookef. 2024. Ava model: An instruction finetuned open-access multilingual language model. _CoRR_, abs/2402.07827.
* van der Maaten and Hinton [2012] Laurens van der Maaten and Geoffrey E. Hinton. 2012. Visualizing non-metric similarities in multiple maps. _Machine Learning_, 87(1):33-55.
* Wang et al. [2023] Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, Ai Ti Aw, and Nancy F. Chen. 2023. SeaEval for multilingual foundation models: From cross-lingual alignment to cultural reasoning. _CoRR_, abs/2309.04766.
* White [1987] Geoffrey M White. 1987. _Proverbs and cultural models: An American psychology of problem solving_. Cambridge University Press.
* Yang et al. [2019] Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3687-3692, Hong Kong, China. Association for Computational Linguistics.
* Yao et al. [2023] Binwei Yao, Ming Jiang, Diyi Yang, and Junjie Hu. 2023. Empowering LLM-based machine translation with cultural awareness. _CoRR_, abs/2305.14328 

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.

## Appendix A Dataset

### Annotations

We recruit crowd annotators through Prolific14 with the requirement of the corresponding language as their first language, and being fluent in English. Expert annotators are Master's, PhD and Post-doc researchers, including the authors of this paper. The annotation process is illustrated in Figure 6.

Footnote 14: [http://prolific.com/](http://prolific.com/)

Instructions to create the conversational context:

**Step 1: Check if the proverb is used correctly in the conversation.**

Note: Sometimes, the proverb is figurative, meaning that the underlying meaning and the literal meaning of the proverb are different! The conversation should fit the figurative usage/meaning of the proverb.

Example:

_Person 1: "I'm scared of my boss." Person 2: "Well, barking dogs seldom bite."_

"Barking dogs seldom bite" -It has a literal meaning of dogs that bark rarely take action and bite you, so you don't need to be afraid of getting hurt. The proverb metaphorically describes people that threaten you a lot rarely take action and harm you. Although this conversation may be missing some contexts, it should be labelled as correct.

Example:

_Person 1: "My dog is barking." Person 2: "Well, barking dogs seldom bite."_

The proverb is used in a literal way when it has a figurative meaning. This should be labelled as wrong.

**Step 2: Re-write the conversation if the proverb is not used correctly from step 1.**

The conversation should be 1-turn (1 round between 2 people), and maximum 2-turn (2 rounds between 2 people).

Note: Please do not produce a conversation where one person is asking about the meaning of the proverb.

Instructions to create the answers:

What does the person mean?

Identify the person who used the proverb in the conversation.

Write down a short sentence in the OPT1 column, state what the person means by the proverb in this conversation.

Write down a negative of OPT1 in the OPT2 column.

### Animal and Food Terms in the Dataset

Table 4 shows selected animal and food concepts across different languages. From the data, we can see that proverbs naturally contain culturally important concepts. For example, we can see that "tiger" is a relatively important concept for Eastern cultures, whereas "lion" is more important for Western cultures; while bread is enjoyed by many people around the world, rice is culturally more important in the East.

### Additional Qualitative Analysis of Proverbs

We provide a qualitative analysis of how similar proverbs are expressed differently across languages and cultures. Similar to the ones in our introduction, many proverbs have a similar variant across cultures but are expressed differently.

Figure 6: The data annotation process of MAPS.

 

These proverbs differ by either using concepts that are familiar with the culture or using a local place name or person name (but this is very rare). Table 5 shows examples.

Next, when proverbs are figurative, different languages and cultures tend to use different types of concepts to draw parallels. We randomly sampled 100 figurative proverbs in English, Indonesian and Chinese, and classified contained concepts into one of the 5 categories, namely: Animals & Insects, Food, Cultural (including religious and spiritual entities, historical figures or names from the local culture), Nature (including metals, plants and other in-animated objects) and Others. Most of the time, a proverb only contains a single type of concept. However, when there are multiple types of concepts, we pick the dominant one (such as part of the object of the sentence). The distributions are in Figure 7. Here, we observe noticeable differences in distributions across different cultures. There are more concepts related to Animals & Insects and Nature in Indonesian than in other languages, which is probably due to Indonesia's unique geographical location.

### Additional Data Statistics

We include additional dataset statistics in Table 6. To calculate the average tokens in the context for Chinese, we take each character as a word.

### Interpreting the KDE Plot

For better comparison, we produce the Kernel Density Estimate (KDE) plot of 400 randomly sampled sentences in each language (2400 sentences in total), from a parallel multilingual dataset Li et al. (2023a) in Figure 8. As the original data is much larger (67k sentences per language), sub-sampled sentences are likely

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Lang** & Avg Tok in Context & Avg Turns \\ \hline English & 28.41 & 1.18 \\ Chinese & 31.30 & 1.14 \\ German & 27.91 & 1.12 \\ Indonesian & 25.35 & 1.15 \\ Russian & 31.25 & 1.47 \\ Bengali & 35.16 & 1.63 \\ \hline \hline  & & \\ \end{tabular}
\end{table}
Table 6: Additional dataset statistics: average number of tokens in the context, and average turns in the context.

\begin{table}
\begin{tabular}{c l} \hline \hline
**Lang** & **Animals \& Food** \\ \hline En & fox, wolf, lion \\  & bread, loaf, cookie \\ \hline De & luchs, wolf, lbwen, adler \\  & (lynnx, wolf, lion, eagle) \\  & brot, kuchen, schinken \\  & (bread, cake, ham) \\ \hline Ru & JLHCB, BOJIK, COORKA, COIOBJH \\  & (fox, wolf, magpie, nightingale) \\ xne6, kaphaai, IMPOT, kBac \\  & (bread, loaf, pie, kVass) \\ \hline Bn & [HTHT], [HT], [HT], [HT], [HT] \\  & (fox, elephant, tiger, snake) \\  & [HT], [HT], [HT] \\  & (rice, ghee, yoghurt) \\ \hline Zh & [HT], [HT], [HT] \\  & (dragon, tiger, phoenix) \\  & [HT] \\  & (rice) \\ \hline Id & buaya, singa, harimau, merak \\  & (croodile, lion, tiger, peacock) \\  & beras, sagu \\  & (rice, sago) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Selected food and animal concepts from the proverbs.

\begin{table}
\begin{tabular}{l p{284.5pt}} \hline \hline
* Kalau di hutan tak ada singa, \\ beruk rabun bisa menjadi raja \\  & (If there were no lions \\ in the forest, the short-sighted \\ monkey could become king.) \\ Zh
 

not translations of each other, but rather topic-coherent.

When sentences are topic-coherent, their embeddings overlap on top of each other and are inseparable (Figure 8). In comparison with the KDE plot of proverb embeddings (Figure 2), we can see the difference in proverbs across languages and cultures.

### Data Examples

We balance the labels in MAPS and we show example data for all languages in Table 7.

## Appendix B Templates

We use _Generate a very short 1-turn dialogue ends with "proverb" in language_ as the template to query GPT3.5 (gpt-3.5-turbo-0301) for the seed conversational data. The model does not strictly generate seed conversation with 1-turn. We also experimented with a translated template and did not observe quality improvements for our task.

Table 8 contains all of the templates used in our memorization experiments in the main section, with the proverb _no pain, no gain_ as the example. For instance, the last word of _no pain, no gain_ is removed. As the prompting results are highly variable based on the input patterns, we created five different prompt patterns. We take the union of memorized examples among 5 patterns as the memorization accuracy.

Table 9 is the template we used for our main inference experiment in the paper. As described in SS4, we perform experiments with the inference task on mLLMs. Figure 9 illustrates the experiment process for non-MLM models.

## Appendix C Cross-lingual Transfer Baselines

For completeness, we provide cross-lingual transfer baselines on MAPS. For cross-lingual transfer baselines, we re-split the English dataset into the train and test set (274/150 data points each) and evaluate on the original test set for other languages (i.e., same as zero-shot). We randomly sampled 20 data points from the training set as validation. We formulate the task as binary classification and experimented with XLM-R-Base (125M)/XLM-R-Large (355M)/XLM-R-XL (3.5B) and mT0-Base (580M)/mT0-Large (1.2B)/mT0-XL (3.7B).

The data input format is: _Context: [context]_

Figure 8: Visualizing embeddings with Kernel Density Estimate (KDE) when the sentences are sampled from a parallel dataset (topic coherent across languages).

Figure 7: Distributions of concepts categories in figurative proverbs.

Figure 9: An example illustrating how the inference is done with mLLMs (excluding MLMs).

 



#### Choices: A: [answer 1] B: {answer 2}.

We use AdamW optimizer (Loshchilov and Hutter, 2019) and conduct a hyperparameter search of the learning rate of [5e-5, 1e-4, 1e-5] and batch size of [8, 10, 16], trained for 30 epochs with bfloat16 precision, on a single A100 GPU.

The zero-shot transfer results are in Table 10 and averaged over 4 random seeds. The final hyperparameters for all models are [lr=1e-4, batch size=10], except for mT0-Large, which is [lr=1e

\begin{table}
\begin{tabular}{l l l l} \hline \hline
**Lang** & Proverb & Context \\ \hline  & & half a load is & Person 1: I didn’t get the promotion I wanted, but at least I got raise. Person 2: Of course, half a lost is better than none. & A: A raise is better monthing. \\ En & & better than none & at least I got raise. Person 2: Of course, half a lost is better than none. & B: A raise is worth nothing. \\  & & & do not have been used to be

4, batch size=8]. Following previous work, we also include results for the translate-test baselines [10] in Table 10.

Similar to our findings in the main paper, the model does not perform well on the task with models under a billion parameters. The performance gap between English and other languages remains significant.

## Appendix D Additional Results

### Further Details on the Memorization Experiment

#### d.1.1 Discussions

Following the defined criteria for identifying memory recall from LLMs in [23], a generalized prediction by LM always has alternatives based on the context to express similar meanings. Specifically, a phrase like _no pain, no___ would elicit multiple possible predictions. Without knowing the proverb, words like "painkiller", "medication" or "suffixee" are highly likely to occur at the last position based on the context. Similarly, a phrase such as _It is better to teach a man fishing than to give him___, similar concepts like "food", "Carps", or even "money" are very reasonable. Hence, a LLM that predicts the correct missing word (the single correct prediction) has likely memorized the data._

Certainly, based on the training method of LLMs, an alternative setup could be to mask words at various locations and have the model predict the missing words. However, such a method is more suitable to mT0 and XLM-R, which explicitly incorporate masked token predictions in pretraining with <extra_id_o> [17] tokens or <mask>tokens.

#### d.1.2 Additional Results

We include the results with randomly masked tokens (1 masked token per proverb) here for completeness for mT0 and XLM-R models. However, we use modified prompts in Table 11 due to our prior prompts were constructed for predicting the last word of the proverbs, and with <extra_id_o> or <mask> as the masked token for mT0 and XLM-R respectively. The results are in Figure 10. Several observations still persist, such as the disparity in memorization between languages, the low memorization rate of mT0 models, and the positive correlation between model size and memorization, etc.

### Memorized versus Not Memorized

We break down the results into memorized groups versus not memorized groups for the three best-performing models. We only show results when there are more than 50 proverbs in a group in Table 12 (which left us with English and Chinese). The benefit of memorization only shows for English, but not for Chinese.

### Additional Results for the Inference Task

#### d.3.1 Additional Prompt Templates

We experimented with 3 additional prompt templates in Table 13 to demonstrate the generality of our findings. Our experiments (Figure 11) show similar trends as in Figure 2(b) of the main section of our paper. We continue to observe that mT0 models perform the best for the inference task, the results improve as the model size increases, and memorization of the proverb is not an indication of performance on the inference task. However,

\begin{table}
\begin{tabular}{l l} \hline \hline
**Template** & \\ \hline
1. Proverb: no pain, no & \\
2. Complete this proverb: no pain, no & \\
3. Finish the proverb: no pain, no & \\
4. What’s the last word of this proverb: no pain, no & \\
5. What’s missing at the end of this proverb: & \\  & no pain, no & \\ \hline \hline \end{tabular}
\end{table}
Table 8: Memorization templates and the coloured portion is the template.

Figure 10: Memorization of proverbs in different languages when masking out words randomly.

 

we'd like to point out that our experiments do not assess the formal reasoning abilities (Huang and Chang, 2023, such as mathematical reasoning etc.) of mLLMs.

#### d.3.2 Additional Models

We include results of the additional following models, including Vicuna-V1.5 (Zheng et al., 2023, 7B, 13B), and Aya-101 (Ustun et al., 2024, 13B).

Similar to what we observe in the main paper, as the model increases in size, the performance on MAPS is better when asking the model to pick the correct answer. Aya-101 model's performance is on par with mT0 13B in most of the languages (both when asking positive and negative questions), but Aya-101 is noticeably better in Bengali.

### _'Negative' Questions_

We experimented with 4 additional versions of _'negative'_ questions/instructions (randomly created), without the use of the word **'not'**, they are:

* Which answer is contrary to what the person means by the proverb?
* Which answer is impossible as the interpretation of what the person means by the proverb?
* Pick the opposite answer to what the person means by the proverb.
* Pick the wrong answer to what the person means by the proverb.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
* & 50.60 & 50.75 & 49.23 & 51.47 & 49.85 & 50.38 \\ XLM-R-Large (355M) &
- & 50.00 & 50.00 & 50.00 & 49.85 & 50.00 & 49.97 \\ XLM-R-XL (3.5B) &
- & 50.90 & 51.20 & 52.31 & 49.85 & 51.47 & 51.15 \\ \hline mT0-Base (580M) &
- & 51.80 & 51.05 & 51.15 & 49.56 & 54.26 & 51.56 \\ mT0-Large (1.2B) &
- & 54.04 & 55.09 & 54.62 & 53.67 & 57.21 & 54.93 \\ mT0-XL (3.7B) &
- & 67.96 & 62.72 & 63.46 & 57.92 & 58.68 & 62.15 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Zero-shot cross-lingual transfer and translate-test baselines. Cross-lingual averages are calculated over all languages except English.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Templates** \\ \hline
1. Fill the missing token: no ‘mask’, no gain Answer: \\
2. What is the missing word in this proverb: no ‘mask’, no gainAnswer: \\
3. What is the masked word in this proverb: no ‘mask’, no gainAnswer: \\ \hline \hline  & \multicolumn{1}{c}{} & & \\ \end{tabular}
\end{table}
Table 11: Additional memorization templates, adjusted the task description to fit the experiment with random words removed from the proverb. The coloured portion is the template. <extra_id_0> is used for mT0 and ¡mask’ is used for XLM-R.

 



## References

* [1] A. B. Krizhevsky, S. M. Girshick, and J. Malik. Learning from incomplete data: a survey of machine learning algorithms. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 235-240, 2016.
* [2] R. G. Hinton, D. L. Hariharan, and E. Hinton. Imagenet classification with deep convolutional neural networks. _arXiv preprint arXiv:1907.08958v1_, 2019.
* [3] C. Szegedy et al., "Batch normalization for stochastic optimization. ICLR," in _NIPS Workshop on Machine Learning_, volume 1, 2015.
* [4] Y. Bengio, V. LeCun, J. Cortes, and J. Dean. Dropout: an imperative question answering system. _CoRR_, abs/1412.1118v1, 2014.
* [5] P. Baud, N. Srivastava, and A. Zisserman. Deep residual learning for image recognition. In _CVPR 2015 conference on Computer Vision and Pattern Recognition_, pages 235-248, 2015.

[MISSING_PAGE_POST]



We use the same prompt template to evaluate the models. The results are in Figure 13. While our work focuses on reasoning with cultural common grounds, this shows the importance and urgent need to improve the model's ability to answer 'negative' questions.

We speculate this is due to the biases in training data. Often, users seek the correct solution to solve problems online (which we refer to as positive biases) rather than the wrong solution. Hence, when using web corpora as training data for LLMs, such positive biases will propagate to the behaviour of LLMs. To demonstrate this further, we conducted an additional experiment _without_ asking a question in the prompt on BLOOMZ, mT0 and LLaMA-2. In an ideal situation, a good model should score nearly random when no question is asked (analogously to human confusion when data is given, but no question is asked). From Figure 14, all LLMs can score above random for multiple languages, which indicates all models _failed_. This failure mode further hints at the inability of mLLMs to handle negative questions maybe due to the nature of the training data.

### Culture Gaps

In addition to the results in SS5.3, we follow the same procedure and perform the experiment with mT0 for En-Zh translated data. We observe similar results in Figure 15, and the culture gap for En-Zh is 5.33.

### Additional Results on LLaMA-2 with Translations

Since LLaMA-2 13B is one of the recent state-of-the-art (English officially) models, we further conducted a zero-shot experiment by translating all data from other languages into English. We used Google Translate for translation and reported the results in Table 14. From the Table, we can see significant performance gaps (to English). It is also interesting to see the gaps increase as the corresponding geographical location of the language moves further away from English. While we consider this gap to be a combination of the language gap and the defined culture gap, a future interesting direction is to closely examine the culture gap in cross-cultural communications and how this is related to the internal representations in LLMs.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Additional Templates** & \\ \hline Temp1: & \\ Proverb: & <proverb> \\ Context: & <contextx \\ Choices: A: & <answer I\textgreater{} B: & *answer 2\textgreater{} \\ Question: What does the person mean by the & \\ Answer: & \\ \hline Temp2: & \\ Question: What is a probable interpretation of this & \\ Proverb: & <proverb> \\ Context: & <contextx \\ Choices: A: & <answer I\textgreater{} B: & *answer 2\textgreater{} \\ Please choose between A and B. & \\ Answer: & \\ \hline Temp3: & \\ Question: How would one interpret this proverb & \\ given the context? & \\ Proverb: & <proverb> \\ Context: & <contextx \\ Choices: A: & <answer I\textgreater{} B: & *answer 2\textgreater{} \\ Answer: & \\ \hline \hline \end{tabular}
\end{table}
Table 13: Additional prompt templates. The coloured portion is the template.

Figure 12: Performance of additional mLLMs on the proposed **MAPS** dataset. The number of parameters is in millions.

 

Figure 14: Performance of mLLMs on the proposed MAPS dataset when only the proverb, context and choices are provided, but without a question. Ideally, all models should score around random guessing. The number of parameters is in billions for LLaMA-2 and in millions for all other models.

Figure 13: Performance of mLLMs on the proposed MAPS dataset when asking the model a ‘negative’ question. The number of parameters is in billions for LLaMA-2 and in millions for all other models.

 



### Few-shot (In-context) Evaluation

For completeness, we also provide evaluation results with few-shot demonstrations. We perform 2-shot and 5-shot experiments by randomly sampling 5 sets of n-shot demonstrations from the few-shot training set (using the same template as zero-shot evaluation by concatenation). We evaluate BLOOMZ 7.1B, mT0-XXL 13B and LLaMA-2 13B models, and Table 15 shows the results.

From Table 15, we do not observe any improvements with few-shot demonstrations compared to zero-shot. In fact, model performances consistently degrade with more demonstrations. Since our task has a very long context that may affect the n-shot performance. Nonetheless, this degradation has been observed recently in other work such as in Li et al. (2023b); Koto et al. (2023) with few-shot evaluations.

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Lang** & **Ori. Lang** & **MT** & \(\Delta_{En}\) \\ \hline En & 78.68 & — & — \\ De & 68.26 & 73.35 & 5.33 \\ Ru & 62.82 & 71.02 & 7.66 \\ Id & 57.47 & 69.79 & 8.89 \\ Bn & 49.11 & 61.76 & 16.92 \\ Zh & 53.59 & 54.19 & 24.49 \\ \hline \hline \end{tabular}
\end{table}
Table 14: Results of machine-translated data with LLaMA-2 13B. \(\Delta_{En}\) is the resulting gap to the model’s performance on English data.

Figure 15: Performance gap between machine-translated, human-translated English data and results in the source language (En), and target language (Zh).





\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Model** & **En** & **De** & **Zh** & **Ru** & **Id** & **Bn** & **Cross-lingual Avg** \\ \hline BLOOMZ 7.1B 2-shot & 59.49 & 61.55 & 56.59 & 53.77 & 51.53 & 50.00 & 52.65 \\ BLOOMZ 7.1B 5-shot & 51.57 & 52.39 & 50.85 & 50.35 & 50.25 & 50.52 & 50.30 \\ \hline mT0-XXL 13B 2-shot & 78.37 & 72.63 & 76.95 & 78.74 & 74.87 & 63.82 & 76.81 \\ mT0-XXL 13B 5-shot & 68.48 & 67.90 & 70.38 & 71.50 & 67.64 & 60.00 & 69.57 \\ \hline LLaMA-2 13B 2-shot & 74.87 & 56.52 & 55.42 & 60.77 & 56.76 & 51.00 & 58.77 \\ LLaMA-2 13B 5-shot & 64.16 & 52.69 & 54.89 & 55.56 & 52.71 & 50.17 & 54.14 \\ \hline \hline \end{tabular}
\end{table}
Table 15: Few-shot evaluation results from MAPS. Cross-lingual averages are calculated over all languages except English.