Multi-lingual and Multi-cultural Figurative Language Understanding
Year: 2023

Abstract:
Figurative language permeates human communication, but at the same time is relatively understudied in NLP. Datasets have been created in English to accelerate progress towards measuring and improving figurative language processing in language models (LMs). However, the use of figurative language is an expression of our cultural and societal experiences, making it difficult for these phrases to be universally applicable. In this work, we create a figurative language inference dataset, MABL, for seven diverse languages associated with a variety of cultures: Hindi, Indonesian, Javanese, Kannada, Sundanese, Swahili and Yoruba. Our dataset reveals that each language relies on cultural and regional concepts for figurative expressions, with the highest overlap between languages originating from the same region. We assess multilingual LMs’ abilities to interpret figurative language in zeroshot and few-shot settings. All languages exhibit a significant deficiency compared to English, with variations in performance reflecting the availability of pre-training and fine-tuning data, emphasizing the need for LMs to be exposed to a broader range of linguistic and cultural variation during training. 1

heading: 1 Introduction
text: When you are feeling happy, do you think that you are “warm” or “cold”? If you are a monolingual English speaker, you will likely answer “warm”, and use expressions like “this really warmed my heart”. However, if you are a native Hindi speaker, you may answer “cold”, and use expressions like दल को ठंडक पढ़ना (“coldness spreads in one’s heart” ) (Sharma, 2017). Linguistic communication often involves figurative (i.e., non-literal) language (Shutova, 2011; Fussell and Moss, 2008; ∗ These authors contributed equally. 1Data and code is released at https://github.com/ simran-khanuja/Multilingual-Fig-QA
Lakoff and Johnson, 1981), which is laden with implicit cultural references and judgements that vary cross-culturally. Differences in figurative expressions used in different languages may be due to cultural values, history, or any number of other factors that vary across where the languages are spoken.2 Understanding figurative language therefore relies on understanding what concepts or objects are considered culturally significant, as well as their sentiment in that culture. Better understanding of figurative language would benefit tasks such as hate speech detection or sentiment classification (ElSherief et al., 2021; van Aken et al., 2018). However, state-ofthe-art language models have been shown to frequently misinterpret both novel figurative expressions and conventionalized idioms, indicating the need for improved methods (Dankers et al., 2022;
2The Hindi example is most likely attributable to climatic conditions, as cold may be seen as comparatively more positive in an area where extreme heat is more common (Sharma, 2017)
8269
Liu et al., 2022). Most empirical results probing language models’ abilities with respect to figurative language have been based on data in English, meaning there is a comparative lack of resources and study in other languages (Chakrabarty et al., 2022; Liu et al., 2022; Pedinotti et al., 2021a). We find English figurative language datasets may not have cultural relevance for other languages (§2). This is a general challenge in NLP, as assumptions of common knowledge and important topics to talk about vary from culture to culture (Hershcovich et al., 2022). In order to better train multilingual models to interpret figurative language, as well as to understand linguistic variation in figurative expressions, we construct a multilingual dataset, MABL (Metaphors Across Borders and Languages), of 6,366 figurative language expressions in seven languages (§3). Examples are shown in Table 1. We use the dataset to conduct a systematic analysis of figurative language patterns across languages and how well they are captured by current multilingual models (§4). We find that figurative language is often very culturally-specific, and makes reference to important entities within a culture, such as food, mythology, famous people, or plants and animals native to specific regions. We benchmark multilingual model performance (§5) and analyze model failures (§6), finding that zero-shot performance of multilingual models is relatively poor, especially for lower-resource languages. According to (Liu et al., 2021), main factors which poses challenges on the performance in such cases are cross-lingual transfer and concept shift across languages. However, we observe that concept shift seems to play a larger role due to culturally specific examples. Adding a few examples in the target language can improve performance of larger models, but this is more beneficial for lower-resource languages. This highlights the importance of including culturally relevant training data, particularly data that highlights not just the existence of a concept, but also how people view that concept within that culture.

heading: 2 Linguistic and Cultural Biases of Existing Figurative Language Datasets
text: To confirm the importance of building a multilingual, multi-cultural figurative language dataset, we first performed a pilot study to examine the feasibility of instead translating an existing figurative language dataset, Fig-QA (Liu et al., 2022), from
English into other languages. While there are wellknown problems with using translation to create multilingual datasets for tasks such as QA (Clark et al., 2020), it is still worth examining these issues in the context of figurative language in particular. We used the Google Translate Python API to translate the development set into languages that the authors of this paper understood.3 These were French, Japanese, and Hindi. Each annotator annotated 100 examples for both correctness (whether or not the translation was accurate), and cultural relevance (whether or not the expression was one that would make sense to a native speaker from the culture where the language is predominant). As seen in Table 2, the number of incorrect examples is large, particularly for Hindi and Japanese. This is mainly due to expressions that don’t translate directly (such as a “sharp” conversation in English). Culturally irrelevant examples are due to implicitly assumed knowledge. For instance, a crowdworker from the US generated the example “it’s as classic as pancakes for breakfast” with the meaning “it’s very classic”. However, most people from Japan would not see pancakes as a traditional breakfast, and the meaning “it’s not classic” would be more appropriate. The shift in topics discussed in cultures associated with different languages can be captured by native speakers familiar with that culture, motivating our collection of natural figurative language examples from native speakers.

heading: 3 The MABL Dataset

heading: 3.1 Language Selection
text: We choose the following seven languages: Hindi (hi), Yoruba (yo), Kannada (kn), Sundanese (su), Swahili (sw), Indonesian (id), and Javanese (jv). The factors we considered while choosing these languages are as follows : i) We aimed to include a range of languages representing the different classes in the resourcebased taxonomy of languages, proposed by Joshi et al. (2020), subject to annotator availability. 3https://pypi.org/project/googletrans/
ii) We chose languages with a sizeable speaker population as shown in Table 5. iii)Our languages come from 5 typologically diverse language families spoken in 4 different countries, which allows us to include a wide range of linguistic and cultural diversity in our data. Details about the characteristics of each language in terms of available training data and number of speakers can be found in Table 5. Additional information on linguistic properties of these languages can be found in Appendix A.

heading: 3.2 Dataset Collection
text: To create culturally relevant examples, we crowdsourced sample collection to two or more native speakers in the seven languages. Theworkers were asked to generate pairedmetaphors that beganwith the same words, but had different meanings, as well as the literal interpretations of both phrases.
Workers were not discouraged from generating novel metaphors, but with the caveat that any examples should be easily understood by native speakers of that language, e.g., “it’s as classic as pancakes for breakfast” would not be valid if pancakes are not a breakfast food in the country in which that language is spoken.
Instructions given to annotators can be found in Appendix B. After collection, each sample was validated by a separate set of workers who were fluent in that language. Any examples that were incoherent, offensive, or did not follow the format were rejected. The number of samples collected per language can be seen in Table 3. Examples of collected data can be seen in Table 1. We note that because of the limited number of samples in each language, we view the samples collected as a test set for each language, meaning there is no explicit training set included with this release.

heading: 4 Dataset Analysis

heading: 4.1 Concepts expressed
text: In the structure mapping theory of metaphor, figurative language involves a source and target concept, and a comparison is made linking some features of the two (Gentner, 1983). Following Liu et al. (2022), we refer to the source as the “subject” and target as “object” .4 We expect objects referenced to be quite differently cross-culturally. We confirm this by translating sentences from our dataset into English, then parsing to find objects. The number of unique concepts per language, including examples, is listed in Appendix C. This may overestimate the number of unique concepts, as some concepts may be closely related (e.g., “seasonal rain” vs. “rainy season”). Despite this, we are able to identify many culturally specific concepts in these sentences, such as specific foods (hi: samosa, hi: sweet gulkand, id: durian, id: rambutan), religious figures (kn: buddha’s smile, sw: king soloman), or references to popular culture (id: shinchan, yo: aníkúlápó movie, en: washington post reporter). We observe that, excluding pronouns, only 6 objects are present in all languages. These are {“sky”, “ant”, “ocean”, “fire”, “sun”, “day”}. Of course, variations of all these concepts and other generic concepts may exist, since we only deduplicated objects up to lemmatization, but this small set may indicate that languages tend to vary widely in figurative expressions. AppendixD indicates the Jaccard similarity between objects in each language, which is an intuitive measure of set similarity. The equation is also given below for sets of objects from language A (𝐿𝐴) and langugage B (𝐿𝐵).
𝐽(𝐿𝐴, 𝐿𝐵) = |𝐿𝐴 ∩ 𝐿𝐵| |𝐿𝐴 ∪ 𝐿𝐵|
(1)
The most similar language based on concepts present is highlighted in Table 4. Languages from the same region tend to group together. The set of concepts in English is actually most similar to Swahili.5 Upon inspection, there were many general terms related to nature, as well as many references to Christianity in the Swahili data, which may explain the similarity to English.6
4This terminology may be confusable with subject and object in linguistics, but was used because the source and target tend to appear in these linguistic positions in a sentence. 5There are no particularly closely related languages to English in our dataset 6Authors of this paper examined unique concepts expressed in English, Swahili, and Kannada. Swahili sentences had

heading: 4.2 Commonsense Categories
text: We follow the commonsense categories defined in Liu et al. (2022) to categorize knowledge needed to understand each sentence: physical object knowledge (obj), knowledge about visual scenes (vis), social knowledge about how humans generally behave (soc), or more specific cultural knowledge (cul). The same sentence can require multiple types of knowledge. Table 6 shows the prevalence of each type of commonsense knowledge as documented by annotators. Social and object knowledge are the most dominant types required, with Yoruba having an especially high prevalence of social examples. Not many examples were marked as cultural. This may be due to differences in what annotators viewed as cultural knowledge: some knowledge may be considered to fall under the object or social category by annotators, but these same examples may seem culturally specific to people residing in the United States because the objects referenced are not necessarily relevant to English speakers in the US.
18/481 Christianity related concepts, while English had 13/954. Kannada did not have any Christianity related concepts but rather concepts related to Hinduism.

heading: 4.3 Cross-lingual concept distribution
text: To better understand the linguistic and cultural distribution of examples, we extract sentence-level representations from two models: i) XLM-Rlarge (Conneau et al., 2019), our best performing baseline model; and ii) LaBSE (Feng et al., 2020), a language-agnostic sentence embedding model, optimized for cross-lingual retrieval. We observed that XLM-R clusters by language, whereas LaBSE clusters sentences from multiple languages together, based on conceptual similarity (as shown in Figure 2). Since LaBSE is optimized for crosslingual sentence similarity, we chose the latter to conduct further analysis. First, we probe different edges of the cluster and observe concepts along each edge, as visualized in Figure 1. For each concept, we observe sentences from various languages clustering together. Further, these sentences portray cultural traits pertaining to each language. For example, rice is commonly mentioned in languages from Indonesia, given that it is a staple food product there.8 Other examples include sentences in Hindi such as This house is as old as a diamond (diamonds have a significant historical background in India) or Your house is worth lakhs (lakh is an Indian English term).9 To qualitatively study cultural references, we further analyse metaphors belonging to universal concepts such as food, weather/season, and friendship, searching for sentences containing these keywords.10 We obtain 230 sentences containing food, 111 sentences containing weather/season and 307 sentences containing friend. A few examples are as shown in Table 7. We observe multiple regional and cultural references, which may not be under-
8https://www.indonesia-investments.com/business/ commodities/rice/item183 9https://en.wikipedia.org/wiki/Indian_numbering_ system 10We do a regex search over the word and its translation in each language to obtain sentences from all languages in the concept, using https://projector.tensorflow.org/
standable by non-native speakers. For example, annotators make references to the weather/season with Peacock and frying fish on asphalt which are innate comparisons in su. With reference to food, Indian food commonly uses Neem and Tamarind as referenced by metaphors in kn and hi. Neem is a bitter medicinal herb and Tamarind is used to add sourness to food. Finally, we see references to mythological and fictional characters across friendship metaphors, where annotators draw from their attributes to describe friendships.

heading: 5 Evaluation and Results

heading: 5.1 Zero-shot

heading: 5.1.1 Zero-shot evaluation
text: Here, we simply fine-tune the Multilingual Pretrained LanguageModels (MPLMs) on the English labelled data and evaluate on all target languages. This was performed in the standard format of inputting each example as [CLS] [sentence] [SEP] [meaning1] [SEP] [meaning2] and using a linear layer on the [CLS] token to classify the answer.

heading: 5.1.2 Zero-shot transfer results
text: We present zero-shot evaluation results in Table 8, noting that there can be two contributors to the gap in performance in these seven languages as compared to English. First, since our fine-tuning language is English, there can be a drop in performance simply due to cross-lingual transfer. Second, there is a concept shift in these metaphors, as evidenced by our analysis in Section 4. To discern the contribution of both, we machine-translate the target test sets to en (we refer to this as translatetest). The difference between translate-test and zero-shot, can be thought of as the cross-lingual transfer gap, while the rest of the difference between translate-test and en test performance can be attributed to the concept shift. Due to possible MT errors, the results here represent upper bounds for concept shift and cross-lingual shift, which is
further discussed in Section 6.1.
The concept shift gap is generally greater than the cross-lingual gap. As reported in Table 8, the concept shift gap is greater than the cross-lingual transfer gap for all languages except Swahili, across all models. This result for sw corroborates our findings in Section 4, where we observe that en shares the greatest proportion of object concepts with sw. Given Swahili’s extremely low-representation in MPLMs (Table 5), and its high concept overlap with English, we cover most of the gap by simply translating sw to en. For Indonesian (id), we observe that zero-shot performance itself is close to en performance (83.6%) for XLM-R, since id is well-represented in this model (Table 5). Hence, translating to en does not help, and the model needs to be competent in better understanding the cultural references specific to id. In mBERT however, id is poorly represented, and translating to en does help improve performance.
Performance increases as model and training data size increase, but moreso for higher resource languages. The smallest model examined, mBERT, has relatively poor performance for all languages, as all languages have < 60% accuracy. Hindi and Indonesian, the two highestresource languages in our dataset, show a high gain in performance when using a larger model, increasing to 67.58% and 78.09% accuracy respectively. This is especially true for Indonesian, which has a relatively high amount of training data as shown in Table 5. However, lower resource languages tend to show a more modest gain in performance.

heading: 5.2 Few-shot

heading: 5.2.1 Few-shot evaluation
text: While it is common to fine-tune MPLMs on English, given its widespread use and availability, several past works have shown how this is suboptimal (Lin et al., 2019; Debnath et al., 2021) and choosing optimal transfer languages is an important research question in itself (Dhamecha et al., 2021). While the design of an ideal allocation of annotation resources is still unknown, Lauscher et al. (2020) demonstrate the effectiveness of investing in few-shot (5-10) in-language task-specific examples, which provides vast improvements over the zero-shot setup. We include between 2-50 labelled pairs of sentences from each target language, in addition to the English labelled data, for fine-tuning themodel. Training details for all models can be found in Appendix E.

heading: 5.2.2 Few-shot results
text: Figure 3 presents the effects of few-shot transfer for each language. Generally, the performance gain is modest. This aligns with results from Lauscher et al. (2020), who found that performance gains were quite small on XNLI. As our task is also an NLI task, we may expect similar improvements. However, we find collecting some cultural examples could disproportionately help low-resource languages. Augmenting with few examples usually does not help muchWe observed that with a few exceptions, the increase in accuracy on the test set gained was small (< 1%). This is likely because of the diversity of facts needed in order to improve performance. As noted in Section 4.1 and Table 1, this dataset contains many unique cultural references that do not repeat, limiting the utility of seeing a few examples. Lower-resource languages benefit more greatly from augmentation However, there are a few exceptions to this trend. In particular, adding 50 paired Kannada examples to XLM-Rlarge improved performance by 3.83%. Swahili also improves by 1.10% with 50 additional examples for XLM-Rbase, and Sundanese improves by 2.33% with 50 examples for mBERTbase.

heading: 5.3 Evaluation of Large Language Models
text: In addition to the three MPLMs we examine in detail, we also examine the zero-shot performance of large pretrained language models. We choose to
examine GPT-3 (text-davinci-003) and BLOOM176B. As these models are autoregressive rather than masked models, we follow the standard procedure of prediction via choosing the answer with a higher predicted probability (Jiang et al., 2021). The performance of GPT-3 is not very good on most languages when tested zero-shot, but we note that it has a reasonable zero-shot performance on the English development set (74.86%), higher than the reported results of text-davinci-002. (Liu et al., 2022). There is a high concept shift gap as with the other models but also a comparatively higher cross-lingual gap as this model is much stronger in English.

heading: 6 Error Analysis

heading: 6.1 Effect of English MT
text: As noted in Section 5.1, there are two major factors that can cause difficulty in cross-lingual trans-
fer: language shift and concept shift. We try to approximate these effects by translating the test set in each language to English. However, this is done with machine translation, so there may be errors. Despite this, translation can still benefit the model if the original language was low-resource. We can divide the model performance into four cases as shown in Table 9.
First, there are easy examples (53%) which are answered correctly in both the original language and translated versions. Next there are linguisti-
cally challenging examples (19%) which are originally answered incorrectly, but switch to being answered correctly after being translated to English.11 There are difficult-to-translate or incorrectly translated examples (15%). It’s likely that these errors can be completely eliminated with a careful enough translation. Lastly, there are hard examples (12%) which are answered incorrectly before and after being translated. These contain many inherently difficult examples, and examples with specific cultural terms. Examples of each type can be found in Appendix G.

heading: 6.2 Cultural Examples
text: We examine the accuracy of XLM-Rlarge on the commonsense categories in Section 4.2. Overall, there is a small difference in accuracy between cultural examples and the overall accuracy, with overall accuracy at 63.99% and accuracy on cultural examples at 61.68%. Accuracy for all languages can be found in Appendix H. This is a preliminary analysis, but may indicate that references to explicit named entities may not be the only issue for the model with regard to culture.

heading: 7 Related Work

heading: 7.1 Figurative Language
text: English-centric: Most previous inference tasks on figurative language have been in English (Chakrabarty et al., 2022; Liu et al., 2022; Pedinotti et al., 2021a). Further, research on figurative language in English centers around training models to detect the presence of metaphors in text (Leong et al., 2020; Stowe and Palmer, 2018; 11Linguistically challenging here means that the language is more challenging for an LM to perform well in, not that the linguistic structure is very difficult.
Tsvetkov et al., 2014). This is done using datasets primarily consisting of idioms and conventionalized metaphors. However, recognizing common metaphorical phrases may not truly test a model’s ability to interpret figurative language. There is limited research on understanding metaphors, which mostly looks at linking metaphorical phrases to their literal meanings through paraphrase detection (Bizzoni and Lappin, 2018) or generation (Shutova, 2010; Mao et al., 2018). Some studies investigate LMs’ ability to understand metaphors, but they do not consider the fact that metaphors have different meanings based on context (Pedinotti et al., 2021b; Aghazadeh et al., 2022). Most recently, Liu et al. (2022) released a dataset which requires a model to infer the correct meaning of metaphor, rather than simply identifying or paraphrasing it, hence calling to test deeper semantic understanding.
Extension to Multilingual: Research in corpus linguistics (Díaz-Vera and Caballero, 2013; Kövecses, 2004; Charteris-Black and Ennis, 2001) suggests that there significant variation in metaphorical language between cultures. There has been some work in detecting metaphors in multilingual text (Tsvetkov et al., 2013; Shutova et al., 2017). These works have focused on three relatively highresource languages: English, Russian and Spanish. Both focused on cross-lingual techniques to identify metaphors from newspapers and dictionaries. Hence, there hasn’t been any large-scale multilingual dataset of figurative language constructed, which would allow one to study cultural variations across metaphors. We fill this gap with the release of our dataset.

heading: 8 Conclusion
text: Despite being relativelywidespread, figurative language is relatively under-studied in NLP. This is especially true for non-English languages. To enable progress on figurative language processing, we create MABL, a figurative inference dataset across seven languages. We find considerable variation in figurative language use across languages, particularly in the unique objects that people invoke in their comparisons, spanning differences in food, mythology and religion, and famous figures or events. This variation is likely due to differences in cultural common-ground between the countries in which these languages are spoken. We find that multilingual models have considerable room for improvement on this task, and cross-cultural shift may play a significant role in the performance degradation from English. We encourage the NLP community to further examine the role that culture plays in language, and note that figurative language can be used as a testbed to examine crosslinguistic and cross-cultural variations.

heading: 9 Limitations
text: First, despite our pursuit of attempting to understand figurative language use across cultures, we have barely scratched the surface in terms of diverse representation. Due to limited scope, budget, and resources, we collect data from 2-3 annotators per language, for seven languages. Further, culture can vary greatly within a language (Hershcovich et al., 2022). Therefore, until we can represent all of the worlds’ people and their languages, there will always be room for improvement. We also acknowledge that the syntax captured in the dataset may not be the most diverse, as many examples follow the template “<X> is like <Y>”. However, we create these simpler examples as a first step, since extension to more complex and naturalistic language can be included in future work. Second, to analyse concept shift, we machine translate test sets into English. However, these translations can be erroneous to varying degrees, which may have resulted in an over-estimation of error attribution to concept shift. This could not be avoided however, due to limited resources of obtaining human translations. Third, English may not be the best language to transfer from in zero-shot evaluation of multilingual models. While we were constrained by training data availability, past works have shown that
machine-translating train sets can help, an avenue we haven’t explored here. Even though we experiment with few-shot evaluation, there may exist an optimal combination of source languages which best transfer to our target languages. Fourth, the English authors recognized culturespecific terms that were not marked as cultural by annotators in the commonsense categorization across all languages. This may be because annotators, being mostly familiar with their own cultures, attributed culturally specific facts and terms as being common sense. Likewise, the Englishspeaking participants may have viewed a separate set of facts as common sense which would not be agreed upon by people from a different culture. It is thus difficult to disentangle common sense and culture in many cases.

heading: A Selected Languages
text: Table 10 contains additional information on languages included in the dataset. Information on languages was collected from the World Atlas of Language Structures (WALS) and Glottolog 4.7 (Hammarström et al., 2022; Dryer andHaspelmath, 2013).
B Instructions for Annotators
In Liu et al. (2022), workers are prompted with random words taken from English metaphorical frames in Lakoff and Johnson (1981). However, as these metaphorical frames are not readily available in other languages, and we did not want to bias workers toward concepts that are only relevant in English, we chose to omit this prompt and have workers generate sentences freely, while encouraging them to emphasize aspects of their culture. Annotators were paid according to their proposed hourly range ($25/hour on average, all above $15/hr). Validators were paid $15/hr. This study was approved by our IRB. No identifying information was collected. Note that this is the English version of the instructions, as instructions were machine-translated to each target language and corrected by native speakers.
Your task is to generate pairs of sentences with opposite or very different meanings, both of which contain metaphors. You can feel free to incorporate creativity into the metaphors, but also make sure that they’re something that could be understood by the speakers of the language that you are generating metaphors for, e.g., “this is as classic as pancakes for breakfast” to mean “this is classic” wouldn’t make sense for a culture in which pancakes aren’t traditionally eaten for breakfast.
You can do this by thinking of a metaphor that conveys a certain meaning, and replacing the metaphorical phrase with another metaphorical phrase of the same type (for instance, noun phrases, verb phrases or adjective phrases) that conveys the opposite meaning.
Here are some examples of metaphors to give you an idea of what we’re looking for: Please write both the metaphor and its meaning for each sentence.
1. The surgeon is (a lumberjack/a ballet dancer). 2. The movie has the depth of (a wading pool/the grand canyon) 3. Her commitment to the cause was as sturdy as (plywood/oak)
If you’re stuck, a general template you can use is <SUBJECT> is <metaphor 1>/<metaphor 2>.

heading: C Unique Concepts in Different Languages
text: Table 11 displays the number of unique concepts and some examples in each language after basic deduplication (lemmatization and casing).

heading: D Jaccard Similarity between Concepts
text: Table 12 contains Jaccard similarities for sets of concepts found in each language. Language pairs with the highest similarity (row-wise) are bolded.

heading: E Training Details
text: A hyperparameter grid search was conducted over values: epochs ∈ {10, 20, 30}, lr ∈ {2 × 10−4, 5 × 10−4, 2 × 10−5, 5 × 10−5, 2 × 10−6, 5 × 10−6}, and batch size ∈ {32, 64}. XLM-Rlarge was trained for 20 epochs with a learning rate of 5 × 10−6 and a batch size of 32. XLM-Rlarge was trained for 30 epochs with a learning rate of 2 × 10−5 and a batch size of 64. mBERTbase was trained for 30 epochs with a learning rate of 5 × 10−5 and a batch size of 64. An A6000 GPU was used for each model. Each training run takes on the order of a few minutes. Most seeds lead to a near-random performance on the English dev set, while a small minority of seeds lead to non-random performance. We took the top 5 seeds from 1-100 found in terms of English dev set performance in order to avoid including results from degenerate seeds. We did not experiment with trying to optimize the hyperparameters for the experiments in Section 5.2.2 but rather used the same ones found previously. This may account for some settings leading to lower performance.

heading: F Few-shot Full Results
text: Table 13 outlines the effect of adding 𝑘 ∈ {2, ..., 50} examples in each target language.

heading: G Four-Quadrant Examples
text: G.0.1 Easy
• नद का पानी क्रस्टल क तरह साफ है।/the water of the river is as clear as crystal
• Ia berjalan layaknya siput/he walks like a snail
• Inú yàrá ìdánwò nàá palọ́lọ́ bí i itẹ́ òkú/inside the exam room was a dead silence
• Vijana ndio taifa la kesho/youth is the nation of tomorrow
• Diamenjalani hidup bak singa di kebun binatang/he lives life like a lion in the zoo
G.0.2 Challenge - linguistic • Àgbẹ̀ náà pa gbogbo ọmọ tí igi nàá bí lánàa ́/the farmer killed all the children that the tree gave birth to yesterday
• Penzi lao ni kama moto wa kibatari kwenye upepo/their love is like fire in the wind
• Kadang jelema teh bisa ipis kulit bengeut/sometimes people can have thin skin
• Si eta kuliah siga nu teu kantos bobo/that college guy looks like he never sleeps
• ಅವರು ನೀಡಿದ್ದ ನೀರು ಸಮುದ್ರದ ನೀರಿನಂತೆ ಉಪ್ಪಾಗಿತ್ತು/the water they gave was as salty as sea water
G.0.3 Challenge - translation • hirup teh kudu boga kaditu kadieu/life must have here and there
• लड़क का क्तत्व गुलाब जामुन क तरह मीठा था/the girl’s personality was as sweet as Gulab Jamun
• Ìṣọ̀lá má ń tún ilé rẹ̀ ṣe ní gbogbo nìgba ̀/honor does not repair his house all the time
• Nek gawewedang kopi Painem kaya disoki suruh/if you make a Painem coffee drink, it’s like being told
• Bapak tirine sifate kaya Gatot Kaca/his stepfather is like Gatot Kaca
G.0.4 Hard • का लदास भारत के शे ख्चली हैं।/Kalidas is Shekhchili of India
• उसके मन का मैल मट क तरह छलनी से नकल गया।/The filth of his mind was removed from the sieve like soil
• Wajahku dan adikku ibarat pinang di belah dua/My face and my sister are like areca nuts split in half.
• Hari ini cuacanya seperti berada di di puncak gunungBromo/Today theweather is like being at the top of Mount Bromo
• Doni karo Yanti pancen kaya Rahwana Sinta ing pewayangan/Doni and Yanti are really like Ravana Sinta in a puppet show

heading: H Accuracy on Annotated Commonsense Categories
text: Table 14 shows the accuracy on commonsense categories across all languages for XLM-Rlarge. Note that Yoruba is not included due to XLM-Rlarge not being trained on this language.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations section on page 9
7 A2. Did you discuss any potential risks of your work? We believe that there are no significant risks posed by this work.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and intro page 2
3 A4. Have you used AI writing assistants when working on this paper? ChatGPT was used for summarizing/shortening areas of the paper
B 3 Did you use or create scientific artifacts? Section 3
3 B1. Did you cite the creators of artifacts you used? throughout paper
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No, however we use only publically released models and data and cite all artifacts used.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? There were no specific individuals named (beyond references to famous individuals such as the president)
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 3/4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3
C 3 Did you run computational experiments? Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix E
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix E
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? In table 8
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix B
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 3
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? We did not directly include this in the instructions, but it was explained through text/in person if annotators were interested.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? We have submitted a request for exemption at this time due to minimal risk posed, but the IRB has not gotten back to us yet.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Table 5, appendix A

heading: Future Work
text: However, we create these simpler examples as a first step, since extension to more complex and naturalistic language can be included in future work. Second, to analyse concept shift, we machine translate test sets into English. However, these translations can be erroneous to varying degrees, which may have resulted in an over-estimation of error attribution to concept shift. This could not be avoided however, due to limited resources of obtaining human translations. Third, English may not be the best language to transfer from in zero-shot evaluation of multilingual models. While we were constrained by training data availability, past works have shown that
machine-translating train sets can help, an avenue we haven’t explored here. Even though we experiment with few-shot evaluation, there may exist an optimal combination of source languages which best transfer to our target languages. Fourth, the English authors recognized culturespecific terms that were not marked as cultural by annotators in the commonsense categorization across all languages. This may be because annotators, being mostly familiar with their own cultures, attributed culturally specific facts and terms as being common sense. Likewise, the Englishspeaking participants may have viewed a separate set of facts as common sense which would not be agreed upon by people from a different culture. It is thus difficult to disentangle common sense and culture in many cases