How Well Do LLMs Identify Cultural Unity in Diversity?
Authors: Jialin Li
Junli Wang
Junjie Hu
Ming Jiang
Venue: COLM (2024)
Keywords: Cultural Unity, LLM Evaluation, Cross-cultural Association, Benchmark
TL;DR: Moving beyond cultural diversity, we introduce a benchmark dataset CUNIT for evaluating decoder-only LLMs in understanding the cultural unity of concepts.

Abstract: Much work on the cultural awareness of large language models (LLMs) focuses on the models' sensitivity to geo-cultural diversity. However, in addition to cross-cultural differences, there also exists common ground across cultures. For instance, a bridal veil in the United States plays a similar cultural-relevant role as a honggaitou in China. In this study, we introduce a benchmark dataset CUNIT for evaluating decoder-only LLMs in understanding the cultural unity of concepts. Specifically, CUNIT consists of 1,425 evaluation examples building upon 285 traditional cultural-specific concepts across 10 countries. Based on a systematic manual annotation of cultural-relevant features per concept, we calculate the cultural association between any pair of cross-cultural concepts. Built upon this dataset, we design a contrastive matching task to evaluate the LLMs' capability to identify highly associated cross-cultural concept pairs. We evaluate 3 strong LLMs, using 3 popular prompting strategies, under the settings of either giving all extracted concept features or no features at all on CUNIT Interestingly, we find that cultural associations across countries regarding clothing concepts largely differ from food. Our analysis shows that LLMs are still limited to capturing cross-cultural associations between concepts compared to humans. Moreover, geo-cultural proximity shows a weak influence on model performance in capturing cross-cultural associations.

cn Published as a conference paper at COLM 2024
How Well Do LLMs Identify Cultural Unity in Diversity?
Jialin Li Junli Wang
Junjie Hu
Tongji University {2233032,
University of Wisconsin-Madison junjie www.ee..wwwss....cmoee
Ming Jiang"
Indiana University Indianapolis mj280@iu.ee edu
Abstract
Much work on the cultural awareness of large language models (LLMs) focuses the models' sensitivity to geo-cultural diversity. However, in addition to cross-cultural differences, there also exists common ground across cultures. For instance, bridal veil in the United States plays similar cultural-relevant role as a honggaitou in China. In this study, we introduce a benchmark dataset CUNIT for evaluating decoder-only LLMs in un- derstanding the cultural unity of concepts. Specifically, CUNIT consists of 1,425 evaluation examples building upon 285 traditional cultural-specific concepts across 10 countries. Based on a systematic manual annotation of cultural-relevant features per concept, we calculate the cultural associa- tion between any pair of cross-cultural concepts. Built upon this dataset, we design contrastive matching task to evaluate the LLMs' capability to identify highly associated cross-cultural concept pairs. We evaluate 3 strong LLMs, using popular prompting strategies, under the settings of either giving all extracted concept features or no features at all on CUNIT. Interestingly, find that cultural associations across countries regarding clothing concepts largely differ from food. Our analysis shows that LLMs are still limited to capturing cross-cultural associations between concepts compared to humans. Moreover, geo-cultural proximity shows a weak influence on model performance capturing cross-cultural associations.
1 Introduction
Recent advances in large language models (LLMs) have significantly empowered the ma- chines' knowledge capacity in a variety of general domains such math (Ouyang et al.,
2022,Stoolfo et al.) 2023), logical reasoning (Liu et al., 2023b Xu et al. (2023), and common- sense (Bang et al. 2023; Bian et al. 2023). While humans share common knowledge, they knowledge, especially in cultural contexts. In- spired by that, recently, scholars have been actively exploring LLMs' cultural awareness (Li & Zhang 2023 Huang& & Yang, 2023; Wu et al., 2023 Hershcovich et al. 2022), to facilitate cultural knowledge dissemination and cultural y-aware communication by LLMs.
also possess diverse, community-speciff
Prior studies generally consider LLMs' cultural awareness from two perspectives: (1)
linguistic-level variations caused by cultural diversity such as word usage (Shaikh et al. 2023) and language style (Kabra et al.) 2023) Liu et al., 2023a), and (2) knowledge-level
variations like social norms (Fung et al. 2023) Ziems et al. 2023a CH-Wang et al. 2023) and
cultural inferences Liu et al. 2202 Yin et al. al.] 2022
& Zhang 2023) across geopolitical
regions. Despite valuable contributions made by existing research, these studies primarily
focus on examining the sensitivity of models to the connections between geopolitical regions
and their associated cross-cultural concepts. The emphasis lies on the machine's 's awareness
Corresponding author
'Ou dataset and code are released publicly at httttp:://githu..com/// htt.hhu..cmmm//l//////

---

Published as a conference paper at COLM 2024
Calculate Similarity Concepis
Cheongsam
vills
Horgpilln
Chooagaa
Bridel
Horpesiton
bridk
welding
good luck
fonale
triée
hride
Coutrastive Matching Task
waldling
werkfing
walding
Question: Pleuse the following Salnural-Specffiie Concagas" in descending order of similarity with Bridal vells user. attendance occasion cultural signifscance. Cultural- Culturd-Speeffe Concepts: Cheongsam, Honggaitou Label: Cheongsam Honggaitou
fentival
good lack
good lack
Chinese
Eaconolism
modesty
happiness
Figure Illustraiive example of a CUNIT data instance for contrastive matching. Given culturally specific concept, "Bridal veils", from query culture (e.g., United States) and two culturally specific concept candidates, "Cheongsam" and "Honggaitou", from the target culture (e.g., China), our goal is to ask an LLM to determine which target concept shares a higher cultural-centered similarity to the query concept. This comparison is based on the concepts' pragmatic features in three categories: users (e.g., bride), cultural-specific occasions (e.g,, wedding), and cultural significance (e.g,, good luck).
of geo-cultural diversity. However, there is scant research exploring the capacity of LLMs to grasp culturally centered associations among concepts specific to diverse geo-cultures. Differing from prior studies that highlight LLMs' awareness of geo-cultural diversity, this alternative perspective shifts its focus towards investigating the models' potential to capture shared aspects across cultures. The goal of this paper is to assess LLMs' competence in facilitating intercultural communication.
In this study, we evaluate LLMs' ability to align cross-cultural concepts by their inherent cultural-centered associations. Although language and culture are highly intertwined, we focus on LLMs pre- pre-trained on Einglish-domnant corpora. This is mainly because English is the most accessible language, with extensive data encompassing massive global knowledge, including concepts from diverse cultures. Specifically, we concentrate on cultural-centered association in three aspects: (1) the social group of the cultural object's users, (2) cultural- specific occasions, and (3) cultural significance. Figure [ shows an illustrative example. traditional bridal veil worn by the Han Chinese brides at their wedding ceremony, like "Bridal veils" in Western culture. Although both items differ in clothing shape and materials, they are used in a similar cultural scenario (i.e., worn by brides at their weddings) and share the same cultural significance (e.g., symbolizing good fortune). Therefore, the concept "Bridal veils" should have more cultural equivalence with the concept "Honggaitou" compared with "Cheongsam" (i.e., a traditional Chinese dress worn by women in special events like wedding ceremonies). This new paradigm of accessing LLMs' cultural awareness will empower machines' mutual understanding of diverse cultural knowledge, benefiting cross-cultural alignment in downstream applications like machine translation (Yao et al. 2023) and multimodal reasoning (Li & Zhang 2023).
The object "Honggaitou"
To conduct this study, we design new contrastive matching task and introduce novel benchmark CUNIT. Specifically, we curate high-quality set of cross-cultural concepts with detailed feature annotations over 10 countries, and create the curated concepts to evaluate LLMs on identifying concept pairs with higher cultural similarity. We perform an in-depth probing analysis of three popularly used decoder-based LLMs by utilizing popular prompting strategies. Considering the potential challenges that the frequency of cross-cultural concepts might pose for LLMs in deducing implicit cultural-centered associations, we evaluate LLMs in two settings with or without providing external features of cultural concepts in the prompt to L.MSs.
pairs from
2

---

Published as a conference paper at COLM 2024
Overall, the GPT model outperforms the open-sourced LLaMA model in terms of prediction accuracy and consistency in both settings. We further investigate the impact of the cross- cultural concepts' frequency and geographical locations on model predictions. Our results reveal that LLMs are still limited to capturing the cultural similarity between low-frequency concepts in the long tail. Besides, LLMs can benefit from additional culture-related annota- tions in the prompt when comparing two concepts from geographically distant cultures.
2 Related Work
Knowledge probing. Advances in pre-trained language models (PLMs) have spurred a variety of probing designs uncover embedded facets of the world in these models
YYusseef et al. 2023: Belinkov 2022). Popular topics include linguistic properties (Shaikh
et al. 2023 Kabra et al. 2023 Liu al. 2023a biases (Hendrycks et al. 2021: Parrish al.
2022 Huang & Xtong 2023), facts (Petroni et al. 2019: Kassner et al. 2021 Jiang et al. 2020)
and commonsense knowledge (Bang et al. 2023 Ouyang et al. 2022)
Early work typically views probing classification problem, where classifier is built on top of the intermediate representations of selected PLMs to predict target properties (Baroni et al. 2014 Conneau et al. 2018 Tenney al. 2019). With the rise of encoder-based PLMs pre-trained by masked language modeling, the cloze-style probing has been widely used to examine the models' knowledge capacity (Zhong et al. 2021 Yu et al. 2023). Recently, following the generative nature of decoder-based PLMs, especially LLMs, the design of probing a guestion-answwring (QA) format has become popular (Singhal et al. 2022 Tack
& Piech 2022 Blair-Stanek et al. 2023). Our study falls into this probing group and focuses
on
S regarding new probing perspective- gee-cullugrllll competence.
Cultural awareness of language models. Research in language models' awareness of
cultural factors has received increasing attention in the NLP community (Jiang & Joshi
2024
Ramezani & Xu 2023 Tha et al., 2023). Existing work in this field covers a variety of
explorations, ranging from cross-cultural differences word usage (Shaikh et al. 2023 Li
et al.
2023 Hu et al. 2023) to dialect-associated biases in PLMs (Kabra et al. [023] Ziems
et al.
2023b Sun et a 2023), and from the correlation between cross-cultural shifts and
cross-
lingual shifts Yao al., 2002), to geo-diverse commonsense reasoning evaluation (Yin)
ltt al.
2022). Despite various investigations, the majority of prior studies emphasize the
sensitivity of language models to cultural diversity regarding geo-political regions.
Differing from prior work concentrating on the models' awareness of geo-cultural differ- ences, we are interested in the models' potential to capture latent common ground across cultures. To the best of our knowledge, only one latest work (Li & Zhang, 2023) has touched on this topic, where the authors propose an automatic method to align any low-resource cross-cule cros--cultural concept with high-resource cross-cultural concept based on their shared semantic category attributes annotated on WordNet. Compared with Li & Zhang (2023) that considers this alignment problem from on-surface semantic perspective, emphasize culture-centered pragmatic-leve associations among cross-cultural concepts. With this emphasis, our ultimate goal is to measure the geo-cultural competence of L.LMs.
CUNIT Benchmark for Cultural Contrastive Matching
3.1 Task Definition
We introduce a contrastive matching task to evaluate LLMs' capability to capture concept- level cultural similarity in different geo-cultures. Specifically, given a triplet of cultural- specific concepts (c9, c())), where el comes from a query culture q, cf and ch come from target culture f, we evaluate an LLM on identifying a concept from of and c¹ that shares higher cultural similarity with the query ef in terms of three pragmatic categories (g1/82, 82.8).. We formulate this task generative question-answering format, where an LLM is asked to generate a text answer to the question like "Please sort the following cultural-specific concepts in descending order of similarity with cq in terms of 81, 82.82.88 83- Cultural-specific
3

---

Published as a conference paper at COLM 2024
A honggaitou (Chinese: KT #k; pinyin: hönggäitou), also shortened to guitou (Chinese: it 9k; pinyin: gäitou; 'head cover') referred to as red veil English, traditional red-coloured bridal veil Chinese brides cover their faces their wedding ceremony before their wedding night...
Sim(
Hongpaton
Bridall vall
0.625
the Han
Sim
Horezziton
bszgu
0.330
Sim
Hongsston
Ryuson
0.308
human annotation
calculate similarity
generale prompt
Features: occasion, cultural significance
Question: Plense sort the following "Cultural- Culnural-Speciffe Concepts" descending order similarity with Bridal veils in terms of user. occasion and cultural significance. Cultural- Cultural-Specific Concepts: Cheongsam, Hoaggaitou Label: Cheongsam Honggaitou
Chinese brides
Occasion: weedligg
Cultural Significanee: good luck, happiness
The pipeline of CUNIT construction. (1) Find relevant descriptions of cultural- specific concepts from Wikipedia. (2) Extract cultural- features of each concept, and map them into a unified feature schema towards manual annotation (e.g,, using 'bride' to represent synonym features like 'Chinese brides', 'brides'). (3) Calculate the cultural similarity between any pair of cross-cultural concepts. (4) Construct testing cases by different prompt strategies for LLM evaluation.
Figure 2:
concepts: ci, cz." Detailed prompting examples are shown in Table 6 7 in A Appendix D Following the standard practice of prior work (Hossain et al. 2023 & Zhang 2023), we categorize different cultures based on their geo-locations at the country level.
3.2 CUNIT Data Curation
To examine LLMs' performance on this contrastive matching task, we construct a benchmark dataset called CUNIT. Figure displays an overview of our data curation pipeline.
Coss-Cultuual Concept Collection. Our pipeline starts by collecting cross-cultural con- cepts and their descriptions from Wikipedia In this study, we narrow our focus to two material cultural categories: clothing and food (more details shown in Ti Appendix(A). This selection is based our preliminary statistics of accessible cross-cultural concepts across various cultural- categories (e.g., architecture, performing arts) that demon- strate strong cross-cultural associations.
To ensure geo-culture diversity, we select 10 countries across 5 continents that have large number of cross-cultural concepts in the two selected categories(shown in Table [T. To further investigate the influence of geo-cultural proximity on the models' capability to capture cultural similarity, with an emphasis on comparing Éastern-centric cultures versus Western-centric cultures, we considered more Asian countries compared others.
Feature Annotation for Cross-Cultural Concepts. Motivated by prior studies in intercul- tural communication (Wiseman 2003 Wang et al.) 2006), aim to measure the cultural similarity between any pair of cross-cultural concepts originating from two different cultures. Specifically, we focus on comparing the pragmatic information between two cross-cultural concepts, especially examining the context of usage in their cultures. As such cultural nuance is hard to extract from pre-trained word embeddings of each concept, we aim to construct list of categorical pragmatic features for each cross-cultural concept, covering three pre-defined feature categories: (1) the social group of the cultural concept's users,
?Clothing: httpp:://en.wwiipppppps http://dddaarrrreeee.ccccaeerooe en.wikipedia wtt.ddii.gggwww////eoeee wttp://it//aatggrrytuuiu *We exclude South America due to the limited availability of its cross-cultural concepts Wikipedia.
by country and Food: https://
by country)
4

---

Published as a conference paper at COLM 2024
Category
China
India
Japan
Korea
Thailand
Australia
America
Mexico
Italy
Nigeria
All
Clothing
25
12
20
24
7
19
24
6
153
Food
17
17
3
14
30
31
5
132
All
42
27
21
49
55
9
285
Table 1: Countries and cultural concepts in the benchmark. Our benchmark CUNIT for evaluation includes a total of 10 countries. We only consider cross-cultural concepts with more detailed descriptions in Wikipedia in the process of constructing CUNIT.
such gender, social class, and job; (2) the cultural-specific occasion of the concept, such as wedding ceremonies, festivals, and workplace; and (3) the cultural sigmificance of the concept, such as national identity, happiness, and wealth.
Notably, we construct this feature list while annotating our collected concepts. Specifically, to avoid intensive manual annotation yet guarantee the annotation quality, for each concept, we start by designing a simple prompt(shown in Appendix D to instruct ChatGP1 to extract list of excerpts from the concept's Wikipedia article that are relevant to each feature category. Based on the extracted excerpts for each concept, we manually extract the key phrases that contain pragmatic information in our three pre-defined feature categories, and use those key phrases as candidate features. To validate the model's performance in extracting excerpts, we conducted preliminary analysis by having an annotator review ChatGPT's outputs for 50 randomly selected concepts using their full Wikipedia articles, achieving recall of 94.42%. To avoid mis-annotations of concepts due to limited texts, we further filter out concepts that contain key phrases in only one feature category. After processing all concepts' Wikipedia articles, we obtain set of key phrases that potentially contain semantically similar phrases due to paraphrasing. Therefore, to mitigate feature sparsity, we manually conducted phrase normalization that converts multiple synonymous key phrases into a single unique term. We ask two human annotators to extract key phrases and conduct the same phrase normalization to create two feature lists, and obtain Kappa coefficient of 0.9391 for the two annotators, suggesting strong agreement. Finally, we manually merge the two feature lists into a list of 164 categorical features (including 98 features for clothing and 66 features for food), and each concept has 4-5 annotated cultural- relevant features on average. Appendix shows the full list of features.
As each concept is annotated with categorical features, we use the Jaccard similarity to calculate the cultural similarity between any pair of cross-cultural concepts (ct,
Sim(c = Fc. 0 Fg
(1)
F
Fel
where Fgp fa, and Fej Ue. PG. denote the sets of the annotated features associated with the concepts e; and çj respectively.
Sampling Testing Triplets for Evaluation.
Given the annotated cross-cultural concepts,
we aim to create set of meaningful concept triplets D {cC9,c), " W=1
Granularity Sim. Difference Clothing Food
for the con- trastive matching task defined in Section 3.1, As cultural similarity varies largely across cross-cultural concept pairs, randomly sam- pling a concept triplet (ct, (c,,ch,))) is likely to result in incomparable cases where both oi and cf both have very low cultural similarity with the query concept ct. To address this issue, we sample concept triplets satisfying the constraint that one candidate concept has a cultural similarity with the query concept e larger than 0.5, whereas the other candidate concept has a similarity smaller than 0.5. Overall, we have a set of 1, 1,425 testing triplets in total, including 700 clothing triplets and 725 food triplets.
Large Middle
(μ + 0.5ir,
231
156
(μ (m-0.33 0.5or + 0.5g]
221
230
Small
[0, 0.5g]
248
339
Table Concept triplets in three different granularities of similarity difference.
We used the model gpt-3.5-turbo-0613 in this study.
5

---

Published as a conference paper at COLM 2024
To provide a fine-grained analysis of LLM performance on D, we further count the dis- tribution of the similarity difference between the pair of (c), versus (c9,c)) across all triplets, i.e, Sim(c9,ci) Sim(c,,c)) c)), V(e,, ( ~ D, and compute the mean μ and stan- dard deviation o of the similarity difference. We then group the triplets with their similarity differences falling within beyond 0.5g from μ, indicating large, middle, or small level of similarity difference. A small level of similarity difference indicates a harder contrastive matching task, which allows us to evaluate LLMs to tackle difficult cultural triplets. Table shows the statistics of cross-cultural concept triplets per category at each level of granularity.
3.3 CUNIT Data Analysis
Considering that regional and continental variations result in differ- ences in the distribution of features associated with cross-cultural concepts, we calculate the average similarity between all pairs of collected concepts from any two different countries (see Figure 3). Interestingly, we observe a clear similarity cluster among concept pairs from two geolocationally close countries, e.g., clothing concepts among Asian countries.
Cross-cultural Unity.
Clothing
Food
China
China
India
0.20
India
jaan
0.17
0.20
0.28
0.00
Knww
B14
971
0.19
0.8
0.7
Thailand
0:20
0.17
0.19
Thailand
0.1
0.15
Australia..
0.12
0.20
0.06
Australia
0.06
0.09
Amenica
0.11
0.13
0.13
America
0.13
0.18
Moxico
raly-
raly
0:21
0:24
Niperia
0.02
0,02
China
Japan
sotea
Thaland
Avstratia
America
Mexico
Raby
Kijeria
CENIV
iniba
Japan
Thailnd
Avsttr tttaaia
Amarica
Mexico
Robl
Nogeria
Figure 3: Average similarity of concept pairs between different countries. We calculate the similarity of different concepts between countries.
Long-tail Distribution of Cross-Cultural Concepts Frequency. We hypothesize that our collected cross-cultural concepts also follow long-tail distribution which may affect the LLM performance on the cultural contrastive matching task. To verify this hypothesis, we use the Google search engine to estimate the frequency of each curated concept. Specifically, we first concatenate the text string of concept with its corresponding country form query as Query = (concept, country), and then count the number of returned webpages as the approximated frequency for the concept. Indeed, we observe long-tail distribution of the curated concepts in Figure in Appendix
Experimental Setting
Prompting Strategies. Considering the sensitivity of LLM performance to prompts, we employ three popularly used prompting strategies to investigate LLMs' awareness of cultural unity between cross-cultural concepts, including (1) the input-output prompt (IO)
in the template of "Question: [...] An Answer: [..]];; (2) the one-shot prompt (One-shot) that provides an input-output exemplar following the target question; and, (3) the chain- of-thought (CoT) prompt that additionally gives a rationale of the answer in the selected exemplar, aiming to guide the model to generate a rational for the target question.
To investigate how much intrinsic knowledge of cultural unity is learned by LLMs during their pre-training stage, we apply the aforementioned prompting strategies in two evaluation settings with or without providing cultural-relevan features of the concepts in the prompt.
6

---

Published as a conference paper at COLM 2024
Category
Model
Large
Middle
Small
IO
One-shot
CoI
10
One-shot
CoI
10
One-shot
CoI
features
human
100.0-0000
95.93_n000
79 29.44_.0.0
Clatbing
gpl-3.5
laama-13b
50.14_0.0
=0.3
49.13:0m
58.37 10:89
49.78 1000
38.01
54.52_00.0
25.88
49.32
10:00
60.78
1000
40.32.10.0
26.03_0.57
44. 20.50
50.00 Lo03
52.41
19.60..00
Ilama-7b
36,80_020
38.10.0m
47.62
30.32..000
39.37
2010
50,00
37.30ga00
41.33
48.39 s003
human
94,87
81.74_.0.00
76.70.0.0
gpt-3.5
65
67
0.00
52.32-0041
57.10
1071
1.34
1..0
27.8
36.68 2077
Nama- 13b
46.47 0030
49.68 40
45.5
10000
48 26.26
47.17 2000
46.5
2000
46.90
46.46.0000
46.46
44.40_
Ilama-7o
40.71
50.32.000
52.24 40,00
38.48
38.26.0m
20.00.a00
42.63..0
41.74.003
47.05
Without features
human
93.51
88 88.69.
66.53.pp.0
Clothing
Ilama-13b
48.27-077
46.10_000
10.73
45.74_009
20.41
g0000
50.83_00.6
46.61 =0.00
54,45
2000
52.94
47.51 20000
46. 46.37
48.39_0.30
50.07 +3.56
29,80
46.77
50,20 _0.00
Ilama-7b
40.04
26.62 i0
37.45
39.14
29. 86 10:0
36.65
40.32
31.65
39.72/000
human
91.67 0.00
64.35 0.00
67.55 10.0
Fond
gpt-3.5
Ilama-13b
48.08
38.55
0.30
37 37.50.000
51.39
46.47
49.36 20:1
68.77 68.77--0.00
50.22 50.22-.0.00
-0.00
$7.61
53. 20.
50.72
57.17 =0:0
54,23 54.23..0.3
48.53
49.90 49.90
48.67
20:37
48.08 18.00.00
51.87 =018
42.31_000
11.03:100
46.15
31.96.000
=000
43.66..0.0
36.73.000
46.31=n00
Table 3: Accuracy of three prompting strategies with or without providing features in the cultural contrastive matching task. The bold number indicates the best-performing model in the same strategy, and underline number indicating the best-performing strategy in the same model.
Specifically, in the first setting (W/ Features), we list cultural-relevant features per concept in the prompt, whereas in the second setting (W/O Features), directly ask the question in the prompt without providing any concept features. To further disentangle the influence of the curated concepts' long-tail distribution property LLMs, replace the concept mention with an artificial phrase (e.g,, concept A, concept B), to see if model can identify similar concepts only based on their features.
Models. We use 3 popular LLMs for evaluation: (1) GPT-3.5 (GPT-3.5-turbo-6133, 175B parameters) which shows a higher quality in handling complex instructions than prior GPT-based models (Brown et al.] 2020); (2) LLaMA-7B and (3) LLaMA-13B which are open-sourced LLMs pre-trained on the English-dominated corpora (Touvron et al., 2023). In addition to LLMs, we also employ 2 human annotators to answer questions with and without features respectively on CUNIT for comparison. The annotators' background is shown in Section
Metrics. We use two evaluation metrics in this study. First, we measure the accuracy by checking if the LLM can correctly identify more similar concept per concept triplet in the contrastive matching task. Second, we hypothesize that LLMs may have intrinsic biases in selecting the first or second candidate concepts regardless of the question. To assess such model biases, we evaluate the consistency of the LLM's prediction for the same question by flipping the order of the two candidate concepts in the prompt.
LLM Evaluation and Results
5.1 How well do LLMs identify similar cross-cultural concepts?
Overall Accuracy. Table 3 shows the accuracy results of humans and LLMs in identifying a cultural concept with a higher similarity the cultural contrastive task. Overall, the accu- racy of human significantly outperforms LLMs, indicating that LLMs still face challenges in identifying the latent cultural unity of concepts compared to humans. Comparatively, GPT-3.5 shows higher prediction accuracy than LLaMA in most testing cases, suggest- ing that this closed-sourced model captures more common ground across cultures than I.L.A--base open-sourced models.
7

---

Published as a conference paper at COLM 2024
Clothing
Food
1.0
1.0
gpt-3.5
Consistency
0.8
fama-2212 1mma-2-1335
lama-2-7b
Consistency
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
CoT 0.0
10
One-shot
10
One-shot
CoT
Figure 4: Consistent performance across different models in experiments with features. Including the consistency performance of the three prompt strategies under gpt-3.5-turbo- 0613, Ilama-2-7b-chat and llama-2-13b-Bn it can be found that --t-b.--turrb has the best consistency performance, significantly better than llama-2-7b-chat and llama-2-13b-che
W/O Features. We further compare the models' and humans' prediction accuracy in the evaluation setting without providing any concept features. Interestingly, unlike humans and GPT-3.5, which show improved prediction accuracy with prompts containing concept features, LLaMA variants, particularly LLaMA-13B, tend to perform more effectively when such features are absent from the prompts. One possible reason for this observation is that LLaMA is not trained robustly to utilize various forms of unseen information in the prompt at test time, and adding such unseen cultural features confuses the model.
Contrastive Matching Granularity. By further looking into the model accuracy among three levels of similarity difference between the candidates and the query concept, we find that humans tend to make more prediction errors when the similarity difference between two candidates with the query concept becomes smaller. However, this data factor does not play clear effect on the models' predictions. We conjecture that although LLMs can identify concept with a higher cultural similarity from a cultural concept triplet, these models cannot measure the numerical values of similarity accurately.
Prompting strategies. Regarding three prompting strategies, we find that CoT prompting generally guides models to achieve the highest prediction accuracy, followed by one-shot prompting and vanilla input-output prompting. Particularly, this pattern occurs more in the evaluation setting with features in the prompt. Our observation indicates that adding Mlain-of-thought rationale and providing exemplar both improve the models for identifying associated cross-cultural concept pairs.
Stability of LLM prediction. Given that prior studies have shown that LLMs may in- volve intrinsic biases to the order of candidate options in multi-choice question answer- ing ingiRonnnson Wingate 2023), we further investigate the consistency L.Ms'' predictions by ilipping the order of two candidate options. Figure 4 shows the consistency of the LLMS under each prompting strategy. We observe that GPT-3.5 obtains a noticeably higher consistency compared with LLaMA variants, This result indicates that the predictions of GPT-3.5 rely more on the semantic understanding of the questions rather than the order of the candidates.
5.2 How Do Data-centric Factors Affect LLMs' Predictions?
Cultural Knowledge Representaativeness. Given that some cross-cultural concepts such as "Bridal veils" are much more well-represented than other concepts like "Honggaitou", we explore the sensitivity of LLM performance to cross-cultural concepts frequency. Specifically, we take the maximum frequency (based on Google Search Engine) of concepts in each triplet sorting testing triplets by their representativeness, we compare the prediction accuracy of the best LLM (GPT-3.5) on the top 1 1/3 of well-represented triplets versus the bottom 1/3 of rndre-tepreeennte triplets. Figure
to denote the triplet's maximum representativeness.
8

---

Published as a conference paper at COLM 2024
Clothing
Food
1.0
1.0
Bottom 33%
0.8
Top
0.8
0.6
0.6
Acc
0.4
0.4
0.2
0.2
0.0
0.0
10
One-shot
CoT
One-shot
CoT
Figure 5: The accuracy of GPT on concept triples(with features) in different long-tail degree groups. We computed the maximum long-tail degree in each triplet, denoted as dmax- And we calculated the average accuracy of dmax positioned within the first 1/3 and the last 1/3 of all triplets.
shows the prediction accuracy on clothing and food, respectively. We find that the model consistently exhibits higher prediction accuracy the well-represented triplets versus the under-represented ones across prompting strategies and cultural categories, indicating that long-tail cultural knowledge could challenge the model for identifying cross-cultural similarity.
Accuracy Inter- vs. Intra-group Geo-cultures
1.0
0.8
Accuracy
0.6
0.2
0.0
Eastem
Westem
Across
Eastern
Westem
Across
Eastem
Wester
Across
Eastern
Westem
Across
(GPT with features)
(GPT without features)
(human with features)
(human without features)
Figure 6: The accuracy between different culture groups. We compare accuracy of GPT and human when the concept triplets are from different culture groups. The length of the box represents the first quartile to the third quartile of the accuracy. The orange line shows the median accuracy. The lines extending from the box extend to 1.5 times the IQR above and below the quartiles, and circles outside the box indicate obvious outliers
Inter- vs. Intra-group Geo-cultural Proximity. We further explore the model prediction accuracy to see if there exists geo-cultural proximity in the model's capability to identify similar cross-cultural concept pairs. We group the cultures into two groups: (1) Eastern and (2) Western cultures. Figure 6 boxplot showing the median (orange bar, with the first quartile to the third quartile in a box) of the accuracy in the cultural contrastive matching task, where we particularly focus on the model's performance against human annotators of Mosseenn-Westen cultures. From the first three columns (GPT
in the inter-/intra-gne with features) Figure we observe a higher median accuracy (yet larger variance) of GPT-3.5 on predictions of concept triplets across Fastern-Western cultures, compared to its predictions of concept triplets within the same cultures. However, the setting without features, GPT-3355 does not show clear difference in terms of the median accuracy from inter- or intra-group of Bastern-Westare
cultures. Besides, human annotators outperform GPT-3.5 in terms of median accuracy in both settings with without cultural features. When concepts originate from Eastern-centric cultures, annotators can provide more accurate predictions, these annotators are from Eastern countries.

---

Published as a conference paper at COLM 2024
6 Conclusion
We propose new benchmark CUNIT, and design contrastive matching task to assess the cultural understanding of LLMs. The cross-cultural concepts collected by CUNIT demon- strate variations among different regions, with greater similarity observed between concepts among Eastern countries. Additionally, there is significant skewness the long-tail dis- tribution of different cross-cultural concepts. The contrastive matching task showcases the potential and consistency of GPT-3.5 in measuring cross-cultural unity, while open- sourced LLMs like LLaMA do not perform robustly to handle ordering and formatting issues. Meanwhile, the CoT prompting strategy demonstrates stronger prediction perfor- mance than vanilla input-output prompting and one-shot prompting. Regarding the data factors influencing cross-cultural similarity measures, we find that long-tail distribution of cross-cultural concepts and regional differences have an impact on models' prediction per- formance. Overall, LLMs still make more mistakes on long-tail taig-taall concepts. In our preliminary work for cross-cultural research, we provide data on cross-cultural concepts and contrastive matching tasks, as well as analysis perspectives on long-tail degree and regional differences, which would facilitate future research in exploring the cross-uultural capabilities of LLMs.
7 Ethical Considerations
The construction of CUNIT involves two stages of human annotation. In the first stage, focus on annotating concept features. Specifically, we employed 12 annotators from China, India, and the United States to identify culturally relevant features of each concept using Wikipedia descriptions (both the full text and ChatGPT-highlighttr excerpts). The annotators are generally bilingual, with some being multilingual. The second stage involves contrastive matching on testing data. To establish upper-bound performance, we collected judgments from two additional annotators, both from China and proficient in English. Given the cultural ties in Asia, they possess sufficient cultural knowledge across Asian countries. We also provided a tutorial on our curated cultural concepts to further enhance their understanding of annotation. Despite that, there may still inevitablely exist potential annotation biases given the annotator' 's cultural background.
Limitations
Considering that CUNIT is constructed the basis of Wikipedia and specifically focuses on two types of material cultures across 10 countries, issues with data biases and represen- tativeness are inevitable. Although could collect data from sources beyond Wikipedia, maintaining data quality would be challenging. Another limitation is the limited cultural background of the annotators, which may affect their understanding of diverse cultural con- cepts, particularly unfamiliar ones, during annotation. To address these issues, a potential solution is launch an open data curation platform that invites crowdsourcing volunteers to share their knowledge on cultural concepts, thereby expanding the valuable resources in CUNIT. In addition, our contrastive matching task is the first step in examining LLMs' understanding of cultural unity. In the future, we plan to extend our exploration to more practical downstream applications, such as machine translation.
9 Acknowledgements
We would like to thank the anonymous reviewers for their valuable feedback. MJ is partially supported by Google and the National Science Foundation (ISS-438420).
10

---

Published as a conference paper at COLM 2024
References
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung- A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. CoRR, abs /2tp//22320002233 2023. doi: 0148550//AXIV22302.000
URL
https://coi.org// 18550/arXixg wttpaarrvvv.222..8.0 023322004223
Marco Baroni, Georgiana Dinu, and Germán Kruszewski. Don't count, predict! A systematic comparison of context-counting vs, context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 1: Long Papers, pp. 238-247, The Association for Computer Linguistics, 2014. doi: DL.115/V11P144-003 URL v1/p14-1023
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Comput. Linguistics, 221222 2022. doi: 10.1162/COLI .00422. URL https://doi.org: 08.1162//co1i.a00422
Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, and Ben He. Chatgpt is a knowl- edgeable but inexperienced solver: An investigation of commonsense problem in large language models. CoRR, abs 2300.166212 2023. doi: (05500/ARXTV.233..162222 URL
https://doi, org/ 10. 48550/ 2303. 16421
Andrew Blair-Stanek, Nils Holzenberger, and Benjamin Van Durme. Can GPT-3 perform statutory reasoning? In Matthias Grabmair, Francisco Andrade, and Paulo Novais (eds.), Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, ICAIL 2023, Braga, Portugal, June 19-23, 2023, pp. 22-31. ACM, 2023. doi: 10.1145 /594536. 3595163.
org/ /00g 1441 3555//559536..6 3595163
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Porbert-Vosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL/ntpps:/prrceedigg.. hash/1457
ips. pc/pap/r/202//
Sky CH-Wang, Arkadiy Saakyan, Oliver Li, Zhou Yu, and Smaranda Muresan. Sociocul- tural norm similarities and differences via situational alignment and explainable textual entailment. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 3548 3564. Association for Computational Linguistics, 2023. URL https://acllantho..og org. 2023. mmnlp-main. 215
Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single \$&!#* vector: Probing sentence embeddings for linguis- tic properties. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 2126-2136. Association for Computational Linguis- tics, 2018. doi: N..18633/V1/PD 1198. URL /tRL/tttt:///aalattooo hwwphhooogyoorgggg/gggo
Yi Fung, Tuhin Chakrabarty, Hao Guo, Owen Rambow, Smaranda Muresan, and Heng Ji. NORMSAGE: multi-lingual multi-cultural norm discovery from conversations on- the-fly. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. URL httphhttt///clanttttt wrgeeeoooorrgeeeee emn emn.p-main...... 941
Association for Computational Linguistics, 2023.
11

---

Published as a conference paper at COLM 2024
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning Al with shared human values. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. httpss:/:///pem
net/forum?id=dNy RKzJacY
Daniel Hershcovich, Stella Frank, Heather C. Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Îlias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. Challenges and strategies in cross-cultural NLP. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, Pp. 6997-7013, Association for Computational Linguistics, 2022. doi: 10.18653 V1/2022. ACL-LONG.482.
www.i11165/v22220 acl- long.-l-ooon... 482
Tamanna Hossain, Sunipa Dev, and Sameer Singh. MISGENDERED: limits of large language models in understanding pronouns. In Anna Rogers, Jordan L. Boyd--Grabee and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 5352-5367. Association for Computational Linguistics, 2023. doi: URL https://d/// org/10 18653/v1/2023 acl long.
Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. A fine-grained comparison of pragmatic language understanding in humans and lan- guage models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 4194-4213. Association for Computational Linguistics, 2023. doi: 10.18653 (01116653/112202 /tt////2222233LLOOGGGC https://doi. org/oioorg//0 1rtp:/o...rrggoo11116///////2 18653/v1/2022
URL
long- 230
Jing Huang and Diyi Yang. Culturally aware natural language inference. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 7591 Association for Computa- tional Linguistics, 2023. URL//ttps///acllnthooooggg/ org//2000...../23333.. http://2222....inngg.ee findings-emnlp 509
Yufei Huang and Deyi Xiong. CBBQ: A chinese bias benchmark dataset curated with human-ai collaboration for large language models. CoRR, abs 162444, 2023. doi: 10.48550/ AA0////2.XII.220112222222 URL URL
wttp////...////r..i....oooo 2306. 16244
Akshita Jha, Aida Mostafazadeh Davani, Chandan K. Reddy, Shachi Dave, Vinodkumar Prabhakaran, and Sunipa Dev. Seegull: A stereotype benchmark with broad geo-cultural coverage leveraging generative models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 9851-9870. Association for Computational Linguistics, 2023. doi: 10.18653 / ACL-LONG.558.. htthhttss///do..rrrr 186538886200000 ac1-long. 548
Ming Jiang and Mansi Joshi. CPopQA: Ranking cultural concept popularity by LLMs. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (Volume 2: Short Papers), pp. 615-630, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.18653/ve 2222222aaallllsootttoo URL https://aclaanthooogy.yoo wwwp:///oooog//2224////// www.aaaacccteeeee
Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. X-FACTR: multilingual factual knowledge retrieval from pretrained language models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 5943-5959. Association for Computational Linguistics, 2020. 4799. URL https: //oii. org/10, 10. 18653/v1/2020
doi: 10.18653. / /V1 emn1p-main.44 479
12

---

Published as a conference paper at COLM 2024
Anubha Kabra, Emmy Liu, Simran Khanuja, Alham Fikri Aji, Genta Indra Winata, Samuel Cahyawijaya, Aremu Anuoluwapo, Perez Ogayo, and Graham Neubig. Multi-lingual and multi-cultural figurative language understandlnng In Anna Jorean L. Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Compulational Linguistics: ACI. 2023, Toronto, Canada, July 9-14, 2023, pp. 2269-8284 Association for Computational Linguistics, 2023, doi: 10. (0.11863//1// /V1 /1tp//1//22222222 HOD.HHINDINNSS-CLEE 10. 10888663122222 Findings-acl.55 525
Jordan
URL https://doi.orrr
Nora Kassner, Philipp Dufter, and Hinrich Schütze. Multilingual LAMA: investigating knowledge in multilingual pretrained language models. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pp. 3250-3258. Association for Computational Linguistics, 2021. doi: 10.18653 /6653/V1 /V1/ 2021. AD2.EEACL-MAIN28. URL httph:ttp:://ol http:tss////doo.rr/// 18653/v1/2021 eacl-main. 284
Liang Li, Ruiying Geng, Chengyang Fang, Bing Li, Can Ma, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. CAIS: A pragmatic chinese answer-to-sequencc dataset with large scale and high quality. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 2983-3000 Associa- tion for Computational Linguistics, 2023. doi: 10.18653 /V155/////// W1122223.CLLLLOCC.C
URL
https://doi. org/10. 18653/v1/2023. -tt lcn- 168
Zhi Li and Yin Zhang. Cultural concept adaptation on multimodal reasoning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 262-276. Association for Computational Linguistics, 2023. URL https://aallanhoolgg org/2023. emn wwwnnpamann.......mmm...mmmmmmmm 18
Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. Are multilingual Ilms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. CoRR, abs/ 20..22399.0999 2023a. doi: 10.48550/ARXIV23309.. ARXIV233090055 URL htttps://ooi..o/ org/ 10. 09.48550/arr11 2389.08591.
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. Visually grounded reasoning across languages and cultures. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 10467-10488 Association for Computational 818. URL /ttt:://oooo.ogr///.116151111111 www.p- main. .
Liciiiinnnnninnnnniiiiii Linguistics, 2021. doi: 10.18653/V1/222 /T2.///VV//2.1.MMLMAAAATT E...L--MIN..NNN
org/10. 11653/v1/2021
Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. Evaluating the logical reasoning ability of chatgpt and GPT-4. CoRR, 2254/2333.03333 2023b. doi: 10.48850/AAN ARXIV.2304.03453
URL httphhtts///doi.orr wtt..////....///........../. 2330//33rxx..2.4....... 03439
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul E. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeuriPS, 2022. URL
nips. ccpppp/apennsssccpppppp files/ hrgp://6235...aa00001111aaaaaaar
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. BBQ: A hand-built bias benchmark for question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ire- land, May 22-27, 2022, pp. 2086-2105, Association for Computational Linguistics, 2022. doi: 10.18663/ W11/202.FINDINNGS-II ACL.16, URL https::///// //oii org/10, 18653/v1/2022 (findings-add acl. 165
13

---

Published as a conference paper at COLM 2024
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. Language models as knowledge bases? In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNIPPIJCCNIP 2019, Hong Kong, China, November 3-7, 2019, pp. 2463-2473, Association for Computational Linguistics, 2019. doi: C116653/V1/D19-II/
URL https://doi..or//// 100.0001180vv1v/09111110
Aida Ramezani and Yang Xu. Knowledge of cultural moral norms in large language models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computationnl Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 428 Linguistics, 2023. doi: 10.18653 /tt/////322...-LO..... 18653/v1/2023 wol-long....
428-446. 446. Association for Computational URL https://doi.orr org/10
Joshua Robinson and David Wingate. Leveraging large language models for multiple choice question answering. In The Eleventh International Conference Learning Rep- resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
Omar Shaikh, Caleb Ziems, William Held, Aryan J. Pariani, Fred Morstatter, and Diyi Yang. Modeling cross-cultural pragmatic inference with codenames duet. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 6550-6569, Association for Computational Linguistics, 2023. doi: 10.18653 V1 FINDIIGSININNG 410. URL https:://doi.org/// 18lo0/111533/3130 Findings-acl.477
Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Schärli, Aakanksha Chowdhery, Philip Andrew Mansfield, Blaise Agüera y Arcas, Dale R. Webster, Gregory S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle K. Barral, Christopher Semturs, Alan Karthikesalingam,, and Vivek Natarajan. Large language models encode clinical knowledge. CoRR, abs 2022. doi: 10.48550/ 0A55500/ARXXV.2222133333 URL http::://to.i..... http:///oooorr///////// 48550/arXi00 2212. 13138
Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf, and Mrinmaya Sachan. A causal framework to quantify the robustness of mathematical reasoning with lan- guage models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACI. 2023, Toronio, Canada, July 9-14, 2023, pp. 545-561. Associa- tion for Computational Linguistics, 2023. doi: https://doi 1t..or//.../..... 188838816622222 acl acl-long. 32
URL
Jiao Sun, Thibault Sellam, Elizabeth Clark, Tu Vu, Timothy Dozat, Dan Garrette, Aditya Siddhant, Jacob Eisenstein, and Sebastian Gehrmann. Dialect-robust evaluation of gen- erated text. In Anna Rogers, Jordan L. Boyd-Graber and Naoaki Okazaki (eds.), Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 6010-6028. Association for Computational Linguistics, 2023. doi: (0.18653/V1/ fttps://doi.orr/// 18653/v1/2022 acl-long 331
URL
Anaïs Tack and Chris Piech. The AI teacher test: Measuring the pedagogical ability of blender and GPT-3 in educational dialogues. In Antonija Mitrovic, Nigel Bosch, Alexan- dra I. Cristea, and Chris Brown (eds.), Proceedings of the 15th International Conference Educational Data Mining, EDM 2022, Durham, UK, July 24-27, 2022. International Ed- org.gg.rrg/22222/
ucational Data Mining Society, 2022. URL wasssshortppppers 54/index html
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie
14

---

Published as a conference paper at COLM 2024
Pavlick. What do you learn from context? probing for sentence structure in contex- tualized word representations. In 7th International Conference on Learning Representa- fions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/ 2307709288, 2023. doi: 10.48550/ SRXXIV2307.09288. URL https //doi. ordoi.org/1..o 41510/.4550/440 arxiv. 2307. 09288
Wen-Ling Wang, Hwei-Ling Lee, and Susan Jane Fetzer. Challenges and strategies of instrument translation. Western Journal of Nursing Research, 28(3):110-321, 2006. doi:
10.1177/0199945995284472 URL https://doi. http:///ooooorg//...0 01/0111117770199999995544772 PMID: 16585807.
L. Richard Wiseman. Intercultural communication competence. In B. William Gudykunst (ed.), Cross-Culturii and Intercultural Communication, chapter 11, pp. 191-208. Sage, 2003,
Winston Wu, Lu Wang, and Rada Mihalcea. Cross-cultural analysis of human values, morals, and biases in folk tales. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 5113-5125. Association for Computational Linguistics, 2023. URL htth:httss//aaathhhtttt
org. ottp://aaanooooooooooo/oo/oo/aaa
311
Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. Are large language models really good logical reasoners? A comprehensive evaluation from deductive, inductive and abductive views. CoRR, 10,48550/ TERXIVV.2306098444
2023. doi: /ttL:ttp:: wrw.:/....rg/11.. 48550/arXi 2386.09841
Binwei Yao, Ming Jiang, Diyi Yang, and Junjie Hu. Empowering Ilm-based machine transla- tion with cultural awareness. CoRR, abs 2bs/2305.14328, 2023. doi: 10.48550 A3XIV 14328. URL hhttts:t/dooiiii wtt.:////.....////.i.ii.o.o// 2305. 14328
Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. Geomlama: Geo-diverse commonsense probing on multilingual pre-trained language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 2039-2055 Association for Computational Linguistics, 2022. doi: 10.18653/VI 2222222EMNL--MAAI bttps://doi.orge/ /00 18653/v1/20277 http://///2222...mmmmmmmmmm 132
URL
Paul Youssef, Osman Alperen Koras, Meijie Li, Jörg Schlötterer, and Christin Seifert. Give me the facts! A survey on factual knowledge probing in pre-trained language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 15588-15605 Association for Computational Linguistics, 2023. URL /tL/ittpss//aalattooloyyy/ www.aaanthhloo.rgg//////. 2023 findings-emmlp. 1043
Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, Chunyang Li, Zheyuan Zhang, Yushi Bai, Yantao Liu, Amy Xin, Nianyi Lin, Kaifeng Yun, Linlu Gong, Jianhui Chen, Zhili Wu, Yunjia
15

---

Published as a conference paper at COLM 2024
Qi, Weikai Li, Yong Guan, Kaisheng Zeng, Ji Qi, Hailong Jin, Jinxin Liu, Yu Gu, Yuan Yao, Ning Ding, Lei Hou, Zhiyuan Liu, Bin Xu, Jie Tang, and Juanzi Li. Kola: Carefully benchmarking world knowledge of large language models. CoRR, abs 220..09226966 2023, doi: 10.48550/ ARXTV.2306.099 09296. URL httph:t/p///// org/ 48550/arXiv 092966
Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [MASK]: learning vs. learning to recall. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Häkkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 5017-5033, Association for Computational Linguistics, 2021. doi: 10.18653/ /V1 W1/202.NAACCLMMII .388. URL httLttt///ooo.org///// 18653/v1/2021 naacl-main. 398
Caleb Ziems, Jane Dwivedi-Yie Yu, Yi-Chia Wang, Alon Y. Halevy, and Diyi Yang. Normbank: A knowledge bank of situational social norms. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 7756-7776. Association for Computational Linguistics, 2023a. doi: 10.18653/ /V1/ 2023.ACI-LLONG.AD0
http::ttss///ooo org/10. (6633v1//022) acl -ong. 429
Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. Multi-value: A framework for cross-dialectal english NLP. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volune 1: Long Papers), ACL 2023, Torotoo, Canada, July 9-14, 2023, pp. 744-768. Association for Computational Linguistics, 2023b. doi: 10. 18653 (0653/V1/20
URL https://doi. 00t.///0..1653/11202220 acl- 44
A Preliminary Statistics
In Table 4 give the number of cultural concepts of food and clothing in Wikipedia in different countries. The countries we select are the top-ranked countries in different continents. We also consider geographical proximity factors, select India, China, Japan, Korea, and Thailand within Asia.
B Features of Clothing and Food
In Table we list the features in clothing and food categories used after normalization.
C Long-tail Distribution of Concepts
In Figure 7 we plot the distribution of webpage counts from the Google Search engine for all cross-cultural concepts, and we see clear long-tail pattern.
D CUNIT Data Examples
The prompt template used to filter concepts from Wikipedia descriptions is:
Question: Please extract sentences phrases from the following context that describe the cultural concept Concept Name, including the users of the cultural concept, the cultural occasions it is commonly used in, and its cultural significance.
Context: Description
In Table 6 7 8 we take Honggaitou, Cheongsam, and Bridal Veils examples to show the prompt templates of different strategies we used. In the experiments without features, we removed the feature information of cultural concepts.
16

---

Published as a conference paper at COLM 2024
Country
Continents
Clothing num
Food num
Sum
India
Asia
351
209
560
China
Asia
143
195
338
Japan
Asia
239
333
Mexico
America
19
255
274
Italy
Europe
216
259
France
Europe
42
212
254
Korea
Asia
133
205
Turkey
Asia
32
172
204
Germany
Europe
38
146
184
Britain
Europe
67
111
178
Indonesia
Asia
44
134
178
Pakistan
Asia
80
98
178
Tran
Asia
59
104
163
America
America
75
162
Greece
Europe
35
115
150
Spain
Europe
21
120
141
Thailand
Asia
25
115
140
Russia
Asia
103
138
Australia
Oceania
46
89
135
Poland
Europe
12
116
128
Vietnam
Asia
23
100
123
Bangladesh
Asia
38
74
112
Nepal
Asia
13
99
112
Portugal
Europe
101
104
Columbia
America
94
99
Ireland
Europe
18
79
97
Norway
Europe
75
92
Nigeria
Africa
10
81
91
Ukraine
Europe
65
90
Peru
America
79
88
Serbia
Europe
83
88
Netherlands
Europe
15
72
87
Algeria
Africa
18
68
86
Syria
Asia
85
86
Brazil
America
84
85
Finland
Europe
79
83
Canada
America
77
82
Austria
Europe
75
Chile
America
70
78
Palestine
Asia
70
77
Switzerland
Europe
67
75
Table 4: Number of clothing and food concepts in all countries collected from Wikipedia. We calculate the quantity of concepts of country in Wikipedia categories 'Clothing by country' and 'Cuisine by country'.
17

---

Published as a conference paper at COLM 2024
Type
Category
Features
male, female, child, noble, commoner, emperor /empress, official, concubine, prince/ ppriceess,, royal, politician, swagman, military, police, judge, farmer, labourer, frontiersman, athlcte, postman, firefighter, rider, cowboy / /cowgirl, fisher, trader, hunter, dancer, musician, servant, guard, scholar, monk, bridc, married, unmarried
people
formal, informal, ceremony, sacrifice, wedding, funeral, graduation, coming of age, worship ancestor, conferring, coronation, religious, gathering, court, political stage, casual, hunting, birthday, festival, celebration, military action, kayaking, drama, workplace, nightwear, swimming, labour, farming, fishing, horse-riding, riding
Clothing occasion
hope, good luck, longevity, wealth, protect, fertility, happiness, morals, brave, virtue, spiritual, modesty, honor, social status, cultural heritage/symbol symbol of nation, identity, unity, relationship, healing, marriage, religious beliefs, personality, emperor's 's favor, respect, pride, solidarity, revolution, welcome, nature, peace, rebellion, authority
meaning
bride, groom, villager, tribe, military, child, stockman, noble, people related to religion
people
ceremony, celebration, party, birthday, wedding, funeral, worship ancestor, feast, gathering, baptism, festival, Epiphany, Winter Solstice, Day of Dead, Passover, beginning of spring, Valentine's Day, Lantern Festival, Hannukah, Easter, Lebaran, Christmas, New Year, Hallows' Day, carnival holidays, Fool's Day, Children's Day, Mother's s Day, Girl's Day, Shabbat, Thanksgiving, Deepavali, Purim, religious beliefs
Food
occasion
religious beliefs, happiness, wealth, hope, fertility, hospitalit, sharing, nature, bless, commemorate, pure, comfort, rural life, gratitude, respect, celebrate, reunion, health, longevity, beginning, good luck, waste, get rid of hardship
meaning
Table 5: Normalized feature words under different categories. We list three feature words under different categories here: (1) the social group of the cultural object's 'ss users; (2) the cultural-specife occasion of the concept; and (3) the cultural significance of the concept.
Stritegy
Examples
Qusttnn: Please spet the fellowing 'Culturaloppeific Cencepts' in descording order of similarity feature overlap between Cultural-aprific Concepts' with Honggaitou in tenma of weanss, athesdanc excasien and symbolic mearing:
Inpot-octput Prompting
Calluna--queefe Concpti Chrongum, Bridal veils Answer Furmat: Honggaitoo and Cheongsam are more similar than Honggailou and Bridal veils terms of wearet, allendance occasion and symbelic meoning. please snswer Cheongam Bridal veils, otherwise answer Cheoggam <Bridal veils. Anster:
w/o features
Questine Please sort the following Cultural-specific Conceptss in descending order of semilantt feature overlap between Cultural-pecfffc Concepts' with Hongraitou terms secnrer, athendance eccesion and symbolic mesning. Culluraf-apect Concytia Chrongum, Rridal wells
Inpuboolput Prompting
with features
Featume of Trridal voils: 1. Wearem: female, bride: 2, Attendance occasion: wedding: Symbolic Meaning; good luck, modesty
Fastures of Houggattor: 1. Wearer: female, bridi; 2. Attendance cccasion: wedding Symbolic Meoning: good lack. happiness
Features of Cheonguaar: Wearer: female: Aftendance occasion: wedding, featival: Symbolic Meaning China nationalism
Ansioer Former: Honggailow and Chemggam are more simolar than Hongaaitu and Bridal veils terms weater, attendance occasion symbolic meaning, pleae snewer Cheongam Bridal veils, otherwise answer C Cheongsam <Bridal veils. Ansoe::
Question: Please soet followlngg Cultural-specilic Concopts' descanding order of similarity feature oveslap between Cultural-spcctin Concepts' with concept A terms wearee, attendancs occasion and symbolic exaning.
Cultural spoctii Conernts: concept concept
Inpot-output Prompting
(w/o names)
Festures of comcept
Festires of comcept A:1, Weaner: female, bnde Attendance eccasion: wedding: Symbolic Meoning:
concept
female: Attendance occasion: wedding
good happiness wedding: Symbosi Meening: nattoralism Attendance concept concept
Meaning: good modlesty concept <concopt
Fornut: concept concept
occasion symbolic meaning, please concopt Sconcept C, alhherwie Ansuer:
Table Input-output prompt strategies' examples. Each instance includes prompt template for evaluating language models, the cultural-specific concepts and the required format for responses.
18

---

Published as a conference paper at COLM 2024
Strategy
Examples
Questiow: sort following Cultural-ppcilic Concepts' descending order similarity feature overlap betwenn
Cultural-ppccfic Concepts' leongiagenn terms wearer, attendance occasion and symbolic meaning.
Cultarsl-qqaijle spoolt Concepts Calceus, Pileus (ial))
Ansaw Formis: Jeanglagwan and Caleus more than Jeongiagwan Pileus (hal) lerms weates, atlendance occasion and symbolic mearing, plense answer Calerus Pileus (fut), othewwise answer Calceus <Pilens (hat). Ansave: Calcess > Pileus (hat)
One-shet Prompting
w/o features
Quesion: Please sort the following "Cultura-specifcc Concepts' in descending order of similarity feature overlap between " Cultural-ppclit Concepts' with Honggailou in lerms of weares, atlendarce occasion and symbolic meaning. Cultural specffe Concepts: Cheongsam, Bridal veils
Ansavy Formel: Hooggaitou and Chrongsam are more similar than Honggaitou and Bridal veils lerms of weares, atlendance occasion and symnholi mearing, please answer Cheongsam > Bridal vils, otherwise wise answer Cheongsa < <Bridal vells. Amorr:
Qhaskion Please sori the fellowing Culluul--ppeilic Concepts' in descending order of similarit feature overlop beloveen Cuhturl-specffic Concepte' with Jeongisgwan in terms of weates, attendance occasion and symbolic meaming. Cultural spacffe Concepts Calceus, Pileus (nat)
Features joingtguse: Fanure Calauss 1. Wearer: lemale; 2 Atlendance occasion: wedding, fesiiall 3 Symbolic Meaningg: China nalicnalism
Weazer: female, bride: 2. Attendance occasion: wedding: 3. Symbolic Meaning: good luck, happiness
Fanures Pileus (ua)): 1. Wearen female, bride, Atlendance occasion: wedding, 3. Symbolic Mearing: good luck, modesly Amsve Format Jeongjagwan and Calceus more similar than Jeungigwwan and Pileus (hat) terms
attendance
occasion and symbulic mearingg please answer Calerus > PPileus (fat), othewwise answer Calceus <Pileus (fut). Amsver. Calceas > Pileus
One-shot Prompting
with features
Quostiow Please sort the following Cuultural speili Concepts' descending order similarity feature overlap beliven weares, attendance occasion and symbolic meaning.
Cultural Concepts' with Honggailou
Cultural-apoifee Conceptic Cheungam,
Fanurs Hongeettuu
female, bride; Attendance oceasion: wedding, Symbolic Meaning:
luck, happiness
Features Cheongsam: Wearer: female, bride; Attendance occasior: wedding: Symbolic Meaning: good luck, modesty
female: Attendance uccasion: wedding, festival: Symbolic Meaning: China nationalism
Featune Bridal
Answer Eormat: Honggallou and Cheongsam more similar than Honggallou and Bridal veils terms wearer, attendance occasion and symbolic mearing, please answer Cheongsam Bridal veils, otherwise answer Cheongam <Bridal veils. Ansawe:
Qusition: Please sort following Cultural-ppeci Concepts' descending order similarity feature overlap beloveen Cullunal-pecilic Concepts' with jooggagean lerms weares, atlendance occasion and symbolik meaning. Caltunsl-pecit Concepts:: Calceus, Tileus (hat)
Features Jeingiguon: Wearer: femalle, bride; 2. Attendance occasion: wedding: 3. Symbolic Meaning; good luck, happiness Fontures of Calemss 1, Weazer: female: 2. Attendance oocasion: wedding, festival, 3. Symbolii Mearing: China nationaliam
Faatures Pileus (ut): 1. Weater: female, bride; Attendance occasion: wedcing 3. Symbolic Mearing: good luck. modesty Ansaw Formal: jsongiagwan and Cakeus are more similar than Jeangiagwan and Pileus (hal) in lerms weares, atlendance occasion and symbollc mearing, please answee Calcous PPlleus (hal), otherwoo see answer Calceus <Pileus (hat). Ansave: Calceus Fileus (hat)
One-shot Prompting
[w/a names)
Queetoo:: Please sort the following Cuuturalspectfic Cuncepts' in descending order of similarity feature overlap betwein Cultural-spccficc Concepts' with concept A in terms of wester, attendunce occasion and symbolic meaning. Cultural spaefle
B, Fraturvs comcrpt Wewer. female, brid;; 2. Attendannce occasion: wedding 3. Symboolic Meaning good luck, happiness Fanuies comcopt 8: Weaer:: female; Atlendance occasion: wedding. festival, 3, Symbolic Meaning: Chira nationalism Faature concept
C
Weater: female, bride: Attendance occasion: wedding: 3. Symbolic Meaning: good luck, mudesty Amove Fornut: concupt and concept are similar than concept and
Cin of attendamce occasion and symbolic mearing, please concept concept C, otherwise answer concapt cconcept Ansoer.
Table 7: One-shot prompt strategies' examples. Each instance includes a prompt template for evaluating language models, the cultural-specific concepts and the required format for responses.
19

---

Published as a conference paper at COLM 2024
Strategy
Examples
Queaioo: Please sort the following CCultural-peccfic Concepts' in descending order of similarity leature overla--betwenn between
Cultrrasspecffcc Concepts' with loongjawwnn in terms of wearer, attendance occasion and symbolic meanning
oplesaai-piejfe Conapti Calceus, Pileus (hal)
Ansavr Formst: Jeongjagwan and Caleuus are more similar than Jeongingwan and Pileus (hat) in terms wearer, atterdance occasion and symbolic meaning, please snswer Calceus PPileus (bat), otherwise Ansavr: Calceus Pileus (hat)
Calceus <Pileus (hat).
Rotoms: Festures Jeongiegwan: wesned by man, the upper class the Joseon period: weared in daily beadgear; do not have any symbolic meaning.
Features of Calquus: weared by males the upper-class of the Roman Republic and Empine; weared in everyday life; the symbol of ronk or social status the weater.
Keatures of Fileus (hal)) weared by infanley, weared during the Saturnalia festival; is the symbols of Liberlas and the goddess representing liberty.
CoT Prompting
2. Wearer. The leongiawwan and Caloeus are both weared by the males in the upper class, the Jeongiagwan and Pileus (hat)
w/o features
have no common wearers' description.
3. Attendance Occasions: The Jemningwan and Caleuus are both weared in everyday life, the Jeongjagwan and Pileus (hat))
occasions' description.
Symbollc Meaning: The Jeangjagwan
have any common symbolic meaning with Calcous Pileus (hat).
Jeangiagwan' and 'Cakeuu'' more similer than Jeungigwann and (fatt), the answer is Caloeus >Pileus (hat).
Qusdiion Please sort the following "Cultaral-ppec" Concepts' in descending order of similarity feature averlap between Culturalsppssie Concepts' with Honggaitou in terms of weares, attendance occasion and symbolic mesning.
Cultural spwejtt Conopis Cheongsam, m. Bridal veils
Ansaer Fermst: If Honggallou and Cheongsam are more similar than Honggallou and Bridal vells in terms of weares, attendance occasion and symbolic meaning, please answerCheoggam > Reidal veils, otherwise answer Cheongiam <Bridal veils. Amsaer:
Question: Please sort the following "Culurrl specific Concepts' descending order of similarity feature overlap between -ppeural-ppeclic Concepts' with Joongjagsan Colhonat-ppoff Comoepts: Calevus, Pileus (hat) Faatures of loongingaan:: 1.
weares, allendance
and symbolic meaning.
female, bride; 2. Attendance occasion: wedding: 3. Symbolic Meaning: good luck, happiness Fartuee of Caliese Wearer: female; 2. Attendance occasion: wedding, festival, 3. Symbollic Meaning China nationaliam
Features of Pilews (hut): 1. Weazer: female, bride; 2. Attendance occasion: wedding: 3. Symbolic Meaning: good lack, modesty Ansaee Farmat: looaglagwan and Caleuss are more similar than Jeongiagwan and Pileus (hal) in lermi of wearer, allendaicee occasion and symbolic mearing, please answer Calceus Pileus (hat), otherwise answer Calceus <Pileus (hat).
Ansaer: Calceus Pillus (bat)
Rassons: Wearer: The Jeongaagan and Colceus are both weared by the males in the upper class, the Jeongiagwan and wearers' description.
Pileus(hat) have
2. Attendance Occasions: The leongjagwan and Calceus are both weared in everyday life. the Jecoglagwan and Pileus (hat)
CoT Prompting
commen occasions' deseription.
with features
3. Symbolic Mearing: The Jeungiagwan
have any common symbolic meening with Calceus Pileus (bat).
Jeongaawann and 'aCleeus' are more similar than Jeonglagwan' and 'Pileus (faty, the answer is Caleeus PPileus (hat).
Qwstton: Please sort the following "Cultural-ppecfic Concepts' in descending order of similarity feature overlaphetween between Cultrra-specffic Concepts' with Honggaiton in terms of wearer, attendance occasion and symbolic mestiig. Caltural spvcife specffc Concptts Cheongsam, Bridal veils
Faatunes Hougusito::
Wenrer female, bride; 2. Attendance ocessior: wedding: 3. Symbolic Meaning good luck, happirees
Features Cheongsam: 1. Wearer: female: Attendance occasion: wedding, festival; Symbolic Mesring:: China nationalism
Fatture Bridal wells: Wearer: female, bride, Atlendance occasion wedding, Syiboolle Meaning, good luck,, modesly Ansver Formut: Honggaitua and Cheonguam are more similar than Hunggaitou
terms wearen, attendance occasion and symbolic meaning, please snswer Cheongsam > Bridal vells, otherwise answer Cheongsam <Bridal veils. Ansave:
Qustion Please sort following Cultural-specfee Concopts' descrnding order of similarity feature averlap between Culturalsspesfic Concepts' with leongigwwn in terms of wearer, attendance occasion and symbolic meaning. Cultural specftt Conoptti Calceus, Pileus (hat)
luck, happiness Features of Caleuus 1. Weazer: Semale; 2. Attendance occasier: wedding, festival, 3. Symbolic Meaning China natioralism
Features of Jaingingun:: 1. Wearer: female, bride; 2. Attendance occasion: wedding: 3. Symbolic Meaning:
Fartunes of Pileus (hat): 1. Wearer: female, bride; 2. Atlendance occasion: wedding: 3. Symbolli Meaning: good luck, modesty Ansver fermut: Jeongingwan and Calceus are similar than Jeungingwan and Pileus (hat) in terms wearer, attendance and symbolic meaning, please
Calceus Pileus (bal), otherwise
Calceus <Pileus
Ansave: Caleesi PPlees
Rosson: Wearer: Jeungagwan Calceus are both wessed by the males upper class, the Jeongiagwan and
Pileus(hat) have no commoni wearers' description.
2. Attendance Occasions: The leongiagwan and Calceus are both weared in everyday life, the Jecoglagwan and Pileus (hat)
CpT Prompting
have no common occasions' deseription.
(w/o names)
3. Symbolic Meaning: The Jeungaagenn do not have any common symbolic meaning with Calceus or Pileus (bat).
4. So 'Joongagwan' and Cakeus are more similar than Jeongjagwan' and 'Pileus (haty, the answer is Caloeus > Pileus (hat).
Question: Please sort the following 'Cultural specific Concepts' descending order of similarity feature overlap between and symbolic meaning.
Cullural-speclic Concepts' with concept Colturat-ppiiff Concepto: concept E, concept C Faitures of concept A: Wearer female 2. Atfendance occasion: wedding, festival, 3. Symholic Meaning: China nationalism
of weares, atlendance
female. bride: Attendance occasion: wedding: Symbolic Meaning: good luck, happiness
Faatuees of comcopt Features of concept C: 1. Wearer: female, bride; 2. Attendancce pocasion: wedding 3. Symbolic Meaning good luck, modesty Ansaer format: concept A and coucopt B are more similar than concept A and concept in lerms of wearer, allendance occasion and symbolic meaning, plense answer concept >conccppt C, otherwise answer concept B <concept C. Ansuxr:
Table 8: Chain-of- Thought prompt strategies' examples. Each instance includes a prompt template for evaluating large language models, the cultural-specific concepts and the re- quired format for responses.
20

---

Published as a conference paper at COLM 2024
Concept Long-tail Degree Distribution
10a
107
106
Long-tail Degree
10
10a
102
10
0
50
100
150
200
250
285
Figure 7: The longtail degree of all cultural-specific concepts. We calculate the long-tail degree of all cultural concepts through Google search engine.
Category
Large
Middle
Small
IO
One-shot
CoT
One-shot
CoT
10
One-shot
CoT
50.14_..0
59.74.06i
57.23
51.81_000
45.51
55.13
48.66.041
47.92
51,55
Clotbing
llama-13b
49.13_030
50.00.0m
50.000000
39.32-000
49.772000
50.00.000
49.19-1.0
50.00
50.00-000
Ilama-7b
36.15_n30
34.20.0m
33.191000
35.52..000
36.43:0m
31.851000
33.60.000
39.451003
33.67
g.5
79.27-015
65.49
66.991026
54.35_000
42.17.001
43.11ga27
45.82.aur
31.07
24.63
1lama113b
50.00-020
50.00.0m
50.00.a00
50.00-..00
50.00:a.0
50.00-a00
50.00:0.00
50.00
50.00.a.0
Ilama-7b
49.36
44.55 :om
27.24 27.24.0000
48.70
48.70
39.13
37.61
54.13 13:3
46. 46.46:000
39.23 :000
Table 9: The results of all experiments without concepts' name. Based on the experiment in Table 3 concept names are removed from the prompt words.
E Experiments without concept names
We also conducted experiment to remove cultural concept names to determine whether the model can make comparisons based solely on the features the concepts, thus eliminat- ing the influence of the concept names. Through this exploration, we hope to mitigate the long-tail effects caused by concept names. Table shows the results.
F
Error analysis
Based our error analysis of model predictions, we identify three major types of model errors: (1) cultural knowledge errors, (2) answer format errors, and (3) concept candidate order bias. Table 10 provides an illustrative example of each error type.
Figure shows the results our error analysis. We observed that GPT-3.5 performs best in terms of answer formatting, while LLaMA-7b shows poor performance. By removing concept names in questions (see details in Appendix(E), we found that both LLaMA variants (7b and 13b) are sensitive to the order of concept candidates.
21

---

Published as a conference paper at COLM 2024
Error Type Examples
Forward prompting
Question: Please sort the following "Cultural-specffic Concepts' in descending order of similarity feature overlap between symbolic meaning.
"Cultura-specffc Concepts' with Sapanpe wearer, attendance
Cultural-pecifcc Concepts: Burzinqa, Alasho
Labei: Burzinqa Alasho
Answer: Burzinqa Alasho
Knowledge
Reverse prompting
Question: Please sort following "Cultural-specffc Concepts" in descending order similarity feature overlap between
Cullurab-specffc Concepts with Sapanpe in terms of wearer, attendance occasion and symbolic meaning.
Cellural-peciff Concepts: Alasho, Burzinga
Buzzinga
Answer: Alasho Burzinga
Reasons: The large language model maintains consistency in two different candidate cuncept onderings.
Question: Please the following 'Cultural specific Concepts' in descending order of similarity feature overlap between
"Cultural speciffc Concepts'
Honggaitou terms wearer, attendance Caltural-spciffc Concepts: Soubane cloth, Isiagu
symbolic meaning.
Lafei: Soubane cloth > Isiagu
Formal
Ansier: Based the information provided, here comparison the cultural-specffic concepts descending order weares, attendance occasion, and symbolic meaning: Soubane cloth
of similarity 'Honggaitou'
>Honggaifou >siiggu
Reasons: The large language model did not answer the question in the given format.
Forward prompting
following 'Cultural-specif Concepts' in descending similarity overlap between "Cultural specific Concepts' with Honggaitou in terms of wearer, attendiace occasion and symbolic meaning. Caltural-speife Concptt Soubane cloth, Aso oke
Question:
Lakel: Soubane cloth
Answer: Soubane cloth Aso oke
Order
Reverse prompting
Questinn: Please sort the following 'Cultural 'Cultural-specife Concepts' in descending order of similarity feature overlap between of attendance eccasion and symbolic meaning.
CCultrra-speeffcc Concepts' with Sapanpe
Zaltural-specife Concrpts: Aso oke, Soubane cloth
Lakel: Aso oke Souhane cloth
Answer Aso oke Soubane cloth
Reasons: Large language medels questions given but the changes after swapping candidates.
Table 10: Examples of different error types. Each instance is derived from real experimental data, showing different error types.
22

---

Published as a conference paper at COLM 2024
coohinngggh--uit---nauue
ciahhing-phh-wihhoot----
knnwledge
formar errors
order bias
Nums
Nums
Nums
One-shol
CuT
Oniesshol
CUT
One-shol
CUT
cothing-llamal1 Ilamaalla30 fiatufe
clothi ng. withoututtttthuuuaauuuuuu
clothing
800
800
800
Inowledge ermoss
formac eroors
600
600
600
Nums
Nums
Nums
400
400
400
200
200
200
One-shot
CoT
One-shot
CoT
One-shot
CoT
clathing lavihbbbith-ssettrrrss
cloth dethhnngllammm/-wittoootttttttt
coltinggllmmaa..wiaad--momm
800
800
800
knowledge errors
600
order bids
600
600
Nums
Nums
Nums
400
400
400
200
200
200
10
One-shot
CUT
10
One-shot
COT
10
One-shot
CDT
foac-gpt-withh henptwwith-ffatrree
hhod-gat-withhute
hocl-gpt-without-namees
knnwledge
forma: emoos
order olas
Nums
Nums
Nums
One-shot
CUT
One-shot
CUT
One-shot
CoT
food Sodd-llaaalla hoctnnallbwihhhh 3b-with-features
looc-llama3 3bb thout/feature
lood-llama13.. llammmm3b.ithoouttammeee.
800
800
800
knowlodge arrors
farma errora
600
600
600
Nums
Nums
Nums
400
400
400
200
200
200
0
One-shot
CDT
One-shot
COT
One-shot
CoT
fond-llama feeuurts
fendllamms/huuthhhuu---suuuuu
foodllams/huuithotteaaees
800
800
R00
knoaledge
600
order blas
600
Nums
Nums
Nums
400
000
400
200
200
200
0
10
Oni-shul
CUT
10
Orie-shol
CUT
One-hhol
CUT
Figure 8: Analysis of error samples of different models. Our work conducts an error anal- ysis of different models under different experimental groups(with features, w/o features, w/o names).
23


Reviews:
[{"review_id": "zC2wSYnPib", "reviewer": "Reviewer_RbmH", "summary": "The paper addresses the important problem of cultural awareness. It introduces a dataset CUNIT with cultural concepts and their cultural associations and similarities across 10 countries to better facilitate cultural awareness of LLMs. The authors evaluate existing LLMs on this dataset and find that they are still limited in capturing cross-cultural associations.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Soundness of evaluation task design.\n\nWe appreciate your insightful comment on the consistency between our ranking-based testing setup for LLMs and their practical applications in contextualized settings. Aligning with your consideration of LLMs' contextualized settings in practice, our task is designed in a single-turn conversational QA format. We hope to disentangle other confounding factors (e.g., context length and style) and focus on the models' capability to capture the relative cultural unity of concepts. This task could benefit downstream practices requiring \"cross-cultural alignment\" of concepts. For example, our task could be used to assess the cultural-centered analogical reasoning ability of an LLM-based MT system in finding appropriate replacements.\n\nQ2: Soundness of cross-cultural similarity measurement.\n\nWe understand your concern with the ability of LLMs and even humans to quantify the absolute cultural similarity. To clarify, our task requires a model or human to identify a more culturally similar concept of two candidates given the query concept. Measuring the values of cultural similarity between any two concepts is only used to obtain the ground truth for the evaluation of our task.\n \nQ3: Details of human annotation.\n\nThanks for your helpful suggestion. We share the annotation details below and will add a new section of Ethical Considerations in the paper to provide these descriptions.\n \nWe have two stages of human annotation:\n\n●Feature extraction. We employed 12 annotators from China, India, and the US to identify culturally relevant features of each concept using Wikipedia descriptions (full text and ChatGPT-highlighted excerpts). The annotators are generally bilingual (at least in English), with some being multilingual. \n\n●Constrastive matching on testing data. To establish upper-bound performance, we gathered judgments from two additional annotators, both from China and proficient in English. Given the cultural ties in Asia, they possess sufficient cultural knowledge across Asian countries. We also provided a tutorial on our curated cultural concepts to further enrich their understanding for annotation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The dataset is an important contribution to the community, because it can help LLMs to create a better common ground on cross-cultural concepts to improve cultural awareness.\n2.  The analysis is comprehensive and the experiment is well-designed. \n3. The authors also show implication on data-centric factors and suggests the focus on long-tailed cultural knowledge.", "reasons_to_reject": "The following are limitations general to many related problems not just to this paper in particular.\n1. The evaluation method is a bit unnatural, it's a relative ranking task \"sort the following cultural-specific\nconcepts in descending order\", but this is not how these LLMs will be used in practice: they will be more used in contextualized settings like a conversation. I wonder if the paper could discuss more about how to translate this task in a more grounded setting. \n2. One thing I am not sure is whether it's the best practice to quantify the similarity with an actual number and how to use this number. LLMs are not super sensitive to numeric values, and it's also hard for humans to annotate if \"bridal veil\" is 30% similar to \"honggaitou\" or 35% similar.", "questions_to_authors": "Can you say more about the annotation process? Do these annotators come from the same cultural background of the data or they will refer to some official resources?", "extra_scores": {}}, {"review_id": "btQS3tqL3p", "reviewer": "Reviewer_Q9tF", "summary": "this paper proposes a framework  to analyze to what extent English-based LLMs  are able to identify cross cultural concepts,  as a way to extend the current research focusing on the ability of LLMs to be culture aware, which so far have been centered mainly in terms of linguistic variations (usage, style). \nfor this, the authors propose a task and a. dataset that encapsulates manually defined cross-cultural concepts and their features across ten countries, from which a probing task is designed to assess how well decoder-only LLMs detect concept with the highest cross cultural similarity. Results are somewhat expected :  i) GPT is overall winner and ii) results are highly linked to frequency across languages", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Multilingual vs. English-centric monolingual setting.\n\nWe fully agree with you about the importance of multilingual setting in cross-cultural unity. As this study is the first step initiating this research problem, we emphasize the English-centric monolingual setting, mainly because \"English is the most accessible language\" (as discussed in Introduction, page 2) covering diverse, high-quality cultural concepts in a wide range of geopolitical regions, especially under-represented ones (e.g., Nigeria, Thailand). We lean towards data quality compared to language coverage in our first work. A natural next step is to study a multilingual setting by translating our curated data into non-English languages or collecting more non-English data.\n\nQ2: Exclusion of cultural concepts in South America.\n\nThanks for your insightful comment. We explored the accessibility of South American cultural concepts (i.e., clothing and food) on the Spanish Wikipedia. The number of raw clothing concepts remains limited (e.g., a maximum of 9 in Peru). After feature filtering, only 3 concepts were left. Although we could gather data from the Internet beyond Wikipedia, maintaining data quality would be challenging. To address this issue and broaden the impact of our work, we will lauch an open data curation platform that invites crowdsourcing volunteers to share their knowledge on cultural concepts and grow the valuable resources in CUNIT.\n\nQ3: Enriching cross-cultural unity analysis by known cultural structure. \n\nOur intention of exploring the influence of data-centric factors (section 5.2), especially geo-cultural proximity, on LLMs' performance, is well aligned with your consideration of the existing cultural structure in society. Particularly, we analyze GPT-3.5's performance against human annotators in the inter-/intra-group of Eastern-Western cultures. Our initial rationale for considering coarse-level cultural group proximity instead of fine-level country-based proximity is that we hope to obtain a sufficient amount of data for analysis. Despite that, we believe your suggestion of delving into some countries with strong cultural associations, particularly those with large data samples, will enrich our LLM analysis on cross-cultural unity. The idea of exploring concept-level cultural unity toward loanwords is also very inspiring. By launching the open data curation platform for collecting more data, we believe these explorations are promising to study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "the topic is really relevant in terms of the amount of personalization LLMs are expected to provide.\n\nthe dataset is represent a very valuable effort in terms of quality of annotations and expected diversity of the cross culture topics and seems to be well constructed and validated .", "reasons_to_reject": ".", "questions_to_authors": "1.Is just focusing on English centric LLMs a bit limiting for the scope of the problem ? Moreover, it seems counterproductive as people from non-English speaking countries are not, in the first place, willing to consume or interact with information provided by an English-based LLM. In that sense, while it consider the current results relevant, are we missing a very, very big picture by not considering a multilingual setting ? \n\n2. I think this is actually a missed opportunity to increase the contribution of the paper:\n\"We exclude South America due to the limited availability of its cross-cultural concepts on\nWikipedia.\"\nAs we know, most countries in South America uses Spanish as main language, but it is highly fragmented given historical and even geographical reasons. Therefore, having a single common language (Spanish), but that has clear cultural differences across more than 10 countries seems like a perfect scenario for these type of experiments (considering Spanish from Spain could even also make things more interesting as there could be an intercontinental setting)\n\n3. i think one way of complementing the analysis and maybe to obtain more diverse results could bring known cultural structure for the analysis. For example, we know there are clusters that could be exploited , such as Korean - Japanese . Based on their historical ties and relationships (beyond their Chinese linguistic root), there are several terms in Korean that derive from Japanese . Is it possible to adapt the analysis to such setting?", "extra_scores": {"ethics_concerns_details": "."}}, {"review_id": "rzl98UNmCZ", "reviewer": "Reviewer_K2wz", "summary": "New benchmark for identifying cross cultural similarity of traditional concepts. Through semi automatic annotation of fine grained features for food and clothing concepts, concepts similarity is defined by shared features. LLMs are evaluated on distinguishing cross cultural concepts with high similarity versus low, a novel task as a step towards cross cultural communication facilitated by LLMs.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Details of ChatGPT support in feature annotation.\n\nModel: gpt-3.5-turbo-0613\n\nQuality: We conducted a preliminary performance analysis by having an annotator review the model's outputs for 50 randomly selected concepts using their full Wikipedia articles, achieving a recall of ~94.42%. To ensure high-quality feature annotation, both wiki text and model-extracted excerpts are provided.\n\nQ2: Alignment of cultural similarity measurement with human judgements.\n\nThe goal of cultural similarity measure is to get the ground truth of contrastive matching on testing data. Therefore, we directly collect human answers to these cases and compare them with machine-generated ground truth. Table 3 shows a high agreement exists (~88% on average, feature given) between human judgments and ground truth.\n\nQ3: Potential arithmetic issue in feature-based LLM assessment.\n\nThanks for pointing this out! The results of LLMs on test cases with masked concept names are in Table 9, Appendix E, and we will reference this in the main paper. We do not explicitly instruct LLMs to calculate Jaccard Index on the features for predictions, as they are still weak at calculations. The ablation study of feature-only assessment aims to compare with the concept-only one, where we find LLMs perform better with features, suggesting they may lack sufficient knowledge of a concept's cultural aspects.\n\nQ4: Annotator background.\n\nPlease see our reply to Reviewer RbmH Q3.\n\nQ5: Intuition of feature extraction.\n\nWe consider feature extraction mainly based on the nature of material culture[1], which is reflected in its social behavior: \"the way the material is used, shared\"[2]. Inspired by this, we propose three pragmatic feature categories to capture an object's cultural scenario (who, when, where) and its cultural significance (symbolic meaning). Festivals like Jewish holidays are considered as cultural scenario features.\n\n[1] Dant, T. (1999). Material culture in the social world. McGraw-Hill Education.\n\n[2] Material culture. Wikipedia.\n\nQ6: Benefits to other cultural dimensions.\n\nOur curated data, e.g., cultural significance features, can benefit the analysis of LLMs on other cultural dimensions. For example, we can explore any model biases toward certain cultural values by measuring LLM performance on concepts sharing the same cultural significance features (e.g., luck) and comparing it across different features.\n\nQ7: Citation, Writing, and Other LLMs.\n\nWe will improve the manuscript accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Well motivated, interesting and even groundbreaking goal statement, to enable cross cultural understanding and common ground facilitated by LLMs. The proposed task effectively operationalizes one aspect of this seemingly intractable goal.\n\nExpansion of traditional cross lingual semantic matching (lexicon induction etc) to the pragmatic level by focusing on concepts used in similar contexts. \n\nCreative and effective annotation process, where fine grained features are used to quantify concepts similarity for relatively objective quantification. There is very high agreement for the features annotation. \n\nInteresting results showing that LLMs lag behind humans in the purposes task.", "reasons_to_reject": "Part of the annotation process is automated with ChatGPT, but the version is not specified, and neither is any indication of the performance (quality) of that automated step. Namely, it is used to summarize Wikipedia articles to extract excerpts relevant to the defined features; but we don't know what is the recall compared to a human doing this task or if there's a performance bias towards concepts from certain countries.\n\nWhile the idea of contrasive classification by number of shared features is interesting, I am not convinced it is justified to treat it as ground truth. Some concepts may be perceived by a well versed human as very similar despite having only a few shared features, maybe because some features are more important than others. And vice versa. The study here does not corroborate the calculated similarity with human judgments.\n\nUnclear experimental setup: a setting is mentioned where concepts names are masked and only features are available, but no results for this setting are presented in the main paper. Further, it seems odd to use this setup, since it seems like it relies purely on the arithmetic ability to calculate the Jaccard index from the lists of features, and does not require any cultural awareness. This also applies to the setting where both features and concepts are provided, since high performance can simply be explained by mathematical ability. \n\nNo details about the annotators except that they are \"from Eastern countries\". \n\nThe feature lists seem somewhat arbitrary and are not well motivated. For example many Jewish holidays are features, though they're not necessarily very cross cultural.", "questions_to_authors": "\"LLMs are still limited to capturing\" should be \"LLMs are still limited in their ability to capture\"?\n\n\"construct this feature list at the same while annotating\" - redundant \"at the same time\"? \n\nIn the framework of Hershcovich et al. [1], the contribution here is limited to the Common Ground dimension. What about the other dimensions? Would facilitating cross cultural communication require addressing Aboutness and Values too?\n\nIs each matrix in figure 3 symmetric? It may be easier to read if you drop everything under the diagonal to avoid repetition. \n\nDid you try Llama 3? It may be worth adding it to the next version. Also GPT-4. \n\nIn figure 4, \"significantly\" refers to what p value and what statistical test? \n\n[1] Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and Strategies in Cross-Cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6997–7013, Dublin, Ireland. Association for Computational Linguistics.", "extra_scores": {"ethics_concerns_details": "No information on annotators or compensation"}}, {"review_id": "ikP0z05Fpx", "reviewer": "Reviewer_zPi4", "summary": "There has been much research regarding whether LLM can detect or even understand cultural diversity. This paper, instead, tries to understand whether LLM is sensitive to cultural unity, i.e., the common ground across cultures. To do this, the authors introduce a new benchmark dataset CUNIT that contains evaluation examples of a variety of concepts across 10 countries.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Assessment of LLMs' perceptions on each isolated cultural-specific concept.\n\nWe share your concern with the potential shortcuts learned by LLMs for prediction cross-cultural unity. Note that we have the chain-of-thought (CoT) baseline where we ask the LLM to predict a concept's cultural features and then perform our task. One easy way to assess LLMs' perceptions on each concept is to measure the accuracy of LLMs' predicted features compared to our extracted features. We show a better performance of CoT than vanilla prompting, indicating that LLMs have certain levels of perceptions on the concepts.\n\nQ2: Annotator background.\n\nYes, the purpose of collecting human judgements is to provide the upper bounds. We agree with your thought that annotators' performance is associated with their familarity with the testing concepts. We provide the details of annotator background in the reply to Reviewer RbmH Q3. \n\nRegarding the annotation quality, we acknowledge that there may inevitabley exist potential annotation biases given the annotator's cultural background. Follow your helpful sugestions, we will broaden the annotator participants by lauching an open flatform to grow the resources of our CUNIT Benchmark.\n\nQ3: Mistaken the granularity\n\nThanks for pointing this out. Indeed, the \"Large\" and \"Small\" in Table 2 should be flipped. We double-check our results and make sure this label mistake only happens in Table 2, and do not occurs in the other tables. We will correct this label mistake in the next version.\n\nQ4: Figure improvement.\n\nWe appreciate your thoughtful feedback and will improve the figures accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is generally well-written. I appreciate the authors include many examples in the main text as well as figures, which can help the reader to better grasp the idea.\n\n- The contribution is impactful. By benchmarking the LLMs' sensitivity to cultural unity, we can better understand the cross-lingual capability of these models.\n\n- The description of CUNIT is quite clear in Section 3.\n\n- The authors evaluate both LLM's results as well as human performance.", "reasons_to_reject": "There are two limitations:\n\n- Before evaluating LLM's sensitivity to cultural unity, I think it is necessary to first qualitatively or quantitively evaluate whether an LLM knows or understands these concepts. \n\n- The authors did not mention the background of the two human annotators. I assume the authors want to set human performance as the upper-bound performance. However, since the concepts are across 10 countries, I am sure how well these human annotators are familiar with these concepts. So maybe a better human upper-bound performance would be to ask different people who are familiar with the concepts involved in each triplet.\n\nSome minor suggestions are listed in the section below.", "questions_to_authors": "Figure 1 looks nice. However, the caption does not introduce the relations between each concept of interest itself and the features below it (And the features themselves). Therefore readers can get confused (I know they are described in later sections, but since the authors choose to place one figure on the first page to sell the paper, the authors should reduce possible confusion as possible as they can).\n\nTable 2: unfortunately, the authors seem not to mention \\mu and \\delta of the difference of similarity of the final triplet samples anywhere in the paper. I believe these are very important indicators of the difficulty in different granularities.\n\nThe author mentioned that \"A small level of similarity difference indicates a harder contrastive matching task\". If I understand correctly from Table 2, \"Small\" indicates that the Sim. Difference is actually **large**, i.e., a value within ($\\mu$+0.5$\\delta$, 1], which should actually make the contrast task easier. It is quite confusing here. Are the authors mistaken the two granularity, i.e., \"Large\" and \"Small\"?\n\nFigure 3: the information is redundant since the similarity of A and B is the same as the similarity of B and A. The authors can remove the upper triangular part.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a new dataset, CUNIT and an evaluation methodology for testing cultural awareness in LLMs. The evaluation dataset is created by identifying several concepts across 10 cultures (corresponding to countries) and identifying features that correspond to these concepts. Based on these features, the method calculates the cultural association between the concepts, which is used to evaluate LLMs by asking them to identify related concepts. The results suggest that LLMs still do not perform as well as humans on this task.

Overall, this work received positive comments from reviewers, who appreciated the novel task setup and dataset. They also appreciated the clarity in writing and explanation, and the fact that the authors compared LLM evaluation to human evaluation. This dataset will be useful to the community when released. There were some concerns about the two annotators who performed the feature annotation, as they may not have been familiar with all the cultural concepts spanning 10 cultures. There was also a concern about a missed opportunity of using multilingual data and not restricting to English, and the authors mentioned that it could be addressed in future work.
Decision: Accept