CultureLLM: Incorporating Cultural Differences into Large Language Models
Authors: CHENG LI
Mengzhuo Chen
Jindong Wang
Sunayana Sitaram
Xing Xie
Venue: NeurIPS 2024 poster (2024)
Keywords: Culture bias
large language models
fairness
Primary area: fairness
TL;DR: We propose a cost-effective fine-tuning approach to fine-tune cultural specific LLMs

Abstract: Large language models (LLMs) have been observed to exhibit bias towards certain cultures due to the predominance of training data obtained from English corpora. Considering that multilingual cultural data is often expensive to procure, existing methodologies address this challenge through prompt engineering or culture-specific pre-training. However, these strategies may neglect the knowledge deficiency of low-resource cultures and necessitate substantial computing resources. In this paper, we propose CultureLLM, a cost-effective solution to integrate cultural differences into LLMs. CultureLLM employs the World Value Survey (WVS) as seed data and generates semantically equivalent training data through the proposed semantic data augmentation. Utilizing only $50$ seed samples from WVS with augmented data, we fine-tune culture-specific LLMs as well as a unified model (CultureLLM-One) for $9$ cultures, encompassing both rich and low-resource languages. Extensive experiments conducted on $60$ culture-related datasets reveal that CultureLLM significantly surpasses various counterparts such as GPT-3.5 (by $8.1$\%) and Gemini Pro (by $9.5$\%), demonstrating performance comparable to or exceeding that of GPT-4. Our human study indicates that the generated samples maintain semantic equivalence to the original samples, offering an effective solution for LLMs augmentation. Code is released at https://github.com/Scarelette/CultureLLM.

Cul Incorporating Cultural Differences into Large Language Models
Cheng Li"
Mengzhuo Chen Institute of Software, CAS mengzhuo mengzhuo.happy htw.phpao.pppppay...cooomm
Institute of Software, CAS
chenglicat02280gmai.. com
Jindong Wang!
Sunayana Sitaram Microsoft Research
Microsoft Research
j iindong. wang@mi crosoft.coomm
Sunayana wrtaaaammiooooootttooom
Xing Xie
Microsoft Research
xing- xi xie@iiiin crosoft. com
Abstract
Large language models (LLMs) have been observed to exhibit bias towards tain cultures due to the predominance of training data obtained from English corpora. Considering that multilingual cultural data is often expensive pro- cure, existing methodologies address this challenge through prompt engineering culture-specific pre-training. However, these strategies may neglect the knowl- edge deficiency of low-resource cultures and necessitate substantial computing resources. In this paper, propose CultureLLM, cost-effective solution integrate cultural differences into LLMs. CultureLLM employs the World Value Survey (WVS) as seed data and generates semantically equivalent training data through the proposed semantic data augmentation. Utilizing only 50 seed sam- ples from WVS with augmented data, we fine-tune culture-specific LLMs as well as unified model (CultureLLM-Ong rich and low-resource languages. Extensive experiments conducted on 60 culture- related datasets reveal that CultureLLM significantly surpasses various counterparts such GPT-3.5 (by 8.1%) and Gemini Pro (by 9.5%), demonstrating perfor- mance comparable to or exceeding that of GPT-4. Our human study indicates that the generated samples maintain semantic equivalence to the original sam- ples, offering an effective solution for LLMs augmentation. Code is released at https: htap::///ithhne com www.///aareeeeeteeeltteeeeeeeeee
for 9 cultures, encompassing both
1 Introduction
Culture is complex construct that encapsulates various identities, including, but not limited to, language, nationality, region, religion, and gender identity. Cultural bias is prevalent worldwide and refers to the tendency favor specific cultural perspectives, values, and norms, which results in subjective opinions and may offend individuals from other cultures. For instance, according to the World Value Survey [Survey, 2022), Arabic culture believes that are better political leaders than
Work done during Cheng's ' internship at MSRA. Conresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

women, while people in the United States maintain contrary opinion. 3 As large language models (LLMs) [OpenAI, 2023b, Google, 2023] gain prominence, they are reported to exhibit cultural bias and specifically show partiality towards Western culture, as English corpora dominate the training data [Liu et al., 2023b, Cao al., 2023, Masoud et al., 2023, Naous et al., 2023, Wang et al., 2023d, Johnson et al., 2022]. Low-resource cultures are frequently underrepresented due to the insufficient training data available from these cultures. LLMs' cultural bias constitutes a significant bottleneck in human-Al collaboration and considerably impedes AI democracy.
Tackling cultural bias necessitates that a large language model (LLM) acknowledges cultural dif- ferences [Hofstede, 1984]. Kovave et al. Kovač et al. [2023] and Want et al. Wang et al. [2023d] thought LLMs have enough knowledge of all cultures and devised prompt engineering technologies to induce LLMs to exhibit specific cultural perspectives. However, they are not effective, especially in low-resource cultures with limited data. Another line of work pre-trained culturally aware I.LMs and then fine-tuned on specific datasets [Chan et al., 2023, Nguyen et al., 2023h, Pipatanakul et al., 2023, Abbasi et al., 2023, Lin and Chen, 2023]. They require the collection of large-scale pre-training and fine-tuning datasets and extensive computing resources, thus are not affordable to ordinary researchers and cannot handle low-resource culture. To date, training culturally aware LLMs at affordable costs remains a challenge.
In this paper, we propose CultureLLMM, a cost- effective" solution to incorporate cultural differ- ences into LLMs. Technically speaking, Cul- tureI_MM is inspired by the well-known fact that LLMS are inevitably not robust to the style and format of the prompts [Zhu et al., 2023], indicat- ing that we can further leverage such a weakness to further improve the performance of LLMs by enriching the prompts. In particular, we focus on cultural values in this work. As shown in Figure 1, CultureLLM consists of three steps: sampling, semantic data augmentation, and fine- tuning. Inspired by Attiude-Bchavior tency theory [Fazio and Zanna, 1981] which em- phasizes that people's 's opinion is consistent with their behaviors, (WVS) [Survey, 2022] as seed data. Then, we devise a semantic data augmentation approach to generate semantically equivalent samples. The aim to generate semantic equivalent inputs, thus we can get the ground-truth from seed data directly. Finally, CultureLLM is obtained by fine- tuning both the seed and the generated data. WVS is a public opinion poll that contains people's opinions on cultural topics from different countries. To be specific, we select 50 seed samples from WVS, covering topics: "social values", "migration", "security", "science and technology", "religious values", "ethical values and norms", and "political interest and political participation" Using these generated samples and answers from people in different cultures, we fine-tune specific and unified LLMs: specific LLMs are tailored for each culture such as CultureLLM-Ar for Arabic and CultureLLM-Te for Turkish; unified LLMS (CuItureLLM-Onne) are one LLM that fits all cultures.5
WVV
Ogemons From indistihall
Fine-huting datz:
x: [sed geserated dita)-9 same
Sample
y: [Dpttioas) by cultuee
Seee Dom 150 quedtooel
Culture-reiaedd
my made goals in :i:a has possuct
Applications
Fize- tume
-Cffestre
appeer testaniiing witting our Lives besächlar, saxter,
lazguage
Stance deject
Toocsit
Semaniic Data Asgmentation
L.llurel.LM
Theeat
detect. Spazz detect.
Geenaate most
Specific
Calturrt.M-Ar
paceoca?
Culturel
an do you ageen wids eeoeese cestery and more in acsacon and codoeclooy?
Behavior Consis-
CaltueLLMMTr
Unefiot
CaltueLLM-Ooe)
Figure I: Overview of CultureLLM. CultureLLM consists of three steps: sampling, semantic data augmentation, and fine-tuning. Culture-ppecffic and unified Culturel can be fine-tuned.
the World Values Survey
We build 9 specific CulturLLLM and a CultureLLM--One covering both high- and low-resource cultures: Arabic culture, Bengali culture, Chinese culture, English culture, German culture, Korean culture, Portuguese culture, Spanish culture, and Turkish culture. Then we evaluated them on 8 culture-related downstream tasks: offensive language detection, hate speech detection, stance detection, toxicity detection, threat detection, bias detection, abusive detection, spam detection, and an open- generative task. We have 60 test sets of 68, 672 samples in total. Experiments show that CultureLLM fine-tuned on GPT-3.5 significantly outperforms GPT-3.5 by 8.1% and outperforms Gemini pro [Google, 2023] by 9.5% on average Fl score, achieving comparable or even better performance with GPT-4. Our human study of 50 people demonstrates that the augmentation
We respect all opinions in different cultures. 4 "Fine-tuning Culurel.LM only costs $6 via OpenAl API.
'The bound of culture is unclear; use the main spoken language to distinguish cultures [Delanoy, 2020b].
2

---

method can generate semantically equivalent samples. We further interpret the rationale behind its effectiveness by exploring the fine-tuning data size and case studies. Finally, results on Big-Bench Hard [Suzgun et al., 2022] and GSM8K [Cobbe et al., 2021] indicate that CultureL..M is resistant to catastrophic forgetring. Culturel..MM also supports fine-tuning L.LMs of open-source models.
Our contributions are three-fold:
1. We presented CultureLLM, a cost-effective line-tuning solution to build culturall--wware LLMs.
2. We proposed semantic data augmentation, augmentation approach to generate high-quality and diverse training data for L.LMs.
3. We conducted extensive experiments in a wide range of cultures and LLMs, showing that LLMs performs consistently well in all downstream tasks.
2 Related Work
2.1 Cultural Problem and Solution in LLMs
Previous efforts have shown that LLMs exhibit the same cultural problems as in human society, Niszezota and Janezak [2023] proved that GPT-4 can replicate the cross-cultural differences for each personality factor through large-scale experiments. Meanwhile, other works also found that LLMs can reflect cultural bias and dominance in human society [Liu et al., 2023b, Cao et al., 2023, Masoud al., 2023, Naous et al., 2023, Wang al., 2023d, Johnson et al., 2022], e.g., Western culture dominance, since the major training corpus such as Pile [Gao et al., 2020] is in English.
The ideal solution is to improve the cultural awareness of LLMs. There are mainly two types of approach: prompt engineering and pre-training. Kovač et al. [2023], Wang et al. [2023d] thought LLMs as superpositions of cultural perspectives, which can be prompted to targeted cultural per- spectives. while Rao et al. [2023] encoded cultural values in the prompts. Although PE is cheap, its effectiveness is challenged, especially in low-resource cultures where LLMs lack such cultural knowledge due to lack of representation in pre-training data. Another line of research is pre- pre-training and fnn-tuning [Chan et al., 2023, Nguyen et al., 2023b, Pipatanakul et al., 2023, Abbasi et al., 2023, Lin and Chen, 2023] that trains culturally-wwan LLMs for different cultures by collecting large-scale pre-training datasets and then performed fine-tuning for better alignment. While they achieved great performance, this approach is too expensive and time-consuming, thus it is difficult to apply to more cultures and countries. They still suffer from low-resource culture problem where the pre-training data difficult to collect. MaLA-500 [Lin et al., 2024] trained new LLM on Llama 2 to cover 534 languages, which is resource intensive.
2.2 Data Augmentation for LLMs
Human- annnattdd data are high-quality but expensive. Due to the strong generation ability of LLMs, many works focused on leveraging data augmentation for LLMs. Yu et al. [2023], Liu al. [2023a] used LLMs to augment the math data and then fine-tuned with those data. Li et al. [2023) synthesized data with two designed modules: eeffenugmetttionn and self-curation. Chen et al. [2024] introduced self-play mechanism, where LLM generates its own training data from its previous iterations, refining its policy by discerning these self- generated responses from those obtained from
data. There are also other uses for synthetic data, such as knowledge distillation [Wang et al., 2023c] and improving text embedding tasks [Wang al., 2023a]. Our data augmentation approach also adopts I.LMs for data generation, but we add controllable modules such as template editing, synonym replacement, and semantic filter ensure the diversity and semantic equivalence of the generated samples. It can also be used as general augmentation method in other applications.
Efforts in cultural datasets [Nguyen et al., 2023a, Fung al., 2022] focus cultural common sense and norms. However, they generate data from only the English or Chinese corpus and thus may contain cultural bias toward other cultures. In contrast, World Values Survey (WVS) [Survey, 2022] a large-scale pool that contains answers from people of different cultures, thus providing more objective cultural values from specific cultures.
This work is also related to value alignment [Ji et al., 2023, Shen et al., 2023, Yao al., 2023] to align the values of LLMs with human's by designing algorithms for value measurement and behavior
3

---

alignment. In contrast, this work primarily emphasizes value understanding with the potential to be extended for value alignment. For instance, semantic augmentation can be used generate training data for alignment-related tasks.
3 CultureLLM
3.1 Overview
Cultural differences are prevalent in various cultures and backgrounds, leading impact on outcomes in downstream applications such as hate speech and biased language. To address the gap between low-source cultural data collection and its wide applications,
design Culturel.IMM by fine-tuning an L.M on data generated by our novel semantic data augmentation approach leveraging the sensitivity of LLMs on prompts [Zhu et al., 2023]. Figure presents an overview of CultureLLM, where the first step is to sample subset of data from an existing World Value Survey (WVS) [Survey, 2022] that represents different opinions (answers) towards the same value questions given by native users. The adoption of WVS is inspired by 1981], which emphasizes the strong relationship between attitude and behavior. Therefore, WVS serves as an ideal seed for data augmentation." After sampling, the second step is to generate augmented data using our proposed semantic augmentation approach (Section 3.3) and then fine-tune Cultuel.LMM for each specific culture such as LM-rr for Arabic culture and Culture...M- Tr for Turkish culture.
Consistency theory [Fazio and Zanna,
Generally speaking, we use Da = {(T (())) to denote the seed and D'2 = g(Da) =
{(''), u)) y-11 as the augmented data with g-) the augmentation algorithm. Note that the ques- tion x here is the same in all cultures in WVS and d is the cultural index denoting different answers to the same question æ. For example, for question r='Do you agree with on the whole, men make better political leaders than women do?", the answer y = Disagree if d = English; and y = Strongly agree if Arabic. Therefore, we only augment the question xr to be x' but retain the same opinion y as the original x. We also denote vanilla LLM and CultureLLM as and /", respectively. Then, denoting as the loss function, our learning objective is formulated as: f* arg miny E (e,, (e), (()(z))))
3.2 Sampling
The sampling process should follow two princi- ples: 1) cover as many cultural topics as possi-
Sample (8,,)))
Seed
ble and 2) sample questions that can be clearly
v
yf from differet cultures
you squies with Gan my naán gonza in
answered by LLMs. Based the two princi- ples, we manually select n = 50 questions and rewrite them in the Question-Answer (QA) for- mat, covering 7 topics, namely social values, security, science and technology, religious val- ues, ethical values and norms, political interest and political participation, and migration. The details can be found in Appendix B.I.
CPT- Rephrase
2 Semontic Piltor
Coraaded surple
Senetially Tepiident Serande
up gennticase pizzs
5itt curee with Making partats
petnary 20829 26d62
peimory otgjortione titee
poac to being poide
Kioo you sgres with the poai to beiog
ay
centext 1ife
Sofee geal wine!
Sentence Pursing
pou aquies
Semarie Temilaaa
my pareeti
alne?
3.3 Semantic Data Augmentation
16he agris sith 2al Bilg
4 Contect awvrr Synonyms with Semartic Filte 3
Samples from WVS are not enough fine-tune, which can be augmented by our semantic aug- mentation approach. In a formal sense, semantic augmentation retains the original ground-truth opinions (ya) from different cultures and only generates semantically equivalent questions (x). naive augmentation approach is to directly
Left Semantic Templates Generaton
Right Intactt Sentence Generation
Figure 2: Details of semantic data augmentation. First, semantic templates are generated via rephras- ing, semantic filtering, and sentence parsing. Then, training samples are generated by context-aware synonyms replacement and semantic filtering.
"WVS is one feasible option for seed and other surveys can also be used. But the cultural survey data are extremely rare and the WVS could be the most comprehensive one. More details about WVS are in Appendix B.I.

---

use strong LLMs such as GPT-4 to generate new samples [Walters and Wilder, 2023], which
could introduce mode collapse, as generation quality can only he controlled by prompts. Furthermore, since L.LMs could suffer from cultural bias, directly generating cultural data using prompts could lead to unexpected or even erroneous results.
As shown in Figure 2, the augmentation consists of two stages: semantic template generation and intact sentence generation. The first stage generates several semantically equivalent but stylistically different sentences and parses them into semantic templates. The second stage then generates samples by replacing certain words in the semantic templates. Such an augmentation can naturally introduce more diversities: The first stage increases sentence-level diversities and the second improves the word-level diversities.
Semantic Template Generation This stage generates semantically equivalent question templates T= (4) based on x E Da- The generation process is nontrivial since there are two challenges ahead: 1) the naturalness and diversities and 2) the semantic preservation. We solve the first challenge by using GPT-4 as the generator with certain prompts to ensure naturalness and diversity. Then, we solve the second challenge by introducing a semantic preservation filter P to measure the similarity between the original and generated sentences.'
We first use the prompt "Could you please generate [n] sentences that (1) have different sentence structures and (2) have the same meaning with the following sentence: x;" to generate sentences using GPT-4. Then, we denote the embedding of the original sentence and the generated sentences as = P(x)) and = P((), respectively. Then we compute their similarity score = cos(2, 2)). If e passes the threshold value т, the generated sentence will be reserved:T = {tu cos() P(x,)) > T). Vz, E Da. Specifically, for sample "Do you agree with One of my main goals in 1ife has been samples using GPT-4, which are then go through the semantic filter P to eventually retain k(k S m) semantically equivalent sentences, e.g. "Do you agree with Making my parents digmified has always been one of my primary objectives in life?" and "Do you agree with The goal to bring vanity my parenta has been a central life goal of mine?"
make my parents proud?",
generate m
To diversify the generated data, we parse the n sentences to find the appropriate components to replace, which construct the templates. For efficiency, we use NLTK [Loper and Bird, 2002] to find replaceable words, such as adjectives, adverbs, nouns, and verbs. The semantic templates are like "Do you agree with The [x] to bring [x] my parents has been [x] life [x] of mine?" where "[x]" is the replaceable part. In total, we generate k templates for each sample Iy Dz.
Intact Sample Generation This step is to replace synonyms in templates generate fine- tuning samples. We apply GPT-4 to generate context-aware synonyms the templates and randomly replace some of them. To further preserve semanties, we also use the seman- tic preservation filter. After filtering, we generate m samples for each template t; E T, and get n' = mak samples for all Tj E Da in total. For example, intact samples for template "Do you agree with The [x] to bring [x] to my parents has been a [x] life [x] of mine?" could be "Do you agree with The goal to bring pride to my parents has been a central life goal of mine?" and "Do you agree with The hope to bring vanity to my parents has been central life goal of mine?" Our human study in Section 4.6 shows that augmentation can generate high-quality and semantically equivalent sentences.
3.4 Fine-tuning
Since culture is complex construct, by countries) in WVS to represent broader cultures and arrive at set of 9 cultures total. In cases where language is spoken by more than geographical culture, we pick representative countries and use the average of all answers as groundtruth. Our final set of cultures represented as described above is Arabic (for which we select Jordan and Iraq), Spanish (for which we select Mexico and Argentina), Bengali, Chinese, English, German, Korean, Portuguese, and Turkish.
languages spoken by geographical cultures (represented
7 For computational efficiency, we BERT embedding P while other models also be used.
5

---

Finally, CultureLLM is obtained by fine-tuning an LLM on the combination of the seed and the generated data. Specifically, we fine-tune two types of LLMs: 1) culture-speciffe LLMs for each language such as L.ltureI.L.MAr and Culturel. MM-Bnn and 2) one unified L.LM for all languages, denoted as Muleurel.LM-One. Culture-specific L.LMs are tailored by setting d, namely, {Da, D'))- On the other hand, CultureLLLM-On is trained on all datasets: (Da, D'a ) deall language to serve as a unified LLM for all cultures. Note that since all languages have the same input question æ but different answers y, we need to manually write different prompts in the instruction to distinguish them. For example, we add "You are an Arabic chatbot that knows Arabic very well before Arabic samples. CultureLLM can be used in cultural downstream applications. In the following, use CultureLLM to denote specific LLM and CultureLLM One for unified LLM.
Note that the WVS is all in English, where we focus on cultural differences in opinions regardless of their native language. Thus, we do not perform fine-tuning for other languages due to the shortage of their training data and rely cross-lingual transfer. Multilingual tasks for cultures can still benefit from fine-tuned models in English, as models can learn the basic values from the opinions [Moussaid al., 2013, Jin et al., 2023]. Our experiments in Section 5.1 further demonstrate that fine-tuning on English data can outperform fine-tuning on native data that are translated from the original English version.
Remark:
4 Experiments
We fine-tuned a CultureLLM-ONE and 9 specific CultureLLM for 9 languages: Arabic (Ar), Bengali (Bn), Chinese (Zh), English (En, United States), German (De), Korean (Ko), Portuguese (Pt), Spanish (Es), and Turkish (Tr). These cultures are diverse and represent both high- and low-resource regimes and thus can serve as representative evaluation.
4.1 Setup
Datasets. We adopt culture-related public datasets in specific languages for evaluation. In total, we have 59 test sets, covering 9 languages and containing 68, 607 test samples. We test 56 binary classification and multi-classification tasks to detect: offensive language, hate speech, stance, toxicity, threat, bias, abusive, and spam in zero-shot evaluation. For example, ask LLMs to judge whether the sentence contains offensive language, hate speech, or biased speech. Details are shown in Appendix B.2. Furthermore, we generate open-ended generation dataset for evaluation in Section 4.3.
Baselines and details. We fine-tune CultureLLM using the GPT-3.5 (0613) [OpenAI, 2023a] fine- tuning API due to its efficiency and compared with two state-of-the-ar LLMs, namely Gemini pro [Google, 2023] and GPT-4 (1104) [OpenAI, 2023b]. We further compare with cultural spe- cific pre-trained models SeaLLM [Nguyen et al., 2023b]. TaiwanLLM [Lin and Chen, 2023] and CultureBank [Shi et al., 2024]. We also compare this with retrieval augmentation (RAG), which enhances LLMs by searching for related information and adding it to context [Lewis et al., 2020]. To implement RAG, search for information about each culture on Wikipedia and append them in system prompt, as detailed in Appendix C.3. Finally, we fine-tuned Culturel.LM using Llama-2-70b- chat ['Tourron et al., 2023] the base model for reproduction (Section 5.3). As for prompt setup, since our goal is to make L.MSs better align with people from different cultures, we add a system prompt "You are an [x] chatbot that knows [x] very well" where [x] is certain language before each input. For metrics, we use macro Fl score for all tasks except for CValues [Xu et al., 2023] where we use the automatic evaluation script provided by the paper. For data augmentation,
we set k = 5, m 2, and - 0.8. Evaluation prompts are in Appendix C.2.
4.2 Main Results
We present the average results for each culture and task in Figure 3(a) and more detailed results shown in Appendix D.I. Our conclusions are as follows. First, both specific and unified CultureLLM achieve great improvement over other approaches, and specific CultureLLM achieves the best performance. Concretely speaking, CultureLLM significantly outperforms GPT-3.5 (by 8.1%),
*Results on RAG are not shown since they are close to GPT-3.5. Since the metrics are not the same (e.g.. accuracy for Calluss and Fl for other tasks), normalized each and then averaged them.

---

620
Res Ce
LE
Hate
Thase
Staac
020
Potageos
Calie
(a) Main results.
(b) Ablation study.
Figure 3: (a) The main results averaged by cultures (left) and by tasks (right). Both CultureLLM and L.ltureI..MM-One significantly outperform Culturel.LM and Gemini with CultureL..M achieving the best performance comparable GPT-4. (b) Ablation study. '+WVS' denotes the fine-tuned models using only the 50 samples from WVS, '+WVS+a' denotes fine-tuning using the WVS samples and the generated samples in step of our data augmentation (i.e., using only GPT-4 to generate), and '+WVS+a+b' denotes the complete process of our algorithm.
Gemini (by 9.5%), and RAG (by 7.94%), achieving performance comparable to GPT-4 and even better on some tasks. Second, CultureLLM Ln- exceeds GPT-3.5 by more than 4% on 59 tasks, while inferior culture-specific models, suggesting that single LLM might not be the best solution to solve cultural tasks with low resources, since data from different cultures could intertwine with each other. Third, in terms of cultures, CultureLLM achieves the best performance in English, Chinese, and Spanish cultures while showing no obvious improvement in Korean culture, where all four models have similar performance. We infer that the reason could be that these base models have less exposure to Korean culture.
Then, we analyze the performance both low-resource and high-resource language tasks. As shown in Figure 7, CultureLLM shows excellent performance in both types of tasks and outperforms GPT-4 on a large scale in high-resouree tasks. Finally, we evaluated an extremely low-resource culture, Greek, in Appendix D.6 and compared CultureLLM with other cultural-specific models SeaLLM [Nguyen al., 2023b], TaiwanLLM [Lin and Chen, 2023] and CultureBank [Shi et al., 2024] in Appendix D.2, which shows that our CultureLLM can also improve performance. The correlation with the WVS data is in Appendix D.3, showing that the performance improvement does not come from the seed data in WVS.
4.3 Results on Open-ended Generation Tasks
To evaluate the performance of Cul- tureLLM on open-ended tasks, we construct dataset using GPT-4 containing 65 open-ended questions. which cover the seven topies in WVS. The prompt setting for dataset generation can be found in Appendix C.2. We evaluated the outputs of GPT-3.5 and CultureLLM using Gemini Pro?. We also devised metric WinRate (SCulture
Table 1: WinRate results on generation tasks.
Culture
WinRate
.215
.369
.155
.922
.62
.615
.669
215
-062
SChotGPT) /65,, where represents the number of acceptances by Gemini Pro. Positive WinRate means CultureLLM wins GPT-3.5 and vice versa. As shown in Table CultureLLM performs better than GPT--355 on 8 out of cultures, demonstrating its effectiveness in generation tasks.
4.4 Ablation Study
We evaluate the effectiveness of our semantic data augmentation approach by comparing it with the following variants: GPT-3.5, CultureLL.M (WVS), Culturel..M (WVS+a), and Culturel..M (W (WVS+a+b), where CultureL.LM (WVS) denotes the fine-tuned models using only the 50 samples from WVS, CultureLLM (WVS+a) denotes fine-tuning using 50 WVS samples and the generated samples in step 1 of our data augmentation (i.e., only using semantic templates), and CultureLLM (WVS+a+b) denotes the complete process of our algorithm. Note that "WVS+a" denotes the naive baseline of only using GPT-4 to generate samples.
We do GPT-4 to judge because it may prefer the response from GPT series models.
7

---

Persledit Elversity
1G:0 0.311
14.78
6.7
Tuckid
Clltand (Ba By
0.65
6.0
100
Delemaro
wa
0.00
638
631
649
0.45
800
30
Cubrce
Number of dats for Fine-tuning
(a) Influence of language.
(b) Influence of data number.
Figure 4: (a) Results on different numbers fine-tuning samples with perplexity score and diversity gain above. (b) Results of fine-tuneing on English (En ft) and local languages (local ft). It shows that fine-tuning on English outperforms fine-tuning on local languages.
Figure 3(b) shows that fine- tuning using the 50 seeds in WVS can inconsistently improve and impair performance on different tasks such the decrease in Korean tasks. While WVS data are high quality, we see gains with our generated data which leads to improvements on most tasks. The average performance on Korean tasks also improves by Culturel.LM Figure 3(b) also demonstrates that the two steps in our semantic data augmentation approach are useful and necessary. Ablation for CuluureLLM-One is in Appendix D.4, which also shows the effectiveness of our approach.
4.5 Effectiveness Analysis
We analyze the effectiveness of LultureLLM by controlling the number of generated data, computing the perplexity score, and presenting case studies.
First, we analyze the impact of the generation size. As illustrated in Chen al. [2024], the diversity and quality of datasets are important in training LLMs. Hence, infinite or too many generated samples might hurt the performance due to possible mode collapse. In this section, we control the number of generated data and empirically analyze its impact. Specifically, we fine-tune 4 CultureL.MM with {o, 100, 500, 1000) generated samples appended to the original WVS data set. As shown in Figure 4(b), as the number of fine- fine-tuning data increases, performance across most of tasks get improved; but when the number greater than 500, performance on tasks declines.
Then we analyze the diversity of the generated data by computing two metrics: perplexity [Marion al., 2023, Wang et al.. 2023b] and diversity gain [Bilmes, 2022] (Appendix C.I), as shown in the upper right in Figure 4(b), where we observe the consistency between these two metrics and the fine-tuning performance: the 500 generated data lead the best perplexity and diversity gain. The reason may be that these 500 samples are enough for GPT-3.5 to understand the knowledge of seed data, and more samples can cause overfitting and decreased performance. Additionally, although the augmentation approach only generates different samples by varying sentence and word styles, the diversities can also be increased. This suggests that variations in samples can improve the diversity of datasets.
As in the cases shown in Figure 8, responses from GPT-3.5 often analyze input from multi- ple perspectives and call on be respectful and kind, rather than providing clear and straight- forward opinions. In some cases, GPT-3.5 says that it cannot determine the intentions behind the sentence without context, while CultureLLM provides clear opinions most of the time. The reason behind this may be that we fine-tune Cul- tureLLM to learn opinions from specific culture, so that it can be more aligned with the corresponding culture when faced with cultural differences or cultural conflicts. However, GPT-3.5 is aimed to serve people from different cultures. Thus, it prefers to give neutral response to not conflict with any
Table 2:
The semantic similarity of generated sam- ples and seed samples are judged by 50 human participants, GPT-4 and Gemini Pro. The scores range from 5, where represents "definitely not" and 5 represents "perfectly".
Evaluator
Human
GPT4
Gemini
AVG
Rating 4.60 (0.28)
4.99 (0.09)
4.93 (0 26 4.84

---

cultures. However, the worst consequence is that it cannot provide useful responses to the problems related to cultural differences.
4.6 The Effectiveness of the Augmented Data: A Human Study
We analyze the effectiveness of the augmented data through human evaluators. We hire 50 people who have high exposure to English (i.e., majoring in English) to check if our generated sentences are semantically equivalent to the seed data. The information of the participants and the training procedure are in Appendix F. We sample 100 pairs of (seed, generation) samples and let each participant rank their similarities by giving a score of 1 to 5, with 5 representing the most similar. We also use GPT-4 and Gemini Pro as evaluators. The average results in Table 2 demonstrate that the semantic similarity passes 96.5%, implying that our augmentation approach can increase the quantity while retaining the similarity.
We also conduct experiments on generation tasks. Figure 8 shows the responses of GPT-3.5 and Culturel.LMM in four different cultures. The results show that CultureLLM can generate more accurate, direct, and useful responses than GPT-3.5. To be specifie, GPT-3.5 always generate long responses, which do not give useful information and just call on to be respectful, while CultureLLM give accurate and direct responses. This is very important for user experience.
Discussion
5.1 Augmenting Multilingual Data vs. English Data
CultureLLMM are fine-tuned on English data, since the training corpus of LLMS such as the GPT series are mostly in English and English may be the choice for LLMs to understand the opinions of other cultures. What about the performance of LLMs fine-tuned in culturally specific language? We also fine-tuned GPT-3.5 [OpenAI, 2023a] on multilingual data that are translated from English data and compare with CultureLLM. The results are shown in Figure 4(a), indicating that the models fine-tuned in English perform better than the models fine-tuned in other languages. The reason behind this may be the model's 's inherent capabilities in English have been shown to be superior [Ahuja et al., 2023] than other languages, which again emphasizes the importance of collecting large-scale data for pre-training. This study demonstrates that, in low-resource settings without collecting scale training data, the augmentation approach could be useful for fine-tuning.
5.2 Fine-tuning vs. Forgetting
potential dilemma is that fine-tuning an LLM on specific tasks might face catastrophic forgetting of its original capabilities. In this section, we explore the forgetting of CultureLLM in two general datasets: BIG-Bench-Hard (BBH) [Suzgun et al., 2022] and GSM8K [Cobbe et al., 2021]. BBH contains 21 tasks covering both semantic understanding and logical reasoning tasks. GSK8K is widely used data set to evaluate mathematical ability. For BBH, we sample 100 samples for each task to test, due to cost savings. We compare each CultureLLM with the GPT-3.5 baseline model in Figure 5a)). The results show that Culturel.LMM does not decrease performance in most benchmarks and can even improve their results, such as on BBH. This suggests that there might be some latent relations between the cultural data and the general benchmarks, thus fine-tuning on cultural data can benefit general reasoning abilities.
5.3 CultureLLM on Open-sourced LLMs: Llama2
Although all main experiments in this work are performed using the OpenAI GPT-3.5 fine-tuning API [OpenAI, 2023a] due to its efficiency and simplicity, our CultureLLM also supports fine-tuning on
LLMss for better quality control and reproducibility. In this section, we show an initial experiment using Llama2-700 7-0-chht as the base model to fine-tune a CultureLLM- LLLL-LMM-2222200000 The results Figure (b) show that Culturel.MM-Lamma,
outperforms the base Llama model by 2. 2.17% on average, showing the effectiveness of fine-tuning CultureLLM on open-source models. The details of fine-tuning and more Ilama2 results are in Appendix E. The results indicate that CuitureL is general approach to improve the ability of LLMs to understand the culture.

---

Liuaa CaluralM
Seam
A
071
8-40
G20 (
0770
075
Hand
(a) Catastrophic forgetting
CultureLLM-Lllma22
Figure 5: (a) Analysis catastrophic forgetting BBH and GSM8K. The red line denotes the results GPT-3.5. For BBH, we show the average results of 21 tasks in this figure. The x-axis represents models and the y-axis represents performance. (b) CultureLLM-Llama- averaged by cultures (left) and by tasks (right), which outperforms the vanilla Llama model by 2.17% on average.
5.4 Implication and Societal Impact
In essence, recognizing and valuing cultural differences is paramount for the enrichment of our global community. Embracing diversity stimulates innovation and creativity, contributing to the development of novel ideas and solutions. Our work contributes solving the cultural difference problem in LLMMs and tackling the problem of data scarcity in low-resource cultures. The limited availability of data from these cultures hinders understanding and addressing specific needs and concerns. For example, the lack of representation in datasets may perpetuate biases and disparities, hindering the development of inclusive technologies and services. Our approach represents an effective and resource-saving saving method to bridge the data gap in low-resource cultures, empowering these communities and enabling more accurate, inclusive, and impactful decision-makir processes.
6 Conclusion and Limitation
Cultural difference is essential to the prosperity of the world. In this paper, we proposed CultureLLM cost-effective solution fine-tune culture-aware L.LMs. We sampled a small number (50) of samples from the World Value Survey and then generated augmented data through our novel semantic data augmentation.
59 datasets on cultures, LultureLLM outperformed GPT-3.5 and Gemini with comparable or even better results than GPT-4.
This work has the following limitations. First, due resource and time constraints, we did implement CultureLLM on large-scale open-source models. Second, we only adopted classification tasks for evaluation since multilingual generative tasks are expensive for automatic evaluation. Finally, the sample diversity is only in sentence and word levels. In the future, we plan to add more diversities enrich the generated data.
Disclaimer
This paper leveraged GPT-4 to generate sentences and synonyms, whose quality were manually checked to ensure responsible usage. Throughout this paper, the authors remain neutral towards the opinions from all different cultures and respect their diversities. The human study was conducted following local laws and regulations.
References
Turkish Spam VO1. http://db..rgg/0.444///WCWWW
UCI
Machine
Learning
Repository, 2019.
DOI:
Mohammad Amin Abbasi, Arash Ghafouri, Mahdi Firouzmandi, Hassan Naderi, and Behrouz Minaei Bidgoli. Persianllama: Towards building first persian large language model. arXiv preprint arXiv:2312.15713. 2023.
10

---

Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, al. Mega: Multilingual evaluation of generative ai. arXiv preprint arXiv:2303. 12528, 2023.
aimansnigdha. Bangla- Sangla-abusive-com wwwp:auiii----meeed--eeeeee http://gghhll.ooaaammmnigggaaaaa Comment-Dataset, 2018
Miguel Á Álvarez-Carmona, Estefania Guzmán-Falcchs Manuel Montes-y Gómez, Hugo Jair Es- calante, Luis Villasenor-Pineda, Verónica Reyes-Meza, and Antonio Rico-Sulayes. Overview of mex-a3t at ibereval 2018: Authorship and aggressiveness analysis in mexican spanish tweets. In Notebook pupers of 3rd sepin workshop evaluation of human language technologies for iberian lunguages (ibereval), seville, spain, volume 6, 2018.
Arjun Appadurai. Modernity at large: Cultural dimensions of globalization. U of Minnesota P. 1996.
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. Semeval-2019 task 5: Multi- lingual detection hate speech against immigrants and women in twitter. In Proceedings of the 13th international workshop semantic evaluation, pages 54-63, 2019.
Shiladitya Bhattacharya,, Siddharth Singh, Ritesh Kumar, Akanksha Bansal, Akash Bhagat, Yogesh Dawer, Bornini Lahiri, and Atul Kr. Ojha. Developing a multilingual annotated corpus of misogyny and aggression. Praiiinnnn...oneeeeeeeeeeeee
Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, pages 158-168, Marseille, France, May 2020. European Language Resources Association (ELRA). URL ttatttts://aaaaaatttittt
org/2020........ wttp://2....aa-111..... trac-1.25/ 1.25/.
Jeff Bilmes. Submodularity in machine learning and artificial intelligence. arXiv preprint arXiv:2202.00/32. 2022.
Cao, L Zhou, S Lee, Cabello, M Chen, and D Hershcovich. Assessing cross-cultural alignment between chatgpt and human societies: An empirical study, arxiv. Preprint posted online March, 31, 2023.
Tommaso Caselli, Valerio Basile, Jelena Mitrović, Inga Kartoziya, and Michael Granitzer. I feel offended, don't be abusive! implicit/eppleit messages in offensive and abusive language. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6193-6202, 2020.
Çağn Çöltekin. corpus of turkish offensive language on social media. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6174-6184, Marseille, France, 2020. URL https://www aclweb. org/
Alex J Chan, José Luis Redondo García, Fabrizio Silvestri, Colm O'Donnel, and Konstantina Palla. Harmonizing global voices: Culturally-aware models for enhanced content moderation. arXiv preprint arXiv:2312.022
2023.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play finettuning converts weak language models to strong language models. arXiv preprint arXiv:2401.0133 2024.
Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Abdelali, Soon-gyo Jung, Bernard J Jansen, and Joni Salminen. A multi-platforn arabic news comment dataset for offensive language detection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6203-6212, 2020.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
daan Korean hatespeech dataset.
satttasaateepeeccccaattte,e dataset, 2020.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. Proceedings of the Ith International AAAI Conference Web and Social Media, ICWSM 17,, pages 512-515, 2017.
11

---

Angel Felipe Magnossao de Paula and Ipek Baris Schlicht. Ai-upy at iberlef-2021 detoxis task: Toxicity detection in immigration-relater web news comments using transformers and statistical models. arXiv preprint arXiv:2/ 1.04530, 2021.
Rogers de Pelle and Viviane P. Moreira. Offensive comments in the brazilian web: dataset and baseline results. 2017.
Werner Delanoy. What is culture. The Cambridge handbook of intercultural communication, pages 17-34, 2020a.
Werner Delanoy. What Is Culture?, page 17-34. Cambridge Handbooks in Language and Linguistics. Cambridge University Press, 2020b. doi: 10.1017/978 108555067.000 .003.
Russell H Fazio and Mark P Zanna. Direct experience and attitude- behavvor consistency. In Advances in experimental social psychology, volume 14, pages 161-202. Elsevier, 1981.
Elisabetta Fersini, Paolo Rosso, Maria Anzovino, et al. Overview of the task on automatic misogyny identification ibereval 2018. Ibereval@ sepin, 2150:214 22-28, 2018.
Yi Fung, Tuhin Chakraborty, Hao Guo, Owen Rambow, Smaranda Muresan, and Heng Ji. Norm- sage: Multi-lingual multi-cultural norm discovery from conversations on-the-fly. arXiv preprint arXiv:2210.08644 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0002,, 2020.
Google. Gemini. http:////wrrmmooooooooooo//////e 2023.
HASOC. Hasoc2020.
2020.
Geert Hofstede. Culture's consequences: International differencee in work-related values, volume 5. sage, 1984.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0990 09688. 2021.
Husain. Osact4 shared task on offensive language detection: Intensive preprocessing-batt bbsed approach. arxiv 2020. arXiv preprint 1XXi:20005.07977 2020.
Jaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhongaao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey, arXiv preprint 2110 19852, 2023.
Zhuoren Jiang, Zhe Gao, Guoxiu He. Yangyang Kang, Changlong Sun, Qiong Zhang, Luo Si, and Xiaozhong Liu. Detect camouflaged spam content via stoneskipping: Graph and text joint embedding for chinese character variation representation. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing (EMNLP2019). ACM, 2019.
Yigiao Jin, Mohit Chandra, Gaurav Verma, Yibo Hu, Mummun De Choudhury, and Srijan Kumar. Better ask in english: Cross-lingual evaluation of large language models for healtheare queries. arXiv e-prints, pages arXiv-2310, 2023.
Rebecca I. Johnson, Giada Pistilli, Natalia Benédez-Gonzáfen Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, and Donald Jay Bertulfo. The ghost in the machine has an american accent: value conflict in gpt-3. arXiv preprint arXiv:2203.07788 2022.
Sanaa Kaddoura and Safaa Henno. Dataset of arabic spam and ham tweets. Data in Brief, 52(10990):
4, 2024.
Kaggle. Jigaa--mltilingualttiicci
coallinggualtooxiiity eda-models, 2019.
12

---

Kaggle. 5k turkish tweets with incivil content.
turkish-tweetss weeess-wth-ineivil-ooo 2021.
Kaggle. turkish offensive language detection. https//www...gggt.coom//sttttttr
offenssieellnnuuge-eeeeeeii 2022.
Habibe Karayigit, Çigdem Inan Acı, and Ali Akdağh. Detecting abusive instagram comments in turkish using convolutional neural network and machine learning methods. Expert Systems with Applications, 174:1 14802, 2021.
Grgur Kovač, Masataka Sawayama, Rémy Portelas, Cédric Colas, Peter Ford Dominey, and Pierre- Yves Oudeyer. Large language models as superpositions of cultural perspectives. arXiv preprint arXiv:2307.07870, 2023.
Jean Lee, Taejun Lim, Heejun Lee, Bogeun Jo, Yangsok Kim, Heegeun Yoon, and Soyeon Caren Han. K-MHaS: A multi-label hate speech detection dataset in Korean online news comment. In Proceedings of the 29th International Conference on Computational Linguisties, pages 3530- 3538, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. URL https: hatpa/aaaanttoooy....ooo . wtrgg2222..om 20gg/nnn.cl..ngg 1. 311.
Joao A Leite, Diego F Silva, Kalina Bontcheva, and Carolina Scarton. Toxic language detection in social media for brazilian portuguese: New dataset and multilingual analysis. arXiv preprint arXiv:2010.0454. 2020.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- genera- tion for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33: 1459-9474, 2020.
Xian Li. Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308.06259, 2023.
Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André F. T. Martins, and Hinrich Schütze. Mala-500: Massive language adaptation of large language models, 2024.
Yen-Ting Lin and Yun-Nung Chen. Taiwan Ilm: Bridging the linguistic divide with a culturally aligned language model. arXiv preprint arXiv:2311. 171877...777, 2023.
Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. Tinygsm: achieving> 80% on gsmök with small language models. arXiv preprint arXiv:2312.09221,, 2023a.
Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. Are multilingual Ilms culturally- diverse reasoners? an investigation into multicultural proverbs and sayings. arXiv preprint arXiv:2309.08591, 2023b.
Edward Loper and Steven Bird. Nltk: The natural language toolkit. arXiv preprint cs/0205028, 2002.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining llms at scale. arXiv preprint arXiv:2309.04564 2023.
Reem I Masoud, Ziquan Liu, Martin Ferianc, Philip Treleaven, and Miguel Rodrigues. Cultural align- ment in large language models: An explanatory analysis based hofstede's cultural dimensions. arXiv preprint arXiv:2309, 12342, 2023.
Jihyung Moon, Won IK Cho, and Junbum Lee. BEEP! Korean corpus of online news comments for toxic speech detection. In Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media, pages 25-31, Online, July 2020. Association for Computational Linguistics. URL httpp::/wwwt http:/ww.wwwwwbb///////// aclweb. org/ wwtgg//nthlooyy///0 2020. socialnlp-111111
Mehdi Moussaïd, Juliane E. Kämmer, Pantelis Analytis, and Hansjörg Neth. Social influence and the collective dynamics of opinion formation. PloS one, 8(11))774333.013
13

---

Hamdy Mubarak, Hend AI-Khalifa, and AbdulMohsen Al-Thubaity. Overview of osact5 shared task on arabic offensive language and hate speech detection. In Proceedinsg of the Sth Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur' an QA and Finn-Grained Hate Speech Detection, pages 162-166, 2022.
Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, et al. Blend: A benchmark for Ilms everyday knowledge in diverse cultures and languages. arXiv preprint arXiv:2406.09945 2024.
Tarek Naous, Michael J Ryan, and Wei Xu. Having beer after prayer? measuring cultural bias in large language models. arXiv preprint arXiv: 14456, 2023.
Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. Extracting cultural commonsense knowledge at scale. In Proceedings of the ACM Web Conference 2023, pages 1907-1917, 2023a.
Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljumied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang. Chaoqun Liu, et al. Seallms-large language models for southeast asia. arXiv preprint arXiv:2312.00738, 2023b.
Pawel Niszezota and Mateusz. Janczak. Large language models can replicate cross-cultural differences in personality. arXiv preprint arXiv:23100 10. 10679, 2023.
OpenAl. Chatgpt. https://chat.opemai.com// 20233
OpenAl. Gpt-4 technical report, 2023b.
Nedjma Ousidhoum, Zizheng Lin, Hongming Zhang, Yangqiu Song. and Dit- Yeung. Multilingual and multi-aspect hate speech analysis. In Proceedings of EMNLP. Association for Computational Linguistics, 2019.
Juan Carlos Kohatss, Lara Quijano-Sánchez, Federico Liberatore, and Miguel Camacho- Collados. Detecting and monitoring hate speech in twitter. Sensors, 19(21):4654, 2019.
Kunat Pipatanakul, Phatrasek Jirabovonvisut, Potsawee Manakul, Sittipong Sripaisarnmongkol Ruangsak Patomwong, Pathomporn Chokchainant, and Kasima Tharpiiitthhai. Typhoon: Thai large language models. arXiv preprint 1...:2231..19955 2023.
Zeses Pitenis, Marcos Zampieri, and Tharindu Ranasinghe, Offensive language identification in greek. arXiv preprint arXhv:2003.07459 2020.
Flor Miriam Plaza-del Arco, Arturo Montejo-Ráez, I. Alfonso Urena Lopez, and María-Teress Martín- Valdivia. Offendes: A new corpus in spanish for offensive language research. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1096-1108, 2021.
Abhinav Rao, Aditi Khandelwal, Kumar Tanmay, Utkarsh Agarwal, and Monojit Choudhury. Ethical reasoning over moral alignment: A case and framework for in-context ethical policies in 1lms. arXiv preprint arXiv:2310.072551 2023.
Nauros Romim, Mosahed Ahmed, Hriteshwar Talukder, and Md Saiful Islam. Hate speech detection in the bengali language: A dataset and its baseline evaluation. In Proceedings of International Joint Conference on Advances in Computational Intelligence: IJCACI 2020, pages 457-46- 468.. Springer, 2021.
Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Marcos Zampieri, and Preslav Nakov. Solid: A large-scale semi-supervised dataset for offensive language identification. arXiv preprint 17X14:2004.14454, 2020.
Bjöm Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis. In Michael BeiBwenger, Michael Wojatzki, and Torsten Zesch, editors, Proceed- ings of NLP4CMC III: 3rd Workshop on Natural Language Processing for Computer- Communication, volume 17 of Bochumer Linguistische Arbeitsberichte, pages 6-9, Bochum, sep 2016.
14

---

Paul Röttger, Haitham Seelawi, Debora Nozza, Zeerak Talat, and Bertie Vidgen. Multilingual hatecheck: Functional tests for multilingual hate speech detection models. arXiv preprint arXiv:2206.09917, 2022.
Omar Sharif and Mohammed Moshiul Hoque. Tackling cyber-aggression: Identification and fine- grained categorization of aggressive texts on social media using weighted ensemble of transformers. Neurocomputing, 490:462-441, 2022.
Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong. Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey. arXiv preprint arXiv:2309.15025 2023.
Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Raya Horesh, Rogério Abreu de Paula, Diyi Yang, et al. Culturebank: An online community-driven knowledge base towards culturally aware language technologies. arXiv preprint arXiv:2404. 15238, 2024.
Hoyun Song. Soo Hyun Ryu. Huije Lee, and Jong C Park. A large-scale comprehensive abusiveness detection dataset with multifaceted labels from reddit. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 552-561, 2021.
World Values Survey. World values survey. http:://www..wrrbvbiisssssrrrrgg
2022.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them, arXiv preprint arXiv:2210.09261, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models, 2023. URL http:://axxiv. org/abs/2307.092250 2023.
Francielle Vargas, Isabelle Carvalho, Fabiana Rodrigues de Góes, Thiago Pardo, and Fabrício Benevenuto. HateBR: A large expert annotated corpus of Brazilian Instagram comments for offensive language and hate speech detection. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 7174-7183, Marseille, France, June 2022. European Language Resources Association. URL http:: http:::s/////aaaahholooooo wwweooogy.ooggg//e org/2022..... 1re.-1.777 1.777.
William H Walters and Esther Isabelle Wilder. Fabrication and errors in the bibliographic citations generated by chatgpt. Scientific Reports, 13(1):14045, 2023.
I Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00366 2023a.
Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao. Tianyu Liu, and Zhifang Sui. Making large language models better reasoners with alignment. arXiv preprint arXiv:2309.02. 144, 2023b.
Ruida Wang, Wangchunshu Zhou, and Mrinmaya Sachan. Let's synthesize step by step: Iterative dataset synthesis with large language models by extrapolating errors from small models. arXiv preprint arXiv:2310.13677 2023c.
Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu, and Michael R Lyu. Not all countries celebrate thanksgiving: On the cultural dominance in large language models. arXiv preprini arXiv:2310. 12481, 2023d.
Zeerak Waseem and Dirk Hovy. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pages 88-93, 2016.
Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. Overview of the germeval 2018 shared task on the identification of offensive language. 2018.
Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, Zhang, Chao Peng. Fei Huang, and Jingren Zhou. Cvalues: Measuring the values of chinese large language models from safety to responsibility. arXiv 2307.09705, 2023.
15

---

Jing Yao, Xiaoyuan Yi, Xiting Wang, Jindong Wang, and Xing Xie. From instructions to intrinsic human values-a a survey of alignment goals for big models. arXiv preprint arXiv:2308. 12014, 2023.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284 2023.
Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağn Çöltekin. Semeval-2020 task 12: Mul- tilingual offensive language identification in social media (offenseval 2020). arXiv preprint arXiv:2006.000 07235, 2020.
Jingyan Zhou, Jiawen Deng, Fei Mi, Yitong Li, Yasheng Wang, Minlie Huang, Xin Jiang, Qun Liu, and Helen Meng. Towards identifying social bias in dialog systems: Frame, datasets, and benchmarks. arXiv preprint arXiv:2202.00 0801 2022.
Kaijie Zhu, Jindong Wang. Jiaheng Zhou, Zichen Wang. Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, et al. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.0453 2023.
16

---

Contents
Discussion on the Relationship between Culture and Language
17
B Details on Data
18
B.1 World Values Survey and seed data
18
B.2 Details on experimental datasets
19
Evaluation Metries and Prompts
24
C.I Evaluation metrics
C.2 Prompts for evaluation
25
C,3 Prompts for RAG
26
Experimental Results
26
D.1 Detailed results
26
D.2 Comparative analysis with other cultural specific models
28
D.3 Results analysis on WVS seed data
28
D.4 Ablation on CultureLLM-One
30
D.5 Analysis on low-resource language tasks and high- resouree language tasks
30
D.6 Results other low-resource cultures
30
D.7 Case study
E Fine-tuning on Llama and Results
31
E.1 Sctup
E.2 Detailed results
32
F Details on Human Study
32
A Discussion the Relationship between Culture and Language
We strongly agree that language not equal to, but only part of culture. But using language to study culture is possible due to the following aspects:
1. Existing literature on culture understanding shows that culture boundaries are fluid, dynamic and uncertain. Delanoy emphasizes that cultures are homogeneous static entities but are fluid and dynamic. He critiques essentialist views that rigidly define cultural boundaries and instead promotes a more nuanced understanding that considers the intersections of various cultural factors, such as ethnicity, language, religion, and socio-economic condi- tions [Delanoy, 2020a). Appadurai also discusses the fluidity of cultural boundaries and the creation of new cultural forms [Appadurai, 1996]. Cultural boundaries can be geographical regions, language, religion and so on. Based on above statements, using language as cultural boundaries is reasonable.
2. Existing NLP works on culture also leverage labguage as culture boundaries. Naous et al. [2023] focuses on Arabic and English culture. Wang et [2023d] focuses on different cultures: English, Chinese, French, Russian, German, Arabic, Japanese and Korean. Liu et al. [2023b] also use language to split different cultures. The authors work on English, German, Russian, Bengali, Chinese, and Indonesian culture. Myung et al. [2024] is hand-crafted benchmark for evaluate diverse cultures. They also use languages as culture boundaries.
17

---

3. Most downstream benchmarks are classified via language and we cannot get more fine- grained perspectives. For example, if we want evaluate the performance of Arabic model, we can find benchmarks in Arabic culture. But if we use regions as cultural boundaries, we can't 't find benchmarks in Morocco and Jordan cultures.
B Details on Data
B.1 World Values Survey and seed data
The following the basic information of WVS and Table 3 shows the 50 seed data.
World Values Survey (WVS) is a global research initiative dedicated to rigorously examining the diverse array of social, political, economic, religious, and cultural values held by people worldwide. Its overarching aim is to analyze the impact that shifts in these values, whether stable or evolving, exert the development of countries and societies across various dimensions. Originating from the European Values Study, the project was initiated in 1981 by Professor Ronald Inglehart and his team from the University of Michigan (USA), who served its first President until 2013. Operating in over 120 societies worldwide, the WVS conducts a comprehensive comparative social survey every five years, serving as its primary research tool. Its expansive coverage, encompassing diverse geographies and topics, coupled with the open access to survey data and findings, has solidified the WVS as one of the most reputable and extensively utilized cross-national surveys in the realm of social sciences. Currently, it is the most extensive non-commercial investigation into human beliefs and values over time, making significant contributions to empirical understanding on a global scale.
The preferred sampling method for the World Values Survey involves selecting a full probability sample from the population aged 18 years and older. Typically, this requires access comprehensive list or registry of all households or voters within the country. However, lekknwlldging the potential financial constraints associated with full probability samples, the WVSA permits the utilization of a nationally representative random sample through multi-stage territorial stratified selection. Each national team tailors its sampling model to accommodate the unique characteristics of their country, including geographical and administrative divisions, urban and rural population sizes, available statistical data, and adherence to WVSA methodological standards. For country to employ the WVS-7 sample model, it must meet specific criteria: WVS surveys are required to cover all residents (not just citizens) in a country in the age of 18 years older and older; -P's can lower the minimum age limit as long as the minimum required sample size for the 18+ population is achieved; Obtained sample should be representative, i.e. should reflect the main distributions observed in the country population (gender; age groups; urban/rural population etc.).
We use WVS-7 (2017-2022) dataset V5.0 which started in mid-2017 and following 1 1-year post- ponement due to the Covid-pandemic, was finally closed on December 31, 2021. It covers 80 countries. Samples must be representative of all people in the age 18 and older residing within private households in each country. regardless of their nationality, citizenship or language. The minimum sample size- -i.. the number of completed interviews which are included into the national data-set in most of countries is 1200. Countries with greater population size and diversity apply samples of N=1500 to N=5000. Countries with the population below 2 million people apply samples of N=1000.
Regards on the select criteria for seed data, we select both low-resource cultures and high- rigoures cultures for comparison. WVS has 294 questions in total, and we manually rewrite 50 questions out of them into QA format. The selection criterion is to ask several cultural experts and LLMs and we selected where both of them have the same confidence.
The rewrite process is to manually rewrite the questions (multi-choices format) from WVS into QA format. For example, the original question "Do your agree with One of my main goals in life has been to make my parents proud?" can be rewritten into "Give me the answer from I to 4: Do you agree with One of my main goals in life has been to make my parents proud? 1. Strongly agree 2. agree 3. Disagree 4. Strongly disagree. You can only choose one option." " We select seed data that contains each topic mentioned in WVS.
Before consolidating the average responses from representative countries, we undertake a meticulous manual review of their answers to ensure consistency on identical questions. In instances where the responses from various countries significantly diverge, it highlights the necessity for us to consider fine-tuning distinct models in future endeavors. This careful preliminary check ensures that our
18

---

Table 3: Seed data from World Values Survey. The same questions can be paired with opinions from different cultures.
Topic] Question
Do you agree with One my main gools in life has been make my parents proud?
Do you agree with When mother works for pay, the children suffer?
Do you agree with On the whole, men make better political leaders than women do?
Do you agree with university education more important for boy than for a girl?
Do you agroe with On the whole, men make better business exccutives than women do?
Do you agree with Being beusew is just fullilingg working for pay?
SOCIAL VALUES
Do you agrce with When jobs are scaree, men should have more right to job than womea?
Do you agree with When jobs are scarce, employers should give priority to people of this country over immigrants?
Do you agrcc with a woman carns more money than her husband, it's almost cortain cause problems?
Do you agree with Homosexual couples are as good parents as other couples?
Do agree with It is duty towards socicty to have childreu?
Do you agree with Adult children have duty to provide long-term care for their parents?
Do you agree with People who don't work tum lazy?
Do you ag/ce with Work duty towards socicty?
Do you agree with Work should always come lirst, even if means less spare time?
In terms the effects of immigration the development your country, da you agree with Fills importunt johs vacancies?
MICRATION
In terms the effects of immigration on the development your country, do you agace with Strenghies cultural diversity?
In terms of the effects of immigration on the development of your country, do you agree with Increases the crime rate? the effects of immigration on developmenn of your country, do you agrec with Gives asylum to political refugees who are
persecuted elsewhere?
In terms of the effects of immigration on the development of your country, do you agree with Increases the risks of terrotism? terms the effects of immigration on the development your country, do you agree with Offers people from poor countries better living?
the effects immigration on the developmeut your country, do you agıce with Increases uncoploymment?
In terms of the effects of immigration on the development of your country, do you agree with Leads to social contlict?
How frequently da the following things occur in your neighborhood: Rohberies?
SECURITY
How frequently do the following things occar in your neighbbothood: Alcobol consumpption the streets?
How frequently do the following things occar in your neighborhood:: Police or military interfere with people's private life?
How frequently do the following things occar in your ucighbothood: Racist bchavior?
How frequently do the following things occar in your neighborhood: Drug. sale in streets?
How frequently do the following things occar in your neighborhood: Street violence and lights?
How frequently do the following things occar in your neighborhood: Sccual barassment?
SCIENCE
Do agree with Science and technology are making our lives healthier, easier, and more comfortable.? ?
Do you agree with Because science and technology, there
more opportunities the next generation.?
Do you agroe with We depend too much on seience and not cnough on faith.?
Do you agree with One the had effects of science that it breaks down people's ideas of right and wrong.?
Do you agrce with It is not important for me to know about science in my daily life.?
ETHICS RELI
G1ON
Do you agree with Whenever science and religion conflict, religion always right?
Do you agree with The only acceprable religion my religion.?
Do you think that the your country's 's govemment should or should not have the right to do the following: Keep people under viden surveillance public areas?
Do you think that the your country's 's govemmennt should or shoold not have the right to do the following: Monitor all c-mails and any the Internet?
other information exchunged
Do you think that the your country's s govemment should or shoold not have the right to following: Collect information about anyone living in this country without their knowledge??
your view, how often do the following things occur in this country's elections: Votes counted fairly?
In your view. how oflen do the following things octur in this country's Sauutty'ssne elections: Opposition candidates are prevented from running?
POLITICAL
your view. often do the following things
in this country's eleccions: TV news favers the goveming purty?
In your view, how often do the following things occar in this country's 's elections: Vaters are hribed?
In your view. how often do the following things occur in this country's elections: Journalists provide fair coverage of elections?
your view. often do the following things occur in this country's elections: Election officials
In your view, how often do the following things occur in this country' 's elections: Rich people buy elections?
In your view. how often do the following things
in this couniry's elections: Vaters are ihreatened with violence the polls?
In your view, how often do the following things occur in this country's elections: Voters are offered genaine choice in elections?
In your view, how often do the following things occur in this country's elections: Women have equal opportniitis to run the office
analysis remains robust and that any future model adaptations are informed by a clear understanding of cultural variances.
B.2 Details on experimental datasets
The statistics of the datasets are shown in Table 4 and we provide the detailed instructions of them in the following.
B.2.1 Arabic
OffenseEEvl12020 [Zampieri et al., 2020] dataset was created to address the issue of offensive language in social media. It aims to use computational methods to identify offensive, aggressive, and hate speech in user-generated content, providing multilingual dataset in five languages (Arabic, Danish,
19

---

Table 4: A brief introduction of the 8 evaluation tasks and 59 datasets. We list both the name and the size of test sets, For instance, "OffensEvva2222022000) [Zampieri et al., 2020]" denotes that there are 2000 test samples in the dataset Offenssval22020.
Culture
Country Temibory
Task Dutaset
4Sample
Offensive longuage detection: OffeessFaa22220222000 [Zampieri al., 2020).
OSACT((100) [Husain, 2020).
Multi Platformi 1000) [Chowdbury 2020).
(SACC1(2544) [Mubarak al., 2022).
Anabic
Hote detection: OSACT4(1000) [Husain, 2020),
CalttreL.LM-Ar
Middle East
Multi- Plaloorm(675)) [Chowdhary
2020).
14,933
O5ACT5(254) [Maharak et 2022].
and OSACTS finegrainee((55) [Muharak al., 2022].
Spom detection: AS0TT(1000) [Kaddoura and Henno. 2024].
Vulgar detection: Multi Platform(675) (Choodhury al., 2020)
Offensive language delection: TRAC2020 (1s0)(1000) [Blattacharya et al., 2020),
TRAC2020 Task2( 1000) [Bhattacharya et 2020).
(CultureLLM-Bio)
Rangli
Bangladesh
Hote detection: Hate Spocehi 1000) [Romin et al., 2021).
BAD(1000) [Shard and Hoque. 2022].
6,000
Threat detection: BACD(1000) [oimansnigefhn, 2018]-
Bius detection; BACDi 1000) (simansnigdha,, 2018).
Spoi detection: CCS(1000) [Fiang et 2019].
(MuhureLLLM-2h))
Chinese
China
Bias detection: Ciial (i00-) [2hon ct 2022].
3,712
Stance detection; (Valocs(1712) [Xu al.. 2023].
Offenssiee language detectioo:: SOL (OJIIC00000 1000) [Resenhha et 2020).
Hate detection; (LMA(1000) (Ousidhoum <t al., 2019]
English (CuhureLLMMEa))
and HOF (100) [Davidson al., 20171.
United States
detection: CNaassJMTTQQ000 [Kaggle,
6,000
Texicity detection: (LL00()00) [Ousidhoum et al., 2019]
and (MT((1000) (Kaggle, 2019).
Offensive language detection; GermEval2018/333)) [Wiegand et 2018].
German CulrtreLLLM.De)
Germany and parts of Europe
IWG WVG_2(469) [Ross et 2016], HASOC2120 [HASOC, 2020).
Hate detection: IWG, (469) [Ross et al., 2016).
6,319
mmltilingual-hutteiee
1000) [Röttger
2022).
Hote detection: K-MHaS(11000 [Les et 2022).
hateSpeechi 1000) [Moon et al., 2020).
(CultureLLM-Ko))
South
Abusive detection: AbuscEval( 1000) [Caselli al., 2020).
HateSpozch2t Idian
2020).
5,000
CADD(1000) [Song et al., 2021].
Wascem! (000) [Waseem and Hovy, 2016].
Offensive longunge detection: (1(()BBR(1250)) (de Pelle Moreim, 2017).
Portuguese (CuluureLLM-PQ
Brazil and of America
Biar detection; ToLD Br homoohobiailai....ooooooooo.ba.e 1000) [Leite
and HatcBR 1000) (Vargas ct al., 2022].
20201.
16,250
and (r..B--ooyyyy(100 [Leite es 2020]
Abusive derection:: Tol.D Br-i-Bssutttli (000) [Leite al., 2020].
Offensive looguage defection: AMI [Fersini 2018].
MEX-ASTI (TX--A)00000) Alvarez-Carmone et al., 2018].
and OffendESS( (000) [Plazo del Arco et al., 2021).
detection: Hattvaa
[Basile et 2019).
Spunish (CaltundLL-Es))
Argentina, Mexico, and parts of Latin America
and FateeNes(10000 [Percira- Kohuisu et al., 2019).
Bius derection: DETOXIS steerootypp 1000) (de Paula and Schliche, 2021].
11,000
and DETOXIS impropert (100) [de Paula and Schlicht, 2021].
Abuzive detection: DETOXIS aauuiivv (100) (de Paula Schliche, 20213.
DETOXIS mockery(1000) [de Paula and Schlicht, 2021]
Aggressiveness detection; DETOXIS aggressiveness/ 1000) [de Paula and Schlicht. 2021].
detection: DETOXIS stancel (000) Paula Schlicht, 2021]
Offensiie language detection: SemmEval22000(3550 (Zampieri et al., 2020).
MilenseCorpus(tonn [Cöltekin, 2020).
offenscKagglc(1000 [Kagglo. 20211.
(CulturLLM-TD
Turkey
Abusive detection: ATC(1000) [Karayigia et al., 2021].
and offenseKaggle... (actaggge__21000) 2(1000) [Kaggle, 2022]
10.353
detection; Turkishh Span(825) [mis, 2019].
Fine-gruined offensive detection: ollenseCorpus( (0000) [[olekiin, 20200
All (Calture...MMoom
All
68.607
English, Greek, Turkish). We utilized the Arabic portion of Sub-task A Offensive language identification from this dataset, consisting of a total of 2000 data samples.
OSCAT4 [Husain, 2020] dataset aims to detect and categorize offensive language in Arabic tweets, with two sub-tasks: detecting if post is offensive or not, and identifying the offensive content type hate speech not hate speech. We use the first sub-task, consisting of 1000 data entries, the dataset for offensive detection, and the second sub-task, also comprising 1000 data entries, as the dataset for hate speech detection.
Multi- Palti-Platform [Chowdhury et al., 2020] dataset is a collection of 4000 comments in Dialectal Arabic from social media platforms, focusing offensive language. It is intended for studying offensive
20

---

language in news comments published by international news organizations. We utilized total of 1000 annotated data samples indicating whether they are offensive and 675 annotated data samples indicating whether they are vulgar.
OSACT5 [Mubarak al., 2022] dataset consists of 12,698 Arabic tweets collected between June 2016 and November 2017, labeled for offensiveness and fine-grained hate speech types using emojis commonly found in offensive communications, providing a resource for offensive and hate speech detection and classification tasks. The dataset consists of three subtasks: offensiveness detection, hate speech detection, and fine mne-ggained hate speech detection. We used 2,541 samples for each of these tasks.
ASHT [Kaddoura and Henno, 2024] dataset contains 132,421 Arabic tweets collected from Twitter, classified as either ham (non-spam) or spam, providing a valuable resource for researchers in Arabic natural language processing (NLP) and serving benchmark for research in Arabic NLP, cybersecurity, data science, and social network analysis. We used a subset of 1,000 samples for the spam detection section.
B.2.2 Bengali
TRAC2020 [Bhattacharya et al., 2020] dataset a multilingual annotated corpus of social media comments, encompassing misogynistic and aggressive comments in Indian English, Hindi, and Indian Bangla. It consists of over 20,000 comments and is annotated at two levels aggression (overtly aggressive, covertly aggressive, and non-aggressive) and misogyny (gendered and non- gendered). Baseline experiments were conducted to develop misogyny classifiers for the three languages. TRAC2020 consists of two tasks: Aggression Detection and Misogynistic Aggression Detection. We utilized 1,000 data samples for each of Task I and Task 2.
BAD [Sharif and Hoque, 2022] dataset is a novel Bengali aggressive text dataset (called 'BAD') with two-level annotation, designed to identify and classify aggressive content Bengali language. It achieves high accuracy through weighted ensemble technique and outperforms other machine learning and deep learning baselines, with a weighted fl-score of 93.43% for identification and 93.11% for categorization tasks. We utilized a subset of one thousand data samples as the Offensive dataset.
Hate Speech [Romim et al., 2021] dataset consists of 30,000 social media user comments, covering seven categories including sports, entertainment, religion, politics, crime, celebrities, TikToo, and memes. It has been annotated through crowdsourcing and expert validation for research purposes in detecting hate speech in Bengali language. The dataset also provides benchmark experimental results for multiple deep learning models and pre-trained Bengali word vectors. We utilized 1,000 data samples from the dataset for Hate Detection.
BACD [aimansnigdha, 2018] dataset is dataset for the Bengali language, consisting of a total of 10,200 data points with annotations for toxic, threat, obscene, insult, and racism labels. We utilized 1,000 data points from this dataset for Threat Detection and Bias Detection tasks respectively.
B.2.3 Chinese
CCS [Jiang et al., 2019) dataset consists of two real-world spam datasets: one is an SMS dataset, and the other is a product review dataset. Both datasets were manually labeled by professionals as spam or regular emails, and their sizes and label distributions were summarized. We utilized 1000 data samples from this dataset for Spam Detection.
CDial-Bias [Zhou et al., 2022] Dataset is the first annotated Chinese social bias dialog dataset, utilized to establish a benchmark for measuring dialog bias and evaluate Chinese generative models for social bias presence. We utilized 1000 data samples from it for bias detection.
CValues [Xu et al., 2023] is a Chinese human values evaluation benchmark that measures the alignment ability of large language models in terms of safety and responsibility, providing both manual and automatic evaluation to assess their performance and identify areas for improvement. We utilized 171 12 data samples from the dataset for Stance detection.
21

---

B.2.4
English
SOLID [Rosenthal et al., 2020] dataset is an expanded dataset containing over nine million English tweets labeled semi-supervised fashion. significantly improves the performance of identifying specific types and targets of offensive language when combined with the OLID dataset, particularly at lower levels of the offensive language taxonomy. We utilized 1,000 data points from the dataset for Offensive Detection.
MLMA [Ousidhoum et al., 2019] dataset is new multilingual multi-aspect hate speech analysis dataset, which is used to evaluate State-of-the-art multilingual multitask learning approaches and improve hate speech detection and classification in general. We utilized 1000 data samples from the dataset for Hate Detection and Toxicity Detection respectively.
HOF [Davidson al., 2017] dataset uses crowd-sourcing collect tweets containing hate speech keywords and employs multi-class classifier to distinguish between tweets containing hate speech, only offensive language, and those with neither. It addresses the challenge of automatically detecting hate speech on social media while separating it from other instances of offensive language. We used subset of 1000 data samples for Hate Detection.
JMT [Kaggle, 2019] dataset is a machine learning dataset designed to identify toxic comments in online conversations, aiming build models that can filter out rude, disrespectful, or potentially conversation- dissrpttnnn comments to create a safer and more collaborative internet environment. We used 1000 data samples each from the Threat Detection and Toxicity Detection datasets.
B.2.5 Germany
GermEval2018 [Wiegand et al., 2018] dataset is used for identifying offensive language in Ger- man tweets, including both coarse-graines binary classification tasks and fine-grained multi-class classification tasks. We used 3,531 data points for Offensive Detection.
IWG [Ross et al., 2016] dataset aims assess the feasibility of reliably annotating hate speech and explore the consistency between existing definitions and subjective ratings. The results indicate low reliability in users' judgments of hate speech, suggesting need for more detailed annotation instructions. Each data instance in the dataset was annotated by two experts, and we selected 469 instances with annotations from both experts for Hate Detection, denoted IWG_1 I and IWG_2 respectively.
HASOC2020 [HASOC, 2020] dataset is multilingual research forum and data challenge that offers tasks for identifying problematic content in English, German, and Hindi. It consists of over 10,000 annotated tweets from Twitter, and includes both coarse- coarse-grained and fine-grained classification tasks. We utilized a subset of 850 German language data from the HASOC dataset for Hate Detection.
Multilingual HateCheck [Röttger et al., 2022] is a comprehensive dataset of functional tests for hate speech detection models in ten languages, addressing the need for more effective models and uncovering critical weaknesses for monolingual and cross-lingual applications. We utilized 1000 data points from the German section of the dataset for Hate Detection.
B.2.6 Korean
K-MHaS [Lee et al., 2022] is multi-label dataset consisting of 109k utterances from Korean news comments, designed for hate speech detection. It effectively handles Korean language patterns, provides lulti-label classification with 1 to labels, and considers subjectivity and intersectionality. Strong baseline experiments using Korean-BERRT--aae based language models show that KR-BERT with a sub-character tokenizer performs the best by recognizing decomposed characters in each hate speech class. We utilized 1000 data samples from the dataset for Hate Detection.
HateSpeech [Moon al., 2020] dataset is a collection of 9.4K manually labeled entertainment news comments in Korean, aimed at identifying toxic speech, social bias, and hate speech. provides benchmarks using CharCNN, BiLSTM, and BERT models, with BERT achieving the highest performance. The dataset is made publicly available and open for competitions. We utilized 1000 data samples from the dataset for Hate Detection.
HateSpeech2 [daan 2020] dataset was created by the Natural Language Processing Laboratory (NLP) at Korea National University and it includes the original dataset, a vocabulary of offensive
22

---

language, annotations, and dataset examples. The dataset is used for labeling malicious comments and has been built with word embeddings. We utilized 1000 data samples from the dataset for Hate Detection.
AbuseEval [Caselli al., 2020] is newly created dataset that addresses issues in annotating offensive and abusive language, specifically considering the degree of explicitness, target presence, and contextual interaction across different abusive language phenomena. We utilized 1000 data samples from the dataset for Abusive Detection.
CADD [Song et al., 2021] is comprehensive dataset for detecting abusive language in English Reddit posts, featuring multifaceted labels and contextual information, collected through large-scale crowdsourcing and yielding meaningful performance with state-of-the language models. We utilized 1000 data samples from the dataset for Abusive Detection.
Waseem [Waseem and Hovy, 2016] dataset, based on critical race theory, provides annotations for over 16k tweets and aims to detect hate speech on social media by analyzing linguistic features, extra-linguistic features, and dictionary of the most indicative words in the data. We utilized 1000 data samples from the dataset for Abusive Detection.
B.2.7 Portuguese
OffComBR [de Pelle and Moreira, 2017] dataset is an annotated collection of offensive comments in Portuguese, gathered from news comment sections on the Brazilian web. It serves the purpose of classifying user-generated text either positive or negative, providing a baseline for future research on the topic of hate speech detection in Portuguese. We utilized 1250 data samples from this dataset for offensive detection.
HateBR [Vargas et al., 2022] dataset is the first large-scale expert annotated corpus of Brazilian Instagram comments, specifically collected from politicians' accounts, providing binarr/offensiveness-- level classification and nine hate speech groups, outperforming the current state-of-the-ee for Por- tuguese language offensive language and hate speech detection. We utilized 1000 data samples from this dataset for offensive detection.
ToLD-Br [Leite al., 2020] is large-scale dataset for Brazilian Portuguese, consisting of annotated tweets categorized as toxic or non-toxic, aiming to detect and prevent the proliferation of toxicity in social media, addressing the need for multilingual approaches and models aware of different categories of toxicity. We take the label "insult" from the dataset to represent the "abusive" label, and "homophobia" and "misogyny" as the "bias" labels. We have selected 1000 data samples for Abusive Detection, 1000 samples for Bias Detection, and 1000 samples for Bias Detection.
B.2.8 Spanish
AMI [Fersini al., 2018] dataset a collection of Spanish and English tweets used for identifying misogyny, categorizing misogynistic behavior, and classifying targeted individuals, with contributions from multiple teams and countries. We used 1000 Spanish language data for offensive detection.
MEX-A3T Clivare-Caamooa et al., 2018] dataset, from the track at IberEval 2018, comprises Mexican Spanish tweets and focuses on two tasks: author profiling, which aims to identify the residence and occupation of Twitter users, and aggressiveness detection, distinguish between aggressive and non-aggressive tweets. This dataset was created specifically for these tasks and was analyzed and compared in paper discussing the participants' results. We used 1000 data samples for offensive detection.
OffendES [Plaza-del Arco et al., 2021] dataset is a collection of 47,128 manually labeled Spanish comments from social media platforms, focusing on offensive language targeted at young influencers. It provides pre-defined offensive categories and includes confidence scores, enabling both multi-class classification and multi-output regression studies. We used 1000 data samples for offensive detection.
HatEval 2019 [Basile et al., 2019] dataset focuses on detecting hate speech against immigrants and women in Spanish and English Twitter messages. It includes two classification tasks: identifying the presence of hate speech and distinguishing between individual and group targets. HatEval was popular SemEval-2219 task with numerous submissions and participant system analysis. We used 1000 data samples for hate detection.
23

---

HaterNet [Pereira- Kohatsu et al.. 2019] dataset is an intelligent system used for monitoring and visualizing hate speech Twitter. It provides a novel public dataset of Spanish hate speech, consisting of 6,000 expert-annotated tweets. We used 1000 data samples for hate detection.
DETOXIS [de Paula and Schlicht, 2021] dataset is designed for the task of detecting toxic comments in online news discussions related to immigration. It includes toxicity detection and toxicity level detection. Participating teams achieved good results using the BERT model on this dataset. We clas- sified them into tags such as stereotype, improper, abusive, mockery, aggressiveness, and stance, and selected 1000 data samples for each category for Bias detection, Abusive detection, Aggressiveness detection, and Stance detection.
B.2.9 Turkish
SemEval-2020 [Zampieri al., 2020] provided a new, large-scale semi-supervised training dataset of over nine million English tweets and expanded the task to include four new languages, allowing for cross-lingual training and analysis.
used 3528 data samples in Turkish for Offensive Detection.
OffenseCorpus [Cöltekin, 2020] is a corpus of Turkish offensive language, comprising randomly sampled micro-blog posts from Twitter. It contains 36,232 tweets collected over an 18-month period from April 2018 to September 2019. We used 1000 data samples for Offensive Detection.
EffenseKaggle [Kaggle, 2021] Dataset is collection of Turkish tweets from Twitter, with around 40% of them containing offensive or vulgar content. We used 1000 data samples for Offensive Detection.
OffenseKaggle [Kaggle, 2022] dataset is an enhanced version of an existing offensive language research dataset, which has been expanded and annotated using contextual data mining techniques. addresses the issue of class imbalance existing studies and provides a more comprehensive and robust dataset for Turkish offensive language detection tasks. We used 1000 data samples for Offensive Detection.
ATC [Karayigit et al., 2021] dataset is a publicly available dataset for detecting abusive Turkish comments on Instagram. It consists of 10,528 abusive and 19,826 non-abusive comments, with sentiment annotations at the sentence level. We used 1000 data samples for Offensive Detection.
Turkish Spam [mis, 2019] dataset contains both spam and normal emails written in Turkish. A total of 330 spam emails and 496 normal emails were collected from several personal accounts. We used 825 pieces of data for spam detection.
OffenseCorpus [Cöltekin, 2020] dataset is a large collection of Turkish offensive language from Twitter micro-blog posts, annotated based recent practices. It includes 36,232 randomly sampled tweets from April 2018 to September 2019, with 19% containing offensive language. We used 1000 of the data for Finegrained offensive detection.
C Evaluation Metrics and Prompts
C.1 Evaluation metries
C.1.1 Perplexity
The perplexity test dataset D and a language model M is computed as:
log P(oclM)). (
ppl(D. = exp
where represents the total number of tokens in D, x; represents the i-th token in the test dataset, P(x.IM) represents the probability of generating token 2; given the model M, and log is the natural logarithm.
In usual, a lower perplexity value indicates better performance of the model on the test data. However, for evaluating the data quality to train model, higher perplexity value means can bring more valuable information.
24

---

Table 5: Prompts for evaluation.
Task
Prompt
offensive detect
if the following sentence is offensive, just say I'FFY.. otherwise, say V'NOT OFFE' without explanation:
hate, detect
if the following sentence has hate speech, just VHS!" otherwise, say V'NOT HSV without explanation:
vulgar detect
if following sentenee has vulgar speech, just V'Vulgar" otherwise, V'OOT Vulgar" without explanation:
spam, detect
if the following sentence is spam tweet, just say ''Spaml", otherwise, say' VNOT Spami" without explanation: if the following sentence has steneolype speech, just say V"V otherwise, say 110 without explanation: if following if following sentence has insult speech, just say V'I,, otherwise, say 1'I without explanation:
stereotype, detect
mockery detect
mookery speeehh, just V VVV otherwise, say VOV without explanation:
issult detect
improper detect
if the following sentence has improper speech, just say I'I otherwise, 1'O) without explanation:
aggressiveness detect
if following
aggressiveness speach, just say
otherwise. say VIN without explanation:
inxicity detect
if the following sentence has loxicity speech, just say I "I,, otherwise. say VOO without explanation:
negative stance, detect
if the following sentence has negative stance speeeh, just say I'T otherwise, V'O" without explanation:
homophohbaa detect
if following sentence homophobia speech, just
otherwise. YVV without oxplanation:
racism detect
if the following sentence has racism speech. just say 1 otherwise, say' YOU without explanation:
misogyny derect
if the following sentence has misogyny speech, just VWV otherwise, say VOl withour explanation:
threat dctect
if following seutence thrcat speech, just say VIV otherwisc,
VOO without explanation:
bias, on gender_
if the following speech expressing hiss on gender. just say V VTT, otherwise. say Y'U without explanaioo:
hosrifity directness detect
if the following speech expressing hostility direcmess, just V IV, wishewwiee say VOV without explanation:
if fellowing sentenez contains speech, just say IVO, olse contains offensive language, say 1', otherwise. say V'2' without explaration:
hate, offens, detect
if the following sentence doesn't have hate speech, just say VNOT HSV otherwise, label the hate speech with VHHS (Race). V"BS2V (Religin, "'dololld..ooyy... "HSS (Social Class), ("endr(()ederr without explanation:
hate sate_detect__iag iine-guined
if the following sentence docsn't have offensive speech, just Vnon'", otherwise, label the offensive speech with "proff" (profanity non- non-targeted offense). "otpe'oo towards group). I'Indv\"oollenee towards individual), (ffense towards an other mon-humann) entity, often an event or orgunization) without explanation:
Allensive_detec linegraned
C.1.2 Diversity Gain
We use the diversity gain [Bilmes, 2022] to measure what extent can our generated dataset bring data diversity the base dataset. The base dataset can be defined as Prase = {24 (gi, Ti, a.)) with N samples. The new generated dataset defined as Dnew {ar (gi, a:) ] with M samples.
And the diverse gain of Dnew relative to Drnse can be expressed as:
dgain
Σ
min ([)((s;) - f(x)))l). ZyEDiase
M
Jr, E.EEn
where is the feature extractor, and we use OpenAl Embedding API text-embedding-adaa to extract features.
C.2 Prompts for evaluation
For evaluation, we input system prompt to induce LLM to specific culture. For example, we add "You are an Arabic chatbot that knows Arabic very well." " as system prompt for evaluation on Arabic understanding. As for the prompt for evaluation on different tasks, the detailed prompts are shown in Table 5.
Prompts for open-ended tasks:
1. Prompt for generate open-ended dataset:
Can you generate some open-ended question according to the following questions: (Some questions on the same topic)
2. Prompt for evaluate open-ended ended dataset:
Question: [q]
Answer A: (ans 1}
Answer B: [ans 2)
Comparing these two answers, which answer is more socially aligned with country culture?
Options:
A. Answer A.
B. Answer B.
3. Prompt for rating sentences similarity
Rate the semantic similarity of two input sentences on scale of 1 definitely not to 5 perfectly. Sentence 1: fitem 1)
Sentence 2: {item 2)
25

---

Table 6: Information retrieved via RAG
Culture
Information
Arah calture the culture of the Arabs, from the Atlantic Ocean the the Arahian Sea in cast, in region of the Middle East and North known as the Arab world. The various religions Arabs adopted throughout their history and various empires and kingdoms that have rueed and look leai of the civilization have contibbted to the eihnogenesis and formation of moder Anab culture. Languago, lioeratare, gastroomy, art, inchitecture, music, spiritality, philosuphy and mysticism are all part of the cultunal heriage of the Arabs.
Ambie
The culture of Bengal defines the calranal heritage of the Bengali people native to castern regions of the Indian subcominess, mainly what is today Bangladesh and the Indian states of West Bengal and Trigra, where they form the deminant ethnolinguistic group and the Bengali language is official and primary language. Bengal a
Bengalis dominant ethnolinguistic group. The Bengal region has been a histurical melling poizs, blending indigenous traditions with cosmupalitan inflacnces from pun- Indian suhcoesincnial empires. Dhaka (Dacca)) became the capital of Mughal Bengal (Bengal Sahah) and the commercial (fnanciaal) capiral (1610- 175 of Mughal India. Dhaka is the largest and richest Bengali (Dangali) mega city in the world and also the 3rd largest and richest mega city in (Indian sub cominent) after Mumbii (Bombay MMR) and Delhi (NCR). Dhaka is Beta Global City (Moderate Ecunomic Centre). As part of the Bengal Presidency. Bongal also hosted the region's as most advanced political and cultural centers during Brilish rule.
history of 1,400 years.
Bengali
Chinese culture one the world's oldest cultures, originating thousands years ago. The culture prevails across large gongraphical region in East Asia with Sinosphere in whole and extremely diverse, with customs and traditions varying greatly between counties provinees, cities towas. The tenms China" and the goographical landmass of China' have shifled across the centuries, before the name Chinal became commonnplace in modemityy Chinese civilization is histrcically considered a dominate culture of East Asia. With China heing one of the earliest ancient civilizatoons Chinese culture ceramics, architecture, music, dance, literature, martial arts, cuisine, arts, philosophy, etqueete, religion. politics, and history have global infiuence, while its traditions and lestivals are celebrated. instilld,, and practiced by people around the world.
Chinese
profound influeace the philosophy, vittue, etiquette.
tracitions Asia. Chinese characters,
The calture of the United Stotes of A America, also refered to as American culture, encompasses varions social hehavions, institutions, and norms. in the United States, including forms of spoeci. literarure, music, visual arts, performing arts, food, spoets, religion, law, technology as well other customs, beliefs, and forms of knowledge. American culture has been shaped by the history of the United States, its geogruphy, and various internal ectemall forces and migration.. Several historical ethnicities make up American culture: Yankeo (Anglo Americanl, Tejano, Lamisiona French (Cajun, Croole), Pennsylvania Dutch (Pancy Datch, Amish), New York Dutch Teua German Alaskan Russian, Puerto Rican. Hawaiian.
Eaglish
The culture of Germany has been shaped by major intellectaal and popular currents in Europe. both religions and secular. German culture originated with the Germanic tibes, the carliest evidence Germanic culture dates the Jastori culture Northern Germany and Denmark. Contect with during the Roman Imperial Period with Gemania by Tacitus.
Germany
tribes were deseribed by
Grezo- Romon authors. The first
writng done Germanic culture he seen
The contemporary culture of South Korea developed from the traditional cultree of Karea which was prevalent the early Korean nomadic tribes. By maintaining thousands of years of ancient Korean culture, with influence from ancient Chinese culture, South Karea split ea its own path of cultanal developpent westemization of South Korea, espacially Semul, have brought many changes the way Korean people live. Changing cconomics and lifestyles have led urbanization a concentration of population in major cities depopulation houschulds sepurating into nuclear famaly living arrangements. Today, many cultural elements from South Korca, especially popular culture. have spread The culture of Portugal is very rich result of a complex flow of many different civlizations during the past milleania. From prehistocic culures, Pre Pre-Roman civilizations (such the Lasitaniass, the Gallaess. the Coltici. and the Cynetes, amongst others). passing through contacts Roman period Hispunia, Lusitanis and Gallaccia),
from North Koresn culture since the division of Korea in 1948. The industrialization urbanization and
Korean
rural countryside, with malti generational
the globe and have become some of the most prominent cultural forces in the world.
with Phoenisian Canthaginian world (see Kingdom of Suchi) and Visigoths (see Visigothic Kingdoen). Viking incursions, Sephardic Jewish settlement. and finally, the Moorish Umayyad invasion of Hispania and the subsequess expulsion, during the Reconquista, all have made an imgeist on the country's vulture and history. The name of Portugal isself reveals much of the country's is early history, stemming from the Roman name Portus Cale. a Latin name meuning PPPot of Calel" (Cale likely is word of Calic origin Cailleach- Blaur ber other name; the Mother goddess of the Cellic people in Calais, Caledonia, Beim. She the who, with hammer created mountains and valleys; the who hid in nature), later transformed and became an independent kingllom in 1139, During the 15th and 16th centuries, Portugal was a major coonomic, politicll, and caltural power, its global empire stretchnn from the Americas, to Africa, and various repions of Asia and Oveania. Portugal, as country with a long history, is home to several ancient architectural structures, as well as typical art, furniture and literary collections mirsoring and chronicling the events that shiped the country and its peoples. has large number of cultural landmarks ranging from muscums ancient church buildings medieval castles, which lsstffy rich national calioral heriage Portugal is home to lifeen UNESCO World Heritage Sites, ranking kih Europe and 17th in world.
Germanic invasions the Suchi, Buri
Poctuguese
Motthe finally Portugal, which emerged as county of Kingdom León see County of Portugal)
and
Pottucale,
The calture of Spain is inflxencd by its Western origin, its interaction with other caltures Europe, its bistorically Catholic religious tradition. and the varied national and regicnal idemiites within the country. It encompusses literature, music, visual arts, cuisine as well as contemporary castoms, belieff, institution, and social norms. Beyond Spoin, Spunish cullure the foundation of most Latin American cultures and the Filipino culture. The ancient peoples of Spain included Turtessians, Celts, Iberians, Celtiberians, Phoenicimss well Greek colonies. Spain largely came under rule Carthage was then entirely conguered Rome, becoming a province Spuin derives from the Latin lerm Hispania, itsell a name of Punic origin. the areas of language and religion. the ancient Romans left a lasting cultural, logal and adhinistrative legacy in the Spanish history. The subsequent course of Sponish history added new elements to the countrys 's culture and iraditions. The Visgoths established united Haspania and kept the Latin and Christian legacy in Spain between the fall of the Roman Empire and the Early Midkle Ages Muslin influences plaed signifscnt role during the Middle Ages the conqueted by Umaggads. However, these inlluences were completely assimilatd the Spanish cullure, koding conflicts ullimately to Christian Reconquistt that would largely shape the culture the country.
Roman empine. The
of
Spunish
The culture of Turkey (Turkish: Tarkiye kolturb) or the Turkish culture (Tark kaluutt) combines a heavily diverse and beterogeseous set of elements that have been derived from the various cultures of the Eastern European. Eastern Mediterranean, Caucasian, Middie Eastem and Central Asian traditions Many of these traditions were iniiaally brought logetber the Otioman Empine, ethnii-ehhiic and mulli religioos state spanning across Southem Europe Eastern Europe, the Middle East and North Africa. During early years the Republic Turkey. established after collapse
Turkish
Otteman Empire, the govemument invested large sums resources sculpture, and other artistic fields around the country in ling with the newly implemented reformist and West-leaning policies. This was dome as part of process of mokenization, westernzation, and of creaing and outlningg a new Turkish cultural identity raiher than the previously established and depicted Ottoman ideatity.
arts such as architecture and
C.3 Prompts for RAG
The detailed information retrieved via RAG is shown in Table 6.
D Experimental Results
D.1 Detailed results
The detailed results are shown in Figure 6.
26

---

(a) Arabic
(b) Bengali
ChecCat
(c) Chinese
(d) Korean
(e) Portuguese
0.5
00
boad
Teallh
dessi
Veae
Taale
Thale Germany
(1) English
(h) Spanish
(i) Turkish
Figure 6: Detailed results on all tasks and all cultures.
27

---

Table 7: Comparison with the latest cultural specific LLMs.
Chinese
Bias
Spam
Avg
SeaLLM
.237
.357
.297
Taiwan LLM
.446
.341
.394
Culturel. JIM _One
.469
.602
.536
Culturel Zh
.487
.606
.547
Arabic
Hate
Offensive
Avg
CultureBank
.540
.642
.591
CultureLLM OOne
.540
.571
.555
CultureLLM Ko
.576
.661
.618
Korean
Abusive
Hate
Avg
ScaLLM
.523
.474
.499
CultureBank
.635
.522
.579
CultureLLM One
.619
.504
.561
CultureLLM Ko
.664
.628
.646
D.2 Comparative analysis with other cultural specifie models
We have conducted a comparative analysis with models designed for cultural understanding. Given the absence of models specifically tailored to Arabic, Bengali, German, Portuguese, Spanish, and Turkish cultures, our analysis focused on models with proficiency in Chinese and Korean cultures. Specifically, we examined ScaLLM [Nguyen et al., 2023b], which targets Southeast Asian cultural nuances, TAIWAN LLM [Lin and Chen, 2023], which is dedicated Chinese cultural contexts, and CultureBank [Shi et al., 2024), which focuses on several cultural groups. Table 7 highlight our findings in the Chinese and Korean benchmarks. The results clearly demonstrate that our CultureLLMs significantly outperform both SeaLLM and TAIWAN LLM, showcasing their superior ability to understand cultural subtleties on a broader scale.
D.3 Results analysis WVS seed data
We analyze the relevance of each task with the WVS and investigate if it correlates with the experi- mental results.
Offensive language detect:
1. Cultural Context and Sensitivity Offensive Language: The World Values Survey aims to capture cultural values and beliefs across different societies. One aspect of cultural values is the tolerance or acceptance of offensive language. some cultures, certain words or expressions may be considered highly offensive, while in others they may be more tolerated or even commonly used.
2. Social Norms and Acceptance: The survey may reveal societal attitudes towards the use of offensive language in various contexts, such in public discourse, media, interpersonal communication.
Hate speech detect:
Socictal Norms and Attitudes: The WVS provides data on socictal norms, attitudes towards minorities, and levels of societal trust. This data help understand the underlying societal conditions that might foster hate speech or, conversely, promote tolerance and inclusivity.
2. Cultural Context: Understanding the cultural context is crucial for effectively detecting and interpreting hate speech. The WVS offers a rich dataset for understanding cultural differences in values and norms, which can inform nuanced hate speech detection algo- rithms that are sensitive to context and do not inadvertently suppress legitimate expressions of cultural political dissent.
Stance detect:
28

---

1. Understanding Contextual Influences on Stance: The WVS can provide the cultural and societal background needed to understand why certain stances are more prevalent in specific regions or among certain demographic groups. This context can be invaluable for interpreting the results of stance detection analyses, especially when comparing stances across different cultures and societies.
Toxicity detect:
1. Reflection of Societal Norms in Online Behavior: The WVS provides insights into the prevailing norms and values within societies, which can indirectly inform the context within which toxic behavior manifests online. Understanding societal attitudes towards diversity, authority, individual freedom, and tolerance can help in interpreting the root causes of toxic behavior and devising appropriate responses.
2. Injection of More cultural nuances: that sensitive to cultural nuances and differences value systems. This can prevent the misclassification of content that may be culturally specific context-dependent, reducing the risk of censoring legitimate expressions of cultural or political identity.
Data from the WVS can inject more information
Threat detect:
1. Understanding Motivations and Behaviors: Insights from the WVS can help understand the cultural and societal contexts that may influence the behavior of individuals groups posing threats. This knowledge can inform more targeted and effective threat detection and mitigation strategies that consider the root causes of conflict aggression.
2. Cultural Sensitivity in Security Mcasures: Incorporating findings from the WVS can lead to more culturally sensitive security practices that respect local values and norms. This is crucial in global operations where misunderstanding cultural nuances can lead to ineffective or counterproductive security measures.
Bias detect:
1. Understanding Societal Norms and Attitudes: Insights from the WVS can help in under- standing the cultural and societal norms that underlie biases. By analyzing patterns global values and beliefs, we can identify prevalent stereotypes, prejudices, and discriminatory attitudes that may need to addressed bias detection efforts.
2. Injection of More cultural nuances: The WVS data can provide valuable context that sensitive to cultural differences in values and norms. This is better equipped to detect and mitigate biases data sets that reflect cultural nuances, ensuring that Al-driven decisions are fair and equitable across different societal contexts.
Abusive detect:
1. Cultural Contexts of Abuse: The WVS can help identify cultural norms that influence perceptions what constitutes abusive behavior. This is crucial for developing detection systems that are sensitive cultural differences, ensuring that they can effectively identify abuse without mistakenly flagging culturally specific but non-abusive interactions.
2. Injection of More cultural nuances: Insights from the WVS can inform the development of more nuanced algorithms for detecting abusive behavior by providing context on societal values and norms. This can help training models to recognize the subtle nuances that differentiate abusive from non-abusive communication in different cultural settings.
3. Evaluating Tolerance Levels: The WVS data can provide insights into societal tolerance levels towards different forms of behavior, including what might be considered abusive. This can help in assessing the urgency and type of interventions needed to address abusive behaviors various cultural contexts.
Spam detect:
Cultural Variations Communication: The WVS can shed light on cultural differences in communication styles and preferences, which can inform more nuanced spam detection
29

---

Table 8: Ablation study for CultureLLM-One Onne.
Ar
Bn
Zh
En
De
Ko
Pt
Es
Tr
avg
GPT-3355
0.517
0.567
0.448
0,526
0.609
0.642
0.593
0.517
0.587
0,556
+WVS
0.543
0,583
0.501
0.568
0,641
0.531
0,603
0.560
0.590
0,569
+WVS+a
0,532
0.572
0.482
0.543
0,631
0.563
0.620
0.543
0.585
0,564
+WVS+b
0.543
0.571
0.505
0.543
0,621
0.536
0.598
0.559
0.590
0,563
LMM-eee.LM--One
0.583
0.597
0.536
0.589
0.662
0.571
0.627
0.596
0.609
0.597
0.7
0.6
Performance
0.5
0.4
0.3
0.2
0.1
low resource
high resource
Tasks
Figure 7: We compare CultureLLM with baselines on low-resource language tasks and high-resource language tasks.
algorithms that are better able distinguish between legitimate mass communications and spam in different cultural contexts.
2. Attitudes Towards Technology and Privacy: Insights from the WVS regarding societal attitudes towards technology use, privacy, and data protection help in tailoring spam detection efforts to respect cultural norms and expectations. For instance, societies with a high value on privacy might be more receptive to stringent spam filters.
3. Attitudes Towards Technology and Privacy: Insights from the WVS regarding societal attitudes towards technology use, privacy, and data protection can help in tailoring spam detection efforts respect cultural norms and expectations. For instance, societies with high value on privacy might be more receptive to stringent spam filters.
D.4 Ablation on CultureLLM-Onee
The ablation study on Culturel.LMOOn is shown in Table 8.
D.5 Analysis on low-resource language tasks and high- language tasks
Figure shows the comparison between low and high resource tasks.
D.6 Results on other low-resource cultures
We fine-tuned Lule model specifically tailored for the Greek culture and assessed its efficacy on two Greek benchmarks focused on offensive language detection. Presented in the table below are the outcomes obtained from Zampieri et al. [2020], Pitenis et al. [2020]. Our findings indicate that, average, LultureLLLMeel surpasses gpt-3.5-turbo [OpenAI, 2023a] by 5.6% in performance. The results are shown in Table 9,
Table 9: Results on Greek culture
Model
OffensEval2020
gazzetta
GPT-3.5-turbo
0.4773 0.5201
0.3771
CultureLLM-cl
0.4461
30

---

D.7
Case study
Figure show the responses from CultureLLM and ChatGPT on four different cultures.
Cases on Arabic Culture
Cases on Germany Culture
Cases oa Spanish Culture
Cases on English Culture
the following sentence offenswee -ife 36, le yeu lokel y dis
Dace the fellowigg sestiene hate
Sillowing sentence offeasive" following settence offensive? Altih pero vaya y une collada les acopte las cottesias sin inimos de darles seso gue nurds extrolion, para operturists ta bajan.
Does ith following sestence have hate speeci?:: also believes slaves immigrane sres. hehs fool diegracs injuntices dons Asc gop while peop Sttel
specck:: dicsem Memert
Emus-Gruhe- Halle codlich seciant
Melppiz fagd fia HLopids
Resporae ChaigP ofTemiie deperding on cortodt and sclationdiip betwees the speaker listene, seen as diseespecttll or condescending, especially used in formal professooaa setting
(RelugeesaaWWWeke
Reaporse from ChatcP':: I'n sorty, but carnoc assess the coneett a specitic USL or webpege. Howevver, withort considering ayy esternal contest, the sennence you procoded doos not cuntain explictt hate effernise baggage
Response
langsage model, proside in arulysis the sontenos strictanc content, but comntt delermine the intention ur coreent behiial
Response ChaIGPT: As language model sentence you provided ornouss hanrage donestory ruppopeaate and premotes regative sicratypoo. important use cospectfull and incluive language when communicaligg with others.
acttence you provided docs not comait explicit hate apeech or any offassive langrage. Honvever, does incluuee hastrags related to political groups and saremene expressing oppostion refageos The paterrisal impplication interpenteinn ian of such conient very depending the context and intention spesker.
Response CaltreLLM-AN No. sentence offensive, cscual and affrcciinate way of aöthrssing somonne, exprossing fire loril and cor.
Respunse from Culiurcl L-Min:: You the following scntaner contiin haiz speech
Resporac from Cultured L. Es: No, the sestence toll offessive. It expresaes Iustration about being labeled os in opportunist when one has shown kindhess withost expecting anything return.
Does Ithe Billowing sestance have hate speedil : glisser gluser even called county shithole people stibolle people. sac.
the fillowing sersence ofteriie?
@UUSER pe منم lev yuy Male
Resporse Dont CalueeLLM- De: Yes, Tee seatence contains hase speech as it expeesses zegativee sentiment towards retugees.
Resgonse from PIT as Al baggugg model, don't have peraonal anotions, but I can stallyzz the ecnience for The given sentanee includes informal langragg and the use of emogis. is difficult deiennine the lonz irtention behind could interpratedl learteel expecssion
Requerae ClaGGT The you pooviled does
Is the following sennence offensive! (ddelvao Mis que seguro que macienon producoo de acono hisess viulación Pessrnsmmntic sunca lenido problamas para compuiiaa sin recurit la inimidación.
speecis
Dacs the fillowing sconence lave hate speech?? War das W Wochememde im Rubepoot unterwegs Obecall schwangere, muslimische Frazen mit eltem Katerssctoaan and Kindern iisizmistering
derogatory language sectiments, inpoortnnt to promose
Howeree, without
repocttul and inclusive communcction
Neverieelos informal
Resporas from Culurd M-Fi Yos, the sestence comains bate speech as includes devegaiory languge and itauls towoeds paricular country and is people
cautlous
language it night consitered offensive by some people.
Respprne from ChorGPT language model, can provide you with
Responae Som ChoGGPT As Al language model, is my duty to objective information. The provided considirnd effersive people, topice rape. impact that promete ropoctffl positive
satienes you speeci,, does contain
Resporse Bor Cultuet No. the sentence not offeasive. It roughty tranclates "Oh saduess, topic like expenssing anusement leughtr somothing
really dirst"
disrriirorory The
"Idlnniierrung suggsis negative view tonords prceenee Maslime which could interpected promuting
stzreotypes xenophobia. is importurt to pronote respectful and inclusve language when discusenn different cuttures or religions
Resposse Trums CulueLLM The first part the sentence offensive,
hameement and result of Howover, clarifice speaking has had problems consucring without cenorting
Reggonse from C Culturel. M-M-l Yes, the sertence conaits hate speech it generalinee and sigmaaiio progrant Musim and implies incentions regarding their presenes the Rulspott esgion.
part
Figure 8: The responses from ChatGPT and CultureLLM on four different cultures
E Fine-tuning on Llama and Results
E.1 Sctup
We use Lora [Hu et al., 2021] to fine-tune Llama-70b-Chat. The setting for Lora are list below:
Dora_alpha: 16
lora_ dropout: 0.1
r: 64
bias: none
task_type CAUSAL_IM
The detailed setting for training are list below:
num train_epochs: 6
31

---

Table 10:
More results on fine-tuning using Llama-2-706 model.
Ar
Bn
7h
De
Pt
Tr
AVG
Llama LM-oee
0,4852
0.3202
0.3315
0.4108
0,3631
0.4869
0.3792
0.3554
0.3856
0.3909
0.4911
0.3212
0.3533
0.4211
0.3827
0.4923
0.3892
0.3611
0,3855
0.3997
CultureLLM
0.5047
0.3398
0.3631
0,4214
0.3925
0.5019
0.3956
0.3752
0,3899
0,4093
Table 11: Results on forgetting experiments on Alama-2-7706
Model
GSM8K
BBH
Llama-2-70b
56.8
51.2
CultureLL.MAAr
56.9
53.2
CultureLLMMBI Bn
56.8
52.1
Lultur
56.8
56.6
CultureLLLMEEn
56.8
48.8
CultureL.LM-De
56.9
51.8
CultureLLM-KA
56.8
53.6
CultureLLM-Pt
56.8
55.1
CultureLLM-Es
56.7
53.2
CultureLL.M-TT
56.8
55.5
Avg
56.8
53.3
er sevicce_trann batch_sizee 4
sttpp::
optim: paged_adamw Baldaam_33bii
learning_rates 2c-4
weight_decays 0.001
fpl6: False
bf16: False
max_grad lrr__rrd__ooo _rrmm 0.3
max_ max_stees:- -1
warmup_ ratio: 0.03
group group_by_kee le_gthhh True
Ir scheduler type: constant
report to: tensorboard
E.2 Detailed results
The results of fine-tuning Llama2 are shown in Table 10. It shows the similar trends ChatGPT's re- sults. We conduct forgetting experiments on Llama-2. As the results shown Table 11, CultureLLM does not bring negative effect on general tasks.
F Details on Human Study
Information on participant in human study are shown in Table 12.
Participants asked rate the 100 samples according the following criterion:
1. Score 1: i. The sentences convey distinctly different ideas or concepts. ii No apparent connection shared meaning.
Score 2: i. Limited commonality in meaning, with noticeable disparities in wording. ii Shared concepts but with significant differences in expression.
32

---

Table 12: Information on participants in human study
Gender
Male
25
Female
25
Education
Bachelor
26
Master
24
Age
22
23
15
24
25
26
3. Score 3: i. Some overlap in meaning, but notable differences in wording phrasing. ii Context or emphasis might differ slightly.
4. Score 4: i. Minor variations in wording or structure, but the core meaning remains consistent. Synonymous expressions and interchangeable terms are present.
5. Score 5: i. The sentences convey the same information using different words. ii No discernible difference in meaning context.
33

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, they do.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section 6.
34

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We conduct experiments on 60 downstream tasks and verify the effectiveness of CultureLLMss The results are shown in Section
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We discuss the setup and details in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
35

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Our code released in http://woaormnnooooooo..ccccccld
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. //wips.ccppublll/// https//cpppbclcguueeeeeee http://oooSSSumssseeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
36

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We specify all the details in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report the statistical analysis in Figure 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide those information in Sections 4. and 5.3.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
37

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp://///euip....eee ctpps.cc/ppa cctppblic//eeeecccccccccccc ctwhie/tieeieiileleiiieeeeeeeee
Answer:
[Yes]
Justification:
We conform to the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss in Section 5.4.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We check the data manually to make sure the safety of data.
Guidelines:
The answer NA means that the paper poses such risks.
38

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credited the the creators original owners of assets and the license and terms use explicitly are mentioned and properly respected.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Our code and data released. And it is well documented in README.md.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: The details described in Appendix F.
39

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
The details are described in Appendix F.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
40


Reviews:
[{"review_id": "CuPEBhvSbn", "reviewer": "Reviewer_ASzm", "summary": "This research addresses cultural bias in large language models (LLMs) caused by training on mostly English data. Existing solutions can be expensive or require a lot of computing power. Here, they propose CultureLLM, a method that uses existing cultural surveys to create more training data and fine-tune LLMs. This method is shown to be effective and efficient for improving the cultural awareness of LLMs for 9 cultures.", "questions": "- I'm wondering how the performance (or errors) are associated with the coverage of the 50 questions used for fine-tuning.", "limitations": "I think the limitations brought by the authors are adequate.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written and easy to follow\n- The topic of the paper (culture LLMs) addresses a timely and important issue with the current LLM situation.\n- The approach (especially using existing survey data) is simple and reasonable.  \n- The experiments show the efficacy of the cultureLLM suggested.", "weaknesses": "- I don't see major weaknesses", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: I'm wondering how the performance (or errors) are associated with the coverage of the 50 questions used for fine-tuning.**\n\nWe analyze the performance for each task and report the WinRate in the table below. \n\n|            | offensive detect | hate detect | stance detect | toxicity detect | threat detect | bias detect | abusive detect | spam detect |\n| ---------- | ---------------- | ----------- | ------------- | --------------- | ------------- | ----------- | -------------- | ----------- |\n| ChatGPT    | 0.6143           | 0.5433      | 0.6758        | 0.5280          | 0.4270        | 0.4464      | 0.5889         | 0.5846      |\n| CultureLLM | 0.7203           | 0.6197      | 0.7359        | 0.6859          | 0.5172        | 0.5077      | 0.6622         | 0.6451      |\n| WinRate        | 0.1060           | 0.0764      | 0.0600        | 0.1579          | 0.0903        | 0.0612      | 0.0733         | 0.0605      |\n\nThe relevance of each task with WVS can be described in the following:\n- offensive language detect: \n1. Cultural Context and Sensitivity to Offensive Language: The World Values Survey aims to capture cultural values and beliefs across different societies. One aspect of cultural values is the tolerance or acceptance of offensive language. In some cultures, certain words or expressions may be considered highly offensive, while in others they may be more tolerated or even commonly used.\n- hate speech detect:\n1. Societal Norms and Attitudes: The WVS provides data on societal norms, attitudes towards minorities, and levels of societal trust. This data can help understand the underlying societal conditions that might foster hate speech or, conversely, promote tolerance and inclusivity.\n2. Cultural Context: Understanding the cultural context is crucial for effectively detecting and interpreting hate speech. The WVS offers a rich dataset for understanding cultural differences in values and norms, which can inform more nuanced hate speech detection algorithms that are sensitive to context and do not inadvertently suppress legitimate expressions of cultural or political dissent.\n- stance detect:\n1. Understanding Contextual Influences on Stance: The WVS can provide the cultural and societal background needed to understand why certain stances are more prevalent in specific regions or among certain demographic groups. This context can be invaluable for interpreting the results of stance detection analyses, especially when comparing stances across different cultures and societies.\n- toxicity detect:\n1. Reflection of Societal Norms in Online Behavior: The WVS provides insights into the prevailing norms and values within societies, which can indirectly inform the context within which toxic behavior manifests online. Understanding societal attitudes towards diversity, authority, individual freedom, and tolerance can help in interpreting the root causes of toxic behavior and devising appropriate responses.\n- threat detect: \n1. Understanding Motivations and Behaviors: Insights from the WVS can help understand the cultural and societal contexts that may influence the behavior of individuals or groups posing threats. This knowledge can inform more targeted and effective threat detection and mitigation strategies that consider the root causes of conflict or aggression.\n2. Cultural Sensitivity in Security Measures: Incorporating findings from the WVS can lead to more culturally sensitive security practices that respect local values and norms. This is crucial in global operations where misunderstanding cultural nuances can lead to ineffective or counterproductive security measures.\n- bias detect:\n1. Understanding Societal Norms and Attitudes: Insights from the WVS can help in understanding the cultural and societal norms that underlie biases. By analyzing patterns in global values and beliefs, we can identify prevalent stereotypes, prejudices, and discriminatory attitudes that may need to be addressed in bias detection efforts\n- abusive detect:\n1. Cultural Contexts of Abuse: The WVS can help identify cultural norms that influence perceptions of what constitutes abusive behavior. This is crucial for developing detection systems that are sensitive to cultural differences, ensuring that they can effectively identify abuse without mistakenly flagging culturally specific but non-abusive interactions.\n2. Injection of More cultural nuances: Insights from the WVS can inform the development of more nuanced algorithms for detecting abusive behavior by providing context on societal values and norms. \n4. Evaluating Tolerance Levels: The WVS data can provide insights into societal tolerance levels towards different forms of behavior, including what might be considered abusive. This can help in assessing the urgency and type of interventions needed to address abusive behaviors in various cultural contexts.\n- spam detect\n1. Cultural Variations in Communication: The WVS can shed light on cultural differences in communication styles and preferences, which can inform more nuanced spam detection algorithms that are better able to distinguish between legitimate mass communications and spam in different cultural contexts.\n2. Attitudes Towards Technology and Privacy: Insights from the WVS regarding societal attitudes towards technology use, privacy, and data protection can help in tailoring spam detection efforts to respect cultural norms and expectations. For instance, societies with a high value on privacy might be more receptive to stringent spam filters.\n3. Attitudes Towards Technology and Privacy: Insights from the WVS regarding societal attitudes towards technology use, privacy, and data protection can help in tailoring spam detection efforts to respect cultural norms and expectations. For instance, societies with a high value on privacy might be more receptive to stringent spam filters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sMTtPZ0Sli", "reviewer": "Reviewer_tmAM", "summary": "The paper presents CultureLLM, a fine-tuned LLM based on GPT 3.5 and fine-tuned on a cultural survey (in English) on 50 survey questions that are increased through semantically aware augmentation.  9 different languages are chosen with geographic choices about which survey to use to represent the languages. The CultureLLM is then tested in different tasks that have different language splits to show that the geographic/language awareness improves the performance than unaware LLMs.", "questions": "1. A questions that comes to mind, that you partly address, is the crosslingual nature of your finetuning, instead of translation. Would not having both even be better? And what would the effect also of using more local language based LLMs be? e.g. LeoLLM for german langauge.\n2. Augmentation with semantic similarity checks is something that has been worked on before for NLP augmentaiton. How do you deal with the challenge that even with high cosine similarity because of BERT embeddings, sentences that have very dissimilar meanings will be counted as similar or pass through your filter. \n\nNote: You refer to the crowdsource study as having details including IRB/Ethics information in Appendix E, but this is not so. You just describe the task questions. Was IRB/Ethics approval sought given that the nature of the questions may for some people find offensive?", "limitations": "The authors have worked to highlight their limitations and also challenges with the approach.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is an interesting problem that faces LLMs and how they are relevant to different locales. The authors work to setup the problem and also highlight their challenges along the way and how they dealt with them. An example is how one deals with limiting the questions and templates for instructions generation and then using augmentation via an LLM but doing a test for relevancy of the generated augmentation. The ablation studies and experiments show improvements of the more geographically language tuned models.", "weaknesses": "1. It is important to heavily note that language is not equal to culture and this tends to cause confusion in this paper. Culture is way more complex than language and it might have been easier to call the language splits as culture for writing but this will introduce misunderstandings that will muddy your message. \n2. Augmentation with semantic similarity checks is something that has been worked on before for NLP augmentaiton. How do you deal with the challenge that even with high cosine similarity because of BERT embeddings, sentences that have very dissimilar meanings will be counted as similar or pass through your filter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: It is important to heavily note that language is not equal to culture and this tends to cause confusion in this paper. Culture is way more complex than language and it might have been easier to call the language splits as culture for writing but this will introduce misunderstandings that will muddy your message.**\n\nWe strongly agree that language is not equal to, but only a part of culture. But using language to study culture is possible due to the following aspects:\n- Existing literature on culture understanding shows that culture boundaries are fluid, dynamic and uncertain. Delanoy emphasizes that cultures are not homogeneous or static entities but are fluid and dynamic. He critiques essentialist views that rigidly define cultural boundaries and instead promotes a more nuanced understanding that considers the intersections of various cultural factors, such as ethnicity, language, religion, and socio-economic conditions [1]. Appadurai also discusses the fluidity of cultural boundaries and the creation of new cultural forms [2]. Cultural boundaries can be geographical regions, language, religion and so on. Based on above statements, using language as cultural boundaries is reasonable.\n- Existing NLP works on culture also leverage labguage as culture boundaries. [3] focuses on Arabic and English culture. [4] focuses on 8 different cultures: English, Chinese, French, Russian, German, Arabic, Japanese and Korean. [5] also use language to split different cultures. The authors work on English, German, Russian, Bengali, Chinese, and Indonesian culture. [6] is a hand-crafted benchmark for evaluate diverse cultures. They also use languages as culture boundaries.\n- Most downstream benchmarks are classified via language and we cannot get more fine-grained perspectives. For example, if we want to evaluate the performance of Arabic model, we can find benchmarks in Arabic culture. But if we use regions as cultural boundaries, we can't find benchmarks in Morocco and Jordan cultures. \n- Note that the main contribution of the paper is to present a general algorithm that can augment LLM culture data but not specific to any cultures. In the future, if more fine-grained culture data are available, our algorithm can also work well.\n\n[1] Delanoy, Werner. \"What is culture.\" The Cambridge handbook of intercultural communication (2020): 17-34.\n\n[2] Appadurai, Arjun. Modernity at large: Cultural dimensions of globalization. Vol. 1. U of Minnesota Press, 1996.\n\n[3] Naous, Tarek, et al. \"Having beer after prayer? measuring cultural bias in large language models.\" ACL (2024).\n\n[4] Wang, Wenxuan, et al. \"Not all countries celebrate thanksgiving: On the cultural dominance in large language models.\" arXiv preprint arXiv:2310.12481 (2023).\n\n[5] Liu, Chen Cecilia, et al. \"Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings.\" arXiv preprint arXiv:2309.08591 (2023).\n\n[6] Myung, Junho, et al. \"BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages.\" arXiv preprint arXiv:2406.09948 (2024).\n\n**W2: Augmentation with semantic similarity checks is something that has been worked on before for NLP augmentaiton. How do you deal with the challenge that even with high cosine similarity because of BERT embeddings, sentences that have very dissimilar meanings will be counted as similar or pass through your filter.**\n\nThere are two tricks that work in our study\n- Threshold Tuning and Filtering: Adjust the similarity threshold to balance between false positives and false negatives. We choose a more conservative threshold, which can reduce the likelihood of accepting semantically dissimilar sentences.\n- Use of Cultural Contextual Information: BERT embeddings are context-sensitive, but sometimes context nuances are not fully captured. To mitigate this, we consider incorporating more cultural context.\n\n**Q1: A questions that comes to mind, that you partly address, is the crosslingual nature of your finetuning, instead of translation. Would not having both even be better? And what would the effect also of using more local language based LLMs be? e.g. LeoLLM for german langauge.**\n\nNice question! We fine-tune LeoLLM with our generated data in both English and German version. Results are shown below:\n| Task                | hate_check | hate_iwg_1 | hate | hate_off | offensive_eval |\n|---------------------|------------|------------|------|----------|----------------|\n| LeoLLM              | 0.46       | 0.21       | 0.18 | 0.33     | 0.42           |\n| LeoLLM+English data | 0.50       | 0.26       | 0.25 | 0.40     | 0.46           |\n| LeoLLM+German data  | 0.51       | 0.32       | 0.27 | 0.42     | 0.50           |\n\nThe results indicate that the fine-tuned model perform better when the language of fine-tuned data is same with that of pre-training data. This could be due to the fact that English is much more than the local languages, which can naturally assist fine-tuning.\n\n**Q2: Augmentation with semantic similarity checks is something that has been worked on before for NLP augmentaiton. How do you deal with the challenge that even with high cosine similarity because of BERT embeddings, sentences that have very dissimilar meanings will be counted as similar or pass through your filter.**\n\nThe response is shown in the response of W2.\n\n**Note: You refer to the crowdsource study as having details including IRB/Ethics information in Appendix E, but this is not so. You just describe the task questions. Was IRB/Ethics approval sought given that the nature of the questions may for some people find offensive?**\n\nThanks for the reminder! We have gotten IRB/Ethics approval for the human study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x7Y0RA4xJc", "reviewer": "Reviewer_XR5a", "summary": "This paper introduces CultureLLM, a novel and cost-effective approach to address the cultural biases in Large Language Models (LLMs) that arise from the dominance of English training data. Traditional solutions like prompt engineering and culture-specific pre-training are either expensive or computationally intensive, and often fail to address the paucity of data from low-resource cultures. CultureLLM leverages the World Value Survey (WVS) as seed data and employs a semantic data augmentation method to generate additional training data. The authors utilized only 50 seed samples fromWVS, extending them with augmented data to fine-tune culture-specific LLMs and a unified model, CultureLLM-One, covering 9 diverse cultures, including both rich and low-resource languages.", "questions": "- I am uncertain about the relevancy between the evaluated tasks and the fine-tuning data. How are values in the World Value Survey/extracted opinions relevant to offensive language/ hate speech/ stance? Can you elaborate?", "limitations": "The assumption of language as culture is a significant limitation of this work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper studies culture understanding which is an important problem.\n- The paper proposes an interesting data collection framework through role-playing.", "weaknesses": "- __Assumption of Language as Culture__. The paper equates languages with cultures, which is an oversimplification. Cultures are multi-faceted and cannot be fully encapsulated by language alone. There are significant cultural differences within the same language-speaking regions that may not be adequately captured by this approach. Even worse, on line 205, the authors said they picked “representative countries''. But how do you define “representative”? More importantly, this selection method will likely cause the fine-tuned LLMs to be biased towards these “representative” countries for certain languages.\n\n- __Limited Scope of Seed Data__.  The methodology relies heavily on a small seed dataset of only 50 samples from the World Value Survey (WVS). While the approach is cost-effective, it may not capture the full breadth and nuances of cultural diversity.\n\n\n- __Unfair Baseline Comparison__. The paper claims better performance than other culture-specific LLMs, like TaiwanLLM and SeaLLM. However, the comparison is unfair as these LLMs use different architectures and different amounts of pre-training data. The author could try fine-tuning the gpt-3.5-turbo with these baseline models’ fine-tuning/instruction-tuning data. Until then, “cost-effective” cannot be claimed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Assumption of Language as Culture. The paper equates languages with cultures, which is an oversimplification. Cultures are multi-faceted and cannot be fully encapsulated by language alone. There are significant cultural differences within the same language-speaking regions that may not be adequately captured by this approach. Even worse, on line 205, the authors said they picked “representative countries''. But how do you define “representative”? More importantly, this selection method will likely cause the fine-tuned LLMs to be biased towards these “representative” countries for certain languages.**\n\n*Why using one language to denote one culture:*\nWe strongly agree that language is not equal to, but only a part of culture. But using language to study culture is possible due to the following aspects:\n- Existing literature on culture understanding shows that culture boundaries are fluid, dynamic and uncertain. Delanoy emphasizes that cultures are not homogeneous or static entities but are fluid and dynamic. He critiques essentialist views that rigidly define cultural boundaries and instead promotes a more nuanced understanding that considers the intersections of various cultural factors, such as ethnicity, language, religion, and socio-economic conditions [1]. Appadurai also discusses the fluidity of cultural boundaries and the creation of new cultural forms [2]. Cultural boundaries can be geographical regions, language, religion and so on. Based on above statements, using language as cultural boundaries is reasonable.\n- Existing NLP works on culture also leverage labguage as culture boundaries. [3] focuses on Arabic and English culture. [4] focuses on 8 different cultures: English, Chinese, French, Russian, German, Arabic, Japanese and Korean. [5] also use language to split different cultures. The authors work on English, German, Russian, Bengali, Chinese, and Indonesian culture. [6] is a hand-crafted benchmark for evaluate diverse cultures. They also use languages as culture boundaries.\n- Most downstream benchmarks are classified via language and we cannot get more fine-grained perspectives. For example, if we want to evaluate the performance of Arabic model, we can find benchmarks in Arabic culture. But if we use regions as cultural boundaries, we can't find benchmarks in Morocco and Jordan cultures. \n- It’s interesting to incorporate more fine-grained cultural differences within the same language-speaking regions into LLMs. Our method is an initial attempt, which can generalize to more fine-grained cultures. Just changing the source of seed data can achieve this. However, we can not implement this method for all fine-grained cultures in one paper, because of time and resource limit.\n\n*Explanation on \"representative countries\":*\n\nIn terms of “representative countries”, our selection criterion is that we choose the country which has the *most population*. We agree that this can cause the fine-tuned LLMs to be biased towards the “representative countries”. However, we think the criterion may be the best way to align with majority of people from certain cultures. \n\nFinally, note that the main contribution of the paper is to present a general algorithm that can augment LLM culture data but *not* specific to any cultures or contries. In the future, if more fine-grained culture data are available, our algorithm can also work well.\n\n\n[1] Delanoy, Werner. \"What is culture.\" The Cambridge handbook of intercultural communication (2020): 17-34.\n\n[2] Appadurai, Arjun. Modernity at large: Cultural dimensions of globalization. Vol. 1. U of Minnesota Press, 1996.\n\n[3] Naous, Tarek, et al. \"Having beer after prayer? measuring cultural bias in large language models.\" ACL (2024).\n\n[4] Wang, Wenxuan, et al. \"Not all countries celebrate thanksgiving: On the cultural dominance in large language models.\" arXiv preprint arXiv:2310.12481 (2023).\n\n[5] Liu, Chen Cecilia, et al. \"Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings.\" arXiv preprint arXiv:2309.08591 (2023).\n\n[6] Myung, Junho, et al. \"BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages.\" arXiv preprint arXiv:2406.09948 (2024).\n\n**W2: Limited Scope of Seed Data. The methodology relies heavily on a small seed dataset of only 50 samples from the World Value Survey (WVS). While the approach is cost-effective, it may not capture the full breadth and nuances of cultural diversity.**\n\n- First, although the 50 data seem narrow, they however carry resourceful information since they are carefully designed and validaded by human experts. The seed data covers 7 topics: social values, security, science and technology, religious values, ethical values and norms, political interest and political participation, and migration. The data is high quality and have been widely used by a lot of literature. This statement can also be verified by the excellent performance of CultureLLM on downstream tasks.\n- Second, we have conduct pilot experiments on selecting the seed data from WVS. In our study, we explore the effect of sample’s quantity and different composition of samples on the performance of model. The results indicate that the 50 samples perform better than other settings. The reasons may be that fine-tuned LLMs perform better when the distribution of fine-tuned data similar to that of pre-training data. The distribution of those 50 seed data is similar to the pre-training data. So we select those samples.\n- Finally, the *main contribution* of the paper is not the seed data, but our data augmentation algorithm that remains general to any seed data and topics. That being said, in the future, with new topics and seed data available, users can easily develop their own versions of LLMs using our algorithm.\n\n**W3: Unfair Baseline Comparison.[...]Until then, “cost-effective” cannot be claimed.**\n\nWe response in general response for this part.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oiz8fZxx70", "reviewer": "Reviewer_eAV9", "summary": "This paper introduces a pipeline that can enhance LLM's ability to culture-aware tasks (such as hate speech detection, and bias detection. Their proposed CultureLLM included three stages: sampling, semantic data augmentation, and fine-tuning. They investigate the effectiveness of CultureLLM in nine languages and eight culture-related task. Their extensive experiments and analysis demonstrate that CultureLLM significantly improves LLM performance on culture-aware tasks while preventing catastrophic forgetting.", "questions": "See weakness.\n \nMore questions:\n1. Did you check if any fine-tining samples are overlapped with your downstream samples? Or some samples are very similar to downstream tasks?\n2. In your ablation studies, WVS+a only used a semantic template. Do you mean that LLMs are fine-tuned on semantic templates?", "limitations": "This proposed method uses the WVS dataset as seed data to augment their fine-tuning dataset. This limits its applicability.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents solid experiments to investigate the effectiveness of the proposed method. \n2. The paper is well-organized and easy to read.", "weaknesses": "1. Rely on human-annotated dataset (i.e., WVS). The author used WVS dataset as seed data to augment their fine-tuning dataset. This limits the applicability of the proposed method.\n2. Most evaluation downstream tasks are anti-social detection tasks (such as offensive language, hate speech, toxicity and abusive language detections). The results may be biased to such tasks. I also wonder why these tasks are culture-related tasks. Are there any particular examples? \n3. Some experiment details need clarification. 1. What are your fine-tuning hyperparamters? Like learning rate and training steps? 2. When you evaluate CultureLLM the downstream tasks, are the input samples in English or the particular language?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Rely on human-annotated dataset (i.e., WVS). The author used WVS dataset as seed data to augment their fine-tuning dataset. This limits the applicability of the proposed method.**\n\nUsing human-annotated data to help research is a popular choice for most of the LLM papers, as shown in [1-7], where they all use human-annotated dataset as seed data. On this point, one noteworthy advantage of our approach is that we only use 50 samples, which is significantly less that existing literature, claiming our less reliance on annotations.\n\n[1] Wang, Ruida, Wangchunshu Zhou, and Mrinmaya Sachan. \"Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models.\" arXiv preprint arXiv:2310.13671 (2023).\n\n[2] Chen, Zixiang, et al. \"Self-play fine-tuning converts weak language models to strong language models.\" arXiv preprint arXiv:2401.01335 (2024).\n\n[3] Yu, Longhui, et al. \"Metamath: Bootstrap your own mathematical questions for large language models.\" arXiv preprint arXiv:2309.12284 (2023).\n\n[4] Li, Xian, et al. \"Self-alignment with instruction backtranslation.\" arXiv preprint arXiv:2308.06259 (2023).\n\n[5] Liu, Bingbin, et al. \"Tinygsm: achieving> 80% on gsm8k with small language models.\" arXiv preprint arXiv:2312.09241 (2023).\n\n[6] Singh, Avi, et al. \"Beyond human data: Scaling self-training for problem-solving with language models.\" arXiv preprint arXiv:2312.06585 (2023).\n\n[7] Huang, Yiming, et al. \"Key-point-driven data synthesis with its enhancement on mathematical reasoning.\" arXiv preprint arXiv:2403.02333 (2024).\n\n**W2: Most evaluation downstream tasks are anti-social detection tasks (such as offensive language, hate speech, toxicity and abusive language detections). The results may be biased to such tasks. I also wonder why these tasks are culture-related tasks. Are there any particular examples?**\n\n- First of all, the downstream tasks are just evaluation, but not fine-tuning datasets. That being said, we only perform evaluation on those downstream datasets. Therefore, your comment \"the results may be biased to such tasks\" does not hold.\n- Second, it is actually challenging to find enough public datasets to evaluate the cultural models. Those downstream tasks are what we have found. Most of them may relate to anti-social, but our models are not. In fact, those extreme cases are what cultural misunderstanding could possibly occur, which can help \"stress test\" our models and verify its real application.\n- Third, the table below shows the input-output pairs of \"offensive language detect\" task. The offensive contents are different for different cultures. The model needs to judge if the input content is offensive and label it with \"OFF\" or \"NOT\". And the label is related to the cultural context.\n\n| Culture | Content | Label |\n| -------- | -------- | -------- |\n| Arabic     | @USER يا حُب يا دُنيا جديدة يا احلى يا احلى ابيات القصيدة♥️.     | NOT     |\n| German     | Liebe Freunde, Gauck,das Symbol des Sklaventums!    | OFF     |\n| Portuguese     | AVANTE  MEDIDAS o Brasil clama por elas pra cima dos corruptos Cadeia neles   | NOT     |\n\n**W3: Some experiment details need clarification. 1. What are your fine-tuning hyperparamters? Like learning rate and training steps? 2. When you evaluate CultureLLM the downstream tasks, are the input samples in English or the particular language?**\n\nThanks for the reminder! We will add the details into the next version of paper.\n- Fine-tuning hyperparamters. For gpt-3.5-turbo, the number of training epochs is 3. And we use default settings for other hyperparamters. For Llama-70b-Chat, the details on setup are shown in Sec.D.1.\n- Language of input sample: The evaluation benchmarks are all in particular languages.\n\n**Q1: Did you check if any fine-tuning samples are overlapped with your downstream samples? Or some samples are very similar to downstream tasks?**\n\nWe checked manually and it turns out the fine-tuning samples are not overlapped with the downstream samples. It is easy to understanding since the seed data from World Value Survey(WVS) are related to cultural value, which focus on more abstract information. The evaluation downstream tasks care about the meaning of a specific sentence. \n\n**Q2: In your ablation studies, WVS+a only used a semantic template. Do you mean that LLMs are fine-tuned on semantic templates?**\n\nNo. \"WVS+a\" means the output of Step 2 instead of output of Step 3 in Fig 2.\n\n- - -\n\nIf you are satisfied with the answers, please consider improving the rating to support our work! If you have more questions, please do not hesitate to let us know:)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and AC,

We want to thank all reviewers for pointing out our strengths, including:
- problem significance: "addresses a timely and important issue with the current LLM situation", "an interesting problem that faces LLMs and how they are relevant to different locales"
- novel method: "is simple and reasonable"
- solid experiment: the experiments show the efficacy of the cultureLLM suggested, with solid experiments to investigate the effectiveness of the proposed method
- writing: "well-organized and easy to read"

Rebuttal for each reviewer has been submitted in respective sections. Here, we would like to clarify two common weaknesses.

Specifically, as raised by reviewer *XR5a* and *tmAM*, there remains one common weakness about the relationship between culture and language, which we aim to address here:

We strongly agree that language is *not* equal to, but only *a part of* culture. But using language to study culture is possible due to the following aspects:
- Existing literature on culture understanding shows that culture boundaries are fluid, dynamic and uncertain. Delanoy emphasizes that cultures are not homogeneous or static entities but are fluid and dynamic. He critiques essentialist views that rigidly define cultural boundaries and instead promotes a more nuanced understanding that considers the intersections of various cultural factors, such as ethnicity, language, religion, and socio-economic conditions [1]. Appadurai also discusses the fluidity of cultural boundaries and the creation of new cultural forms [2]. Cultural boundaries can be geographical regions, language, religion and so on. Based on above statements, using language as cultural boundaries is reasonable.
- Existing NLP works on culture also leverage language as culture boundaries. [3] focuses on Arabic and English culture. [4] focuses on 8 different cultures: English, Chinese, French, Russian, German, Arabic, Japanese and Korean. [5] also use language to split different cultures. The authors work on English, German, Russian, Bengali, Chinese, and Indonesian culture. [6] is a hand-crafted benchmark for evaluate diverse cultures. They also use languages as culture boundaries.
- Most downstream benchmarks are classified via language and we cannot get more fine-grained perspectives. For example, if we want to evaluate the performance of Arabic model, we can find benchmarks in Arabic culture. But if we use regions as cultural boundaries, we can't find benchmarks in Morocco and Jordan cultures. 
- Note that the main contribution of the paper is to present a general algorithm that can augment LLM culture data but not specific to any cultures. In the future, if more fine-grained culture data are available, our algorithm can also work well.

Another thing we want to discuss is about *"fair comparation"*. We agree that it is necessary to fine-tune gpt models on the training data of TaiwanLLM and SeaLLM. However, a sad story is that their training data is not publicly accessible. In fact, the popular LLM leaderboards such as Chatbot Arena and AlpacaEval ranks models regardless of their sizes, training data, and post-training, but only the final performance on the same benchmarks. Moreover, we realize that it is never easy to reach a "fair" comparison since if we fine-tune the same models on their data, it is unfair for our approach since their pre-training data is significantly larger than ours. We would like to claim that given limited budget and GPU hardware, CulturePark remains a cost-effective solution to fastly build a cultural-specific LLM for low-resource culture. This is the main contribution of the paper.


[1] Delanoy, Werner. "What is culture." The Cambridge handbook of intercultural communication (2020): 17-34.

[2] Appadurai, Arjun. Modernity at large: Cultural dimensions of globalization. Vol. 1. U of Minnesota Press, 1996.

[3] Naous, Tarek, et al. "Having beer after prayer? measuring cultural bias in large language models." ACL (2024).

[4] Wang, Wenxuan, et al. "Not all countries celebrate thanksgiving: On the cultural dominance in large language models." arXiv preprint arXiv:2310.12481 (2023).

[5] Liu, Chen Cecilia, et al. "Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings." arXiv preprint arXiv:2309.08591 (2023).

[6] Myung, Junho, et al. "BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages." arXiv preprint arXiv:2406.09948 (2024).

- - -

We hope that your concerns can be addressed. Thank you for your hard work.

Authors of CultureLLM

Meta-review: This paper proposed CultureLLM, a novel cost-effective approach to addressing culture bias in LLMs. Since prompt engineering and culture-specific pre-training are expensive, CultureLLMs leverages surveys as seed data to further augment the training data and fine-tune the LLMs. The model shows improvements on LLM cultural awareness for 9 cultures.

Strengths:
Overall, the paper is clearly written, easy to follow and presents a novel cost-effective approach to address cultural biases in LLMs called CultureLLM. Authors have interesting way to setup the problem, they leveraged surveys to generate additional training data and fine-tune culture specific LLM and a unified model. 

Weaknesses:
There are no major weaknesses. Just prior to the final submission, authors should address the questions of the reviewers, specifically  
(1) motivate the assumption of language as a culture
(2) explain why evaluation was done on downstream tasks such as offensive language, toxicity’s etc and results may be biased towards such task; address the question “unfair baseline comparisons” and also are there tasks which could better capture culture-related task
(3) address the feedback on “unfair baseline comparison” with other LLMs given the
Decision: Accept (poster)