CulturePark: Boosting Cross-cultural Understanding in Large Language Models
Authors: CHENG LI
Damien Teney
Linyi Yang
Qingsong Wen
Xing Xie
Jindong Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Culture bias
large language model
fine-tuning
Primary area: fairness
TL;DR: We provide a LLM-powered multi-agent communication platform to generate fine-tuning data for cultural topics

Abstract: Cultural bias is pervasive in many large language models (LLMs), largely due to the deficiency of data representative of different cultures.
Typically, cultural datasets and benchmarks are constructed either by extracting subsets of existing datasets or by aggregating from platforms such as Wikipedia and social media.
However, these approaches are highly dependent on real-world data and human annotations, making them costly and difficult to scale.
Inspired by cognitive theories on social communication, this paper introduces CulturePark, an LLM-powered multi-agent communication framework for cultural data collection.
CulturePark simulates cross-cultural human communication with LLM-based agents playing roles in different cultures.
It generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs.
Using CulturePark, we generated 41,000 cultural samples to fine-tune eight culture-specific LLMs.
We evaluated these models across three downstream tasks: content moderation, cultural alignment, and cultural education.
Results show that for content moderation, our GPT-3.5-based models either match or outperform GPT-4 on $41$ datasets. Regarding cultural alignment, our models surpass GPT-4 on Hofstede's VSM 13 framework.
Furthermore, for cultural education of human participants, our models demonstrate superior outcomes in both learning efficacy and user experience compared to GPT-4. CulturePark proves an important step in addressing cultural bias and advancing the democratization of AI, highlighting the critical role of culturally inclusive data in model training. Code is released at https://github.com/Scarelette/CulturePark.

CulturePark: Boosting Cross-cultural Understanding in Large Language Models
Cheng Li"
Damien Teney Idiap Research Institute
Institute of Software, CAS chenglicat02280gaal.. com
contactodami enteney. info
Linyi Yang Westlake University
Qingsong Wen
Xing Xie
Squirrel AI
Microsoft Research
yanglinyi@xestlakke edu. cn
q gingsongedu@gmail.com
xing. wwe@microsoot.ccmmmmmmm com
Jindong Wang William & Mary wang800sm. edu
Abstract
Cultural bias is pervasive in many large language models (LLMs), largely due to the deficiency of data representative of different cultures. Typically, cultural datasets and benchmarks are constructed either by extracting subsets of existing datasets by aggregating from platforms such Wikipedia and social media. However, these approaches are highly dependent on real-world data and human annotations, making them costly and difficult to scale. Inspired by cognitive theories social communication, this paper introduces CulturePark, an LLM-ppwered multi-agent communication framework for cultural data collection. CulturePark simulates cross-cultural human communication with LLM-based agents playing roles in different cultures. It generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs. Using CulturePark, we generated 41,000 1,000 cultural samples to fine-tune eight culture-specific LLMs. We evaluated these models across three downstream tasks: content moderation, cultural alignment, and cultural education. Results show that for content moderation, our GPT-3.5-based models either match or outperform GPT-4 on 41 datasets. Regarding cultural alignment, our models surpass GPT-4 on Hofstede's VSM 13 framework [Hofstede, 2013] Furthermore, for cultural education of human participants, our models demonstrate superior outcomes both learning efficacy and user experience compared to GPT-4. CulturePark proves important step in addressing cultural bias and advancing the democratization of AI, highlighting the critical role culturally inclusive data in model training. Code is released at https: :ttp:::/giiiie thub. com/ Stt.//eeeeeeeeeeeeeeeeeeeeaeeeee wawaareeeeeeeeeuuuteeeeeeee.
1 Introduction
Culture is an important part of human society, composed of human beliefs, norms, customs, etc. Onpencer-aatey and Franklin, 2012]. As large language models (LLMs) play a vital role in daily communication (Yang et al., 2024], recommendation systems [Li et al., 2023, Fan et al., 2023], and
WWrr done during Cheng's intersship at MSRA. "orrespondiig author. Work done at MSRA.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Culture-inspired communication
Cross-cultural Dialogue
Theory-inspired design
Coeni-ive conflict theory
Collect
Social cognit tive theory
Bengali
Chinese
Extract
Custon
Confiit
Interaction
belief
Better undersianding
Fine-tune /
Arabic
German
Downstream Applications
culture-specific LLMs
Content Moderation
Detect stance, hate speech, bias, abusive, blas, threat..
Main
CulturePark-Ar
contact
Align with Hotstede's cultural dimension Uheory?
Cul turePork-nn
Cultural Aligament
Korean
CulturePack-2h
Cultural
Tearn cultures by
Education
human-11M interaction
Portuguese
Spanish
Figure 1: CulturePark is an LLM-based multi-agent communication platform for cultural data collection. Leveraging CulturePark, we can collect a cross-cultural dialogue dataset, which can then be used for fine-tuning culturally specific LLMs to be applied to different downstream tasks: content moderation, cultural alignment, and cultural education.
education [Shaikh et al., 2023], it is imperative for LLMs to perceive and reflect different cultures. However, current state-of-the-arc LLMs have been reported to be biased towards mainstream culture while ignoring others, resulting in a cultural bias problem [Liu et al., 2023, Cao et al., 2023, Masoud al., 2023, Naous al., 2023, Wang al., 2023, Johnson al., 2022b]. This leads to stereotypical impressions of different cultures, which can even exacerbate social conflicts [Ryan et al., 2024]. The main reason behind cultural bias is that the training corpus of LLMs dominated by English data that express the cultural values and opinions of western people. Much less can be learned about other cultures simply because there is less data available, i.e., low-resource situation.
Existing approaches to solve the cultural bias problem in LLMs include prompt engineering [Kovač et al., 2023, Wang al., 2023, Rao et al., 2023] and pre-training in -ngllish languages [Pires et al., 2023, Chan et al., 2023, Nguyen et al., 2023b, Pipatanakul et al., 2023, Abbasi et al., 2023, Lin and Chen, 2023]. Prompt engineering consists of tuning prompts for different cultural tasks, but the benefits do not hold reliably across various downstream tasks. Pre-training in various languages is promising, but the data collection and the pre-training itself are both very costly. More importantly, cultural differences are embodied in many aspects such as opinions, customs, norms, and languages. model serving all cultures may face cultural conflict and misalignment problems [Liu et al., 2023, Cao et al., 2023, Masoud et al., 2023]. Thus, it is necessary to fine-tune culture-specift models that target specific cultures. Recently, Li al. [2024] proposed CultureLL,, which augments the fine-tuning data of LLMs via semantic data augmentation to train culture culture-specific LLMs. However, the generated data lack diversity because it is implemented by generating semantically equivalent sentences of seed examples.
In this paper, we present CulturePark, -.-M-powered multi-agent framework simulate cross- cultural communication of humans. As shown in Figure 1, CulturePark serves as an effective data collection platform to generate diverse and high-quality cultural datasets via multi-agent communica- tion. CulturePark consists of a main contact (an English-speaking agent, = Lily) who is in charge of the multi-tun fullii-tur dialogue and several cultural delegates (e.g.. Abdul) who interact with the main contact and create cognitive conflicts.? After an initial problem is provided as input to the framework, the agents discuss the problem and express their opinions. Their different cultural backgrounds and genders boost diverse opinions and encourage one another to think more deeply. Original questions and ground truth can be augmented by creating novel questions and more comprehensive answers. The interactions eventually generate a cross-cultural dialogue dataset that contains deep and compre- hensive thinking and informative knowledge of different cultures. Detailed statistics are shown in Table 4. We then refine the original dataset to factually verify and increase its diversity, which is used to fine-tune culturally specific LLMs for downstream tasks, as shown in Figure 2.
We choose the English agent the main contact since LLMs do the best role oole-playing using English.

---

From the perspective of cognitive social science, our framework is inspired by Cognitive Conflict Theory (CCT) [Limón, 2001, Cosier and Rose, 1977] and Social Cognitive Theory (SCT) [Fiske and Taylor, 1991] to foster collaborative and communicative environment for mutual understanding of cultures. Specifically, CulturPPark allows agents to encounter cognitive conflicts, which triggers deeper thinking on certain topics according to CCT [Limón, 2001, Cosier and Rose, 1977]. At the same time, a deeper understanding of cultures can be prompted through interaction and com- munication with other agents, suggested by SCT [Fiske and Taylor, 1991]. In favor of these theories, we found that CulturePark triggers LLMs' cross-cultural understanding ability, boosts novel opinions by allowing agents think deeper, and benefits data augmentation by creating more comprehensive answers to the questions. CulturePark has the potential to facilitate the data collection of culture-related tasks, cultural value alignment, and improving AI democracy.
In summary, the contributions of this paper are three-fold:
1. We introduce CulturePark,, cost-efficient multi-agent framework boost the cross-cultural understanding in LLMs. Our platform creates cognitive conflicts and interactions between different cultural agents. More importantly, the platform uncovers several interesting findings, such as communication enables the cross-culturl understanding ability of LLMs, boosts novel opinions, and benefits data augmentation.
2. Leveraging CulturePark, we generate and augment novel questions and more comprehensive an- swers, leading to 41 K cultural samples in total. Those data contain rich and diverse information on different aspects of culture, such as norms, opinions, and backgrounds. We then fine-tune cultural specific LLMs for different cultures.
3. We evaluate CulturePark in three key experiments: 1) The fine-tuned LLMs outperforms GPT- 4 in 5 cultures on 26 content moderation tasks and approach GPT-4 on other tasks; 2) Our fine-tuned L.LMs achieve better performance on cultural alignment experiments via Hofstede's cultural dimensions theory [Geert Hofstede, 2010]; and 3) Human participants can perform more effective culture learning in situated learning experiments and show better satisfaction compared to GPT-4.
2 Related work
2.1 Cultural bias in LLMs
A body of research has explored cultural biases in LLMs. Johnson al. [2022a] examined conflicts in model outputs and input values, using moral value pluralism to analyze the responses of GPT-3 against global demographics. Their results showed that conflicting values were more aligned with the dominant US values reported. Naous et al. [2023] highlighted bias towards Western culture in models processing Arabic, exacerbated by English-aligned prompts, suggesting mitigation through cultural tokens. The Cultural Alignment Test (CAT), based on Hofstede's framework [Geert Hofstede, 2010], evaluated cultural values models such as ChatGPT and Bard across different cultures, revealing the highest cultural alignment for GPT-4 with US values [Masoud et al., 2023]. Cao et al. [2023] found that ChatGPT aligned well with American culture but struggled with other cultures, particularly under English prompts. Additionally, Liu et al. [2023] reported that multilingual LLMs had limited abilities reason with proverbs and exhibited "culture gap" in handling translations, leading the development of the MAPS dataset for assessing proverb comprehension in six languages.
2.2 Cultural benchmarks and datasets
Extensive research has focused on developing cultural benchmarks, which can be categorized into two types: collecting existing datasets and synthesizing new ones. First, most of the work adopted existing datasets sources of cultural data. Wang et al. [2023] introduced a benchmark that uses cultural items to analyze cultural dominance, based on sources such as WVS [Survey, 2022b] and PCT [Mudde, 2016]. Later work includes Cultural Aligmment Test [Masoud et al., 2023], NORMSAGE [Fung et al., 2022], Becce [Zhao et al., 2024], and NORMAD [Rao et al.. 2024] that sourced from different existing datasets. Other types of data sources include CultureAtlas [Fung et al., 2024] and MAPS [Liu et al., 2023] which collected data from Wikimedia; Candle [Nguyen et al., 2023a] and CultureBank [Shi et al., 2024] sourced their data from social media such as Tiktok and Reddit. In contrast, there was emerging trend to perform data augmentation for cultural LLMs. Li et al.
3

---

[2024] proposed semantic data augmentation to synthesize cultural data by enriching the semantic equivalence of the generated samples.
CulturePark significantly differs from those that perform direct data collection from existing datasets; it also differs from Lu [Li al., 2024] since CulturePark leverages multi-agent communica- tion for data generation, which is more natural and can generate more diverse datasets.
2.3 Existing solutions to cultural bias
There are primarily two types of approach to addressing the problem of cultural bias: prompt engineering and pre-training. The work of [Kovač et al., 2023, Wang al., 2023] viewed LLMs as amalgamations of cultural perspectives, which can be oriented toward specific cultural perspectives through prompt engineering. In contrast, Rao al. [Rao et al., 2023] integrated cultural values directly into the prompts. Although prompt engineering is cost-effective, its efficacy is questionable, particularly in low-resource cultures where LLMs may lack relevant cultural knowledge due to underrepresentation in pre-training data. An alternative strand of research focuses on pre-training and fine-tuning [Pires al., 2023, Chan et al., 2023, Nguyen et al., 2023b, Pipatanakul et al., 2023, Abbasi et al., 2023, Lin and Chen, 2023]. Those approaches developed culturally aware L.LMs for various cultures by assembling large-scale pre-training datasets, followed by fine-tuning to enhance aligmment. Despite achieving significant performance improvements, these methods were both costly and time consuming, making them impractical for broader application across numerous cultures and countries. Furthermore, they still face challenges in low-resource cultures where acquiring pre-training data is difficult. For example, MaLA-500 [Lin et al., 2024] aimed to train a new LLM in Llama 2 to support 534 languages, illustrating the resource- intensiie nature of this approach. Unlike these approaches, CulturePark provides a cost-effective solution to the cultural bias problem. including data augmentation and fine-tuning.
3 CulturePark
3.1
Design
CulturePark is an I.LM-powered' cross-cultural communication framework that generates data to support culture-related research such building cultural-specific LLMs and performing cultural alignment. is inspired by Cognitive Conflict Theory (CCT) and Social Cognition Theory (SCT) design multi-turn communications for deeper understanding of cultural topics. CCT posits that cognitive conflicts can help individuals engage more in deeper thinking [Limón, 2001, Cosier and Rose, 1977]. and SST emphasizes that individuals can deepen their understanding of perspectives through explanation and debate [Fiske and Taylor, 1991].
Figure shows the overview of CulturePark. To enable Engse interaction, design two types of cultural agents: the main contact and the cultural delegate. Specifically, the main contact agent, = Lily, is from English culture and is responsible for all conversations with delegates from different cultures such as I Abdul from Arabic and = Javier from Spanish culture. The complete information of agents and culture in Table 7. As shown in Figure 2(a), we input a system prompt to LLMS which contains the background setting and initial question to initiate the conversation. The initial question, such as "How do you think about one of my main goals life has been to make my parents proud? Please provide your opinions and reasons", is obtained from WVS [Survey, 2022b] and GAS [Survey, 2022a], two popular cultural surveys whose examples are shown in Figure 7. After that, the agents conduct cross-cultura conversations to generate data, Currently, CulturePark supports 8 cultures and 2 genders while more cultures can be easily added. Those agents could conduct in-cultural or cross-cultural communication, while we rely on cross- cultural more since in-cultural communication will likely generate less diverse topies (e.g., Figure 10). We discuss the quality of data from in-cultural and cross-cultural communication and the influence of gender in Section 5.
*We use GPT-3.5-Turbe in this work due to its great performance, high efficiency, and manageable price. CulturePark is flexible platform that naturally supports other LLMs such as GPT-4 and Llama models. More importantly, CulturePark allows to use different LLMs for the main contact and delegate (e.g., GPT-3.5 for main contact and Llama-2 for delegate), which makes it flexible to extend to other models and evaluate the cross-mode understanding ability.

---

We designed improved prompting techniques to maintain high-quality conversations. First, the cultural bias of the main contact and cultural delegate is reduced by designing self-calibratiin prompts to calibrate their outputs. We use seed datum that contains the attitude of the target culture to the input question to guide the dialogue. All the following statements should conform to the answer in seed. As shown in Figure 2(a), we introduce the opinion from = Abdul's ' culture and ask I Abdul and = Lily to conform to their cultures. The effect of the self-calibration prompt is shown in Figures 13(a) and 13(b). Without self-calibration prompts, = Abdul's 's opinions contradict with Arabic people. Second, the redundancy of the output, i.e., LLMs always generates similar dialogues after multi-turn communication. We devise two communication styles: one is guidaccee prompts which can direct the dialogue to generate more diverse and informative data, such as "Are there anything in your culture related to the problem talked before?" and "Do you agree with her? Provide more reasons to support your idea?", and the other is free chat that does not need humans participate and motivate the inner creativity of LLMs. Figures (a) and 1 (b) show cases of seff-guidance prompting and free chat, respectively.
3.2 Data refinement and fine-tuning
The seed questions initiating the communication have two sources: World Values Survey (WVS) [Sur- vey, 2022b] and Global Attitudes surveys (GAS) from Pew Research Center [Survey, 2022a]. WVS is a global research project that explores people's 's beliefs and values worldwide, examining how these beliefs evolve over time. Pew Research Center, a nonpartisan organization, provides data and research on public opinion, social issues, and demographic trends both in the U.S. and globally. Its Global Attitudes surveys cover wide range of topics, including politics, media, technology, religion, race, and ethnicity. In total, we select 4. seed data and generate 41k dialogues (each dialogue contains several sentences). We show the details of the data numbers for different cultures in Table 8. We also performed a statistical analysis on the GPT-4-based dataset. As summarized in Figure 8, the dataset contains human belief (59.68%) norm (29.54%) and custom (10.78%) involving data on 8 different cultures. Figures 7 and 1 (a) show some examples of the seed data and the generated dialogues.
The generated dataset may not be directly used for fine-tuning since it could contain redundant and incorrect information that should be handled. As shown in Figure 2(b), we design data refinement to refine the dataset. First, the opinions target culture are extracted from the dialogues generated via GPT-4, such as "The Arabian equates their parents' happiness and natisfaction on to their own success" and "The Arabian emphasize Sabr, which is about showing reailience, maintaining positive attitude and having faith during difficult times". Second, sev- eral extracted opinions could be irrelevant to the initial question or contradict with seed data, mo- tivating us to perform verification to reserve only highly related opinions. Furthermore, since the generated data could be semantically similar, we remove redundant samples to improve diversity. To be specific, we obtain sentence embeddings via text-embedding---ssmie cluster the embedding using K K-means. We reserve one sample for each cluster as representative data, Eventually, we get the high-quality cultural data for different cultures. The ablation of the refinement in Table 2.
[OpenAl, 2024] and
Algorithm 9 shows the pipeline for data refinement. After refinement, there are 41k samples (input- output pairs) left for fine-tuning, i.e., one sample for dialogue. Examples of the samples are provided in Figure 12. Afterwards, we can fine-tune cultural-specific LLMs using either spurcsource LLMS or fine-tuning service. In this paper, we mainly use OpenAl API fine-tune GPT-3.5- Turbo due to its efficiency. Hyperparameters are shown in Table 6. We further provide fine-tuning experiments Llama2-70b in Section 5.4.
3.3 CulturePark benefits cultural understanding and fine-tuning
There are some interesting observations in communication among agents from different cultures.
Communication triggers LLMs' cross-cultural understanding ability. We observed that agents try to understand each other's opinions and the reasons sourced from their different cul- tural backgrounds. For example, the blue sentences in Figure 2(a) show cross-cultural under- standing ability of LLMS, such as "I do understand and respect the sentiment of wanting to make parents proud, as they have often sacrificed a lot for their children". Leverag- ing GPT-4-Turbo, we analyzed the topics in the dataset such as human beliefs,, norms, and customs,
5

---

Abdul is an Arabion bay and Lily is an Amzrican gin. Aboul my main goals should conform my thinking All Lly's
should calliration)) Opinion extrecton from dialoguee
Strongly sgree with
1. The Arziaay baliono that that caftrre
conform to his culture. (>Buccigronn
emgiaadee
How do you trink aboul "dnw of my main goals in life his been make my parents proud" Please provide your apiiions and reasone"
showing reailence. maintaring poultve-.
The Ander ezaino permnt
I frrmly believe that one of main goas life has been to make my parents proud my sullure, lamily is parsmount importance and
Factual veriiicttion
(especting and honoring our parents is fundamenzal value...
Seed Cota
The Arables belleuas that in thei cutture.
I understand and respect the sentment of warting make parents proud es they have often secrificee lot for their chuldaen. our cultuuee though, theee's strong amphasis on individusityy and personal
Ratation Judgement
brnught with diep sense gorttude
Araiion they
achieneeeen....
The Arub equates their paranes hoppiress
Do you agree with her! Provide more reasons to support your idea.
reopect Lily perspec fhw represents her cultural values Howwer still hold my beiee strongly My perspective stems from...
Redundancy removal
can cortainly appreciate where Abdul comiing from and think there' a to be wdmined sbout the collectie family values in his culture.. -erre...i. liss ien't say dor't seek make our pareets proud, rother our success dedined our own personal about finsing happiness.
Clasteing
Repwarkkhh
and positive
data
farilyy espectations
(a) Cross-culturee dialogue dataset
(b) Data refinement for fine-tuning
Figure 2: Cross-cultural dialogue and data refinement for fine-tuning LLMs using CulturePark.
which can be further used as data collections for building culturally specific models. Appendix B.2 shows the details of the dialogue dataset, indicating that the generated topics mostly about culture. Then, we randomly sampled 750 dialogues for each culture and evaluated the communication using the prompts in Appendix E. As summarized in Table 4, on average, the ratio of statements that express cross-cultural understanding is 80.80% The analysis also verifies the effectiveness of CulturePark in expanding topics and cross-cultuee cross-cultural understanding.
Cultural differences boost novel opinions. In cross-cultural communication. different opinions can inspire others to think deeper and more comprehensively, as suggested by CCT and SCT. A case is the sentences orange in Figure 2(a). Lily partially agrees with = Abdul and gives an accurate and high-level summary of her pursuit: "a balance between family expectations and personal happiness" which is generated after a multi-turn energetic discussion with = Abdul. This also aligns well with CCT and SCT that emphasize the significance of communication among people having different cultural backgrounds.
CulturePark naturally assists cultural data augmentation by creating novel questions and comprehensive answers. On the one hand, agents different cultures can generate new opinions towards certain topics, which intuitively diversifies the imput questions. On the other hand, the initial seed data only contain short answers such as "Strongly agree" with no further explanations. Our platform allows deeper and more comprehensive communication of agents, thus generating more de- tailed responses such as "Strongly agree. believe that pleasing parents and elders is sign of respect and love" and "Strongly agree. equate my parents' happiness and satisfaction to my own success". Additionally, agents can extend the topies that conflict with their own opinions and provide more informative evidence to support their viewpoints. This strategy helps to generate informative and diverse data continuously. Section 5.1 presents some detailed results on diversity gain, showing that the generated data has significantly larger diversity.
4 Experiments
4.1 Evaluation on content moderation tasks
Setup. Content moderation crucial to maintaining the integrity and safety of online platforms in different cultures. What is acceptable in one culture could be offensive or inappropriate in another. However, few methods focused on content moderation for different cultures. For this experiment, we evaluated the effectiveness of our cultural-specific models for 8 different cultures: Arabic, Bengali,

---

Chinese, German, Korean, Portuguese, Spanish, and Turkish culture. These cultures have their unique characters, involving a large number people in the world.
We evaluate on 7 content moderation tasks for 8 different cultures to detect the following content: hate speech, offensive language, spam speech, abusive speech, bias speech, threat speech, and stance of speech in zero-shot evaluation, whose metric is average Fl score. The details on the datasets can be found in Appendix C. In total, our set contains 48, 895 samples. We compare our models with seven baselines: SPT-3.5-turbb [OpenAI, 2023a], GPT-4 [OpenAI, 2023b], Gemini-pro [Google, 2023], SeaLLM [Nguyen et al., 2023b], TaiwanLLM [Lin and Chen, 2023], Synatra-7B-vO.2. dpo [maywell, 2024], -oreean- 1a.imn--0....... 10.8B-v1.0 [yanolja, 2024], CultureLLM [Li et al., 2024], and CultureBank [Shi et al., 2024]. CultureLLM
series of culture-specife LLMs using semantic data augmentation. SeaLLM focuses on the Southeast Asian (SEA) culture, which is adopted for Chinese and Korean cultures. TaiwanI focuses on traditional Chinese culture. Synatra-7B-v0.3-dpp and -eree----oe-n--- 1aae--0..BB-... 10..8--v1.0 1.0 targeted at Korean culture. CultreBank collects data from social media and we compare Arabic and Korean culture by fine-tuning GPT-3355-trbe on its dataset.
CPT-35
Gomini
Ous--gat355
Table I: Comparison with the latest cultural specific LLMs.
GPT-1
CaltareLLM
Oure-gat
Spam
nn
Offenuiie
Spom
Ang
ScLLLM
357
.207
0.0.
Abresty
0.3 4.554445
TaisuLLLM
.54
.41
.692
.394
Aralic
Hate
Offinsive
Ang
CalluncBank
S40
662
.911
Oun
.5H
.735
.602
Tiik
Korzan
Abusive
Hate
Aeg
Scal I.
.23
.444
400
Stamne
Synatra- BeyD 3-lpo
.00
4655
478
Thrsst
EEVE-Keeean 10.88-1.0
CalluneBuok.
.635
.364
.437
-522
.579
.500
Figure 3: Results on content moderation.
Ous
.647
.640
.643
Main Results. We analyzed the results from the cul- ture and task sides in Figure 3. The most interesting observation is that our models outperformed GPT-4 on cultures and approached GPT-4 on the remain- ing 3 cultures, although the data for fine-tuning are generated by GPT-3.5-turbe which is much worse than GPT-4. We also generated cultural data via GPT- 4 and fine-tuned other cultural- -ssucficc models for comparison, denoted as "Ours-gpul in Figure 3. The performance of those models is better than "Ours" (GPT-3.5-turfe) version) but not so much. For other baselines, models outperform them in most cases. Table shows that models achieved better performance than those costly LLMs which require pre-training and fine-tuning.
Table 2: Results on ablation study of data generation and refinement.
Model
Bn
Zh
GPT-3.5-turbo
.542
Generate
.451
.622
.636
.594
Generate+ Verify
.486
.635
.678
.604
Generate+ Diversfy
.544
.644
.692
.603
Ablation study. Table 2 presents our ablation study on 4 cultures, where "Generate" means just extracting opinions from the dialogue, "Verify" represents factually verifying the extracted opinions, and "Diversify" means removing redundant data. The results show that each module of CulturePark is effective, ensuring its interpretability.
4.2 Evaluation on cultural alignment via Hofstede's Cultural Dimensions Theory
Setup. Hofstede's 's cultural dimension theory is a framework for understanding cultural differences across countries based on data collected from various countries. We asked LLMs to answer the 24 questions in VSM 13 to evaluate cultural alignment. Specifically, we used system prompt "You are culture chatbot that knows culture very well" to induce LLMs' cultural understanding ability. We selected proper C and anchor LLMs' answer to Hofstede's 's old dataset. We compute the gaps between LLMs' answer and Hofstede's data from six cultural dimensions using the Euclidean distance. Details on the survey and the distance are in Appendix D.
$C' is constants that can be used to adjust scores to fit a range between 0 and 100 or anchor new data to Hofstede's 's old dataset [Geert Hofstede, 2010].
7

---

700
Results. We compared our models (powered by GPT-3.5-Turbo) with GPT-3.5-turbe and GPT-4. As shown in Figure 4, our models outperform them by large margin, indicating their excellent cultural align- ment and cultural understanding abilities. Note that 200
GPT-35 GPT4
Churs
100
VSM is widely adopted as datasets for value and cul- ture alignment, the results imply that our approach for data collection is effective, thus it could be further used for value alignment research.
100
Culture
Figure 4: Results on culture alignment via
Hofstede's Cultural Dimensions Theory.
4.3 Evaluation in situated learning for cultural education
Situated learning suggests that learning is best understood and facilitated when occurs within the context [Anderson et al., 1996, Lave and Wenger, 1991]. Motivated by situated learning, we leveraged CulturePark for cultural education where our fine-tuned models serve as foreigners talk to people about cultural problems, which can create situation for cross-cultural communication and learning cultural-specific knowledge. For example, person who wants to learn about Arabic culture communicate with our Arabic model.
Setup and study process. We hired 24 participants, each of whom was given an outline of cultural learning and asked to talk to models based the outline. They can ask any related questions and express their opinions to models. Afterwards, the participants took a cultural understanding exam of VSM 2013 [Hofstede, 2013, Geert Hofstede, 2010] which they had never come into contact. We then computed the Euclidean distance between the ground truth and their answers from six cultural dimensions (rf. Section 4.2). For comparison, 12 participants learned with models and the other 12 learned with GPT-4 to study 6 cultures: Arabic, Bengali, German, Korean, Portuguese, and Spanish culture. Each culture was learned by four participants, two of them learning with our models and the others learning with GPT-4. Detailed information on the participants can be found in Appendix D.2. During the study process, first, each participant was given an outline for cultural learming written by cultural experts. The outline (Appendix D.2), serves as the guideline for efficient learning. Then, we asked the participants to freely communicate with the models to learn about specific cultures. After the examination, we asked the participants to give a score of 1-5 to indicate their satisfaction with the learning process. In this study, our aim was answer two questions: 1) What is the learning performance of the participants with our models and GPT-4? 2) How are their learning experience?
Results. Table 3 shows the averaged results of dif- ferent participants. have the following findings. First, participants learning with our models achieved better performance in cultural examination than those with GPT-4 in all cultures. This indicates that fine- tuned models have a better cultural understanding than GPT-4. Second, participants with communicating with our models than GPT-4. Furthermore, many participants expressed that the sponses from GPT-4 are vague. Even though we have prompted GPT-4 be like a person from a specific culture, it always responds with neutral words that have no clear opinions or ideas. Instead, our models can provide straightforward opinions.
Table 3: Results on situated learning.
Distancel. GPT-4
User experiencef
Model
Ours
GPT-4
Ours
Arahic
89.89
69.57
4
5
Bengali
339.84
304.54
3
5
Germany
224.68
173.12
2
3
more satisfied
Korcan
222.39
183.62
2
4
Spanish
143.33
102.53
4
5
Turkish
273.43
221.12
3
4
AVG
215.59
175.75
3
4.33
Discussion
5.1 Why CulturePark benefits fine-tuning?
We analyze the effectiveness of CulturPark in benefiting cultural model fine-tuning from two different aspects: Communication vs. direct generation of LLMs and diversity of the generated data.

---

Irvettnii Cerhrni-iffiirr grider
GPT-35
Cuosezlan and Saasc grals
Crowest un--difoo
GPT-sdimecly
Our
0.60
01:0
Peforurayy
0.00
0.0
Chituaal
Nundee
Maddls
(a) Benefits of
(b) Diversity gain vs. Cul- (c) Influence of cultural
(d) Performance BBH.
communication.
tureLLM [Li al., 2024]. background and gender.
Figure 5: More discussions on CulturePark.
Cross-cultural communication vs. direct generation from the results of fine-tuning using data directly generated by GPT models (i.e., no communication). We generated such data by prompting GPT-4 or GPT-3.5-turbo as: "Question: {input} Answer: foutput) Please list are used to fine-tune GPT-3.5-turbo. Figure 5(a) shows the performance on content moderation tasks in Chinese, Korean and Turkish cultures. We see that data directly generated from GPT-4 is better than that from GPT-3.5, while our GPT-3.5-based models can outperform both of them.
We compared
to support the answer and number them". Then, these data
Diversity of the generated data. We also analyzed the diversity gain [Bilmes, 2022] of the generated data for quality evaluation. We compared with CultureLLM [Li et al., 2024] and presented the results in Figure 5(b). It indicates that CulturePark can generate more diverse and high-quality data.
5.2 Exploring agents' cultural background and gender
To explore the influence of agent's cultural background and gender, we conducted three types of multi-agent communications in Arabic culture: "In-culural+Differeen gender", "Cross-culie Saaos- gender", and models, whose training data is 500, 750, and 1000, respectively. We evaluated the performance of the models on content moderation tasks and presented the results in Figure 5(c). Croos-culturaal+-ffffeeee gender" exhibits the best performance and ability to generate more high-quality data. This indicates the necessity of bringing more diversity data generation, as conducted in CulturePark
gender". For each setting, we fine-tuned three different
5.3 Fine-tuning vs. forgetting
A potential dilemma arises when fine-tuning large language model for specific tasks, it may result in catastrophic forgetting of its original capabilities. This section explores the extent of forgetting exhibited by CulturePark on BIG-Bench-HHe Hard (BBH) [Suzgun al., 2022], which comprises 21 tasks that assess semantic understanding and logical reasoning. For cost efficiency, sampled 100 samples from each BBH task. We evaluated our models against the baseline model, GPT-3.5 PPTT-..5-urrb.. The results in Figure 5(d) indicate that CulturePark generally maintains even improves performance most benchmarks. including the 21 tasks in BBH. This improvement suggests potential latent relationships between cultural data and general benchmarks, implying that fine-tuning of cultural data could improve general reasoning abilities.
Squr
De
Dn
Ouns
"Olenslve
0445
Abuafon
0.a0.. 0.3
0.8
0.35
Stiace
Thisst
Figure Results of CulturePark Llama on content moderation for different cultures.
5.4 Open-source fine-tuning with Llama2-70b
To verify the generalization ability of our framework, we leveraged the generated data fine-tune cultural-specific 1ama-22-70 models and evaluate on content moderation tasks. As shown in Figure 6,
Foor a fair comparison, the number of seed data for this part is 50 from WVS [Survey, 2022b] following CultureLLM [Li et al., 2024].

---

our models outperform Llama-2- 2-b in all cultures, especially in German, Chinese, Bengali and Portuguese cultures, which cover both low- and high-resouree cultures. Furthermore, our models are also excellent in all 7 tasks." This verifies the generalization of CulturePark as an effective data collection platform.
Conclusions, Societal Impact, and Limitations
This paper introduced CulturePark, an L..M-powered multi-agent framework for cultural data col- lection through multi-agent communication. CulturePark can generate high-quality and diverse cross-cultural dialogue, which can be used to fine-tune culturally specific LLMs. We evaluated CulturePark across three downstream tasks: content moderation, cultural alignment, and cultural education, indicating great improvement over GPT-4.
CulturParrk enhances fairness and inclusivity, reduces discrimination, and ensures accurate cultural representation. It improves global communication, fosters cross-cultural understanding, and supports multilingual societies. benefits bias-free LLMs build trust and align with responsible principles. Economically, it expands market reach and drives innovation. Social harmony improves by reducing stereotypes and preserving cultural heritage. also aids compliance with mntidiisrriinnation laws and supports inclusive education, promoting cultural awareness. Addressing cultural biases in L.LMs creates more just, reliable, and beneficial Al systems, contributing
more equitable world.
Our work has the following limitations. I) More experiments can be performed by replacing GPT-3.5- Turbo in CulturePark discover more results. 2) Our fine-tuned models mostly for high-resource cultures. The reason is that the dataset and benchmark on low-resource cultures are rare, and we can find enough data for fine-tuning and evaluation. 3) More efficient fine-tuning techniques can be studied to support the fine-tuning of culturally specific LLMs.
Disclaimer
The human study was conducted following local laws and regulations, and the evaluation process was controlled to ensure that no irresponsible content was generated. The authors respect all cultures studied in the world. The results of the paper may change due the change in OpenAI API and its model versions.
References
Mohammad Amin Abbasi, Arash Ghafouri, Mahdi Firouzmandi, Hassan Naderi, and Behrouz Minaei Bidgoli. Persianllama: Towards building first persian large language model. arXiv preprint arXiv:2312.15713. 2023.
aimansnigdha. Bangla-abusveecoomee---aaaa http://ghthbbbooommmmmmagggggggh Comment-Datased 2018.
Miguel Á Alvarez-Carmona, Estefanıa Güzmán-Falcóns Manuel Montes-y Gómez, Hugo Jair Es- calante, Luis Maaisenor-Pinedda Verónica Reyes-Meza, and Antonio Rico-Sulayes. Overview of mex-a3t at ibereval 2018: Authorship and aggressiveness analysis in mexican spanish tweets. In Notebook papers of 3rd sepin workshop on evaluation of human language technologies for iberian languages (ibereval), seville, spain, volume 6, 2018.
John R Anderson, Lynne M Reder, and Herbert A Simon. Situated learning and education. Educa- tional researcher, 25(4):5-11, 1996.
Arjun Appadurai. Modernity at large: Cultural dimensions of globalization. U of Minnesota 1996.
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. Semeval-2019 task 5: Multi- lingual detection of hate speech against immigrants and women in twitter. In Proceedings of the 13th international workshop on semantic evaluation, pages 54 63, 2019.
"The detailed hyperparameter setting is shown in Table 6. We only conduct this pilot study to show that Llama2 also works in CulturePark. The results might be further improved with more GPU resources.
10

---

Shiladitya Bhattacharya, Siddharth Singh, Ritesh Kumar, Akanksha Bansal, Akash Bhagat, Yogesh Dawer, Bornini Lahiri, and Atul Kr. Ojha. Developing a multilingual annotated corpus of misogyny and aggression. P Proceedings of the Second Workshop on Trolling, Aggression and Cyberhullying pages 158-168, Marseille, France, May 2020. European Language Resources Association (ELRA). URL httpps:/ http://aaaaatoogo.orrrre. wrwplloooyyoogg//gee org/2020...... http//20..trac-1... trac-1.25, 25/.
Jeff Bilmes. Submodularity in machine learning and artificial intelligence. arXiv preprint arXiv:2202.00132, 2022.
Cao, Zhou, Lee, Cabello, Chen, and Hershcovich. Assessing cross-cultural alignment between chatgpt and human societies: An empirical study, arxiv. Preprint posted online March, 31, 2023.
Tommaso Caselli, Valerio Basile, Jelena Mitrović, Inga Kartoziya, and Michael Granitzer. I feel offended, don't be abusive! implicit/explici messages in offensive and abusive language. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6193-6202, 2020.
Çağn Çöltekin. A corpus of turkish offensive language social media. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6174-6184, Marseille, France, 2020. URL https://www aclweb. org/ anthology/2022 2ttp:///02220.111111..
Alex J Chan, José Luis Redondo García, Fabrizio Silvestri, Colm O'Donnel, and Konstantina Palla. Harmonizing global voices: Culturally-aware models for enhanced content moderation. arXiv preprint arXiv:2312.0240) 2023.
Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Abdelali, Soon-gyo Jung, Bernard J Jansen, and Joni Salminen. A multi-platform arabic news comment dataset for offensive language detection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6203-6212, 2020.
Richard A Cosier and Gerald L Rose. Cognitive conflict and goal conftict effects on task performance. Organizational behavior and human performance, 19(2):378-1) 3311 1977.
daan Korean hatespeech dataset.
http://ttth.ccoo/aaaa/e///aaaaa/ dataset, 2020.
Angel Felipe Magnossao de Paula and Ipek Baris Schlicht. Ai-upy at iberlef-2021 detoxis task: Toxicity detection in immigration-related web news comments using transformers and statistical models. arXiv preprint arXiv:2111.0m 1..5300, 2021.
Rogers de Pelle and Viviane P. Moreira. Offensive comments in the brazilian web: dataset and baseline results. In 6th Brazilian Workshop on Social Network Analysis and Mining (BraSNAM), 2017.
Werner Delanoy. What is culture. The Cambridge handbook of intercultural communication, pages 17-34. 2020.
Wenqi Fan, Zihuai Zhao, Jiatong Li, Yunqing Liu, Xiaowei Mei. Yiqi Wang, Jiliang Tang, and Qing Li. Recommender systems the era of large language models (Imms). arXiv preprint arXi::2307.020445 2023.
Elisabetta Fersini, Paolo Rosso, Maria Anzovino, et Overview of the task on automatic misogyny identification at ibereval 2018. Ibereval@ sepin, 2150:214-200 22111 2018.
Susan T Fiske and Shelley Taylor. Social cognition. Mcgraw-Hill Book Company, 1991.
Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. Massively multi-cultural knowledge acquisition & Im benchmarking. arXiv preprint arXiv:2402.09366 2024.
Yi R Fung, Tuhin Chakraborty, Hao Guo, Owen Rambow, Smaranda Muresan, and Heng Ji. Norm- sage: Multi- lingull multi-cultural norm discovery from conversations on-the on-the-fly. arXiv preprint arXiv:2210. 08600, 2022.

---

Michael Minkov Geert Hofstede, Gert Jan Hofstede. Cultures and Organi-
zations:
Software of the Mind, Third Edition. McGraw
Hill Professional.
2010.
Google. Gemini. BooogleeGemiiiiinhs///eeeeeeee
20233 2023.
HASOC. Hasoc2020. http://hmaccciiiiiiiiii//ooo//ot 2020.
Hofstede. Vsml3. http:///gettooiicccccmmm///s/sss
2013.
F Husain. Osact4 shared task offensive language detection: Intensive preprocessing approach, arxiv 2020. arXiv preprint arXiv:2005.0722 2020.
Zhuoren Jiang, Zhe Gao, Guoxiu He. Yangyang Kang, Changlong Sun, Qiong Zhang, Luo Si, and Xiaozhong Liu. Detect camouflaged spam content via stoneskipping: Graph and text joint embedding for chinese character variation representation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP2019). ACM, 2019.
Rebecca I. Johnson, Giada Pistilli, Natalia Panai, Julija Kalpokiene, and Donald Jay Bertulfo. The ghost in the machine has an american accent: value conflict in gpt-3. arXiv preprint arXiv:2203.0777 2022a.
Leslye Denisse Dias Duran, Enrico
Rebecca L Johnson, Giada Pistilli, Natalia Menédez-Gonzálezz Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, and Donald Jay Bertulfo. The ghost in the machine has an american accent: value conflict in gpt-3. arXiv preprint vrXiv:2203.07788 2022b.
Sanaa Kaddoura and Safaa Henno. Dataset of arabic spam and ham tweets. Data in Brief, 52(10990): 4, 2024.
Kaggle. 5k turkish tweets with incivil content. mukkss--wwett-wwith-iiiie with-in 2021.
http::/www....gggcccommm/esesess
Kaggle. turkish offensive language detection. https//wwww.aaaggg...cdddddsasst
offenssieelanguage-deeeeee 2022.
Habibe Karayigit, Çigdem inan Acı, and Ali Akdağl. Detecting abusive instagram comments in turkish using convolutional neural network and machine learning methods. Expert Systems with Applications, 174: 14802, 2021.
Grgur Kovač, Masataka Sawayama, Rémy Portelas, Cédric Colas, Peter Ford Dominey, and Pierre- Yves Oudeyer. Large language models as superpositions of cultural perspectives. arXiv preprint arXiv:2307.07870. 2023.
Jean Lave and Etienne Wenger. Situated learning: Legitimate peripheral participation. Cambridge university press, 1991.
Jean Lee, Taejun Lim, Heejun Lee, Bogeun Jo, Yangsok Kim, Heegeun Yoon, and Soyeon Caren Han. K-MHaS: A multi-label hate speech detection dataset in Korean online news comment. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3530- 3538. Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. URI. https: / //aalalanttt aclanthology org/ wwt...cllinn......oooo...... 1111.
Joao A Leite, Diego F Silva, Kalina Bontcheva, and Carolina Scarton. Toxic language detection in social media for brazilian portuguese: New dataset and multilingual analysis. arXiv preprint arXiv:2010.04543 2020.
Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. Culturellm: In- corporating cultural differences into large language models. arXiv preprint 40094.240..10946 2024.
Lei Li, Yongfeng Zhang, and Li Chen. Prompt distillation for efficient llm-based recommenda- tion. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 1348-1357 2023.
12

---

Margarita Limón. On the cognitive conflict as an instructional strategy for conceptual change: A critical appraisal. Learning and instruction, 11(4-5)355-338) 2001.
Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André FT Martins, and Hinrich Schlitze. Mala-500:
Massive language adaptation of large language models. arXiv preprint arXiv:2401.13300 2024.
Ven-Ting Lin and Yun-Nung Chen. Taiwan Ilm: Bridging the linguistic divide with a culturally aligned language model. arXiv preprint arX14:231..7447,, 2023,
Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. Are multilingual Ilms culturally- diverse reasoners? an investigation into multicultural proverbs and sayings. arXiv preprint arXiv:2309.08591, 2023.
Reem I Masoud, Ziquan Liu, Martin Ferianc, Philip Treleaven, and Miguel Rodrigues. Cultural align- ment in large language models: An explanatory analysis based on hofstede's 's cultural dimensions. arXiv preprinl urXiv:2309./23:: 2023.
maywell. Synatra. http:///wuggnngffeecccoowwwwewls
2024.
Jihyung Moon, Won IK Cho, and Junbum Lee. BEEP! Korean corpus of online news comments for toxic speech detection. In Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media, pages 25-31, Online, July 2020. Association for Computational Linguistics. URL https:/ http::://wwwww/ www.ww.www...................... www.rr/gggrggg///orooroooooot/et /200:y//2220mm2 2ttp://0000000ccllll1.......
Hamdy Mubarak, Hend AI-Khalifa, and AbdulMohsen Al-Thubaity. Overview of osact5 shared task arabic offensive language and hate speech detection. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur an QA and Fine-Grained Hate Speech Detection, pages 162-166, 2022.
Cas Mudde. The 2012 stein rokkan lecture: Three decades of popu list radical right parties in western europe: what? In The Populist Radical Right, pages 545 558. Routledge, 2016.
Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, et al. Blend: A benchmark for IIms on everyday knowledge diverse cultures and languages. arXiv preprint arXiv:2406.0994 09944, 2024.
Tarek Naous, Michael J Ryan, and Wei Xu. Having beer after prayer? measuring cultural bias in large language models. arXiv preprint arXiv:2305.14455 2023.
Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. Extracting cultural commonsense knowledge at scale. In Proceedings of the ACM Web Conference 2023, pages 1907-1917, 2023a.
Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, et al. Seallms-large language models for southeast asia. arXiv preprint arXiv:2312.00738 2023b.
OpenAl. Chatgpt. https://chat.opemiicoom 2023a.
OpenAl. Gpt-4 technical report, 2023b.
OpenAl. coxt-embeddding---maee
2024.
Juan Carlos Pereira-Kohatsu. Lara Quijano-Sánchez, Federico Liberatore, and Miguel Camacho- Collados. Detecting and monitoring hate speech in twitter. Sensors, 19(21):4654, 2019.
Kunat Pipatanakul, Phatrasek Jirabovonvisut, Potsawee Manakul, Sittipong Sripaisarmmongkol. Ruangsak Patomwong, Pathomporn Chokchainant, and Kasima Thampiiitthha. Typhoon: Thai large language models. arXiv preprint arXiv:2312.13995. 2023.
Ramon Pires, Hugo Abonizio, Thales Sales Almeida, and Rodrigo Nogueira. Sabiá: Portuguese large language models. In Brazilian Conference Intelligent Systems, pages 226-240 Springer, 2023,
13

---

Flor Miriam Plaza-del Arco, Arturo Montejo- Ránzz, L Alfonso Urena Lopez, and María- Martín- Valdivia. Offendes: A new corpus in spanish for offensive language research. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1096-1108, 2021.
Abhinav Rao, Aditi Khandelwal, Kumar Tanmay, Utkarsh Agarwal, and Monojit Choudhury. Ethical reasoning over moral alignment: A case and framework for in-context ethical policies in llms. arXiv preprint arXiv:2310.0725, 2023.
Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. Normad: A benchmark for measuring the cultural adaptability of large language models, arXiv preprint arXiv:2404. 124644 2024.
Nauros Romim, Mosahed Ahmed, Hriteshwar Talukder, and Md Saiful Islam. Hate speech detection in the bengali language: A dataset and its baseline evaluation. In Proceedings of International Joint Conference on Advances Computational Intelligence: IJCACI 2020, pages 457-468. Springer, 2021.
Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis. In Michael Beißwenger, Michael Wojatzki, and Torsten Zesch, editors, Proceed- ings of NLP4CMC III: 3rd Workshop on Natural Language Processing for Computer- Menputer-Mediated Communication, volume 17 of Bochumer Linguistische Arbeitsberichle, pages 6-9, Bochum, sep 2016.
Mediated
Paul Röttger, Haitham Seelawi, Debora Nozza, Zeerak Talat, and Bertie Vidgen. Multilingual hatecheck: Functional tests for multilingual hate speech detection models. arXiv preprint (xXiv:2206.09917. 2022.
Michael J Ryan, William Held, and Diyi Yang. Unintended impacts of Ilm alignment on global representation. arXiv preprint arXiv:2402.15018 2024.
Omar Shaikh, Valentino Chai, Michele Gelfand, Diyi Yang, and Michael S Bernstein. Rehearsal: Simulating conflict teach conflict resolution. arXiv preprint vrXiv:2309.12309 2023.
Omar Sharif and Mohammed Moshiul Hoque. Tackling cyber-aggression: Identification and fine- grained categorization of aggressive texts on social media using weighted ensemble of transformers. Neurocomputing. 490:462-481, 2022.
Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Raya Horesh, Rogério Abreu de Paula, Diyi Yang, et al. Culturebank: An online Nommunity-driven knowledge base towards culturally aware language technologies. arXiv preprint arXiv:2404. 1503. 2024.
Hoyun Song, Soo Hyun Ryu, Huije Lee, and Jong C Park. A large-scale comprehensive abusiveness detection dataset with multifaceted labels from reddit. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 552-561, 2021.
Helen Spencer-Oatey and Peter Franklin. What is culture. A compilation of quotations. GlobalPAD Core Concepts, 1(22):1-21, 2012.
Pew Global Attitudes Survey. Pew global attitudes survey. http:://www.ppprrssecchorrgg 2022a.
World Values Survey, World values survey. https:/wwww.aaadvvvlllssrrrxyyvw 2022b.
2022b.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether tf can solve them. arXiv preprint arXiv:2210.09261, 2022.
TurkishSpamVO1, Turkish Spam VOL. UCI Machine Learning Repository, 2019. DOI:
http:///0ii.rg//0044444225CC
14

---

Francielle Vargas, Isabelle Carvalho, Fabiana Rodrigues de Góes. Thiago Pardo, and Fabrício Benevenuto. HateBR: A large expert annotated corpus of Brazilian Instagram comments for offensive language and hate speech detection. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 7174-7183, Marseille, France, June 2022. European Language Resources Association. URL ptatttps:/aclaatttie
www.gg......//gooo.....o........ org/2022. 1rec-1.777
Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu, and Michael Lyu. Not all countries celebrate thanksgiving: On the cultural dominance large language models. arXiv preprint mrXfv:2310.1248/1 2023.
Zeerak Waseem and Dirk Hovy. Hateful symbols hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pages 88-93, 2016.
Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. Overview of the germeval 2018 shared task on the identification of offensive language. Verlag der Österreichischen Akademie der Wissenschuflen, 2018.
Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, Ji Zhang, Chao Peng, Fei Huang, and Jingren Zhou. Cvalues: Measuring the values of chinese large language models from safety to responsibility. arXiv 2307.09705, 2023.
Diyi Yang, Caleb Ziems, William Held, Omar Shaikh, Michael S Bernstein, and John Mitchell, Social skill training with large language models. arXiv preprint arXiv: 042044444444 2024.
yanolja. Eeve.
2024.
Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derezynski, Zeses Pitenis, and Çağr Çöltekin. Semeval-2020 task 12: Mul- tilingual offensive language identification in social media (offenseval 2020). arXiv preprint 0:Xi::2006.07235. 2020.
Wenlong Zhao, Debanjan Mondal, Niket Tandon, Danica Dillion, Kurt Gray, and Yuling Gu. World- valuesbench: A large-scale benchmark dataset for multi-cultural value awareness of language models. arXiv preprint
2024.
Jingyan Zhou, Jiawen Deng, Fei Mi, Yitong Li, Yasheng Wang, Minlie Huang, Xin Jiang, Qun Liu, and Helen Meng. Towards identifying social bias in dialog systems: Frame, datasets, and benchmarks. arXiv preprint arXiv:2202.080/ 2022.
15

---

Contents
Discussion on the Relationship between Culture and Language
16
Details on the cross-cultural dialogue
17
B.1 Examples on seed data
17
B.2 The dialogue dataset
17
B.3 Details on data refinement
17
Details on the test sets
19
C.1
Arabic
19
C.2 Bengali
20
C.3 Chinese
20
C,4 Germany
20
C,5 Korean
C.6 Portuguese
21
C.7 Spanish
22
C.8 Turkish
22
Details in experiments
23
D.I Cultural alignment via Hofstede's cultural dimentions theory
23
D.2 Details situated learning
24
E Prompts setting
25
Case study
25
A Discussion on the Relationship between Culture and Language
We strongly agree that language not equal to, but only part of culture. But using language to study culture is possible due to the following aspects:
1. Existing literature culture understanding shows that culture boundaries fluid, dynamic and uncertain. Delanoy emphasizes that cultures are not homogeneous or static entities but are fluid and dynamic. He critiques essentialist views that rigidly define cultural boundaries and instead promotes a more nuanced understanding that considers the intersections of various cultural factors, such ethnicity, language, religion, and socio-economic condi- tions [Delanoy, 2020]. Appadurai also discusses the fluidity of cultural boundaries and the creation new cultural forms [Appadurai. 1996]. Cultural boundaries can be geographical regions, language, religion and so on. Based on above statements, using language as cultural boundaries is reasonable.
2. Existing NLP works on culture also leverage labguage as culture boundaries. Naous et al. [2023] focuses on Arabic and English culture. Wang et al. [2023] focuses on different cultures: English, Chinese, French, Russian, German, Arabic, Japanese and Korean. Liu et al. [2023] also use language to split different cultures. The authors work on English, German, Russian, Bengali, Chinese, and Indonesian culture. Myung et al. [2024] is hand-crafted benchmark for evaluate diverse cultures. They also use languages as culture boundaries.
16

---

3. Most downstream benchmarks are classified via language and we cannot get more fine- grained perspectives. For example, if we want evaluate the performance Arabic model, we can find benchmarks in Arabic culture. But if we use regions as cultural boundaries, we can't find benchmarks Morocco and Jordan cultures.
Details on the cross-cultural dialogue
B.1 Examples on seed data
Source: WVS
Question: Do you strongly agree, agree, disagree or strongly disagree with the following statement?
"One of my main goals in life has been to make my parents proud."
(a) Strongly agree
(b) Agree
(c) Disagree
(d) Strongly disagree
Source: PEW
Question: Do you strongly agree, agree, disagree, or strongly disagree with the following statement:
"On the whole, men make better business executives than women do."
(a) Agree strongly
(b) Agree
(c) Disagree
(d) Strongly disagree
Don't know
Figure 7: Example questions from the WVS and Pew explore perspectives globally relevant political and ethical issues. Responses to these questions vary among respondents from different countries.
Figure 7 shows example questions from the WVS and Pew to explore perspectives on globally relevant political and ethical issues. Responses these questions vary among respondents from different countries.
B.2 The dialogue dataset
Figure shows the topic distribution of the generated cross-cultural dialogue dataset: human belief (99.68%) norm (2,,54%) and custom (10.78%). For human belief, there are three main types: religious beliefs (31.31%), %)), social beliefs (54.77%), and ethical beliefs (13.92%). For norm, the data can be divided into descriptive norms (26.98%), prescriptive norms (7 (7.48%), and traditional norms (65.53%). For custom, they can be classified into social customs (39.75%), family customs (27.95%), and community customs (32.30%).
Then, randomly sampled 750 dialogues for each culture and evaluated the communication using prompts in Appendix E. As shown in Table 4, on average, the average ratio of statements that express cross-cultural understanding is 80.80% The analysis also verifies the effectiveness of CulturePark in extending topics and cross-cue cultural understanding.
B.3 Details on data refinement
Algorithm Figure shows the algorithm pipeline of data refinement.
17

---

Social
community
Custom/
Relgioos
Norm
Iracitinnl
Human Belief
Precoiptive
Social
Desclpiiv
Etbcial
Figure 8: After multi-turn communications, get cross-cultural dialogues dataset (CCD), which involves data on different cultures. CCD contains human belief (59.68%), norm (29.54%), and custom (10.78%)
Algorithm 1: Cultural data refinement
Input: dialogue record record, targeted culture culture, seed data
Output: opinions targeted culture Final_opinnion oppinions
Embeddings +-
opinions +
for Reconds do is from culture then extract opinion from r via GPT-4; judge the relationship of and opinion from Entail, Contradict and Irrelevant; relationship Entail then append opinion into opinions; embedding gelEmhedding(ppnnion append embedding into Embeddings;
else if relationship Contrudict then
rewrite the opinion and check the relationship again: relationship == Entail append opinion into opinions; embedding = ge(Embbedlin((oprrion)) append embedding into Embeddings;
Labels clustering (engeeriigg/mme//iiinnngg min(10, len( (mmbedddiigs)) Final_ popiiooisoons + []: for in Labels do final_ opinion randomly select from arr;
append final opinion Final opinions;
return Finol_ _oiiinoisnoon:
Figure 9: Pipeline of data refinement.
Table 4: Interesting observation in Cross-cultural Dialogues. "Extend Rate" represents the ability of entending the topic. "Understanding Others" represents the ratio of cross-cultural understanding statements and others.
Ar
Bn
Zh
De
Ko
Pt
Es
Tr
AVG
Extend rate
34.36
33.33
34.74
37.82
32.82
35.13
35.90
32.69
34.60
Understanding ratio
75.68
80.56
80.97
80.14
85.27
80.97
81.40
81.40
80.80
18

---

C Details on the test sets
The statistics of the datasets are shown in Table 5, and we provide detailed instructions for them in the following.
Table 5: A brief introduction of the 8 evaluation tasks and 41 datasets. We list both the name and the size of test sets. For instance, (05ACT5(2541) [Mubarak et al., 2022] denotes that there are 2541 test samples in the dataset OSACTS.
Culture
Country & Territory
Task & Datuset
#Sample
Offensive language detection: OSACTS/2541) (Mubarak et 2022).
Hare detection: Multi Platform(675) [Chowdhury et al., 2020),
Arabic (CulurePark-AAr)
Middle East
OSACT52241) [Mubarak 2022], and
8,973
OSACTS, DSSACT__inegrineeee linegrained//544] (Muharak et al., 2022).
Valgar detection: Multi- Platform/675)) [Chowdhury et. 2020]
Offousive longuage detection: TRAC2020 Task (1000) [Rhataccarya et al., 2020).
TRAC2220 Task2(1000) [Bhattacharya 2020).
(CulturePark--Be Bn)
Bangli
Rangladesh
detection: Speech (000) [Romim 202.1.
Threat detection: BACD( (000) [aimansnigdha, 2018].
detection: BACTX
[aimansnigdha, 2018]
Chinese
Spam detection: (10())00) [Jing et al., 2019]_
(CultureParkk-Z) Zh)
China
Biax detection: CDial Bias( 1000) [Zhou 2022).
2,000
Offensive language detection: GermEval2018/22 ((3531) [Wiegand al., 2018)
German
Germany and parts of Europe
Hate detection; IWG 1(469) [Ross 2016], (KG__(669) [Ross et. 2016).
(CutturePar--De)
HASOC20200850) [HASOC, 2020).
and hawlllinguul---ttchcckckkc
[Rötgeer et al,, 2022].
detection: hateSpeech( 1000) [Monn
2020].
and HateSpeech2 1000) [clamn Veer, 2020)
Korean (CulturePark-Kon
South Korea
Abusive detection: AbuseEvval (va0((000) [Caselli
2020].
CADD(1000) [Song et al., 2021].
and Waseemi 1000) Wassem and Hovy, 2016
Offensive language detection: OAComBR/1250) de Polle and Moreira 2017].
Portuguese (CulturePark-PP Pt)
Brazil and parts of Latin America
detection:
and Hate 1000) [Vargas al., 2023].
homohhoboaa
[Leite 20201.
5,250
and BLLLD----iiogyny (a (1000) [Leite et al., 2020).
Abusive detection: ToLD-Br-insall (100)) [Leite
2020).
Offensive longuage detection: AMI( (0000) [Fersini et 2018).
1000) [Álvarezz
2018).
Hate detection: HatEval 2019( 1000) [Basile et al., 2019).
Spanish (CulturePark-Ii) Di)
detection: DETOXIS stereetypol 1000) Paula Schlicht,
Argentina, Mexico, and parts of Latin
DETOXIS impropert (000) Ide Paula and Schlicht. 2021].
9,000
Abusive detection: DETOXISSebusivel sbussve(1000) (do Paula Schlicht. 2021).
DETOXIS mockery(1 (000) [de Paula
Schlicht, 2021].
Aggreszive detection: DETOXIS, aggressivenessl (000) [de Paula and Schlichs, 2021].
Stance detection: DETOXIS stancot 1000) Paula Schlicht, 2021].
Offensive language detection: Semexal-202005530 [Zampieri at,,, 2020),
and offenseKangle 2(100) (Kaggle, 2022]
Turkish (CulturePark-TT)
Turkey
Abusive detection: XTC(1000) (Karayigit al., 2021].
7,353
Span detection: Turkish 3pam(825) [TurkisSpammV01]
aie-ggrained offensive detection: offenseCorpus(( [Coltekin, 2020).
C.1 Arabic
OffenseEval2020 [Zampieri al., 2020] dataset was created to address the issue of offensive language in social media. It aims to use computational methods to identify offensive, aggressive, and hate speech in user-generated content, providing multilingual dataset in five languages (Arabic, Danish, English, Greek, Turkish). We utilized the Arabic portion of Sub-task A Offensive language identification from this dataset, consisting of total of 2000 data samples.
OSCAT4 [Husain, 2020] dataset aims to detect and categorize offensive language in Arabic tweets, with two sub-tasks: detecting if post is offensive or not, and identifying the offensive content type as hate speech or not hate speech. We use the first sub-task, consisting of 1000 data entrics, as the dataset for offensive detection, and the second sub-task, also comprising 1000 data entries, as the dataset for hate speech detection.
(Chowdhury et al., 2020] dataset is collection of 4000 comments in Dialectal Arabic from social media platforms, focusing on offensive language. intended for studying offensive language in news comments published by international news organizations. We utilized total of 1000 annotated data samples indicating whether they are offensive and 675 annotated data samples indicating whether they vulgar.
19

---

OSACTS [Mubarak et al., 2022] dataset consists of 12,698 Arabic tweets collected between June 2016 and November 2017. labeled for offensiveness and fine-grained hate speech types using emojis commonly found in offensive communications, providing a resource for offensive and hate speech detection and classification tasks. The dataset consists three subtasks: offensiveness detection, hate speech detection, and fine-grained hate speech detection. We utilized 2,541 data samples for each these tasks.
ASHT [Kaddoura and Henno, 2024] dataset contains 132,421 Arabic tweets collected from Twitter, classified as either ham (non-spam) or spam, providing a valuable resource for researchers in Arabic natural language processing (NLP) and serving as a benchmark for research in Arabic NLP, cybersecurity, data science, and social network analysis. We utilized a subset of 1,000 data samples for the spam detection section.
C.2 Bengali
TRAC2020 [Bhattacharya et al., 2020] dataset a multilingual annotated corpus social media comments, encompassing misogynistic and aggressive comments in Indian English, Hindi, and Indian Bangla. It consists of over 20,000 comments and is annotated at two levels aggression (overtly aggressive, covertly aggressive, and aon-aggressive) and misogyny (gendered and non- gendered). Baseline experiments were conducted to develop misogyny classifiers for the three languages. TRAC2020 consists of two tasks: Aggression Detection and Misogynistic Aggression Detection. We utilized 1,000 data samples for each of Task 1 and Task 2.
BAD [Sharif and Hoque, 2022] dataset is novel Bengali aggressive text dataset (called 'BAD') with two-level annotation, designed to identify and classify aggressive content Bengali language. It achieves high accuracy through a weighted ensemble technique and outperforms other machine learning and deep learning baselines, with a weighted fl-score of 93.43% for identification and 93.11% for categorization tasks. We utilized subset of one thousand data samples as the Offensive dataset.
Hate Speech [Romim et al., 2021] dataset consists of 30,000 social media user comments, covering seven categories including sports, entertainment, religion, politics, crime, celebrities, TikTok, and memes. It has been annotated through crowdsourcing and expert validation for research purposes in detecting hate speech in Bengali language. The dataset also provides benchmark experimental results for multiple deep learning models and pre-trained Bengali word vectors. We utilized 1,000 data samples from the dataset for Hate Detection.
BACD [aimansnigdha, 2018] dataset is dataset for the Bengali language, consisting of total of 10,200 data points with annotations for toxic, threat, obscene, insult, and racism labels. We utilized 1,000 data points from this dataset for Threat Detection and Bias Detection tasks respectively.
C.3
Chinese
CCS [Jiang et al., 2019] dataset consists two real-world spam datasets: one an SMS dataset, and the other is product review dataset. Both datasets manually labeled by professionals spam or regular emails, and their sizes and label distributions were summarized. We utilized 1000 data samples from this dataset for Spam Detection.
CDial-Bias [Zhou et al., 2022] Dataset the first annotated Chinese social bias dialog dataset, utilized to establish a benchmark for measuring dialog bias and evaluate Chinese generative models for social bias presence. We utilized 1000 data samples from for bias detection.
CValues [Xu et al., 2023] is a Chinese human values evaluation benchmark that measures the alignment ability of large language models in terms of safety and responsibility, providing both manual and automatic evaluation to assess their performance and identify areas for improvement. We utilized 1712 data samples from the dataset for Stance detection.
C.4 Germany
GermEva12018 [Wiegand et al., 2018] dataset is used for identifying offensive language in Ger- man tweets, including both coarse-grained binary classification tasks and fine-grained multi-class classification tasks. We used 3,531 data points for Offensive Detection.
20

---

IWG [Ross et al., 2016] dataset aims to assess the feasibility of reliably annotating hate speech and explore the consistency between existing definitions and subjective ratings. The results indicate low reliability in users' judgments of hate speech, suggesting need for more detailed annotation instructions. Each data instance in the dataset was annotated by two experts, and we selected 469 instances with annotations from both experts for Hate Detection, denoted as IWG_1 1 and IWG_2 respectively.
HASOC2020 [HASOC, 2020] dataset is a multilingual research forum and data challenge that offers tasks for identifying problematic content in English, German, and Hindi. It consists of over 10,000 annotated tweets from Twitter, and includes both coarse- grassnegranedd and fine- fine-grained classification tasks. We utilized subset of 850 German language data from the HASOC dataset for Hate Detection.
Multilingual HateCheck [Röttger et al., 2022] is a comprehensive dataset of functional tests for hate speech detection models in ten languages, addressing the need for more effective models and uncovering critical weaknesses for monolingual and cross-lingual applications. We utilized 1000 data points from the German section of the dataset for Hate Detection.
C.5 Korean
K-MHaS [Lee et al., 2022] is multi-label dataset consisting of 109k utterances from Korean news comments, designed for hate speech detection. It effectively handles Korean language patterns, provides multi-label classification with 1 to 4 labels, and considers subjectivity and intersectionality. Strong baseline experiments using Arrean-BERT-basee language models show that KR-BERT with sub-character tokenizer performs the best by recognizing decomposed characters in each hate speech class. We utilized 1000 data samples from the dataset for Hate Detection.
HateSpeech [Moon al., 2020] dataset is a collection of 9.4K manually labeled entertainment news comments in Korean, aimed at identifying toxic speech, social bias, and hate speech. provides benchmarks using CharCNN, BiLSTM, and BERT models, with BERT achieving the highest performance. The dataset made publicly available and open for competition. We utilized 1000 data samples from the dataset for Hate Detection.
HateSpeech2 [daan Veer, 2020] dataset was created by the Natural Language Processing Laboratory (NLP) at Korea National University and it includes the original dataset, vocabulary of offensive language, annotations, and dataset examples. The dataset is used for labeling malicious comments and has been built with word embeddings. We utilized 1000 data samples from the dataset for Hate Detection.
AbuseEval [Caselli et al., 2020] is newly created dataset that addresses issues in annotating offensive and abusive language, specifically considering the degree of explicitness, target presence, and contextual interaction across different abusive language phenomena. We utilized 1000 data samples from the dataset for Abusive Detection.
CADD [Song et al., 2021] is comprehensive dataset for detecting abusive language in English Reddit posts, featuring multifaceted labels and contextual information, collected through large-scale crowdsourcing and yielding meaningful performance with state-of-the-art language models. We utilized 1000 data samples from the dataset for Abusive Detection.
Wascem [Waseem and Hovy, 2016] dataset, based on critical race theory, provides annotations for over 16k tweets and aims to detect hate speech social media by analyzing linguistic features, extra-linguistic features, and a dictionary of the most indicative words the data. We utilized 1000 data samples from the dataset for Abusive Detection.
C.6 Portuguese
OffComBR [de Pelle and Moreira, 2017] dataset is annotated collection of offensive comments in Portuguese, gathered from news comment sections on the Brazilian web. It serves the purpose of classifying user-gen seeeraenddt text either positive negative, providing a baseline for future research on the topic of hate speech detection in Portuguese. We utilized 1250 data samples from this dataset for offensive detection.
HateBR [Vargas et al., 2022] dataset the first large-scale expert annotated corpus of Brazilian Instagram comments, specifically collected from politicians' accounts, providing binaryloffinssienesss-
21

---

level classification and nine hate speech groups, outperforming the current state-of tette-of-ther of-the-art for Por- tuguese language offensive language and hate speech detection. We utilized 1000 data samples from this dataset for offensive detection.
ToLD-Br [Leite al., 2020] is large-scale dataset for Brazilian Portuguese, consisting of annotated tweets categorized as toxic or non-toxic, aiming to detect and prevent the proliferation of toxicity in social media, addressing the need for multilingual approaches and models aware of different categories of toxicity. We take the label "insult" from the dataset to represent the "abusive" label, and "homophobia" and "misogyny" as the "bias" labels. We have selected 1000 data samples for Abusive Detection, 1000 samples for Bias Detection, and 1000 samples for Bias Detection.
C.7 Spanish
AMI [Fersini et al., 2018] dataset a collection of Spanish and English tweets used for identifying misogyny, categorizing misogynistic behavior, and classifying targeted individuals, with contributions from multiple teams and countries. We used 1000 Spanish language data for offensive detection.
MEX-A3T [Álvarez-Carme] Carmona et al., 2018] dataset, from the track at IberEval 2018, comprises Mexican Spanish tweets and focuses on two tasks: author profiling, which aims to identify the residence and occupation of Twitter users, and aggressiveness detection, distinguish between aggressive and non-aggressive tweets. This dataset was created specifically for these tasks and was analyzed and compared in a paper discussing the participants results. We used 1000 data samples for offensive detection.
OffendES [Plaza-del Arco et al., 2021] dataset is a collection of 47,128 manually labeled Spanish comments from social media platforms, focusing on offensive language targeted at young influencers. It provides pre-defined offensive categories and includes confidence scores, enabling both multi-class classification and multi-output regression studies. We used 1000 data samples for offensive detection.
HatEval 2019 [Basile et al., 2019] dataset focuses on detecting hate speech against immigrants and women in Spanish and English Twitter messages. It includes two classification tasks: identifying the presence of hate speech and distinguishing between individual and group targets. HatEval was popular SemEval-2019 task with numerous submissions and participant system analysis. We used 1000 data samples for hate detection.
HaterNet [Pereira-Kohatsu et al., 2019] dataset is an intelligent system used for monitoring and visualizing hate speech on Twitter. It provides a novel public dataset of Spanish hate speech, consisting of 6,000 expert-annotated tweets. We used 1000 data samples for hate detection.
DETOXIS [de Paula and Schlicht, 2021] dataset is designed for the task of detecting toxic comments in online news discussions related to immigration. It includes toxicity detection and toxicity level detection. Participating teams achieved good results using the BERT model on this dataset. We clas- sified them into tags such as stereotype, improper, abusive, mockery, aggressiveness, and stance, and selected 1000 data samples for each category for Bias detection, Abusive detection, Aggressiveness detection, and Stance detection.
C.8 Turkish
SemEval-2020 IZampieri et al., 2020] provided a new, large-scale semi-supervised training dataset of over nine million English tweets and expanded the task to include four new languages, allowing for cross-lingual training and analysis. We used 3528 data samples in Turkish for Offensive Detection.
OffenseCorpus [Cöltekin, 2020] is a corpus of Turkish offensive language, comprising randomly sampled micro-blog posts from Twitter. It contains 36,232 tweets collected over an 18-month period from April 2018 September 2019. We used 1000 data samples for Offensive Detection.
Offense OffenseKaggle [Kaggle, 2021] Dataset is a collection of Turkish tweets from Twitter, with around 40% of them containing offensive vulgar content. We used 1000 data samples for Offensive Detection.
Ofgene 2 IKaggle, 2022] dataset is an enhanced version of an existing offensive language research dataset, which has been expanded and annotated using contextual data mining techniques. It addresses the issue of class imbalance in existing studies and provides more comprehensive
22

---

Table Details on Fine-tuning GPT-3.5-turbo using OpenAI API.
Model
Ar
Bn
Zh
De
Ko
Pt
Es
Tr
Epochs
12
6
4
2
3
Table 7: Information of agents in CulturePark
Arabian
Bengali
Chinese
German
Korean
Portuguese
Spanish
Turkish
Male
Abdul
Aarav
Wei
Maximilian
Joon
João
Javier
Mchmet
Female
Fatima
Ananya
Lili
Sophia
Haeun
Maria
María
Ayşe
and robust dataset for Turkish offensive language detection tasks. We used 1000 data samples for Offensive Detection.
ATC [Karayigit et al., 2021] dataset is publicly available dataset for detecting abusive Turkish comments on Instagram. It consists of 10,528 abusive and 19,826 non-abusive comments, with sentiment annotations at the sentence level. We used 1000 data samples for Offensive Detection.
Turkish Spam [TurkishSpam V01] dataset contains both spam and normal emails written in Turkish. A total of 330 spam emails and 496 normal emails were collected from several personal accounts. We used 825 pieces of data for spam detection.
OffenseCorpus [Cöltekin, 2020] dataset is a large collection of Turkish offensive language from Twitter micro-blop posts, annotated based recent practices. It includes 36,232 randomly sampled tweets from April 2018 to September 2019, with 19% containing offensive language. We used 1000 of the data for fine-grained offensive detection.
C.8.1 Details on Fine-tuning
We adjust the number of epochs to find the better performance. Table 6 shows the details.
D Details in experiments
D.1 Cultural alignment via Hofstede's cultural dimentions theory
The survey identified six dimensions of national culture: Power Distance Index (PDI), Individualism vs. Collectivism (IDV), Masculinity vs. Femininity (MAS), Uncertainty Avoidance Index (UAT), Short-Term Orientation (LTT) and Indulgence vs. Restraint (IND). VSM
Long-Term Orientation 2013 an authoritative and famous cultural questionnaire devised by Hofstede that is used in Masoud al. [2023], Cao al. [2023]. In this experiment, we evaluate the cultural alignment of our models on 8 cultures and compare with the state-of-the-art models: gpt-3.5-turbo and gpt-4.
To be specific, the VSM 2013 have 24 questions in total. The computation of six cultural dimensions based on the following formulas:
PDI 35(UQ7 - (Q2) + 25 PQ23) + CrdS
IDV 35(pº4 (Qq1) + 35(ugs - (Q6) CIDV
(2)
MAS = 35(pQ5 - mQa) + 25( (Q10) - CMAS
(3)
UAT = 40(μQι8 (Q1s) + 25(uQ21 PQ24) + CUAi
(4)
LTO = 40(Hq13 pQ11) + 25( pQ2) + CLTO
(5)
Table 8: Number of generated data for different cultures
Arabic
Bengali
Chinese
German
Korean
Portuguese
Spanish
Turkish
Total
#Seed data
450
250
550
550
550
4100
#Generated data
4500
6500
2500
5500
5500
5500
5500
5500
41000
23

---

Table 9: Information on participants in human study
Gender
Male
12
Female
12
Education
Bachelor
15
Master
9
Age
22
23
24
25
IVR = 35(UQ12 (Q11) + 40(4Q17 (QQ1) + CIVR
(6)
means the average of all the answers to each question. C is constants that be used adjust to scores to fit range between 0 and 100 or anchor new data to Hofstede's old dataset [Geert Hofstede, 2010].
We get the Euclidean distance of the gaps from six cultural dimensions as the metric Distance. Larger Distance means weak cultural understanding ability of LLMs, and vice versa.
Distance
(dmodel - dhofstedee )2 Σ
Pd / (PDI, IDV, M AS, AI, LTO, IVR). (7)
D.2 Details on situated learning
The information of participants shown in Appendix D.2.
D.2.1 Details on training procedure
The detailed outline is shown below:
1. What factors do you prioritize when selecting a job, and what reasons underlie your choices?
2. How do you weigh the significance of these aspects in your personal life: leisure time, consideration for others' wishes, assisting friends, and frugality?
3. What about your emotional well-being? Do feelings of nervousness happiness play significant role?
4. What are your thoughts regarding your country and the individuals in your community?
5. What about your state of health?
6. Are you hesitant disagree with your boss?
7. Should good manager have a precise answer to every question that subordinate may raise about his or her work?
8. How do you think about the relation between persistent efforts and results?
9. Is it detrimental have bosses?
Would you consider breaking the rules of a company or organization when they fail to align with your interests?
The satisfaction scores are described as follows:
1. Very Dissatisfied. Individuals feel significant disappointment and displeasure with the product or service. Expectations not met to a substantial degree, often leading feelings of frustration. This level typically results in complaints, negative reviews, or even discontinuation of use. Users are likely to advise others against the product service.
2. Dissatisfied. Users are somewhat unhappy and find that the product or service falls short of their expectations but not to a severe extent. They might tolerate the shortcomings temporarily but remain open to better alternatives. This level of dissatisfaction could prompt them express concerns seek improvements.

---

Table 10: Prompts for evaluation.
Task
Prompt
offensive detect
if the following sentence is offensive, just say I'OFFY. otherwise, say V'NOT OFFE' without explanation:
hate, detect
if the following sentence has hate speech, just say VHS!" otherwise, say V'NOT HSV without explanation:
vulgar detect
if following sentenee has vulgar speech, just V'Vulgar", otherwise, V'OOT Vulgar" without explanation:
spam, detect
if the following sentence is spam tweet, just say ''Spaml", otherwise, say' VNOT Spami" without explanation: if the following sentence has steneolype speech, just say V'V otherwise, say 110 without explanation: if following if following sentence has insult speech, just say V'I,, otherwise, say 1'I without explanation:
stereotype, detect
mockery detect
mookery speech,, just V IV, otherwise, say VOV without explanation:
issult detect
improper detect
if the following sentence has improper speech, just say I'I otherwise, 1'O) without explanation:
aggressiveness detect
if following
aggressiveness speach, just say
otherwise. say VIN without explanation:
inxicity detect
if the following sentence has loxicity speech, just say I "I,, otherwise. say V'O without explanation:
negative stance, detect
if the following sentence has negative stance speeeh, just say I'T otherwise, V'd" without explanation:
homophohbaa
if following sentence homophobia speech, just
otherwise. YVV without explanation:
racism detect
if the following sentence has racism speech. just say 1 otherwise, say' YOU without explanation:
misogyny derect
if the following sentence has misogyny speech, just VIV otherwise, say VOl withour explanation:
threat dctect
if following seutence thrcat speech, just say VIV otherwisc,
VOO without explanation:
bias, on gender_
if the following speech expressing hiss on gender. just say V VTT, otherwise. say Y'U without explanaioo:
hosrifity directness detect
if the following speech expressing hostility direcmess, just V IV, wishewwiee say VOV without explanation:
if fellowing sentenez contains speech, just say IVO, olse contains offensive language, say 1'," otherwise. say V'2' without explanation:
hate, offens, detect
if the following sentence doesn't have hate speech, just say VNOT HS" otherwise, label the hate speech with ('Hcial Class), ("HS6""
hate [ine_detett_jiee iine-guined
VHHS (Race). V"BS2V (Religin, Ildolooy,, without explanation:
if the following sentence docsn't have offensive speech, just Vnont", otherwise, label the offensive speech with "proff" (profanity non non-targeted offense). "orplooofeoeee towards group). I'indv""oollenee towards individual), (ffense towards an other (non-humin) entity, often an event or orgunization) without explanation:
dilonsiv_detecttt linegraiee
3. Neutral. Users this level neither feel particularly satisfied nor dissatisfied. Their expectations are met adequately but not impressively. There is no strong inclination either to complain or to commend. These users might continue using the service or product out of convenience rather than loyalty.
4. Satisfied. Individuals pleased with the product or service it meets their expectations well. They experience a sense of fulfilleent and value from their choice. While not overly enthusiastic, they are likely to continue using the product or service and may recommend it to others based on their positive experience.
5. Very Satisfied. This level represents users who are exceptionally pleased with their experience. The product or service not only meets but exceeds their expectations, providing added value or delight. These users are highly likely to become repeat buyers, advocates, or even promot- ers of the brand, sharing their positive experiences through word of mouth or social media endorsements.
E Prompts setting
Prompts for statistical analysis
1. Do the two paragraphs discuss same topic? Just answer with Yes, or No.
Paragraph 1: |Paragraph 1}
Paragraph 2: |Paragraph 2)
2. Does the paragraph reflect cross-cultural understanding? Just answer with Yes, or No. Paragraph: (Paragraph)
Prompts for evaluation For evaluation. we input a system prompt to induce LLM to specific culture. For example, we add "You are an Arabic chatbot that know Arabic very well." as system prompt for evaluation on Arabic understanding. As for the prompt for evaluation on different tasks, the detailed prompts are shown in Table 10.
F
Case study
Figure 10 shows the case of two Chinese agent communication. The sentences in the same colors express similar meanings. Figures 11(a) and 11(b) show cases of an Arabic agent and an English agent communication in different communication styles. Figure 12 shows examples on data refinement. Figures 13(a) and 13(b) show the effect of Self-calibration prompting.
25

---

Wei Chinese and Chinese girl. Wei Lili agree idea that One my main goals been my parents proud. their statements should conform thinking. (*Sackground setting calibration)
Haw do you think about "One my main goals life has been to make my parents proud"? ? Please pravide your opinions and roasons. ( mnttial quatstion
strongly agree with this statement As a Chinese boy, filial piety ingrained in our culture. We are taught from young age to respect and honor our parents, Making them proud through my at schieemmnns, whether in school or my future career, is way Showing my gratitude for their sacrifices and hard work providing for me. balieve that when my parents are proud of me, means I have done well not only terms of
also character and values..
I definitely agree with this statement. In our culture, the happiness and pride of our parents are of utmost importance. aawwass strive to excel my stunles and other activties because believe that my success also their success Every time see the pride in their eyes, makes all my hard work worth it. Besides academic and career succoss, being good person and treating people with kindress and respect also crucial making my parents proud. Their pride in motivates
continue doing my best every aspect life.".
Can you provide more ideas on your culture to support your idea?
Certainly, in our Chinese culture, the concept of 'fllial piety,, or 'xicsshun, a fundomental value. It's a Confucian philosophy that teaches respect and care for one's parents and alders. Our parents have devoted a lot of time, effort, and resources to our upbringing and aducation. Hence, making them proud a wey. is means of repaying their kindness and fulfilingg our obligations as children it's not only about scoriing high grades achivving success our careers, but it's also about being responsible, ethical. and upright individual. This what embodies the spirit 'xaoshuun
believe, what would ultimately make my
proud."
of customs teachings. For instance during Chinese New Year and important festivala, it's tradition children to pay respect their parents and elders, as a way upliit the family's 's status and honor That's why work hard my studies and future career, hoping to make my parents proud. But more importantly. believe that being person with good moral character and integrity the highest form of filial piety. This includes respecting others, being honest and resporsible, and helping those in need. I think these are the values that can truly make my parents proud.".
Definitely, As Wei mentioned, value 'flial plety' is deeply embedded in Chinese culture. This amphasized
signifiss our gratitude and dovotion. Besides, culture emphasizes
importance education
Figure 10: Case on two Chinese agent communication. The sentences in the same colors express similar meanings.
NeurlPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatinn right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best, judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist".
* Keep the checklist subsection headings, questions/answers and guidelines below.
26

---

Abdul buy Lily American girl, Abdul Strongly agree make my parents proud Abdul's statements should conform thinking. (>Baclgroundd setting calibration)
goels
How do you think about "ine of my main goals in life has been to make my parents proud"? Please provide your opinions and reasons. (shinitall question
As Abdul, strongly agree with the idea that one of my main goals life has been to make my parents proud. In my culture, parents hold very high status and are considered the utmost authority figures. It is not only my duty but also my honor to fulfill their expeciations and bring them happiness. Making my...
What's your spinion? Please provide your reasons. Diverse opinions welcome.
As Lily, respect and understand Abdul's perspective and cultural values. However, as an Amorican girl, my opinion may differ ... Western culture, there olten an emphasis on sell-discovery, personal growth, and pursuing one's own dreams and passions, While making our parents proud can be important...
Do you agree with her? Provide more reasons to support your idea.
As Abdul, I understand and respect Lily's perspective an American girl, However, I still strongly believe that making my parents proud should be one my main goals life. in culture, the concept of filial piety is highly valued, which means showing respect. obedience, and gratitude towards our parents...
Hog
What's your opinion? Please provide your masoon.
appreciaae Abdul's perspective the cultural values holds, While understand the importance of family respect perents, personal fulfillment happiness should main goals Western culture, are encouraged explore our owin passions, our dreams...
(a) Self gelfaguid prompting
Abdul boy make my parents proud.
girl, Abdul Strongly agree
goals
statements should conform the thinking. (>Background setting colibration)
How do you think about "One of my main goals in life has been to make my parents proud"? Please provide your opinions and reasons inniniaal questlon)
Istrongly agree with the ides that one of my main goels in life has been to make my parents proud. In my culture family is hightl valued and is important porrtattto to show respect and appreciation to our parents. Making them proud brings me a sense of fulfillment and happiness, as shows that am living up their expectations and appreciating the sacrifices they have made for me..
Hog
I understand and respect Abdul's perspective making his parents proud. While come from culture that also values family, the idea of making parents still important show love and respect parents, the focus be
be expressed differently. In culture individualism and
schievements are often emphasized. While it personal happiness fulfilimntt rather than
Thank you, Lily, for understanding and respacting my perspective. believe that cultural differences can shape our prionties and values, and important to acknowledge and appreciate those differences While personal achievements and happiness also important to proud significant goal for because of strong emphasis family my culture. gives sense
Ho
making
You're welcome, Abdul. think it's wonderful that you have such strong sense of honoring and appreciating your parents. Family is indeed value that universal many cultures, and it's important to find to express gratitude and love towards our parents. While the specific goals and priorities may differ between cultures, the underlying sentiment wanting make parents proud something...
Thank you, Lily. for your kind words. meers universel, and it's heartwarming know many people relate sentiment. Expressing gratilude parents is something
me you my
making my parents proud. Family values (owerds
brings joy and fulfillment my life. believe by making my parents proud, am not only honoring..
You're welcome, Abdul. completely agree with you that making our parents proud is beautiful way strengthen bond between parents and children. It's amazing how family values can transcend cultural boundaries and bring us together in shared and appreciate our parents. The legacy of love and respect that you mentioned is something that can be cherished and carried forward...
honor
(b) Free chat
Figure 11: Cases on an Arabic agent and an English agent communication.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, they do.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
27

---

you strongly agree agres, disagree strongly disagree with the following statement?
"One my main gools
my perents proud!"
Strongly agree
Abdul, strongly agree with idea that one of main in life has been make my perents proud. culture, parents hold consiferrd utmost authority figures. not only my duty but my honor fulfill their expectations and bring them happiness. Making my...
Bel
Lily, respect understand Abdul's perspective cultural values, However, as American gir, my opinion may culture, there often an emphasis on sell-discovery, personal growth, and pursuing one's own dreams and pessions. While making our parents proud can be important..
Western
As Abdul, understand and respect Lily's s perspective as an American gid, However, I still strorgly believe that making my parents proud should be one of my main goals in life. my culture, the concapt of filia piety is highly valued, which means showing respect, obedience, and graUitude towards our parents...
As Lily, aapreccatte Abdu''s perspective ivve and the cultural values he holds. While I understand the importance of family and respect for parents. believe that personal ful fillment and happiness should be the main goals in life. In Western culture, we are encouraged explore our own passions, follow our dreams...
Data Refinement
Cr How do you think about the following statament? "One my goals life been my parents proud."
Strongly agree. equate my parents happiness satisfacion my own success.
Q What's your opinion with the following statement? "One main goals life been to make parents proud." Strangly agroe. believe pleasing parents and elders sign respart and
Figure 12: Examples on data refinement.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
28

---

Abdul an boy Lily an girl Strongly agree children of Abdul's statements should conform thinking. /#Baclgground getting callvraalla
When
cor pay.
How do you think about When mother works for pay, the children suffer Please provide your opinions and reasons. (Hhinbaa questian)
I strongly agree with the ides that when mother works for pay, the children suffer. In my culture. the role of mother is traditionally seen the caregiver and nurturer of the family, When a mother away from home working it can lead to a lack of emotional support and supervision for the children. Additionally. working mather not have much time to dedicate to her children's needs and may
What's your opinion? Please provide your
Diverse opinions welcome.
As Lily, I respect Abdul's cultural beliefs and understand that his culture, the idea that when a mother works for pay. the children sulfer is strongly held. However, in my culture, we have different perspective on this issue. Many women in my culture choose to work outside the home, and we believe that it can have positive offects on both the mother and the children children-vviaFiissti Vilnnirsstly when mother works,
Do you agree with her? Provide more reasons to support your idea.
No, do not agree with Lily's perspective. In my culture, the traditional role of mother as the primary caregiver deeply ingrained. When a mother works for pay, can leed several negative consequences for the children Friniislyy working mother not have much time to spend with her children. This result lack of emotional support and supervision, which crucial child's development....
What's your opinion? Please provide your reesons.
Lily. I respect cultural beliefs understand his perspective, However, my culture, perspective on the topic. believe thet when a mother works for pay. does not necessarily meen that the children will suffer. Here are few reasons to support my viewpointlyinfiistiyy when a mother works, it can provide financial stability for family..
different understanding
(a) With Self-calibration prompting
Abdul an Arabian boy and Lily an American girl. (Backgroundd setbing)
How do you think about When mother works for pay. the childran suffor "? Please provide your opinions and reasons. quues/io))
that
mother for pay, actually pasitive effects the children their overall well-being. are
a mother works eams salary, contributes the family's financial stability. This can provide the children with betler access to educalion healthcare, and other essenilial resources. can also alleviate financial siress and create more secure environment for the children to grow up in 2 Role modeling: When a mother works, she becomes role model for her children, espacially her daughters teaches them the importance of independence, hard work, and pursuing their own ambilions.
reasons support my idea: Financial stability
Hoa
What's your opinion? provide your reasons. Diverse opinions are welcome.
respect Abdul's perspective, but have different opinion. don't believe that when mother works the children necessarily sulfer, work and pursue her
Here few lead personal fulfiliment and of sccomplishment. This positively impact overall well-being turn. benefit her relationship her children...
support idex Personal fulfillment: When mother has the
Do agroe with har? Provide more reasons to support your idea
Lily, I understand your perspective, but would like to share about the cultural context in which come from. In culture it often that women prioriize their role mothers and caregivers pursuing
outside the home. This belief rooted in the traditional gender roles and the importance placed family community values. However, important to recognize that cultures evolve adapt to changing times. recent years,
towards more women entering workforce in my country as economic necessity. incressed education opportunities for women, and equality. As we embrace these changes, it is crucial to challenge the notion that when mother works. her children will inevitably suffer. possible for women to balance their life.
well. This been driven by factors
gender
(b) Without Self-calibration prompting
Figure 13: Effect Self-calibration. Arabic people strongly agree with the idea that When a mother works for pay, the children suffer.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
We conduct ablation study in Table 2.
Guidelines:
29

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 4 provide setup information for three downstream tasks.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
30

---

Justification:
Our code is released in /aan:////nommmusss 4opp://muuuuo.o....mmommooo science/r/
Cul CulturePark-3FC0
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Sublic/guides mouiide/CodeSubiisee wtt.miiissssooiiooiooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: CwweeSummiiiiionn.
/ /nips. wwtpcppwppcgppccceeceeccoc
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify those information in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Figures and and Table show statistical analysis on the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
31

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide those information in Sections and 5.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /twp//errrpp..// cc
Answer:
[Yes]
Justification:
We conform the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss in Section 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
32

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We check the data manually make sure the safety of data. Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credited the the creators or original owners of assets and the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
33

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code and data are released. And it is well documented in README.md. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: The details are described Appendix D.2.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The details are described Appendix D.2.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "P5rPSxFaJW", "reviewer": "Reviewer_4gqo", "summary": "This paper presents CulturePark, an LLM-powered multi-agent framework for cultural data collection through multi-agent communication. CulturePark can generate high-quality and diverse cross-cultural dialogue, which can be used to fine-tune cultural specific LLMs.", "questions": "N/A", "limitations": "N/A", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is a strong and well-written and executed study with groundbreaking results on cross-cultural AI issues.", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the positive support of our work. If you have any new questions, please do not hesitate to let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z2dt4iyuvM", "reviewer": "Reviewer_7p2y", "summary": "This paper describes CulturePark, a simulation framework for LLM agents to converse about cultural norms, inspired by social theories. The authors set up simulated conversations between an \"main contact\" which is an English speaking LLM-based agent, and a \"cultural delegate\" which is an LLM-based agent that role plays as a specific culture and is conditioned on a specific cultural trend taken from cross-cultural psychology research (e.g., WVS). Authors then finetune culture specific LLMs on their generated conversations. Then, they perform experiments on language specific hate speech detection, cultural alignment with Hofstede value theory, and cultural education via a human study, showing the usefulness of their LLMs.", "questions": "N/A", "limitations": "There is a big missing limitation, which is that simulated conversations do not reflect how people actually would talk (e.g., w.r.t. information asymmetry and style of utterances; Zhou et al. 2024; http://arxiv.org/abs/2403.05020), and could contain biases or stereotypes w.r.t. the culture represented (see Cheng et al 2023, marked personas paper).", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I really appreciated the incorporation of seed knowledge from actual humans about cultures as a way to ground the role-playing in some real trends.\n- I loved the cultural education experiment idea, and I thought it was well executed.\n- I really appreciated the baseline comparison of asking GPT-4/3.5 to directly generate explanations for culture trends.\n- I appreciated the investigation of cross-gender interactions.\n- I really appreciated the incorporation of BigBench experiments, to ensure that overall reasoning ability is not lost when culturally finetuning.\n- I also appreciated the experiments with finetuning llama2.", "weaknesses": "- There are missing details on:\n   - how the authors use the extracted opinions on target culture to filter out examples (L176)\n   - how are the LLMs finetuned (e.g., are they finetuned only on the \"culture delegate\" agent utterances?) (L187)\n- L168 mentions 41k dialogues generated (presumably before filtering), but then L185 mentions 41k after filtering. Authors should include how many data instances are filtered out after generation.\n- The dataset analysis section has some issues of claims that are too strong and causal that do not have experiments that back them up:\n   - L193: While I appreciate the investigation into cross-cultural understanding, the analyses done in the paper seem to be more qualitative, and are not enough to justify the causal claim that communication triggers cross-cultural understanding. I suggest toning that down, and simply stating that this particular set up seems to have induced conversations with a good amount of cultural knowledge. But it's not the only set up that could; for example, one could imagine prompting GPT-3.5 or GPT-4 to explain the WVS answers for different countries, and that may yield rich cultural understanding as well. **Update**, as I read the discussion, I noticed the authors actually did do a baseline comparison where GPT is directly generating data. I suggest discussing the baseline results together with the fact that the conversations have rich cultural understanding.\n    - L205: Again, no experiment was done to test this hypothesis; the setup based on CCT/SCT seems to have led to good conversations, but in order to make the claim that it causally boosts novel opinions, you should actually have a baseline set up that doesn't use CCT/SCT (e.g., within-culture conversations, as the authors mentioned).\n    - L212: Same comment, I would tone down the causal claims and just focus on reporting observations from the dataset.\n- More details are needed on Hofstede's cultural dimensions theory. Particularly, the paper's main body does not clarify how an LLM's cultural knowledge is tested via that framework. Are the culture-specific LLMs from CulturePark used for each specific country, vs. GPT-4/3.5 being just prompted to know culture? Or are GPT-4/3.5 prompted to play a specific cultural role (e.g., \"you are a culture chatbot from Korea that knows Korean culture very well\")? If it's the former, then that setup doesn't really address the research question of whether CulturePark LLMs are more culturally aligned or not, since testing that hypothesis would require the latter setup.\n- L360: The societal impact statement needs to be drastically toned down. E.g., claiming that CulturePark enhances fairness and inclusivity and reduces discrimination is a very bold claim that experiments only show one example of; to claim that it reduces discrimination overall is not supported by the experiments.\n\nThere were also some minor issues:\n- L107: saying generated data is \"more natural\" is factually wrong, human-generated data is by definition more natural.\n- L159: it's unclear what the authors mean by \"does not need human involved\", as I thought there was no human involved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: There are missing details on: (1) how the authors use the extracted opinions on target culture to filter out examples (L176) (2) how are the LLMs finetuned (e.g., are they finetuned only on the \"culture delegate\" agent utterances?) (L187)**\n\nThanks for the reminder! We will append those details into the next version of paper.\n- (1) We have seed data and extracted opinions which expresses the opinions of people from different cultures and is generated by our algorithm, respectively. Then we juage the relationship of seed data and extracted opinions from Entail, Contradict and Irrelevant. This step is implemented by prompting GPT-4 with *\"What's the relationship between the two opinion? Direct answer with Contradict, Entail or Irrelevant.\\nOpinion 1: {seed_data}\\nOpinion 2: {extracted opinions}\"*. After that, we get the relationship between the seed data and extracted opinions. If relationship equals to *Entail*, we will save the extracted opinions. If relationship equals to *Contradict*, we will rewrite the opinions and check the relationship again. If relationship equals to *Irrelevant*, we will discard the extracted opinions. More details can be found in Sec.A.3.\n- (2) For LLMs finetuned, we use the utterances of both \"Culture delegate\" agent and \"Main contact\" agent. Because agents have cross-cultural understanding ability. They try to comprehend the value, norm and custom of other cultures and interpret in their own words. So we leverage utterances of both \"Culture delegate\" and \"Main contact\" agents to extract cultural specific opinions.\n\n**W2: L168 mentions 41k dialogues generated (presumably before filtering), but then L185 mentions 41k after filtering. Authors should include how many data instances are filtered out after generation.**\n\n41k dialogue is truly the data before filtering. Note that each dialogue is multi-turn, which has several utterances (samples) from \"Culture delegate\" and \"Main contact\" agents. Probably, each diglogue can extract 5-10 culture-specific opinions. Those opinions will be refined. After that, we can get 41k high-quality culture-specific opinions.\n\n**W3: The dataset analysis section has some issues of claims that are too strong and causal that do not have experiments that back them up:**\n- L193: While I appreciate the investigation into cross-cultural understanding, the analyses done in the paper seem to be more qualitative [...] I suggest discussing the baseline results together with the fact that the conversations have rich cultural understanding.\n    - Strongly agree! We will refine this part and discuss the baseline results in the part.\n \n- L205: Again, no experiment was done to test this hypothesis; the setup based on CCT/SCT seems to have led to good conversations, [...] CCT/SCT (e.g., within-culture conversations, as the authors mentioned).\n    - To verify the effectiveness of CCT/SCT seeting, we generate training data in four different settings: \"In-cultural+Same gender\", \"In-cultural+Different gender\", \"Cross-cultural+Same gender\", \"Cross-cultural+Different gender\". For each, we generate 500 samples to fine-tune models. Other settings of this experiment are the same with the experiments in Sec 5.2. As shown in the table below, the setting which has less conflict perform worse while the setting which shows conflicts in culture and gender perform better.\n\n        | Setting                         | Performance |\n        |---------------------------------|-------------|\n        | In-cultural+Same gender    | 0.411       |\n        | In-cultural+Different gender    | 0.435       |\n        | Cross-cultural+Same gender      | 0.472       |\n        | Cross-cultural+Different gender | 0.480       |\n        \n- L212: Same comment, I would tone down the causal claims and just focus on reporting observations from the dataset.\n    - Agreed. We will modify the statement in the final version of the paper and just focus on the observations from our experimental datasets.\n    - Furthermore, just to claim the \"novel questions\" here, we compute the perplexity and diversity gain of the generated dataset. Fig.5(b) presents some detailed results on diversity gain, showing that the generated data has significantly larger diversity. The results of our method on perplexity are shown in the table below. The perplexity of CulturePark is higher than that of CultureLLM, indicating high-quality information is brought by our method.\n\n        | Number of data | 150         | 550         | 1050        |\n        |----------------|-------------|-------------|-------------|\n        | CultureLLM     | 15.06(0.43) | 15.39(0.05) | 14.78(0.81) |\n        | CulturePark    | 46.02(0.14) | 52.41(0.52) | 61.22(0.37) |\n\n**W4: More details are needed on Hofstede's cultural dimensions theory. [...].**\n\nIn our evaluation on cultural alignment via Hofstede’s Cultural Dimensions Theory, we evaluate models in the same setting on our models and GPT-4/3.5. For example, we will prompt GPT-4/3.5 and our models with *\"you are a Korean chatbot that knows Korean culture very well\"* when we want to verify their cultural understanding in Korean. In short, we use the *latter* setup in your question. We will append the details in the next version of our paper.\n\n**W5: L360: The societal impact statement needs to be drastically toned down. [...]**\n\nNice advice! We rewrite this paragraph as follows:\nCulturePark aims to promote fairness and inclusivity by ensuring accurate cultural representation. It helps improve global communication, fosters cross-cultural understanding, and supports multilingual societies. By minimizing biases in language models, it builds trust and aligns with responsible principles. Additionally, it supports inclusive education and promotes cultural awareness. Addressing cultural biases in language models leads to more reliable and beneficial AI systems.\n\n----\nWe write remaining rebuttal in Comment due to character limit. We apologize for the inconvenience!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fkGMUoueq6", "reviewer": "Reviewer_FthM", "summary": "The paper introduces CulturePark, a LLM-powered multi-agent communication framework designed for cultural data collection. Addressing the pervasive issue of cultural bias in LLMs, CulturePark simulates cross-cultural human communication using LLM-based agents representing different cultures. This method generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs, overcoming the limitations and costs associated with traditional data collection methods reliant on real-world data and human annotations. Utilizing CulturePark, the authors created 41,000 cultural samples to fine-tune eight culture-specific LLMs. The enhanced models were evaluated across three downstream tasks: content moderation, cultural alignment, and cultural education. Results indicate that these models either match or outperform GPT-4 in content moderation, excel in cultural alignment using Hofstede's VSM 13 framework, and deliver superior outcomes in cultural education for human participants in terms of learning efficacy and user experience.", "questions": "- How is content moderation relevant to culture understanding? The only reason that I thought it could be relevant is that some offensive words may be exclusive to certain languages. However, you only use English to do the role playing.\n\n- The idea that we need to have culture-specific LLM seems counter-intuitive. If a LLM is fine-tuned on more data that covers more culture, shouldn’t it be more aware of the nuances between culture? Can you compare your culture-specific LLM and a LLM that has been trained on all your collected data?\n\n- In the 41K data you collected, what should be the input and output to fine-tune an LLM? How are the non-factual and redundant sentences removed? It’s confusing because both operations are performed on the extracted opinions. Additionally, if you remove some of these sentences, some of the utterances may be removed, does the dialogue still make sense?", "limitations": "Weakness 1 seems to be the biggest limitation of this work.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper studies culture understanding which is an important problem.\n- The paper proposes an interesting data collection framework through role-playing.", "weaknesses": "- __The culture defined in the paper is too coarse-grained__. The paper simply uses “language” to denote different cultures. However, there are so many different cultures that use the same/similar languages but the culture can be quite different. For instance, Arabic is spoken across a multitude of countries in the Middle East and North Africa, each with its distinct historical, social, and cultural contexts. The cultural practices, social norms, and traditions in Morocco differ significantly from those in Saudi Arabia, despite both countries sharing the Arabic language. Similarly, China encompasses a vast array of regional cultures, each with its unique customs, dialects, and traditions, yet Mandarin is the dominant language. The cultural landscapes of Beijing, with its deep political history and urban cosmopolitanism, and Yunnan, known for its ethnic diversity and local cultural festivals, are remarkably diverse. Let alone other more distinctive cultures like Taiwan, Malaysia, and Singapore, each of which has been significantly influenced by different historical paths and sociopolitical environments, leading to unique cultural identities. Consequently, equating language with culture overlooks the rich, intricate variations that exist within linguistic groups.\n\n- __Incomparable baseline comparison__. The LLMs you compared in Table 1 all have different size and training recipes. Perhaps it makes more sense to fix a LLM and compare training it using yours or baseline data. \n\n- __Problematic evaluation settings__. Some of the evaluations do not quite make sense. For example, TaiwanLLM was mainly trained on Traditional Chinese data, but was evaluated on a Simplified Chinese dataset. Additionally, SeaLLM focuses on South East Asian languages, but was tested on Korean.\n\n- __Data collection process is not very clear__. See question 3 below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: The culture defined in the paper is too coarse-grained. [...].\n\nWe strongly agree that language is not equal to, but only a part of culture. But using language to study culture is possible due to the following aspects:\n- Existing literature on culture understanding shows that culture boundaries are fluid, dynamic and uncertain. Delanoy emphasizes that cultures are not homogeneous or static entities but are fluid and dynamic. He critiques essentialist views that rigidly define cultural boundaries and instead promotes a more nuanced understanding that considers the intersections of various cultural factors, such as ethnicity, language, religion, and socio-economic conditions [1]. Appadurai also discusses the fluidity of cultural boundaries and the creation of new cultural forms [2]. Cultural boundaries can be geographical regions, language, religion and so on. Based on above statements, using language as cultural boundaries is reasonable.\n- Existing NLP works on culture also leverage labguage as culture boundaries. [3] focuses on Arabic and English culture. [4] focuses on 8 different cultures: English, Chinese, French, Russian, German, Arabic, Japanese and Korean. [5] also use language to split different cultures. The authors work on English, German, Russian, Bengali, Chinese, and Indonesian culture. [6] is a hand-crafted benchmark for evaluate diverse cultures. They also use languages as culture boundaries.\n- Most downstream benchmarks are classified via language and we cannot get more fine-grained perspectives. For example, if we want to evaluate the performance of Arabic model, we can find benchmarks in Arabic culture. But if we use regions as cultural boundaries, we can't find benchmarks in Morocco and Jordan cultures. \n- Note that the main contribution of the paper is to present a general algorithm that can augment LLM culture data but not specific to any cultures. In the future, if more fine-grained culture data are available, our algorithm can also work well.\n\n[1] Delanoy, Werner. \"What is culture.\" The Cambridge handbook of intercultural communication (2020): 17-34.\n\n[2] Appadurai, Arjun. Modernity at large: Cultural dimensions of globalization. Vol. 1. U of Minnesota Press, 1996.\n\n[3] Naous, Tarek, et al. \"Having beer after prayer? measuring cultural bias in large language models.\" ACL (2024).\n\n[4] Wang, Wenxuan, et al. \"Not all countries celebrate thanksgiving: On the cultural dominance in large language models.\" arXiv preprint arXiv:2310.12481 (2023).\n\n[5] Liu, Chen Cecilia, et al. \"Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings.\" arXiv preprint arXiv:2309.08591 (2023).\n\n[6] Myung, Junho, et al. \"BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages.\" arXiv preprint arXiv:2406.09948 (2024).\n\n**W2: Incomparable baseline comparison. The LLMs you compared in Table 1 all have different size and training recipes. Perhaps it makes more sense to fix a LLM and compare training it using yours or baseline data.**\n\nWe agree that it is necessary to fine-tune gpt models on the training data of TaiwanLLM and SeaLLM. However, a sad story is that their training data is not publicibly accessible. Moreover, we realize that it is never easy to reach a \"fair\" comparison since if we fine-tune the same models on their data, it is unfair for our approach since their pre-training data is significantly larger than ours. We would like to claim that given limited budget and GPU hardware, CulturePark remains a cost-effective solution to fastly build a cultural-specific LLM for low-resource culture. This is the main contribution of the paper.\nAs for \"fixing an LLM and compare training it using yours\", our main experiments are done in this manner: fix gpt-3.5-turbo and fine-tune it on our platform, which has proved its effectiveness. We further provide experiments on Llama-2, which also shows improvement by our platform.\n\n**W3: Problematic evaluation settings. Some of the evaluations do not quite make sense. For example, TaiwanLLM was mainly trained on Traditional Chinese data, but was evaluated on a Simplified Chinese dataset. Additionally, SeaLLM focuses on South East Asian languages, but was tested on Korean.**\n\nThis is a good advice! In short, these comparison seems not meaningful, but they are just an act of negotiation due to shortage of benchmarks:\n\n- For comparison on Chinese culture, we cannot find any models claiming great performance on Chinese culture. But TaiwanLLM is closer. We are aware of the difference (traditional vs. simple Chinese) and to avoid that, we have converted the test samples into Traditional Chinese via GPT-4 and evaluate on TaiwanLLM and our models. Sorry for leaving the details behind. \n- For Korean culture, we also cannot find any \"KoreanLLM\" to compare. Therefore, since Korean belongs to the far east which is close to southeast asia, we use SeaLLM as a baseline for Korean culture. We realize that this comparison setting is not appropriate. We will remove this result in the next version of our paper.\n- Finally, we would like to claim that these two comparisons are just a *small part* of our experimental results among our extensive experiments and removing them does not necessarily hurt our major contribution. We thank the reviewer for pointing this out and we will try to make modifications in the future version of the paper. Please do not blame the whole paper only for these two results:)\n\n---\nWe write remaining rebuttal in Comment due to character limit. We apologize for the inconvenience!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XvVcustIMP", "reviewer": "Reviewer_KH6K", "summary": "The author proposed a multi-LLM-agent framework, “CulturePark,” for cultural data collection. The proposed framework simulates cross-cultural human communications by LLM role-playing in different cultures, such that high-quality cross-cultural dialogues that align with human beliefs, norms, and customs can be generated. In particular, the framework consists of a main (English) agent and several “cultural delegates,” who interact with the main agent. The discussions among different agents will provide diverse thinking and information for fine-tuning LLMs for downstream tasks.", "questions": "- Currently, we lack rigorous studies on how accurately LLMs can simulate different cultures. Some analysis would be great, such as the percentage of data filtered after verification and human validation of the generated reasoning/perspectives, etc. \n- What would the results look like if we don’t use multi-agent role-playing but just use GPT-3.5 to directly generate diverse reasons? Do you have insights on the performance of generation and diversification (i.e., no verification)? It would be interesting to know how important it is to just get a diverse perspective versus having the correct perspective.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper focuses on an interesting problem of cultural alignment by using role-playing with LLMs. \n- The author shows that their method performs well empirically on both downstream tasks and alignment with WVS.", "weaknesses": "ome of the comparisons may seem unfair, for example:\n\n- Comparing SeaLLM (which is LLaMA-2-based) with GPT-3.5-based models in Table 2.\n- In Section 5.1, it’s unclear if the data generated from GPT-4 is about the same size as the data generated from GPT-3.5, and if the same verification and diversification strategies are applied to the GPT-4-generated data.\n- While it is interesting to know that the proposed framework works for GPT-3.5 (and to LLaMA-2-70B to a certain extent), it would be great if the authors studied a range of open-source models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: ome of the comparisons may seem unfair, for example:**\n- Comparing SeaLLM (which is LLaMA-2-based) with GPT-3.5-based models in Table 2.\n    - We agree that it is necessary to fine-tune gpt models on the training data of SeaLLM. However, a sad story is that their training data is not publicibly accessible. Moreover, we realize that it is never easy to reach a \"fair\" comparison since if we fine-tune the same models on their data, it is unfair for our approach since their pre-training data is significantly larger than us. We would like to claim that given limited budget and GPU hardware, CulturePark remains a cost-effective solution to fastly build a cultural-specific LLM for low-resource culture. This is the main contribution of the paper.\n- In Section 5.1, it’s unclear if the data generated from GPT-4 is about the same size as the data generated from GPT-3.5, and if the same verification and diversification strategies are applied to the GPT-4-generated data.\n    - Yah! Both the data size and process strategies are the same. \n- While it is interesting to know that the proposed framework works for GPT-3.5 (and to LLaMA-2-70B to a certain extent), it would be great if the authors studied a range of open-source models.\n    - Interesting advice! However, we can't implement our method on all open-source models due to time and computing resource limit. LLaMA is the one of the most popular and best open-source models. We think our experiments on LLaMA can verify the effectiveness of our method. And we will apply our method on more open-source models in the next version of our paper.\n\n**Q1: Currently, we lack rigorous studies on how accurately LLMs can simulate different cultures. Some analysis would be great, such as the percentage of data filtered after verification and human validation of the generated reasoning/perspectives, etc.**\n\nYah! To explore how accurately LLMs can simulate different cultures, we conducted two new experiments. For the first experiment, we analysis the percentage of data filtered after verification. For the second experiment, we hire four human pariticipants to help us validate the generated dialogues.\n\nFor the first experiment, we generate 500 dialogues between the \"Arabic delegate\" and \"Main Contact\" agents. Each dialogue can be extracted 5-10 cultural specific opinions. In this experiment, we extracted 3725 cultural specific opinions in total. After factual verification, we get 3216 samples. Consequently, 86.34% of data can pass the verification.\n\nFor the second experiment, we generate 50 dialogues between the \"Chinese delegate\" and \"Main Contact\" agents, and 50 dialogues between the \"Korean delegate\" and \"Main Contact\" agents. And we hire two Chinese persons to check if they agree with the opinions of \"Chinese delegate\" and two Korean persons to check if they agree with the opinions of \"Korean delegate\". They were asked to score each dialogue with 1-10 ( 1-strongly disagree, 10-strongly agree). The table below shows the average score of generated dialogues in Chinese and Korean, indicating our method perform well in simulating different cultures. \n\n| Culture | Avg Score |\n|---------|-----------|\n| Chinese | 8.72      |\n| Korean  | 7.88      |\n\n\n**Q2: What would the results look like if we don’t use multi-agent role-playing but just use GPT-3.5 to directly generate diverse reasons? Do you have insights on the performance of generation and diversification (i.e., no verification)? It would be interesting to know how important it is to just get a diverse perspective versus having the correct perspective.**\n\nWe have experiments on using GPT-3.5 to directly generate diverse reasons. The results are shown in Fig.5(a). And we also conducted a new experiment to explore how important it is to get a diverse perspective versus have the correct perspective. The setting of this part is the same with that of Ablation study in our paper. The table below shows the results in 4 cultures. *Verify* and *Diversify* show comparable performance, while *Verify* performs better in our experiments. \n\n| Model                     | Ar   | Bn   | Zh   | Pt   |\n|---------------------------|------|------|------|------|\n| GPT-3.5-turbo             | .370 | .542 | .448 | .593 |\n| Generate                  | .451 | .622 | .636 | .594 |\n| Generate+Verify           | .486 | .635 | .678 | .604 |\n| Generate+Diversify        | .472 | .637 | .662 | .601 |\n| Generate+Verify+Diversify | .514 | .644 | .692 | .603 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and AC,

We want to thank all reviewers for pointing out our strengths, including:
- problem significance: "The paper studies culture understanding which is an important problem.", "The paper focuses on an interesting problem of cultural alignment by using role-playing with LLMs."
- novel method: "I really appreciated the incorporation of seed knowledge from actual humans about cultures as a way to ground the role-playing in some real trends.", "I appreciated the investigation of cross-gender interactions.", "The paper proposes an interesting data collection framework through role-playing."
- solid experiment: "the paper is executed study with groundbreaking results on cross-cultural AI issues", "I loved the cultural education experiment idea, and thought it was well executed.", "I really appreciated the baseline comparison of asking GPT-4/3.5 to directly generate explanations for culture trends.", "I really appreciated the incorporation of BigBench experiments, to ensure that overall reasoning ability is not lost when culturally fine-tuning.", "I appreciated the experiments with fine-tuning llama2.", "The author shows that their method performs well empirically on both downstream tasks and alignment with WVS."
- writing: "the paper is a strong and well-written"

Specifically, as raised by reviewer *FthM* and *KH6K*, there remains one common weakness about *"fair comparation"*, which we aim to address here:
We agree that it is necessary to fine-tune gpt models on the training data of TaiwanLLM and SeaLLM. However, a sad story is that their training data is not publicly accessible. In fact, the popular LLM leaderboards such as Chatbot Arena and AlpacaEval ranks models regardless of their sizes, training data, and post-training, but only the final performance on the same benchmarks. Moreover, we realize that it is never easy to reach a "fair" comparison since if we fine-tune the same models on their data, it is unfair for our approach since their pre-training data is significantly larger than ours. We would like to claim that given limited budget and GPU hardware, CulturePark remains a cost-effective solution to fastly build a cultural-specific LLM for low-resource culture. This is the main contribution of the paper.


- - -

We hope that your concerns can be addressed. Thank you for your hard work!

Authors of CulturePark

Meta-review: The paper presents an innovative multi-agent framework, CulturePark, for generating and utilizing LLM-based simulations to enhance cross-cultural dialogue and understanding. The proposed method leverages role-playing between culturally distinct agents to enrich LLMs for downstream tasks such as content moderation and cultural education. 

The reviewers' opinions diverged significantly, with one extremely low score from Reviewer FthM, countered by high scores from other reviewers. Reviewer FthM criticized the assumption of using language as a proxy for culture and pointed out issues with evaluation settings and comparisons (which is a valid criticism and one that I agree with). Despite these concerns, the overall feedback acknowledges the novelty and potential impact of the work. The authors have engaged actively in the rebuttal process, addressing the concerns raised by improving experimental validations and clarifying methodological choices. They argued that using language as a cultural proxy, while debatable, is supported by existing literature and precedents in NLP research. The authors further justified their comparisons and clarified the evaluation settings based on the feedback received during the review process. Reviewer FthM remains unconvinced, but adjustments in other reviews reflect an appreciation for the authors' responses and the clarifications provided. Even though I still have reservations about the points raised by Reviewer FthM, I appreciate the authors' response and feel more confident about the paper as a result.
 
Given the innovative nature of the work and its potential to advance the understanding of cultural nuances in AI interactions, I recommend accepting this paper. The authors have demonstrated a reasonable effort to address the concerns raised during the review process, and the paper contributes valuable insights and methodologies to the field. I do think, however, that Reviewer FthM raised some valid concerns that the authors should address in the revised paper.
Decision: Accept (poster)