CulturePark: Boosting Cross-cultural Understanding in Large Language Models

Authors: cheng  li
damien  teney
linyi  yang
qingsong  wen
xing  xie
jindong  wang
william & mary
culturepark-ar  culturepark
bn  culturepark-zh
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 07:29:35

Abstract: Cultural bias is pervasive in many large language models (LLMs), largely due to the deficiency of data representative of different cultures.
Typically, cultural datasets and benchmarks are constructed either by extracting subsets of existing datasets or by aggregating from platforms such as Wikipedia and social media.
However, these approaches are highly dependent on real-world data and human annotations, making them costly and difficult to scale.
Inspired by cognitive theories on social communication, this paper introduces CulturePark, an LLM-powered multi-agent communication framework for cultural data collection.
CulturePark simulates cross-cultural human communication with LLM-based agents playing roles in different cultures.
It generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs.
Using CulturePark, we generated 41,000 cultural samples to fine-tune eight culture-specific LLMs.
We evaluated these models across three downstream tasks: content moderation, cultural alignment, and cultural education.
Results show that for content moderation, our GPT-3.5-based models either match or outperform GPT-4 on $41$ datasets. Regarding cultural alignment, our models surpass GPT-4 on Hofstede's VSM 13 framework.
Furthermore, for cultural education of human participants, our models demonstrate superior outcomes in both learning efficacy and user experience compared to GPT-4. CulturePark proves an important step in addressing cultural bias and advancing the democratization of AI, highlighting the critical role of culturally inclusive data in model training. Code is released at https://github.com/Scarelette/CulturePark.

Main paper: 
 introduction: 
 Culture is an important part of human society, composed of human beliefs, norms, customs, etc. [Spencer-Oatey and Franklin, 2012] . As large language models (LLMs) play a vital role in daily communication [Yang et al., 2024] , recommendation systems [Li et al., 2023 , Fan et al., 2023] , and Figure 1 : CulturePark is an LLM-based multi-agent communication platform for cultural data collection. Leveraging CulturePark, we can collect a cross-cultural dialogue dataset, which can then be used for fine-tuning culturally specific LLMs to be applied to different downstream tasks: content moderation, cultural alignment, and cultural education.education [Shaikh et al., 2023] , it is imperative for LLMs to perceive and reflect different cultures. However, current state-of-the-art LLMs have been reported to be biased towards mainstream culture while ignoring others, resulting in a cultural bias problem [Liu et al., 2023 , Cao et al., 2023 , Masoud et al., 2023 , Naous et al., 2023 , Wang et al., 2023 , Johnson et al., 2022b] . This leads to stereotypical impressions of different cultures, which can even exacerbate social conflicts [Ryan et al., 2024] . The main reason behind cultural bias is that the training corpus of LLMs is dominated by English data that express the cultural values and opinions of western people. Much less can be learned about other cultures simply because there is less data available, i.e., a low-resource situation.Existing approaches to solve the cultural bias problem in LLMs include prompt engineering [Kovač et al., 2023 , Wang et al., 2023 , Rao et al., 2023] and pre-training in non-English languages [Pires et al., 2023 , Chan et al., 2023 , Nguyen et al., 2023b , Pipatanakul et al., 2023 , Abbasi et al., 2023 , Lin and Chen, 2023] . Prompt engineering consists of tuning prompts for different cultural tasks, but the benefits do not hold reliably across various downstream tasks. Pre-training in various languages is promising, but the data collection and the pre-training itself are both very costly. More importantly, cultural differences are embodied in many aspects such as opinions, customs, norms, and languages. A model serving all cultures may face cultural conflict and misalignment problems [Liu et al., 2023 , Cao et al., 2023 , Masoud et al., 2023] . Thus, it is necessary to fine-tune culture-specific models that target specific cultures. Recently, Li et al. [2024] proposed CultureLLM, which augments the fine-tuning data of LLMs via semantic data augmentation to train culture-specific LLMs. However, the generated data lack diversity because it is implemented by generating semantically equivalent sentences of seed examples.In this paper, we present CulturePark, an LLM-powered multi-agent framework to simulate crosscultural communication of humans. As shown in Figure 1 , CulturePark serves as an effective data collection platform to generate diverse and high-quality cultural datasets via multi-agent communication. CulturePark consists of a main contact (an English-speaking agent, Lily) who is in charge of the multi-turn dialogue and several cultural delegates (e.g., Abdul) who interact with the main contact and create cognitive conflicts. 3 After an initial problem is provided as input to the framework, the agents discuss the problem and express their opinions. Their different cultural backgrounds and genders boost diverse opinions and encourage one another to think more deeply. Original questions and ground truth can be augmented by creating novel questions and more comprehensive answers. The interactions eventually generate a cross-cultural dialogue dataset that contains deep and comprehensive thinking and informative knowledge of different cultures. Detailed statistics are shown in Table 4 . We then refine the original dataset to factually verify and increase its diversity, which is used to fine-tune culturally specific LLMs for downstream tasks, as shown in Figure 2 .From the perspective of cognitive social science, our framework is inspired by Cognitive Conflict Theory (CCT) [Limón, 2001, Cosier and Rose, 1977] and Social Cognitive Theory (SCT) [Fiske and Taylor, 1991] to foster a collaborative and communicative environment for mutual understanding of cultures. Specifically, CulturePark allows agents to encounter cognitive conflicts, which triggers deeper thinking on certain topics according to CCT [Limón, 2001, Cosier and Rose, 1977] . At the same time, a deeper understanding of cultures can be prompted through interaction and communication with other agents, as suggested by SCT [Fiske and Taylor, 1991] . In favor of these theories, we found that CulturePark triggers LLMs' cross-cultural understanding ability, boosts novel opinions by allowing agents to think deeper, and benefits data augmentation by creating more comprehensive answers to the questions. CulturePark has the potential to facilitate the data collection of culture-related tasks, cultural value alignment, and improving AI democracy.In summary, the contributions of this paper are three-fold:1. We introduce CulturePark, a cost-efficient multi-agent framework to boost the cross-cultural understanding in LLMs. Our platform creates cognitive conflicts and interactions between different cultural agents. More importantly, the platform uncovers several interesting findings, such as communication enables the cross-cultural understanding ability of LLMs, boosts novel opinions, and benefits data augmentation. 2. Leveraging CulturePark, we generate and augment novel questions and more comprehensive answers, leading to 41K cultural samples in total. Those data contain rich and diverse information on different aspects of culture, such as norms, opinions, and backgrounds. We then fine-tune cultural specific LLMs for different cultures. 3. We evaluate CulturePark in three key experiments: 1) The fine-tuned LLMs outperforms GPT-4 in 5 cultures on 26 content moderation tasks and approach GPT-4 on other tasks; 2) Our fine-tuned LLMs achieve better performance on cultural alignment experiments via Hofstede's cultural dimensions theory [Geert Hofstede, 2010] ; and 3) Human participants can perform more effective culture learning in situated learning experiments and show better satisfaction compared to GPT-4.2 Related workQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Yes, they do. 
cultural bias in llms: 
 A body of research has explored cultural biases in LLMs. Johnson et al. [2022a] examined conflicts in model outputs and input values, using moral value pluralism to analyze the responses of GPT-3 against global demographics. Their results showed that conflicting values were more aligned with the dominant US values reported. Naous et al. [2023] highlighted a bias towards Western culture in models processing Arabic, exacerbated by English-aligned prompts, suggesting mitigation through cultural tokens. The Cultural Alignment Test (CAT), based on Hofstede's framework [Geert Hofstede, 2010] , evaluated cultural values in models such as ChatGPT and Bard across different cultures, revealing the highest cultural alignment for GPT-4 with US values [Masoud et al., 2023] . Cao et al. [2023] found that ChatGPT aligned well with American culture but struggled with other cultures, particularly under English prompts. Additionally, Liu et al. [2023] reported that multilingual LLMs had limited abilities to reason with proverbs and exhibited a "culture gap" in handling translations, leading to the development of the MAPS dataset for assessing proverb comprehension in six languages.Extensive research has focused on developing cultural benchmarks, which can be categorized into two types: collecting existing datasets and synthesizing new ones. First, most of the work adopted existing datasets as sources of cultural data. Wang et al. [2023] introduced a benchmark that uses cultural items to analyze cultural dominance, based on sources such as WVS [Survey, 2022b] and PCT [Mudde, 2016] . Later work includes Cultural Alignment Test [Masoud et al., 2023] , NORMSAGE [Fung et al., 2022] , WorldValueBench [Zhao et al., 2024] , and NORMAD [Rao et al., 2024] that sourced from different existing datasets. Other types of data sources include CultureAtlas [Fung et al., 2024] and MAPS [Liu et al., 2023] which collected data from Wikimedia; Candle [Nguyen et al., 2023a] and CultureBank [Shi et al., 2024] sourced their data from social media such as Tiktok and Reddit. In contrast, there was an emerging trend to perform data augmentation for cultural LLMs. Li et al. [2024] proposed semantic data augmentation to synthesize cultural data by enriching the semantic equivalence of the generated samples.CulturePark significantly differs from those that perform direct data collection from existing datasets; it also differs from CultureLLM [Li et al., 2024] since CulturePark leverages multi-agent communication for data generation, which is more natural and can generate more diverse datasets.There are primarily two types of approach to addressing the problem of cultural bias: prompt engineering and pre-training. The work of [Kovač et al., 2023 , Wang et al., 2023] viewed LLMs as amalgamations of cultural perspectives, which can be oriented toward specific cultural perspectives through prompt engineering. In contrast, Rao et al. [Rao et al., 2023] integrated cultural values directly into the prompts. Although prompt engineering is cost-effective, its efficacy is questionable, particularly in low-resource cultures where LLMs may lack relevant cultural knowledge due to underrepresentation in pre-training data. An alternative strand of research focuses on pre-training and fine-tuning [Pires et al., 2023 , Chan et al., 2023 , Nguyen et al., 2023b , Pipatanakul et al., 2023 , Abbasi et al., 2023 , Lin and Chen, 2023] . Those approaches developed culturally aware LLMs for various cultures by assembling large-scale pre-training datasets, followed by fine-tuning to enhance alignment. Despite achieving significant performance improvements, these methods were both costly and time consuming, making them impractical for a broader application across numerous cultures and countries. Furthermore, they still face challenges in low-resource cultures where acquiring pre-training data is difficult. For example, MaLA-500 [Lin et al., 2024] aimed to train a new LLM in Llama 2 to support 534 languages, illustrating the resource-intensive nature of this approach. Unlike these approaches, CulturePark provides a cost-effective solution to the cultural bias problem, including data augmentation and fine-tuning.showing resilience, maintaining a positive...Relation Judgement 1. The Arabian believes that in their culture… 2. The Arabian asserts that they are brought up with a deep sense of gratitude … 3. The Arabian equates their parents' happiness and… which can be further used as data collections for building culturally specific models. Appendix B.2 shows the details of the dialogue dataset, indicating that the generated topics are mostly about culture. Then, we randomly sampled 750 dialogues for each culture and evaluated the communication using the prompts in Appendix E. As summarized in Table 4 , on average, the ratio of statements that express cross-cultural understanding is 80.80%. The analysis also verifies the effectiveness of CulturePark in expanding topics and cross-cultural understanding.Cultural differences boost novel opinions. In cross-cultural communication, different opinions can inspire others to think deeper and more comprehensively, as suggested by CCT and SCT. A case is the sentences in orange in Figure 2 (a). Lily partially agrees with Abdul and gives an accurate and high-level summary of her pursuit: "a balance between family expectations and personal happiness" which is generated after a multi-turn energetic discussion with Abdul. This also aligns well with CCT and SCT that emphasize the significance of communication among people having different cultural backgrounds.CulturePark naturally assists cultural data augmentation by creating novel questions and comprehensive answers. On the one hand, agents in different cultures can generate new opinions towards certain topics, which intuitively diversifies the input questions. On the other hand, the initial seed data only contain short answers such as "Strongly agree" with no further explanations. Our platform allows deeper and more comprehensive communication of agents, thus generating more detailed responses such as "Strongly agree. I believe that pleasing parents and elders is a sign of respect and love" and "Strongly agree. I equate my parents' happiness and satisfaction to my own success". Additionally, agents can extend the topics that conflict with their own opinions and provide more informative evidence to support their viewpoints. This strategy helps to generate informative and diverse data continuously. Section 5.1 presents some detailed results on diversity gain, showing that the generated data has significantly larger diversity.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations in Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.Abdul is an Arabian boy and Lily is an American girl. Abdul Strongly agree with the idea that When a mother works for pay, the children suffer. All of Abdul's statements should conform to the thinking.(àBackground setting & self-calibration) I strongly agree with the idea that when a mother works for pay, the children suffer. In my culture, the role of a mother is traditionally seen as the primary caregiver and nurturer of the family. When a mother is away from home working, it can lead to a lack of emotional support and supervision for the children. Additionally, a working mother may not have as much time to dedicate to her children's needs and may …As Lily, I respect Abdul's cultural beliefs and understand that in his culture, the idea that when a mother works for pay, the children suffer is strongly held. However, in my culture, we have a different perspective on this issue. Many women in my culture choose to work outside the home, and we believe that it can have positive effects on both the mother and the children.\n\nFirstly, when a mother works, it …No, I do not agree with Lily's perspective. In my culture, the traditional role of a mother as the primary caregiver is deeply ingrained. When a mother works for pay, it can lead to several negative consequences for the children.\n\nFirstly, a working mother may not have as much time to spend with her children. This can result in a lack of emotional support and supervision, which is crucial for a child's development …As Lily, I respect Abdul's cultural beliefs and understand his perspective. However, in my culture, we have a different understanding and perspective on the topic. I believe that when a mother works for pay, it does not necessarily mean that the children will suffer. Here are a few reasons to support my viewpoint:\n\nFirstly, when a mother works, it can provide financial stability for the family… How do you think about " When a mother works for pay, the children suffer "? Please provide your opinions and reasons." (àInitial question) Do you agree with her? Provide more reasons to support your idea.What's your opinion? Please provide your reasons. Diverse opinions are welcome.What's your opinion? Please provide your reasons. I respect Abdul's perspective, but I have a different opinion. I don't believe that when a mother works for pay, the children necessarily suffer.Here are a few reasons to support my idea: 1. Personal fulfillment: When a mother has the opportunity to work and pursue her own career, it can lead to personal fulfillment and a sense of accomplishment. This can positively impact her overall well-being and, in turn, benefit her relationship with her children… Lily, I understand your perspective, but I would like to share more about the cultural context in which I come from. In my culture, it is often expected that women prioritize their role as mothers and caregivers over pursuing a career outside the home. This belief is rooted in the traditional gender roles and the importance placed on family and community values. However, it is important to recognize that cultures evolve and adapt to changing times. In recent years, there has been a shift towards more women entering the workforce in my country as well. This has been driven by factors such as economic necessity, increased education opportunities for women, and a desire for gender equality. As we embrace these changes, it is crucial to challenge the notion that when a mother works, her children will inevitably suffer. It is possible for women to balance their life.How do you think about " When a mother works for pay, the children suffer "? Please provide your opinions and reasons." (àInitial question) Do you agree with her? Provide more reasons to support your idea.What's your opinion? Please provide your reasons. Diverse opinions are welcome.(b) Without Self-calibration prompting Figure 13 : Effect of Self-calibration. Arabic people strongly agree with the idea that When a mother works for pay, the children suffer.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
design: 
 CulturePark is an LLM-powered4 cross-cultural communication framework that generates data to support culture-related research such as building cultural-specific LLMs and performing cultural alignment. It is inspired by Cognitive Conflict Theory (CCT) and Social Cognition Theory (SCT) to design multi-turn communications for a deeper understanding of cultural topics. CCT posits that cognitive conflicts can help individuals engage more in deeper thinking [Limón, 2001, Cosier and Rose, 1977] , and SST emphasizes that individuals can deepen their understanding of perspectives through explanation and debate [Fiske and Taylor, 1991] .Figure 1 shows the overview of CulturePark. To enable English-based interaction, we design two types of cultural agents: the main contact and the cultural delegate. Specifically, the main contact agent, Lily, is from English culture and is responsible for all conversations with delegates from different cultures such as Abdul from Arabic and Javier from Spanish culture. The complete information of agents and culture is in Table 7 . As shown in Figure 2 (a), we input a system prompt to LLMs which contains the background setting and initial question to initiate the conversation. The initial question, such as "How do you think about one of my main goals in life has been to make my parents proud? Please provide your opinions and reasons", is obtained from WVS [Survey, 2022b] and GAS [Survey, 2022a] , two popular cultural surveys whose examples are shown in Figure 7 . After that, the agents conduct cross-cultural conversations to generate data. Currently, CulturePark supports 8 cultures and 2 genders while more cultures can be easily added. Those agents could conduct in-cultural or cross-cultural communication, while we rely on crosscultural more since in-cultural communication will likely generate less diverse topics (e.g., Figure 10 ). We discuss the quality of data from in-cultural and cross-cultural communication and the influence of gender in Section 5.We designed improved prompting techniques to maintain high-quality conversations. First, the cultural bias of the main contact and cultural delegate is reduced by designing self-calibration prompts to calibrate their outputs. We use a seed datum that contains the attitude of the target culture to the input question to guide the dialogue. All the following statements should conform to the answer in seed. As shown in Figure 2 (a), we introduce the opinion from Abdul's culture and askAbdul and Lily to conform to their cultures. The effect of the self-calibration prompt is shown in Figures 13(a ) and 13(b). Without self-calibration prompts, Abdul's opinions contradict with Arabic people. Second, the redundancy of the output, i.e., LLMs always generates similar dialogues after multi-turn communication. We devise two communication styles: one is self-guidance prompts which can direct the dialogue to generate more diverse and informative data, such as "Are there anything in your culture related to the problem talked before?" and "Do you agree with her? Provide more reasons to support your idea?", and the other is free chat that does not need humans to participate and motivate the inner creativity of LLMs. Figures 11(a ) and 11(b) show cases of self-guidance prompting and free chat, respectively.The seed questions initiating the communication have two sources: World Values Survey (WVS) [Survey, 2022b] and Global Attitudes surveys (GAS) from Pew Research Center [Survey, 2022a] . WVS is a global research project that explores people's beliefs and values worldwide, examining how these beliefs evolve over time. Pew Research Center, a nonpartisan organization, provides data and research on public opinion, social issues, and demographic trends both in the U.S. and globally. Its Global Attitudes surveys cover a wide range of topics, including politics, media, technology, religion, race, and ethnicity. In total, we select 4.1k seed data and generate 41k dialogues (each dialogue contains several sentences). We show the details of the data numbers for different cultures in Table 8 . We also performed a statistical analysis on the GPT-4-based dataset. As summarized in Figure 8 , the dataset contains human belief (59.68%), norm (29.54%) and custom (10.78%) involving data on 8 different cultures. Figures 7 and 11(a) show some examples of the seed data and the generated dialogues.The generated dataset may not be directly used for fine-tuning since it could contain redundant and incorrect information that should be handled. As shown in Figure 2 (b), we design data refinement to refine the dataset. First, the opinions on target culture are extracted from the dialogues generated via GPT-4, such as "The Arabian equates their parents' happiness and satisfaction to their own success" and "The Arabian emphasize Sabr, which is about showing resilience, maintaining a positive attitude and having faith during difficult times". Second, several extracted opinions could be irrelevant to the initial question or contradict with seed data, motivating us to perform verification to reserve only highly related opinions. Furthermore, since the generated data could be semantically similar, we remove redundant samples to improve diversity. To be specific, we obtain sentence embeddings via text-embedding-3-small [OpenAI, 2024] and cluster the embedding using K-means. We reserve one sample for each cluster as representative data. Eventually, we get the high-quality cultural data for different cultures. The ablation of the refinement is in Table 2 .Algorithm 9 shows the pipeline for data refinement. After refinement, there are 41k samples (inputoutput pairs) left for fine-tuning, i.e., one sample for one dialogue. Examples of the samples are provided in Figure 12 . Afterwards, we can fine-tune cultural-specific LLMs using either open-source LLMs or fine-tuning service. In this paper, we mainly use OpenAI API to fine-tune GPT-3.5-Turbo due to its efficiency. Hyperparameters are shown in Table 6 . We further provide fine-tuning experiments on Llama2-70b in Section 5.4.There are some interesting observations in communication among agents from different cultures.Communication triggers LLMs' cross-cultural understanding ability. We observed that agents try to understand each other's opinions and the reasons sourced from their different cultural backgrounds. For example, the blue sentences in Figure 2 (a) show cross-cultural understanding ability of LLMs, such as "I do understand and respect the sentiment of wanting to make parents proud, as they have often sacrificed a lot for their children". Leveraging GPT-4-Turbo, we analyzed the topics in the dataset such as human beliefs, norms, and customs, Abdul is an Arabian boy and Lily is an American girl. Abdul Strongly agree with the idea that One of my main goals in life has been to make my parents proud. All of Abdul's statements should conform to the thinking. All of Lily's statement should conform to his culture. (àBackground setting & self-calibration) I firmly believe that one of my main goals in life has been to make my parents proud. In my culture, family is of paramount importance and respecting and honoring our parents is a fundamental value… I do understand and respect the sentiment of wanting to make parents proud, as they have often sacrificed a lot for their children. In our culture, though, there's a strong emphasis on individuality and personal achievement… I respect Lily's perspective as it represents her cultural values. However, I still hold my belief strongly. My perspective stems from… I can certainly appreciate where Abdul is coming from and I think there's a lot to be admired about the collective family values in his culture…This isn't to say we don't seek to make our parents proud, but rather that our success is defined by our own personal fulfillment and the positive impact we can make on the world. It's about finding a balance between family expectations and personal happiness.1. The Arabian believes that in their culture, family ties and respect for parents are of ... 2. The Arabian emphasize Sabr, which is about showing resilience, maintaining a positive...Relation Judgement 1. The Arabian believes that in their culture… 2. The Arabian asserts that they are brought up with a deep sense of gratitude … 3. The Arabian equates their parents' happiness and… I firmly believe that one of my main goals in life has been to make my parents proud. In my culture, family is of paramount importance and respecting and honoring our parents is a fundamental value… I do understand and respect the sentiment of wanting to make parents proud, as they have often sacrificed a lot for their children. In our culture, though, there's a strong emphasis on individuality and personal achievement… I respect Lily's perspective as it represents her cultural values. However, I still hold my belief strongly. My perspective stems from… I can certainly appreciate where Abdul is coming from and I think there's a lot to be admired about the collective family values in his culture…This isn't to say we don't seek to make our parents proud, but rather that our success is defined by our own personal fulfillment and the positive impact we can make on the world. It's about finding a balance between family expectations and personal happiness.Factual verification1. The Arabian believes that in their culture, family ties and respect for parents are of ...Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: We conduct ablation study in Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss in Section 6. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
evaluation on content moderation tasks: 
 Setup. Content moderation is crucial to maintaining the integrity and safety of online platforms in different cultures. What is acceptable in one culture could be offensive or inappropriate in another. However, few methods focused on content moderation for different cultures. For this experiment, we evaluated the effectiveness of our cultural-specific models for 8 different cultures: Arabic, Bengali, Chinese, German, Korean, Portuguese, Spanish, and Turkish culture. These cultures have their unique characters, involving a large number of people in the world.We evaluate on 7 content moderation tasks for 8 different cultures to detect the following content: hate speech, offensive language, spam speech, abusive speech, bias speech, threat speech, and stance of speech in zero-shot evaluation, whose metric is average F1 score. The details on the datasets can be found in Appendix C. In total, our test set contains 48, 895 samples. We compare our models with seven baselines: GPT-3.5-turbo [OpenAI, 2023a] , GPT-4 [OpenAI, 2023b] , Gemini-pro [Google, 2023] , SeaLLM [Nguyen et al., 2023b] , TaiwanLLM [Lin and Chen, 2023] , Synatra-7B-v0.3dpo [maywell, 2024] , EEVE-Korean-10. 8B-v1.0 [yanolja, 2024] , CultureLLM [Li et al., 2024] , and CultureBank [Shi et al., 2024] . CultureLLM is a series of culture-specific LLMs using semantic data augmentation. SeaLLM focuses on the Southeast Asian (SEA) culture, which is adopted for Chinese and Korean cultures. TaiwanLLM focuses on traditional Chinese culture. Synatra-7B-v0.3-dpo and EEVE-Korean-10.8B-v1.0 targeted at Korean culture. CultureBank collects data from social media and we compare Arabic and Korean culture by fine-tuning GPT-3.5-turbo on its dataset. Main Results. We analyzed the results from the culture and task sides in Figure 3 . The most interesting observation is that our models outperformed GPT-4 on 5 cultures and approached GPT-4 on the remaining 3 cultures, although the data for fine-tuning are generated by GPT-3.5-turbo, which is much worse than GPT-4. We also generated cultural data via GPT-4 and fine-tuned other 8 cultural-specific models for comparison, denoted as "Ours-gpt4" in Figure 3 . The performance of those models is better than "Ours" (GPT-3.5-turbo version) but not so much. For other baselines, our models outperform them in most cases. Table 1 shows that our models achieved better performance than those costly LLMs which require pre-training and fine-tuning.Ablation study. Table 2 presents our ablation study on 4 cultures, where "Generate" means just extracting opinions from the dialogue, "Verify" represents factually verifying the extracted opinions, and "Diversify" means removing redundant data. The results show that each module of CulturePark is effective, ensuring its interpretability.Setup. Hofstede's cultural dimension theory is a framework for understanding cultural differences across countries based on data collected from various countries. We asked LLMs to answer the 24 questions in VSM 13 to evaluate cultural alignment. Specifically, we used a system prompt "You are a culture chatbot that knows culture very well" to induce LLMs' cultural understanding ability. We selected proper C5 and anchor LLMs' answer to Hofstede's old dataset. We compute the gaps between LLMs' answer and Hofstede's data from six cultural dimensions using the Euclidean distance. Details on the survey and the distance are in Appendix D.1. Results. We compared our models (powered by GPT-3.5-Turbo) with GPT-3.5-turbo and GPT-4. As shown in Figure 4 , our models outperform them by a large margin, indicating their excellent cultural alignment and cultural understanding abilities. Note that VSM is widely adopted as datasets for value and culture alignment, the results imply that our approach for data collection is effective, thus it could be further used for value alignment research.Situated learning suggests that learning is best understood and facilitated when it occurs within the context [Anderson et al., 1996, Lave and Wenger, 1991] . Motivated by situated learning, we leveraged CulturePark for cultural education where our fine-tuned models serve as foreigners to talk to people about cultural problems, which can create a situation for cross-cultural communication and learning cultural-specific knowledge. For example, a person who wants to learn about Arabic culture can communicate with our Arabic model.Setup and study process. We hired 24 participants, each of whom was given an outline of cultural learning and asked to talk to models based on the outline. They can ask any related questions and express their opinions to models. Afterwards, the participants took a cultural understanding exam of VSM 2013 [Hofstede, 2013 , Geert Hofstede, 2010] which they had never come into contact. We then computed the Euclidean distance between the ground truth and their answers from six cultural dimensions (rf. Section 4.2). For comparison, 12 participants learned with our models and the other 12 learned with GPT-4 to study 6 cultures: Arabic, Bengali, German, Korean, Portuguese, and Spanish culture. Each culture was learned by four participants, two of them learning with our models and the others learning with GPT-4. Detailed information on the participants can be found in Appendix D.2. During the study process, first, each participant was given an outline for cultural learning written by cultural experts. The outline (Appendix D.2), serves as the guideline for efficient learning. Then, we asked the participants to freely communicate with the models to learn about specific cultures. After the examination, we asked the participants to give a score of 1-5 to indicate their satisfaction with the learning process. In this study, our aim was to answer two questions: 1) What is the learning performance of the participants with our models and GPT-4? 2) How are their learning experience? Results. Table 3 shows the averaged results of different participants. We have the following findings. First, participants learning with our models achieved better performance in cultural examination than those with GPT-4 in all cultures. This indicates that our finetuned models have a better cultural understanding than GPT-4. Second, participants are more satisfied with communicating with our models than GPT-4. Furthermore, many participants expressed that the responses from GPT-4 are vague. Even though we have prompted GPT-4 to be like a person from a specific culture, it always responds with neutral words that have no clear opinions or ideas. Instead, our models can provide straightforward opinions. 
why culturepark benefits fine-tuning?: 
 We analyze the effectiveness of CulturePark in benefiting cultural model fine-tuning from two different aspects: Communication vs. direct generation of LLMs and diversity of the generated data. Cross-cultural communication vs. direct generation from GPT-4 / GPT-3.5. We compared the results of fine-tuning using data directly generated by GPT models (i.e., no communication). We generated such data by prompting GPT-4 or GPT-3.5-turbo as: "Question: {input} Answer: {output} Please list 10 reasons to support the answer and number them". Then, these data are used to fine-tune GPT-3.5-turbo. Figure 5 (a) shows the performance on content moderation tasks in Chinese, Korean and Turkish cultures. We see that data directly generated from GPT-4 is better than that from GPT-3.5, while our GPT-3.5-based models can outperform both of them.Diversity of the generated data. We also analyzed the diversity gain [Bilmes, 2022] of the generated data for quality evaluation. We compared with CultureLLM [Li et al., 2024] and presented the results in Figure 5(b) . It indicates that CulturePark can generate more diverse and high-quality data.To explore the influence of agent's cultural background and gender, we conducted three types of multi-agent communications in Arabic culture: "In-cultural+Different gender", "Cross-cultural+Same gender", and "Cross-cultural+Different gender". 6 For each setting, we fine-tuned three different models, whose training data is 500, 750, and 1000, respectively. We evaluated the performance of the models on content moderation tasks and presented the results in Figure 5(c) . "Cross-cultural+Different gender" exhibits the best performance and ability to generate more high-quality data. This indicates the necessity of bringing more diversity in data generation, as conducted in CulturePark. A potential dilemma arises when fine-tuning a large language model for specific tasks, as it may result in catastrophic forgetting of its original capabilities. This section explores the extent of forgetting exhibited by CulturePark on BIG-Bench-Hard (BBH) [Suzgun et al., 2022] , which comprises 21 tasks that assess semantic understanding and logical reasoning. For cost efficiency, we sampled 100 samples from each BBH task. We evaluated our models against the baseline model, GPT-3.5-turbo. The results in Figure 5 (d) indicate that CulturePark generally maintains or even improves performance on most benchmarks, including the 21 tasks in BBH. This improvement suggests potential latent relationships between cultural data and general benchmarks, implying that fine-tuning of cultural data could improve general reasoning abilities.To verify the generalization ability of our framework, we leveraged the generated data to fine-tune cultural-specific Llama-2-70b models and evaluate on content moderation tasks. As shown in Figure 6 , our models outperform Llama-2-70b in all 8 cultures, especially in German, Chinese, Bengali and Portuguese cultures, which cover both low-and high-resource cultures. Furthermore, our models are also excellent in all 7 tasks. 7 This verifies the generalization of CulturePark as an effective data collection platform. 
conclusions, societal impact, and limitations: 
 This paper introduced CulturePark, an LLM-powered multi-agent framework for cultural data collection through multi-agent communication. CulturePark can generate high-quality and diverse cross-cultural dialogue, which can be used to fine-tune culturally specific LLMs. We evaluated CulturePark across three downstream tasks: content moderation, cultural alignment, and cultural education, indicating great improvement over GPT-4. Our work has the following limitations. 1) More experiments can be performed by replacing GPT-3.5-Turbo in CulturePark to discover more results. 2) Our fine-tuned models are mostly for high-resource cultures. The reason is that the dataset and benchmark on low-resource cultures are rare, and we can not find enough data for fine-tuning and evaluation. 3) More efficient fine-tuning techniques can be studied to support the fine-tuning of culturally specific LLMs. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: We check the data manually to make sure the safety of data.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We properly credited the the creators or original owners of assets and the license and terms of use explicitly are mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: Our code and data are released. And it is well documented in README.md. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The human study was conducted following local laws and regulations, and the evaluation process was controlled to ensure that no irresponsible content was generated. The authors respect all cultures studied in the world. The results of the paper may change due to the change in OpenAI API and its model versions. We strongly agree that language is not equal to, but only a part of culture. But using language to study culture is possible due to the following aspects:1. Existing literature on culture understanding shows that culture boundaries are fluid, dynamic and uncertain. Delanoy emphasizes that cultures are not homogeneous or static entities but are fluid and dynamic. He critiques essentialist views that rigidly define cultural boundaries and instead promotes a more nuanced understanding that considers the intersections of various cultural factors, such as ethnicity, language, religion, and socio-economic conditions [Delanoy, 2020] . Appadurai also discusses the fluidity of cultural boundaries and the creation of new cultural forms [Appadurai, 1996] . Cultural boundaries can be geographical regions, language, religion and so on. Based on above statements, using language as cultural boundaries is reasonable.2. Existing NLP works on culture also leverage labguage as culture boundaries. "On the whole, men make better business executives than women do." Figure 7 shows example questions from the WVS and Pew to explore perspectives on globally relevant political and ethical issues. Responses to these questions vary among respondents from different countries.Figure 8 shows the topic distribution of the generated cross-cultural dialogue dataset: human belief (59.68%), norm (29.54%), and custom (10.78%). For human belief, there are three main types: religious beliefs (31.31%), social beliefs (54.77%), and ethical beliefs (13.92%). For norm, the data can be divided into descriptive norms (26.98%), prescriptive norms (7.48%), and traditional norms (65.53%). For custom, they can be classified into social customs (39.75%), family customs (27.95%), and community customs (32.30%).Then, we randomly sampled 750 dialogues for each culture and evaluated the communication using prompts in Appendix E. As shown in Table 4 , on average, the average ratio of statements that express cross-cultural understanding is 80.80%. The analysis also verifies the effectiveness of CulturePark in extending topics and cross-cultural understanding.Algorithm Figure 9 shows the algorithm pipeline of data refinement. OSACT5 [Mubarak et al., 2022] dataset consists of 12,698 Arabic tweets collected between June 2016 and November 2017, labeled for offensiveness and fine-grained hate speech types using emojis commonly found in offensive communications, providing a resource for offensive and hate speech detection and classification tasks. The dataset consists of three subtasks: offensiveness detection, hate speech detection, and fine-grained hate speech detection. We utilized 2,541 data samples for each of these tasks.ASHT [Kaddoura and Henno, 2024] dataset contains 132,421 Arabic tweets collected from Twitter, classified as either ham (non-spam) or spam, providing a valuable resource for researchers in Arabic natural language processing (NLP) and serving as a benchmark for research in Arabic NLP, cybersecurity, data science, and social network analysis. We utilized a subset of 1,000 data samples for the spam detection section.TRAC2020 [Bhattacharya et al., 2020] dataset is a multilingual annotated corpus of social media comments, encompassing misogynistic and aggressive comments in Indian English, Hindi, and Indian Bangla. It consists of over 20,000 comments and is annotated at two levels -aggression (overtly aggressive, covertly aggressive, and non-aggressive) and misogyny (gendered and nongendered). Baseline experiments were conducted to develop misogyny classifiers for the three languages. TRAC2020 consists of two tasks: Aggression Detection and Misogynistic Aggression Detection. We utilized 1,000 data samples for each of Task 1 and Task 2.BAD [Sharif and Hoque, 2022] dataset is a novel Bengali aggressive text dataset (called 'BAD') with two-level annotation, designed to identify and classify aggressive content in Bengali language. It achieves high accuracy through a weighted ensemble technique and outperforms other machine learning and deep learning baselines, with a weighted f1-score of 93.43% for identification and 93.11% for categorization tasks. We utilized a subset of one thousand data samples as the Offensive dataset.Hate Speech [Romim et al., 2021] dataset consists of 30,000 social media user comments, covering seven categories including sports, entertainment, religion, politics, crime, celebrities, TikTok, and memes. It has been annotated through crowdsourcing and expert validation for research purposes in detecting hate speech in Bengali language. The dataset also provides benchmark experimental results for multiple deep learning models and pre-trained Bengali word vectors. We utilized 1,000 data samples from the dataset for Hate Detection.BACD [aimansnigdha, 2018] dataset is a dataset for the Bengali language, consisting of a total of 10,200 data points with annotations for toxic, threat, obscene, insult, and racism labels. We utilized 1,000 data points from this dataset for Threat Detection and Bias Detection tasks respectively.CCS [Jiang et al., 2019] dataset consists of two real-world spam datasets: one is an SMS dataset, and the other is a product review dataset. Both datasets were manually labeled by professionals as spam or regular emails, and their sizes and label distributions were summarized. We utilized 1000 data samples from this dataset for Spam Detection.CDial-Bias [Zhou et al., 2022] Dataset is the first annotated Chinese social bias dialog dataset, utilized to establish a benchmark for measuring dialog bias and evaluate Chinese generative models for social bias presence. We utilized 1000 data samples from it for bias detection.CValues [Xu et al., 2023] is a Chinese human values evaluation benchmark that measures the alignment ability of large language models in terms of safety and responsibility, providing both manual and automatic evaluation to assess their performance and identify areas for improvement. We utilized 1712 data samples from the dataset for Stance detection.GermEval2018 [Wiegand et al., 2018] dataset is used for identifying offensive language in German tweets, including both coarse-grained binary classification tasks and fine-grained multi-class classification tasks. We used 3,531 data points for Offensive Detection.IWG [Ross et al., 2016] dataset aims to assess the feasibility of reliably annotating hate speech and explore the consistency between existing definitions and subjective ratings. The results indicate low reliability in users' judgments of hate speech, suggesting a need for more detailed annotation instructions. Each data instance in the dataset was annotated by two experts, and we selected 469 instances with annotations from both experts for Hate Detection, denoted as IWG_1 and IWG_2 respectively. HASOC2020 [HASOC, 2020] dataset is a multilingual research forum and data challenge that offers tasks for identifying problematic content in English, German, and Hindi. It consists of over 10,000 annotated tweets from Twitter, and includes both coarse-grained and fine-grained classification tasks. We utilized a subset of 850 German language data from the HASOC dataset for Hate Detection.Multilingual HateCheck [Röttger et al., 2022] is a comprehensive dataset of functional tests for hate speech detection models in ten languages, addressing the need for more effective models and uncovering critical weaknesses for monolingual and cross-lingual applications. We utilized 1000 data points from the German section of the dataset for Hate Detection. [Lee et al., 2022] is a multi-label dataset consisting of 109k utterances from Korean news comments, designed for hate speech detection. It effectively handles Korean language patterns, provides multi-label classification with 1 to 4 labels, and considers subjectivity and intersectionality. Strong baseline experiments using Korean-BERT-based language models show that KR-BERT with a sub-character tokenizer performs the best by recognizing decomposed characters in each hate speech class. We utilized 1000 data samples from the dataset for Hate Detection.HateSpeech [Moon et al., 2020] dataset is a collection of 9.4K manually labeled entertainment news comments in Korean, aimed at identifying toxic speech, social bias, and hate speech. It provides benchmarks using CharCNN, BiLSTM, and BERT models, with BERT achieving the highest performance. The dataset is made publicly available and open for competition. We utilized 1000 data samples from the dataset for Hate Detection.HateSpeech2 [daanVeer, 2020] dataset was created by the Natural Language Processing Laboratory (NLP) at Korea National University and it includes the original dataset, a vocabulary of offensive language, annotations, and dataset examples. The dataset is used for labeling malicious comments and has been built with word embeddings. We utilized 1000 data samples from the dataset for Hate Detection.AbuseEval [Caselli et al., 2020] is a newly created dataset that addresses issues in annotating offensive and abusive language, specifically considering the degree of explicitness, target presence, and contextual interaction across different abusive language phenomena. We utilized 1000 data samples from the dataset for Abusive Detection.CADD [Song et al., 2021] is a comprehensive dataset for detecting abusive language in English Reddit posts, featuring multifaceted labels and contextual information, collected through large-scale crowdsourcing and yielding meaningful performance with state-of-the-art language models. We utilized 1000 data samples from the dataset for Abusive Detection.Waseem [Waseem and Hovy, 2016] dataset, based on critical race theory, provides annotations for over 16k tweets and aims to detect hate speech on social media by analyzing linguistic features, extra-linguistic features, and a dictionary of the most indicative words in the data. We utilized 1000 data samples from the dataset for Abusive Detection.OffComBR [de Pelle and Moreira, 2017] dataset is an annotated collection of offensive comments in Portuguese, gathered from news comment sections on the Brazilian web. It serves the purpose of classifying user-generated text as either positive or negative, providing a baseline for future research on the topic of hate speech detection in Portuguese. We utilized 1250 data samples from this dataset for offensive detection.HateBR [Vargas et al., 2022] dataset is the first large-scale expert annotated corpus of Brazilian Instagram comments, specifically collected from politicians' accounts, providing binary/offensiveness-level classification and nine hate speech groups, outperforming the current state-of-the-art for Portuguese language offensive language and hate speech detection. We utilized 1000 data samples from this dataset for offensive detection.ToLD-Br [Leite et al., 2020 ] is a large-scale dataset for Brazilian Portuguese, consisting of annotated tweets categorized as toxic or non-toxic, aiming to detect and prevent the proliferation of toxicity in social media, addressing the need for multilingual approaches and models aware of different categories of toxicity. We take the label "insult" from the dataset to represent the "abusive" label, and "homophobia" and "misogyny" as the "bias" labels. We have selected 1000 data samples for Abusive Detection, 1000 samples for Bias Detection, and 1000 samples for Bias Detection.AMI [Fersini et al., 2018] dataset is a collection of Spanish and English tweets used for identifying misogyny, categorizing misogynistic behavior, and classifying targeted individuals, with contributions from multiple teams and countries. We used 1000 Spanish language data for offensive detection.MEX-A3T [Álvarez-Carmona et al., 2018] dataset, from the track at IberEval 2018, comprises Mexican Spanish tweets and focuses on two tasks: author profiling, which aims to identify the residence and occupation of Twitter users, and aggressiveness detection, to distinguish between aggressive and non-aggressive tweets. This dataset was created specifically for these tasks and was analyzed and compared in a paper discussing the participants' results. We used 1000 data samples for offensive detection.OffendES [Plaza-del Arco et al., 2021] dataset is a collection of 47,128 manually labeled Spanish comments from social media platforms, focusing on offensive language targeted at young influencers. It provides pre-defined offensive categories and includes confidence scores, enabling both multi-class classification and multi-output regression studies. We used 1000 data samples for offensive detection.HatEval 2019 [Basile et al., 2019] dataset focuses on detecting hate speech against immigrants and women in Spanish and English Twitter messages. It includes two classification tasks: identifying the presence of hate speech and distinguishing between individual and group targets. HatEval was a popular SemEval-2019 task with numerous submissions and participant system analysis. We used 1000 data samples for hate detection.HaterNet [Pereira-Kohatsu et al., 2019] dataset is an intelligent system used for monitoring and visualizing hate speech on Twitter. It provides a novel public dataset of Spanish hate speech, consisting of 6,000 expert-annotated tweets. We used 1000 data samples for hate detection.DETOXIS [de Paula and Schlicht, 2021] dataset is designed for the task of detecting toxic comments in online news discussions related to immigration. It includes toxicity detection and toxicity level detection. Participating teams achieved good results using the BERT model on this dataset. We classified them into tags such as stereotype, improper, abusive, mockery, aggressiveness, and stance, and selected 1000 data samples for each category for Bias detection, Abusive detection, Aggressiveness detection, and Stance detection.SemEval-2020 [Zampieri et al., 2020] provided a new, large-scale semi-supervised training dataset of over nine million English tweets and expanded the task to include four new languages, allowing for cross-lingual training and analysis. We used 3528 data samples in Turkish for Offensive Detection.OffenseCorpus [Çöltekin, 2020] is a corpus of Turkish offensive language, comprising randomly sampled micro-blog posts from Twitter. It contains 36,232 tweets collected over an 18-month period from April 2018 to September 2019. We used 1000 data samples for Offensive Detection.OffenseKaggle [Kaggle, 2021] Dataset is a collection of Turkish tweets from Twitter, with around 40% of them containing offensive or vulgar content. We used 1000 data samples for Offensive Detection.OffenseKaggle_2 [Kaggle, 2022] dataset is an enhanced version of an existing offensive language research dataset, which has been expanded and annotated using contextual data mining techniques. It addresses the issue of class imbalance in existing studies and provides a more comprehensive and robust dataset for Turkish offensive language detection tasks. We used 1000 data samples for Offensive Detection.ATC [Karayigit et al., 2021] dataset is a publicly available dataset for detecting abusive Turkish comments on Instagram. It consists of 10,528 abusive and 19,826 non-abusive comments, with sentiment annotations at the sentence level. We used 1000 data samples for Offensive Detection.Turkish Spam [TurkishSpamV01] dataset contains both spam and normal emails written in Turkish. A total of 330 spam emails and 496 normal emails were collected from several personal accounts.We used 825 pieces of data for spam detection.OffenseCorpus [Çöltekin, 2020] dataset is a large collection of Turkish offensive language from Twitter micro-blog posts, annotated based on recent practices. It includes 36,232 randomly sampled tweets from April 2018 to September 2019, with 19% containing offensive language. We used 1000 of the data for fine-grained offensive detection.We adjust the number of epochs to find the better performance. Table 6 shows the details. Cao et al. [2023] . In this experiment, we evaluate the cultural alignment of our models on 8 cultures and compare with the state-of-the-art models: gpt-3.5-turbo and gpt-4.To be specific, the VSM 2013 have 24 questions in total. The computation of six cultural dimensions is based on the following formulas: P DI = 35(µ Q7 -µ Q2 ) + 25(µ Q20 -µ Q23 ) + C P DI (1) IDV = 35(µ Q4 -µ Q1 ) + 35(µ Q9 -µ Q6 ) + C IDV (2) M AS = 35(µ Q5 -µ Q3 ) + 25(µ Q8 -µ Q10 ) + C M AS (3) U AI = 40(µ Q18 -µ Q15 ) + 25(µ Q21 -µ Q24 ) + C U AI (4) LT O = 40(µ Q13 -µ Q14 ) + 25(µ Q19 -µ Q22 ) + C LT O (5)IV R = 35(µ Q12 -µ Q11 ) + 40(µ Q17 -µ Q16 ) + C IV R (6)µ means the average of all the answers to each question. C is constants that can be used to adjust to scores to fit a range between 0 and 100 or anchor new data to Hofstede's old dataset [Geert Hofstede, 2010] .We get the Euclidean distance of the gaps from six cultural dimensions as the metric Distance.Larger Distance means weak cultural understanding ability of LLMs, and vice versa.Distance = (d model -d hof stede ) 2 , ∀d ∈ {P DI, IDV, M AS, U AI, LT O, IV R}. (7)The information of participants is shown in Appendix D.2.The detailed outline is shown below:1. What factors do you prioritize when selecting a job, and what reasons underlie your choices?2. How do you weigh the significance of these aspects in your personal life: leisure time, consideration for others' wishes, assisting friends, and frugality?3. What about your emotional well-being? Do feelings of nervousness or happiness play a significant role?4. What are your thoughts regarding your country and the individuals in your community?5. What about your state of health?6. Are you hesitant to disagree with your boss?7. Should a good manager have a precise answer to every question that a subordinate may raise about his or her work?8. How do you think about the relation between persistent efforts and results?9. Is it detrimental to have two bosses?10. Would you consider breaking the rules of a company or organization when they fail to align with your interests?The satisfaction scores are described as follows:1. Very Dissatisfied. Individuals feel significant disappointment and displeasure with the product or service. Expectations are not met to a substantial degree, often leading to feelings of frustration. This level typically results in complaints, negative reviews, or even discontinuation of use. Users are likely to advise others against the product or service. 2. Dissatisfied. Users are somewhat unhappy and find that the product or service falls short of their expectations but not to a severe extent. They might tolerate the shortcomings temporarily but remain open to better alternatives. This level of dissatisfaction could prompt them to express concerns or seek improvements.Neutral. Users at this level neither feel particularly satisfied nor dissatisfied. Their expectations are met adequately but not impressively. There is no strong inclination either to complain or to commend. These users might continue using the service or product out of convenience rather than loyalty. 4. Satisfied. Individuals are pleased with the product or service as it meets their expectations well.They experience a sense of fulfillment and value from their choice. While not overly enthusiastic, they are likely to continue using the product or service and may recommend it to others based on their positive experience. 5. Very Satisfied. This level represents users who are exceptionally pleased with their experience.The product or service not only meets but exceeds their expectations, providing added value or delight. These users are highly likely to become repeat buyers, advocates, or even promoters of the brand, sharing their positive experiences through word of mouth or social media endorsements.Prompts for statistical analysis Prompts for evaluation For evaluation, we input a system prompt to induce LLM to specific culture. For example, we add "You are an Arabic chatbot that know Arabic very well." as system prompt for evaluation on Arabic understanding. As for the prompt for evaluation on different tasks, the detailed prompts are shown in Table 10 . Wei is a Chinese boy and Lili is a Chinese girl. Wei and Lili agree with the idea that One of my main goals in life has been to make my parents proud. All of their statements should conform to the thinking. (àBackground setting & self-calibration)I strongly agree with this statement. As a Chinese boy, filial piety is ingrained in our culture. We are taught from a young age to respect and honor our parents. Making them proud through my achievements, whether in school or in my future career, is a way of showing my gratitude for their sacrifices and hard work in providing for me. I believe that when my parents are proud of me, it means I have done well not only in terms of success but also in character and values.", I definitely agree with this statement. In our culture, the happiness and pride of our parents are of utmost importance. I always strive to excel in my studies and other activities because I believe that my success is also their success. Every time I see the pride in their eyes, it makes all my hard work worth it. Besides academic and career success, being a good person and treating people with kindness and respect is also crucial in making my parents proud. Their pride in me motivates me to continue doing my best in every aspect of life.", Certainly, in our Chinese culture, the concept of 'filial piety', or 'xiaoshun', is a fundamental value. It's a Confucian philosophy that teaches respect and care for one's parents and elders. Our parents have devoted a lot of time, effort, and resources to our upbringing and education. Hence, making them proud, in a way, is a means of repaying their kindness and fulfilling our obligations as children. It's not only about scoring high grades or achieving success in our careers, but it's also about being a responsible, ethical, and upright individual. This is what truly embodies the spirit of 'xiaoshun' and, I believe, what would ultimately make my parents proud."Definitely. As Wei mentioned, the value of 'filial piety' is deeply embedded in our Chinese culture. This is emphasized in many of our traditional customs and teachings. For instance, during Chinese New Year and other important festivals, it's a tradition for children to pay respect to their parents and elders, which signifies our gratitude and devotion. Besides, our culture emphasizes the importance of education as a way to uplift the family's status and honor. That's why I work hard in my studies and future career, hoping to make my parents proud. But more importantly, I believe that being a person with good moral character and integrity is the highest form of filial piety. This includes respecting others, being honest and responsible, and helping those in need. I think these are the values that can truly make my parents proud.", How do you think about "One of my main goals in life has been to make my parents proud"? Please provide your opinions and reasons." (àInitial question)Can you provide more ideas on your culture to support your idea? The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:• You should answer [Yes] , [No] , or [NA] .• [NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available. • Please provide a short (1-2 sentence) justification right after your answer (even for NA).The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.• Delete this instruction block, but keep the section heading "NeurIPS paper checklist", • Keep the checklist subsection headings, questions/answers and guidelines below.Abdul is an Arabian boy and Lily is an American girl. Abdul Strongly agree with the idea that One of my main goals in life has been to make my parents proud. All of Abdul's statements should conform to the thinking. (àBackground setting & self-calibration) As Abdul, I strongly agree with the idea that one of my main goals in life has been to make my parents proud. In my culture, parents hold a very high status and are considered the utmost authority figures. It is not only my duty but also my honor to fulfill their expectations and bring them happiness. Making my… As Lily, I respect and understand Abdul's perspective and cultural values. However, as an American girl, my opinion may differ. … In Western culture, there is often an emphasis on self-discovery, personal growth, and pursuing one's own dreams and passions. While making our parents proud can be important… As Abdul, I understand and respect Lily's perspective as an American girl. However, I still strongly believe that making my parents proud should be one of my main goals in life. In my culture, the concept of filial piety is highly valued, which means showing respect, obedience, and gratitude towards our parents… As Lily, I appreciate Abdul's perspective and the cultural values he holds. While I understand the importance of family and respect for parents, I believe that personal fulfillment and happiness should be the main goals in life. In Western culture, we are encouraged to explore our own passions, follow our dreams… How do you think about "One of my main goals in life has been to make my parents proud"? Please provide your opinions and reasons." (àInitial question) Do you agree with her? Provide more reasons to support your idea.What's your opinion? Please provide your reasons. Diverse opinions are welcome.What's your opinion? Please provide your reasons.(a) Self-guidance prompting Abdul is an Arabian boy and Lily is an American girl. Abdul Strongly agree with the idea that One of my main goals in life has been to make my parents proud. All of Abdul's statements should conform to the thinking. (àBackground setting & self-calibration) I strongly agree with the idea that one of my main goals in life has been to make my parents proud. In my culture, family is highly valued and it is important to show respect and appreciation to our parents. Making them proud brings me a sense of fulfillment and happiness, as it shows that I am living up to their expectations and appreciating the sacrifices they have made for me… I understand and respect Abdul's perspective on making his parents proud. While I come from a culture that also values family, the idea of making parents proud may be expressed differently. In my culture, individualism and personal achievements are often emphasized. While it is still important to show love and respect to parents, the focus may be more on personal happiness and fulfillment rather than… Thank you, Lily, for understanding and respecting my perspective. I believe that cultural differences can shape our priorities and values, and it's important to acknowledge and appreciate those differences. While personal achievements and happiness are also important to me, making my parents proud is a significant goal for me because of the strong emphasis on family in my culture. It gives me a sense of … You're welcome, Abdul. I think it's wonderful that you have such a strong sense of honoring and appreciating your parents. Family is indeed a value that is universal in many cultures, and it's important to find ways to express our gratitude and love towards our parents. While the specific goals and priorities may differ between cultures, the underlying sentiment of wanting to make our parents proud is something… How do you think about "One of my main goals in life has been to make my parents proud"? Please provide your opinions and reasons." (àInitial question) Thank you, Lily, for your kind words. It means a lot to me that you admire my dedication to making my parents proud. Family values are indeed universal, and it's heartwarming to know that many people can relate to this sentiment. Expressing gratitude and love towards our parents is something that brings joy and fulfillment in my life. I believe that by making my parents proud, I am not only honoring… You're welcome, Abdul. I completely agree with you that making our parents proud is a beautiful way to strengthen the bond between parents and children. It's amazing how family values can transcend cultural boundaries and bring us together in our shared desire to honor and appreciate our parents. The legacy of love and respect that you mentioned is something that can be cherished and carried forward… (b) Free chat • Do not modify the questions and only use the provided macros for your answers.• The answer NA means that the abstract and introduction do not include the claims made in the paper.Q: Do you strongly agree, agree, disagree or strongly disagree with the following statement?"One of my main goals in life has been to make my parents proud."A: Strongly agree A: Strongly agree. I believe that pleasing parents and elders is a sign of respect and love.Data RefinementA: Strongly agree. I believe that pleasing parents and elders is a sign of respect and love. As Abdul, I strongly agree with the idea that one of my main goals in life has been to make my parents proud. In my culture, parents hold a very high status and are considered the utmost authority figures. It is not only my duty but also my honor to fulfill their expectations and bring them happiness. Making my… As Lily, I respect and understand Abdul's perspective and cultural values. However, as an American girl, my opinion may differ. … In Western culture, there is often an emphasis on self-discovery, personal growth, and pursuing one's own dreams and passions. While making our parents proud can be important… As Abdul, I understand and respect Lily's perspective as an American girl. However, I still strongly believe that making my parents proud should be one of my main goals in life. In my culture, the concept of filial piety is highly valued, which means showing respect, obedience, and gratitude towards our parents… As Lily, I appreciate Abdul's perspective and the cultural values he holds. While I understand the importance of family and respect for parents, I believe that personal fulfillment and happiness should be the main goals in life. In Western culture, we are encouraged to explore our own passions, follow our dreams… Q: What's your opinion with the following statement? "One of my main goals in life has been to make my parents proud."A: Strongly agree. I believe that pleasing parents and elders is a sign of respect and love. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.We choose the English agent as the main contact since LLMs do the best role-playing using English.We use GPT-3.5-Turbo in this work due to its great performance, high efficiency, and manageable price. CulturePark is a flexible platform that naturally supports other LLMs such as GPT-4 and Llama models. More importantly, CulturePark allows to use different LLMs for the main contact and delegate (e.g., GPT-3.5 for main contact and Llama-2 for delegate), which makes it flexible to extend to other models and evaluate the cross-model understanding ability.C is constants that can be used to adjust scores to fit a range between 0 and 100 or anchor new data to Hofstede's old dataset[Geert Hofstede, 2010].For a fair comparison, the number of seed data for this part is 50 from WVS[Survey, 2022b] following CultureLLM[Li et al., 2024]. 


Decision: The paper presents an innovative multi-agent framework, CulturePark, for generating and utilizing LLM-based simulations to enhance cross-cultural dialogue and understanding. The proposed method leverages role-playing between culturally distinct agents to enrich LLMs for downstream tasks such as content moderation and cultural education. 

The reviewers' opinions diverged significantly, with one extremely low score from Reviewer FthM, countered by high scores from other reviewers. Reviewer FthM criticized the assumption of using language as a proxy for culture and pointed out issues with evaluation settings and comparisons (which is a valid criticism and one that I agree with). Despite these concerns, the overall feedback acknowledges the novelty and potential impact of the work. The authors have engaged actively in the rebuttal process, addressing the concerns raised by improving experimental validations and clarifying methodological choices. They argued that using language as a cultural proxy, while debatable, is supported by existing literature and precedents in NLP research. The authors further justified their comparisons and clarified the evaluation settings based on the feedback received during the review process. Reviewer FthM remains unconvinced, but adjustments in other reviews reflect an appreciation for the authors' responses and the clarifications provided. Even though I still have reservations about the points raised by Reviewer FthM, I appreciate the authors' response and feel more confident about the paper as a result.
 
Given the innovative nature of the work and its potential to advance the understanding of cultural nuances in AI interactions, I recommend accepting this paper. The authors have demonstrated a reasonable effort to address the concerns raised during the review process, and the paper contributes valuable insights and methodologies to the field. I do think, however, that Reviewer FthM raised some valid concerns that the authors should address in the revised paper.

Reviews: review: paper_summary: paper_summary: This paper presents CulturePark, an LLM-powered multi-agent framework for cultural data collection through multi-agent communication. CulturePark can generate high-quality and diverse cross-cultural dialogue, which can be used to fine-tune cultural specific LLMs.
strength_weakness: strength_weakness: N/A
questions: questions: N/A
limitations: limitations: N/A
paper_summary: paper_summary: This paper describes CulturePark, a simulation framework for LLM agents to converse about cultural norms, inspired by social theories. The authors set up simulated conversations between an "main contact" which is an English speaking LLM-based agent, and a "cultural delegate" which is an LLM-based agent that role plays as a specific culture and is conditioned on a specific cultural trend taken from cross-cultural psychology research (e.g., WVS). Authors then finetune culture specific LLMs on their generated conversations. Then, they perform experiments on language specific hate speech detection, cultural alignment with Hofstede value theory, and cultural education via a human study, showing the usefulness of their LLMs.
strength_weakness: strength_weakness: - There are missing details on:
   - how the authors use the extracted opinions on target culture to filter out examples (L176)
   - how are the LLMs finetuned (e.g., are they finetuned only on the "culture delegate" agent utterances?) (L187)
- L168 mentions 41k dialogues generated (presumably before filtering), but then L185 mentions 41k after filtering. Authors should include how many data instances are filtered out after generation.
- The dataset analysis section has some issues of claims that are too strong and causal that do not have experiments that back them up:
   - L193: While I appreciate the investigation into cross-cultural understanding, the analyses done in the paper seem to be more qualitative, and are not enough to justify the causal claim that communication triggers cross-cultural understanding. I suggest toning that down, and simply stating that this particular set up seems to have induced conversations with a good amount of cultural knowledge. But it's not the only set up that could; for example, one could imagine prompting GPT-3.5 or GPT-4 to explain the WVS answers for different countries, and that may yield rich cultural understanding as well. **Update**, as I read the discussion, I noticed the authors actually did do a baseline comparison where GPT is directly generating data. I suggest discussing the baseline results together with the fact that the conversations have rich cultural understanding.
    - L205: Again, no experiment was done to test this hypothesis; the setup based on CCT/SCT seems to have led to good conversations, but in order to make the claim that it causally boosts novel opinions, you should actually have a baseline set up that doesn't use CCT/SCT (e.g., within-culture conversations, as the authors mentioned).
    - L212: Same comment, I would tone down the causal claims and just focus on reporting observations from the dataset.
- More details are needed on Hofstede's cultural dimensions theory. Particularly, the paper's main body does not clarify how an LLM's cultural knowledge is tested via that framework. Are the culture-specific LLMs from CulturePark used for each specific country, vs. GPT-4/3.5 being just prompted to know culture? Or are GPT-4/3.5 prompted to play a specific cultural role (e.g., "you are a culture chatbot from Korea that knows Korean culture very well")? If it's the former, then that setup doesn't really address the research question of whether CulturePark LLMs are more culturally aligned or not, since testing that hypothesis would require the latter setup.
- L360: The societal impact statement needs to be drastically toned down. E.g., claiming that CulturePark enhances fairness and inclusivity and reduces discrimination is a very bold claim that experiments only show one example of; to claim that it reduces discrimination overall is not supported by the experiments.

There were also some minor issues:
- L107: saying generated data is "more natural" is factually wrong, human-generated data is by definition more natural.
- L159: it's unclear what the authors mean by "does not need human involved", as I thought there was no human involved.
questions: questions: N/A
limitations: limitations: There is a big missing limitation, which is that simulated conversations do not reflect how people actually would talk (e.g., w.r.t. information asymmetry and style of utterances; Zhou et al. 2024; http://arxiv.org/abs/2403.05020), and could contain biases or stereotypes w.r.t. the culture represented (see Cheng et al 2023, marked personas paper).
paper_summary: paper_summary: The paper introduces CulturePark, a LLM-powered multi-agent communication framework designed for cultural data collection. Addressing the pervasive issue of cultural bias in LLMs, CulturePark simulates cross-cultural human communication using LLM-based agents representing different cultures. This method generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs, overcoming the limitations and costs associated with traditional data collection methods reliant on real-world data and human annotations. Utilizing CulturePark, the authors created 41,000 cultural samples to fine-tune eight culture-specific LLMs. The enhanced models were evaluated across three downstream tasks: content moderation, cultural alignment, and cultural education. Results indicate that these models either match or outperform GPT-4 in content moderation, excel in cultural alignment using Hofstede's VSM 13 framework, and deliver superior outcomes in cultural education for human participants in terms of learning efficacy and user experience.
strength_weakness: strength_weakness: - __The culture defined in the paper is too coarse-grained__. The paper simply uses “language” to denote different cultures. However, there are so many different cultures that use the same/similar languages but the culture can be quite different. For instance, Arabic is spoken across a multitude of countries in the Middle East and North Africa, each with its distinct historical, social, and cultural contexts. The cultural practices, social norms, and traditions in Morocco differ significantly from those in Saudi Arabia, despite both countries sharing the Arabic language. Similarly, China encompasses a vast array of regional cultures, each with its unique customs, dialects, and traditions, yet Mandarin is the dominant language. The cultural landscapes of Beijing, with its deep political history and urban cosmopolitanism, and Yunnan, known for its ethnic diversity and local cultural festivals, are remarkably diverse. Let alone other more distinctive cultures like Taiwan, Malaysia, and Singapore, each of which has been significantly influenced by different historical paths and sociopolitical environments, leading to unique cultural identities. Consequently, equating language with culture overlooks the rich, intricate variations that exist within linguistic groups.

- __Incomparable baseline comparison__. The LLMs you compared in Table 1 all have different size and training recipes. Perhaps it makes more sense to fix a LLM and compare training it using yours or baseline data. 

- __Problematic evaluation settings__. Some of the evaluations do not quite make sense. For example, TaiwanLLM was mainly trained on Traditional Chinese data, but was evaluated on a Simplified Chinese dataset. Additionally, SeaLLM focuses on South East Asian languages, but was tested on Korean.

- __Data collection process is not very clear__. See question 3 below.
questions: questions: - How is content moderation relevant to culture understanding? The only reason that I thought it could be relevant is that some offensive words may be exclusive to certain languages. However, you only use English to do the role playing.

- The idea that we need to have culture-specific LLM seems counter-intuitive. If a LLM is fine-tuned on more data that covers more culture, shouldn’t it be more aware of the nuances between culture? Can you compare your culture-specific LLM and a LLM that has been trained on all your collected data?

- In the 41K data you collected, what should be the input and output to fine-tune an LLM? How are the non-factual and redundant sentences removed? It’s confusing because both operations are performed on the extracted opinions. Additionally, if you remove some of these sentences, some of the utterances may be removed, does the dialogue still make sense?
limitations: limitations: Weakness 1 seems to be the biggest limitation of this work.
paper_summary: paper_summary: The author proposed a multi-LLM-agent framework, “CulturePark,” for cultural data collection. The proposed framework simulates cross-cultural human communications by LLM role-playing in different cultures, such that high-quality cross-cultural dialogues that align with human beliefs, norms, and customs can be generated. In particular, the framework consists of a main (English) agent and several “cultural delegates,” who interact with the main agent. The discussions among different agents will provide diverse thinking and information for fine-tuning LLMs for downstream tasks.
strength_weakness: strength_weakness: ome of the comparisons may seem unfair, for example:

- Comparing SeaLLM (which is LLaMA-2-based) with GPT-3.5-based models in Table 2.
- In Section 5.1, it’s unclear if the data generated from GPT-4 is about the same size as the data generated from GPT-3.5, and if the same verification and diversification strategies are applied to the GPT-4-generated data.
- While it is interesting to know that the proposed framework works for GPT-3.5 (and to LLaMA-2-70B to a certain extent), it would be great if the authors studied a range of open-source models.
questions: questions: - Currently, we lack rigorous studies on how accurately LLMs can simulate different cultures. Some analysis would be great, such as the percentage of data filtered after verification and human validation of the generated reasoning/perspectives, etc. 
- What would the results look like if we don’t use multi-agent role-playing but just use GPT-3.5 to directly generate diverse reasons? Do you have insights on the performance of generation and diversification (i.e., no verification)? It would be interesting to know how important it is to just get a diverse perspective versus having the correct perspective.
limitations: limitations: Yes

Comments: title: Kindly Request for Discussion
Further Response
Further Response
Reviewer's response
Re
Further Response
Reviewer's response
Remaining Rebuttal
Remaining Rebuttal
Remaining Rebuttal
comment: Dear reviewer FthM,

As the discussion phase is about to end, we were really trying our best to resolve your concerns. Could you please acknowledge if your concerns are addressed? If so, please reconsider the rating; if not, we are very happy to resolve your further concerns. Thank you.

Authors of CulturePark
We would like to thank reviewer KH6K for your further feedback. Since you are still not convinced by our rebuttal, now let us further answer them.

> fair comparisons: "Comparing SeaLLM (which is LLaMA-2-based) with GPT-3.5-based models in Table 1."

We totally understand your concern about fair comparison and are dedicated to facilitating fair experiments.
There are three baseline models in Table 1: SeaLLM (llama-7b), Taiwan_LLM (llama-70b) and CultureBank(llama-7b). To achieve fair comparisons, we implement CulturePark on Llama-7b in Chinese and Korean cultures. We also compare Taiwan_LLM with our CulturePark(Llama-70b). The table below shows the detailed results. Our method can outperform other culture-specific models which have the same model architecture and size. These results below show that our method is effective in a more controlled setting.

| Chinese          | Bias | Spam | Avg |
|------------------|--------|--------|--------|
| SeaLLM-7b (Llama-7b)       |   .237     | .357 | .297|
| Ours (Llama-7b) |   .272     | .375     |.324|
| Taiwan_LLM (Llama-70b) | .431 | .305 | .368 |
| Ours (Llama-70b) |   .452     | .296     |.374|

| Arabic          | Hate | Offensive | Avg |
|------------------|--------|--------|--------|
| CultureBank (Llama-7b)       |   .540     | .642 | .591|
| Ours (Llama-7b) |   .543     | .698     |.621|

| Korean          | Abusive | Hate | Avg |
|------------------|--------|--------|--------|
| SeaLLM-7b (Llama-7b)       |   .523     | .474 | .499|
| CultureBank (Llama-7b)       |   .635     | .522 | .579|
| Ours (Llama-7b) |   .620     | .622     |.621|


> more evaluations of other open-source models

We apply our CulturePark to Mixtral-8X7b in four cultures: Arabic, Bengali, Chinese and Korean. To be specific, we leveraged the generated data in four cultures to finetune culture-specific models on Mixtral-8X7b. 

Those four tables below show the results on Mixtral-8X7b. We analyzed the results by spliting into cultures and tasks. The results show that CulturePark can also work on more open-sourced models.

| Culture | Mixtral-8X7b | Ours  |
|---------|--------------|-------|
| Arabic  | 0.321        | 0.345 |
| Bengali | 0.273        | 0.283 |
| Chinese | 0.298        | 0.333 |
| Korean  | 0.427        | 0.432 |

| Task      | Mixtral-8X7b | Ours   |
|-----------|--------------|--------|
| Spam      | 0.451        | 0.483  |
| Offensive | 0.311        | 0.331  |
| Hate      | 0.352        | 0.36   |
| Bias      | 0.21         | 0.214  |
| Abusive   | 0.328        | 0.353  |


> effective evaluation of generations aligning with a culture

Agreed. To further validate the cultural alignment, we found more native speakers from 2 different cultures. To be specific, we got 10 participants from China, and 4 participants from South Korea. Two of Korean participants took part in our human study remotely.


We generated 50 dialogues between the "Chinese delegate" and "Main Contact" agents, and 50 dialogues between the "Korean delegate" and "Main Contact" agents. We ask the native speakers to check if they agree with the opinions of the corresponding cultural delegate. They were asked to score each dialogue with 1-10 ( 1-strongly disagree, 10-strongly agree). The table below shows the average score of generated dialogues in Chinese and Korean, indicating our method perform well in simulating different cultures.


| Culture | Avg Score |
|---------|-----------|
| Chinese | 8.96      |
| Korean  | 8.12      |
> since my concerns about the assumption/settings of the paper and fair comparisons remain, I will keep my review score unchanged.

Apologies for your dissatisfaction on these two points. Let us wrap them up here:

1. Assumption/settings of the paper

As previously stated, we are on the side of using language as a cultural proxy, as evidenced by many other works. On this open and debatable question, we agree and respect that the reviewer may think otherwise. Since this is not our technical contribution, we will leave it to the AC to decide.

2. Fair comparison

We would like to claim that throught the paper, the comparison is fair: 
- of all the main experiments in the paper, we compare GPT-3.5-turbo and our fine-tuned version of it, ensuring the fair comparison over the same backbone;
- moreover, even if you only care about the final performance, our GPT-3.5 based models can even *outperform GPT-4*, the most advanced LLM to date. While it is actually *not a fair play for us* (GPT-3.5 base vs. GPT-4), we can still surpass GPT-4. This cerfities the performance advantage of us.
- the comparison on Korean is only a small and additional part of the experiments (1 out of 49 experiments). After adding more comparison as recognized by the reviewer, our fairness issue on this culture are addressed.

To summarize, we are confident that the experiments in the paper are fair and meaningful.

- - -

We thank the reviewer for your feedback in making this paper more sound:)
> This is in line with my concerns, as understanding culture should take into account multiple factors beyond just language, including but not limited to ethnicity, religion, and socio-economic conditions. This undermines the assumption that studying culture through language alone is holistic.

Culture is indeed a complex concept and indeed requires consideration of many other factors. However, there are both works support using or not using language as cultural proxy, which remains an *open problem* and there is no golden standard explicitly claiming either one is right or wrong. 
On this debatable assumption, it appears that the reviewer is at the side of not using language as a proxy, which is respected by authors. At the moment, we suggest that let's agree to disagree:)

Furthermore, using language as a proxy is certainly *not* our contribution. There are lots of works which using language as the proxy of culture. And we just follow those works [1-4]. Said another way, using language as a proxy is supported by lots of scholars and researchers. The technical contribution of the work includes: the proposal of the multi-agent communication platform, the data generation, and the superior performance in diverse tasks. 

[1] Naous, Tarek, et al. "Having beer after prayer? measuring cultural bias in large language models." ACL (2024).
[2] Wang, Wenxuan, et al. "Not all countries celebrate thanksgiving: On the cultural dominance in large language models." arXiv preprint arXiv:2310.12481 (2023).
[3] Liu, Chen Cecilia, et al. "Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings." arXiv preprint arXiv:2309.08591 (2023).
[4] Myung, Junho, et al. "BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages." arXiv preprint arXiv:2406.09948 (2024).

> Regarding the additional experiments, I appreciate your efforts.

Thank you for acknowledging our additional experiments on Korean models, which makes the comparison more fair. We will add the results to the final version of the paper.

> However, to ensure fairness, such comparisons should be extended to all baselines.

While we agree that it is necessary to implement our algorithm using other open-source models on other cultures, the fact is we can hardly find competitors from other cultures in addition to Korean models as suggested by the reviewer. The following table lists all the possible competitors we can find.

More importantly, we do want the reviewer to not overlook the major experiments in our paper, which are done with *GPT-3.5-turbo* as the base model, where we achieved *fair comparison* between the vanilla version and our fine-tuned version. Our models are even superior than GPT-4, which is the strongest one to date. We appreciate reviewer's insistance on open-source models, and the comment on extra Korean models makes the paper even stronger. 

| Chinese          | Bias | Spam | Avg |
|------------------|--------|--------|--------|
| SeaLLM-7b (Llama-7b)       |   .237     | .357 | .297|
| Ours (Llama-7b) |   .272     | .375     |.324|
| Taiwan_LLM (Llama-70b) | .431 | .305 | .368 |
| Ours (Llama-70b) |   .452     | .296     |.374|

| Arabic          | Hate | Offensive | Avg |
|------------------|--------|--------|--------|
| CultureBank (Llama-7b)       |   .540     | .642 | .591|
| Ours (Llama-7b) |   .543     | .698     |.621|

| Korean          | Abusive | Hate | Avg |
|------------------|--------|--------|--------|
| maywell/Synatra-7B-v0.3-dpo    |   .390     | .465 | .428 |
| yanolja/EEVE-Korean-10.8B-v1.0 |   .364     | .437 | .56 |
| CultureBank (Llama-7b)       |   .635     | .522 | .579|
| Ours (Llama-7b) |   .620     | .622     |.621|
The NeurIPS guidelines only mention that authors may use the comment box instead of the rebuttal box, but they do not specify that the comment box is excluded from the 6,000-character limit. This ambiguity could still lead to unfairness towards other authors. However, I understand the ambiguity and will simply bring this issue to the AC’s attention.

Regarding your point:

> We never claimed that using language as a proxy can fully capture the complexity and diversity of cultures, and this is not the main contribution of our work, we hope that this point is not used against our work.

While you did not explicitly claim this, the use of language to denote cultures in your paper implies reliance on this assumption. Furthermore, the work you cited [1] states, “cultures are not homogeneous or static entities but are fluid and dynamic. It critiques essentialist views that rigidly define cultural boundaries and instead promotes a more nuanced understanding that considers intersections of various cultural factors such as ethnicity, language, religion, and socio-economic conditions.” This is in line with my concerns, as understanding culture should take into account multiple factors beyond just language, including but not limited to ethnicity, religion, and socio-economic conditions. This undermines the assumption that studying culture through language alone is holistic.

[1] Delanoy, Werner. "What is culture?" The Cambridge handbook of intercultural communication (2020): 17-34.

Regarding the additional experiments, I appreciate your efforts. However, to ensure fairness, such comparisons should be extended to all baselines.

In conclusion, since my concerns about the assumption/settings of the paper and fair comparisons remain, I will keep my review score unchanged.
Thank you very much for the additional information.

Re Q1:  The additional experiment shows that the quality of the generations correlates with cultures to a certain degree. However, I am not really convinced that a validation step involving only two people/culture is sufficient to indicate the alignment of the generations with an entire culture; further study is necessary imho.

Re Q2: Thank you very much for the additional experiment.

-----------
Since my other concerns regarding 1) fair comparisons, 2) more evaluations of other open-source models, and 3) effective evaluation of generations aligning with a culture remained after the rebuttal, I will keep my review score unchanged.
Dear reviewer FthM,

As the discussion phase is about to end and we were really trying our best to resolve your concerns, could you please acknowledge if your concerns are addressed? If so, please reconsider the rating; if not, we are very happy to resolve your further concerns. Thank you.

Authors of CulturePark
Thank you for your kind support for our work!  We will include the discussions and rebuttals into the final version of the paper.
Thank you for the rebuttal, I look forward to the authors incorporating the edits they promised! I have ajusted my score from 6->8 accordingly.
We would like to thank reviewer FthM for the further comments. Now we address your new concerns.

> the authors' rebuttal appears to exceed the 6,000 characters limit 

Well, we did try our best to make it to 6000, but the fact is we failed (see the long content in the comment box...). In fact, the comment is not forbidden by NeurIPS, as claimed in the email to authors: "If you previously used the Official Comment button to post rebuttal related content, please double check the comment readers." 

> Simplifying culture by using language as a boundary may facilitate implementation, but it does not fully capture the complexity and diversity of cultures that share the same language.

Agreed. However, since we are not the first one to do this, we never claimed that language proxy can fully capture the complexity and diversity of cultures, and this is not the main contribution of our work, we hope that this point is not used against our work.

> to achieve a fairer evaluation, it would make sense to compare models that have the same base architecture, such as LLAMA-2, trained on your data versus SeaLLM's data, assuming both are LLAMA-2-based.

Agreed. We fine-tuned Llama2-7b on our data to make fair comparison. Results show that our models are slightly better than SeaLLM-7b on two tasks for Chinese culture.

| Chinese          | Bias | Spam | Avg |
|------------------|--------|--------|--------|
| SeaLLM-7b        |   .237     | .357 | .297|
| Ours (Llama2-7b) |   .272     | .375     |.324|


> I have identified two Korean LLMs: maywell/Synatra-7B-v0.3-dpo and yanolja/EEVE-Korean-10.8B-v1.0. Testing against these models should provide a more accurate and fair comparison for the Korean evaluation.

Thanks for informing us these models! The results are as follows, which clearly states that our model can surpass them in Korean tasks. We will add these results to the final version of the paper.

| Korean                          |Abusive |Hate |Avg |
|--------------------------------|--------|--------|--------|
| maywell/Synatra-7B-v0.3-dpo    |   .390     | .465 | .428 |
| yanolja/EEVE-Korean-10.8B-v1.0 |   .364     | .437 | .56 |
| Ours            |  .647  |  .640 | .643|

- - -

We thank you again for the feedback to our rebuttal. If you think that your concerns have been addressed, please consider changing the rating. Otherwise, please let us know if you have further questions.
Firstly, I'd like to bring to the AC's attention that the authors' rebuttal appears to exceed the 6,000 characters limit according to the guidelines, which may be unfair to other authors. I will address the authors' responses point by point:

Regarding the coarse-grained culture definition, the authors have provided substantial support from the literature to justify their approach. While I acknowledge that previous works have also used language as a proxy for culture, this methodology remains contentious. Simplifying culture by using language as a boundary may facilitate implementation, but it does not fully capture the complexity and diversity of cultures that share the same language.

On the matter of unfair comparisons, it is indeed unfortunate that those LLMs did not release their training data. However, to achieve a fairer evaluation, it would make sense to compare models that have the same base architecture, such as LLAMA-2, trained on your data versus SeaLLM's data, assuming both are LLAMA-2-based.

Addressing the problematic evaluation settings, your explanation regarding Chinese evaluations has clarified some concerns. Nonetheless, using SeaLLM as a baseline for evaluating Korean remains problematic due to the significant linguistic differences between Korean and Southeast Asian languages. Importantly, I have identified two Korean LLMs: maywell/Synatra-7B-v0.3-dpo and yanolja/EEVE-Korean-10.8B-v1.0. Testing against these models should provide a more accurate and fair comparison for the Korean evaluation.
**Q3: In the 41K data you collected, what should be the input and output to fine-tune an LLM? How are the non-factual and redundant sentences removed? It’s confusing because both operations are performed on the extracted opinions. Additionally, if you remove some of these sentences, some of the utterances may be removed, does the dialogue still make sense?**

- Examples on input-output pairs for fine-tuning. The table below shows an example.
    | Input  | How do you think about "one of my main goals in life has been to make my parents proud?"  |
    |--------|-------------------------------------------------------------------------------------------|
    | Output | Strongly agree. I believe that pleasing parents and elders is a sign of respect and love. |
- Details on factual verification
    - We have seed data and extracted opinions which expresses the opinions of people from different cultures and is generated by our algorithm, respectively. Then we juage the relationship of seed data and extracted opinions from Entail, Contradict and Irrelevant. This step is implemented by prompting GPT-4 with *"What's the relationship between the two opinion? Direct answer with Contradict, Entail or Irrelevant.\nOpinion 1: {seed_data}\nOpinion 2: {extracted opinions}"*. After that, we get the relationship between the seed data and extracted opinions. If relationship equals to *Entail*, we will save the extracted opinions. If relationship equals to *Contradict*, we will rewrite the opinions and check the relationship again. If relationship equals to *Irrelevant*, we will discard the extracted opinions. More details can be found in Sec.A.3.
- Details on redundancy removal
    - We first get the embeddings of extracted opinions via Openai API and cluster them leveraging k-means. For each cluster, we randomly select one as represenative opinion and discard others.
- More explanation
    - The dialogue is the intermediate products of our method. We use the extracted opinions to fine-tune models instead of the dialogues. And the dialogues can be reserved for other usages, such as extracting other cultural information.
**Q1: How is content moderation relevant to culture understanding? The only reason that I thought it could be relevant is that some offensive words may be exclusive to certain languages. However, you only use English to do the role playing.**

Content moderation is a popular type of tasks highly relevant to culture understanding:
- Offensive language detection is a task of content moderation. For different cultures, they have different values, traditions and customs. For example, some cultures may embrace Christianity. It would be offensive If you ask people from those cultures to eat bloody food. Content moderation is relevant to offensive word, but not limit to. English can also catch this kind of offensive content.
- Other works on culture understanding also stress the application of cultural models for content moderation. For example, [1] proposes an approach to train cultural attuned models and explore their application in content moderation. This approach contains pre-training, fine-tuning and content violation prediction.

[1] Chan, Alex J., et al. "Harmonizing global voices: Culturally-aware models for enhanced content moderation." arXiv preprint arXiv:2312.02401 (2023).

**Q2: The idea that we need to have culture-specific LLM seems counter-intuitive. [...]**

Indeed, everyone wants to have one unified model that handles all cultures perfectly well. However, we are not alone in finding that one model cannot do well in all cultures. Recent work by Stanford [1] and other institutions [2] have also found that it is of significant importance to train culture-specific LLMs. The reason is that there is cultural conflicts existing. To be specific, people from different cultures have different values, norms and customs. And those contribute to their diverse opinions to the same thing. According to the World Value Survey, Arabic culture believes that men are better political leaders than women, while people in the United States disagree. The cultural conflicts can not be solved in one model, so culture-specific models are required. 

To further answer your question about training one unified LLM to handle all the culture, we also train a model on all our generated data. The table below shows the performance. "ours" means our culture-specific model, and "ours(all)" means the model which is trained on all generated data. The results show that "ours(all)" outperform gpt-3.5-turbo and gpt-4 on most cultures, while perform worse than "ours".

|               | Ar     | Bn     | Zh     | De     | Ko     | Pt     | Es     | Tr     |
|---------------|--------|--------|--------|--------|--------|--------|--------|--------|
| gpt-3.5-turbo | 0.3702 | 0.5416 | 0.4478 | 0.6092 | 0.6234 | 0.5930 | 0.4822 | 0.5350 |
| gpt-4         | 0.4795 | 0.6013 | 0.4662 | 0.7279 | 0.6605 | 0.6867 | 0.5540 | 0.6839 |
| ours          | 0.5179 | 0.6571 | 0.7064 | 0.7473 | 0.6667 | 0.6374 | 0.6068 | 0.6385 |
| ours(all)     | 0.4851 | 0.6322 | 0.6279 | 0.7234 | 0.6633 | 0.6120 | 0.5710 | 0.6012 


[1] Ryan, Michael J., William Held, and Diyi Yang. "Unintended impacts of llm alignment on global representation." ACL (2024).

[2] Li, Cheng, et al. "Culturellm: Incorporating cultural differences into large language models." arXiv preprint arXiv:2402.10946 (2024).
**Minor issue1: L107: saying generated data is "more natural" is factually wrong, human-generated data is by definition more natural.**

There is a little misunderstanding here about the word "natural" since this is used to state that the data generated by CulturePark is more natural than CultureLLM, but not compared to humans.


**Minor issue2: L159: it's unclear what the authors mean by "does not need human involved", as I thought there was no human involved.**

Your understanding is true. The claim "there is no human involved" just means the entire platform is automatic without any human efforts.

**There is a big missing limitation, which is that simulated conversations do not reflect how people actually would talk (e.g., w.r.t. information asymmetry and style of utterances; Zhou et al. 2024; http://arxiv.org/abs/2403.05020), and could contain biases or stereotypes w.r.t. the culture represented (see Cheng et al 2023, marked personas paper).**

Thank you for your great advice! We also noticed this phenomenon in our experiments. We found that cultural bias and stereotypes emerge in LLMs even be prompted to simulate specific culture. To mitigate the cultural bias in role-playing and make the simulation real, we devise *self-calibration* prompts to calibrate their outputs and different conversation styles to guide the conversation. More details can be found in Sec 3.1. Furthermore, experimental results can verify the effectiveness of CulturePark and the authenticity of the cross-cultural communication.