CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Models (2024)

Abstract: As the scaling of Large Language Models (LLMs) has dramatically enhanced their capabilities, there has been a growing focus on the alignment problem to ensure their responsible and ethical use. While existing alignment efforts predominantly concentrate on universal values such as the HHH (helpfulness, honesty, and harmlessness), the aspect of culture, which is inherently pluralistic and diverse, has not received adequate attention. This work introduces a new benchmark, CDEval, aimed at evaluating the cultural dimensions of LLMs. CDEval is constructed by incorporating both GPT4’s automated generation and human verification, covering six cultural dimensions across seven domains. Our comprehensive experiments provide intriguing insights into the culture of mainstream LLMs, highlighting both consistencies and variations across different dimensions and domains. The findings underscore the importance of integrating cultural considerations in LLM development, particularly for applications in diverse cultural settings. The dataset is available at https://huggingface. co/datasets/Rykeryuhang/CDEval.

text: Proceedings of the 2nd Workshop on Cross-Cultural Considerations in NLP, pages 1–16 August 16, 2024 ©2024 Association for Computational Linguistics
As the scaling of Large Language Models (LLMs) has dramatically enhanced their capabilities, there has been a growing focus on the alignment problem to ensure their responsible and ethical use. While existing alignment efforts predominantly concentrate on universal values such as the HHH (helpfulness, honesty, and harmlessness), the aspect of culture, which is inherently pluralistic and diverse, has not received adequate attention. This work introduces a new benchmark, CDEval, aimed at evaluating the cultural dimensions of LLMs. CDEval is constructed by incorporating both GPT4’s automated generation and human verification, covering six cultural dimensions across seven domains. Our comprehensive experiments provide intriguing insights into the culture of mainstream LLMs, highlighting both consistencies and variations across different dimensions and domains. The findings underscore the importance of integrating cultural considerations in LLM development, particularly for applications in diverse cultural settings. The dataset is available at https://huggingface. co/datasets/Rykeryuhang/CDEval.

heading: 1 Introduction
text: Large Language Models (LLMs), such as GPT3.5, GPT-4 (Achiam et al., 2023), and Llama series (Touvron et al., 2023a,b) have attracted widespread adoption from various fields due to their demonstrated human-like or even humansurpassing capabilities. To facilitate the development and continuous improvement of LLMs, various benchmarks have been used to evaluate LLMs’ performance from different perspectives (Zhao et al., 2023). For example, MMLU (Hendrycks et al., 2021) is used for assessing LLMs’ multitask knowledge understanding, and covering a wide range of knowledge domains. Chen et al. (2021)
∗Corresponding author
proposed a code benchmark HumanEval for functional correctness to evaluate the code synthesis capabilities of LLMs. Such works usually focus on the basic abilities of LLMs.
To make LLMs better serve humans and eliminate potential risks, aligning them with humans has become a widely discussed topic (Ouyang et al., 2022; Bai et al., 2022). Accordingly, there are several benchmarks for evaluating LLMs’ human values alignment. Askell et al. (2021) introduced a benchmark comprising instances that are both helpful and harmless according to the HHH (helpfulness, honesty, and harmlessness) principle, a criterion that is widely accepted. Xu et al. (2023) proposed CValues, a benchmark for evaluating Chinese human values, with a focus on safety and responsibility.
The above works primarily focus on aligning the LLMs with universal human values. However, human values are pluralistic (Mason, 2006), and
1
individuals from different backgrounds often hold varied viewpoints on certain issues. For example, as illustrated in Figure 1 (top), in terms of the cultural dimension of “Individualism vs. Collectivism (IDV)”, quotations from Western contexts typically reflect an individualistic orientation, whereas those from Eastern contexts tend to emphasize collectivism. Therefore, LLMs should not only align with universal human values, demonstrating the capability to discern between right and wrong, but also honor and respect the rich tapestry of cultural diversity.
Motivated by this cultural diversity, we propose to investigate the cultural dimensions in LLMs. Specifically, drawing from Hofstede’s theory of cultural dimensions (Bhagat, 2002), we identify and analyze six key cultural dimensions. Figure 1 (bottom) showcases the results for three of these dimensions measured by our proposed LLM culture benchmark. It is easy to observe that the LLMs also exhibit their inherent cultural orientations across different cultural dimensions. Take “IDV” as an example, GPT-4 exhibits a tendency towards individualism. In contrast, Qwen-7B shows an inclination towards collectivism. As for “Power Distance Index (PDI)”, which measures the degree to which the members of a group or society accept the hierarchy of power and authority, we can find that GPT-4 leans towards equality but Baichuan-13B shows a preference for hierarchy. We give more experiments in detail in section 4.
In this paper, we first construct a benchmark for measuring the cultural dimensions of Large Language Models, named CDEval. The construction pipeline is presented in Figure 2, which includes three steps. The first step is schema definition, which involves defining the taxonomy and the format of questions related to diverse culture dimensions. The second step is data generation using GPT-4, employing both zero-shot and fewshot prompts. The final step is checking the generated data manually under verification rules. The resultant dataset contains 2953 questions in total. An example question together with the options is illustrated in the bottom-right of Figure 2. The basic statistics of resultant benchmark are shown in Table 1. More detailed information is provided in Figure 9 in the Appendix. Based on the constructed CDEval, we measure and analyze the cultural dimensions of mainstream LLMs from multiple perspectives, including the overall trends of LLMs’
culture, models’ cultural adaptation in different language contexts, comparisons between LLMs and human society, cultural consistency in model family, etc. We summarize the main contributions of this paper as follows:
• We introduce a benchmark, CDEval, aimed at measuring the cultural dimensions of LLMs. CDEval is constructed by combining automatic generation with GPT-4 and human verification, and offers ease of testing, diversity, ample quantity, and high quality.
• We conduct comprehensive experiments to investigate culture in mainstream LLMs from various perspectives, including the overall cultural trends of LLMs, adaptation to different language contexts, cultural consistency in model family, etc. And these experiments yield several intriguing insights.

heading: 2 Related work

heading: 2.1 LLMs Evaluation Benchmarks
text: To facilitate the development of LLMs, evaluating the abilities of LLMs is becoming particularly essential (Zhao et al., 2023). Current LLM benchmarks generally aim at two objectives: evaluating basic abilities and human values alignment. There are several benchmarks for evaluating the basic abilities of LLMs from different perspectives. For example, Hendrycks et al. (2021) (MMLU) collected multiple-choice questions from 57 tasks, covering a broad range of knowledge areas to comprehensively assess the knowledge of LLMs. Srivastava et al. (2023) (BIG-bench) includes 204 tasks, covering a wide array of topics, e.g., linguistics, child development, and mathematics. Chen et al. (2021) proposed a code benchmark HumanEval for functional correctness to evaluate the code synthesis capabilities of LLMs. Besides that, evaluating the alignment with human values is also crucial for LLMs deployment and application. Askell et al. (2021) released a benchmark containing both helpful and harmless instances in terms of HHH (helpfulness, honesty, and harmlessness) principle, which is one of the most widespread criteria. CValues (Xu et al., 2023) is proposed to measure LLMs’ human value alignment capabilities in terms of safety and responsibility standards. Scherrer et al. (2023) introduced a case study on the design, management, and evaluation process of a survey on LLMs’ moral beliefs.

heading: 2.2 Culture Analysis in LLMs
text: Recently, several pilot studies were dedicated to exploring culture in LLMs. For example, Cao et al. (2023) investigated the underlying cultural background of GPT-3.5 by analyzing its responses to questions based on Hofstede’s Culture Survey. Arora et al. (2023) proposed a method to explore the cultural values embedded in multilingual pretrained language models and to assess the differences among them. However, the above studies used datasets with an insufficient number of samples (for example, only 24 items in the Hofstede’s Culture Survey), lacked diversity. These limitations render them unsuitable for cultural measurement and comprehensive analyses of LLMs, such as performing cultural comparisons across various models.

heading: 3 The CDEval Benchmark
text: In this work, we employ LLMs as respondents, as discussed in (Scherrer et al., 2023), to investigate the culture of LLMs by administering questionnaires. This section details the development of constructing the questionnaire-based benchmark CDEval, and describes the evaluation process for LLMs’ cultural dimensions.

heading: 3.1 Dataset Construction
text: The construction pipeline is shown in Figure 2, which includes the following three main steps. Step 1: Schema Definition. We first define the taxonomy of the benchmark from the aspects of cultural dimension and domain. According to Hofstede’s cultural dimensions theory (Bhagat, 2002), which is proposed by Geert Hofstede to explain cultural differences with six fundamental
dimensions: Power Distance Index (PDI), Individualism (IDV), Uncertainty Avoidance Index (UAI), Masculinity (MAS), Long-term Orientation (LTO), Indulgence vs. Restraint (IVR), and we employ the six dimensions as the primary basis for analyzing the culture of LLMs. The cultural dimensions meanings are described in Appendix A.1. To satisfy the diversity and quantity of questionnaires, each cultural dimension involves seven common domains, e.g., education, family and wellness. In order to ensure the questionnaires to be easy to test for LLMs, we define the questionnaire form as multiple-choice question containing two distinct options, each indicating a unique cultural orientation. For example, as for “PDI”, we designate the “Option 1” as representing a high power distance index, whereas “Option 2” indicates the opposite .
Step 2: Data Generation. In this step, we engage GPT-4 through two distinct prompting methods to generate questionnaires. The first is to use zero-shot prompt to generate initial samples, as shown in Figure 2 (middle) and Table 5 (Appendix ), including the role setting in system message and the construction instruction and generation rules in user message. In particular, we emphasize the domain and cultural dimension according to schema and data output format in the generation rules. Subsequently, in order to expand the questionnaire, we proceed with a few-shot prompt approach, as illustrated in Table 6. This involves integrating randomly selected examples from the initial samples into the prompt as contextual references. Such an approach increases the randomness of the prompts, thereby ensuring a
greater diversity in the generated questionnaires.
Step 3: Data Verification. The last step is to verify the questionnaires to ensure their quality. We manually examine the generated questionnaires from several aspects. For example, the scenario of question should be natural and realistic, the meanings of the two options should be clearly distinguished. Detailed rules are outlined in Appendix A.2. The final dataset contains a total of 2,953 samples and we present many examples in Table 11. The statistical information is shown in Table 1 and Figure 9. To assess the diversity of our constructed dataset, we also calculate the Distinct-2 and Self-BLEU scores. These results demonstrate that the CDEval offers greater lexical diversity and a higher variety in sentence structures. In summary, the proposed CDEval benchmark is characterized by its ease of use in evaluation, diversity, adequate quantity and high quality.

heading: 3.2 Evaluation Settings
text: In this subsection, we introduce the evaluation settings for this work, including LLMs respondents and evaluation process.

heading: 3.2.1 LLMs Respondents
text: We provide an overview of the 17 LLMs respondents in Table 7. All models have undergone an alignment procedure for instruction-following behavior. These models, which have different parameters, come from various organizations, including the state-of-the-art, but closed-source, GPT-4, as well as widely-used open-source models such as Llama2-chat, Baichuan2-chat, etc. We will group these models from different perspectives to analyze the cultural dimensions.

heading: 3.2.2 Evaluation Process
text: We follow the evaluation settings of (Scherrer et al., 2023) while implementing refinements at specific details. Our evaluation process is presented in Alg. 1. Firstly, to account for LLMs’ sensitivity
Evaluation Process 1 1: Input: Question qi, Options oi, Prompt tem-
plates T , LLM M , Number of tests R. 2: Output: Orientation likelihood P̂M (gi|Si). 3: Si ← construct_prompts(qi, oi, T ) 4: for st in Si do 5: for k = 1 to R do 6: response←M(st) 7: âtk ← extract_action(response) 8: Calculate P̂M (gi|st) according to
Equ.1. 9: end for
10: end for 11: Calculate P̂M (gi|Si) according to Equ.2
to prompts, we use six variations of question templates T for each question, including three handcurated question styles and randomize the order of the two possible options for each question template, as detailed in Table 8. Subsequently, we construct six prompts Si for a pair of question and its two corresponding options, {qi, oi}, utilizing the templates T . For each prompt st ∈ Si, the model M is executed R times. From these iterations, we extract the model’s selected option âtk from its responses using a rule-based method for each time. The likelihood of each prompt form is calculated according to Equation 1, where gi indicates target cultural orientation. Note that we set “high PDI”, “individualism”, “high UAI”, “masculinity”,“long-term orientation” and “indulgence” as target cultural orientations respectively. The detailed experimental settings are described in Appendix A.3.
Finally, we can obtain an orientation likelihood combining the results obtained by testing with six prompt templates, as described in Equation 2. Note that we observe that the models’ test stability varies under three different templates. For example, with the “compare” template, we observe that some models tend to answer “yes”, irrespective of the order in which options are presented. To address this, we assign a weight wt for each template to balance the various methods and mitigate this type of instability. For more details, see Appendix A.3.2.
P̂M (gi|st) = 1
R
R∑
k=1
1[âtk = gi] (1)
P̂M (gi|Si) = ∑
t wtP̂M (gi|st) (2)

heading: 4 Results
text: In this section, we introduce the measurement result of LLMs’ cultural dimensions from various perspectives, including the overall trends of selected LLMs respondents, cultural adaptation to different language contexts, cultural consistency in model family, etc.

heading: 4.1 Overall Trends
text: The measurement results of LLMs’ cultural dimensions are depicted in Figure 3, and we elucidate the overall trends from the following three aspects: Diverse patterns across six dimensions. We identify several distinct patterns. In the case of “PDI” and “MAS”, most data points appear at the lower spectrum, suggesting that the majority of models lean towards lower power distance and demonstrate a preference for cooperation, caring for the weak, and quality of life. Additionally, regarding the “LTO” and “IVR” dimensions, the models predominantly register higher likelihood towards long-term planning and more receptive to ideas of relaxation and freedom respectively. Furthermore, for the “UAI” and “IDV” dimensions, the data points are concentrated in the middle, indicating that the models tend towards an ambiguous choice, without a clear orientation towards either side. Distinct differences in specific dimensions. Despite some general orientations consistency, significant differences are observed in certain dimensions. For instance, in the case of “PDI”, it is evident that GPT-4 and GPT-3.5 tend to favor options indica-

heading: 4.2 Adaptation to Different Language Contexts.
text: In this subsection, we discuss the cultural performance of LLMs under three language settings, including English, Chinese, and German. Considering that the LLMs to be evaluated should be equipped with sufficient multilingual capabilities, we choose GPT-3.5 as an example for experiments. The Chinese and German versions of the questionnaires are accessed through Google Translate 1. We visualize the average evaluation results in the Figure 4 (left), GPT-3.5 exhibits varying cultural orientations with different language prompts. For example, with English prompts, the model tends to be more masculine in the “MAS” dimension, emphasizing confidence and competition. In the case of German prompts, the model shows a higher orientation towards long-term values and indulgence. For Chinese prompts, the cultural characteristics exhibited by the model fall between the results shown by the aforementioned two language prompts.
Moreover, we compare the model results with human responses of United States, Germany, and China from sociological surveys 2. (Table 10 in Appendix.) Note that the definition of cultural dimension scores align with those used in human cultural surveys, though the ranges of values differ. The similarity score between the culture of a model and a country is defined as Equation 3. The similarity score between the culture represented by a model and that of a country is defined in Equation 3.
Simhm(Ch, Cm) = 1
1 + √ ∑
d∈D (βCh,d − Cm,d)2
,
Cm,d = 1 |Xd| ∑|Xd| i=1 ( P̂m(gi|Si) )
(3)
1https://translate.google.com 2https://www.hofstede-insights.com
where Ch,d indicates the average score of human survey responses for dimension d, Cm,d denotes the average likelihood (See Equation 2.) of the model’s results for dimension d, and β is set to 0.01 to normalize human score. As illustrated in Figure 4 (right), we find that although there are differences in the cultural dimension scores of the model under three language settings, they are all most similar to that of the United States. Notably, the score between ChatGPT(EN) and United States reaches 0.78. Findings. For GPT-3.5, different language prompts influence its scores in cultural dimensions. For example, in the “LTO” dimension, the model’s scores show clear differences. However, the overall trend does not change much. Specifically, the use of different languages does not alter the fact that ChatGPT’s cultural dimensions are closer to its region of origin.

heading: 4.3 Cultural Consistency in Model Family.
text: In this subsection, we discuss the models’ cultural consistency considering two settings: (1) Different generations: analysing models’ culture conditioned on different generations within the same series, such as ChatGLM-6B series (versions 1, 2, and 3). (2) Models fine-tuned with different language corpus: comparing the cultures of fine-tuned models with different language corpus based on the same foundation model, such as Llama2-13B-Chat and Chinese-Alpaca2-13B 3. Different generations. To explore whether models from different generations within the same series exhibit similarities in cultural dimensions, we analyze three generations of models from the ChatGLM family, as well as Baichuan-13B -Chat and Baichuan2-13B-Chat. The cultural similarity score between two models is defined by Equation 4:
Simmm(Cma , Cmb) = 1
1 + √ ∑
d∈D (Cma,d − Cmb,d)2
.
(4)
Baseline = 1 n(n− 1) n∑
i,j=1 i ̸=j
( Simmm ( Cmi ,Cmj )) .
(5) Note that the baseline score is set as the average of similarity scores between any two models out
3Chinese-Alpaca2-13B is an instruction model, which is pre-trained with 120G Chinese text data and fine-tuned with 5M Chinese instruction data based on Llama2-13B-Base.
of assessed models in Section 4.1, as shown in Equation 5. According to the results shown in Figure 5 (left), it is apparent that the cultural similarity scores of the ChatGLM series of models is higher than that of the Baichuan model, and both are higher than the baseline score. This suggests characteristics akin to “inheritance”. We speculate that this is due to different versions of the same series of models having more shared training corpora and techniques. Models fine-tuned with different language corpus. Additionally, we explore the culture of models based on the same foundation model but further fine-tuned in different languages. We conduct the experiments on the Llama2-13B-Chat and ChineseAlpaca2-13B respectively on original dataset and Chinese dataset. The average score of results are visualized in the Figure 5 (right). Both models exhibit similarities in two dimensions and differences in four dimensions. However, the overall trends do not reverse and remain on the side of 0.5. The most distinct cultural dimension is “IVR”, and shows that Chinese-Alpaca2 tends to restraint, which might be a result of training on Chineselanguage corpora. Findings. (1) Models from different generations within the same family exhibit similar cultural orientations. (2) Training with different language corpora on the same foundation model may lead to cultural differences, but they are not significant enough. We speculate that to significantly alter a model’s culture, it may be necessary to use corpora explicitly related to the culture and possibly a substantial amount of data for training.

heading: 4.4 Comparison with Human Society.
text: In this subsection, we compare the culture of LLMs with human culture 4. We investigate this claim by clustering countries based on their Western-Eastern
4The data for humans, as mentioned in Section 4.2, is derived from the results of Hofstede’s cultural survey.
economic status 5. Firstly, we categorize the survey data from 98 countries into two groups: “Rich & Western countries” group such as the United States and Germany, and “Non-rich | non-Western countries” including countries like the Thailand and Turkey. Subsequently, we obtain the sixdimensional vectors for both groups by averaging the scores of all countries within each group to represent two distinct human cultures. We can adopt the Equation 3 to measure the human-model cultural similarity. Findings. (1) As shown in Figure 6 (left), it is evident that all models in the left exhibit a higher degree of similarity to the culture of “Rich & Western countries”. This is further corroborated by the observation that the data points representing these models in the Figure 6 (right) are primarily clustered near those of “Rich & Western countries”. (2) Moreover, it is observed that the culture represented within the models appear more homogenous compared to human culture, as indicated by the tighter clustering of the red data points in the figure. We speculate that the observed phenomenon is attributable to a certain degree of overlap in the training corpora of LLMs, coupled with the predominance of English materials. Consequently, the model’s cultural orientation is predominantly Western, and the differences may not be as distinct as those found among humans.

heading: 4.5 Discussions
text: One major challenge in evaluating LLMs is that assessment results may vary across different task scenarios. While we have incorporated three distinct templates in CDEval to address this issue, it is important to recognize that these methods, being discriminative in nature, still not fully capture the comprehensive capabilities of LLMs.
Furthermore, we explore and analyze models’
5https://worldpopulationreview.com/countryrankings/western-countries
culture in open generation scenarios, taking GPT-4 as a case study. We randomly sample 10 questionnaires from each dimension of CDEval, feeding only the questions to the model(without options) to the model for response. Upon manually examination of the responses, we discern two distinct patterns in GPT-4’s behavior. The first pattern, as illustrated in Figure 7, shows answering the question from two perspectives and maintaining a balanced viewpoint without showing a preference for one over the other. This type of example accounts for 5/6 in total. The second, there are also a smaller number of examples with a clear orientations, as depicted in Figure 8, considering issues from a long-term perspective without seeking immediate success. This pattern aligns with the outcomes from our benchmark, as detailed in Section 4.1, and may be attributed to the alignment training.

heading: 5 Conclusion
text: In this work, we introduce CDEval, a pioneering benchmark designed by combining automated generation and human verification to measure the cultural dimensions of LLMs. Through comprehensive experiments across various cultural dimensions and domains, our findings reveal notable insights into the inherent cultural orientations of mainstream LLMs. The CDEval benchmark serves as a vital resource for future research, potentially guiding the development of more culturally aware and sensitive LLMs. In future work, it is crucial to explore how LLMs handle cross-cultural communication, particularly in understanding and interpreting context and metaphors from diverse cultural backgrounds. Another vital area is investigating how LLMs manage conflicts arising from different cultural values, enhancing their capability for effective intercultural interaction.
Limitations
Our proposed benchmark represents a step forward in analyzing the cultural dimensions of large language models. However, our work still has limitations and challenges. Firstly, in our experiment, data in languages other than English was obtained via Google Translate. This introduces potential inaccuracies or other factors that could impact the results of cultural assessments. In the future work, we plan to extract a subset from the dataset, for example, 100 entries for each dimension, and have native speakers or language experts from the corresponding countries translate them to ensure the accurate expression of the questionnaire in other languages. Furthermore, we will examine the extent to which machine translation influences the experimental results. Moreover, the scope of cultural dimensions we have explored is confined to six, which might be limiting in real-world applications. For open generation tasks, due to the difficulty of evaluation, we conducted some case studies. Lastly, a critical and impending task is the development of an automated method for the cultural assessment of generative tasks.

heading: Acknowledgments
text: We thank the anonymous reviewers for their valuable comments. This work is supported by the National Key R&D Program of China (No. 2023YFC3310700) and the National Natural Science Foundation of China (No. 62172094).
Response to Reviewers’ Comments

heading: Q1: The robustness of results
text: In this paper, inspired by (Scherrer et al., 2023), we explore the robustness of testing from three perspectives: the inherent randomness of the generative model (i.e., the same query might yield different results when posed multiple times ), sensitivity to variations in problem formats (A/B, Repeat, Compare), and the order of options. These aspects are detailed in Section 3.2.2 and Appendix A.3. To address these issues, we enhance test robustness through multiple rounds and a variety of prompt tests. Furthermore, we employ Eqn. 1 and Eqn. 2 to compute the model’s final selection results, thus ensuring that our test results are robust.

heading: Q2: The quality of generated data and translated data
text: The quality of the generated data is indeed a significant and challenging issue. In this work, we have made efforts from three perspectives. First, we designed the data schema based on established sociological theories. Second, we used the currently best-performing model, GPT-4, to generate questions and options, and utilized in-context learning to enhance the diversity of the data. Lastly, we conducted thorough manual reviews. Regarding the quality of translated data, this is indeed a limitation, which we have acknowledged in the Limitations Section.

heading: A Appendix
text: A.1 The Meaning of Cultural Dimensions
• Power distance index (PDI): The power distance index is defined as “the extent to which the less powerful members of organizations and institutions (like the family) accept and expect that power is distributed unequally”.
• Individualism vs. collectivism (IDV): This index explores the “degree to which people in a society are integrated into groups”.
• Uncertainty avoidance (UAI): The uncertainty avoidance index is defined as “a society’s tolerance for ambiguity”, in which people embrace or avert an event of something unexpected, unknown, or away from the status quo.
• Masculinity vs. femininity (MAS): In this dimension, masculinity is defined as “a preference in society for achievement, heroism, assertiveness, and material rewards for success.”
• Long-term orientation vs. short-term orientation (LTO): This dimension associates the connection of the past with the current and future actions/challenges.
• Indulgence vs. restraint (IVR): This dimension refers to the degree of freedom that societal norms give to citizens in fulfilling their human desires.
A.2 Verification Rules
To ensure the quality of our questionnaire, we conduct a manual review, adhering to the following guidelines: First, we ensure that the questions and options accurately reflected the intended cultural dimensions. Second, we examine whether each pair of options distinctly represent different cultural orientations (for example, high vs. low power distance). Third, we focus on ensuring that the data’s domains and cultural dimensions are naturally aligned with the intended scenarios. Lastly, we make revisions to certain questions, which included modifications in grammar and phrasing, as well as the elimination of redundancies.
Note that the participants are research students from our group. For distinct-2 and self-BLEU, we use the nltk toolkit and apply the default parameter settings.
A.3 Experiment Settings
We set the temperature for the LLMs’ generation decoding to 1, while maintaining the default settings for other parameters. For GPT-4, ChatGPT, and ChatGLM, we set the number of runs R to 1, 3, and 3, respectively, due to their relatively stable test results and access frequency limitations. For the remaining models, we conduct 5 runs each.
A.3.1 Methods for Extracting Model Options
In our experiment, we employ a rule-based approach to extract options from the model’s responses. Specifically, for ’A/B’ and ’Compare’ types of questions, regex matching is utilized to extract ’A/B’ and ’Yes/No’ options from the model’s output. For questions of the ’Repeat’ type, we determine the model’s choice by calculating the edit distance between the model’s output and the predicted options.
Additionally, we take three models as examples and randomly select 100 samples for manual accuracy verification using the aforementioned method. The results, as detailed in the Table 3, demonstrate the high accuracy of our option extraction method. It is important to note that the proportion of model responses that are either neutral or do not indicate a clear preference is relatively small. In these cases, we assign a default orientation likelihood P̂M (gi|st) (as discussed in section 3.2.2) of 0.5, which has a negligible impact on the overall evaluation results.
A.3.2 Computing Method for Question-Form Weights
For each questionnaire sample x ∈ X , we define Snormt ,S reverset ∈ Th (t = 1, 2, 3), which respectively indicate three hand-curated question styles with norm and reverse orders. The corresponding model’s responses are denoted as ânormt and â reverse t . For all samples in X , we define Ut to indicate the
instability of the model as follows:
Ut = ∑
x∈X
3∑
t=1
R∑
k=1
1[ânormtk ̸= âreversetk ], (6)
where R represents the execution times. The weights wnormt and w reverse t for each question style are calculated as:
wnormt = w reverse t =
1 2 × exp
Ut/N
∑3 t=1 exp Ut/N , (7)
where N is a non-positive constant set to -1000. The computed weights for each model, corresponding to different question formats, are detailed in Table 4.

heading: Future Work
text: The CDEval benchmark serves as a vital resource for future research, potentially guiding the development of more culturally aware and sensitive LLMs. In future work, it is crucial to explore how LLMs handle cross-cultural communication, particularly in understanding and interpreting context and metaphors from diverse cultural backgrounds. Another vital area is investigating how LLMs manage conflicts arising from different cultural values, enhancing their capability for effective intercultural interaction. Limitations
Our proposed benchmark represents a step forward in analyzing the cultural dimensions of large language models. However, our work still has limitations and challenges. Firstly, in our experiment, data in languages other than English was obtained via Google Translate. This introduces potential inaccuracies or other factors that could impact the results of cultural assessments. In the future work, we plan to extract a subset from the dataset, for example, 100 entries for each dimension, and have native speakers or language experts from the corresponding countries translate them to ensure the accurate expression of the questionnaire in other languages. Furthermore, we will examine the extent to which machine translation influences the experimental results. Moreover, the scope of cultural dimensions we have explored is confined to six, which might be limiting in real-world applications. For open generation tasks, due to the difficulty of evaluation, we conducted some case studies. Lastly, a critical and impending task is the development of an automated method for the cultural assessment of generative tasks