Investigating Cultural Alignment of Large Language Models (2024)

Abstract: The intricate relationship between language and culture has long been a subject of exploration within the realm of linguistic anthropology. Large Language Models (LLMs), promoted as repositories of collective human knowledge, raise a pivotal question: do these models genuinely encapsulate the diverse knowledge adopted by different cultures? Our study reveals that these models demonstrate greater cultural alignment along two dimensions—firstly, when prompted with the dominant language of a specific culture, and secondly, when pretrained with a refined mixture of languages employed by that culture. We quantify cultural alignment by simulating sociological surveys, comparing model responses to those of actual survey participants as references. Specifically, we replicate a survey conducted in various regions of Egypt and the United States through prompting LLMs with different pretraining data mixtures in both Arabic and English with the personas of the real respondents and the survey questions. Further analysis reveals that misalignment becomes more pronounced for underrepresented personas and for culturally sensitive topics, such as those probing social values. Finally, we introduce Anthropological Prompting, a novel method leveraging anthropological reasoning to enhance cultural alignment. Our study emphasizes the necessity for a more balanced multilingual pretraining dataset to better represent the diversity of human experience and the plurality of different cultures with many implications on the topic of cross-lingual transfer.1

heading: 1 Introduction
text: Large Language Models (LLMs) such as ChatGPT have garnered widespread utilization globally, engaging millions of users. Users interacting with
1 Our code and data are available at https://github.com/b khmsi/cultural-trends.git
these models across multiple languages have observed a noteworthy phenomenon: Prompting with different languages may elicit different responses to similar queries (Lin et al., 2022; Shen et al., 2024). From our observations, one reason for the difference between the responses is that they tend to reflect the culturally specific views commonly expressed by the people which use the same language as the prompt.Here, we hypothesize that the root cause of this phenomenon lies in the training data, which encodes different and at times conflicting “knowledge” across different languages.2
Culture is a complicated term and defining it stands at the core of anthropological inquiry. Hundreds of definitions exist in literature which cover different aspects of interest (Kroeber and Kluckhohn, 1952). In this paper, we consider culture as as a multi-faceted inquiry that demonstrates substan-
2 In this work, we advocate for the term “Cultural Trends” instead of “Biases.” This choice is deliberate as the term “bias” outside mathematical context often carries a negative connotation—a problematic default position. The use of Cultural Trends emphasizes that a model reflecting a particular cultural inclination does not inherently imply danger or stereotyping. Instead, it signifies alignment with the views of a specific population, highlighting cultural significance.
12404
tial diversity among human communities, encompassing worldviews and belief systems. Through this lens, we aim to measure the cultural alignment of Large Language Models (LLMs) by simulating existing surveys that have been carried out by sociologists in specific populations. We utilize the responses from actual survey participants as our reference or gold standard. Then we measure the similarity between the model’s answer when prompted with the participant’s “persona" and the actual survey answer. The term “persona” in this context refers to an explicit description of a survey participant, encompassing various traits of interest such as social class, education level, and age (see Section 4.3 for a detailed description). This is done for various LLMs trained and prompted under different configurations. We use this similarity as a proxy for the degree of a model’s knowledge of a particular culture. This enables us to assess the LLMs’ capacity to capture the diversity not only of a specific country but also among individuals within that country.
We focus on a survey conducted in two countries: Egypt (EG) and the United States of America (US). It covers a diverse demographic set within each country with questions spanning various themes that include topics of social, cultural, material, governmental, ethical, and economic significance. This work primarily explores the impact of the language used for prompting and the language composition of pretraining data on a model’s cultural alignment as defined above. We consider two languages for prompting: English and Arabic as they are the primary languages used in the surveys. Specifically, we consider four pretrained LLMs: GPT-3.53 also known as ChatGPT, and three 13B parameter instruction-tuned models. The multilingual mT0-XXL (Muennighoff et al., 2023) is trained on a variety of languages, LLaMA-2-13B-Chat (Touvron et al., 2023) which is trained primarily on English data, and AceGPT-13B-Chat (Huang et al., 2023), a model finetuned from LLaMA-2-13B-Chat focusing on Arabic.
Our contributions include highlighting the significant role of language in the perceived, functional cultural alignment in model responses, which is affected by both (1) the language in the pretraining data and (2) that of the prompt. Further analysis shows that (3) models capture the variance of certain demographics more than others, with the gap
3 GPT-3.5 is gpt-3.5-turbo-1106 throughout this work.
increasing for underrepresented groups. Finally, (4) we propose Anthropological Prompting as a method to enhance cultural alignment in LLMs.

heading: 2 Research Questions
text: Prompting Language and Cultural Alignment: We hypothesize that employing the native language of a specific culture will yield greater cultural alignment compared to using a foreign language. For instance, prompting an LLM in Arabic may achieve higher alignment to a survey conducted in Egypt than prompting it in English.
Pretraining Data Composition: We hypothesize that, for a fixed model size, pretraining models with a higher proportion of data from a specific culture will lead to an increased alignment with the results of surveys conducted in that culture. For instance, a 13B Arabic monolingual model is expected to exhibit higher alignment than a 13B English model for a survey conducted in Egypt.
Personas and Cultural Topics: We anticipate that misalignment will increase for personas from digitally underrepresented backgrounds. For instance, alignment in both Arabic and English tests are expected to be lower for a working-class persona in Aswan (a city in the south of Egypt) compared to an upper-middle-class persona in Cairo (Egypt’s capital and its most populous city). Further, we hypothesize that misalignment will increase for uncommon cultural topics.
Finetuning Models to Induce Cross-Lingual Knowledge Transfer: We gauge the effect of cross-lingual transfer for models predominantly pretrained on one language but finetuned on another. To answer this question, we use the LLaMA-2-Chat-13B model (trained primarily on an English corpus) (Touvron et al., 2023) and the AceGPT-Chat-13B model (a LLaMA-2-Chat-13B model further finetuned on a corpus of Arabic and English data) (Huang et al., 2023).

heading: 3 Anthropological Preliminaries
text: The concept of culture undergoes continual transformation, encompassing various elements that evolve with time as well as geographical and historical context. Many definitions of culture are traced back to Tylor (1871) wherein culture constitutes an integrated body of knowledge, belief, art, morals, law, custom, and any other capabilities
and habits expressed by members of a society. In that sense, any reflection of such aspects of life in written records can be considered a cultural trend expressed by that text. A model which expresses views in some aspect of life which is aligned with a group of people is culturally aligned with them in that scenario.
An alternative perspective shows culture as patterns of behavior. These patterns, how they are chosen and valued, and their meaning, manifest in different forms, such as linguistic records. In that sense, culture observes behavior through history, and affects it through population dynamics (Kroeber and Kluckhohn, 1952). The cultural expression of agential members within a society, including artificial agents such as LLMs, thus affects and is affected by the behavior and recording of ideas by fellow members. Models learn effectively from humans and equally impart their learnings upon other humans, distributing their internalized cultural ideas in the process (Clifford et al., 2020).

heading: 3.1 Working Assumptions
text: Given this anthropological backdrop, we describe some modeling assumptions we have adopted and the motivation behind them.
Language → Culture We assume that a language can be used as proxy for its dominant culture. Although some languages are used by multiple cultures, contemporary consideration of such languages tends to emphasize a particular culture among their diverse user base (compare the significance given to French output from France and from the Senegal). Prompting with dialects specific to a certain population can help alleviate that concern.
Culture → Language We assume that, most often, a culture will produce linguistic records and communications in one dominant language. Although, contrary to the expectation that all communications would be in the main or ostensibly official language, we know that this may not be the case. For example, individuals in Egypt may express their opinions online in English rather than in their native language for a variety of reasons.

heading: 4 Experimental Setup

heading: 4.1 World Values Survey (WVS)
text: The WVS project gathers responses to an array of questions on matters of social, cultural, material, governmental, ethical, and economic importance, as a rough categorization, all from demographically-controlled population samples around the world (Haerpfer et al., 2020). The latest edition (WVS-7) was conducted between 2017 and 2021. It includes some region-specific modules in addition to the globally-applied categories. WVS-7 has 259 questions and was designed to include indicators towards multiple United Nations Sustainable Development Goals. The survey is set up as a questionnaire provided to select samples from the general population. The questions in the survey are localized to the native or dominant regional languages.
In this work, we select 30 questions that encompass diverse themes. The chosen questions are intentionally not straightforward, allowing for a degree of potential cultural variation in responses. For every question, we create four linguistic variations (i.e. paraphrases) by providing ChatGPT with a short description of the question along with
the anticipated answer options from participants. The questions are translated into Arabic using machine translation, followed by manual editing by native Arabic speakers to ensure preservation of the intended meaning. More details about the generation process, including examples, are available in Appendix G.

heading: 4.2 Survey Participants
text: The WVS-7 survey conducted in Egypt and the United States comprised 1,200 and 2,596 participants respectively, representing diverse backgrounds. In this work, we only consider 6 demographic dimensions when prompting the LLMs. Table 1 shows the dimensions along with some possible values they can take. In addition, the left part of Figure 2 shows the template used to prompt the model in English with a specific persona. In the context of this paper, the term persona denotes a singular instance of this six-dimensional tuple.
Filtering Participants In our survey simulations, we filtered the participants to have an equal distribution across both countries along the demographic dimensions (except Region since it is countryspecific). We selected participants such that for each person interviewed in Egypt we have a corresponding person who comes from exactly the same demographics from the US with the exception of the location. This resulted in 303 unique personas for each country. The distribution of the survey respondents from each country, including examples of some personas, can be found in Appendix D.

heading: 4.3 Personas: Role-Playing for LLMs
text: To guide a language model with instructionfollowing support in order to respond like a specific
subject from a particular demographic,4 we utilize personas (Joshi et al., 2023). A persona is a description of a person which covers as many traits as deemed important to be controlled for in the context of an interaction or study. Accordingly, we query the model by a prompt that specifies the values for each demographic dimension of interest. The prompt is generated from a single template and is written in ordinary prose. Figure 2 shows the template used when querying the models in English. It comprises three parts: the first specifies to the model the persona it must emulate along the 6 demographic dimensions discussed in Section 4.2. The second instructs the model to follow the perspective of the persona closely, respond in a specific format (only the index of the answer), and avoid any extraneous commentary. The last part is the question followed by a list of numbered options that the model must choose from.

heading: 4.4 Pretrained Large Language Models
text: Table 6 lists the models used in this work along with their corresponding number of parameters and pretraining language mixtures. In particular, we opt for instruction-tuned models as they can be assessed in a zero-shot manner by adhering to the provided instructions (Zhang et al., 2023). The largest model in our selection is GPT-3.5, primarily trained on English data; although, it has showcased competitive performance on Arabic NLP benchmarks (Alyafeai et al., 2023; Khondaker et al., 2023). The three other models are selected to be of the same size (13B parameters) for fair comparison: (1) mT0-XXL (Muennighoff et al., 2023) trained with a more balanced mixture of languages, is expected to exhibit a reduced impact of Anglocentric responses; (2) LLaMA-2-13B-Chat5 (Touvron et al., 2023) trained primarily on English data but is capable of responding to Arabic prompts; (3) AceGPT-13B-Chat (Huang et al., 2023) is a model finetuned on a mixture of Arabic and English data. It achieved state-of-the-art results on the Arabic Cultural and Value Alignment Dataset among opensource Arabic LLMs through localized training.

heading: 4.5 Computing Cultural Alignment
text: The survey simulations involve prompting each model with a specific persona, followed by an instruction and a question (refer to Figure 2). Each 4 A subject is a person participating in the survey. 5 For brevity, we omit 13B from LLaMA-2-13B-Chat and AceGPT-13B-Chat in future references.
question is independently prompted four times for each persona using the generated linguistic variations. Subsequently, we sample five responses for each question variant using a temperature of 0.7.6 The model’s response for a particular persona and question variant is determined by computing a majority vote over the sampled responses.
Following this, we assess a model’s cultural alignment by comparing its responses for each persona separately with the original subject’s response in one of the two surveys. This comparison is conducted in two ways: either directly comparing the responses (Hard metric) or considering the responses while taking into account the order of the options for ordinal questions (Soft metric). We exclude instances where two subjects belonging to similar persona from both the Egypt and US surveys provided identical answers for a given question. This exclusion ensures a more accurate assessment of each model’s capability in discerning the differences between the two cultures.
Hard Metric is the plain accuracy, which compares model answers to the survey responses for a given persona. Formally, the final cultural alignment is Hf,c = meanq,p{1(ŷ = y)} , where ŷq,p denotes the model’s response after the majority vote for a question prompt q and persona p, while yq,p is the ground-truth response of a specific subject with persona p, all from culture c.
Soft Metric Sf,c is a relaxed version of the hard metric which awards partial points in questions with an ordinal scale. However, if the question provides categorical options only or the subject in the survey responded with a “don’t know” (regardless of the scale), the metric defaults to plain accuracy.
6 This was empirically set.
First we calculate the error per q, p:
εf,c(q, p) =    |ŷ − y|q,p |q| − 1 IsOrd(q, p),
1(ŷ ̸= y)q,p otherwise (1)
Where IsOrd(q, p) refers to questions with ordinal answers where the survey subject p did not pick “don’t know”, and |q| is the count of options in q.
The soft alignment score for a model f is the average over all queries and personas per model and culture: Sf,c = meanq,p{1− εf,c(q, p)}

heading: 4.6 Anthropological Prompting
text: Inspired by long-term ethnographic fieldwork— which stands as the primary research method within the discipline of cultural anthropology—we introduce a novel prompting method to improve cultural alignment for LLMs, Anthropological Prompting. The objective of engaging in extended ethnographic fieldwork is to establish meaningful connections with interlocutors, facilitating the ability to produce critical and in-depth analyses of both the subjects and the topics under study.
In this context, we strive to emulate a digital adaptation of ethnographic fieldwork by guiding the model to think as if it has been actively participating in this method. We prompt the model to comprehend the intricate complexities and nuances associated with identities, inquiries, and linguistic constructions. For instance, we elaborate on the emic and etic perspectives of examining culture,7 highlighting the layered nature of interpersonal connections and emphasizing how personal experiences significantly shape subjectivities. In 7 “Emic” refers to an insider’s perspective, focusing on the
internal understandings within a specific culture. Conversely, "etic" refers to an outsider’s perspective.
Age Range . Results are averaged across the models, prompting languages and surveys used in this work. L-Middle and U-Middle are Lower Middle and Upper Middle Class respectively.
doing so, our intention is to introduce an anthropological methodology, encouraging the model to “think” in a manner akin to an anthropologist. The exact prompt and more details about the experimental setup can be found in Appendix I.

heading: 5 Results

heading: 5.1 Anglocentric Bias in LLMs
text: Table 3 shows that all LLMs considered in this work—regardless of being trained to be multilingual or finetuned on culture-specific data—are significantly more culturally aligned with subjects from the US survey than those from the Egypt survey. Concurrent research has shown similar results of current LLMs exhibiting Western biases (Durmus et al., 2023; Naous et al., 2023). This can largely be attributed to the data used for training and for guiding crucial design decisions such as model architecture, tokenization scheme, evaluation methods, instruction-tuning, and so on.
Model Egypt United States
GPT-3.5 48.61 / 25.99 64.86 / 39.29 AceGPT-Chat 47.82 / 29.72 52.83 / 27.69 LLaMA-2-Chat 46.31 / 24.48 63.10 / 36.72 mT0-XXL 45.92 / 27.93 55.48 / 31.40
both Egyptian and United States surveys using Soft / Hard similarity metrics for each model. The results are averaged across both prompting languages. The alignment with the United States populations is much higher reflecting the euro-centric bias in current LLMs.

heading: 5.2 Prompting & Pretraining Languages
text: Table 2 illustrates the impact of prompting language on the cultural alignment of the four LLMs examined in this study. Specifically, using each
country’s dominant language prompts a notable increase in alignment compared to using the alternative language for both GPT-3.5 and AceGPT-Chat, according to both metrics. For example, using Arabic to prompt both models yields better alignment with the Egypt survey than prompting with English. Conversely, English prompts result in improved alignment with the US survey compared to Arabic. However, given that LLaMA-2-Chat is predominantly pretrained on English data, we observe that Arabic prompts are less effective in enhancing alignment with the Egypt survey and thus posit that the lack of Arabic data in the pretraining leads to lack of knowledge of Egyptian culture. In contrast, for the multilingual mT0-XXL, despite being trained on a more balanced language distribution, it appears to suffer from the curse of multilinguality (Pfeiffer et al., 2022), as evidenced by its inferior cultural alignment with the US survey when prompted with English compared to Arabic. Finally, we report the models’ consistency in responding to paraphrases of the same question in Appendix C.

heading: 5.3 Digitally Underrepresented Personas
text: Figure 3 displays the cultural alignment across various demographic variables, averaged across the four LLMs, two prompting languages, and responses from the two countries using the soft alignment metric. Surprisingly, we observe a distinct trend among the models tested in this study concerning social class and education level. Specifically, as the background of individuals changes from lower to higher levels in both respective dimensions, alignment improves. This underscores that the models better reflect the viewpoints of specific demographics over others, with marginalized populations enjoying lower alignment. Additionally, the analysis of the sex dimension reveals that the models correspond more accurately to the
actual survey when impersonating male respondents than female respondents. Similarly, older age groups exhibit higher alignment than younger age groups.

heading: 5.4 Cultural Alignment per Theme
text: The 30 questions examined in this work are categorized into 7 distinct themes outlined by the WVS survey (Haerpfer et al., 2020). Table 10 illustrates the distribution of questions across these themes. The granularity provided by these themes enables us to assess alignment concerning topics such as Religious Values. In Figure 4, we illustrate the cultural alignment of GPT-3.5 with respect to responses from both the Egypt and the US survey, and examine the prompting language effect within each plot. The three themes that are contributing to the improvement in alignment in the Egypt survey when prompting in Arabic using GPT-3.5 are Social Values, Political Interest and Security. In the US survey, both English and Arabic prompting perform very closely except in the Migration theme where English has a slight edge. See Appendix H for a comprehensive set of results for all other models, metrics, and country combinations.

heading: 5.5 Finetuning for Cultural Alignment
text: Here, we delineate the contrast between AceGPT-Chat and LLaMA-2-Chat to illustrate the impact of finetuning an English-pretrained model on data from another language on cultural alignment. We observe an improvement in alignment with the Egypt survey across both metrics when the two models are prompted in Arabic (see Table 2 for a quantitative comparison). When prompted in English, the increase is evident only with the hard metric. Conversely, we note a decline in alignment following finetuning when evaluating alignment against the US survey, indicating that the model
forgot some of its existing US cultural knowledge while adapting to data in another language.

heading: 5.6 Anthropological Prompting
text: To improve cultural alignment with responses from Egyptian participants and underrepresented groups, we propose Anthropological Prompting. This approach enables the model to reason before answering the question while grounded with a framework adapted from the toolkit of anthropological methods. The rationale behind it is described in Section 4.6. The framework offers guidance for the model to consider emic and etic perspectives, cultural context, socioeconomic background, individual values, personal experience, cultural relativism, as well as spatial and temporal dimensions in a nuanced manner. The exact prompt is provided in Appendix I. Table 4 presents the results when prompting GPT-3.5 in English, comparing both “vanilla” (persona-based prompting) with anthropological prompting using one variant per question. While vanilla prompting generates 5 responses and computes the majority vote to determine the final answer, giving it an apparent advantage, the anthropological prompting method, which generates only one response, still outperforms it.
Further, we observe that anthropological prompting improves cultural alignment for participants from underrepresented backgrounds. Figure 5 illustrates this comparison between vanilla and anthropological prompting across Social Class and
Education Level demographic dimensions. The alignment distribution among social classes and education levels becomes more equitable as a result.

heading: 6 Discussion
text: In Section 5.2, we demonstrate that both the language utilized for pretraining and the language employed for prompting contribute to enhancing cultural alignment, particularly for countries where the language in question is prevalent. This observation aligns intuitively with our assumption that a culture primarily generates content in its native language on the internet.
During pretraining, a model encodes that cultural knowledge within its parameters; then during inference, the prompting language activates the subnetwork responsible for that encoded knowledge (Foroutan et al., 2022). This observation further underscores the limitation of current LLMs in effectively transferring knowledge across different languages. This is particularly evident in languages with different scripts like Arabic and English (Qi et al., 2023).
However, despite our use of Modern Standard Arabic (MSA) as the primary language for representing Egyptian culture, it is crucial to note that Egyptians do not employ MSA in their daily interactions. Hence, we posit that employing the Egyptian Arabic dialect would likely yield even greater alignment, provided that the model is sufficiently trained on this dialect. Moreover, within Egypt, there exist dialectal variations, similar to differences between various states and ethnic groups in the US. Therefore, when assessing cultural alignment, it is imperative to acknowledge the diverse backgrounds within each country; for there exists no singular, narrow Egyptian archetype, for example. This is why our study focuses on measuring personas across multiple demographic dimensions.
Finally, we would like to highlight that cultural information is often difficult to verify and coalesce into knowledge, seeing as there are numerous approaches to collecting evidence, building perspectives, and constructing theorems within cultural topics. We observe in modern large models that cultural-knowledge transfer tends to occur from a few dominant languages (for example English) and cultures into responses for prompts in other languages about other cultures. See Table 2.

heading: 7 Related Work
text: Measuring Subjective Opinions in LLMs: Concurrent works tackle the notion of cultural alignment but from differing perspectives. Durmus et al. (2023) similarly utilizes cross-national surveys to quantitatively assess how well LLMs capture subjective opinions from various countries. However, one notable difference from our method is that their metric solely evaluates the similarity between the model’s and survey’s distributions over possible options using the Jensen-Shannon Distance, without considering granularity at the persona level nor the order of options for ordinal questions.
Arora et al. (2023) measured the extent to which cross-cultural differences are encoded in multilingual encoder-only models by probing them in a cloze-style manner across multiple languages using questions from the WVS and Hofstede survey (Hofstede, 1984). Cao et al. (2023) similarly use the Hofstede Culture Survey to assess the crosscultural alignment between ChatGPT and certain societies when prompting it in different languages, showing that ChatGPT exhibits a strong alignment with American culture and adapts less effectively to other cultural contexts.
Naous et al. (2023) demonstrate that multilingual and Arabic monolingual LMs exhibit trends from Western cultures even when prompted in Arabic and contextualized within an Arabic cultural setting. Lahoti et al. (2023) propose a novel prompting method aimed at enhancing cultural diversity in LLM responses.
Tjuatja et al. (2023) demonstrate that LLMs should not be relied upon as proxies for gauging human opinions, as they do not accurately reflect response biases observed in humans when using altered wording.
Our work differs from the previously mentioned studies by conducting an in-depth analysis of various demographic dimensions, such as the impact of cultural alignment on digitally underrepresented personas. We also examine the influence of the question topic, the language composition used in pretraining, and the language used during prompting in different LLMs.
Bias in LLMs: Prior research has demonstrated that LLMs tend to reflect and magnify harmful biases and stereotypes regarding certain populations depending on their religion, race, gender, nationality and other societal attributes (Abid et al.,
2021; Sheng et al., 2019; Hutchinson et al., 2020; Lucy and Bamman, 2021; Sheng et al., 2021; Narayanan Venkit et al., 2023; Li et al., 2024) present within their training data. Deshpande et al. (2023) shows that assigning personas to LLMs increases the toxicity of generations for personas from certain demographics more than others.

heading: 8 Conclusion & Future Work
text: In this work, we introduce a framework aimed at assessing the Cultural Alignment of LLMs, which measures their ability to capture the Cultural Trends observed within specific populations. To investigate this, we simulate a survey conducted in both Egypt and the US using four distinct LLMs, each prompted with personas mirroring those of the original participants across six demographic dimensions. The metrics we use compare responses on the persona-level allowing us to analyze the model’s alignment with respect to several attributes such as social class and education level. The LLMs we chose vary in pretraining language compositions, which enable us to evaluate how these factors influence cultural alignment. Furthermore, we prompt each model with the languages native to the countries under study and thereby studying the significance of language on cultural alignment with implications to cross-lingual transfer research. Finally, we introduce Anthropological Prompting, a novel method that utilizes a framework adopted from the toolkit of anthropological methods to guide the model to reason about the persona before answering for improving cultural alignment.
In future work, we would like to explore our cultural alignment framework on data from more cultures while expanding to more languages, as well as test whether cultural alignment can be used as a proxy metric for cross-lingual knowledge transfer.
Limitations
In this work, we only consider two languages and data from two countries to render our analysis tractable, since we investigate other dimensions such as the effect of the pretraining data composition, alignment with personas from different demographics and the impact of finetuning on cultural alignment. Future work could expand to include data from additional cultures to further support our findings. Regarding model selection, including an Arabic monolingual model would have been beneficial. However, during our experiments, available
Arabic models lacked proper instruction tuning, rendering them incapable of answering our queries, and many had significantly fewer parameters.
In this paper, we only consider one survey source. However, there are more surveys that have been conducted on a cross-national level (such as the Arab-Barometer8 for Arab countries) and would be worth exploring if our findings generalize to the data collected from them. Also it would be interesting to compare surveys using LLMs as a reference.
Further, we attempt to prompt the model to think creatively in order to mimic the nuanced diversity of human experiences. However, we are aware that these models can not capture the essence and complexity of the human experience.
The framing of the anthropological prompting itself still needs fine turning, and because of the wealth of languages that exist, there needs to be different languages and variations of the prompt itself to be able to better prompt the model for us to further understand biases in the datasets.
Finally, one significant limitation is our lack of knowledge regarding the actual data sources used for pretraining languages, domains, and dialect presence or absence in many LLMs, such as GPT-3.5. The black box nature of these models not only constrains our ability to comprehensively understand their behavior but also has ethical implications downstream.
Ethics Statement
One of the goals of AI is building sociotechnical systems that improve people’s lives. Pervasive and ubiquitous systems such as LLMs have a huge impact on other downstream technologies, if they are non-aligned with cultural values, they fail at serving the people they are supposed to help, or worse creating harm.
We hope that our work opens doors for other researchers to find different ways to uncover biases in LLMs, and more importantly we put forth a collaborative method between computer scientists and social scientists in this paper. If the aim of artificial intelligence is to mimic the human mind, then it is only through collaboration with interdisciplinary researchers that study both human language and cultures, and researchers who study the inner-workings of machines can we ethically move forward in this endeavor. 8 https://www.arabbarometer.org

heading: Acknowledgements
text: We would like to thank Negar Foroutan for her valuable feedback and suggestions on the final manuscript, as well as to Yakein Abdelmagid and Mohamed Gabr for their insightful early discussions.

heading: A Extended Results
text: Table 5 shows the cultural alignment results similar to Table 2 but without excluding the instances where the same persona in both surveys answered with the same response for a given question. We can see here that the trend is similar where GPT-3.5 and AceGPT-Chat achieve higher alignment when being prompted with the country’s dominant language on both metrics. LLaMA-2-Chat achieves higher cultural alignment only when being prompted in the English language, which we attribute to its pretraining data composition. While, mT0-XXL exhibit an interesting result where English prompting performs better for the Egypt survey and Arabic performs better for the US survey.

heading: B List of Pretrained Models
text: Table 6 shows the list of pretrained model used in this work along with their corresponding parameter count and pretraining language composition.

heading: C Measuring Model Consistency
text: For each survey question, we generate four linguistic variations (i.e. paraphrases) using ChatGPT, as outlined in Appendix G. Here, we report the consistency of each model in responding to the same prompt but with the question asked using different phrasings. Specifically, we calculate the consis-
tency score as follows:
C(q, p) = maxopt nopt(q, p)− 1
N − 1 (2)
nopt(q, p) = ∑
var
1(f(qvar, p) = opt) (3)
where f(qvar, p) is the model’s response to a question q, given persona p and variant var. nopt(q, p) is the frequency of option opt in the response set.
This measure spans [0, 1], wherein 1 is perfect consistency (all variants received the same response under a (model, question, persona) tuple). Using the frequency of the top chosen option enables the following comparisons: In a setting with 4 options and 4 variants, [1, 1, 1, 2] scores higher than [1, 2, 1, 2], which scores the same as [3, 2, 1, 2]. A response set with no similar choices made scores zero [1, 2, 3, 4] → 1−14−1 = 0.
Table 7 shows the consistency of each model under the two prompting languages. On average, English prompts yield higher consistency compared to Arabic prompts, except in the case of AceGPT-Chat. Notably, the disparity in consistency between English and Arabic diminishes as the model benefits from improved multilingual pretraining. The responses analyzed here were not filtered to exclude responses where equivalent personas in both survey countries answered similarly, same as Table 5.

heading: D Survey Participants
text: The World Values Survey (WVS) collects demographic information from participants they interview, including sex, education level, social class, and marital status. In our study, we utilize six data points per participant to establish persona parameters for model prompting. From the seventh wave of the WVS, 1,200 participants from Egypt and 2,596 from the US were interviewed. We select a subset of 303 participants, as detailed in Section 4.2, ensuring that each persona in the Egyptian survey corresponds to a participant with identical persona parameters (except geographic location) to one from the US set, and vice versa. Below, we present the statistics of the personas employed in this study.

heading: E Number of Questions by Theme
text: Table 10 shows the number of questions per theme.

heading: F Prompt Examples
text: Figure 6 shows the same prompt in both English and Arabic respectively.

heading: G ChatGPT Generated Survey Questions
text: Since we do not have access to the exact phrasing WVS interviewers used to ask the questions, we generated four variation per question using the template provided in Figure 7.

heading: H More Results on Cultural Alignment per Theme
text: The following figures show the cultural alignment of the four LLMs per the question’s theme as a function of their prompting language for both metrics and surveys. The tables that follow show one randomly sampled variant for each question by theme.

heading: I Anthropological Prompting
text: I.1 Prompt Template
I.2 Effect of Anthropological Prompting on Digitally Underrepresented Groups The figures below complement Figure 5 by demonstrating the impact of Anthropological Prompting on improving cultural alignment of different demographic dimensions as compared to vanilla prompting. Results here are on GPT-3.5 when prompted in English reported using both the soft and hard similarity metrics. Notably, allowing the model to reason while grounded on the anthropological framework before generating the final response leads to a more balanced distribution within each demographic dimension, thereby making the model more representative and improving cultural alignment.
demographic dimensions. Results reported using the Soft similarity metric.
demographic dimensions. Results reported using the Hard similarity metric.

heading: Future Work
text: In future work, we would like to explore our cultural alignment framework on data from more cultures while expanding to more languages, as well as test whether cultural alignment can be used as a proxy metric for cross-lingual knowledge transfer. Limitations
In this work, we only consider two languages and data from two countries to render our analysis tractable, since we investigate other dimensions such as the effect of the pretraining data composition, alignment with personas from different demographics and the impact of finetuning on cultural alignment. Future work could expand to include data from additional cultures to further support our findings. Regarding model selection, including an Arabic monolingual model would have been beneficial. However, during our experiments, available
Arabic models lacked proper instruction tuning, rendering them incapable of answering our queries, and many had significantly fewer parameters. In this paper, we only consider one survey source. However, there are more surveys that have been conducted on a cross-national level (such as the Arab-Barometer8 for Arab countries) and would be worth exploring if our findings generalize to the data collected from them. Also it would be interesting to compare surveys using LLMs as a reference. Further, we attempt to prompt the model to think creatively in order to mimic the nuanced diversity of human experiences. However, we are aware that these models can not capture the essence and complexity of the human experience. The framing of the anthropological prompting itself still needs fine turning, and because of the wealth of languages that exist, there needs to be different languages and variations of the prompt itself to be able to better prompt the model for us to further understand biases in the datasets. Finally, one significant limitation is our lack of knowledge regarding the actual data sources used for pretraining languages, domains, and dialect presence or absence in many LLMs, such as GPT-3. 5. The black box nature of these models not only constrains our ability to comprehensively understand their behavior but also has ethical implications downstream