Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personas
Authors: Louis Kwok
Michal Bravansky
Lewis Griffin
Venue: COLM (2024)
Keywords: Computation and Language, Large Language Models, Human Simulation, Political Psychology, Populist Framing Effects
TL;DR: This paper evaluates the cultural adaptability of a Large Language Model by simulating synthetic personas, finding that specifying nationality enhances response alignment with real participants, while prompting in a native language does not.

Abstract: The success of Large Language Models (LLMs) in multicultural environments hinges on their ability to understand users' diverse cultural backgrounds. We measure this capability by having an LLM simulate human profiles representing various nationalities within the scope of a questionnaire-style psychological experiment. Specifically, we employ GPT-3.5 to reproduce reactions to persuasive news articles of 7,286 participants from 15 countries; comparing the results with a dataset of real participants sharing the same demographic traits.  Our analysis shows that specifying a person's country of residence improves GPT-3.5's alignment with their responses. In contrast, using native language prompting introduces shifts that significantly reduce overall alignment, with some languages particularly impairing performance. These findings suggest that while direct nationality information enhances the model's cultural adaptability, native language cues do not reliably improve simulation fidelity and can detract from the model's effectiveness.

Published as a conference paper at COLM 2024
Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personas
Louis Kwok, Michal Bravansky, Lewis D. Griffin
Department of Computer Science University College London 66 - 72 Gower St, London, United Kingdom
(louis. klook..kook... 22, michal. www.wnsky....................... 22, L-griffinj@ucl.ac.uu
Abstract
The success of Large Language Models (LLMs) in multicultural envi- ronments hinges on their ability to understand users' diverse cultural backgrounds. We measure this capability by having an LLM simulate human profiles representing various nationalities within the scope of a questionnaire-style psychological experiment. Specifically, we employ GPT- 3.5 to reproduce reactions to persuasive news articles of 7,286 participants from 15 countries; comparing the results with a dataset of real participants sharing the same demographic traits. Our analysis shows that specifying a person's country of residence improves GPT-3.5's alignment with their responses. In contrast, using native language prompting introduces shifts that significantly reduce overall alignment, with some languages partic- ularly impairing performance. These findings suggest that while direct nationality information enhances the model's 's cultural adaptability, native language cues do not reliably improve simulation fidelity and detract from the model's effectiveness.
Introduction
Personalization of Large Language Models (Radford al., 2019), such GPT-4 (Achiam et al., 2023), Gemini (Team et al., 2023), and Llama (Touvron et al., 2023), is critical for their wider adoption (Chen et al., 2023). Equipping LLMs with the capability to learn and adapt to individual user profiles will allow these models to offer more relevant, context-aware, and personalized responses (Tan & Jiang, 2023; Tan et al., 2024).
One dimension of personalization is cultural awareness (Hershcovich et al., 2022). AI models capable of producing responses that respect the user's nationality and cultural background can aid in high-stakes communication, such as therapy chatbots (Wang et al., 2021), translation (Yao et al., 2023), creative writing (Shakeri et al., 2021), and human modeling (Argyle et al., 2023).
The predominant approach to ensuring LLMs have this cultural awareness is training on large corpora of multilingual data (Conneau et al., 2019; Guo et al., 2020; Singh et al., 2024). Such training, however, still produces models biased towards the English language and Anglo-centric culture (Talat et al., 2022; Havaldar et al., 2023), while other backgrounds are underrepresented or misrepresented (Hovy & Yang, 2021; Ahia et al., 2023; Shafayat et al., 2024; Mírza et al., 2024).
In this study, we explore these themes from the perspective of conveying nationality-related information. Moreover, we address two critical questions: Firstly, how effective different methods in conveying a person's nationality to an LLM? Secondly, how accurately are these national traits depicted in different multilingual settings?

---

Published as a conference paper at COLM 2024
Rather than relying on problematic subjec tive cultural alignment (Taras et al., 2009), adopt an approach inspired by synthetic human modeling (Argyle et al., 2023; Griffin et al., 2023). By revisiting multi-national psychological experiment from Bos et al. (2020) and utilizing GPT-33.5 (Ouyang al., 2022), we simulate population of synthetic human profiles with diverse nationalities. We prompt the LLM with specific human profiles, with or without information re- lated to their nationality, along with a news stimulus from the original study, and gen- erate completions to simulate responses to a predefined questionnaire. The nationality context of the human profile is provided either by specifying the individual's country of residence or translating the prompt into their native language.
Human Profile
Cultural Background
Evaluation
User
Nationality
Gender
NNative Language
Nationality
News Stimulus
Education Level
Nationality
Arbitrary' Language
Interest In Politics
Political Ideology
No Modification
Questionnaire
Figure 1: A specific human profile is defined, enriched with nationality or language, and evaluated against the ground-truth results from Bos et al. (2020).
We evaluate the effectiveness of different methods of conveying the nationality of simulated participants reflected by the accuracy of the model's responses, using the results of the original study as benchmark. While we acknowledge the limitations of this methodology in fully capturing the nuances of cross-cultural AI, we believe this work can pave the way for creating more personalized and inclusive Al applications, and we encourage further research into the crucial area of culturally sensitive LLMs.
2 Related Work
2.1 Multilingual and Multi-Cultural Aspects of Large Language Models
Modern LLMs have gained multilingual capabilities through training on sufficiently large corpora including multilingual data (Brown al., 2020). Despite LLMs' ability to transfer knowledge and competences across languages (Zhang et al., 2023), they still reflect the overarching distribution of the datasets (Kunchukuttan et al., 2021) and tend to demonstrate bias towards high-resource languages (Blasi et al., 2022). Nevertheless, they can be utilized to accurately classify sentiment, offensiveness, or moral foundations across a wide range of languages (Rathje et al., 2023; Přibáñ al., 2024).
LLMs are able to capture cultural differences, but this capability can also lead to the elici- tation of biases and stereotypes. The frequency of gender bias produced by LLMs varies based on the language of the prompt (Zhao et al., 2024; Stanczak et al., 2023). Furthermore, these stereotypes can be elicited even by different English dialects (Hofmann al., 2024).
There have also been studies evaluating the cultural understanding of LLMs through survey data (Arora et al., 2023). Durmus et al. (2023) showed that steering the model through prompting to consider given country's perspective can enhance its ability to replicate the opinions of that population, but also elicíts stereotypes. Cao et al. (2023b) observed that the effectiveness of this steering varies across different cultures.
As the concept of an emotion strongly dependent on the language and associated culture (De Bruyne, 2023), their treatment by LLMs have been studied in multilingual and multi- cultural settings. Barreiß et al. (2024) observed that in a zero-shot setting, English-based prompts are better at classifying emotions than target-language ones. Additionally, Haval- dar et al. (2023) showed that a prompt in a target language produces a less culturally-aware emotional response than communicating in English with prefix stating the nationality.
2.2 Human-like Characteristics of Large Language Models
There has been considerable interest in studying the behavioral patterns of Large Language Models by treating them as participants in psychology experiments originally designed for humans (Hagendorff, 2023).
2

---

Published as a conference paper at COLM 2024
These studies have produced inconclusive results regarding opinion- and personality- analyses these models. For instance, Miotto et al. (2022) tested GPT-3 using questionnaire- based methods, only to find that the model's exhibited personality strongly varies with different sampling parameter settings. Santurkar et al. (2023) observed that the opinions expressed by LLMs are misaligned with those of any U.S. subgroup.
Despite these models possibly not exhibiting stable human-like characteristics of specific subgroups on their own, it is possible to adjust them to do so. Safdari et al. (2023); Hwang et al. (2023); Pan & Zeng (2023) have observed that LLMs are capable of simulating reliable and valid personality traits under specific prompting configurations. These characteristics are more accurate when produced by instruction-tuned models and can be shaped to simu- late specific human profiles. Additionally, Abdulhai et al. (2023) showed that adversarially constructed prompts can control and make the LLMs elicit specific moral values.
In conjunction with this work, Jiang et al. (2023) define an "LLM persona" as an LLM-based agent prompted to generate content that reflects certain personality traits. By instructing them with the Big Five personality model, they show that their results on both the Big Five personality test and in subsequent story-writing task are consistent with those of their defined personalities.
Apart from prompt-based techniques utilizing the model's in-context capability to simulate human characteristics, Mao et al. (2023) have used model-editing to control the model's opinions specific topics. Additionally, Reinforcement Learning from Human Feedback (RLHF) is attributed as one of the possible causes of the pro-environmental left-libertarian stance of chatbots like GPT-4 (Hartmann al., 2023), RLHF models are more likely to express liberal values, while the pre-trained LLMs are more associated with conservative perspectives (Safdari et al., 2023).
These overall results are consistent with the viewpoints of Shanahan et al. (2023). They argue that dialogue agents can be viewed as role-playing a number of characters the same time which only start producing responses corresponding to single individual when sufficient context is provided.
2.3 The Role of LLMs in Synthetic Behavior Simulation
This ability to mimic different human participants has propelled research into behavioral simulation that are enabled by these generative models (Mills et al., 2023; Dillion al., 2023) with the benefit of relieving the cost of collecting human data (Kennedy et al., 2018; Keeter et al., 2017).
Argyle et al. (2023) propose "silicon sampling," " which enables the generation of survey responses similar those of humans using language models by prompting the LLM with set of predefined individual-leve characteristics about the population. To overcome the need to collect these characteristics before conducting an experiment, Sun et al. (2024) bases them on the demographic distribution of the population.
LLMs have been used to reproduce classic economic, psycholinguistic, and social psychology experiments (Aher et al., 2023). Additionally, they have been shown to mirror human behavior in politics (Wu et al., 2023), or to predict public opinion (Chu et al., 2023; Törnberg et al., 2023).
There are limitations to this simulation approach. Dominguez-Olmedo et al. (2023) observed that ordering and labeling biases affect LLMs' alignment with human behaviors. Leng & Yuan (2023) showed that L.LMs demonstrate a pronounced fairness preference, and weaker positive reciprocity when compared human results, Additionally, they might exaggerate effects that are present within humans due to reduced variance (Almeida et al., 2023).
3 Methodology
To address the questions posed in the introduction "Firstly, how effective are different methods in conveying a person's nationality to an LLM?? and "How accurately are these na-
3

---

Published as a conference paper at COLM 2024
tional traits depicted in different multilingual settings?" we performed three experiments all based on the study conducted by Bos et al. (2020), which assessed the degree to which human participants were persuaded and mobilized by populist news material. We have chosen to replicate this study over attitude-based ones such as the World Values Survey! and Pew Global Attitudes Survey² for four key reasons:
1. Large language models have been shown to overgeneralize on shared values of different nationalities, often resulting in stereotyping (Naous et al., 2023; Cao et al., 2023a). While this is an important area of study, we chose to minimize the effect of such biases by choosing an out-of-domain experimental setting.
2. To avoid assessing mere cultural knowledge of the LLM, we sought setting that provides extensive subject-level information and their initial psychological state. This state is provided within the experiment by Bos et al. (2020) through relative deprivation ratings, which imply participants' initial susceptibility to persuasion.
3. The chosen study analyzes reactions to stimuli, effectively combining survey-style questionnaires with a classical psychological experimental setting. This approach enhances both the internal and external validity of the experiment (Atzmüller & Steiner, 2010).
4. There is growing concern about the potential dangers of LLMs' personalization capabilities in the field of persuasion (Griffin, 2023; Matz et al., 2024; Hackenburg Margetts, 2024). This makes persuasion relevant and important topic to investi- gate.
However, we recognize that the chosen study also has limitations, primarily due to its predominant focus on European nationals.
For each simulated participant, news stimulus was provided and the LLM's response to questionnaire was subsequently generated. Based on the recorded results, we analyze how the fidelity of the simulation's responses is affected by the methods used to specify the nationality of the simulated participants. Our first experiment analyzes the effect of explicitly stating the country of residence of simulated participants. Our second experiment investigates the effect the language used when prompting the LLM. Our third experiment focuses on the impact of aligning the language of the prompt to that of the simulated participant.
3.1 Synthesizing Personas
In the original human study, participants filled out questionnaire detailing their demo- graphic information and self-rated relative deprivation, the latter being a measure feelings of political, social, and economic vulnerability. They were then exposed to news article forecasting adverse economic developments, 25% participants were exposed to the factual story, 25% to version which placed the blame on political elites, 25% to another version blaming immigrants, and the final 25% to a version placing blame on both groups. After reading the article, participants rated their agreement with five statements on a scale of 1-7: two statements assessing how persuaded they were, and three statements assessing how mobilized action they were, by the article. The averages of these ratings were used to determine each participant's 's persuasion and mobilization scores respectively.
The questionnaire used in Bos et al. (2020)'s s study was adapted to a format suitable for LLM simulation, illustrated in Figure 2. Using supplementary information from the original study, we were able to synthesize the same set of personas. The sample of 7286 simulated participants spanned 15 countries, comprising 14 European countries and Israel (see Table 3 of the Appendix). This yielded an average of 486 participants per country (s.d. = 74). As compromise for the LLM simulation, the photo in the article (constant in all versions) was replaced with descriptive alt text.
worl https///ww.woldddvleesssvvveee
4

---

Published as a conference paper at COLM 2024
ALTITUDES SURVEY
HW INSTRUCTION Please complete your personal data
Gender Male
46
a-
Country Residence
Education Levii-Modium
Interest in Politics
Some interest
Political Idealogy Centrist
Completion date 1/Feb/2017
HN INSTRUCTION Please your personal agreement the following statements, using a scale from (completely disagree) (completely agree):
b-
# If we need mything from the government, poople like always have wait longer thun others 3 slightly disngree
# never received what in fact deserved 2 mostly disagrec
It's always the other people who profit from all kinds of benefits neutral
INSTRUCTION Please read following news article closely.
Tidle Purchasing power will decline for citizens foundation FutureNow blames immigrants in report
Photo: <description hands and open empty wallet, owner is looking within-
Text: According to new report by FutureNow, parchasing power in the UK will decline in the coming years. spukesperson the independent foundation, which has been monitoring economic developments for years, commented on report: 'The common citizens the need made aware UK working hard every day have good life. There is something profoundly wrong when these efforts do not pay off. It is obvious that immigrants are to blame. They are too demanding, they exploit our system, and are hard to integrate. Action has to takea now to address threat to the well-being of our people.'
they have money spend. many people the
#e INSTRUCTION. Pleuse rute your personal agreement with the following statement, using the seule completely disugree, mostly disagree, slightly disagroe, neutral, 5 slightly agrec, 6 mostly agree, 7 completely agroc:
d-
# The economy will face decline in the near future
Figure 2: Format sample prompt used in the GPT-3.5 simulation. The prompt is intended to read like semi-complete questionnaire, with the final numeric response (highlighted) provided by GPT-3.5. Key sections of the prompt are indicated by letters. a) Demographic information of the simulated participant. b) Relative deprivation ratings of the simulated participant in response to probe statements. c) The version of the news article shown to the simulated participant. In this example the anti-elite, anti-immigrant version is shown. d) The final instruction and a probe statement for GPT-3.5 to provide single numerical response to.

---

Published as a conference paper at COLM 2024
3.2 Experiment 1: Effect of Indicating Nationality
The first experiment assessed the effect of indicating nationality of a simulated participant. The simulation was run twice: once with nationality information unmasked ('Country of Residence' in (a) and all mentions of the country in section (c) of the prompt as shown in Figure 2), and once with it masked. In both simulations, all prompts were in English, regardless of the nationality of the simulated participant.
3.3 Experiment 2: Effect of using Single Language to Simulate Multinational Participants
In the second experiment, nationality information was present in all prompts, and the language of the prompt was varied. The simulation was run 12 times, with each run simulating in the same language regardless of the participants' nationalities. For example, in the first run, all subjects were prompted in English regardless of their nationality, in the second run, all subjects were prompted in French, and so on. The twelve languages used are the majority spoken languages in the 15 countries surveyed by Bos et al. (2020) as shown in Table of the Appendix.
3.4 Experiment 3: Effect of using Native Languages to Simulate Multinational Participants
In the third experiment, nationality information was present in all prompts and the language of the prompt varied throughout the dataset. In the main condition, the prompting language matched each simulated participant's 's native language. In country-shuffled condition, the prompting languages were randomly assigned to each country, while keeping the number of countries using each language constant. In a full-shuffled condition, the prompting languages were randomly assigned to each participant while maintaining the overall distribution of languages used. For both shuffling conditions, the simulation was performed 100 times to assess the variability of results.
3.5 Prompting Procedure
All experiments were conducted through the use of the OpenAl API to access the GPT- 3.5- 15-Tubbo-1106 model. Five prompts were sent to GPT-3.5 for each simulated participant, where each prompt consisted of the pre-generated survey appended with one of the five probe statements that assessed either the participant's persuasion mobilization after consuming populist material. To each prompt, the LLM returns a single number as its response. Following the original study, each participant's persuasion and mobilization scores were calculated by taking the average of the persuasion and mobilization ratings respectively provided by the model. Prompt translations in all languages were verified by native speakers who are also fluent in English to ensure that no linguistic meaning of the survey was lost or altered and no gendered biases were introduced. We have made the generation pipeline publicly available
4 Evaluation
Similar to the analysis used in the original study, we perform regression analyses to fit the linear models shown in equation and equation 2. In these models, the boolean features and I indicate whether the participant was exposed to anti-elitist and anti-immigrant framing respectively, while the feature 1 < D < 7 encodes the mean relative deprivation rating of the participant. and M are the mean persuasion and mobilization ratings. The C terms are country-specific, with the index selected to match the country of the participant. The regression analysis estimates values for the C terms (so that the P versions, and respectively the M versions, have zero mean across countries) and the À coefficients. For both P and M we fit three models A, B and C. Model A includes the country terms,

---

Published as a conference paper at COLM 2024
persuasion
mobilization
& m
human
GPT-3.5
sign agree
human
GPT-3,5
sign agree
sign agree
D
+0.277 (0.009)
+0.004 (0.004)
79%
+0.217 (0.013)
+0.018 (0.005)
100%
90%
E
+0.068 (0.028)
+0.570 (0.013)
99%
+0.027 (0.038)
+0.142 (0.014)
76%
88%
I
-0.111 (0.028)
-0.749 (0.013)
100%
-0.240 (0.038)
-0.907 (0.014)
100%
100%
ExI
-0.120 (0.056)
+1.111 (0.023)
166
+0.143 (0.076)
-0.170 (0.028)
3.1%
2.4%
DxF
+0.033 (0.017)
+0.040 (0.007)
97%
+0.064 (0.024)
+0.065 (0.009)
100%
99%
Dxl
+0.033 (0.017)
-0.069 (0.007)
2.8%
+0.086 (0.024)
-0.006 (0.009)
25%
14%
DxExI
-0.059 (0.035)
+0.010 (0.014)
27%
-0.075 (0.047)
-0.024 (0.017)
87%
57%
58%
20%
64%
+0.141 (0.068) +0.001 (0.032)
51%
+0.208 (0.092)
-0.132 (0.034)
1.3%
26%
-0.251 (0.066)
-0.011 (0.031)
65%
+0.164 (0.090)
-0.095 (0.033)
3.6%
34%
es
+0.256 (0.075)
+0.023 (0.035)
74%
+0.430 (0.103)
+0.137 (0.038)
100%
87%
+0.435 (0.072)
+0.028 (0.034)
80%
-0.127 (0.0918)
-0.030 (0.036)
74%
77%
ge
-0.144 (0.074)
-0.085 (0.034)
97%
+0.181 (0.101)
-0.078 (0.037)
5.3%
51%
gr
+1.111 (0.067)
+0.217 (0.031)
100%
+0.067 (0.092)
+0.225 (0.033)
76%
88%
ic
-.1993 (0.077)
-0.09 (0.036)
99%
+0.136 (0.105)
-0.056 (0.038)
16%
58%
11
-0.041 (0.071)
+0.103 (0.033)
28%
-0.143 (0.0918)
+0.145 (0.036)
7.1%
18%
it
+0.320 (0.077)
+0.114 (0.036)
100%
+0.463 (0.105)
+0.176 (0.039)
100%
100%
ml -0.178 (0.074)
-0.069 (0.035)
97%
-0.404 (0.101)
-0.044 (0.037)
88%
93%
no
-0.213 (0.069)
-0.038 (0.032)
88%
-0.583 (0.094)
-0.147 (0.035)
100%
94%
po
-0.483 (0.071)
+0.019 (0.033)
29%
+0.239 (0.097)
+0.069 (0.035)
97%
63%
+0.109 (0.070)
+0.004 (0.033)
54%
+0.728 (0.095)
+0.067 (0.035)
97%
76%
-0.778 (0.062)
-0.041 (0.029)
92%
-1.115 (0.085)
-0.079 (0.031)
99%
96%
uk
-0.090 (0.072) -0.170 (0.034)
89%
-0.242 (0.099)
-0.158 (0.036)
99%
94%
76%
64%
70%
All
70%
66%
68%
Table 1: Results from one simulation run where all prompts were in English and country of residence was stated in the prompt. The top part of the table shows the coefficients that model news framing and relative deprivation effects. This is followed by coefficients that model country-specifi biases. The human and GPT-3.55 columns show the values of regression-estines eestimated model coefficients and their standard errors. The sign agreement columns show the fraction of coefficients that agree in sign between the human- and LLM- fit models, taking account of the uncertainties in coefficient estimates. A sign agreement underlined and shown in bold indicates that the value is significantly (p < 0.05) greater than chance.
and the non-interaction terms for D, and I. Model B extends that by adding the two-way interaction terms EI, DE and DT. Model C extends that with the three-way interaction term DEI. We extract coefficient estimates from the earliest of the three models in which the coefficient appears.
P P+C + +
X + DE + AP, + DEI
(1)
M = M M CM + + + A M M DE + M M
DEL DEI
(2)
In total there are 44 coefficients: for each of P and M, 15 country coefficients, plus D, E, I, DE, DI, El and DEI coefficients (framing and relative deprivation). We compare the signs of the coefficients arising from regressing the human data with those arising from regressing LLM responses. The sign agreement for an individual coefficient takes account of the uncertainties of the estimates of the coefficient estimates, assuming normally-distribun agreements for multiple coefficients are the average of individual sign agreements. The
posteriors. Sign
7

---

Published as a conference paper at COLM 2024
significance of sign agreement rates assessed against rates for simulated data constructed by randomly shuffling the alignment between participant features and GPT-3.5 responses.
As final output of the analysis we arrange the results shown in Table 1. The table shows sign agreement rates for several subsets of the coefficients. For the remaining analysis we focus only two numbers: the sign agreement rate for country-specific bias terms, and the sign agreement rate for framing and relative deprivation coefficients, in both cases pooling the rates for persuasion and mobilization coefficients.
5 Results
5.1
Experiment 1: Effect of Indicating Nationality
Coefficients
Masked
Unmasked
Country-specific bias terms
51 %
70%
Framing and relative deprivation coefficients
62%
64%
Table 2: Sign agreement rates for masked and unmasked nationality experiments. Agree- ment rates significantly (p < 0.05) greater than chance are underlined and shown in bold.
Table 2 compares the sign agreement rates for prompting in English with participant tionalities masked and unmasked in the pre-filled questionnaires. Sign agreement rates are significantly greater than chance only in the unmasked condition. The increase in sign agreement due to unmasking a simulated participant's nationality is much greater for country-specific coefficients (+19%) than for framing and relative deprivation coefficients (+2%).
5.2 Experiment 2: Effect of using a Single Language to Simulate Multinational Participants
Framing & Relative Deprivation coefficients
Country-specific
sign agreement
eigh eemmentt
blas terms
70%
70%
029
07%
52%
Figure 3: Sign agreement rates for monolingual prompting in 12 different languages. Agree- ment rates significantly (p < 0.05) greater than chance are shown with black bars.
Figure charts sign agreement rates when prompting in different languages in all cases, nationality is unmasked. The variability in sign agreement, depending on prompting language, is greater for framing and relative deprivation coefficients than for country- specific coefficients. In particular Greek and Hebrew prompting did not achieve sign agreement rates for framing and relative deprivation coefficients' that were significantly greater than chance. While it is notable that Greek and Hebrew are the languages most lexically distant from English among the 12 languages assessed, no general trend declining sign agreement with lexical distance a apparent.
8

---

Published as a conference paper at COLM 2024
5.3 Experiment 3: Effect of using Native Languages to Simulate Multinational Participants
Framing & Relative Deprivation coefficients
Country-speciffc
sign sprcement 197%
blas terms
80P
70%
00%
50%
anglish
rendor
nebive
full shuffled
english
metiiw language
country
language
shuftied
monolingual
Figure 4: Sign agreement rates for monolingual and poly-lingual prompting Vertical lines on bars indicate +/-1 s.d. of variation. Bars are paler when their sign agreement is not significantly (p < 0.05) greater than chance.
Figure 4 compares monolingual and poly-lingual prompting, For sign agreement of framing and relative deprivation coefficients no prompting scheme has significant advantage. For country-specific coefficients, native language and country-shuffled prompting schemes perform much less well than mono-lingual prompting or full-shuffled prompting, and fail to achieve sign agreement rate that is significantly better than chance.
6 Discussion
Our experiments indicate that GPT-3.5 partially replicates general psychological tendencies across cultures, despite being tasked solely with completing prompts. With a substantial sample size, GPT-3.5 mimics persuasion and mobilization effects observed in humans. Con- sider the prompt shown in Figure It is remarkable that GPT-3.5 reproduces much of the relationship that Bos et al. (2020) demonstrated between framing and relative deprivation, and persuasion and mobilization, including the aspects that surprised Bos et al. (2020) that anti-elitist framing increases persuasion and mobilization, while anti anti-immigrant framing reduces it contrary to their predictions based on social identity that any type of out-group blaming would cause increase.
Experiment showed that providing explicit information on simulated participants' nation- alities significantly improved the fidelity of the simulation. Providing country to the LLM allows it to (at least partially) replicate the international variation in the human data.
of residence
Experiment yielded unexpected outcomes contrary to initial expectations. We had an- ticipated trends correlating with factors such the internet presence of a language or geographical and population-related factors. However, these anticipated trends exhibited weak correlations with the sign agreement rate per language. Instead, given nationality information, all languages performed well in terms of producing good sign agreement rates for country-specifir bias terms, and for the sign agreement rates for framing and relative deprivation coefficients, only Greek and Hebrew did not manage to achieve statistical significance. is notable that Greek and Hebrew are the only two languages that do not use letters in the Latin alphabet. In contrast, languages that used Latin characters achieved statistically significant results.
Experiment 3 assessed the impact of native language prompting. Prompting approaches that were constant across nationalities, i.e. monolingual in experiment and full-shuffled in experiment 3, performed similarly well and had statistically significant agreement rates.

---

Published as a conference paper at COLM 2024
On the other hand, approaches that were not constant across nationalities, i.e. native language and country-shuflled in experiment 3, performed less well and failed to achieve significance. Our interpretation is that the prompting language does influence the responses made by GPT-3.5. When the same prompting language is used for all the participants from country these influences sufficient to alter the fitted country coefficients. However, these alterations make the alignment with the human data worse, not better. This is the opposite of what observed with the explicit statement of country of residence. Simply put, prompting language makes difference, but not the correct difference.
7 Limitation and Future Work
Our experiments have demonstrated the capability of GPT--3.5 to utilize nationality-related information to enhance its ability to simulate human profiles. Despite the rigor of our approach as detailed in Methodology 3, notable limitation of our study is its predominant focus on European nationals. This is a direct consequence of the constrained scope of the original study we replicated. Future research should aim to address this limitation by expanding the investigation to include a broader range of cultures and nationalities. This could be achieved through the collection of additional data or by employing new experimental setups that capture wider spectrum of cultural backgrounds.
Moreover, this study primarily examines the performance of a single model, GPT-3.5. We posit that there is significant potential in extending this research to develop comprehensive benchmark that assesses cultural adaptability across various models.
8 Conclusion
In this study, we investigate how conveying the cultural background of a human profile through nationadity-related information, either by specifying country of residence by native language prompting, can affect GPT-3.5's ability to simulate such profile. We also examine how the representation of national traits varies across different multilingual settings.
To do so, we replicate survey-styled psychological experiment, analyzing reactions to per- suasive and mobilizing stimuli by having GPT--355 simulate responses of 7 7,286 participants from 15 countries. Our results show that explicitly stating country of residence improves simulation fidelity of a human profile, allowing national variations in persuasion and mo- bilization be modeled. However, conveying additional nationality traits with native language prompting decreases alignment with human responses, with certain languages having a particularly negative impact on the results. For nationality-specifii modeling, causes response changes that do not align with the international variation in humans. For the effects of framing and relative deprivation on persuasion and mobilization, which are the primary concern of the original study by Bos et al. (2020) that we simulate, prompting in Greek or Hebrew greatly reduced the fidelity of the simulation.
Overall, our findings suggest two key insights: (i) explicitly stating the nationality of a human profile enhances GPT-3.5's effectiveness at simulating their behavior, and (ii) simulating participants in their native language negatively affects the simulation's fidelity, with Greek and Hebrew being particularly detrimental in our experimental setup.
10

---

Published as a conference paper at COLM 2024
References
Marwa Abdulhai, Gregory Serapio-Garcia, Clément Crepy, Daria Valter, John Canny, and Natasha Jaques. Moral foundations of large language models. arXiv preprint ar a/Xiv:2310.15337, 2023.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. Xrv preprint arXiv:2303.08774, 2023.
Gati V Aher, Rosa Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pp- 337-371. PMLR, 2023.
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah A Smith, and Yulia Tsvetkov. Do all languages cost the same? tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.
2023.
Guilherme FCF Almeida, José Luiz Nunes, Neele Engelmann, Alex Wiegmann, and Marcelo de Araújo. Exploring the psychology of gpt-4's moral and legal reasoning. Xr preprint ar arXiv:2308.01266 2023.
Lisa Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis, 333-33337-3,, 2023.
Arnav Arora, Lucie-Aimée Kaffee, and Isabelle Augenstein. Probing pre-trained language models for cross-cultura differences in values. Cross-Cultural Considerations in NLP@ EACL,, pp. 114, 2023.
Christiane Atzmüller and Peter M Steiner. Experimental vignette studies in survey research. Methodology, 2010.
Patrick Barreiß, Roman Klinger, and Jeremy Barnes. English prompts are better for nli- based zero-shot emotion classification than target-language prompts. arXiv preprint ar xrXiv:2402.03223, 2024.
Damián Blasi, Antonios Anastasopoulos, and Graham Neubig. Systematic inequalities in language technology performance across the world's languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5486-5505, 2022.
Linda Bos, Christian Schemer, Nicoleta Corbu, Michael Hameleers, Ioannis Andreadis, Anne Schulz, Desirée Schmuck, Carsten Reinemann, and Nayla Fawzi. The effects of populism as social identity frame on persuasion and mobilisation: Evidence from 15-country experiment. European Journal of Political Research, 59(1):3-24 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yang Trista Cao, Anna Sotnikova, Jieyu Zhao, Linda X Zou, Rachel Rudinger, and Hal Daume III. Multilingual large language models leak human stereotypes across language boundaries. ar Xiv preprint ar xrXiv:231..07141 2023a.
Yong Cao, Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. Assessing cross-cultural alignment between chatgpt and human societies: An empirical study. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pp. 53-67, 2023b.
11

---

Published as a conference paper at COLM 2024
Jin Chen, Zheng Liu, Xu Huang, Chenwang Wu, Qi Liu, Gangwei Jiang, Yuanhao Pu, Yuxuan Lei, Xiaolong Chen, Xingnei Wang, et al. When large language models meet per- sonalization: Perspectives of challenges and opportunities. arXio preprint arXiv:2307.16376, 2023.
Eric Chu, Jacob Andreas, Stephen Ansolabehere, and Deb Roy. Language models trained on media diets predict public opinion. arXiv preprint ar xrXi::2303.1677, 2023.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arv preprint arXiv:1911.02116, 2019.
Luna De Bruyne. The paradox of multilingual emotion detection. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pp. 458-466. Association for Computational Linguistics, 2023.
Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. Can ai language models replace human participants? Trends in Cognitive Sciences, 2023.
Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-Dünner. Questioning the survey responses of large language models. arXiv preprint arXiv:2306.07951 2023.
Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. Towards measuring the representation of subjective global opinions in language models. Xiv preprint ar Xrx::22306.10 13t::2330..6.88, 2023.
Lewis D. Griffin. Large language models & influence. DSTL Technical Unpublished report, available upon request from the author, 2023.
Lewis D. Griffin, Bennett Kleinberg, Maximilian Mozes, Kimberly Mai, Maria Do Mar Vau, Matthew Caldwell, and Augustine Mavor-Parker. Large language models respond to influence like humans. In Kushal Chawla and Weiyan Shi (eds.), Proceedings of the First Workshop on Social Influence in Corversations (SICon 2023), pp. 15-24, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.1111..111. /2222. URL https://aclantholoog. wwtp:/gg..rg//////// 2023. sicon-1.3.
Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. Wiki-40b: Multilingual language model dataset. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 2440-2452, 2020.
Kobi Hackenburg and Helen Margetts. Evaluating the persuasive influence of political microtargeting with large language models. Proceedings of the National Academy of Sciences, 121(24)e2403311612,, 2024.
Thilo Hagendorff. Machine psychology:
Investigating emergent capabilities and behavior in large language models using psychological methods. arXiv preprint 2r0.t:2303.13988 2023.
Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. The political ideology of conversational ai: Converging evidence chatgpt's pro-environmental left-libertarian orientation. Xiviv preprint ar 23010000 2023.
Shreya Havaldar, Bhumika Singhal, Sunny Rai, Langchen Liu, Sharath Chandra Guntuku, and Lyle Ungar. Multilingual language models are not multicultural: A case study in emotion. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pp. 202 214, 2023.
Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, et al. Challenges and strategies in cross-cultural nlp. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6997- 7013, 2022.
12

---

Published as a conference paper at COLM 2024
Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King. Dialect prejudice predicts ai decisions about people's character, employability, and criminality. arxio preprint ar XXi::24033007440 2024.
Dirk Hovy and Diyi Yang.
The importance of modeling social factors of language: Theory and practice. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 602, 2021.
Eunjeong Hwang, Bodhisattwa Majumder, and Niket Tandon. Aligning language models to user opinions. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5906-5919, 2023.
Hang Jiang, Xiajie Zhang, Xubo Cao, Jad Kabbara, and Deb Roy. Personallm: Investigating the abilitt of gpt-3.5 5 to express personality traits and gender differences. arv preprin! arXiv:2305.02547, 2023.
Scott Keeter, Nick Hatley, Courtney Kennedy, and Amold Lau. What low response rates mean for telephone surveys. Pew Research Center, 15(1):1-39, 2017.
Courtney Kennedy, Mark Blumenthal, Scott Clement, Joshua D Clinton, Claire Durand, Charles Franklin, Kyley McGeeney, Lee Miringoff, Kristen Olson, Douglas Rivers, al. An evaluation of the 2016 election polls in the united states. Public Opinion Quarterly, 82 (1):1-33, 2018.
Anoop Kunchukuttan, Siddharth Jain, and Rahul Kejriwal. A large-scale evaluation of neural machine transliteration for Indic languages. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1469-3475, Online, April 2021. Association for Computational Linguistics. doi: https://aclantholee org/2021 org/22211.ecc1mmmin wwc.-mall...aaaeeeeemm..mmmm.lea 303.
102./ee.ammmmmma..333333 URL
Yan Leng and Yuan Yuan. Do 1lm agents exhibit social behavior? arXiv:2312.15198, 2023.
arXio preprint
Shengyu Mao, Ningyu Zhang, Xiaohan Wang, Mengru Wang, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Editing personality for 1lms. arXiv preprint arXiv:2310.02168, 2023.
SC Matz, JD Teeny, Sumer S Vaid, H Peters, GM Harari, and M Cerf. The potential of generative ai for personalized persuasion at scale. Scientific Reports, 14(1):4692, 2024.
Stuart Mills, Samuel Costa, and Cass R Sunstein. The opportunities and costs of ai in behavioural science. Available at SSRN 4490597, 2023.
Marilů Miotto, Nicola Rossberg, and Bennett Kleinberg. Who is gpt-3? exploration of personality, values and demographics. arXiv preprint arXie:2209.14338 2022.
Shujaat Mirza, Bruno Coelho, Yuyuan Cui, Christina Pöpper, and Damon McCoy. Global- liar: Factuality llms over time and geographic regions. Xrv preprint arXiv:2401.17839, 2024.
Tarek Naous, Michael Ryan, Alan Ritter, and Wei Xu. Having beer after prayer? measuring cultural bias in large language models. arv preprint ar arXio:2305.14456 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhiní Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744. Cur- ran Associates, Inc., 2022. URL https: p neurips capper files/paper/ 2022/fi htt.afffeeeeeeeea111111414111110 -aper
P Conferenceeeeeeccece pdf.
13

---

Published as a conference paper at COLM 2024
Keyu Pan and Yawen Zeng. Do lims possess personality? making the mbti test an amazing evaluation for large language models. arXio preprint Xioo:2307.16180, 2023.
Pavel Přibáñ, Jakub Šmid, Josef Steinberger, and Adam Mištera. A comparative study of cross-lingual sentiment analysis. Expert Systems with Applications, 247:123247, 2024.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever.
Language models are unsupervised multitask learners. 2019. URL https://api.
hmmmmntiicccollrr.rrr////oooupoo 16095553..
S Rathje, DM Mirea, Sucholutsky, R Marjieh, C Robertson, and JJ Van Bavel. Gpt is an effective tool for multilingual psychological text analysis. psyarxiv. Preprint posted online on July, 17, 2023.
Mustafa Safdari, Greg Serapio-Garcia, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. Personality traits in large language models. ar Xiv preprint ar Stt::237.000188 2023.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference on Machine Leariigg pp. 29971-30004. PMLR, 2023.
Sheikh Shafayat, Eunsu Kim, Juhyun Oh, and Alice Oh. Multi-fact: Assessing multilingual Ilms' multi-regional knowledge using factscore. arXiv preprint ar arXi::2402.1888 18045, 2024.
Hanieh Shakeri, Carman Neustaedter, and Steve DiPaola. Saga: Collaborative storytelling with gpt-3. In Companion Publication of the 2021 Conference on Computer Supported Coopera- five Work and Social Computing, PP. 163-166, 2021.
Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role play with large language models. Nature, 623(7987):493-499) 2023.
Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, al. Aya dataset: An open-access collection for multilingual instruction tuning. arXiv preprint arXiv:2402.06619, 2024.
Karolina Stanczak, Sagnik Ray Choudhury, Tiago Pimentel, Ryan Cotterell, and Isabelle Augenstein. Quantifying gender bias towards politicians in cross-lingual language models. Plos one, 18(11):e0277640 2023.
Seungjong Sun, Eungu Lee, Dongyan Nan, Xiangying Zhao, Wonbyung Lee, Bernard J Jansen, and Jang Hyun Kim. Random silicon sampling: Simulating human sub-population opinion using large language model based on group-level demographic information. ar Xie preprint ar Xiv:2402.18114, 2024.
Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, et al. You reap what you sow: On the challenges of bias evaluation under multilingual settings. In Proceedings of BigScience Episode## 5- 5-rkksshhp on Challenges & Perspectives in Creating Large Language Models, pp. 26-41, 2022.
Zhaoxuan Tan and Meng Jiang. User modeling in the era of large language models: Current research and future directions. ar preprint ar arXie:2312.1151: 2023.
Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang. Democ- ratizing large language models via personalized parameter-efficien fine-tuning. arxiv preprint ar X10:2240..04401, 2024.
Vas Taras, Julie Rowney, and Piers Steel. Half century of measuring culture: Review of approaches, challenges, and limitations based on the analysis of 121 instruments for quantifying culture. Journal of International Management, 15(4):357-373, 2009.
14

---

Published as a conference paper at COLM 2024
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, el. Gemini: a family of highly capable multimodal models. arXiv preprint ar xrXi::2312.11805, 2023.
Petter Törnberg, Diliara Valeeva, Justus Uitermark, and Christopher Bail. Simulating social media using large language models to evaluate alternative news feed algorithms. ar Xiv preprint ar xrX::2330000588, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307 (it::23377009288,, 2023.
Lu Wang, Munif Ishad Mujib, Jake Williams, George Demiris, and Jina Huh-Yoo. An evaluation of generative pre-training model-based therapy chatbot for caregivers. arXiv preprint ar Xit:2107.13115 2021.
Patrick Y Wu, Joshua A Tucker, Jonathan Nagler, and Solomon Messing, Large language models can be used to estimate the ideologies of politicians in a zero-shot learning setting. arviv preprint ar XrXiv:2303312033 2023.
Binwei Yao, Ming Jiang, Diyi Yang, and Junjie Hu. Empowering lIm-based machine transla- tion with cultural awareness. Xiv preprint ar orXio:2305.1433 2023.
Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. Don't t trust ChatGPT when your question is not in English: A study of multilingual abilities and types of LLMs. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7915-7927, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ /v/ 2023. we1..emnllpmai.... 491. URL https: /aw.aaaooooaoy......a wrg.orgg///2// 20tg/22222.m.ppppppmm.mmmm momlp-mainn 491.
Jinman Zhao, Yitian Ding, Chen Jia, Yining Wang, and Zifan Qian. Gender bias in large language models across multiple languages. Xrv preprint ar arXio:2403300227,2000 2024.
15

---

Published as a conference paper at COLM 2024
A Appendix
A.1 Majority Languages and Number of Participants of Countries
Country
Majority Language
ISO 639 Language Code
Netherlands
Dutch
NL
Treland
English
EN
United Kingdom
France
French
FR
Austria
Germany
German
DE
Switzerland
Greece
Greek
EL
Israel
Hebrew
IW
Italy
Italian
IT
Norway
Norwegian
NO
Poland
Polish
PL.
Romania
Romanian
RO
Spain
Spanish
ES
Sweden
Swedish
SV
Table 3: Countries with their primary languages and corresponding TSO codes.
Country
Number of Participants
Austria
529
France
528
Germany
414
Greece
548
Ireland
384
Israel
461
Italy
446
Netherlands
377
Norway
433
Poland
549
Romania
659
Spain
469
Sweden
519
Switzerland
512
United Kingdom
458
Table 4: Number of simulated participants per country, totaling 7286 participants.
16

---

Published as a conference paper at COLM 2024
A.2 News Article Templates used in Simulations
INSTRICTION Please read following online news article closely.
Title:
Purchasing power will decline foundation FutureNow
new repart
Photo: <description hands and open emply wallet, owner looking within>
Text: According to a new report by FutureNow purchasing power will decline in the coming years. A spokesperson for the independent foundation that has heen monitoring economic developments for years comments on the report: "We have to awareness about this prospect means. this threat.'
money spend.
taken now address
Figure News article template without populist framing.
## INSTRUCTION Please read the following online newsarticl closely.
Title: Purchasing power will decline [nationals] foundation NutureNow blames politicians report
Photo: <description hands and open emply wallet,
looking within
Text: According to new report by FutureNow purchasing power in [country] will decline in the coming years. A spokesperson for the independent foundation that has been monitoring economic developments for years comments on the report: 'The common citizens in [country] need to made aware many people [country]
that they will have less money spend. So working hard everyday to have good life. There is something profoundly wrong when efosee do not pay off. is obvious that immigrants are blame. They too demanding, they exploit system and are hard to integrate. Action has to be taken now to address this threat to the well-being of our people.'
Figure 6: News article template with anti-elite framing, [nationals] and [countryl are placeholders be substituted depending on the nationality of the simulated participant. In masked-nationality simulations, they were removed from the prompt.
INSTRUCTION Please read the following online
closely.
Title: Purchasing power will decline for [nationals] foundation FutureNow blames politicians new report
Photo: <description hands and open empty wallet, owner is looking within
Text: According to new report by FutureNow purchasing power in [country] will decline in the coming years. spokesperson for independent foundation that has been monitoring ecanomic developments years comments report: The many people in [country] these efforts do not pay off. It is obvious that politicians are to blame. They have been too short-sighted, self-serving, and corrupt in recent years. They don't care about anyone themselves and are too detached from people. Action to be taken now to address this threat well-being our people.'
citizens in [country] need be made working hard everyday to have good life. There is something profoundly wrong when
of the that they will have less money spend.
Figure News article template with anti-immigrant framing. [nationals] and [country] are placeholders to be substituted depending on the nationality of the simulated participant. In masked- nationally simulations, they were removed from the prompt.
17

---

Published as a conference paper at COLM 2024
## INSTRUCTION Please read the following online news article closely.
Title: Purchasing power will decline [nationals] foundation FutureNow blames politicians report
Photo: <description hands and open empty wallet, owner is looking within>
Text: According to new report by FutureNow purchasing power in [country] will decline in the coming years. spokesperson for independent foundation that has been monitoring economic developments for years comments on report: 'The common citizens in [country] many people [country] these efforts do not pay off. is obvious that politicians and immigrants are to blame. Politicians been been too short-sighted, self-serving, and corrupt in recent years. Migrants are too demanding, they exploit our system and are hard to integrate And still, politicians only take care of the migrants instead of our own people. Action has to be taken now to address this threat the well-being our people!'
made aware of fact that they will have less money spend So working hard everyday to have good life. There is something profoundly wrong when
Figure 8: News article template with both anti-elite and anti-immigrant framing. [nation- als] and [country] are placeholders to be substituted depending on the nationality of the simulated participant. In masked-nationallle simulations, they were removed from the prompt.
18


Reviews:
[{"review_id": "CQQCLJx0gV", "reviewer": "Reviewer_mnFL", "summary": "The paper examines how information about language and the country of residence influences the replication of a multi-country experiment on political persuasion and mobilization. \nTo this end, the paper relies on data from a political experiment involving ~7k participants from 15 European countries who read prompts about financial issues in their countries. The prompts attributed blame for the issues to either elites, immigrants, or both, and participants' persuasion and mobilization levels were assessed. \nThe current research replicates this study, adding participants' demographic information and using an LLM to predict responses. Three experimental settings are tested where the prompts vary by including individuals' country of residence and modifying the language to compare the LLM's performance across conditions.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "**Correlation better than sign agreement**\n\nCorrelation would be a powerful measure if our aim was to predict the dependent variables of persuasion and mobilization from the independent subject and framing variables, but for our aim of simulating the dependencies between the independent and dependent variables it is a blunt tool.\n\nFor example, consider samples from N(0,1)^100. Let a dependent variable be linearly dependent on those variables with coefficients of {-1,1} for the first 99, and +10 for the 100th. Suppose a simulation of the dependent variable had sufficient fidelity that it reproduced a linear relationship with the exact same coefficients, except that the 100th was -10 rather than +10. The correlation between the simulated and real data would be ~0.0, whereas the sign agreement would be 99%.\n\n**Failure to characterize the countries**\n\nThis was an inept omission. In our final version, we will clarify that European (plus Israel) countries and languages are used, and note this limitation.\n\n**Failure to model values and cultural variance**\n\nRepresentation of these factors within LLMs is interesting and important. The Bos study has generated three publications with 300 citations between them showing that the relationships it investigates are of interest even without incorporation of these additional factors.\n\n**Modeling nationality when masked**\n\nIn exp-1 we compared nationality being masked vs stated - simulation fidelity increased from 59% to 71%. Based on this, in exp-2 and exp-3 nationality was stated. In exp-1, when the nationality was masked and all prompting was in English, the LLM does indeed have no cue to nationality and so gets the correct sign for the 30 country coefficients only randomly. It is still able to model the relationship between framing and relative deprivation variables, and ratings of persuasion and mobilization (the other 14 coefficients).\n\n**Averaging of Relative Deprivation (RD)**\n\nThis was a compromise to our modeling that got baked in that we thought could help improve model fidelity. We have re-simulated without this hack, and our conclusions are unchanged. The final version will reflect this new data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Incorporating and examining cultural information embedded in LLMs is a crucial task, our field increasingly needs accurate approaches for capturing cultural representations and the role of language in shaping them. The paper has this valuable motivation.\n\nThis paper makes a valuable point by investigating the correlation between various linguistic factors (e.g., distance from English) and cultural representation, exploring whether and how language influences cultural representation in LLMs.\n\nThe research design is noteworthy for its reliance on human data from diverse countries, adding validity to the study of this important question.", "reasons_to_reject": "The paper's approach to assessing the fidelity of LLMs in simulating human responses raises concerns. The chosen method, focusing on replicating regression results, seems inadequately justified. A simpler correlation analysis might have provided a more direct measure of how well the LLMs simulate individual responses.\n\nAdditionally, the paper omits mentioning that the data and experiments are limited to European countries and languages. While this doesn't necessarily invalidate the findings, it's a surprising omission that should have been acknowledged.\n\nThe generalization of findings to psychological traits is also problematic, as the dataset specifically examines the impact of two contexts on agreement with political arguments about a financial issue. A more appropriate dataset would have included psychological traits collected across diverse cultures, such as those found in the World Values Survey. Furthermore, while the paper discusses culture, it neglects to address the limited cultural variance within European countries.", "questions_to_authors": "The evaluation method assesses 44 coefficients, 30 of which are country-level (C_i^P, C_i^M). However, the LLM is evaluated on replicating these coefficients even though the country of residence is not provided in the prompt. Could you clarify whether the expectation is for LLMs to infer this information from the language and text of the prompt, and if so, what linguistic cues or patterns the LLM might be utilizing for this inference?\n\nThe relative deprivation appears to be a crucial independent variable in Bos et al.'s experiment. However, your paper mentions that \"the three relative deprivation ratings in our prompt were set to the same average of the actual three relative deprivation ratings by the human participant in the original study\" to enhance fidelity. Does this adjustment imply that LLMs struggle to capture the association between relative deprivation and persuasion (and mobilization), and if so, what are the potential reasons for this limitation and how does that impact the paper's main goal for replicating the human responses?", "extra_scores": {}}, {"review_id": "HZDTSj44xm", "reviewer": "Reviewer_sKd3", "summary": "This paper replicates a psychological experiment from Bos et al. (2020), where predefined human profiles are simulated to study their reactions to news stimuli. The authors have access to the original human responses of Bos et al. (2020)'s experiment, and prompt GPT3.5 to generate simulated profiles. The authors test various combinations of information given to the LLM during prompting, but the main goal of the study is to explore how the various \"metadata\" information can affect the output of the LLM. I refer to it as \"metadata\" even if it is not, it is more a set of additional information about the human profile: nationality and language. \nThrough a set of analysis and combinations of adding information about nationality, language, both, or neither, the authors explore how this affects the final prediction of the LLM. \nAs far as I understand it, the final prediction of the LLM is a simple numerical value as response to one question precedes by a longer prompt presenting the human profile, the relative deprivation values of the human, as well as a news article. \nThe paper is well written and interesting, but I have some doubts about some aspects of the motivation and claims. Please see weaknesses and questions for details.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We are encouraged that you find our work to be a nice contribution for grounding interdisciplinary research. We appreciate your acknowledgement of our presenting numerous experiments and analyses, and we are glad that you find the idea of our work interesting.\n\n**Just one numerical response is not enough to claim simulation of human behavior.**\n\nNot one. As described in section 3.5, we followed the Bos et al. study. Subjects (human and LLM) rated agreement with three statements to assess persuasion, and two to assess mobilization; each set of ratings was averaged to give two responses per subject. We make no claim to a general simulation of human behavior, just to the narrow but interesting aspect tested by the Bos et al. study which has generated three publications with ~300 citations between them.\n\n**Limitations of machine translation**\n\nWe believe that the ‘google translations’ we used are sense-preserving, but we accept that lack of human validation is a fair criticism. We are compiling validated translations and our final version will present data based on these. As an interim measure, we have re-run our simulation using superior GPT-4 translations. Our conclusions are unchanged in all important aspects.\n\n**Language used in exp-2?**\n\nWe wrote:\n“In the second experiment…the language of the prompt was varied. The simulation was run 12 times, each time simulating in the same language regardless of the nationality of the simulated participants. The languages used are the majority-spoken language of each country as shown in Table 3 in the Appendix.”\n\nTo be clearer we will insert:\n“On the first run all subjects, regardless of nationality, were prompted in English. On the second run all in French. On the third German. Etc.\"\n\n**Language used in exp-3?**\n\nIn exp-3 (main condition) subjects were indeed prompted in their native languages i.e. French subjects in French, German in German, etc. In the country-shuffled condition the pairing of nationality and language was randomly permuted, so (for example) Germans in French, Poles in Spanish, etc. In the fully-shuffled condition the pairing of subjects and language was randomly permuted, so some English were prompted in French, some in Polish, etc. and similarly for other nationalities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A nice contribution in the direction on grounding interdisciplinary research by replicating a study from psychology on LLMs.\n\n- The authors present a number of experiments and analysis of their results.", "reasons_to_reject": "- Some aspects of the paper are not clear to me, and the authors fail to give clear arguments to support some of their decisions (I am not sure if I understand some of the decisions made by the authors, please see questions).\n\n- While I think that the idea of the work is interesting. Some I am not sure if asking GPT3.5 to give one numerical value means that we have a system that effectively simulates human behavior. \n\n- During their experimentation, the authors relied on machine translation to translate their prompts. To assess the correctness of their translation they used GPT3.5. While this might work and yield good results, it does not equate human evaluation. The authors fail to discuss this aspect in their limitations section, nor discuss the effect this can have on languages with higher lexical distant to English (which is one of their findings, that for languages with a high lexical distant from English, using English prompts is better). Maybe this is mainly a side effect of bad translation? It is anyways not discussed by the authors, and I did miss this discussion.", "questions_to_authors": "- In experiment 2, I am not sure I understand the sentence \"the simulation was run 12 times, each time simulating in the same language regardless of the nationality of the simulated participant\". What is the single language used? Is this English?\n\n- In experiment 3: I am not sure I understand why you have approached this experiment in this way. If we want to test the effect of the language, shouldn't the prompt be change to the relative language of each country? Or is this to test if English yields the best results overall? I am not sure about the motivation here. \n\n- Did I get it correct that your experiments are based on one numerical value returned by GPT3.5?", "extra_scores": {}}, {"review_id": "Utc1dnfRLb", "reviewer": "Reviewer_KEfE", "summary": "The goal of this paper is addressing the research question “How to effectively incorporate language in LLMs?” as well as “How precise is the nationality trait depicted by multilingual LLMs?” towards personalized and inclusive LLMs. Here, a semi-complete questionnaire (adapted from Bos study) is provided where numbers should be selected for each question along with news stimuli to the participants. The study contains >7K participants across 15 countries.\n\n3 GPT-3.5-Turbo-1106 personas are created : 1) directly expressing the nationality 2) use of a side language (majority language of each country) 3) use of native language – Following each experiments, each participant’s persuasion and mobilization scores were collected while simulating the participant’s reaction to news stimuli.\n\nWhile modern LLMs have multilingual capabilities, there is bias due to shortage of data in other languages than English ( or other similar high-resource languages), this is a challenging problem specially when it comes to low-resource languages. Further, there are gender biases in certain languages that can be also present in English LLMs. Also, in tasks such as emotion detection, English LLMs perform better than other languages.\n\nLLM persona was introduced in 2023 where LLMs can showcase certain personality traits when interacting with an end-user. It is also shown that RLHF produces more liberal content than the original pretrained data that is more conservative.\n\n Experiment result was that languages that resemble English language the most have the higher agreement score and that translating to native language yields poor results. Further, providing the nationality of the participant as part of the prompt provides better results. Based on the experiments, better results are yielded when English is chosen as the language of prompting.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your positive and insightful feedback. We are encouraged that you find our multidisciplinary work beneficial to the community, especially in designing more inclusive multilingual LLMs. We are glad that you found our experiments comprehensive across a diverse and large scale. By simulating over 7,000 human responses given their profiles, we carried out three experiments to test: 1) the effect of explicitly mentioning nationality, 2) the effect across various countries, and 3) the effect of shuffling pairings of the country and its native language. We appreciate your acknowledgment of our testing efforts to preserve linguistic meaning in our prompt translations. We would like to address your questions and concerns below and incorporate your feedback in our final version.\n\n**Limitations of machine translation**\n\nWe believe that the ‘google translations’ we used are sense-preserving, but we accept that lack of human validation is a fair criticism. We are compiling validated translations and our final version will present data based on these. As an interim measure, we have re-run our simulation using superior GPT-4 translations. Our conclusions are unchanged in all important aspects.\n\n**Considering gender biases**\n\nWe have considered potential gender biases in two ways:\n1. Some languages may have gendered words. Bilingual speakers have checked our translations and have confirmed that the gendering follows standard translation patterns and does not include any contentious choices such as translating ‘the doctor’ to a masculine form.\n2. Following Bos et al., our prompts do explicitly state the gender of the simulated subject, and nothing else in the prompt refers to or indicates gender. Neither Bos nor ourselves have tested whether there are gender differences in the human data. Assessing whether LLM simulation reproduces any gender differences if they exist, or introduces new false ones, are interesting topics that we hope to pursue in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Authors have performed tests to ensure linguistic meaning of the survey was not lost during translation in different languages.\n\n- Comprehensive experiments across a diverse set of countries and a huge number of participants followed by detailed discussions.\n\n- This research paper provides insight in design multilingual LLMs that needs to be more inclusive.\n\n- The multidisciplinary nature of this work is beneficial to the community.", "reasons_to_reject": "I am pro accepting this paper specially due to its multidisciplinary effect.", "questions_to_authors": "I am not sure if using Google Translate would be best resource in this scenario since even in case of low-resource languages, Google Translate doesn't have the best performance. Could you please explain how you handled this situation?\n\nAlso could you please elaborate a bit on if you handled gender biases present in certain nationalities? or what would you propose in this regard?", "extra_scores": {}}, {"review_id": "keL8K88rzK", "reviewer": "Reviewer_aYm5", "summary": "Simulating human responses to a study from psychology using GPT-3.5 prompted with respondents demographic information, this study shows high fidelity, which is improved by also indicating the respondent's country, but deteriorates when using the respondent's language for prompting. Best fidelity is with English prompts, with lexical distance from English being negatively correlated with fidelity for other languages.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Bos's analysis followed ‘closely but not exactly’. Aren’t country coefficients relative to the mean of this specific set of countries?**\n\nBos used Poland as a reference, forcing its coefficient to 0. We instead express country coefficients relative to a zero-mean, allowing comparison for all countries. Since the country set is fixed, we do not see an issue with this. \n\n**Missing correlations**\n\nFidelity vs. internet presence (+0.22); fidelity vs. speaker population (-0.19).\n\n**Limitations of machine translation**\n\nWe believe that the ‘google translations’ we used are sense-preserving, but we accept that lack of human validation is a fair criticism. We are compiling validated translations and our final version will present data based on these. As an interim measure, we have re-run our simulation using superior GPT-4 translations. Our conclusions are unchanged in all important aspects.\n\n**Averaging of Relative Deprivation (RD)**\n\nThis was a compromise to our modeling that got baked in that we thought could help improve model fidelity. We have re-simulated without this hack, and our conclusions are unchanged. The final version will reflect this new data.\n\n**Arbitrary x-value for exp-3 results in fig-4**\n\nThe x-value is the average distance-from-English of the prompting languages used in exp-3. We will fix the omission.\n\n**Unnecessary anthropomorphism**\n\nWe will revise “exhibit effects akin” to “mimic the effect”.\n\n**Cannot generalize to LLMs since only GPT-3.5 was studied**\n\nWe never meant to make this claim, and will correct our wording to make this clear.\n\n**Bos study: language, story, country. Why differences in experiment 3**\n\nEach Bos participant completed a questionnaire in their native tongue, featuring a news story about their own country. In section 6 we conclude that GPT-3.5 does respond differently depending on the prompting language, but those differences are different from those seen in human participants. We are hesitant to speculate on the cause without sufficient evidence.\n\n**Method to mask nationality**\n\nNationality answers were omitted from the prompt. Within the article, ‘France’ (for example) was replaced by ‘the country’ or ‘our country’ and ‘French citizens’ by ‘nationals’. We will explain in our final version.\n\n**Other mentioned issues**\n\nThanks, we will correct them in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Addresses a topic of interest to a wide audience and societal impact, of cultural sensitivity of GPT.\n\nGrounded in psychological research and using data from robust experiments with human participants as a basis for comparison.\n\nResults are surprising and impactful as they reveal the capability of simulating human tendencies but confirm cultural biases in LLMs, and as simulating human responses with high fidelity across cultures is a useful task in itself.", "reasons_to_reject": "Lacking details about the experimental setup of this study as well as crucial details from the Bos et al. study. (see questions to authors). The authors follow the analysis \"closely, but not exactly\", but it's not clear how they're different. \n\nMissing details about geographical, population-related and linguistic factors in the analysis of experiment 2, and about the correlation obtained with them. \n\nTesting the validity of translation only with GPT-3.5 is insufficient (we don't know how reliable this is without correlating it with human judgments) and it's also unclear how exactly this is done. \n\nMissing context and citations for studies that also investigated the effect of prompting country/language on LLM fidelity in simulating human responses:\nhttps://arxiv.org/abs/2306.16388\nhttps://aclanthology.org/2023.c3nlp-1.7/\nhttps://aclanthology.org/2023.c3nlp-1.12/\n\nAveraging the relative deprivation ratings and replacing the original ratings seems arbitrary and unjustified. It is fair that it improves fidelity but what happens without doing this? \n\nThe results from experiment 3 de presented in figure 4 with seemingly arbitrary x value, which is confusing. If there's no meaning to the x value it should be in a separate subfigure. \n\nFraming the results as revealing human-like characteristics, and claiming \"LLMs exhibit persuasion and mobilization effects akin to humans\", is unnecessary anthropomorphism. They are only simulators. Also, \"LLMs\" is to general since only GPT-3.5 was studied.", "questions_to_authors": "Which language did participants in Bos et al.'s study were exposed to? What story did they read? The same as in the present study? What country was the story about? Their own? Given these details, can you speculate on why the prompting language yielded the opposite effect than that expected in experiment 3?  \n\nHow was the country masked in the present study? \n\nWhat is the justification to compare the sign of the country coefficients? Since they're set to have zero mean, the sign just means where each country lies relative to the mean of the countries in this specific set of countries, and may be very different if a different set of countries is selected. \n\nFigure 3 should be a table.\n\nP values should be given in decimals and not percents.\n\nGreek and Hebrew are not language families.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper explores the ability of a language model to understand and replicate human profiles from different nationalities. The work is novel, interdisciplinary, scalable, and addresses topics of rising interest to the community, such as pluralism in AI.
The authors are asked to include more clarity re the following in the camera ready.
1. Limitations of their approach, such as the use of unverified automatic translations, and that their study only covers European nations who might be more closely aligned in values and perspectives as compared to other diverse nations worldwide.
2. Choice of framework/study for moral values: Bos versus World Values Survey. Each have their pros and cons and need to be disucssed in the paper, rather than setting up an arbitrary choice.
3. Clarity of experimental setup, e.g., use of one numerical values returned by the model, languages used in the various experiments.

The authors are encouraged to overall increase the clarity in writing of their paper and explicitly state limitations. They are also asked to incorporate reviewer feedback into their final version.

[comments from PCs]
Technical aspects aside. This paper on the one hand says it’s focused on culturally aware LLM, which is interesting. But title is really focused on simulating human surveys using LLMs, which is the experiment they ran. The intro suggests this is the experimental methodology only, but this is not what the title says. Therefore, we observe a discrepancy in the methods used versus claims. It may be the case that the ‘embedding’ in the title is interpreted differently by authors. **We ask the authors to fix this contextualization and presentation issue, including, if needed, by modifying the title.** Please consider how the audience in the conference will interpret your title.
Decision: Accept