Quantifying the Persona Effect in LLM Simulations (2024)

Abstract: Large language models (LLMs) have shown remarkable promise in simulating human language and behavior. This study investigates how integrating persona variables—demographic, social, and behavioral factors—impacts LLMs’ ability to simulate diverse perspectives. We find that persona variables account for <10% variance in annotations in existing subjective NLP datasets. Nonetheless, incorporating persona variables via prompting in LLMs provides modest but statistically significant improvements. Persona prompting is most effective in samples where many annotators disagree, but their disagreements are relatively minor. Notably, we find a linear relationship in our setting: the stronger the correlation between persona variables and human annotations, the more accurate the LLM predictions are using persona prompting. In a zero-shot setting, a powerful 70b model with persona prompting captures 81% of the annotation variance achievable by linear regression trained on ground truth annotations. However, for most subjective NLP datasets, where persona variables have limited explanatory power, the benefits of persona prompting are limited.1

heading: 1 Introduction
text: Annotation questions such as “how do you feel emotionally after reading this text” are subjective - there are rarely definitive right or wrong answers (Ovesdotter Alm, 2011). This subjectivity is increasingly being recognized within the NLP community. Subjective NLP tasks are typically characterized by low inter-annotator agreement, making label aggregation inappropriate (Ovesdotter Alm, 2011; Plank, 2022; Cabitza et al., 2023). Previous research has established the significant influence of sociodemographic variables on the annotations of these tasks (Sap et al., 2022; Santy et al., 2023; Pei and Jurgens, 2023, inter alia).
1Code and data will be released at https://github. com/cambridgeltl/persona_effect
One approach to model these persona variables2
is to use LLMs. LLMs have been effectively utilized for role-playing and simulating human behavior, primarily by defining the persona of interest within the prompt (Aher et al., 2023; Horton, 2023; Kovač et al., 2023; Argyle et al., 2023). Their success has even spurred debates on whether LLMs could replace human subjects (Dillion et al., 2023; Grossmann et al., 2023). However, there are also concerns about such “persona prompting” methodology (Figure 1) (Beck et al., 2024), citing ecological fallacy (Orlikowski et al., 2023), and LLMs’ susceptibility to caricatures (Cheng et al., 2023), misportrayal and erasure of subgroup heterogeneity (Wang et al., 2024).
Existing studies have often sought to measure the effects of individual persona variables, overlooking a holistic analysis of the potential explanatory power of persona variables on annotation variance. It is then hard to contextualize the models’ ability to utilize persona information. To address this issue, our research explores the following questions: RQ1: How much variance in human annotation could persona variables explain? Understanding this will help us assess the overall influence of persona variables on human annotation, providing context to our subsequent investigations. We propose employing a linear regression analysis to predict annotations using persona variables and examine the resulting R2 values. We find that persona variables explain relatively little variance (<10%) for many NLP tasks (Section 3). This general framework can be useful in understanding the potential effectiveness of LLM simulations prior to conducting large-scale experiments when some amounts of human data are available.
2In our work, we adopt a broad definition of persona variables to include not only demographic and social variables but also other variables that could help describe a persona, such as variables relating to attitudes, behaviors, lived experiences, and values. It is worth noting that most NLP datasets have no information of any kind available about the annotators.
10289
RQ2: Can incorporating persona variables via prompting improve LLMs’ predictions? Building on RQ1, we assess how much the explained variance by persona variables translates into prediction gains in LLMs. We find that incorporating persona variables provides modest but statistically significant improvements (Section 4). RQ3: For what types of samples is persona prompting most useful? To better understand the utility of persona prompting, we examine its impact across sample types, in terms of annotation entropy and standard deviation. We identify that most gains occur in samples characterized by frequent annotator disagreements within a relatively narrow range (high entropy-low standard deviation), suggesting that models can adjust their annotation to suit the persona, though not drastically (Section 5). RQ4: How effectively can LLMs simulate personas when the importance of persona variables varies? Using a set of survey questions, where persona variables explain the responses to varying degrees, we apply persona prompting to LLMs. We find a linear relationship in our setting: the more persona variables are correlated with the outcome variable, the better LLMs predictions are using persona prompting. Large, preference-tuned models perform best and can explain up to 81% of variance found in human responses. However, when the utility of persona variables is low, persona prompting has little effect. Regrettably, most subjective NLP
datasets fall into this category, casting doubt on the efficacy of persona prompting in the current NLP context (Section 6). Similar methodologies could be applied across different domains to better understand the simulation capabilities of LLMs.

heading: 2 Related Work

heading: 2.1 The Relationship between Persona Variables and Annotation Outcome
text: The role of persona variables, such as demographics and lived experiences, in influencing annotations in NLP tasks is well established. Many studies have highlighted how persona variables affect tasks like hate speech detection (Kumar et al., 2021; Sap et al., 2022; Pei and Jurgens, 2023; Santy et al., 2023; Hettiachchi et al., 2023; Lee et al., 2023), sentiment analysis (Ding et al., 2022; Biester et al., 2022), and irony detection (Frenda et al., 2023). While these studies shed light on the subjectivity of NLP annotations in many tasks, they often stop short of a holistic account of the explanatory power of persona variables on annotation variance. By contrast, in social science, the impact of persona variables on attitude are long studied and quantified (Bobo and Licari, 1989; Bartels, 2002). In our work, we analyze the utility of the persona variables in explaining annotation outcomes across subjective NLP tasks.

heading: 2.2 Modeling Persona Variables and LLM for Simulation
text: Some studies make use of persona variables only to enhance the diversity of model output, often without a strict emphasis on the accuracy or fidelity of persona representation in the output (Hämäläinen et al., 2023; Park et al., 2023; Liu et al., 2024). Meanwhile, several other works emphasize the accuracy of persona representation and have sought to account for the differences between individual annotators or the group-level attributes of annotators through adding individual (group) specific layers (Mostafazadeh Davani et al., 2022; Gordon et al., 2022; Fleisig et al., 2023; Orlikowski et al., 2023, inter alia), or via prompting (Beck et al., 2024). Results from these studies have been mixed, with some work indicating success using grouplevel persona variables (Gordon et al., 2022; Fleisig et al., 2023), while others cast doubt on the effectiveness of such methods (Orlikowski et al., 2023; Cheng et al., 2023; Beck et al., 2024). Simultaneously, in the social sciences, a multitude of studies have been employing use persona prompts in LLMs to simulate human behavior (Horton, 2023; Argyle et al., 2023; Kim and Lee, 2023; Törnberg et al., 2023), while others have pointed out the lack of fidelity and diversity in such simulations (Bisbee et al., 2024; Park et al., 2024; Wang et al., 2024; Taubenfeld et al., 2024).
Our work builds on the uncertainty raised by these mixed results, focusing on the potential of persona prompting with LLMs for simulating different perspectives in NLP tasks, which is currently understudied. Furthermore, our work aims to isolate the evaluation of persona prompting from the impact of text samples in the modeling process, a separation that has not been much explored in previous studies.

heading: 2.3 Persona Prompting and AI Alignment
text: Apart from the research focused on incorporating demographic factors into NLP models and using LLMs for simulations, another line of studies has examined persona prompting in the context of AI alignment (Santurkar et al., 2023; Durmus et al., 2023). These studies have employed LLMs to answer multiple-choice survey questions concerning societal values and attitudes, comparing the LLMgenerated answer distribution with actual human response distribution derived from survey data representing diverse demographic groups. In contrast
to these studies, our work aims to explore the efficacy of LLMs in leveraging persona variables to inform task predictions, rather than the degree to which LLM responses to survey questions mirror those of specific demographic groups.

heading: 3 RQ1: How much variance in human annotation could persona variables explain?
text: Methodology Given the relative gap in literature in a holistic understanding of the impact of persona variables on annotation variance, we investigate to what extent persona variables explain human annotation variance. This analysis would provide valuable context to any modeling exercise of incorporating persona variables.
We employ a mixed-effect linear regression model3 to assess how much variance in annotation can be explained by persona variables (fixed effect), while controlling for the text-specific variability in the text sample (random effect) by fitting a random intercept for each text. Using a mixed-effect linear regression allows us to separate the impact of persona variables from the inherent variation of the text being annotated. We also consider incorporating an additional random effect term to capture individual annotator differences; however, the fixed effect estimates are very similar. Consequently, we opt to include only a random effect for the text sample. We evaluate 10 subjective NLP datasets which provide unaggregated annotations and annotator persona variables. We also consider the presidential vote question in the ANES 2012 public opinion survey (ANES), in which every human subject answers the same question and therefore does not require a text random effect, for comparison.
Results We show a comparison of the tasks, sources of data, annotation methods, sizes, types of persona information included, and the regression R2 values in Table 1.
We observe that the datasets mostly come from social media sources and annotations are collected through crowd-sourcing. They vary substantially in size, persona variables provided and R2 values. While persona variables (fixed effect) do significantly explain some variance in annotation outcomes, they account for just 1.4%-10.6% of the total variance (Marginal R2), even when controlling for text variation. Conversely, variabil-
3In R notation, annotation ∼ persona variables + (1 | text_id)
Task Dataset Data Source Annotation Size Persona Variables
R2Cond. R 2 Marg.
Toxicity Detection annWithAttitudes (Sap et al., 2022)
Twitter 5-point MTurk
N=626 A=5.5a
U.S.
Basic Attitude
0.611 0.045
Offensiveness Rating POPQUORN (Pei and Jurgens, 2023)
Reddit 5-point Prolific
N=1,500 A=8.7 U.S.
Basic 0.319 0.029
Politeness Rating POPQUORN (Pei and Jurgens, 2023)
Email 5-point Prolific
N=3,718 A=6.7 U.S.
Basic 0.454 0.014
Toxicity Detection Kumar et al. (2021) Twitter Reddit 4chan
5-point MTurk
N=106,035 A=5.1 U.S.
Basic Attitude Behavior
0.349 0.106
Sentiment Analysis Diaz et al. (2018) Twitter 5-point N=14,071 A=4.2 U.S.
Basic Attitude
0.329 0.036
Social Acceptability Social-Chem-101 (Forbes et al., 2020)
Reddit 5-point MTurk
N=9,740 A=6.1
Mostly U.S.
Basic 0.432 0.097
Social Acceptability NLPositionalityb
(Santy et al., 2023) Reddit 5-point Opt-in volunteer N=291 A=50.2
87 countries
Basic 0.513 0.005
Toxicity Detection NLPositionalityb
(Santy et al., 2023) Twitter 3-point Opt-in volunteer N=299 A=29.6
Basic 0.432 0.017
ity inherent to individual texts (random effect) can explain up to 70% of the total variance, i.e. ∼ (Conditional R2 − Marginal R2). For comparison, in the ANES dataset, persona variables explain more than 70% human response variance.
The marginal R2 values provide a baseline indication of the variance in annotations that persona variables could explain. The regression model assumes a linear relationship between persona variables and annotation and does not consider any interaction between the persona variables. There-
fore, while it is straightforward and interpretable, for LLMs, it should be considered a weak baseline.
Acknowledging that a substantial portion of variance remains unexplained (25%-70%) by either the text or persona variables across all tasks considered is crucial. This unexplained variance could be attributed to theoretically measurable persona factors such as personality traits and complex moral and political beliefs, which are not currently collected in existing datasets. Additionally, it could be due to hard-to-measure factors like the anno-
tators’ lived experiences, interpersonal dynamics, and other personal variables.
The elevated R2 value in the ANES dataset may be attributed to the escalating degree of polarization in U.S. politics in recent years. This rise in polarization has lead to more predictable voting patterns (Pew Research Center, 2014) and the increasing tendency of U.S. voters to behave in a manner consistent with their in-groups (Graham and Haidt, 2010).
In contrast, tasks such as assessing the hatefulness of a tweet offer more room for personal interpretation, leading to diverse opinions. Thus, persona factors may account for a lesser portion of the variance in annotation for such tasks.
We argue that regression analysis offers a valuable framework for setting realistic expectations regarding the fidelity of persona prompting with LLMs. Specifically, when some level of annotated data is available, this approach offers preliminary insights into potential simulation results, allowing researchers to gauge the likely performance of persona-prompted LLMs for a new application. This then enables an informed decision-making process before committing to costly large-scale simulation runs.

heading: 4 RQ2: Can incorporating persona variables via prompting improve LLMs’ predictions?
text: Methodology Since persona variables can explain a small but significant amount of human annotation variations, we then explore whether persona prompting would improve LLM’s predictions.
As depicted in Figure 1, we prepend each text sample with persona variables in a zero-shot prompting setup. We prompt the LLMs twice: once with persona variables, and once without, to zeroshot predict individual annotations on AnnotatorwithAttitude (Sap et al., 2022), Kumar et al. (2021), EPIC (Frenda et al., 2023) and the politeness rating task in POPQUORN (Pei and Jurgens, 2023). We preserve the original language of the persona descriptions to the extent possible, adopt a multiplechoice format, include a description of the question and the answer choices, and predict only the next token as the model’s response, as done in prior work (Santurkar et al., 2023; Durmus et al., 2023). Due to cost constraints, we sample 600 instances from each dataset. The details of the prompt format are provided in the Section B.1.
We additionally perform a set of robustness experiments by swapping the order of persona variables in the prompt or paragraphing the language used to describe each persona variables and repeat the experiments on Kumar et al. (2021). The detailed setting can be found in Section D.
To evaluate, we compare model predictions with individual human annotations using R2 value, Cohen’s Kappa (Cohen, 1960), mean absolute error (MAE) for multi-class classification or macro F1 score for binary classification, given the class imbalance in Frenda et al. (2023). Our primary objective is to observe the performance changes induced by persona prompting, rather than focusing on the absolute performance of each model.
Result We show the results in Table 2. The first row shows the “Target” R2 values, which refer to the conditional (and marginal) R2 value of the mixed-effect regression on the sampled data computed as in Table 1, while the R2 in subsequent rows are from a fixed-effect linear regression predicting the human annotation with model predictions4. While these two R2 values cannot be compared directly, the “Target” R2 gives context to the fixed-effect R2 values. To evaluate the statistical significance of performance differences between models incorporating and excluding persona variables, we conduct a bootstrap analysis (Efron, 1992) with 1,000 replications. We denote with asterisks those instances where incorporating persona variables leads to statistically significant performance improvements. For the overall improvement (last row), we aggregate predictions from the 6 models and apply the same bootstrapping procedure to assess the collective effect of persona variables. As the 7b and 13b models exhibit much weaker performance, we only feature results from 70b models in the main text, while the results from smaller models are included in Table 4.
At the aggregate level, persona prompting provides varying levels of statistically significant improvement across at least one metric in each of the four datasets. However, these improvements are generally modest. For instance, in EPIC, where persona variables could explain up to 9% of annotation variance, persona prompting only provides 1% gain on average. The effectiveness of persona prompting also varies across models: for each dataset, persona prompting improves the performance of some models but not others, echoing the results
4in R notation, annotation ∼ prediction
in Beck et al. (2024). We note that overall, with and without persona prompting, GPT-4 consistently outperforms all other models in every task. Tulu-2 models outperform Llama-2 with performance on par with GPT-3.5. The Llama-2 models are, on the other hand, much more sensitive to persona variables, arguably to an excessive degree. For example, on AnnotatorwithAttitudes, persona prompting improves the R2 by as much as 0.23 even though persona variables only has a marginal Target R2 of 0.03. We show the robustness experiment result in Table 6. The model performances are consistent across variations in the ordering and language use of the persona variables.

heading: 5 RQ3: For what types of samples is persona prompting most useful?
text: Methodology To better understand persona prompting as a technique, we aim to investigate its effectiveness on data samples with varying degrees of annotation entropy and standard deviation. We focus on Kumar et al. (2021), as persona variables play a relatively more important role in explaining annotation variances in this dataset.
We create a new subsample of the dataset with four categories: low entropy-low standard deviation (most annotators agree with one another and the magnitude of the disagreement is small, e.g. 1, 1, 1, 1, 2); low entropy-high standard deviation (e.g. 0, 4, 4, 4, 0); high entropy-low standard deviation (e.g. 1, 1, 2, 2, 3); and high entropyhigh standard deviation (e.g. 0, 1, 2, 3, 4). The low/high division is based on the medians of en-
tropy and standard deviation. Then, we further stratify samples from each category into four bins according to their average annotation value. We then randomly sample 150 from each bin, culminating in a total of 600 samples per category. This approach is implemented to mitigate extreme class imbalances within certain categories. For instance, the low entropy-low standard deviation category would predominantly include samples with a rating of 0 (Not at all toxic). We then run the LLMs twice, once with persona prompting, once without, in the same setting as described in Section 4, on Llama-2-70b, Llama-2-70b-chat, Tulu-2-70b, and Tulu-2-dpo-70b.
Result We show in Figure 2a the mean improvement in MAE between models with and without persona prompting, averaged across the four models, in each of the four categories, with darker color indicating a greater degree of improvement in predictions when persona prompting is used. To reduce the possibility of finding a dataset-specific effect, we also repeat the same experiment on POPQUORN-Politeness dataset (Pei and Jurgens, 2023), and show the same plot Figure 2b.
Our findings indicate that including persona information leads to only slight changes in the model’s predictions for data with low entropy. This is as expected - with or without persona prompting, a capable LLM should already capture the consensus among annotators if there is one, thus only necessitating minor adjustments to individual predictions.
On the contrary, we observe larger shifts in pre-
diction when annotations have high entropy but low standard deviation. These instances often involve substantial disagreement among individuals, though within a small margin. The integration of persona variables may then enhance the model’s ability to refine its predictions. An example in this would be a prediction transition from 3 (without persona variables) to 4 (with persona variables).
However, when both entropy and standard deviation are high, the task of adjusting predictions based on persona information becomes considerably more challenging, as this would require significant shifts in the predicted values from the “mean” level, when no persona variables are provided. For instance, imagine a case where a prediction needs to change from 0 (without persona variables) to 4 (with persona variables).
While varying in magnitude, the MAE improvements when including persona variables are significant in all four categories in both datasets, based on bootstrapping 1,000 runs. Additionally, to determine whether there are statistically significant differences in improvements across the four categories, we perform a one-way ANOVA and find significant differences in the improvements. Finally, we conduct Tukey’s range test for each pair of categories. For Kumar et al. (2021), the high entropylow standard deviation category statistically significantly outperforms the other three categories, while the differences between the other three categories are not significant. For the POPQUORNPoliteness dataset, the high entropy-low standard deviation setting consistently shows the most improvement. While not all comparisons reach statistical significance, it never performs significantly worse than any other category. High entropy-high standard deviation and low entropy-low standard deviation show some variation in performance in terms of statistical significance, but neither shows more improvements than high entropy-low standard deviation setting. Low entropy-high standard deviation consistently yields statistically the least MAE improvements in all comparisons.

heading: 6 RQ4: How effectively can LLMs simulate personas when the importance of persona variables varies?
text: Motivation Within the context of NLP annotation, both the text sample and the persona variables may vary across instances (Figure 1). Both factors, along with their interactions, could potentially in-
fluence model predictions. To understand the models’ capacity for simulating different perspectives with persona prompting, we designed a case study that minimizes the impact of the text sample.
Methodology We use the ANES dataset (ANES), a comprehensive U.S. national-level election survey, as a data source for this section. This dataset offers a wealth of persona variables from a large sample of respondents. From the perspective of NLP annotation, surveys can be seen as having a large number of individuals (typically >1,000) annotating a small number of sentences, each representing a question. One key difference is that the survey questions, carefully crafted and tested by seasoned professionals, are designed to eliminate ambiguity common in social media-based NLP text annotation datasets. Therefore, by running experiments on the ANES dataset, we can minimize the impact of the randomness in the text samples.
We select a number of questions from ANES 2012 as the text sample, or the questions to be predicted, using a fixed set of persona variables. We ensure that these questions have varying predictability from persona variables, indicated by R2 values. Further details of the dependent and independent variables considered are included in Section C. After filtering out respondents who did not answer some of the questions of interest and performing random downsampling, we arrive at a sample size of 600 human respondents and 21 questions. Each question is paired with two levels of persona variables, resulting in 42 combinations of persona variables and questions, each with a different level of target R2 (see Section C for details). We then run the LLMs with persona prompting.
We also perform a robustness check with the presidential vote prediction question from ANES by swapping the order of persona variables in the prompt or paragraphing the language used to describe each persona variables. The detailed setting can be found in Section D.
Result We visualize the relationship between the predicted and target R2 values in Figure 3 of Tulu2-70b-dpo and Llama-2-7b-chat. The results for other models are provided in the Figure 4. Each point in the scatter plot represents an experiment result, where the x-coordinate signifies the target R2 and the y-coordinate denotes the predicted R2. The line Y = X is also included to represent the maximum linear regression model performance, where
High SD
Low SD
High Entropy Low Entropy Entropy
S D
0.15
0.20
0.25
0.30
Improvement (MAE)
(a) Kumar et al. (2021)
High SD
Low SD
High Entropy Low Entropy Entropy
S D
0.06 0.07 0.08 0.09 0.10 0.11
Improvement (MAE)
(b) POPQUORN-P
Figure 2: Mean improvement in MAE with persona prompting across four 70b models in annotations characterized by low/high entropy and standard deviation, with darker colors denoting more substantial improvement in predictions.
y = 0.81x − 0.09 R2 = 0.74
0.0
0.2
0.4
0.6
0.8
0.0 0.2 0.4 0.6 0.8
Target R2
P re
di ct
ed R
2
Perfect Prediction Tulu−2−dpo−70b
(a) Tulu-2-dpo-70b
y = 0.36x − 0.05 R2 = 0.48
0.0
0.2
0.4
0.6
0.8
0.0 0.2 0.4 0.6 0.8
Target R2
P re
di ct
ed R
2
Perfect Prediction Llama−2−7b−chat
(b) Llama-2-7b-chat
Figure 3: Comparison of predicted R2 and target R2. Each point in the X-Y plane represents an experimental result with persona prompting, where the x-coordinate signifies the target R2 and the y-coordinate denotes the predicted R2. We then fit a linear regression line and also plot the maximum linear regression model performance line y = x in the same figure.
predicted R2 equals target R2. We additionally fit a linear regression line to the data points and show the fitted equation and R2 in the figure.
Our results show a positive correlation between the target and predicted R2 values - the higher the target R2 value, the higher the predicted R2. Tulu2-70b-dpo, one of the best-performing models on the 70b scale, can capture 81% of the target R2. The other 70b models, except for the base model Llama-2-70b (Figure 4), have similar simulation capabilities, while the smaller models (7b and 13b) do much worse. However, it is important to note that no model surpasses the y = x line, suggesting that persona prompting still falls short compared to a trained linear model. The complex relationship
between persona and target variables, including interaction terms, implies that the true target R2 is likely much higher. Furthermore, even the best models fail to utilize the persona information effectively when target R2 is low, especially when R2 < 0.1. Nevertheless, when persona variables are sufficiently predictive of target variables, persona prompting can lead to somewhat accurate simulations on large models.
Considering that most existing NLP datasets, as discussed in Section 1, have marginal R2 < 0.1, we argue that persona prompting cannot reliably simulate different perspectives within existing NLP tasks. This finding may explain the modest gain of persona prompting observed earlier in
Section 2 and in Beck et al. (2024). We propose two potential explanations as to why LLMs, however powerful they are in other tasks, may be deficient in simulating diverse perspectives:
1) The persona variables typically accessible to researchers are group-level, while people form their identity based on both individual and grouplevel characteristics (Marsden and Pröbster, 2019). Therefore, there could be an inherent mismatch between the group-level variables we provide and individual perspectives we aim to simulate.
2) LLM generations can be understood as simulating the medium of a group, rather than individuals (West et al., 2023). Therefore, LLMs can have the tendency to represent a group as a monolith in simulation, thereby failing to capturing the inherent within-group heterogeneity (Wang et al., 2024). While using more fine-grained group-level persona variables may in theory bring us closer to individual ratings, it remains to be seen whether this could lead to true individualization in practice.
We show the robustness experiment result in Table 6. The model performances are consistent across variations in the ordering and language use of the persona variables descriptions.

heading: 7 Conclusion and Recommendation
text: Our study reveals that persona variables account for less than 10% of variance in human annotations across most NLP datasets we consider. The use of persona prompting offers modest yet significant improvements across different tasks. The improvement is most pronounced in cases where the annotators largely disagree but only by a small margin (high entropy-low standard deviation). By running a case study with U.S. opinion survey data, we uncover a linear relationship between target and predicted R2 values. Alarmingly, when the target R2 value falls below 0.1, the predicted R2 often drops to zero. This could explain the small improvements observed in NLP tasks with persona prompting, as existing datasets often have R2 values smaller than 0.1.
Based on these insights, we offer the following recommendations:
1) Exercise Caution in LLM-Based Simulations: In light of our findings, we advise caution for researchers intending to use LLMs for simulation purposes, especially in NLP tasks where persona variables’ influence is likely weak (low target R2). If the goal is to merely improve model generation
diversity, without prioritizing the fidelity of the model output towards the persona variables, applying persona prompting as is may suffice. However, if the goal is to faithfully simulate human behavior, achieving high fidelity could be challenging. Unvalidated, zero-shot simulations with LLMs may not yield reliable results. Therefore, thorough validation and potentially fine-tuning are essential to ensure simulation fidelity.
2) Implement More Strategic Dataset Design: The collection of persona information should be driven by clear objectives. If the aim is to understand how different groups annotate data, collecting only demographic information might be adequate but limited in scope for generalization of findings beyond the specific dataset. For behavioral simulation, a careful selection of persona variables is needed to increase the target R2 and achieve better predictability. Future datasets could include more nuanced and targeted questions probing individual characteristics such as attitudes, beliefs, and behaviors. Moving forward, it is crucial to expand dataset collection efforts to encompass diverse cultural perspectives and multiple languages, especially those from non-U.S. contexts, to make language technologies more equitable globally.

heading: 8 Limitations
text: We recognize the inherent subjectivity in human behavior and the multitude of contextual factors that influence decision-making, many of which are difficult to quantify. While incorporating more fine-grained persona variables could potentially reduce error margins significantly, some level of error is likely to persist. Furthermore, when collecting fine-grained persona information, researchers should carefully consider the ethical implications. In crowdsourcing environments, there is also a risk of obtaining intentionally inaccurate responses to sensitive questions (Huang et al., 2023).
While we exerted considerable effort to include a diverse range of datasets, the vast majority of available datasets with persona information from annotators have been collected in the U.S., featuring persona questions primarily relevant to this particular context. Consequently, we can only speculate about the effectiveness of persona prompting for questions that are specifically tailored to other countries. Furthermore, to the best of our knowledge, as of the time of writing this paper, we have not identified any publicly available datasets
that include annotator persona variables in a language other than English. Considering that even the most sophisticated LLMs still exhibit significant performance disparities between English and nonEnglish languages (Ahuja et al., 2023), it is highly probable that the ability of LLMs to simulate different perspectives based on persona information is considerably weaker in non-English languages. Additionally, many terms used to denote identities are deeply rooted in specific cultural and societal contexts, which cannot be readily translated into other languages. Thus, it is crucial to evaluate the simulation capabilities of an LLM independently for each language, without translation.
The zero-shot simulation ability of LLMs largely depends on their extensive training data, essentially a compressed digital snapshot of the internet. However, previous studies have indicated that the pretraining corpora used by LLMs are riddled with various social biases (Gao et al., 2020; Dodge et al., 2021; Bailey et al., 2022; Hu et al., 2023, inter alia). Consequently, LLM simulations could potentially be tainted by biases and stereotypes, among other issues.
We did not carry out extensive prompt engineering due to computational limitations and the targeted scope of our study. Instead, we presented the same prompts with persona information using language that closely mirrors how questions were asked of human participants. We believe this constitutes a fair setting for comparing LLMs. Additionally, we conducted a robustness check and found little variation for different persona variable orders and the exact wordings used to describe each variable (Section D).

heading: 9 Ethical Considerations
text: We utilize persona variables from publicly available datasets, which have been anonymized prior to their release. Therefore, no human participants were involved or personal data collected in this study. The research acknowledges the potential risks associated with the use of LLMs for simulation purposes, including issues such as identity fraud and manipulation. We sternly denounce such nefarious applications of this technology. We also acknowledge the concerns related to categorizing individuals into different demographic groups. However, we argue that our study merely utilizes existing datasets and does not involve any original data collection. Furthermore, the categorizations employed within
these datasets adhere to established best practices, such as those used by the U.S. Census Bureau, thereby ensuring their appropriateness. In addition, the use of these demographic categories is only aimed at understanding and demonstrating the potential for LLMs to simulate diverse perspectives.

heading: Acknowledgements
text: T.H is supported by Gates Cambridge Trust (grant OPP1144 from the Bill & Melinda Gates Foundation). This work was performed using resources provided by the Cambridge Service for Data Driven Discovery (CSD3) operated by the University of Cambridge Research Computing Service (www.csd3.cam.ac.uk), provided by Dell EMC and Intel using Tier-2 funding from the Engineering and Physical Sciences Research Council (capital grant EP/T022159/1), and DiRAC funding from the Science and Technology Facilities Council (www.dirac.ac.uk). We thank Chen Cecilia Liu, Songbo Hu, Alan Ansell, Philipp Borchert, Manoel Horta Ribeiro, Dirk Hovy, Zihao Fu, Yara Kyrychenko, Yijiang Dong, Shun Shao, Wen Wu, Meiru Zhang and Yinhong Liu for helpful feedback and discussions at various stages of the project.

heading: B Supplementary Results for Section 4
text: B.1 Prompt Template
AnnwAttitudes
**Your Profile**: You race/ethnicity is %s. Your gender identity is %s. You are %s years old. In terms of your political orientation, on a fivepoint Likert scale of -1 (Strongly liberal) to 1 (Strongly conservative ), you are %s. **Question**: How toxic/hateful/ disrespectful or offensive does this post seem to you? (Note: The
purpose is to understand how disrespect/offense can show up in language, not making statements about the actual content of the posts.) "%s"
(A) not offensive at all (B) mildly inappropriate (C) somewhat offensive (D) moderately offensive (E) very offensive **Answer**: (
Kumar et al. (2021)
**Your Profile**: You race is %s. Your gender is %s. You %s describe yourself as transgender. You consider yourself to be %s. You are %s years old. The highest degree or level of school that you have completed is %s. Religion is %s in your life. Generally speaking, you consider yourself politically %s. You %s a parent. You %s use social networking sites (Facebook, Twitter) . You %s use video sites (YouTube,
5https://github.com/openai/openai-python 6https://github.com/TimDettmers/bitsandbytes
Twitch). You %s use news sites (CNN, Fox, NYT, WSJ). You %s use community forums (Reddit, Craigslist , 4chan). In general, you rate the impact of technology on peoples lives as %s. Based on your experience, toxic posts or comments are %s.
**Question**: How toxic do you personally find this comment? "%s" (A) Extremely toxic (B) Very toxic (C) Moderately toxic (D) Slightly toxic (E) Not at all toxic **Answer**: (
EPIC
**Your Profile**: You ethnicity is %s. Your gender is %s. You are %s years old. Your country of birth is %s. Your country of residence is %s. You are a national of %s. You %s a
student. You are %s. Irony is a figurative language device
that conveys the opposite of literal meaning, profiling intentionally a
secondary or extended meaning. For instance, message: "if ur homeless u probably
wouldn’t have a phone." reply: "Yes, and all your belongings
would be in a handkerchief tied at the end of a stick." --> irony: yes
message: "if ur homeless u probably wouldn’t have a phone." reply: "Yes, you’re right."--> irony: yes **Question**: Is the reply ironic in the following message and reply pair? message: "%s" reply: "%s" (A) Ironic (B) Not ironic **Answer**: (
POPQUORN-P
**Your Profile**: In terms of race or ethnicity, you are %s. You are a %s. You are %s years old. Occupation-
wise, you are %s. Your education level is %s.
**Question**: Consider you read this email from a colleague, how polite do you think it is? **Email:**: "%s" (A) not polite at all (B) barely polite (C) somewhat polite (D) moderately polite (E) very polite **Answer**: (
B.2 Persona Variables
We list the persona variables used in Section 4 in Table 3. To assess the importance of each persona
variable, we conducted a “leave-one-out” experiment. In this experiment, we initially fit a regression model using all persona variables. We then iteratively remove each variable and report the change in R2 value as the importance of each person variable.
B.3 Results from All Models Due to space constraints, we could not include the results from all the models in the main text. Here, we present the full results in Table 4. Our analysis indicates that smaller models (7b and 13b) generally exhibit weaker performance compared to their larger counterparts, both with and without persona prompting.

heading: C Supplementary Results for Section 6
text: We include a list of the target variables we considered in Section 6 in Table 5. The persona template used are:
Prompt 1:
**It is 2012. Your Profile**: Racially, you are %s. You are a %s. You are %s years old. Ideologically, you are %
s. Politically, you are %s. It makes you feel %s when you see the American flag flying. You %s attend church. You are %s interested in politics and public affairs.
Prompt 2:
**It is 2012. Your Profile**: Racially, you are %s. You are a %s. You are %s years old. It makes you feel %s
when you see the American flag flying. You %s attend church. You are %s interested in politics and public affairs.
We additional include the Predicted R2 - Target R2 plot for all models in Figure 4.

heading: D Robustness Test
text: For Kumar et al. (2021) and ANES, we conduct a set of robustness checks. Specifically, we alter the order of the persona variables in the prompt across five configurations (Order 1-5) or use GPT-4 to come up with five distinct paraphrases of the prompt template, each intended to maintain the same semantic meaning (Semantics 1-5). The results are shown in Table 6. While there are variations between each Order or Semantics setting, the variations are minimal.
Dataset Feature Importance (∆R2)
AnnwAttitudes Race 0.0066 AnnwAttitudes Political leaning 0.0048 AnnwAttitudes Gender 0.0175
Kumar et al. (2021) Race 0.0243 Kumar et al. (2021) Gender 0.0037 Kumar et al. (2021) Political affiliation 0.0178 Kumar et al. (2021) Impact of Technologya 0.0070 Kumar et al. (2021) Parental status 0.0113 Kumar et al. (2021) Education 0.0093 Kumar et al. (2021) Age range 0.0041 Kumar et al. (2021) Uses social media sitesb 0.0088 Kumar et al. (2021) Uses news media sitesb 0.0001 Kumar et al. (2021) Religion importantc 0.0128 Kumar et al. (2021) Toxic Content a Problemd 0.0131 Kumar et al. (2021) Uses community forumsb 0.0033 Kumar et al. (2021) LGBTQ status 0.0187 Kumar et al. (2021) Identify as transgender 0.0109 Kumar et al. (2021) Uses video sitesb 0.0100
EPIC Ethnicity 0.0037 EPIC Sex 0.0091 EPIC Age 0.0002 EPIC Country of birth 0.0060 EPIC Country of residence 0.0272 EPIC Nationality 0.0150 EPIC Student status 0.0138 EPIC Employment status 0.0187
POPQUORN-P Race 0.0044 POPQUORN-P Gender 0.0008 POPQUORN-P Age 0.0194 POPQUORN-P Occupation 0.0038 POPQUORN-P Education 0.0020
a In general, how much impact do you think technology has on people’s lives? b Adapted from the following question: What types of sites do you use? [Checkbox] ◦ Social Networking (Facebook, Twitter) ◦ Video (YouTube, Twitch) ◦ News (CNN, Fox, NYT, WSJ) ◦ Community Forums (Reddit, Craigslist, 4chan) ◦ Email or messaging (Gmail, WhatsApp, Facebook Chat) c How important is religion in your life? d Based on your experience, to what degree are toxic posts or comments a
problem?
Table 3: Persona variables considered and their importance scores. The importance score for each variable is calculated as the difference in the R2 value before and after removing that variable, using a leave-one-out regression approach. Note that while more persona variables are available in some datasets, they were not included in our study due to prompt formatting limitations.

heading: Future Work
text: Future datasets could include more nuanced and targeted questions probing individual characteristics such as attitudes, beliefs, and behaviors. Moving forward, it is crucial to expand dataset collection efforts to encompass diverse cultural perspectives and multiple languages, especially those from non-U. S. contexts, to make language technologies more equitable globally