How well do simulated populations with GPT-4 align with real ones in clinical trials? The case of the EPQR-A personality test

Authors: gregorio  ferreira
jacopo  amidei
rubén  nieto
andreas  kaltenbrunner
Venue: KDD_org_2024_Workshop_AIDSH
Published: 2024-06-04 16:22:28

Abstract: In this paper, we test if GPT-4o can simulate populations for clinical trials. We performed two experiments, with the Eysenck Personality Questionnaire-Revised (EPQR-A) in three different languages (Spanish, English, and Slovak). Our results show that GPT-4o displays specific personality traits which may vary depending on different parameter settings and questionnaire language. Furthermore, the question of whether simulated populations (mimicking real ones) can be created and used for testing questionnaires is still inconclusive. 
 While we find encouraging results in some personality traits and differences between genders and study fields, we also observe that results for the virtual population answering the questionnaire differ from the ones found in real populations. Accordingly, further research is needed to test how to reduce the differences between virtual and real populations.

Main paper: 
 introduction: 
 In this paper, we explore to what extent Large Language Models (LLMs) can mimic human characteristics and if they can help researchers and/or healthcare professionals simulate populations for testing surveys or questionnaires. Questionnaires and surveys are efficient research methods for acquiring information about individuals and are beneficial for unearthing information that is not directly measurable.Defining sound surveys and questionnaires is not reduced to providing a sequence of questions. The overall structure, flow, coherence, and adequacy of the questions have to be taken into account [41] . The two most important and time-consuming steps AIDSH-KDD'24, August 26, 2024, Barcelona, Spain 2024.when defining a questionnaire are the pre-test (or pilot test), and the test of psychometric properties (for example, reliability and validity). Both steps are complex, since they involve the computation of multiple indices and applying the questionnaire to different populations. LLMs could simulate such populations for testing surveys or questionnaires, facilitating these two steps and reducing time consumption.To advance on that, we studied to what extent GPT displays a specific personality pattern and can simulate populations that could be used to assess questionnaires. In particular, we used the EPQR-A questionnaire [12] that was designed to measure Extraversion (degree of being outward, and socially engaged), Neuroticism (degree of emotional stability), and Psychoticism (degree of impulsiveness, difficulty in accepting and following rules). This questionnaire is particularly compelling for our analyses because it is widely used in psychology and has been translated into several languages, allowing us to perform multi-language experiments.In detail, our study aims to: 1) Test the baseline personality of GPT by using different experimental presentations and language versions (English-EN, Spanish-SP, and Slovak-SK) of the questionnaire. 2) Test the properties of the questionnaire in a simulated population sample -mimicking the characteristics of the real population from [15] and, 3) Check if specific groups of virtual personas (i.e. type of studies and genders) match the expected personality for them (taking into account available literature). 
related work: 
 The recent surge in employing psychology methodologies within the LLMs framework (see for example [20, 22, 27] ) has led [19] to coin the term machine psychology. Similarly, [37] introduces the term AI Psychometrics. According to Hagendorff, machine psychology seeks to uncover emergent abilities in LLMs that traditional natural language processing benchmarks cannot detect. Thus, the term aims to encompass various approaches that utilize psychological methods to analyze LLMs' behavior under one umbrella.Among the various directions that contribute to the development of machine psychology, a popular one is studying the personality of LLMs. For example, the EPQR-A were used by [13] whereas the Big Five factors [9] were used, among others, by [26, 38, 37, 33] to quantify the personality traits of LLMs. Similarly, IPIP-NEO [17] was used in [38] , the Myers-Briggs Type Indicator (MBTI) [4] test was used in [28] , and Short Dark Tetrad (SD4) [36] was used in [37] . In a slightly different fashion, [18] investigates LLM´s behavioral profile in a dynamic context instead of a static one. While the outcomes of the aforementioned studies may vary depending on the LLMs and questionnaires used, there is enough support to draw the promising and optimistic conclusion that personality assessments for LLMs are valid and reliable. These findings hold significance, considering that personality tests are tailored for humans, and there is no guarantee beforehand that they will yield valid and reliable results for LLMs.A further endeavour, pursued by researchers such as [26, 24, 38] , involves exploring the potential for adjusting LLMs' personalities. The objective is to study whether tailored prompts can push LLMs to replicate human personality traits. Encouragingly, findings from these papers suggest that LLMs can indeed be molded to imitate particular personality profiles.A more general direction has brought researchers to utilize LLMs to create simulated humans, serving as experimental participants, survey respondents, or other agents. Simulated populations can streamline experiments, reduce time and costs, and can be utilized in studies unsuitable for human involvement. This concept, referred to by various names such as guinea pigbots [23] , silicon samples, or homo silicus [21] , has found application across various domains within social science. For example, the employment of LLMs as substitutes for human participants was studied in psychological research [23, 10, 35] , political polling [39] , software engineering research [16] , teaching research [32] , economics [21] , social media platforms design [34, 44] , market research to understand consumer preferences [3] and more generally social science research [2] . These studies yielded mixed results. While some outcomes closely mirrored the behaviors observed in real human counterparts, other research raised questions about the suitability of replacing human participants with LLMs in various social science contexts (for example, [35, 39] ).This paper extends the results of [13] investigating the feasibility of utilising GPT-4o as a substitute for human populations in healthcare questionnaires and surveys. To our knowledge, we are the first to evaluate the personality traits of GPT-4o using the EPQR-A personality test and to assess its consistency across different languages, genders, and study fields. 
methods: 
 We prompt GPT-4o to answer the Eysenck Personality Questionnaire-Revised (EPQR-A), which is an abbreviated version of the Eysenck Personality Inventory [12] , containing 24 items for assessing four different scales (6 items each): Extraversion (E), Neuroticism (N), Psychoticism (P), Lie (L) [14] . Each item has a dichotomous response (yes or no), and a score for each scale can be computed by summing individual items (resulting in a range from 0 to 6). The EPQR-A was originally tested in English obtaining relevant results [14] , and has been validated in many different languages such as Spanish [40, 15] , Brazilian [42] , Slovak [11] , French [29] , Turkish [25] , Greek [1] or Urdu [30] .We designed the following two experiments performed sequentially (the GPT prompts used in our experiment are described in detail in Section A.1):• Testing the personality of GPT-4o and the consistency of the EPQR-A scores. We used English instructions, but provided the questionnaire in three different languages: English (EN), Spanish (SP), and Slovak (SK). The rationale for this setting is given by initial tests that indicated that GPT-4o performed more effectively when instructions were given in English, despite the questionnaire's language variation, leading to more precise and coherent responses. We report means and standard deviations of 100 runs to measure potential variability in the answers. • Testing the properties of the EPQR-A when administered to a sample population generated by GPT-4o: we first use GPT to generate a simulated sample of students (examples in Section A.2) equivalent in size (826 students), age and gender composition as the one used in [15] and instruct GPT to assume these personas when answering the questionnaire. Again we used English instructions but provided the questionnaire in the three aforementioned languages.To obtain the sample population from GPT-4o, using Python's statistics library, we first computed the age and gender distribution, sampling from a truncated normal distribution with the corresponding statistics from the real-world sample used in [15] (females (655): 18.9 ± 1.56, males (171): 1936 ± 1.99).We then provided the age and gender information to GPT4o and tasked it to generate a corresponding persona description for a student from a university in Spain. This prompt is listed in Appendix A.1. Examples of virtual students can be seen in Appendix A.2.Once the virtual population had been created, we sent the EPQR-A personality test questionnaires in the three different languages, and for the different combinations of temperature and model version, using OPENAI's API with the task of answering it by impersonating each of the 826 generated student personalities. In more detail, we first specify a personality as a system role in the API call which is followed by the prompt as shown in Appendix A.3, which includes the sample English questionnaire listed in Appendix A.3.3.With the answers from the 826 virtual personas, we compute descriptive statistics for each of the 4 scales of the EPQR-A, and we tested reliability by computing Cronbach's 𝛼 [8] values. This is an index frequently used to evaluate the internal consistency of a set of items [43] . Cronbach's 𝛼 is a way of assessing reliability by comparing the amount of shared variance, or covariance, among the items making up a scale to the amount of overall variance. The idea is that if the scale is reliable, there should be a great deal of covariance among the items relative to the variance [7] . Cronbach's 𝛼 is considered poor if it is below 0.70; fair when it is between 0.70 and 0.79; good when it is between 0.80 and 0.89; and excellent when it is above 0.90 [6] . 
testing the personality of plain gpt: 
 4.1.1 Testing different temperatures and models. We start by analyzing the response of GPT to the EPQR-A test. Using the model's temperature parameter, which controls the "creativity" or randomness of the text generated, we first analyze the impact of different temperatures -0 (more focused), 0.5, 1, 1.5 (more creative) -to use on the responses of three different versions of GPT, that is gpt-3.5-turbo-0125 (GPT-3.5), gpt-4-turbo-2024-04-09 (GPT-4), and gpt-4o-2024-05-13 (GPT-4o). As can be seen in Table 1 , in all temperatures GPT-3.5 scores high in E, and low in N. Scores in P and L were lower in GPT-4 for all temperatures, while GPT-4o has higher scores in these two scales. Furthermore, we also observe a considerably larger standard deviation for GPT-4o.Globally, our analysis suggests that the GPT versions adopt a personality characterized by a tendency towards extroversion (especially when using GPT-3.5), emotional stability (especially with GPT-3.5 and 4), low levels of psychoticism (especially in GPT-4) and trying to follow social norms (with high levels of desirability). As expected setting GPTs temperature to 0 leads to less variability in the answers (lower standard deviation). The differences between a parameter setting of 0.5, 1, or 1.5 are less clear and we have thus opted to use the default setting of GPT for the temperature (i.e. a temperature of 1) in the remainder of this paper.After evaluating the results, we concluded that models depict similar personalities as measured through the EPQR-A test; but results for GPT-3.5 are closer to the extremes for E and N, the same is true for GPT-4 for P and L while the ones for GPT-4o are closer to the mean in the Spanish population of the reference study [15] . We thus decided to use GPT-4o for our remaining experiments.Table 1 : GPT personality Mean (± sd) of the EPQR-A test in Spanish for different temperatures. 100 iterations per model. GPT-3.5 stands for gpt-3.5-turbo-0125, GPT-4 stands for gpt-4-turbo-2024-04-09 and GPT-4o stands for gpt-4o-2024-05-13. 2 ). In general GPT-4o scores low in N and P but high in E, and L. The difference between Spanish and English/Slovak in the L scale can be explained by the change in the scoring introduced by [40] , who reversed the L scores, so lower scores indicate greater social desirability in the Spanish version of the EPQR-A questionnaire, while the contrary holds for English and Slovak. It is thus interesting to observe the slightly higher score of Slovak on this scale. Slovak also shows a noteworthy higher score for E, indicating that the language of the questionnaire changes some of the answers to the questionnaires by GPT-4o.4.2.1 Testing the EPQR-A when administered to different languages. Table 3 (top 4 rows) indicates that simulated population samples answering the questionnaire in Spanish and English consistently displayed similar scores in all scales. High scores in L1 were accompanied by low scores in P (from 0.85 to 2.11) and scores ranging from 2.83 to 3.28 in N and from 2.21 to 3.23 in E. When using the Slovak questionnaire, again some variations were observed. Specifically, although the L scale remained high, compared to the Spanish samples, the scores for P and E were higher, while the score for N was lower. When comparing the real population sample [15] with the simulated sample in Spanish, we found that E, P and L scores were significantly lower in the simulated sample, while N scores were higher. Table 4 shows that Cronbach's 𝛼 values are excellent for the E and N scales, regardless of language. However, for the P scale, Cronbach's 𝛼 was fair for EN but very poor for SK and poor for SP aligns with existing literature [40, 14] . Regarding the L scale, Cronbach's 𝛼 was fair for EN but very poor for the SP and poor for SK, similarly suggesting a need for item and relationship review.The gender analysis (see bottom rows of Table 3 ) reveals that the primary significant differences are in the N and P scales, with females scoring slightly higher than males in N and E, and slightly lower in P, consistent with findings in [15] . This difference is only significant in N in English and Slovak and was also significant for L in the real population sample.Finally, we grouped the EPQR-A scores (in Spanish) of the simulated sample of 826 students into five academic fields according to their personas descriptions. Table 5 shows that students from Business & Management displayed the highest scores in E, while Our findings partially align with existing literature on student personality traits across different fields of study [45] , for example showing that students in Arts & Design score higher in N. Female virtual personas also displayed slightly higher scores in E and N, and lower in P as has been reported in the literature for real populations [46, 5, 31] . This would indicate that to some extent the system is adopting congruently different profiles.This preliminary analysis hints about the possibility of using AI to simulate human persona and understand personal traits across different academic disciplines. 
conclusion: 
 The results in this paper suggest that GPT-4o exhibits human-like personality traits. Our tests indicate a tendency towards sociability (medium Extroversion scores), emotional stability (low Neuroticism), and non-aggressiveness with adherence to social norms (low Psychoticism). GPT-4o consistently shows high social desirability, similar to a human aiming to meet social expectations. Values are quite similar to the mean values found for the general population, except for Extroversion (values are lower for GPT). Interestingly, its personality traits appear stable across different languages. GPT-4o seems at least to be partially able to simulate populations and reproduce some gender differences and variations related to the academic fields. However, there were significant differences in the simulated populations compared to the real ones. This suggests that more refinement in the process of generating virtual personas is needed. An important issue is also the reliability (i.e. Cronbach's 𝛼) of the measure, since it was quite low for some scales, contrasting with another study that found higher reliability in psychological measurements [38] . However, the authors of [38] did not create a simulated population inspired by a real one. Importantly, the questionnaire we used was short, penalizing reliability measurement through Cronbach's 𝛼. Furthermore, reliability in real samples was also not adequate for the P scale.The practical implications of our findings are diverse and show great potential. Our results indicate that systems like GPT-4o can generate simulated populations, which could be valuable in creating new questionnaires and surveys, as well as validating existing ones across different languages or populations. This capability can aid various clinical settings. Indeed, asking these simulated populations to answer new questionnaires, could help in identifying issues to be improved before conducting clinical tests on real populations. For example, simulated populations can be used to identify issues related to internal consistency, or they can be used to test theoretically driven relationships between important variables (e.g., personality dimensions and the way people cope with an illness). Consequently, simulated populations can save researchers time by allowing them to focus their real experiments on the insights gained from these virtual populations. However, future research should improve prompting strategies to refine simulated populations and increase their variability. Our results also support, in line with previous research [26, 38, 37, 33, 13, 38, 28, 37] , that questionnaires developed for humans can be useful when applied to LLMs. This has a clear application in the development of AI-based systems since it can help to evaluate them and avoid inadequate behaviour. The implications of this study underscore the potential of Data-Centric AI (DCAI) in transforming healthcare. The study's innovative approach in simulating virtual populations and applying psychological assessments like the EPQR-A demonstrates the value of integrating DCAI principles to ensure that AI models can generalize well across diverse scenarios without overfitting, thus providing more accurate and actionable insights in clinical settings. Moreover, this study highlights the importance of refining data preprocessing and prompting strategies to reduce biases and enhance the authenticity of simulated personas. This work not only contributes to the academic discourse on AI and healthcare but also offers practical solutions for real-world applications, making significant strides toward the realization of reliable, data-centric AI in healthcare. 
future work: 
 This study underscores the need for future research focused on defining more precisely distinct virtual personas to better represent real populations. In this line, future research can investigate the impact of varying the amount of text and features used to define personalities, seeking to enhance the depth and authenticity of the personas generated. A strategy can be to implement improved prompts that ensure GPT-4o reflects the characteristics of the virtual personas accurately. For example, asking for longer descriptions of virtual personas, including more details, could help in the process of impersonating them to answer the required questionnaires.Furthermore, it could have been useful to simulate a bigger sample, including not only students. We wanted to have a comparable sample to a real one in the literature, and this was the reason why we constrained the sample size and type. However, further studies should include more virtual personas, with different age ranges, education, and social contexts. Second, it would be interesting to test the factorial structure of the questionnaire, to study whether the real populations can be considered a sample of the resulting distributions.Finally, testing with other questionnaires and surveys is needed. Although incipient available literature supports our hypothesis, it would be needed to replicate our experiments with more questionnaires assessing other types of variables of interest in the psychology and health field, such as personal cognitions, mood states, and behaviours. 
 
 Appendix: As explained before for Spanish (SP), the L scale has a reversed scoring system. 



Reviews: review: title: title: Good paper, accept
main_review: main_review: In this paper, the authors performed two experiments on EPQR-A in 3 different languages. The study shows that GPT-4o displays specific personality traits and may vary depending on different parameter settings and questionnaire language. 
Pros:
* Systematic experiments: robust design with two distinct experiments.
* Language variations: tested across English, Spanish, and Slovak for broader applicability, and adds depth and originality with multilingual testing.
* Novel application: unique use of GPT-4 in simulating clinical trial populations.

Cons:
* Inconclusive findings: limits practical applicability due to lack of definitive alignment results.
* Lack of details in postprocessing: insufficient explanation in some methodological sections.
* Generalization issues: specific to EPQR-A, limiting generalizability to other assessments.
* Preliminary nature: limits immediate impact on practical applications.
Overall, the paper is well-constructed and contributes to the emerging intersection of AI and psychology. While the results are not conclusive, the study sets a foundation for future research in simulating populations with AI for clinical trials and other applications.
title: title: Review for #35
main_review: main_review: ### Summary

This paper explores the potential of using GPT-4o to simulate populations for clinical trials, specifically through the use of the Eysenck Personality Questionnaire-Revised (EPQR-A). The authors conducted experiments in three languages (Spanish, English, and Slovak) and analyzed the results to determine the alignment between simulated and real populations.

### Quality

Solid experimental settings and comprehensive results for three languages. The presentation is decent but could be clarified further with illustrative figures.

### Originality and Significance

As a prompt engineering empirical study, the originality lies in the defined problem. There are several works that have explored LLMs' psychology portrayal, e.g.:

- PsychoBench -- On the Humanity of Conversational AI: Evaluating the Psychological Portrayal of LLMs (ICLR 2024 Oral)
- Open Models, Closed Minds? On Agents' Capabilities in Mimicking Human Personalities through Open Large Language Models (arXiv 2024)

There are no apparent differences between these works.

### Writing

Overall, it's clear, but considering the relationship with clinical trials, there is a lack of discussion on how this work's findings address the challenges in clinical trials.

### Pros

- Well-designed experiments on the use of multiple languages with clear objectives.
- Comprehensive analysis with detailed statistical evaluations.
- Latest models are evaluated, e.g., GPT-4o.

### Cons

- Mainly on writing about the impact on clinical trials, as mentioned above.