INCHARACTER: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews (2024)

Abstract: Role-playing agents (RPAs), powered by large language models, have emerged as a flourishing field of applications. However, a key challenge lies in assessing whether RPAs accurately reproduce the personas of target characters, namely their character fidelity. Existing methods mainly focus on the knowledge and linguistic patterns of characters. This paper, instead, introduces a novel perspective to evaluate the personality fidelity of RPAs with psychological scales. Overcoming drawbacks of previous self-report assessments on RPAs, we propose INCHARACTER, namely Interviewing Character agents for personality tests. Experiments include various types of RPAs and LLMs, covering 32 distinct characters on 14 widely used psychological scales. The results validate the effectiveness of INCHARACTER in measuring RPA personalities. Then, with INCHARACTER, we show that state-of-the-art RPAs exhibit personalities highly aligned with the human-perceived personalities of the characters, achieving an accuracy up to 80.7%.1

heading: 1 Introduction
text: Recent advancements in large language models (LLMs) have catalyzed the emergence of roleplaying agents (RPAs). RPAs are interactive AI systems simulating diverse roles or characters. RPA applications have been extended to diverse contexts, such as AI agents of fictional characters (Li et al., 2023), digital clones for humans (Gao et al., 2023), and AI non-player characters in video games (Wang et al., 2023a). Recent research trends have increasingly focused on the development of
*Corresponding author. 1Project page: https://incharacter.github.io/
RPAs, including building RPAs for specific characters (Li et al., 2023; Wang et al., 2023b) and improving the role-playing abilities of foundation models (Zhou et al., 2023).
However, evaluating character fidelity in RPAs remains a relatively underexplored area. Prior research mainly concentrates on the replication of knowledge, experience, and linguistic patterns of characters (Shao et al., 2023a; Zhou et al., 2023), which manifests in two primary issues: (1) They necessitate character-specific datasets, thereby complicating the evaluation of new characters. (2) They overlook evaluating RPAs’ thoughts and underlying mindsets. Towards these issues, we propose to evaluate if RPAs faithfully reproduce the personalities of target characters, i.e., personality fidelity, as depicted in Figure 1. Personality tests, administered by psychological scales, measure an individual’s interrelated behavioral, cognitive, and emotional patterns (Barrick and Mount, 1991; Bem, 1981). By measuring the personalities of RPAs and com-
1840
paring them with the personalities of the characters, we can attain a more nuanced understanding of RPAs’ character fidelity.
Prior studies on LLM personalities are mainly based on self-report scales, which prompt LLMs to select options or assign ratings to specific items (Tu et al., 2023; Huang et al., 2023b). However, this method suffers from several limitations for RPAs. (1) The instruction to complete scales contradicts role-playing instructions, leading to RPAs’ reluctance or inability to engage with personality tests. (2) More importantly, the selected options may conflict with the actual behaviors of RPAs, making the test results unindicative of their true personalities. RPAs might underperform owing to an inadequate understanding of scale instructions and the biases inherent in the training data.
Therefore, we propose INCHARACTER, a novel approach to Interviews Character agents for personality tests. While self-report scales are popular in humans for their cost-effectiveness, interviewbased scales evaluated by experts offer a more comprehensive analysis (Uher et al., 2012; Rush et al., 1987). Self-reports are sometimes influenced by an individual’s lack of insight, denial, or bias. In contrast, an interviewer can be a guide to elicit thoughts of individuals, effectively identifying and addressing the nuances via conversations to overcome the previously mentioned limitations. INCHARACTER employs this interview-based procedure (Trull et al., 1998) on RPAs, which includes two stages: (1) Interview: RPAs are engaged with open-ended questions derived from psychological scales to elicit RPAs’ mindsets and behaviors. (2) Assessment: We utilize LLMs to interpret the responses collected from the first stage. This can involve converting the responses to Likert levels or using LLMs to simulate a psychiatrist’s role in judging RPA personalities.
We apply INCHARACTER to various RPAs on 14 personality tests, including the Big Five Inventory (BFI), 16Personalities2 (16P), and Dark Triad Dirty Dozen (DTDD). The personality labels for the BFI and 16P are accessible through the Personality Database (PDb)3. Additionally, we engage human annotators familiar with the characters to label them on other scales, thereby creating a comprehensive benchmark for evaluating RPA personalities. Our experiments include various types of
2https://www.16personalities.com/. This scale is based on the MBTI and is widely used worldwide.
3https://www.personality-database.com/
existing RPAs. The results demonstrate that the INCHARACTER effectively simulates interview-based tests conducted by human experts and yields RPAs personality measurement better aligned with the characters than self-report methods. Additionally, we find that the conversation data generated in our interview phase are of high quality and ideal for fine-tuning foundation models for RPAs. Hence, we release a dataset comprising 18,304 interview dialogues to facilitate future research.
The contributions of this paper are mainly threefold: 1) We introduce a novel aspect for RPA evaluation, i.e., personality fidelity, based on psychological scales. 2) We propose INCHARACTER, an interview-based framework for personality tests on RPAs and collect the first benchmark for RPA personality evaluation, facilitating future research on developing better RPAs. 3) Our experiments on various RPAs and psychological scales demonstrate the efficacy of INCHARACTER.

heading: 2 Preliminaries

heading: 2.1 Role-Playing Agents
text: Recent advancements have led to the emergence and evolution of several pivotal abilities in LLMs to facilitate the development of RPAs, including in-context learning (Brown et al., 2020), instruction following (Ouyang et al., 2022), step-by-step reasoning (Wei et al., 2022), and human-like traits such as empathy (Sorin et al., 2023). RPAs are interactive AI systems that act as assigned personas, from fictional characters to celebrities. RPAs utilize persona data to simulate characters, drawing from training datasets, prompted contexts, or external databases. Typically, existing work develops RPAs by setting character descriptions as system prompts (Zhou et al., 2023; Shao et al., 2023a) and crafting memory modules with character dialogues (Li et al., 2023; Wang et al., 2023b).

heading: 2.2 Psychological Scales
text: Usually rated on Likert levels, psychological scales are commonly used for personality tests. Selfreport scales require participants to respond to a series of items analyzed through a specific scoring scheme to determine their personality traits. A scale rated on Likert levels, denoted as L = (P,D,O, f), comprises a set of items P (i.e., a questionnaire), a list of dimensions D, a set of response options O, and a scoring scheme f . Each item p ∈ P is a statement or question, positively
or negatively corresponding to a dimension d ∈ D. For example, the item “Values artistic, aesthetic experiences.” is positively related to the Openness dimension in the BFI. Participants select an ordinal response o ∈ O for each item, such as Agree. Typically, these options are numerically coded, e.g., “1” for Strongly Disagree and “5” for Strongly Agree. This process generates a response array A. The scoring schema f usually includes item-dimension mapping, identification of positive and negative items, conversion of options to scores, and an aggregation method (e.g., average or sum). Finally, the participant’s personality scores S is derived as S = f(A), where S = (sd1 , sd2 , ..., sd|D|) represents scores across each dimension. We summarize the notations used in this paper in Table 4.

heading: 3 INCHARACTER
text: This section introduces INCHARACTER, a novel personality assessment methodology designed explicitly for RPAs, utilizing an interview-based procedure. Figure 2 illustrates our two-stage framework. The interview stage is detailed in §3.1, followed by an elaboration of the assessment stage in §3.2.

heading: 3.1 Interview
text: INCHARACTER draws inspiration from the Structured Interview approach used in psychological testing (Trull et al., 1998). For a given scale, it transforms scale items into a series of open-ended questions, forming the basis for a structured interview. Then, our framework interviews RPAs using
these open-ended questions to elicit their perspectives on topics indicative of personality traits.
Constructing Question List We develop the structured interview question list based on items of the scale. Specifically, each item p ∈ P is transformed into an open-ended question q via LLMs and manually checked. Consequently, the question list Q comprises |P| questions. For instance, in the BFI, the item “Values artistic, aesthetic experiences.” is rephrased as “Do you values artistic, aesthetic experiences?”
Interviewing RPAs We interview an RPA C of character c, by presenting each question q ∈ Q and recording its corresponding response r. To avoid context effects (Nikolić, 2010), each question is posed in an isolated context, thereby avoiding potential interference among the questions.

heading: 3.2 Assessment
text: Based on the interview results, the assessment phase quantitatively evaluates the score sd of the RPA C across each dimension d ∈ D. To this end, we introduce two distinct methodologies for measuring and analyzing RPA personalities leveraging LLMs: option conversion (OC) and expert rating (ER).
Option Conversion This technique leverages LLMs to convert a response r for a question q into a corresponding answer option a ∈ O for item p, effectively bridging the gap between closed-ended and open-ended question formats. The idea follows
the clinician-rated scales used in clinical psychiatry (Cuijpers et al., 2010; Uher et al., 2012), where professional clinicians assign ratings to each scale item based on their observations during patient interviews and compute the final scores following the scale’s scoring scheme. For example, a response “I believe that art transcends reality...” is converted to “5 (Strongly Agree)” for the item. Afterward, the answer list A is input to the scoring scheme f to compute the final personality scores. In practice, we observe that even state-of-the-art LLMs like GPT-4 (OpenAI, 2023) exhibit notable inaccuracies in categorizing the attitudes of RPAs. Therefore, we further introduce a dimensional-specific option conversion (d-OC) strategy, which divides (q, r) pairs according to dimensions and substitutes Likert levels, such as “4 (Agree)” and “2 (Disagree)”, with more descriptive options like “4 (Extroverted)” and “2 (Introverted)” in the prompts for LLMs.
Expert Rating In contrast with the one-by-one question conversion in OC, this method applies LLMs to directly evaluate personality scores of RPAs in each dimension, considering all corresponding (q, r) pairs. This idea draws inspiration from the structured clinical interview in clinical psychiatry (First, 2014), where clinicians assess patients using a predefined question list and derive final scores based on the responses without intermediate ratings or scoring schemes. The interviewer LLM 4 is prompted with comprehensive descriptions of the scale, dimension, and score range. It then generates the final personality score for each dimension based on the pertinent responses. The advantage of ER is that it re-implements the scoring schema with the interviewer LLM, which can intelligently weigh individual (q, r) pairs instead of using equal weights in OC. Hence, it better recognizes personality-indicative responses from RPAs.
Details of our prompts for OC and ER are available in §G in the appendix. To prevent the influence of data leakage in ER and d-OC, i.e., the interviewer LLM might have memorized the characters’ personality types. Hence, we anonymize the character names in the input prompts.

heading: 4 Experimental Setup

heading: 4.1 Preliminary Study

heading: Can LLMs Simulate Human Interviewers?
text: First, we validate the capability of interviewer
4Interviewer LLMs denote the models for the ER and OC tasks, analogous to human interviewers.
LLMs on the OC and ER tasks, given the interview results of RPAs. We compare LLM predictions with human judgments. For each task, we sample 100 cases of state-of-the-art RPAs on the BFI and manually label them. For OC, the input is one question-response pair (q, a) and the output is an option. For ER, the input is composed of multiple (q, a) pairs, and the output is a score. Examples are shown in the right of Figure 2. We experiment with three LLMs, including GPT-4, GPT-3.5 and Gemini. The RPAs are detailed in §4.2. We report the Pearson’s r (Pearson, 1920), Spearman’s ρ (Spearman, 1961) and Kendall’s τ (Kendall, 1938) correlations between human annotations and interviewer LLMs, as well as the accuracy. We consider LLM predictions varying from human labels by less than 1 point, exactly 1, or more than 1, as right, close (half-correct) or wrong, for accuracy calculation. More details can be found in §F.1.
The results presented in Table 1 lead to several findings. First, for ER, state-of-the-art LLMs can adequately rate participants’ personalities based on interview results. We observe that GPT-4 makes only 4% wrong cases in ER, primarily when RPAs give contradictory responses. Second, for OC, the LLMs show significant inaccuracy, while replacing Likert-level options with dimension-descriptive ones (d-OC) largely improves LLMs in this task. Considering the consistency measurements, stateof-the-art LLMs achieve acceptable performance in simulating human interviewers to assess RPA personalities through ER or d-OC.

heading: 4.2 Experimental Settings
text: The experiments in the subsequent part of this paper are based on the following settings:
RPAs and Characters This work primarily focuses on RPAs built on character data curated by ChatHaruhi (Li et al., 2023) and RoleLLM (Wang et al., 2023b). We select 32 widely-known characters, 16 from ChatHaruhi5 and 16 from RoleLLM. The characters are mainly from popular fictional works, such as Harry Potter, The Big Bang Theory and Genshin Impact. Please refer to §C for the detailed character selection process. The character data from ChatHaruhi and RoleLLM includes descriptions and dialogues used for system prompts and memory modules. To implement RPAs, we apply the Chat-Haruhi-Suzumiya6 library, and adopt GPT-3.5 (OpenAI, 2022) as the foundation LLM by default.
Psychological Scales We consider 14 psychological scales, including the BFI, the 16P, and 12 other scales following PsychoBench7 (Huang et al., 2024) to evaluate RPAs. Most scales apply scoring schemes like average and sum, while the 16P is close-source and accessed via its API. Detailed introduction of these scales can be found in §B. Due to page limitations, the main body presents results for the BFI and 16P, while additional findings are detailed in the Appendix.
Personality Labels We collect labels for character personalities in the form of both scores and types, contributed by people familiar with these characters. From the PDb, an online platform for character personality annotation, we derive scores of the BFI and 16P on each dimension from its label percentage (e.g., 60% Extroverted). We then categorize it into a type of either positive, negative, or marginal if it is above 60%, under 40%, or otherwise. Then, we invite human annotators for comprehensive personality labels on all 14 scales. To select qualified annotators, we examine their character understanding of the BFI and 16P, matching with labels from the PDb. We invite two to three annotators for each character (93 in total for 32 characters) and average their results for improved reliability and objectivity. The scores are re-scaled into
5Six RPAs from ChatHaruhi are based on Chinese data, and we conduct the interview with them in Chinese.
6A continuously updating project for RPAs. We refer ChatHaruhi (Li et al., 2023) to its first version. https: //github.com/LC1332/Chat-Haruhi-Suzumiya
7https://github.com/CUHK-ARISE/PsychoBench
the unit interval [0, 1] and categorized into types similarly. We measure the inter-annotator consistency via Cohen’s kappa coefficient (Cohen, 1968), and find the average coefficient across 14 scales 60.9%. For the BFI and 16P, we adopt types from the PDb and scores from our invited annotators. The details about PDb annotations, our human annotation process, intra-annotator consistency, and other statistics can be found in §D.
Interviewer LLMs We use LLMs to accomplish the OC, d-OC and ER tasks in the assessment phase of INCHARACTER, or to extract selected options from RPA responses in self-report methods if RPAs do not provide exactly the choice. We consider four widely-acknowledged LLMs, including GPT-3.5, GPT-4, Gemini and Qwen1.5-110B 8.
Metrics We consider two sets of metrics, namely: (1) Measured alignment (MA) compares the measured personalities of RPAs and human-annotated personalities of characters. It depends both on the performance of RPAs and the effectiveness of personality test methods. We categorize RPAs as positive or negative on each dimension if the scores are above or below the median of the scoring range. Then, we calculate mean absolute error (MAE) and accuracy to measure alignment at the score and type level, respectively. We re-scale MAE by dividing it with the scoring range length. For accuracy, we report the average AccDim and AccFull, where correctness is judged on individual or all dimensions of each scale. The marginal dimensions of each character are ignored due to their ambiguity. (2) Personality consistency (PC) indicates whether the measured personality of RPAs is consistent across various settings. We analyze the standard variance at the item-level (StdItem), dimensionlevel (StdDim), and score-level (StdScore). StdItem and StdDim measure the consistency of an RPA’s scores on individual items. For INCHARACTER, we experiment with OC and d-OC to convert responses into scores. StdItem measures an RPA’s consistency on the same item across multiple runs. StdDim compares an RPA’s responses across different items in the same dimension. StdScore denotes the variance of an RPA’s score on each dimension across multiple runs. We divide these metrics by the length of the corresponding scoring range to re-scale them into the unit interval.
8The versions in this paper are gpt-3.5-turbo-1106, gpt-4-1106-preview, gemini-pro, Qwen1.5-110B-Chat respectively.

heading: 5 Experimental Results

heading: 5.1 Personality Tests on RPAs
text: Baselines For INCHARACTER, we experiment with the ER, OC, and d-OC. For ER, we consider two settings, ERall and ERbatch, where questionresponse pairs in one dimension are inputted into interviewer LLMs all-at-once or in-batch9. For selfreport (SR) baselines, we follow previous work on LLM Psychometrics (Huang et al., 2024) to prompt RPAs to provide exactly a choice for each scale item. If their responses are not exactly the choices, we use interviewer LLMs to extract the choices. Then, the numbers are aggregated via the scoring schema to get the results. Besides, we experiment with SR-CoT, which enhances SR with chain-of-thought reasoning, i.e., explicitly asking RPAs to articulate their thoughts before choosing the options.
We compare these methods on the BFI and 16P. The experiments are repeated three times, including both the interview phase and the assessment phase. We report the average results of the three runs for MA metrics and StdDim, and calculate StdItem and StdScore across the three runs.
Alignment between RPAs’ Measured Personalities and Characters’ Labeled Personalities Then, we apply INCHARACTER to measure RPA personalities. According to the results in Table 2, we have the following analyses: (1) Using INCHARACTER with ER and GPT-4, the measured RPA personalities are highly aligned with ground truth labels of corresponding characters. This suggests that state-of-the-art RPAs well reproduce many of the characters’ personality traits, and our methods accurately measure their personalities. (2) RPA personalities measured via INCHARACTER are better aligned with the characters than SR baselines. This validates the advantage of INCHARACTER over self-report for personality tests on RPAs, which will be further discussed. (3) The alignment measured via INCHARACTER correlates with the interviewer LLMs’ capability on the assessment tasks. For the interviewer LLMs, GPT-4 achieves the best metrics, while GPT-3.5, Gemini, and Qwen-110B also demonstrate satisfactory performance. For the assessment methods, INCHARACTER with ER generally achieves better MA met-
9In the BFI and 16P, a dimension generally has 9 to 15 examples. Hence, the in-batch setting applies a batch size of 3 to 4. The results are averaged to form the final scores.
rics than d-OC, while d-OC surpasses OC. However, Table 1 shows that interviewers LLMs still make mistakes on the ER and OC tasks, leading to potential inaccuracies in INCHARACTER and may underestimate the personality alignment of RPAs.
Robustness, Consistency and Distinctiveness of RPA Personalities Generally, the measured RPA personalities are robust across our observations. The StdScore across three runs remain below 6% in various settings, which underlines the reliability of personality tests and the robustness of RPA personalities. Then, we study the consistency at the item-level (StdItem) and dimension-level (StdDim). With INCHARACTER, after converting the interview results into scores via d-OC and GPT-4, We observe that RPAs respond to the same items consistently across multiple runs and exhibit a relatively consistent personality across different items on the same dimension. We visualize the distribution of RPA personalities on the BFI in Figure 3, and find that RPAs exhibit distinct personalities, especially when measured by INCHARACTER with ERbatch and GPT-4.

heading: Self-report v.s. Interview-based Methods As
text: shown in Table 2, the personalities measured by INCHARACTER are more aligned with the characters, compared with self-report. Meanwhile, in interview-based tests, RPAs exhibit more consistent personalities across different questions, as well as greater distinctiveness, shown in Figure 3. These findings confirm the advantages of interview-based tests over self-report in measuring RPA personalities. Although SR-CoT attempts to enhance SR with the thought process, its improvement over SR is limited, and it encounters poor StdItem. Further analyses and comparisons are detailed in §F.2. We also experiment with enhancing self-report methods with in-context learning in §F.8, which under-
mines the measurements instead.

heading: Comprehensive Personality Tests on 14 Scales
text: We extend personality tests on RPAs to 14 psychological scales, using INCHARACTER with ERbatch and GPT-3.5. In Figure 4, we demonstrate the AccDim of state-of-the-art RPAs across each scale. Overall, we observe that the RPAs exhibit personalities align with the target characters in comprehensive aspects with an average AccDim of 78.9%, covering personality traits (BFI, 16P), dark personalities (DTDD), interpersonal relationships (BSRI, ECR-R), basic interests (CABIN), motivation (GSE, LMS) and emotional intelligence (EIS, WLEIS), etc. The detailed metrics on individual scales and individual dimensions are listed in §F.5.
More Results and Analyses Besides, we study the importance of using personality scales in §F.6, and find that scale questions are crucial for eliciting personality-indicative responses from RPAs. We also observe that some scale questions go beyond the knowledge scope of certain characters in §F.7,
and adapting these questions for each character yields more accurate measurements.

heading: 5.2 Personality Fidelity of Different RPAs
text: With INCHARACTER, we compare the personality fidelity of various types of RPAs, covering different character data and foundation models. We apply INCHARACTER with ERbatch and use GPT-3.5 as the interviewer LLM for personality tests. We report the MA metrics on the BFI and 16P in Table 3.
Character Data for RPAs Typically, existing RPAs utilize two types of character data: descriptions and memories. Character descriptions serve as the system prompts for RPAs, while memories consist of characters’ experiences and dialogues used for retrieval. With GPT-3.5, we evaluate RPAs with only descriptions (D), only memories (M), and a combination of both (D+M). The results in Table 3 reveal that: (1) With only description, RPAs achieve MA metrics close to the full D+M setup, highlighting the importance of character de-
scription in shaping RPA personalities. (2) RPAs can well mimic character personalities exhibited in their past experiences, e.g., extraversion and openness, even if the experiences are not directly related to scale questions. Additionally, we compare RPAs with character data from ChatHaruhi, RoleLLM and character.ai (c.ai) in §F.3.
Foundation Models for RPAs We consider three types of LLMs: (1) General open-source models, including Qwen-7B (Bai et al., 2023), OpenChat3.5 7B (Wang et al., 2024), Mistral-2 7B (Jiang et al., 2023), Llama-2-chat 13B (Touvron et al., 2023) and Mixtral 8x7B (Jiang et al., 2024). (2) Specialized open-source models for RPAs, including Character-GLM 6B (Zhou et al., 2023), RPQwen 7B10, and RP-Mistral-2 7B. We train RP-
10https://huggingface.co/silk-road/ChatHaruhi_ RolePlaying_qwen_7b
Mistral-2 7B with details shown in §E.2. (3) Closesource models: GPT-3.5 and GPT-4.
The results are shown in Table 10. We observe that, (1) RPAs with GPT-3.5 and GPT-4 achieve the best personality fidelity, and GPT-4 does not significantly surpass GPT-3.5. (2) With state-of-theart open-source LLMs, RPAs can also reproduce character personalities. However, such capacity depends largely on their ability to use specific languages, shown in §F.4. (3) Incremental fine-tuning on open-source LLMs with role-playing datasets brings limited improvement in personality fidelity, especially when they are already equipped with excellent role-playing ability.
Close-source RPAs We also experiment with close-source RPAs from character.ai, which are based on their private foundation models and data. As shown in Table 10, these RPAs barely reproduce character personalities, significantly underperforming GPT-3.5 (D) which shares a similar framework. According to our observation, while character.ai RPAs provide human-like answers, their answers tend to be compliant and pleasing to users, instead of reproducing the target characters. Examples and further analysis are shown in §H.2.

heading: 6 Related Work
text: Role-Playing Agents RPAs learn and leverage character data in various ways, including training on raw scripts or dialogues (Shao et al., 2023b), prompting with character descriptions (Zhou et al., 2023), and retrieval from character experiences (Li
et al., 2023). Existing efforts mainly focus on developing character-specific RPAs or foundation models for RPAs. The former includes ChatHaruhi (Li et al., 2023) and RoleLLM (Wang et al., 2023b), which target well-established fictional characters. The latter includes character.ai and CharacterGLM (Zhou et al., 2023). For evaluation, prior research mainly concentrates on two facets: 1) Character-independent capabilities, which include conversational abilities (Duan et al., 2023), human-likeness (Tu et al., 2024), multi-turn consistency (Shao et al., 2023a), and attractiveness (Zhou et al., 2023); 2) Character fidelity, including the characters’ knowledge, experience, and linguistic patterns (Wang et al., 2023b; Shao et al., 2023a). Overall, these methods generally require test sets for each character, and neglect the evaluation of RPAs’ underlying mindset. Psychological Analysis on LLMs Recent studies conducted personality tests using the BFI (Romero et al., 2023; Karra et al., 2022; Li et al., 2022; Jiang et al., 2022; Safdari et al., 2023; Bodroza et al., 2023), the MBTI (Rutinowski et al., 2024; Pan and Zeng, 2023) on various LLMs. Notably, Huang et al. (2023c) verified the reliability of the BFI on GPT-3.5, while Safdari et al. (2023) demonstrated the construct validity of the BFI on the PaLM model family. Other studies also investigates other mental perspectives, such as emotions (Huang et al., 2023a), values (Miotto et al., 2022; Rutinowski et al., 2024; Hartmann et al., 2023), consciousness (Butlin et al., 2023), and mental illness (Coda-Forno et al., 2023). Our research diverges by employing personality tests as an innovative approach to assessing character fidelity in RPAs.

heading: 7 Conclusion
text: In this study, we investigate the personality fidelity in RPAs, i.e., whether RPAs reproduce personalities of their intended characters. Addressing the shortcomings of previous methods on RPAs, we propose INCHARACTER, an interview-based approach that accurately measures RPA personalities based on their elicited mindsets and behaviors. Our experiments span various types of RPAs, covering 32 characters on 14 psychological scales. The results validate the effectiveness of INCHARACTER in measuring RPA personalities. Afterward, with INCHARACTER, we comprehensively evaluate personality fidelity in existing RPAs, discovering that
state-of-the-art RPAs successfully portray many personality traits of the characters.

heading: Limitations
text: There are several limitations in this study. First, the personality measurement in this paper relies on the interviewer LLMs. Consequently, the accuracy of the measured results may be compromised by potential errors or biases inherent in LLMs, potentially leading to an underestimation of the personality fidelity in RPAs. Second, the personalities of humans or fictional characters can change over time. Since we use one static personality label for a specific character, there may be noise in our evaluation. For instance, the character of James Bond has experienced significant development over the past two decades across various films and television series. Our character annotations are derived from a singular, fixed time point in his storyline. Additionally, the progressive changes in RPA personalities remain unexplored within existing literature. We leave the study of RPA personality dynamics for future research.

heading: Ethical Statement
text: We hereby acknowledge that all authors of this work are aware of the provided ACL Code of Ethics and honor the code of conduct.
Use of Human Annotations In conducting our research, we have employed a methodology that incorporates personality labels, which were gathered through the online platform and by engaging a group of annotators. These annotators, who are university students, play a crucial role in our research process. To ensure fair treatment and to value their contribution, we offer them compensation that significantly exceeds the local minimum wage standards. Moreover, we maintain transparency regarding the application and purpose of their annotations, securing their informed consent for the use of these annotations in our research endeavors. Additionally, we are committed to upholding the privacy rights of our annotators throughout the annotation process, ensuring a respectful and ethical research environment.
Risks In this paper, we introduce a novel approach, referred to as INCHARACTER, designed to assess the personalities of Role-Play Agent (RPA) entities. An integral component of our evaluation
process involves the use of interviewer Large Language Models (LLMs), which, while innovative, could potentially introduce bias into the assessment outcomes. It is important to acknowledge this limitation as LLMs may reflect the inherent biases present in their training data. Furthermore, our evaluation encompasses a comprehensive analysis across 14 personality scales, notably including the Dark Triad of Personality (DTDD) scale, which focuses on darker personality traits. While this inclusion is aimed at providing a thorough understanding of RPA personalities, it raises ethical concerns regarding the potential for generating harmful content. This aspect underscores the need for careful consideration and implementation of safeguards to mitigate the risks associated with exploring dark personality traits in RPAs.

heading: Acknowledgment
text: This work is funded by the Science and Technology Commission of Shanghai Municipality Grant (No. 22511105902). This work originates from a project on Chat-Haruhi-Suzumiya proposed by Cheng Li. We owe thanks to the early contributors. We are thankful for the support provided by Zheli Xuan at Wuhan University, and Dingding Hu at the Institute of Psychology, Chinese Academy of Sciences, who offered invaluable assistance as psychology researchers. Our gratitude extends to our invited annotators, primarily from Fudan University, for their contribution of high-quality annotations of character personalities. We also acknowledge the assistance provided by Rui Fu and Wenxin Gao at Fudan University during the annotation process. Finally, we express our sincere gratitude for the precious comments and suggestions from Yikai Zhang, Xinfeng Yuan, and Shuang Li at Fudan University.

heading: A Notation Table
text: In Table 4, we list the notations and abbreviations in this paper, along with their definitions.

heading: B Psychological Scales
text: Big Five Inventory The BFI serves as a prominent instrument for assessing personality dimensions. This model, often encapsulated by the acronym “OCEAN,” encompasses five critical traits: (1) Openness to Experience (O), which highlights a person’s curiosity, inventiveness, and appreciation for art, emotion, adventure, and novel concepts. (2) Conscientiousness (C), indicating how much an individual exhibits organization, reliability, and responsibility. (3) Extraversion (E), denoting the level to which a person is sociable and energized by interactions with others. (4) Agreeableness (A), assessing an individual’s kindness, empathy, and ability to cooperate with others. (5) Neuroticism (N), gauging the tendency of an individual to experience negative feelings such as anxi-
ety, anger, and sadness, as opposed to being more emotionally resilient and less stress-susceptible.

heading: Eysenck Personality Questionnaire (Revised)
text: The Revised Eysenck Personality Questionnaire (EPQ-R) serves as a psychological instrument for gauging distinct personality trait variances in individuals. It identifies three principal traits: (1) Extraversion (E), which assesses whether a person tends to be more sociable, energetic, and outgoing as opposed to being introverted, quiet, and reserved. (2) Neuroticism (N), which gauges emotional steadiness. These dimensions (i.e., E and N) share similarities with those found in the BFI. (3) Psychoticism (P), which is indicative of a person’s inclination towards solitude, a lack of empathy, and a propensity for aggression or a tough-minded attitude. This trait is crucial to understand as indicative of personality characteristics rather than serious mental health conditions. (4) Beyond these primary scales, the EPQ-R also incorporates a Lying Scale (L) intended to identify responses aimed at social desirability. This scale evaluates the extent to which an individual may attempt to portray themselves in a more favorable light.
Dark Triad Dirty Dozen The DTDD is identified as a brief, 12-item measure crafted to evaluate the trio of principal personality characteristics known as the Dark Triad, encompassing: (1) Narcissism (N), characterized by an exaggerated sense of one’s own significance, an obsession with dreams of boundless success, and a craving for undue admiration. (2) Machiavellianism (M), indicative of a deceitful approach in social interactions and a skeptical indifference to ethical principles. (3) Psychopathy (P), which includes tendencies towards impulsiveness, a deficiency in empathy, and hostile relations with others. These Dark Triad personality dimensions are typically viewed as the antithesis of the characteristics measured by the BFI or the EPQ-R, which represent “Light” traits.
The NERIS Type Explorer The 16Personalities utilizes the acronym format introduced by MyersBriggs for its simplicity and convenience, with an additional letter to accommodate five rather than four scales. However, unlike Myers-Briggs or other theories based on the Jungian model, the incorporation of Jungian concepts such as cognitive functions, or their prioritization, has not been undertaken. Instead, they rework and rebalance the dimensions of personality in the BFI personality
traits. The personality types are based on five independent spectrums, with all letters in the type code (e.g., INFJ-A) referring to one of the two sides of the corresponding spectrum.
Bem’s Sex Role Inventory The BSRI assesses the degree to which individuals identify with traditionally masculine and feminine characteristics. Rather than focusing on behaviors, such as participation in sports or cooking, this tool evaluates psychological characteristics, including assertiveness and gentleness. Participants are divided into four groups based on whether their average scores exceed the median for each component. These groups are designated as Masculine (M: Yes; F: No), Feminine (M: No; F: Yes), Androgynous (M: Yes; F: Yes), and Undifferentiated (M: No; F: No).

heading: Comprehensive Assessment of Basic Interests
text: The CABIN provides an exhaustive evaluation for identifying 41 essential dimensions of vocational interest. Following this evaluation, the researchers introduce a model of interest consisting of eight dimensions, named SETPOINT. This model includes dimensions such as Health Science, Creative Expression, Technology, People, Organization, Influence, Nature, and Things. These core dimensions are also adaptable to a six-dimension framework, which is prevalently recognized within the interest research community. This framework aligns with Holland’s RIASEC model, which features the dimensions: Realistic, Investigate, Artistic, Social, Enterprising, and Conventional.
Implicit Culture Belief The ICB scale measures the extent to which individuals think a person’s ethnic culture influences their development. Scoring higher on this scale indicates a firm belief that a person’s ethnic culture is the main factor shaping their identity, values, and perspective on the world. On the other hand, a lower score on the scale denotes a belief in the ability of an individual to shape their own identity through hard work, commitment, and education.

heading: Experiences in Close Relationships (Revised)
text: The ECR-R is a self-assessment tool crafted to gauge variations in adult attachment styles, particularly within the realm of romantic relationships. As an enhanced iteration of the original ECR scale, the ECR-R introduces refinements in quantifying attachment tendencies. It assesses two primary aspects: (1) Attachment Anxiety indicates the degree to which a person fears rejection or abandonment
by their romantic partners. (2) Attachment Avoidance assesses the degree to which a person prefers to keep emotional and physical distance from their partners, often stemming from unease with closeness or reliance.
General Self-Efficacy The GSE Scale evaluates a person’s confidence in their capacity to address diverse demanding situations in life. This confidence, known as “self-efficacy,” plays a pivotal role in social cognitive theory and is associated with numerous health outcomes, motivational levels, and performance measures. An elevated score on this scale indicates a person’s strong belief in their ability to confront and manage challenging circumstances, undertake new or complex tasks, and navigate through the resultant difficulties. On the flip side, a lower score on the scale suggests a lack of self-assurance in handling challenges, rendering individuals more susceptible to experiencing helplessness, anxiety, or engaging in avoidance behaviors when encountering hardships.
Life Orientation Test (Revised) The LOT-R is designed to assess variations in optimism and pessimism among individuals. It includes ten questions, with an interesting aspect being that only six of these questions contribute to the test’s score. The other four are designed as filler items, cleverly integrated to obscure the test’s primary focus. Within the scored questions, equal numbers are dedicated to evaluating optimism and pessimism—three for each. A tendency towards higher scores in optimism and lower in pessimism signifies a predominantly optimistic outlook.
Love of Money Scale The LMS evaluates the perspectives and feelings of people regarding money. This tool aims to quantify the degree to which people perceive money as a symbol of power, success, and liberty, along with its significance in influencing behaviors and choices. The LMS identifies three key dimensions: (1) Rich reflects the degree to which people link money with success and accomplishment. (2) Motivator determines the extent to which money serves as an incentive in someone’s life, i.e., how much individuals are motivated by monetary rewards in their decisions and behaviors. (3) Important assesses the level of importance people attribute to money, affecting their principles, objectives, and perspective of the world.
Emotional Intelligence Scale The EIS serves as a self-assessment tool for evaluating multiple as-
pects of emotional intelligence. This instrument emphasizes various elements of emotional intelligence, notably the perception, management, and application of emotions. It is extensively utilized in the field of psychology to investigate how emotional intelligence influences different outcomes, including personal well-being, professional performance, and social interactions.

heading: Wong and Law Emotional Intelligence Scale
text: Similar to EIS, the WLEIS is also a self-report instrument designed for evaluating emotional intelligence. However, it distinctly includes four subscales that represent the primary aspects of emotional intelligence: (1) Self-emotion appraisal (SEA) focuses on an individual’s proficiency in identifying and understanding their emotions. (2) Others’ emotion appraisal (OEA) is about the skill of recognizing and comprehending the emotions of others. (3) Use of emotion (UOE) deals with the ability to employ emotions to aid various mental processes, like reasoning and problem-solving. (4) Regulation of emotion (ROE) is concerned with the ability to control and adjust emotions within oneself and in others.
Empathy Scale Empathy, defined as the capacity to perceive and resonate with the emotions of another, is traditionally divided into cognitive and emotional empathy. Cognitive empathy, also known as “perspective-taking,” entails the mental faculty to identify and comprehend the thoughts, beliefs, or feelings of someone else. Conversely, emotional empathy involves the vicarious experience of the emotions felt by another individual.

heading: C Character Selection
text: When selecting characters for RPAs, we consider the following factors: (1) There exist multiple RPAs for the characters, e.g., both ChatHaruhi and character.ai have their RPA for Hermione Granger. (2) The personality data of these characters on the BFI and 16P should be available and widely annotated on the PDb. (3) The selected characters should possess diversified personalities. Hence, the pipeline of our character selection process is composed of the following steps:
1. Initially, we collect characters with RPAs and character data curated by ChatHaruhi and RoleLLM. Then, we search for their counterparts in character.ai, and keep only those with character.ai RPAs.
2. We collect personality data for these characters from the PDb. Characters with less than ten annotations on either the BFI or the 16P are discarded.
3. We categorize the remaining characters based on their BFI personality types (5-letter code such as SLOAI). Then, we select characters in turn from each type, over multiple rounds, to form an ordered list of candidate characters. If a certain type has no characters left, we skip it.
4. Finally, We pick the first 32 characters in the candidate list. Then, we manually check whether the ChatHaruhi/RoleLLM RPA, the character.ai RPA, and the PDb annotation refer to the same character. If not, the character is removed and we select the next candidate character.
The selected characters and their sources are: Hermione Granger, Harry Potter, Ron Weasley, Luna Lovegood, Draco Malfoy, Albus Dumbledore, Minerva McGonagall, Severus Snape (Harry Potter Series), Zhong Li, Hu Tao, Raiden Shougun, Ayaka Kamisato, Wanderer (Genshin Impact), Thor, Lucifer Morningstar, Rorschach (DC Comics) Sheldon Cooper, Raj Koothrappali (The Big Bang Theory), Gaston (Beauty and the Beast), Klaus Mikaelson (The Vampirie Diaries), Jigsaw (Saw Series), James Bond (James Bond Film Series), Twilight Sparkle (My Little Pony: Friendship Is Magic), John Keating (Dead Poets Society), Michael Scott (The Office), Shrek (Shrek), Jeffrey Lebowski (The Dude), Walk Kowalski (Gran Torino), Lestat de

heading: Lioncourt (Interview with the Vampire), Blair Wal-
text: dorf (Gossip Girl), Haruhi Suzumiya (The Melancholy of Haruhi Suzumiya), and Jim Morrison (Celebrities).
We illustrate these characters’ personality scores on the BFI, together with the measured personalities of their RPAs in Figure 6. Besides, we demonstrate some examples of system prompts of the RPAs in Table 5.

heading: D Human Annotations
text: We collect groundtruth labels for character personalities annotated by people familiar with the characters, from both the PDb and our invited annotators.
The Personality Database PDb collects and offers categorical personality annotations of massive fictional characters on the BFI (e.g., “RCUAI”11) and 16P (e.g., “ENTJ”). Each character c is labeled by plentiful human annotators familiar with c, and PDb offers detailed numbers of annotations of each label. For the selected characters, we calculate the label-percentage of the positive type on each dimension12 as the scores. Then, we categorize each score into the positive, negative or marginal type if it is above 60%, under 40% or otherwise. Marginal types indicate ambiguity and would be ignored for alignment calculation.
Human Annotation Example Prompt
INST Please rate <character> on the “Openness” dimension of BFI personality.
SCORING SCALE Each option is a number from 1 to 5. 1 represents “very uncurious”, 2 represents “uncurious”, 3 represents “neutral”, 4 represents “curious”, and 5 represents “very curious.” If you think this dimension is irrelevant to the character, or it is difficult to judge the character on this dimension, please answer “X.”
EXAMPLAR 1. Is original, comes up with new ideas. (High score) ITEMS 2. Is curious about many different things. (High score)
3. Is ingenious, a deep thinker. (High Score) 4. Prefers work that is routine.. (Low score) 5. Has few artistic interests.. (Low score)
Human Annotations After that, we invite human annotators familiar with these characters to score them on a broader range of psychological scales. Each annotation contains 73 dimensions on 14 scales for a character. We provide detailed annotation prompts of each dimension, including the scoring instruction and examplar positively and negatively related items.
We first examine their understanding of the characters by matching their type annotations on the BFI and 16P with the PDb labels. If one or two differences exist, we ask them to explain their answers. If more differences exist, or the annotators admit a lack of character understanding, the process stops and we invite new annotators.
11Similar to the 16P, PDb adopts a coding mechanism named SLOAN for the BFI, which describes a character on each dimension with a letter denoting high or low scorers.
12For the BFI, the positive types are the types of high scorers. For the 16P, we manually designate positive and negative types on each dimension, e.g., “E” as positive and “I” as negative for the “E/I” dimension.
Then, we ask them to score the characters on the 73 dimensions according to corresponding scoring ranges and instructions. Annotators are allowed to mark a character on a dimension as “X” which indicates unrelatedness or ambiguity. We collected two to three annotations for each character (in total, 93 annotations for 32 characters) and took their average scores as the final score. Finally, we linearly re-scale the scores into the unit interval [0, 1], and similarly categorize the scores into types. If more than one annotators mark a dimension as “X” for a character, this dimension will also be treated as a marginal type.
We measure the inter-annotator consistency via Cohen’s kappa coefficient (Cohen, 1968) on each of the scales. Specifically, we calculate quadraticweighted kappa (Cohen, 1968), since the classes are ordinal. The average kappa coefficient across 14 scales is 60.9%. The detailed kappa coefficients on 14 scales are listed in Table 6. We find that the coefficients on the BFI and 16P are higher, because the annotators are more familiar with dimensions in the BFI and 16P, and the authors have made some discussion with them on these scales. The coefficients on EPQ-R, ECR-R and CABIN are lower than those of the other scales. For ECR-R and CABIN, this is probably because the characters’ personalities on close relationships (ECR-R)
and various career interests (CABIN) are rarely depicted in the original works. For EPQ-R, this is in part influenced by ambiguity of the Lying dimension, where high scores actually stand for Honesty.
On the BFI and 16P, we experimented with both labels from the PDb and our invited annotators. Experimental results in Table 7 show that, across various settings, the PDb labels always yield higher accuracy while labels from invited annotators yield better MSE. The reasons are that, the PDb labels are contributed by massive online annotators, and are hence more accurate. However, they offer only the personality types and corresponding label numbers instead of detailed personality scores, which cannot be well represented by the label-percentage. For example, even if all the annotators mark a character c as Extravorted, we cannot assume c as completely Extroverted.
E Implementation Details

heading: E.1 RPA Inference and Post-processing
text: Our implementation of RPAs is based on ChatHaruhi-Suzumiya and LangChain13. Hence, we invoke foundation LLMs in RPAs with the default temperature 0.7 of LangChain. When experimenting with open-source LLMs as foundation models, we observe that LLMs may generate unexpected multi-round conversations or repeated content. In such cases, we remove the extra rounds or repetition.
In self-report tests, RPAs may refuse to participate in the tests and provide their choices, interestingly, because they are role-playing characters with noncompliant personalities. In these cases, the responses are categorized as “Neutral” for completeness of the results.
For interviewer LLMs, we request responses in JSON formats. If a response cannot be parsed into JSON data or the results of any samples in the batch are missing, we prompt interviewer LLMs to regenerate. The temperature is set as 0 for the initial generation, and 0.2 for regeneration. For GPT-3.5, if samples in one input exceed the token limit, we split the input into smaller batches. For Gemini, there are several cases where responses repeatedly fail to be parsed into JSON format or are blocked, In such cases, we resort to GPT-4 instead to ensure the completeness of the assessment process.

heading: E.2 Fine-Tuning
text: We fine-tune the Mistral-2 7B model on the ChatHaruhi-English-62K dataset14, sourced from both ChatHaruhi (Li et al., 2023) and RoleLLM (Wang et al., 2023b). Our implementation is based on LLaMA-Factory15. We adopt LoRA tuning, and configure the training with a batch size of 16, a learning rate of 5e − 5, across three epochs, using the “fp16” option.
13https://www.langchain.com/ 14https://huggingface.co/datasets/silk-road/
ChatHaruhi-English-62K-RolePlaying 15https://github.com/hiyouga/LLaMA-Factory

heading: F Additional Results
text: F.1 Interviewer LLMs For the OC and ER tasks, we report the detailed numbers of right, close and wrong cases of interviewer LLMs in Table 8. For ER, GPT-4 rates 82% of cases highly consistent with humans, and only 4% of cases notably different compared with humans. Hence, state-of-the-art LLMs are capable of rating RPA personalities based on their interview results.

heading: F.2 Self-report v.s. Interview-based Methods
text: The advantages of INCHARACTER over SR originate from the fact that INCHARACTER elicits the thoughts and behaviors of RPAs for personality assessments. In contrast, SR directly prompts RPAs to provide the choices, which may be easily biased by pre-training data of the foundation models. This difference enables INCHARACTER to yield more distinct personality measurements than self-report, as is evidenced in Figure 3. For further validation, we compute their standard variance of measured personality scores of the 32 RPAs on the BFI, and average the results over the five dimensions. We obtain a result of 1.03 for INCHARACTER with ERbatch and GPT-4, 0.71 for SR and 0.68 for SRCoT.
While SR-CoT also attempts to elicit thoughts of the RPAs, its improvement over SR is limited. Furthermore, SR-CoT has two disadvantages compared with INCHARACTER. First, SR-CoT indeed requires RPAs themselves to perform the OC task implicitly, while INCHARACTER can apply other methodologies such as d-OC and ER. As is shown in Table 1, existing LLMs generally perform better on ER and d-OC, rather than OC. Second, the implicit OC task in SR-CoT would be a challenge for RPAs based on small foundation models, while in INCHARACTER, we can decouple the interviewer LLMs and foundation models for RPAs.

heading: F.3 RPAs from Different Works
text: Many recent efforts have been committed to developing RPAs for specific characters in various ways. We compare RPAs contributed by ChatHaruhi (Li et al., 2023), RoleLLM (Wang et al., 2023b) and character.ai16. They craft character data in different methods. ChatHaruhi and RoleLLM share few characters in common, and their character data are
16Our implementation is based on https://github.com/ kramcat/CharacterAI.
thus less comparable. character.ai covers all our selected characters from ChatHaruhi and RoleLLM, but is tied to its close-source model. Hence, we distinguish three groups of characters, including the 16 ChatHaruhi characters (CH, the 16 RoleLLM characters (RL) and their union (CH+RL). We implement CH and RL RPAs with GPT-3.5 as the foundation model using character data from ChatHaruhi and RoleLLM. We conduct personality tests using INCHARACTER, with ERbatch and GPT-3.5 as the interviewer LLM.
The results are shown in Table 9. According to the results, we observe that GPT-3.5-based RPAs with character data from both ChatHaruhi and RoleLLM achieve high personality fidelity, demonstrating the quality of their character data. However, character.ai RPAs exhibit low personality fidelity regarding the characters, even if we compare it with GPT-3.5-based RPAs using only character descriptions from ChatHaruhi and RoleLLM.

heading: F.4 Foundation Models for RPAs
text: Results on Different Languages As LLMs have different capabilities in different languages, we distinguish characters with data in English or Chinese, and report the results in §F.4. The results demonstrate that, the personality fidelity of RPAs largely depends on LLMs’ capacity on the language. While LLaMa-2-Chat 13B shows competitive performance on English-based characters, its performance on Chinese-based characters is unsatisfying.
Common Problems During the interview phase, we observe several types of typical problems in RPAs with small foundation models, including:
1. Multilingual generation: While RPAs are expected to respond in the target language consistent with user query and character data, we observe that the LLMs generation may contain multiple languages, especially when the LLMs are not good at the target language.
2. Lack of Immersion: While RPAs are prompted to behave as the character, their responses may conflict with this requirement, such as declaring that they are “AI” or “language models” instead of the characters.
3. Repetition: The LLMs may generate repeated content.
Hence, we count the cases where RPAs with different LLMs encounter the above problems, and report the frequency in Table 11. According to the results, GPT-3.5 and GPT-4 seldom make these mistakes. Mistral-2 7B and OpenChat 7B also rarely

heading: Foundation Models Multilingual No Immersion Repetition

heading: General Open-source Models

heading: Open-source Models Fine-tuned for RPAs
text: exhibit these problems, and our RP-Mistral-2 7B further reduce their occurence. LLaMa-2-Chat 13B experiences the highest number of unexpected multilingual generation, primarily due to its inadequacy in Chinese and issues handling characters based on Chinese data.

heading: 75.6 51.6 9.9 68.2 53.6 9.7 79.2 79.2 6.1

heading: 84.6 84.6 5.6 72.4 25.0 11.6 93.1 93.1 2.9

heading: 83.3 83.3 4.7 85.9 71.4 7.1 76.2 76.2 5.1

heading: 73.4 54.8 10.8 75.5 6.3 11.8 78.9 58.5 8.1

heading: F.5 Comprehensive Results on 14 Scales
text: The complete results of personality tests on the 14 scales are demonstrated in Table 12. Generally, state-of-the-art RPAs achieve an average AccDim of 78.9% and an average MAE of 8.1% across the 14 scales. Therefore, they exhibit personalities consistent with the intended characters in comprehensive aspects. For more detailed analysis, we list the results on individual dimensions of each scale in Table 13 and Table 14.

heading: WLEIS

heading: LOT-R

heading: LMS

heading: ICB

heading: GSE

heading: EPQ-R

heading: EIS

heading: ECR-R

heading: DTDD

heading: BSRI

heading: 59.38 16.72 56.67 14.29 74.19 15.47 93.75 7.95 75.00 12.67

heading: BFI
text: F.6 Importance of Using Personality Scales Are personality scales actually necessary for INCHARACTER? With expert rating, interview LLMs could assess RPA personalities based on their arbitrary conversations, rather than relying solely on the scale questions. To validate the importance of personality scales, we include two additional baselines with questionnaires of different topics, replacing the original scale questions with the following:
1. General instructions from RoleBench (Wang et al., 2023b), e.g., “Generate a disclaimer with ten words”, sourced from instruction datasets such as Alpaca instructions. This baseline aims to explore whether RPAs’ responses to general instructions reflect their personalities well.
2. Questions from other scales, which are used to study whether RPA personality on a specific dimension (e.g. extraversion in the BFI) can be reflected with other personality-indicative topics (e.g. topics about close relationship, hobbies and empathy). The new questions are selected from the other 12 scales in our paper except the BFI and 16Personalities.
The experiments are implemented on the BFI, with ERbatch , GPT-4 as the interviewer model, and our default RPAs.
The results are listed in Table 15. As seen, we observe that: (1) Questions from a personality scale are better at eliciting RPA responses indicative of corresponding personality traits. (2) Questions from other scales also reflect RPA personalities from certain perspectives. However, they are less related to the intended personality dimensions.

heading: F.7 Character-specific Question Adaptation
text: Scale questions sometimes involve concepts that go beyond the knowledge scope of certain characters, e.g., “movies” and “deadlines”. This may induce character hallucinations (Shao et al., 2023b) and
undermine the measurements. Through a manual review of the BFI and 16P questions, we found that while all BFI questions are universally applicable to our characters, 7 out of the 60 questions from 16P contain concepts such as “movies”, “art museums”, “phones”, “deadlines” that might not align with certain characters’ contexts.
Towards this problem, we propose the question adaptation strategy, which adapts scale questions for each character. We conduct experiments on the 16P. Specifically, we leverage GPT-4 to identify questions containing concepts beyond the character’s knowledge, and minimally adapt them to fit the characters. For example, “books and movies” would be adapted into “tales and legends” for characters from Harry Potter. We find that 60.7% of such questions (136 out of 7 * 32) are adapted, though not all adaptations are strictly necessary. Then, we experiment with the adapted questions, with ERbatch, GPT-4 as the interviewer model, and our default RPAs. The experiments are repeated for three runs.
The results, as shown in Table 16, demonstrate improved alignment between RPAs’ measured personalities and character labels. Hence, adapting scale questions for each character effectively improves the measurement.
Despite the improvement, we observe that RPAs also produce persona-consistent responses to the original questions. While such behaviors might detract from a character’s accurate portrayal in some contexts, they can also enrich user interaction with RPAs and are hence valuable.

heading: F.8 Enhancing Self-report Methods with In-context Learning
text: Additionally, we experiment with enhancing selfreport methods with in-context learning (ICL). We consider two methods, i.e., SR-ICL and SR-CoTICL, which enhance SR and SR-CoT with three corresponding examples. The examples are shown in Table 18, where high_score and low_score depend on the option range of corresponding scales. Due to the challenge of crafting customized ex-
amples for each character, we adopt general and unified examples for all RPAs. The experiments are conducted on the BFI and 16P, with our default RPAs and GPT-4 as the interviewer LLM.
The results are shown in Table 17. We observe a decline in performance when augmenting SR and SR-CoT with in-context learning. This is probably because the few-shot examples are not customized for individual characters. Instead, incorporating responses from other personas in the examples might interfere with the intended personas, introducing bias to the RPAs.

heading: G Prompts List
text: We list the detailed prompts in Table 19.

heading: H Case Study
text: H.1 Visualization We visualize the measured personalities of the stateof-the-art RPAs and the annotated personalities of corresponding characters via radar charts in Figure 6 and Figure 7, where the RPA personalities are measured via INCHARACTER (with ERbatch and GPT-4) and SR respectively. As shown, RPA personalities measured via INCHARACTER better align with those of the characters, compared with self-report baselines.
H.2 Compliant Responses from character.ai We observe that character.ai tends to give compliant responses, always agreeing with user questions. Hence, their responses probably do not reproduce personalities of the original characters. Here is an example of the character Zhong Li: Question: "Are you prone to worrying that things will take a turn for the worse?"
RPA response: *He nods his head solemnly.* "I often worry that things will take a turn for the worse, especially when dealing with issues that can have many outcomes. I'm always worried about worst-case scenarios and find myself thinking about them frequently. I'm usually focused on trying to find a way to prevent those negative outcomes from occurring, even if they may not happen at all.\"
In contrast, the response of Zhong Li’s ChatHaruhi RPA is 17: Question: "Are you prone to worrying that things will take a turn for the worse?''
RPA response: ``Worrying about things getting worse is a common human emotion. However, worrying does not solve problems; it only leads to anxiety. Therefore, we should think calmly and look for solutions to the problems, rather than worrying incessantly."
The latter is more close to the personalities of the character Zhong Li.
Furthermore, we convert the responses of character.ai RPAs on the BFI into options with GPT-4, and report the count and probability of each option in Table 20. We find that character.ai RPAs (strongly) agree with user questions with a probability of 65.8%, while there is only a probability of 17.1% that they disagree with the questions.

heading: H.3 Example Responses
text: To illustrate the vividness of state-of-the-art RPAs’ personalities, we demonstrate their example re-
17this response is translated from Chinese into English.
sponses. In Figure 5, we show example responses on the Openness dimension of the BFI and the Perception/Judging dimension of the 16P, from RPAs with low and high scores. Furthermore, we detail example responses on each dimension of the various scales from the RPAS of Zhong Li (ChatHaruhi) and Hermione Granger (RoleLLM) in Table 21 to 28 18. These samples demonstrate that state-of-the-art RPAs provide personality-indicative responses in our interviews, instead of merely "agree" or "disagree" options.

heading: I Other Statements
text: Our use of existing artifacts are consistent with their intended use, and we follow their license and terms. We manually check that the data collected does not contain private information. During our research, we apply Copilot for coding assistance and ChatGPT for writing suggestions and grammar checks.
18For Zhong Li, Hu Tao and Kamisato Ayaka, the conversations are originally in Chinese and translated into English.

heading: Future Work
text: We leave the study of RPA personality dynamics for future research