CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation (2024)

Abstract: Recently, the advent of large language models (LLMs) has revolutionized generative agents. Among them, Role-Playing Conversational Agents (RPCAs) attract considerable attention due to their ability to emotionally engage users. However, the absence of a comprehensive benchmark impedes progress in this field. To bridge this gap, we introduce CharacterEval, a Chinese benchmark for comprehensive RPCA assessment, complemented by a tailored high-quality dataset. The dataset comprises 1,785 multi-turn role-playing dialogues, encompassing 11,376 examples and featuring 77 characters derived from Chinese novels and scripts. It was carefully constructed, beginning with initial dialogue extraction via GPT-4, followed by rigorous human-led quality control, and enhanced with in-depth character profiles sourced from Baidu Baike. CharacterEval employs a multifaceted evaluation approach, encompassing thirteen targeted metrics on four dimensions. To facilitate the convenient evaluation for these subjective metrics in CharacterEval, we further developed CharacterRM, a role-playing reward model based on human annotations, which has a higher correlation with human judgment compared to GPT-4. Comprehensive experiments on CharacterEval demonstrate that Chinese LLMs exhibit more promising capabilities than GPT-4 in Chinese role-playing conversation. Source code, data source, and reward model will be publicly accessible at https://github.com/ morecry/CharacterEval.

heading: 1 Introduction
text: The development of large language models (LLMs) has marked the beginning of a new era in conversational AI (Zhao et al., 2023; Chang et al., 2023), and opened up a wide range of application possibilities, particularly in agent-based interactions (Park
∗*Corresponding authors:Xin Gao and Rui Yan (xin.gao@kaust.edu.sa, ruiyan @ruc.edu.cn)
et al., 2023; Wang et al., 2023a; Gao et al., 2023). The automated agents, equipped with the emerging capabilities of LLMs such as planning (Silver et al., 2022; Ge et al., 2023; Song et al., 2023), reasoning (Wei et al., 2022; Wang et al., 2022), and in-context learning (Dong et al., 2022; Brown et al., 2020), can perform complex tasks for humans without any supervision. Among the diverse agents, the Role-Playing Conversational Agent (RPCA), designed to offer emotional value instead of productivity, attracts an amount of interest.
RPCA represents a unique category within the realm of conversational agents, distinguished by their capability for immersive interaction (Li et al., 2023). Different from traditional dialogue systems, which typically focus on chit-chat (Yan et al., 2022), knowledge-based (Chen et al., 2020), personalized (Zheng et al., 2019) and empathetic dialogue (Ma et al., 2020), RPCAs engage users in dynamic scenarios, where LLM agents are assumed as specific characters or roles, often derived from existing composition such as novels, films, cartoons, and games. The development of connections between fictional characters and humans has the potential to not only deepen the impact of cultural works but also improve human engagement. Furthermore, RPCAs hold significant application value in their ability to offer emotional value to users, positioning fictional characters as virtual friends. The multifaceted nature of RPCAs has sparked considerable attention, leading to a surge in both research (Shao et al., 2023; Wang et al., 2023d; Tu et al., 2023; Zhou et al., 2023) and application development (e.g., Character AI1, Tongyi Xingchen2 and Glow3). However, these implementations of RPCAs vary significantly in both approach and objectives, presenting a challenge in systematically assessing and comparing their capabilities. There-
1https://beta.character.ai 2https://xingchen.aliyun.com/xingchen 3https://www.glowapp.tech/
1 11836
fore, we propose the CharacterEval, a Chinese role-playing conversation benchmark for advancing RPCA development.
To develop a benchmark, the primary problem is the construction of a dataset. While there are existing datasets (Shao et al., 2023; Wang et al., 2023d; Tu et al., 2023; Zhou et al., 2023; Chen et al., 2023; Shen et al., 2023), their quality is concerning, which are either generated by LLMs or suffering from significant noise due to the extractive methods. These limitations render the evaluation results unreliable for the RPCA’s actual capabilities. To address it, we constructed a Chinese role-playing conversation dataset comprising 1,785 multi-turn roleplaying dialogues, encompassing 11,376 examples and 77 leading characters, drawn from diverse Chinese novels and scripts. Our process began with the collection of well-known sources across various genres. After that, GPT-4 was employed to extract dialogue scenes, utterances, and behaviors of the leading roles of these sources. Following basic preprocessing and the removal of dialogues with fewer turns, we invited annotators to assess the quality of the dialogues. Their task was to identify and retain high-quality dialogues while discarding those of lower quality. Additionally, we crawled detailed character profiles from Baidu Baike4, composing a comprehensive dataset for RPCA evaluation. The example from the dataset is as Figure 1 shows.
Otherwise, role-playing conversation is a compli-
4https://baike.baidu.com/
cated task that requires not only mimicking a character’s behavior and utterance but also maintaining the character’s knowledge, as well as excellent multi-turn ability. Considering this, we proposed a multifaceted evaluation approach including thirteen specific metrics on four dimensions for a fair and thorough assessment of RPCAs, Our evaluation approach considered conversational ability, character consistency, and role-playing attractiveness, and utilized a personality back-testing method to evaluate the personality accuracy of an RPCA. To assess conversational ability, we measured conversational fluency, coherence, and consistency at both the sentence and conversation levels (Chen et al., 2017). Character consistency is the most crucial in role-playing conversation. Hence, we evaluated knowledge and persona consistency to measure how vividly an RPCA can simulate a character. This involves assessing knowledge exposure, accuracy, and hallucination for knowledge consistency, and evaluating behavior and utterance consistency for persona consistency. Considering that RPCAs are entertainment-oriented, role-playing attractiveness is also an important element. We assessed this through human-likeness, communication skills, expression diversity, and empathy. Finally, we introduced personality back-testing. With the collected Myers-Briggs Type Indicator(MBTI) (Myers, 1962) personality types as a reference, we let RPCAs do the MBTI assessment and calculate the MBTI accuracy (personality back-test) as implemented in Wang et al. (2023b).
2
For convenient re-implementation, we invited 12 annotators to score responses generated by different models for the subjective metrics in our evaluation system. Based on the human judgments, we developed a role-playing reward model—CharacterRM, whose correlation with humans could surpass stateof-the-art LLM GPT-4. On CharacterEval, We conducted comprehensive evaluations for existing LLMs, encompassing both open- and closedsource models. Experimental results show the broad prospect of existing Chinese LLM while GPT-series models do not take the predominance in Chinese role-playing conversation.
In summary, our contributions are as follows:
• We create a large-scale, high-quality dataset for RPCA evaluation, consisting of 1,785 multi-turn role-playing dialogues, and 11,376 examples, featuring 77 leading characters from diverse Chinese novels and scripts.
• We propose CharacterEval, a new benchmark for RPCAs, which contains a comprehensive set of evaluation principles, encompassing thirteen specific metrics on four dimensions.
• We develop CharacterRM, a role-playing reward model for evaluating RPCAs in several subjective metrics, achieving better performance than GPT-4 in correlation with humans.
• We conducted thorough evaluations of existing LLMs on CharacterEval, including openand closed-source, and derived valuable findings from the results.

heading: 2 Related Work

heading: 2.1 Knowledge-based Dialogue
text: Knowledge-based dialogue systems integrate external knowledge resources, such as knowledge graphs or unstructured documents, into dialogue systems (Zhao et al., 2020; Li et al., 2020). Recent efforts have focused on improving the understanding and utilization of knowledge within these dialogues. For instance, Xue et al. (2023) introduced K-DIAL, which incorporates additional Feed-Forward Network (FFN) blocks into Transformers (Vaswani et al., 2017) to enhance factual knowledge expression and consistency in dialogue. Similarly, Chen et al. (2020) proposed a knowledge distillation-based training strategy to optimize the knowledge selection decoder. While these methods significantly advance knowledge selection and
utilization, they primarily address general knowledge. Role-playing dialogues, however, demand a more intricate approach, encompassing personalized knowledge, style, behavior, etc.

heading: 2.2 Personalized Dialogue
text: Personalized dialogue systems, which generate responses based on specific personas, represent another relevant area of research (Den Hengst et al., 2019; Zhong et al., 2022). Zheng et al. (2019) pioneered this field by creating the first large-scale personalized dialogue dataset, complete with persona labels. This dataset has spurred further advancements in the field. Additionally, Zheng et al. (2020) developed a pre-trained personalized dialogue model, which could generate coherent responses using persona-sparse dialogue. Although these studies begin to explore persona in dialogue, the personal profiles they utilize are typically limited to short-term, person-related information like name, age, and location, which are considered personalized knowledge in essence.

heading: 2.3 Character-based Dialogue
text: The most closely related research to this work involves recent developments in character-based dialogue systems, which aim to mimic the behavior and utterance style of specific characters (Shao et al., 2023; Wang et al., 2023d; Zhou et al., 2023). Shao et al. (2023) gathered character profiles from Wikipedia and generated character-based dialogues by prompting ChatGPT (OpenAI, 2022). Wang et al. (2023d) used GPT-4 to create character descriptions and developed detailed instructions for prompting ChatGPT to produce character-based dialogues. However, these approaches primarily rely on ChatGPT’s generative capabilities and may not accurately reflect the true personality of the characters. Li et al. (2023) addresses this by extracting role-playing dialogues from novels, scripts, and games, which better preserve the characters’ original traits. Despite this, their approach suffers from a lack of human-in-the-loop refinement and a scarcity of multi-turn dialogues in the dataset. Otherwise, Chen et al. (2023) develop a role-playing dataset focused on Harry Potter. However, the scarcity of diversity makes it hard to comprehensively evaluate the generalized RPCA.
3

heading: 3 Problem Formulation
text: The Role-Playing Conversational Agent (RPCA) is designed to engage in conversations with users by emulating specific characters. These characters are defined by their knowledge, behavior, and style of response. To achieve this, the RPCA utilizes a character profile, denoted as P , and the current dialogue context represented as Cn = [q1, r1, q2, r2, . . . , qn]. Here, qi and ri correspond to the i-th question and response in the dialogue, respectively. The goal for the RPCA is to generate a response rn that is consistent with the character’s profile, which can be represented as:
rn = RPCA(Cn, P ), (1)
where rn is composed of two elements: behavior and utterance. The behavior aspect is enclosed in brackets and provides a detailed description of the character’s actions, expressions, and tone. This separation allows for a fine-grained evaluation of the RPCA’s ability to not only generate appropriate utterances but also unique behavioral traits.

heading: 4 Data Collection
text: In this section, we detail the methodology for constructing the character-based, multi-turn dialogue dataset with high quality. Prior to initiating data collection, adherence to the following four principles is important:
• Fidelity to Source Material: It is crucial that all dialogues are in line with the original works, ensuring the character’s authenticity.
• Diversity in Distribution: The dataset must encompass a wide range of scenarios to thoroughly assess the role-playing capabilities.
• Multi-Turn Feature: The dataset should predominantly consist of multi-turn dialogues, rather than being limited to single-turn ones.
• Human-in-the-Loop: Active human involvement is necessary to guarantee the quality, as reliance solely on LLMs is insufficient.
The pipeline of data collection includes four steps: plot division, dialogue extraction, quality filtering, and human annotation.
Plot Division: The plots in narrative text such as novels and scripts are extremely complex, making it challenging to divide the text into meaningful
chunks. Using the sentence tokenization tool, without considering semantics, will result in breaking a conversation mid-way. To address this, we first employ GPT-4 to identify plot twists—sentences that signify the end of a continuous plot. These plot twists are then used to segment the text into chunks, each containing a complete plot.
Dialogue Extraction: Once we have the plot chunks, GPT-4 is utilized again, this time to extract role-playing dialogues. We design prompts for GPT-4 to perform information extraction, preserving characters’ utterances, behaviors, and scenes from the plots.
Quality Filtering: Dialogues in novels and scripts often involve more than two characters. Simply retaining dialogues between two characters and omitting others will distort the dialogue structure. Therefore, we opt to preserve dialogues following an ABAB pattern (dialogue between two characters) until a third character joins. This approach, while straightforward, helps maintain the original dialogue structure more effectively. Besides, we only keep the dialogue exceeding five turns (six sentences) reserved, filtering the short dialogues.
Human Annotation: Although LLMs have the capability to perform basic information extraction tasks, the randomness still affects data quality. To mitigate this, we invite human annotators to assess the coherence and quality of dialogues and eliminate any problematic instances.

heading: 5 Evaluation Metric
text: Different from traditional chatbots, we contend that RPCAs require a more comprehensive evaluation framework to assess their role-playing capabilities. As shown in Figure 2, we have devised a four-dimensional evaluation system, which includes conversational ability, character consistency, role-playing attractiveness, and personality backtesting, including thirteen metrics.

heading: 5.1 Conversational Ability
text: Basic conversational ability is the first consideration in role-playing conversation. Inspired by previous neural metrics, which evaluate the responses based on well-trained neural models, we introduce a similar approach to assess the fundamental conversational abilities of RPCAs. We focus on three key objectives for generated responses: fluency, coherency, and consistency (Zhang et al., 2021; Mesgar et al., 2020).
4
• Fluency (Flu.) measures the grammatical correctness of a response, indicating whether a response is readable and free from obvious grammatical errors.
• Coherency (Coh.) evaluates the topic relevance between the response and the context. Generally, when the user submits a query on a specific topic, an RPCA should respond following the topic instead of providing an irrelevant response.
• Consistency (Cons.) assesses the stability of RPCAs during a conversation. Responses of an RPCA should not contradict their own responses in previous turns.

heading: 5.2 Character Consistency
text: Character consistency plays a crucial role in evaluating the role-playing ability of the RPCAs. It will bring the most intuitive experience to users when the character consistency of RPCAs varies. Specifically, we evaluate character consistency from two levels, knowledge consistency and persona consistency. The former evaluates if an RPCA could respond based on the character’s knowledge, which includes knowledge exposure, accuracy, and hallucination metrics. The latter assesses if a RPCA’s reflection is in line with the character’s personality, including the behavior and utterance metrics.
• Knowledge-Exposure (KE). For assessing the informativeness of a response, it’s crucial
for an RPCA to reflect knowledge in its responses, as this supports the subsequent evaluation of its knowledge expression capabilities.
• Knowledge-Accuracy (KA). Once the RPCA demonstrates the ability to generate responses with specific knowledge, it’s important to assess whether this knowledge aligns with the character. The goal is for the RPCA to accurately generate responses based on the knowledge from the character’s profile.
• Knowledge-Hallucination (KH). Drawing inspiration from recent studies on hallucinations in LLMs (Rawte et al., 2023; Zhang et al., 2023), we include knowledge hallucination in the evaluation of role-playing dialogue. To enhance the user experience, the RPCA should maintain consistency with the character’s identity and avoid responding to queries involving unknown knowledge.
• Persona-Behavior (PB). A character’s behaviors, typically described within brackets, improve the embodied feeling of users by portraying fine-grained actions, expressions, and tones. Consistent behavior is indicative of an effective RPCA.
• Persona-Utterance (PU). Alongside behavior, a character’s speaking style is also important. Each character has unique expression habits. Therefore, the RPCA’s utterances should align with these habits to adeptly mimic the character.

heading: 5.3 Role-playing Attractiveness
text: As a conversational agent in the entertainment field, it is essential for an RPCA to be sensitive to the user’s emotions. Therefore, we introduce the character attractiveness dimension to assess the attraction of an RPCA during conversation. From the user’s perspective, we consider four key dimensions: human-likeness, communication skills, expression diversity, and empathy.
• Human-Likeness (HL). In the era of publicly available LLMs, these models often suffer from a perceived ’machine-like’ quality in their responses. Most LLMs, designed primarily for information seeking, tend to provide robotic and emotionless answers. However, in role-playing conversations, it is crucial for the
5
RPCA to exhibit a more human-like persona to minimize user resistance.
• Communication Skills (CS). In human society, the ability to skilfully communicate, often referred to as Emotional Quotient (EQ), significantly influences an individual’s likability. Accordingly, users are more likely to engage with an RPCA that demonstrates higher EQ, mirroring the popularity of individuals with strong communication skills in daily life.
• Expression Diversity (ED). The dialogues within CharacterEval are sourced from existing novels, scripts, and various literary works, featuring characters with rich and diverse expressive abilities in both their behaviors and utterances. Therefore, an RPCA should strive to express this diversity in conversation to provide users with a more immersive experience.
• Empathy (Emp.). While the primary role of an RPCA is not that of an emotional counselor, its ability to express empathy can significantly impact its favorability of users. Evaluating empathy in role-playing conversations advances the RPCA to come across as a more warm and friendly conversational partner.

heading: 5.4 Personality Back-Testing
text: Following the recent works on LLM personality testing (Pan and Zeng, 2023; Huang et al., 2023; Wang et al., 2023c), we conducted personality back-testing to assess the role-playing capability of the RPCA within the context of personality dimensions. In this study, we employed the MyersBriggs Type Indicator (MBTI) (Myers, 1962), a well-established personality classification method. To obtain the necessary labels, we collected MBTIs of characters featured in CharacterEval from an archive website5, which hosts a substantial character’s MBTIs. Using these MBTIs as ground truth, we evaluated the accuracy of the MBTI assessment 6 of RPCAs.

heading: 6 Experiment

heading: 6.1 Dataset Statistic
text: We split our CharacterEval into the training set and test set based on examples instead of conversations, where an example is composed of a tu-
5https://www.personality-database.com/ 6https://www.16personalities.com/
ple (Character, Context,Response). The statistic of the dataset is as Table 1 shows. Specifically, we display the turns distribution of test set in Figure 3 to explore the dataset feature. It is notably that over 20% examples have more than 10 turn in dialogue. These statistic demonstrates the multi-turn property of CharacterEval, satisfying the evaluation of RPCA’s performance at longer turns.

heading: 6.2 Experimental Setting
text: CharacterEval employs a comprehensive set of fine-grained subjective metrics (twelve metrics in conversational ability, character consistency, and role-playing attractiveness dimensions) to assess the multi-dimensional capabilities of an RPCA. However, it is important to note that a single evaluated example may not adequately represent all facets of RPCAs. Therefore, we introduce annotators to sparsely evaluate the performance matrix. This approach entails that each example in CharacterEval is assessed using a subset of these subjective metrics, leading to more differentiated evaluation results. Then, based on these selected metrics for each example, we recruit 12 annotators to score responses generated by different models on a fivepoint scale. The human judgments are used to develop a role-playing reward model (CharacterRM), with Baichuan2-13B-base as the backbone. Experi-
6
mental result shows that our CharacterRM exhibits a higher correlation with humans than GPT-4, as Table 3 shows. Although the performance of GPT4 will improve with the number of demonstration increase, the cost of it makes evaluation hard to implement. Consequently, we utilize our CharacterRM for subsequent evaluation of subjective metrics. In the personality back-test, we collect 54 ground MBTIs of characters in our dataset. The RPCAs should answer the MBTI questionnaires and then the accuracy will be computed.

heading: 6.3 Evaluated LLMs
text: In this work, we assess the performance of 10 baselines with different parameters, encompassing both open-source and closed-source models. For the open-source models, we evaluate their chatversion instead of base-version. For the closed-
source models, we utilize their official APIs to conduct performance evaluations. Specifically, we employ the gpt-4 version as the GPT-4, and gpt-3.5-turbo-1106 version as GPT-3.5 in our experiments. Among the evaluated models, CharacterGLM, MiniMax, Xingchen, and BC-NPC-Turbo are tailored for role-playing conversations, while the remaining models are designed for general chat applications. Notably, GPT-4 and GPT-3.5 stand out as the only two models trained on the dataset primarily composed of the corpus with the English language. We consistently employ the same prompt for each model, with minor adjustments made only for closed-source models.
Significantly, GPT-3.5 demonstrates the weakest performance in CharacterEval. Its tendency to generate overly safe responses, such as “I am just an AI assistant and cannot perform role-playing,” highlights its limitations for role-playing applications. This issue stems from the over-alignment by RLHF (Christiano et al., 2017), making it unsuitable for dynamic role-playing interactions.

heading: 6.4 Overall Performance
text: The results across four dimensions are clearly illustrated in Figure 4. BC-NPC-Turbo outperforms in three of these dimensions, whereas GPT-4 is distinguished in personality back-testing. Models specifically designed for role-playing dialogues, such as Xingchen, MiniMax, and BC-NPC-Turbo, demonstrate superior outcomes due to their targeted training.
In the realm of open-source models, InternLM20B and Baichuan2-13B show impressive potential. Despite lacking specialized customization for roleplaying conversations, these models present commendable results in most evaluation dimensions. In contrast, GPT-4’s effectiveness diminishes in
7
0.2
0.4
0.6
0.8
1.0
Conversational Ability
Character Consistency
Role-playing Attractiveness
Personality Back-Testing
Chinese role-playing conversations. Its primary training in English corpus limits the adaptability in complex role-playing scenarios and the deep understanding of Chinese culture.

heading: 6.5 Detailed Result
text: The detailed performance across thirteen metrics is presented in Table 4.
Regarding conversational capabilities, BC-NPCTurbo exhibits superior performance, evidenced by its excellent conversational consistency, as well as comparative fluency and coherency. In contrasting open-source and closed-source models, it is difficult to declare a definitive winner in this dimension. However, when we compare the homogeneous models, such as Qwen-7B versus Qwen14B, and XVERSE-7B versus XVERSE-13B, examining models of the same series, such as Qwen7B versus Qwen-14B, and XVERSE-7B versus XVERSE-13B, it becomes obvious that an increase in the number of parameters can enhance conversational abilities. In the category of models with fewer than 10 billion parameters, Baichuan2-7B and InternLM-7B demonstrate comparable competencies. In the role-playing specialized models, MiniMax stands out for its performance and only falls behind BC-NPC-Turbo. In contrast, GPT-4 and GPT-3.5 do not exhibit a marked superiority in this dimension. Furthermore, it is posited that complex role-playing conversations and scenarios in Chinese might challenge the GPT series, potentially leading to their diminished performance.
In terms of character consistency, the most crucial aspect for role-playing conversations, BCNPC-Turbo still leads significantly. It exhibits the highest accuracy in knowledge accuracy, minimal knowledge hallucinations, and consistent utterances and behaviors when acting as a character. Otherwise, MiniMax also shows notable performance, compared with the open-sourced models and remaining closed models. Once again, the GPT series falls short compared to Chinese LLMs in this dimension. Nonetheless, it is important to acknowledge that GPT-4 excels in knowledge exposure, underlining its strengths in knowledge-intensive tasks. Despite this, in the realm of knowledge accuracy, particularly concerning the understanding of Chinese classical characters, GPT-4 does not exhibit distinct superiority.
Furthermore, BC-NPC-Turbo stands out in roleplaying attractiveness, as demonstrated by its outstanding human-likeness and diverse expressions. As a state-of-the-art LLM, GPT-4 exhibits remarkable performance in communication skills, significantly surpassing other models. This reflects its powerful generalization ability, even in the Chinese role-playing scenario. Interestingly, InternLM-20B emerges as the leader in empathy, highlighting its unique potential to provide emotional support.
Similar conclusions are also observed in the personality back-test, where BC-NPC-Turbo, MiniMax, and GPT-4 demonstrate comparable levels of accuracy. In this particular dimension, the models are required to respond to multi-choice questions that are designed to reveal the underlying values of the roles they are portraying. Since this task does not demand extensive expression in the character’s text style, GPT-4 exhibits the best performance. This result highlights their ability to accurately embody a character’s fundamental personality traits and values.

heading: 7 Conclusion
text: In this work, we aim to build a comprehensive benchmark to evaluate recent Role-Playing conversational Agents (RPCAs). We introduce GPT-4 to extract the dialogues from the existing novels and scripts, proceeding with strict human filtering. After a series of processing, we release a high-quality multi-turn role-playing dataset. Besides, we construct a comprehensive evaluation system to assess the multi-dimensional ability of RPCAs. We also collect human annotation to train a character-based
8
reward model to measure the subjective metrics, for later convenient re-implementation. Extensive experimental results indicate that Chinese LLMs entail more promising capabilities than GPT-4 in Chinese role-playing conversations.
Limitations
The CharacterEval benchmark for Role-Playing Conversational Agents (RPCAs) in Chinese presents several limitations: (1) Dataset Diversity: The dataset primarily focuses on characters from specific Chinese novels and scripts, which may not fully represent the diversity of role-playing scenarios; (2) Subjectivity in Evaluation: Despite using a multifaceted approach, the evaluation’s reliance
on subjective human judgment can lead to inconsistent outcomes; (3) Cultural and Linguistic Scope: The benchmark’s focus on Chinese dialogues limits its applicability to other linguistic and cultural contexts. These limitations highlight the need for ongoing updates to the dataset and evaluation methods, as well as efforts to broaden the benchmark’s cultural and linguistic relevance.
Ethical Consideration
In developing CharacterEval, a benchmark for assessing Chinese Role-Playing Conversational Agents (RPCAs), we have carefully considered several ethical dimensions to ensure our research adheres to high ethical standards:
9
(1) Data Privacy and Permissions: We confirm that all materials used, especially dialogues derived from copyrighted Chinese novels and scripts, have been utilized in non-commercial purpose, respecting copyright laws and privacy policies.
(2) Fairness and Transparency in Annotation: In creating the CharacterRM (role-playing reward model), we have implemented a rigorous selection and training process for our annotators to ensure the fairness and transparency of their contributions. We have taken measures to address potential biases and ensure the annotations are consistent, highquality, and reflective of diverse perspectives.
(3) Responsible Use of RPCAs: Aware of the potential for emotional engagement and the risks associated with the misuse of AI-generated content, we will outlin ethical guidelines for the deployment of RPCAs. Our research includes safeguards to prevent the misuse of these agents, ensuring they are used in ways that are beneficial and respectful to users.

heading: Acknowledgement
text: This work was supported by the National Natural Science Foundation of China (NSFC Grant No. 62122089), Beijing Outstanding Young Scientist Program NO. BJJWZYJH012019100020098, and Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the “Double-First Class” Initiative, Renmin University of China, the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China.

heading: A Robustness Analysis
text: To evaluate the robustness of RPCAs, we select a range of models—InternLM-20B, MiniMax, BCNPC-Turbo, and GPT-4—for analysis. We aim to assess their effectiveness in different stages of a conversation. As illustrated in Figure 5, there is a noticeable trend where most models demonstrate a decline in performance as conversations progress. Remarkably, InternLM-20B maintains consistent performance in terms of character consistency and conversational ability. This could be attributed to the fact that these models, primarily designed for role-playing, have not significantly focused on longer dialogue sequences. This oversight is likely due to the challenges associated with collecting extensive role-playing conversation data. Similarly, GPT-4 exhibits a declined trend under longer conversations, affected by the complex Chinese role-playing scenarios. Our findings indicate that future advancements in RPCA development should focus on enhancing capabilities for longer conversational scenarios, ensuring more stable and consistent role-playing interactions.

heading: B Evaluation Result by GPT-4
text: Although GPT-4 has demonstrated the selfenhancement bias (Zheng et al., 2023) and has a lower correlation with human judgement 3, we present the evaluation result by GPT-4 in a 2-shot setting for reference, as shown in Table 5.

heading: C FAQs
text: Q1: Only Chinese novels and scripts limit the diversity of role-playing scenarios and characters. We are in the process of expanding the size of our dataset and will include a broader range of
language types. Please stay tuned, as this work is intended to be a starting point, not the endpoint. Your mention is valuable in guiding our efforts to enhance the diversity and richness of our dataset.

heading: Q2: Subjective metrics rely on a limited number
text: of human annotations, potentially introducing biases. We are committed to continuously augmenting our dataset with additional human annotations to ensure our evaluation results are more reliable, consistent, and unbiased. We acknowledge the importance of addressing these concerns and are actively working to improve the robustness of our evaluation methodology.

heading: Q3: Since the data sources are from the public
text: domain (e.g., TV shows), a comprehensive analysis of data leakage should be conducted. While our data is indeed derived from actual domains, we hold the point that evaluating role-playing capabilities by human for entirely novel characters poses a significant challenge due to the lack of prior knowledge about these characters. Therefore, our dataset construction leverages existing data sources to test the model’s ability to utilize known information for role-playing. Additionally, our evaluation metrics are designed to be reference-free, thereby minimizing the impact of data leakage. This approach ensures a more accurate assessment of a model’s ability to engage in role-playing scenarios while relieving concerns related to data leakage.

heading: Q4: The data only contains Chinese data, making it of limited interest to the broader audience.
text: As mentioned in response to Q1 of Reviewer s1sR, we are currently expanding our dataset to include a wider range of language types. This expansion is part of our ongoing effort to make our research more inclusive and relevant to a global audience. We believe that diversifying the dataset will significantly enhance its utility and appeal, facilitating broader applications and research in role-playing and character analysis. CharacterEval is intended to be a starting point, not the endpoint.

heading: Q4: The data only contains Chinese data, making it of limited interest to the broader audience.
text: As mentioned in response to Q1 of Reviewer s1sR, we are currently expanding our dataset to include a wider range of language types. This expansion is part of our ongoing effort to make our research more inclusive and relevant to a global audience. We believe that diversifying the dataset will significantly enhance its utility and appeal, facilitating broader applications and research in role-playing
13
and character analysis. CharacterEval is intended to be a starting point, not the endpoint.

heading: Q5: The entire evaluation process is unclear, including how all evaluation metrics, human
text: judgments, CharacterRM, and GPT4 are scored. Can these scoring results truly represent the original meanings of the metrics? Our evaluation process is primarily based on predefined annotated examples, which can be found in https://github. com/morecry/CharacterEval.
(1) Regarding the human annotation, we recruited 12 annotators including 8 males and 4 females with age from 18 to 26, divided into two groups. Each group scores all samples across specified metrics following the guidelines of the pre-
defined annotated examples. Consequently, each (Role information, context, response, metric) tuple receives two scores. In cases of scoring discrepancies, the respective annotators discuss to reach a consensus on the score.
(2) Regarding the GPT-4 evaluation, we employ the predefined annotated examples as demonstration for In-context learning to compute the scores. The API version we used is gpt-4-preview-1106.
(3) CharacterRM is developed based on the human annotation results on training set of CharacterEval. The annotated scores are normalized to [0,1] by dividing by 5. We then enhance Baichuan13B-Base with a classifier, applying a template to the (Role information, context, response, metric)
14
tuple, processing it through Baichuan-13B-Base and the classifier. We choose the classifier output of the last token and normalize it with a sigmoid function to obtain the reward score. The training aims to fit these normalized human scores using mean squared error loss. We use LoRA to optimzer the CharacterRM with such config that lora rank is 32, lora alpha 16, lora dropout is 0.05, trainable layers are all W_pack and o_proj layers, learning rate is 2e-5, epoch is 5, batch_size is 8, and compute dtype is bfloat16. The inference follows a similar procedure as training.
15

heading: Future Work
text: Our findings indicate that future advancements in RPCA development should focus on enhancing capabilities for longer conversational scenarios, ensuring more stable and consistent role-playing interactions