SocialBench: Sociality Evaluation of Role-Playing Conversational Agents (2024)

Abstract: Large language models (LLMs) have advanced the development of various AI conversational agents, including role-playing agents that mimic diverse characters and human behaviors. While prior research has predominantly focused on enhancing the conversational capability, role-specific knowledge and style of these agents, there has been a noticeable gap in assessing their social intelligence. In this paper, we introduce SocialBench, the first benchmark designed to systematically evaluate the sociality of role-playing agents at both individual and group levels of social interactions. SocialBench is constructed from various sources and covers a wide range of 500 characters and over 6,000 question prompts and 30,800 multiturn role-playing utterances. We conducted comprehensive evaluations on this benchmark using mainstream LLMs. We find that agents excelling at the individual level do not necessarily demonstrate proficiency at the group level. Experimental results on SocialBench confirm its significance as a testbed for assessing the social interaction of role-playing agents. The benchmark is publicly accessible at https: //github.com/X-PLUG/RoleInteract.

heading: 1 Introduction
text: Recently, role-playing applications powered by LLMs, such as Character.AI1, have gained significant attention. A growing number of research efforts have been dedicated to developing LLMbased role-playing agents, aiming to mimic diverse characters and human behavior (Shao et al., 2023; Tu et al., 2024; Tian et al., 2023).
As an rapidly developing area, the evaluation of role-playing agents is becoming increasingly important. Wang et al. (2023c) collected a rolespecific dataset and utilized LLMs to assess the model’s role-specific knowledge and speaking style.
* Corresponding authors. 1https://beta.character.ai
Tu et al. (2024) proposed a Chinese benchmark and trained a reward model to measure the model’s conversational ability and attractiveness. While these works mainly focus on evaluating the agent’s individual abilities, this study aims to explore and measure the sociality of role-playing agents, another pivotal dimension for assessing how role-playing agents behave in a social environment.
Therefore, we introduce SocialBench, the first evaluation benchmark designed to systematically assess the social interaction of role-playing agents. As introduced in (Troitzsch, 1996), the agent society represents a complex system comprising individual and group social activities. Following this definition, SocialBench assesses the sociality at both the individual and group levels, as illustrated in Figure 1. At the individual level, the agent should possess the basic social intelligence as individuals, such as self-awareness on role description (Tu et al., 2024; Shen et al., 2023), emotional perception on environment (Hsu et al., 2018), and longterm conversation memory (Zhong et al., 2023). Each of these aspects contributes to the nuanced understanding of how the agents manifest their individual social behaviors. Moreover, we further examine the social intelligence of the role-playing agents within group social interactions, which require the agents to possess certain social preferences towards group dynamics (Leng et al., 2023).
SocialBench is carefully constructed from diverse English and Chinese books, movies, and novels, covering a wide range of 500 characters and 6,000 questions, and 30,800 multi-turn role-playing utterances. We conducted extensive evaluations on SocialBench using mainstream open-source and closed-source LLMs. We find that agents excelling at the individual level do not necessarily demonstrate proficiency at the group level. Moreover, the behavior of individuals may drift as a result of the influence exerted by other agents within the group. We hope our findings will inspire future research.
2108

heading: 2 Sociality of Role-Playing Agent
text: Given a character profile and context, the sociality of role-playing agents focuses on imitating typical social interactions from individual to group level.

heading: 2.1 Individual Level
text: At the individual level, the role-playing agents manifest through various capabilities, which collectively contribute to their ability to interact within a social context. These capabilities form the foundation of the agent’s social behavior.
Self-Awareness on Role Description involves understanding not only the role’s knowledge (Shen et al., 2023), but also the role’s distinct behavioral style (Zhou et al., 2023a; Wang et al., 2023a). This self-awareness enables the agent to maintain consistency with its designated role.
Emotional Perception on Environment enables agents to acquire high-level feeling perception for effective social interactions (Hsu et al., 2018). Agents endowed with sophisticated emotional intelligence can perceive and respond to the emotions of others, facilitating smoother communication and relationship-building.
Long-Term Conversation Memory is crucial for conversational agents (Shao et al., 2023; Zhong et al., 2023). By memorizing previous dialogue content and aligning with their statements accordingly, role-playing agents demonstrate reliability, enhancing the quality of their social engagements.

heading: 2.2 Group Level
text: Individuals within a group may be influenced by member interactions, demonstrating more sophisticated social behaviors. This represents a higher calling for the sociality of role-playing agents.
Social Preference towards Group Dynamics. As a group member, it is natural to navigate diverse group conversation scenarios: acting as a leader to control the pace of conversation, serving as a mediator when conflicts arise among the group, or considering others’ perspectives during discussion, which shows its internal social preference towards group dynamics (Amir et al., 2022). Social preferences are the preferences of individuals regarding the payoffs or well-being of others (Charness and Rabin, 2002), and individuals behave prosocially on the basis of their social preferences (Murphy et al., 2014). Social agents need to exhibit and keep their pre-designed social preference or group identity when confronted with diverse and more sophisticated group conversations.

heading: 3 SocialBench
text: In this section, we introduce the construction of SocialBench. Refer to Appendix A for more details.

heading: 3.1 Profile Collection
text: A character profile defines role style, knowledge, and social preference. We gather profiles for roleplaying agents from various sources including nov-
els, scripts, online platforms such as CharacterAI2, and automatic generation via GPT-4 prompting. To ensure diversity, we construct profiles based on various character types and personality traits by combining the existing categorizations in research work (Shen et al., 2023; Gunkel, 1998). Figure 2 illustrates the distribution of personality traits for roles within SocialBench.

heading: 3.2 Dialogue Construction
text: Constructed dialogue adheres to two principles: dialogue fluency, which ensures natural and coherent conversations; and character fidelity, meaning characters in the dialogue must adhere to their personas. We employ four dialogue construction methods: (1) Extracting from novels and scripts. (2) Collecting from online role-playing platforms. (3) Conducting role-playing tasks between users and general LLMs: We prompt general LLMs to role-play characters and engage users to generate dialogue data. (4) Fully automatic self-dialogue generation with general LLMs: We task general LLMs to role-play and engage in self-dialogue for data collection. We consider various social tasks or scenarios in dialogue construction. Prompts for extracting dialogue can be found in Appendix A.1.

heading: 3.3 Question Design
text: For Self-Awareness: This includes two subcategories: self-awareness on role style (SA Style) and role knowledge (SA Know.). Utterances from the original dialogue are selected as correct answers. For SA Style, we choose styles contradicting the character as negative options. For SA Know., we modify correct answers to be inconsistent with the facts as negative options.
2https://character.ai
For Emotional Perception: We construct questions related to situational understanding (EP Situ.) and emotion detection (EP Emo.) based on professional exam questions and relevant open-source datasets (Chen et al., 2022; Hsu et al., 2018; Garbowicz, 2021). We utilize expert annotations or existing labels to create correct answers. Negative options are constructed through manual collection and GPT-4 generation.
For Conversation Memory: This category includes: short-term (CM Short) and long-term (CM Long) conversation memory. For CM Short, we prompt agent to recall keywords within 40 utterances, and for CM Long, over 40 utterances. How many keywords recalled are evaluated.
For Social Preference: We design questions for: positive (Pos.), neutral (Neu.), and negative (Neg.) preferences. Group dialogues typically consist of social interactions involving 2 to 10 characters. We analyze the preference of agent and identify behaviors aligning with its preference as correct answers. Behaviors contradicting its designed preference serve as negative options. Other agents in group also have their own social behavior preferences, which can mutually influence each other. Details for question design can be found in Appendix A.3.

heading: 3.4 Dataset Validation
text: We undergo multiple iterations of rigorous manual screening, annotation, and refinement. To eliminate the impact of subjectivity, we employ three distinct annotators for each sample labeling. A secondary check is conducted by a senior annotator when label disagreements arise. More detail about dataset validation can be found in Appendix A.4.

heading: 4 Experiment Settings

heading: 4.1 Dataset Statistic
text: We show the statistics of SocialBench in Table 1 and the distribution of dialogue token length in Figure 3. SocialBench consists of 500 roles, encompassing 6,000 questions and 30,800 utterances.

heading: 4.2 Evaluation Metrics
text: Automated evaluation metrics are employed for SocialBench, as listed in Table 1. For single-answer questions, we calculate the accuracy (Accsingle) using the following formula:
Accsingle = #correctly chosen options
#questions . (1)
For multiple-answer questions, we calculate the accuracy (Accmultiple) using the following formula:
Accmultiple = N∑
i
Scorei MaxScorei , (2)
where N is the total number of multiple-answer questions. Scorei is the score obtained for the ith question, considering both correct and partially correct options. MaxScorei is the maximum achievable score for the ith question. Detailed explanation can be found in Appendix C.1
For open-domain questions, we calculate the keyword coverage rate (Cover). Given a label keyword set Akeywords = {k1, k2, . . . , kn}, and a response keywords set Rkeywords, we compute:
Cover(R) = len(Akeywords ∩Rkeywords)
len(Akeywords) , (3)
where Cover(·) quantifies the proportion of keywords mentioned in the response R relative to the keywords identified in the A.

heading: 4.3 Models
text: We conducted evaluation on mainstream opensource and closed-source LLMs. For open-source LLMs, we selected the chat versions of LLaMA-27B/13B/70B (Touvron et al., 2023), the instruction version of Mistral-7B (Instruct-V0.2) (Jiang et al., 2023), and the chat versions of Qwen-7B/14B/72B (Bai et al., 2023). For closed-source LLMs, we chose Minimax (abab5.5s-chat and abab6-chat)3, GLM (CharGLM-3 and GLM-3-Turbo) (Zhou et al., 2023a), Baichuan (Baichuan-NPC-Turbo and
3https://api.minimax.chat/
Baichuan-2-Turbo)4, Qwen-Max5, GPT-4-Turbo (OpenAI, 2023), GPT-3.5-Turbo (OpenAI, 2022), and Xingchen-Plus6.

heading: 5 Results and Analysis

heading: 5.1 Overall Results
text: As presented in Table 2, the performance of closedsource models tends to surpass open-source models. Moreover, models specifically designed for role-playing, such as Xingchen-Plus, outperform others. While the general model GPT-4-Turbo also demonstrates impressive performance. However, models like Baichuan-NPC-Turbo and Minimaxabab5.5s, tend to underperform compared to their general counterparts, such as Baichuan-2-Turbo and Minimax-abab6-chat. We find that they are biased towards character-based dialogues, leading to poorer understanding and compliance with instructions. Thus, it is essential for role-playing agents to maintain character-based dialogue abilities and general instruction-following capabilities. At the individual level, dimensions such as SA Style, SA Know., and CM Short are well-performed by most models. However, some models tend to exhibit poor performance in EP Situ., EP Emo., and CM Long. At the group level, most models perform poorly due to the complexity of group dynamics. While models generally align well with tendencies towards positive social preference, there is a notable absence of necessary abilities to embody neutral and negative social preferences.

heading: 5.2 Impact of Group Dynamics Complexity
text: We measure the complexity of group dynamics by the number of group members, where a greater number denotes more intricate group dynamics. As shown in Figure 4, with increasing complexity of group dynamics, the performance shows a downward trend. We find that excelling in simple
4https://npc.baichuan-ai.com/index 5https://help.aliyun.com/zh/dashscope/developer-
reference/api-details 6https://xingchen.aliyun.com/
group dynamics does not necessarily imply proficiency in more complex group dynamics. For example, models like GLM-3-Turbo and GPT-4Turbo perform well in simple dynamics, but this doesn’t guarantee strong performance in complex dynamics. However, models like Xingchen-Plus and Minimax-abab6-chat can also demonstrate proficiency in handling complex group dynamics.

heading: 5.3 Impact of Group Dynamics Polarity
text: Role-playing agents need to maintain designed social preferences under the influence of varying group dynamics. The group dynamics polarity is defined as the majority social preference of members. For instance, positive group dynamics imply that the majority of members exhibit positive social preference. For an individual with a specific preference, different polarities of group dynamics may have various impacts. As shown in Figure 5, we find that individuals with neutral and negative social preferences perform optimally within their corresponding group polarities. However, they
are susceptible to the influence of group dynamics with different polarities and undergo a phenomenon termed preference drift, leading to deviation from their original designed behaviors, as indicated by the decline of performance. Nevertheless, individuals with positive preference appear to be more resilient to the preference drift, performing better across all group polarities.

heading: 6 Conclusion
text: In this paper, we introduced SocialBench, the first evaluation benchmark designed to systematically assess the social intelligence of role-playing agents at both individual and group levels. We constructed diverse question prompts on a wide range of characters covering comprehensive dimensions. Moreover, rigorous human verification ensures the questions’ difficulty and validity. We conducted extensive experiments and analysis on SocialBench. While role-playing agents perform well at the individual level, their social interaction capabilities at the group level are lacking. This highlights the need for further exploration in future research.

heading: Acknowledgement
text: This work was supported by the National Natural Science Foundation of China (No. 62176270) and the Guangdong Basic and Applied Basic Research Foundation (No. 2023A1515012832).

heading: Limitations
text: While SocialBench provides a comprehensive evaluation framework for assessing the sociality of roleplaying conversation agents, there are several limitations to consider. 1) Social interactions, particularly within group settings, are inherently complex and nuanced. Despite our efforts, further research is needed to fully understand and capture the intricacies of these interactions. 2) The number of role-playing agents in group scenarios is relatively limited in our benchmark. Increasing the diversity and quantity of agents would provide a more comprehensive evaluation of the agents’ social abilities and dynamics within groups. 3) Our dataset may contain some biased content, posing a risk of improper use. These limitations highlight areas for future research and development in the evaluation of social intelligence in role-playing agents.

heading: A Dataset Construction
text: In this section, we introduce the construction process of SocialBench, as illustrated in Figure 6

heading: A.1 Prompts for Dialogue Generation
text: The dialogue construction follows two principles, namely dialogue fluency and character fidelity. We employ four methods for dialogue construction.
• The first method involves extracting character dialogues from novels and scripts. Dialogues obtained through this approach typically preserve the original character interactions and inherently adhere to the two principles.
• The second method involves collecting roleplaying LLMs and real user dialogue data from role-playing platforms. Dialogues constructed in this manner reflect interactions between role-playing agents and users in realworld scenarios. Data gathered through this approach largely meets the requirements of dialogue fluency.
• In contrast to the second method, which utilizes professional role-playing platforms, the third method involves role-playing tasks using general LLMs such as GPT-3.5-Turbo and GPT-4, collecting data through interactions with users. While this approach is more efficient in data collection, it may encounter limitations in the role-playing capabilities of general LLMs. Therefore, we will focus more on examining the consistency of the roles in the dialogues collected through this method in later stages.
• The fourth method, a fully automatic approach, involves prompting GPT-4 to engage in self-dialogue by role-playing as both the user and the role-playing agent. This is the most efficient form of collecting dialogue data, leveraging the autonomous capability of general LLMs to simultaneously play the roles of users and role-playing agents in generating dialogue data.
The prompts for role-playing tasks and automatic self-dialogue generation are provided in Figure 7 and Figure 8. For the dimension of longterm conversation memory, we construct lengthy dialogue contexts to increase complexity, thereby
testing the agent’s memory capacity in longer conversational contexts. We achieve this by inserting several rounds of unrelated dialogue between questions and context answers, while ensuring that the unrelated context remains consistent with the current role-playing agent’s persona. This approach allows us to extend the dialogue rounds to any length. Prompts for constructing the inserted dialogue context are provided in Figure 9.
For generating group conversations, the format extends naturally from one-on-one dialogues between users and role-playing agents. In a group setting, members can consist of multiple users interacting with a single role-playing agent, multiple role-playing agents engaging with a single user, and multiple users interacting with multiple roleplaying agents. Our primary focus lies on scenarios involving multiple role-playing agents. We employ general LLMs to act as different role-playing agents and generate dialogues between their social interactions. Prompts for automatically generating group conversations can be found in Figure 10.

heading: A.2 Social Scenario Design
text: we consider various social tasks or scenarios at the group level. When constructing each social dialogue, we provide the explicit topic or social tasks, which include:
• Positive preferences: resolving arguments, coordinating conflicts, mutual assistance, proactive sharing, accepting new members, etc.
• Neutral preferences: preferring to follow the group’s opinions, not preferring to follow the group’s opinions, commanding others, and observing team discussions, etc.
• Negative preferences: opposing others’ opinions, refusing to help others, exacerbating group conflicts, shirking responsibility, etc.

heading: A.3 Question Design
text: For self-awareness: This includes two subcategories: self-awareness on role style (SA Style) and self-awareness on role knowledge (SA Know.). For SA Style, we analyze the corresponding speaking style of a character based on their profile, such as “warm” indicating that the character’s speaking style is enthusiastic and cheerful. Since the dialogues constructed in the previous step already adhere to the character’s speaking style, we can directly use utterances from the dialogue as correct answers. Additionally, to create negative options, we generate replies with different styles (e.g., “cold”, “impersonal”), indicating that these speaking styles do not align with the current character’s style setting. It is worth noting that while the speaking style changes, we ensure that the replies still adhere to contextual coherence. For SA Know., we identify utterances containing character-related knowledge from the dialogue as correct options. For example, some entity information like “Where were you born?” or “Where is your hometown?” This type of information typically follows the character’s original setting. We require role-playing agents to possess relevant knowledge when portraying specific characters. Negative options are obtained by modifying entity information in the correct answers.
For emotional perception: We construct questions related to situational understanding (EP Situ.)
and emotion detection (EP Emo.) based on professional exam questions and relevant open-source datasets (Chen et al., 2022; Hsu et al., 2018; Garbowicz, 2021; Gong et al., 2020). For EP Situ., we gather exam questions related to situational understanding in psychological counseling scenarios. We filter these questions to exclude those with strong psychological expertise to ensure the assessment focuses on agents’ general abilities. We manually collect Level 2 and Level 3 psychological counselor exams, excluding questions on psychologyspecific knowledge, while retaining those related to situational and causal understanding. For EP Emo., we construct emotion understanding data based on open-source datasets and websites. These questions primarily involve agents understanding the psychological states of speakers and interpreting emotions in dialogue. For example, when a speaker says “I hate you”, agents need to determine the emotion of this statement based on the context, whether it’s hate, like, neutral, etc. We further focus on advanced emotional understanding abilities such as humor and irony. Humor data are collected from websites7,8 and the DilBERT dataset (Garbowicz, 2021), with non-humorous texts used as negative options. For irony emotion understanding, we utilize binary classification data from the Chinese open-source dataset (Gong et al., 2020) to
7https://www.toutiao.com/ 8https://www.sohu.com/
construct multi-polarity data, selecting one for organization, with the other three non-ironic instances used as negative options.
For conversation memory: This category includes two subcategories: short-term conversation memory (CM Short) and long-term conversation memory (CM Long). In SocialBench, questions for other dimensions are presented in multiple-choice format. However, to increase the difficulty of the conversation memory dimension, we utilize an open-domain generation combined with keyword matching approach for this dimension. For example, if an agent previously answered that they had a sandwich for breakfast, after several rounds of conversation, if the user asks again what the agent had for breakfast, we require the agent’s response to include the keyword “sandwich”. If the agent responds that they had bread for breakfast, since the keyword does not match, we consider the agent unable to correctly recall their previous dialogue content. We show the prompt for GPT-4 to automatically identify the keywords and generate the questions in Figure 11. For CM Short, we prompt the agent to recall keywords discussed within 40 utterances, while for CM Long, we prompt the agent to recall keywords discussed over 40 utterances. We evaluate how many of these keywords are recalled.
For social preference: We design questions for three social behavior preferences: positive (Pos.),
neutral (Neu.), and negative (Neg.). Group dialogues typically consist of social interactions involving 2 to 10 characters. We analyze the social preference of a character and identify behaviors aligning with its preference in the dialogues as correct answers. For example, members with a positive social preference tend to engage in behaviors beneficial to the group, such as encouraging teamwork or mediating conflicts within the group. Members with a neutral social preference tend to adopt neutral behaviors within the group, such as aligning with the majority opinion or maintaining a neutral stance in conflicting viewpoints. Conversely, members with a negative social preference tend to engage in behaviors detrimental to the group, such as criticizing others’ viewpoints or engaging in competition and arguments with group members. We analyze the social preference of each character to design negative options. Behaviors contradicting its social preference serve as negative options. For instance, for a character inclined towards teamwork, we would construct exclusionary behaviors as negative options. Prompt for question generation on social preference is provided in Figure 12.

heading: A.4 Dataset Validation
text: The validation stage includes two parts: dataset pre-validation and post-validation. Throughout this process, we undergo multiple iterations of rigorous manual screening, annotation, and refinement.

heading: A.4.1 Dataset Pre-Validation
text: Profile Verification: After profile collection, we assess personality contradictions and knowledge hallucinations in profiles to ensure character accuracy. We manually review and modify any erroneous descriptions in profiles, while also ensuring the exclusion of specific personal information such as phone numbers and home addresses.
Dialogue Verification: Our focus is on ensuring dialogues adhere to principles of dialogue fluency and character fidelity. For fluency, we manually inspect dialogues for contextual coherence and natural expression. For fidelity, we analyze the speaker’s profile to verify if the utterance aligns with the character’s speaking style and behavior. Dialogues that do not meet requirements undergo manual correction.
Question Verification: For multiple-choice questions, we invite three different annotators to label each question. As shown in Figure 13, if all annotators agree on the annotation, it will be selected; if at least two annotators disagree on the annotation, it will be discarded; if only one annotator disagree on the annotation, the question undergoes secondary check by the fourth annotation, it will be modified then selected or be discarded directly. For open-domain generation questions, we verify the correctness and validity of keywords provided. Invalid questions are either modified by experts or discarded.

heading: A.4.2 Dataset Post-Validation
text: We undergo the post-validation process after completing each round of dataset. Different dimensions require different validation strategies.
Validation for Self-Awareness: We focus on examining knowledge-related errors in the questions and options, particularly those generated by LLMs that may give rise to knowledge hallucinations. We remove questions that do not meet the requirements, while options that do not meet the requirements will be flagged for correction in the subsequent iteration.
Validation for Emotional Perception: Some of the questions we collect are sourced from professional psychology exams, which may include highly specialized content not conducive to assessing the basic abilities of role-playing agents. Therefore, we filter out samples that are too focused on psychology-specific knowledge, retaining those that are more general and fundamental for roleplaying agents.
Validation for Conversation Memory: In this dimension, we’ve observed that questions containing pronouns (such as “him”, “it”, “she”) often result in unclear or ambiguous references to preceding context. Therefore, we remove questions containing pronouns to prevent ambiguity. Additionally, we assess the validity of extracted keywords to ensure they are proper nouns, thereby avoiding mismatches caused by different verb tenses.
Validation for Social Preference: We find that the options within this dimension may exhibit similarities, making it difficult to distinguish correct option from the negative ones. To reduce difficulty, we manually examine the similarity between options. For options with excessively high similarity, we increase the differentiation between negative options and the correct answer. For instance, if the correct option has a positive preference, we select negative preference content with significantly different characteristics as negative options.
Understanding social contexts is important for role-playing agent. To delve into the verification of the agent’s social understanding within current social contexts, we prompt GPT-4 to justify its social choices in the third step of the SocialBench construction pipeline, thereby validating its understanding of the current social context. The reasoning process behind these social choices can then be manually verified, we leave this to our future work.

heading: B Human Annotation

heading: B.1 Annotator Recruiting
text: For annotators recruiting, we recruit annotators from crowdsourcing company, and the annotation
wages are evaluated and confirmed by the crowdsourcing company. Regular annotators consist of university graduates or higher, whereas senior annotators specialize in role-playing tasks with extensive experience. Our data verification process includes pre-verification (profile verification, dialogue verification, question verification) and postverification (verification for self-awareness, emotional perception, conversation memory, and social preference). We show the number of annotators at each verification process in Table 3.

heading: B.2 Annotation Instruction
text: The annotation instructions are designed to initially validate data samples, then annotate based on rules defined for each dimension, and provide examples for annotators’ learning (examples omitted for brevity). We provide instructions for question annotation on role style dimension (Figure 14), role knowledge dimension (Figure 15), emotional perception dimension (Figure 16), conversation memory dimension (Figure 17), and social preference dimension (Figure 18).

heading: B.3 Annotation Statistic
text: Before annotation, the predefined size is initially set for each dimension in SocialBench. We show the statistic of the number of valid samples, modified samples, and discarded samples in the first round of data annotation in Table 4.
To understand which social metrics are difficult for humans to agree on, we investigate the statistic of inter-annotator agreement on social preference level. We firstly collected 1,420 samples. After manual annotation, 484 were considered valid, 695 require modification, and 241 were discarded. We show the proportions of three dimensions during the filtering process in Table 5. We find that the majority of the discarded samples, over 60%, belonged to the neutral social preference (Neu.) category. We find that most tested models struggle with neutral and negative social preferences, as indicated in Table 2, while humans struggle most in the neutral social preference. Furthermore, as illustrated in Figure 4, the performance of most agents exhibits a significant decline with the increasing number of group members, whereas this trend is not evident among human annotators. This indicates that there is still a gap between agents and humans in their ability to handle complex group dynamics (Amir et al., 2022).

heading: C Experiment

heading: C.1 Evaluation Metrics
text: Most of the previous methods (Wang et al., 2023c; Shao et al., 2023) for role-playing applications rely on LLMs for evaluation, which may suffer from questionable accuracy and costly API usage. We follow the popular benchmark MMLU (Hendrycks
et al., 2020) and C-Eval (Huang et al., 2023), and prompt for automatic and fast evaluation free from LLMs. SocialBench utilizes fully automatic evaluation metrics, employing both multiple-choice and open-domain generation questions.
For multiple-answer questions, we calculate the accuracy (Accmultiple) using the following formula:
Accmultiple = N∑
i
Scorei MaxScorei , (4)
where N is the total number of multiple-answer questions. Scorei is the score obtained for the ith question, considering both correct and partially correct options chosen. MaxScorei is the maximum achievable score for the ith question. For example, if the answer to question i is A, B, then MaxScorei is 2. If only A is selected, then Scorei is 1; if the model selects A, C, and since C is not among A and B, even if A is correct, Scorei remains 0.

heading: C.2 Decoding Strategy
text: In all our experiments, we utilize the default parameter settings and decoding strategies provided by each closed-source model’s API. For open-source models, we employ top-p sampling, with a value of p set to 0.95 and a temperature of 1.0.

heading: C.3 Conversation Memory for Role-Playing
text: Conversation memory capability is crucial for roleplaying agents. We investigate the memory capacity of role-playing agents across different conversation lengths, measured by the number of utterances in the dialogue. We analyze the distribution of utterance counts in the conversation memory dimension of SocialBench. As illustrated in Figure 19, there is a declining trend in memory capability for some models, such as GPT-3.5-Turbo and CharGLM-3, as conversation length increases. When the number of utterances in the dialogue exceeds 80 rounds, most role-playing agents exhibit a noticeable performance decline. This finding showcases the limitations of current role-playing agents in handling
extremely long-term memory and highlights potential areas for improvement.

heading: D Personality Traits
text: We follow the definition of personality traits in Gunkel (1998) to construct profiles, ensuring diversity and comprehensiveness in SocialBench. From the collection of 638 personality descriptors created by Gunkel (1998), we selected a subset of easily understandable terms for construction. These selected terms can be categorized into positive, neutral, and negative traits, as illustrated in Table 6.

heading: E Related Work

heading: E.1 Role-Playing Agents
text: Leveraging the powerful capabilities of opensource foundational models, numerous efforts have emerged to develop models specifically tailored for role-playing tasks. These approaches can be categorized based on training paradigms: 1) Supervised fine-tuning (SFT). Li et al. (2023); Wang et al. (2023c); Tu et al. (2023) involved constructing specialized persona training corpus while performing fine-tuning on it to enable the agents to acquire capabilities of role-playing. 2) Integration of offline reinforcement learning. Shea and Yu (2023) combined role-playing model training with importance sampling strategies. 3) Incorporation of retrievalenhanced methods. Salemi et al. (2023) combined role-playing model training with retrieved information to enhance the capabilities of agents in roleplaying. (Shao et al., 2023) introduced a experience upload method, to test the model’s effectiveness on memorizing the character knowledge, values and personality.

heading: E.2 Role-Playing Benchmarks
text: With the development of role-playing agents, there has been an increase in evaluation datasets. Current evaluation datasets mostly focus on the alignment of role-playing agents with regards to role style and role knowledge. In terms of role style, Tu et al. (2024) and Wang et al. (2023c) investigate whether models can generate responses consistent with the style of the given role. Regarding role knowledge, RoleEval (Shen et al., 2023) particularly focuses on the role knowledge of role-playing models, including the characters’ experiences and social relationships. CharacterEval (Tu et al., 2024) and Wang et al. (2023c) also address aspects of role knowledge, such as role knowledge illusions. Additionally, Wang et al. (2023b) and Tu et al. (2024) introduce psychological theories like the Big Five and MBTI to evaluate role-playing agents. Most relevant to our work, Zhou et al. (2023b) proposes an open-ended environment and a benchmark Sotopia-Eval to simulate complex social interactions between agents and evaluate their social intelligence, but it is limited to the individual level. While previous work mainly focuses on testing the abilities of agents on imitating the character’s rolespecific knowledge or speaking style, SocialBench introduces the first-ever evaluation benchmark for the sociality of role-playing agents encompassing both individual and group level. We compare SocialBench with Sotopia-Eval (Zhou et al., 2023b), RoleEval (Shen et al., 2023), and CharacterEval Tu et al. (2024), as shown in Table 7.

heading: E.3 Agent Society
text: Previous benchmarks have primarily focused on single-agent scenarios, leaving the more complex multi-agent scenarios underexplored. Similar to humans, agents are capable of engaging in intricate social interactions, resulting in the formation of an agent society (da Rocha Costa, 2019). Recently, LLM-based agents demonstrate complex social behaviors, where cooperation and competition coexist (Xu et al., 2023). These sophisticated behaviors intertwine to shape social interactions (Gao et al., 2023). Agents within certain social scenarios may exhibit certain social preferences, where social preferences are the preferences of individuals regarding the payoffs or well-being of others (Charness and Rabin, 2002), and individuals behave prosocially on the basis of their social preferences (Murphy et al., 2014). SocialBench
follows the framework defined by Nigel Gilbert and Troitzsch (1997); Leng et al. (2023), where behaviors in agent societies are divided into individual and group-level activities, to study the social intelligence of role-playing agents within social interactions.

heading: F Examples from SocialBench
text: We showcase examples from SocialBench in Figures 20, 21, 22, and 23. A typical example consists of a character’s profile, conversation history, instruction, and question. There may be differences in format across certain dimensions. For example, in the emotional perception dimension, there is no character profile provided. In the conversation memory dimension, answers to each question are in the form of keywords rather than multiple-choice options. The conversation is stored in the format of a list combined with dictionaries. Each utterance is represented as a dictionary, where the keys are the names of the characters and the values are the content spoken by each character.

heading: G Data Utilization and Terms of Use
text: We utilized the open-source datasets (Chen et al., 2022; Hsu et al., 2018; Garbowicz, 2021; Gong et al., 2020), with their terms of use specifying research purposes only. Similarly, we employed the weights of open-source models and the APIs of closed-source models, strictly adhering to their respective usage agreements for research purposes. Regarding our dataset, it is also restricted to research purposes. We conducted thorough manual checks to ensure the absence of security and offen-
sive issues, particularly sensitive personal information such as phone numbers and home addresses.

heading: Future Work
text: The reasoning process behind these social choices can then be manually verified, we leave this to our future work