CoSER: Coordinating LLM-Based Persona Simulation of Established Roles
Authors: Xintao Wang
Heng Wang
Yifei Zhang
Xinfeng Yuan
Rui Xu
Jen-tse Huang
Siyu Yuan
Haoran Guo
Jiangjie Chen
Shuchang Zhou
Wei Wang
Yanghua Xiao
Venue: ICML 2025 poster (2025)
Keywords: Role-Playing Language Models
LLM Persona
Dataset
Evaluation
Primary area: applications->language_speech_and_dialog
TL;DR: This paper studies Role-Playing Language Agents (RPLAs) for established characters, and presents CoSER, a collection of authentic character datasets for RPLAs, along with open state-of-the-art models and evaluation protocols using such data.

Abstract: Role-playing language agents (RPLAs) have emerged as promising applications of large language models (LLMs). However, simulating established characters presents a challenging task for RPLAs, due to the lack of authentic character datasets and nuanced evaluation methods using such data. In this paper, we present CoSER, a collection of a high-quality dataset, open models, and an evaluation protocol towards effective RPLAs of established characters. The CoSER dataset covers 17,966 characters from 771 renowned books. It provides authentic dialogues with real-world intricacies, as well as diverse data types such as character experiences and internal thoughts. Drawing from acting methodology, we introduce given-circumstance acting for training and evaluating role-playing LLMs, where LLMs sequentially portray multiple characters in book scenes. Using our dataset, we develop CoSER 8B and CoSER 70B, i.e., advanced open role-playing LLMs built on LLaMA-3.1 models. Extensive experiments demonstrate the value of the CoSER dataset for RPLA training, evaluation and retrieval. Moreover, CoSER 70B exhibits state-of-the-art performance surpassing or matching GPT-4o on our evaluation and three existing benchmarks, i.e., achieving 75.80% and 93.47% accuracy on the InCharacter and LifeChoice benchmarks respectively. Our code, dataset and models are available at: https://github.com/Neph0s/CoSER.

CoSER: Coordinating LLM-Based Persona Simulation of Established Roles
Xintao Wang' Heng Wang² Yifei Zhang'
Xinfeng Yuan
Rui Xu'
Jen-tse Huang Siyu Yuan Yanghua Xiao
Haoran Guo
Jiangjie Chen
Shuchang Zhou?
Wei Wang'
Abstract
Conversation
Setting
Scenario: Horry, Ron, and Hermione decided to retrieve the Philosapher's Stone Stone on
Role-playing language agents (RPLAs) have emerged promising applications of large lan- guage models (LLMs). However, simulating tablished characters presents challenging task for RPLAs, due to the lack of authentic char- acter datasets and nuanced evaluation methods using such data. In this paper, we present COSER, collection of a high-quality dataset, open models, and evaluation protocol towards effective RPLAs of established characters. The COSER dataset covers 17,966 characters from 771 renowned books. It provides authentic dia- logues with real-world intricacies, as well as di- verse data types such as conversation setups, char- acter experiences and internal thoughts. Drawing from acting methodology, we introduce given- circumstance acting for training and evaluating role-playing LLMs, where LLMs sequentially por- tray multiple characters in book scenes. Using our dataset, we develop COSER 8B and COSER 70B, i.e., advanced open role-playing LLMS built on LLaMA-3.1 models. Extensive experiments demonstrate the value of the CoSER dataset for RPLA training, evaluation and retrieval. More- over, COSER 70B exhibits state-of-the-art per- formance surpassing matching GPT-4o on our evaluation and three existing benchmarks, i.e.. achieving 75.80% and 93.47% accuracy on the InCharacter and LifeChoice benchmarks respec- tively. Our code, dataset and models are available https:/githab..om//eeehdCCCCCCS
their own.
Characters' Motivations
We have stop from
obtaining the Philosopher's Stone
Dialogue
[I'm nervous, breath) Alright, this is it. Are you both sure you want come? could be dangerous.
this.] deep
Harry
[I'm scared, but can't let Harry face this alone.]
(Nods firmly) course we're coming, mate. You can't by yourself...
Ron
[We need plan.] Wait! (Pulls out her wond) Harry
Hermione
door opened, the headed dog Fluffy was already asleep, with horp self-playing beside.
Environment
Plot
Conversations
list of structured conversations.
Summary
Harry, Ron, and Hermione through the trapdoor to protect the Sorcerer's Stone. face face series of magical challenges, including Devil's Snore, Giant Wizord's Chess
Characters Experiences
Leads the mission to protect the Stone, showing brovery throughout
Harry Potter and the Philosopher's Stone
Extract plot and conversation data
Book
LLM-based Pipeline
Figure I. An example from COSER dataset, which provides com- prehensive data types such conversation dialogues settings, plot summaries, character motivations and inner thoughts, authen- tically sourced from renowned books.
1. Introduction
Shhnnhaai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan Uni- versity "StepFun 'School of Data Science, Fudan Uni- versity "Johns Hopkins University. Correspondence to: Yanghua Xiao <shawyh@ Studan.edu.cm.. First author's email <xtwang21 ww..udan.cdu.cco
Recent advances in large language models (LLMs) have facilitated the emergence of anthropomorphic cognition in AI (Kosinski, 2023; Shanahan et al., 2023). Role-playing language agents (RPLAs), i.e., LLMs that simulate specific personas based on relevant data, have hence been popu- lar (Park et al., 2023). RPLAs have been adopted to simulate personas of various types, including demographics, charac- ters, or daily individuals (Chen al., 2024a). They have
Proceedings of the = International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

COSER: Coordinating J.MM-Basce Persona Simulation of Established Roles
inspired extensive applications including character chatbots, agents in video games, and digital clones for humans. This paper studies RPLAs for established characters, which represent a crucial yet challenging task beyond the naive portrayal of individual traits or stereotypes. Specifically, RPLAs should faithfully align with their characters' com- plex backgrounds and capture their nuanced personalities.
Mc C M. As result, we develop COSER 8B and 70B, built LLaMA-3.1 models (Dubey et al., 2024), which demonstrate true-to-life character portrayal and state-of-the- art performance on multiple RPLA benchmarks. For evalua- tion, GCA involves two steps: multi-agent simulation and penalty-based LLM judging. Given a test conversation M, we: 1) create multi-agent system to simulate a conversa- tion M, where the actor LLM portrays each character e in the same setting S: 2) assess M using penalty- based L.M critics, leveraging detailed rubrics and the original conver- sation M. GCA evaluation offers three advantages: First, it comprchensively reflects actor LLMs' abilities via multi- agent simulation; Second, it is based on authentic scenes and groundtruth dialogues. Third, it provides expert-curated rubrics to guide LLM critics.
Towards effective RPLAs, two major challenges persist in: 1) Data: High-quality datasets are lacking. Existing datasets are limited to dialogues between two characters, and lack necessary dialogue contexts and knowledge in other forms. Moreover, many datasets are synthesized by LLMs, com- promising authenticity and fidelity to the origins (Wang et al., 2024a; Lu et al., 2024); 2) Evaluation: Current meth- ods fall short in assessing complex character portrayals of LLMs. They typically focus on single-tur interactions with pre-defined question sets, and rely on either LLM-based judges or multi-choice questions. The former lack nuanced discrimination and suffer from bias issues (Li et al., 2024), while the latter only assess specialized aspects (Xu et al., 2024). Overall, there is a lack of authentic character data and appropriate evaluation methods based on such data.
Our contributions are summarized as follows:
1. We introduce the COSER dataset and models for RPLA research and applications. Our dataset com- prises 29,798 authentic conversations and comprehen- sive types of data from 771 renowned books. Leverag- ing this dataset, we develop COSER 8B and COSER 70B, which are state-of-the-art models for RPLAs.
In this paper, we introduce CoSER, collection of authentic character data, along with open state-of-the -frte--rrr models and evaluation protocol based on such data, for Coordinating LLM-Based Persona Simulation of Established Roles. The COSER dataset is sourced from narratives and dialogues in 771 renowned books, processed via our LLM- -bsed pipeline. COSER differs from existing datasets in two fundamental ways: 1) COSER extracts authentic, multi-character dia- logues from acclaimed literary works, in contrast to LLM- synthesized question- question-answer pairs in previous work. Hence, our dataset maintains high source fidelity while exhibiting greater quality and complexity. 2) COSER incorporates comprehensive types of data, as shown in Fig. 1: i) Besides character profiles and dialogues, CoSER encompasses plot summaries, character experiences, and conversation back- grounds, supporting various purposes including prompting, retrieval, model training and evaluation. ii) Conversations in COSER capture characters' actions and internal thoughts beyond surface- speech, enabling RPLAs to simulate sophisticated cognitive and behavioral processes of humans, such as "[I'm nervous, but we have to do this] (Takes a deep breath) Alright, we..." We provide a clear comparison between CoSER and existing datasets in Table 7.
2.
We propose given- ewee--crcuusstnce acting for training and evaluating role-playing I.MMs, drawing from estab- lished acting theory. Our evaluation comprehensively tests actor LLMS via multi-character simulation, while providing original dialogues and detailed rubrics to enhance LLM-based assessment.
Results of extensive experiments demonstrate the sig- nificant value of our dataset for the training, retrieval and evaluation of RPLAs Notably, the COSER mod- els achieve offte-of-the-ar performance across four benchmarks and human evaluation for RPLAs
2. Related Work
Using LLMs to simulate human personas represents a pio- neering research direction. Early studies develop prototypes of RPLAs for fictional characters (Shao et al., 2023) and multi-agent systems that simulate human society (Park et al., 2023), while exploring the nature (Shanahan et al., 2023) and potential limitations (Liu et al., 2024a; Deshpande et al., 2023) of LLM role-play. Chen et al. (2024a) present a com- prehensive survey of relevant research. Our work focuses on the role-play of established characters, with comparison to PersonaHub (Chan al., 2024) in $A.6.
We introduce given-circumstance acting (GCA) for train- ing and evaluating role-playing LLMs, leveraging COSER dataset. Given a conversation with messages M, characters and setting S, GCA requires an actor L.M to sequentially portray each character
Specifically, an RPLA leverages LLMs to create a simulated persona Te that simulates a real character c based on its persona data D... Effective RPLAs require both compre- hensive, high-quality data De and advanced role- pole-playing L.MSs. Among various persona types, we focus on RPLAS
C recreate the conversation, as illustrated in Fig. 2. During training, we train LLMs to portray each character c, on their authentic utterances
2

---

CoSER: Coordinating LLM-BBascd Persona Simulation of Established Roles
Table Overview of COSER and existing RPLA datasets. For characters, Num. count characters with profiles, and Expr. denotes structured character experiences. For conversations, Auth. indicates authentic dialogues or behaviors from the books, and Multi-Chan denotes involving more than 2 characters. Num. (number), Conv. (conversation), and Summ. (summary) are abbreviations.
Book
Character
Conversation
Message
Plot
Dataset
Num.
Num.
Profile
Expr.
#Conv.
#Turns
Setting Auth. Multi-Chara.
Speech Thought Action Summ,
Charater-LLM
9
14,300
13.2
ChatHaruhi
32
54,726
>2
RoleLLM
100
140,726
2
HPD
7
113
1,191
13.2
4
LifeChoice
388
1,462
V
1,462
2
CruSS-MR
126
126
445
2
/
CharacterGLM
250
1,034
15,8
EharacterEval
77
1,785
9.3
/
DITTO
4,002
7,186
5.1
MMRole
85
14,346
4.2
CharacterBencch
3,956
13,162
11.3
COSER
771
17,966
V
29,798
13.2
for established characters, which should faithfully align with their characters' complex backgrounds and nuanced personalities.
ases, e.g., length and position bias (Li et al., 2024), and may lack the necessary knowledge for character-specific evalua- tion. Other benchmarks evaluate role-playing LLMs through multiple-choice questions, assessing specific aspects such as knowledge (Shen al., 2023), decision making (Xu et al.. 2024; Yuan et al., 2024), personality fidelity (Wang et al., 2024b) and point-in-time portray (Ahn et al., 2024).
Datasets for RPLAs Persona data D. describe the real persona through various representations, including pro- files (Yuan et al., 2024), dialogues (Wang et al., 2024a). periences (Li et al., 2023) and multimodal information (Dai et 2024), etc. As shown in Table 7, existing datasets have several limitations. 1) Many are synthesized via LLMs' responses to general instruction sets (Wang et al., 2024a; Chan al., 2024) or character-specific questions (Shao et al., 2023), such as RoleBench (Wang et al., 2024a). However, I.LM-synthesized data compromise authenticity and fidelity to original sources. 2) Human-annotate datasets such as CharacterEval (Tu et al., 2024) and CharacterDial (Zhou et al., 2023) offer improved quality, but are expensive and difficult to scale. 3) Several efforts extract authentic dia- logues from fictional works, such as ChatHaruhi (Li et al., 2023) and HPD (Chen al., 2023). However, they rely on human efforts for individual sources and are hence hard to scale as well. 4) Furthermore, existing datasets offer limited representations and forms, i.e., mainly consisting of two- character or user-character question-answer annser pairs. These datasets support various purposes, including prompting, training, retrieval augmentation, and evaluation RPLAs
3. CoSER Dataset
In this section, we introduce the COSER dataset, which cov- ers authentic data of 17,966 characters from 771 renowned books. COSER features its authentic, non-synthesized dia- logues with real-world intricacies, and comprehensive data representations supporting various usages. In Table 1, we provide a comprehensive comparison with existing datasets. We illustrate our dataset's design principles in $3.1, curation pipeline in $3.2, and statistical analysis in $A.1.
3.1. Design Principles
As shown in Table 1, COSER differs from previous RPLA datasets mainly in its: 1) rich data types. 2) internal thoughts and physical actions in messages, 3) environment as role.
Rich Types of Data The persona data D. can represent character from fictional works in diverse forms, e.g., nar- ratives, profiles, dialogues, experiences, etc. Previous work focuses primarily on profiles and dialogues, which represent limited knowledge. Hence, we propose a more comprehen- sive set data types that are: 1) Comprehensive: covering extensive knowledge about characters and plots from the books; 2) Orthogonal: carrying distinct, complementary information with little redundancy; 3) Contextual- pro- viding sufficient context to enable n. to faithfully reproduce c's behaviors and responses in given scenarios.
Evaluation for RPLAs Existing evaluation methods are based on either LLM judges multi-choice questions (Chen et al., 2024a). LLM-jddged methods typically elicit LLMs' role-playing performance via predefined questions, and score the performance using LLM judges or reward mod- els (Chen et al., 2024a). They assess various dimensions, including Character-independenn aspects such as anthropo- morphism (Tu et al., 2024) and attractiveness (Zhou al., 2023). as well as character-specific traits such as language style, knowledge, and personality (Wang et al., 2024a; Shao et al., 2023). However, LLM judges suffer from inherent bi-
Specifically, we organizes knowledge from books hierarchi-
3

---

CoSER: Coordinating JLM-Basce Persona Simulation of Established Roles
Dataset
Training
Curation Pipeline
Select a target role
to play
Chusk
Extract
Compose role-playing instruction
Profiles
You are Ron Weasley, a wizard from You are about to enter the trapdoor protect ... You think: dangerous, but can't let Harry alone'.
Books
Plot and Conversation Data
Character Data
Scenario
Goodreads
Chunks
Dataset Overview
Instruction
Here are the information other characters.. Derive a training sample
Motivation
Rich Data Types
Associated
Original Dialogue
Plot
Conversation
Character
Content
Dialogue
Profiles
Summary
Scenario
Expr. in Plots Conversations
Supervised Fine-tuning
Characters and Expe-
Characters'
Mot ivvatiionn....
Evaluation
Thought& & Action Dialogues
Role-play each character
[We need plan.] Wait! (Pulls her wand) ...
Multi-agent Simulation
Harry, you go first, you have Hagrid's flute
Profiles
next speaker
Example Conversation
RPLA
Book: Harry Potter and Philosopher's Stone
Scenario
If Dumbledore here...
NSP
Chapter: Through the Trapdoor
Motivation
Turn's
Scenario: Harry, Ron, and Hermione decided to retrieve the Philosopher's Stone on their own.
Penalty-based LLM Judging
score the simulated by identifyine according to detailed rubrics.
dialogue four dimansions,
Characters' Motivations: (Harry .... Ron : Hermione ..
flaws
Original Dialogue:
Ne can train and evaluate
Original Dialogue
- Anthropamarphiim Character Fidelity
zole-playing LLMe vis
givai-circusstane acting.
Simulated Dialogue
LLM Critic
Storyline Quality
Sttoyyline Consistency
Figure Overview of COSER's dataset, training and evaluation. Left: The COSER dataset is sourced from renowned books and processed via LLM- pipeline. It contains rich data types on plots, conversations and characters. Right: We apply given-circumstance acting to train and evaluate role-playing LLMs using these conversations. For training, each sample trains the LLM to portray a specific character in conversation, using their original dialogue. For evaluation, we build multi-agent system for conversation simulation given the same scenario, and assess the simulated dialogue via penalty-based LLM critics.
cally via three interconnected elements: plots, conversations and characters. Each plot comprises its raw text, summary. conversations in this plot, and key characters' current states and experiences in this plot. conversation contains not only the dialogue transcripts, but also rich contextual set- tings including scenario descriptions and characters' moti- vations. Characters are associated with their conversations and plots, based on which we craft their profiles.
trigger downstream events in multi-agent systems.
Thought represents internal thinking processes, which enable RPLAs to simulate sophisticated human cogni- tion. Thoughts should be invisible to others, forming information asymmetry (Zhou et al., 2024).
Environment as a Role In RPLA applications like Al TRPG (Liang et al., 2023), LLMS often serve world simulators that respond to players' actions. To promote this ability, we consider environment as a special role e, which provide environmental responses such as physical changes and reactions from unspecified characters or crowds.
Thoughts and Actions in Messages Previous RPLA stud- ies typically restrict RPLAs' output space to verbal speech alone, limiting their ability to fully represent human inter- actions. In this paper, extend the message space of RPLAs and character datasets into three distinct dimen- sions: speech (L), action (AA, and thought (T), significantly enriching the expressiveness. For instance, an RPLA can convey silence by generating only thoughts and actions with- out verbal speech. The three dimensions are distinguished by markup symbols and function mechanisms:
3.2. Dataset Curation
We curate the COSER dataset through a systematic LLM- based pipeline that transforms book content into high- quality data for RPLAs 2, The details are follows.
Source Selection Our dataset is sourced from most ac- claimed literary works to ensure data quality and character depth. We identify the top 1,000 books on Goodreads's s Best
Speech is for verbal communications of characters.
Action captures physical behaviors, body language, facial expressions, etc. Similar to tool use in agents (Weng, 2023), actions can be programmed to
'Tabletop Role-Playing Games II this paper, we employs Claude-3.5 3.5-ee-e-- (20240620).

---

COSER: Coordinating LLM-BBacc Persona Simulation of Established Roles
Books Ever list 2 and obtain the content for 771 books. As shown in Table 8, these books offer characters and narratives with literary significance and widespread recognition across diverse genres, time periods, and cultural backgrounds.
are trained and judged through performance within speci- fied conditions including environmental context, historical events and personal conditions (Stanislavski, 2008).
We propose to adapt this approach to a framework that trains and evaluates LLMs' role-playing skills, leveraging the com- prehensive data in COSER. In this framework, given a con- versation with dialogue messages M, involved characters C, and contextual setting S, an actor LLM sequentially plays the role of each character e C to simulate the conversation.
Chunking We segment book contents into chunks to fit in LLMs' context window. We employs both static, chapter- based strategy and dynamic. plot-based strategy. Initially, we use regular expressions identify chapter titles as nat- ural chunk boundaries. Then, merge adjacent small chunks and split large chunks to ensure moderate chunk sizes. However, static chunking neglects the storyline and truncates important plots or conversations. To address this, we implement dynamic plot-based chunking, i.e,, during data extraction, we also prompt L.LMs to identify truncated plots or trailing content in the current chunk, and concate- nate them with the subsequent chunk to ensure plot integrity.
4.2. GCA Training and COSER Models
We fine-tune LLMs' role-playing abilities through GCA. Each training sample is derived from conversation and one of its character in CoSER dataset, and LLMs are trained on c's utterances Mc. Specifically, we first compose role- playing instruction ie comprising the scenario description, the character's profile Pe and motivation, and profiles of other involved characters, which provide comprehensive context for role-playing. The original dialogue messages are denoted as M = [m1.... mrl, where T is the number of turns. Then, the training sample (E., m.,.... mT] mT] is a concatenation of the instruction and messages. where the character's messages M. C M are treated as outputs for optimization, and the other parts serve as inputs.
Data Extraction We employ LLMs to extract plot and conversation data from book chunks, including (I) con- tents, summaries and character experiences of plots, and (2) dialogues and background settings of conversations. The extracted data representations are illustrated in Fig. and introduced in $3.1. In the messages, speeches are always extracted from the original dialogues, while actions and thoughts can either be extracted or inferred by LLMs based on the context. For evaluation purposes, we hold out data from the final 10% plots in each book.
We train COSER 8B and COSER 70B based on LLaMA 3.1 Instruct models (Dubey et al., 2024), using 90% books in our dataset. To effectively support diverse use cases of RPLAS, our training samples cover extensive settings: i) The COSER dataset contains massive characters and conversa- tion settings from extensive books. We train models on all characters in each conversation, ranging from major charac- ters with detailed profiles to minor roles driven only by the context; ii) To simulate real use cases, we incorporate role- playing instructions in diverse formats through instruction templates of varying formats. Besides, we consider different combinations of available data by including or excluding: profiles of other characters, plot summaries, and characters' motivations; iii) We train models both with and without characters' internal thoughts in the extracted dialogues.
Organizing Character Data Based on the extracted data, we form the knowledge bases for characters in three steps. First, we unify character references by establishing name mappings between aliases and canonical names using LLMs, e.g., mapping Lord Snow to an unified identifier Jon Snow. Second, we aggregate relevant plots and conversations for each character. Finally, we leverage LLMs to generate char- acter profiles based on their extracted data, describing them from multiple perspectives including background, experi- ences, physical characteristics, personality traits, core moti- vations, relationships, character arcs, etc.
For technical details, including our prompts, engineering implementation, and handling mechanisms for exception caused by LLMs, please refer to SA.
We extend COSER's training beyond character role-playing to develop complementary capabilities in environment mod- eling and next speaker prediction (NSP), which facilitates RPLA applications. To maintain models' general abilities, we augment our training data with the Tulu-3 dataset (Lam- bert et al., 2024). Please refer to $B for more details.
4. Training and Evaluation via GCA
In this section, we introduce given-cireumstance acting (GCA) for training and evaluating ..LMs' role-playing abili- ties using the COSER dataset, as shown in Fig. 2.
4.3. GCA Evaluation
4.1. CivvenCCrrummstncce Acting
Evaluating role-playing LLMS remains a significant chal- lenge. primarily in two aspects: i) providing appropriate scenarios to elicit role-playing performance, and ii) prop- erly assess the performance. Towards these challenges, we
In Konstantin Stanislavski's acting methodologies, given- circumstance acting is a fundamental approach where actors
http::/wwww.ggoodrrassso///////s Bttph///wwwwwwtwooeeeeeeeee Best_Bookss_Evee
5

---

COSER: Coordinating LLM--Bace Persona Simulation of Established Roles
propose GCA evaluation for actor LLMs' role-playing abil- ities, comprising two stages: multi-agent simulation and penalty-based LLM judging, illustrated in Fig. 2.
Individual Agent
Referenced
Character
Anthropo-
Inherent
Fidelity
morphism
Multi-agent Simulation For a test conversation M, we build multi-agent system to simulate a conversation M, in the same setting as M. We create an RPLA Te for each char- acter E C. using the actor LLM. We provide RPLAs with comprehensive data as described in $4.2: scenario descrip- tions and involved character profiles offer crucial context, and character motivations promote RPLA proactiveness and a natural conversation flow. Following $3.1, RPLAs are instructed to output in the speech-action-thougee format. Each RPLA's motivations and inner thoughts are inaccessi- ble to other RPLAs. We adopt an NSP model to select the speaker of each turn from C'U {e), and another LLM as the environment model Ne to provide environmental feedback. The simulation ends upon an <END> signal from NSP, or reaching the maximum of 20 turns. In this way, we obtain multi-turn, multi-character simulation that comprehensively reflects the actor LLMs' role-playine abilities.
Storyline
Storyline
Consistency
Quality
Overall Simulation
Figure 3. We divide the RPLA evaluation into four quadrants (dimensions). The X-axis represents the evaluation perspective: Referenced (comparing with data the source book) versus In- herent (assessing standalone quality). The Y-axis indicates the evaluation scope: Individual Agent versus Overall Simulation.
Anthropomorphism: Evaluates whether RPLAS be- have in a human-like manner, with rubrics covering self-identity, emotional depth, persona coherence, and social interaction.
2. Character Fidelity: Assesses whether RPLAs faith- fully portray their characters, with rubrics examining language style, knowledge and background, personal- ity and behavior, and social relationships.
In addition, we introduce a continue-from parameter k; where the simulation starts from the first & original mes- sages in M. Setting k > 0 controls the story direction and language style, similar to in-context learning. Hence, it en- ables more controlled evaluation and reduces the influence of different language styles of LLMs.
3. Storyline Quality: Evaluates whether the simulated conversation develops naturally, with rubrics focusing on narrative flow and logical consistency.
Penalty-basee LLM Judging In this stage, we assess the simulated dialogue M via LLM critics. Different from previous LLM--as--juddge methods for RPLA evaluation, our LLM critics: i) apply penalty-based scoring by iden- tifying role-playing flaws following detailed rubrics, and ii) leverage the original conversation M as reference.
4. Storyline Consistency: Measures alignment between the simulated conversation and original dialogue
M, i.e., whether RPLAs' reactions (emotions, attitudes, behaviors) remain consistent with the original.
As longer simulations naturally make more flaws, we im- plement length correction to reduce bias in LLM judging following Dubois et al. (2024). Specifically, we obtain the length corrected score as s = 100 - 5 Σ Uy + A[M), where À is set to 1.5 based on statistical analysis in SC.3, For detailed prompts and rubrics, please refer $G.
Specifically, we employ LLM critics to identify flaw in- stances in M of specific rubrics, such as "deviate from the original conversation" or "lack initiative and goals", instead of directly outputting a score in previous work (Wang et al., 2024a; Tu et al., 2024). Each flaw f is assigned severity vy from (minor) to 5 (severe). The initial score for each dimension is calculated as 3 = 100 5 ΣSEF Vy.
5. Experiments
We consider four evaluation dimensions, as shown in Fig. The rubrics are derived from human-annotated issues in extensive human-RPLA conversations from real users and multi-agen aulti-agent simulations, as well dimensions from pre- vious work (Shanahan et al., 2023; Shao et al., 2023; Chen et al., 2024a). For more informed evaluation, LLM critics are provided with additional materials, i.e., the original con- versation M and plot summary, besides data available to actor L.Mss. Each dimension is assessed in independent LLM requests. The evaluation dimensions and their rubrics summarized as follows:
5.1. Settings
Evaluation Protocol We evaluate LLMs' role-playing abilities through GCA on COSER Test, a test set of held-out conversations from the final 10% of each book. COSER Test contains 200 conversations, with 100 from books used in COSER training and 100 otherwise. We employ GPT-4o as LLM critics and environment model, and CoSER 70B for NSP. We exclude characters' inner thoughts for LLM critics, and set the continue-from parameter k = 0. The details are in SC.
Metrics We report LLM-judged scores for each dimen- sion, and their average as the overall score. For analysis, we
*This paper uses GPT-4o as the critic LLM by default.
6

---

COSER: Coordinating J.MM-Basce Persona Simulation of Established Roles
also evaluate two traditional metrics based on N-gram, i.e., BLEU (Papineni et al., 2002) and ROUGE-L((in & Och, 2004) compared against original dialogues. Additionally, we report win rates versus GPT-3.5 and GPT-4o in $D.
over their LLaMA 3.1 baselines. In contrast, Higgs-LLaMA- 70B, fine-tuned on synthesized dialogues, performs below LLaMA-3.1 70B, These results highlight the importance of
high-quality, authentic role-playing data for LLM training.
Models Our experiments cover numerous LLMs: i) Close models, including Minimax Abab7-preview 5. Doubao- pro 9 Stee--1-Flash and Step-2 7, GPT-3.5 (OpenAI, 2022). 9 Claude- 19; fi) Open models, in- cluding COSER 8B and 70B, -LaMA-3.1-Instrucc 8B and 70B (Dubey et al., 2024). Qwen-2-Instrus 7B and 72B (Yang et al., 2024), Vicuna-13B (Zheng et al., 2023), Mixtral-8x7B (Jiang et al., 2024), DeepSeek-V3 (Liu et al., 2024b) and Higgs-Llama-3-709 11_
Conversation Continuation Enables More Controlled Evaluation Table 14 shows experiment results when multi- agent systems start from the first k 3 original mes- sages, In this setting, model obtain higher scores compared to simulations from scratch (k = 0), with reduced perfor- mance gaps between different models, especially for BLEU and ROUGE-L results. For example, the average score gap between Qwen-2 72B and 7B decreases from 1 11.5% (A = 0) to 8.8% (k = 3). This improvement occurs because the = 3 original messages guide the story direction and language style, particularly benefiting smaller models that typically struggle with complex role- role-playing instructions.
GPT-40 and GPT-4o Mini 8, Gemini-I.5-PP
3 3-Haiku and Claude-3.5-Soman
Other Benchmarks We also evaluate COSER models on existing RPLA benchmarks based on multi-choice ques- tions instead of L.LM judges, including InCharacter (Wang et al., 2024b) for personality tests, LifeChoice (Xu et al., 2024) for decision becision-making, and CroSS (Yuan et al., 2024) for motivation recognition. For InCharacter, we report its accuracy on the Big Five Inventory (BFT).
Results on Other Benchmarks We evaluate COSER and other models on existing benehmarks for RPLAs based on multi-choice questions. As shown in Table 3, COSER 70B achieves state-of-the-ar performance across these bench- marks. Notably, COSER 70B achieves 93.47% accuracy on LifeChoice, surpassing GPT-4o by 23%. These results ex- hibit CoSER models' strong capability in nuanced portrayal of characters' personalities and behaviors.
5.2. Main Results
Performance of Various LLMs on COSER Test We apply COSER Test to evaluate extensive LLMss The results shown in Table 2 are averaged across three runs, from which observe that: Z) CoSER 70B achieves oftte-of-the-art performance across both M.uged and N-gram-based metrics. For LLM-judged metrics, COSER 70B outper- forms all open models and shows competitive performance with GPT-4o. COSER 8B similarly outperforms models of comparable scale. For N-gram-basee based metrics, COSER models demonstrate substantial improvements over existing models, exceeding the second-best performance by 58% BLEU; 2) Among all models, GPT-4o, Gemini Pro, Claude- 3.5-Sonnet, Doubao-pro, 2tep- QweB-2-72B, and COSER 70B demonstrate superior performance, achieving average scores above 57%; 3) Table 13 presents LLM performance separately for test splits from books included or excluded for COSER training. The results show consistent trends across both splits, validating that COSER models maintain strong performance on out-of-domain characters.
5.3. Human Evaluation
We conduct human evaluation to further validate the perfor- mance of COSER models and the reliability of LLM critics in GCA evaluation.
Annotations We randomly select 60 samples from CoSER Test, and evaluate 7 representative LLMs (shown in Table 4) based on their simulation results. Three anno- tators independently score the LLMs on 20 samples each, using a 10 scoring scale, given the background context and original conversations. During this process, the anno- tators note that 1) GCA simulation deeply reflects LLMs' role-playing abilities, compared with previous single-turn benchmarks; 2) Manual evaluation is highly challenging and time-consuming. It requires careful learning of complex background and abundant dialogues, and takes 15 minutes to evaluate 7 models for one sample on average. We report models' average scores and win rate against other models.
Authentic Conversations from High-quality Novels Im- prove LLMs' Role-playing Ability According to Table 2, COSER models demonstrate significant improvements
Results and Analysis According to Table 4, we find that: 1) The results generally align with GCA evaluation, validat- ing the reliability of our LLM- evaluation protocol; 2) COSER 70B maintain superior performance, obtaining the highest score (6.783) and win rate (86.9%); 3) Human evaluators show less preference for GPT models compared to GPT-4o as LLM judges, which likely stems from self- preference bias in LLM judges (Wataoka et al., 2024).
$ hhtps//wwwwmmnimaxi.comm com/
" htwp::awwwddbbba.cooooooo Version:241215
7 httpp://wwww.stffuu..comm
"https///pemai..oo//fff www./innwwwhhlllogg-ooooooo Version:240806
9 "nttps:/ddeppmind http:///pwm.oogooo////c/o///////
10 "htps://ooson.alliigg-////
7

---

COSER: Coordinating ALMMBasccd Persona Simulation of Established Roles
Table Performance (%) various L. given-circumstance acting using CoSER Test. Bold underlined values indicate hest performance across all models and open- source models, respectively.
Based on LLM Judges
Based on N-gram
Model
Storyline
Anthro-
Character
Storyline Quality
Average Score
BLEU
ROUGE-L
Consistency
pomorphism
Fidelity
Close-source Models
Abab7-preview
56.81
1..2
43.83+2.71
74.83
54.92=0.57
4.96#0.07
11.50+0.06
Doubao-pro
60.95
49.72.1..3
47.02.11.10
79.28
59.24-0.30
6.38
12.95
Step-1-Flash
57.75
18.12...39
44.48
75.93
56.57
5.95
10.11
Step-2
61.43
49.06.1.69
17.73
77.96
58.94
5.75
12.50
GPT-3.5
57.22 10.13
43.3010.48
42.2911.47
73.91
54.1810.88
1.58
11.80
GPT-4o
61.59 10.66
48.93
48.95
80.3310.55
59.95 10.50
5.9010.16
12.11 10.13
GPT-4o Mini
60.0910.60
18.21
44.88
78.5510.1
57.93
3.90
10.81
Gemini Pro
59.1 10.82
52.41
47.83
17.59
0..25
5.39
11.65
Claude-3-Haie Haiku
1.0.2
44.6611.72
41.88
1..14
54.7 20.84
4.80
12.02
Claude-3.5. -.5-Sonnet
$7.4540.90
48.50
45.69
77.23
57.22
5.17
11.45
Open-source Models
Mistral-7B
59.90=1.33
40.00+0.74
44.75
11.93
51.64-0.56
2.71 10.10
9.28:10.1
Qwen-2-7B
51.96=0.67
35.48
31.51
63. 03.18
=5.53=0.69
4.21
10.71
LLAMA-3.1-8B
14.10+1.63
15.3
40.22.11.16
72.29
52.99-1.20
4.59
10.18:0009
CoSER-8B
58.61
17.23
46.90 46.90112.06
73.04 1.37
56.45_0.56
9.40
14.21
Vicuna-13B-B 13B-1.5
52.7511.64
39.12.1.211
38.04
60.43
47.58
1.67
5.59
Mixtral-8x7B
51.25
38.44.1.115
36.92.2.65
67.69
48.58
5.28
11.66.10.05
Qwen-2-72B
57.75+1.26
47.28+0.87
46.62+1.69
76.60=0.36
57.06=1.00
5.38+0.00
11.85+0.03
1-0B
57.46+1.05
45,95+1.30
43.72+1.17
70.84
55.49=0.33
4.82=0.06
10.98=0.06
Higgs-Llama-3-70B
57.10+1.12
43,82+2.18
42.41+1166
75.62=0.15
54.74=1.26
3.99
10.92+0.66
CoSER-70B
58.66+1.34
53.33
48.75+1.43
75.49=0.94
59.06=0.22
10.10=0.04
14.78=0009
DeepSeek-V3
56.40+0.95
47.87+1.10
44.02++0.18
16.66+1.20
56.24=0.46
4.54
11.02+0.15
Table 3. LLM performance (%) across three existing RPLA bench- marks. I.T. denotes inner thoughts. For InCharacter, report accuracy for individual (Dim) and full (Full) dimensions on BFL
Table 4. Human evaluation on LLMs' GCA simulation.
Model
Score
Win Rate (%)
GPT-3.5
3.117
10.6
Model
Incharacter Dim Full
Life Choice
CroSS
LLaMA-3.I-SB
3.600
19.4
MR
Abab7-preview
4.533
37.5
LLaMA-3.1-88 --BB
64.97
15.62
61.10
30.15
COSER-8B
4.567
38.6
CoSER-8B
75.80
21.88
69.54
44.94
GPT-4o
4.967
47.2
trained w/o I.T.
70.70
15.62
59.92
43.14
Slaude3.55Soome
6.200
73.9
LLaMA-3.1-70B
72.16
31.25
86.48
61.30
CoSER-70B 6.783
86.9
Higgs-Llama-3-777
74.52
28.12
74.03
60.12
CoSER-70B
75.80
34.38
93.47
64.49
trained w/o I.T.
73.12
32.14
93.18
63.14
Qwen-2-72B
74.52
31.25
81.14
62.57
We also compare with BLEU and ROUGE-L. The alignment is measured via model-wise preferences (SC.4).
GPT-3.5
71.20
21.88
78.07
30.09
GPT-4o
76.54
32.62
75.96
64.49
Claude-3, 3.5-Sonnet
72.61
21.88
86.07
30.59
The results in Table 5 reveal that: /) All components of GCA evaluation contribute to improved alignment; 2) Reasoning models excel as judges, with DeepSeek-RI achieving the highest alignment; 3) BLEU and ROUGE-L remain effec- tive indicators for role playpnaying evaluation.
Alignment We further study the alignment between LLM and human judges. We evaluate three judge models, GPT- 4o. DeepSeek-V3 and DeepSeek-RI (Guo et al., 2025). We consider standard GCA evaluation and ablation variants that remove: (i) original dialogues as reference, (ii) scoring rubrics, (iii) length correction, and (iv) dimension separation.
5.4. Ablation Studies
Inner Thoughts and Motivations Enhance RPLAs at Test Time
Table 6 compares LLMss overall scores on COSER
8

---

CoSER: Coordinating L.M-Based Persona Simulation of Established Roles
Table Aligmment (%) between human judges and LLM judges (or automatic metrics).
Expr.10 Conv.
Average Score
Expr.3 Conc.
Scoring Method
Alignment
60-
Expr.3
Conv.
Raw Text
GCA (GPT-4o)
68.6
Base
w/o reference
64.3
w/o rubrics
65.1
w/o length correction
64.5
w/o dimension separation
65.2
LLaMA
LLaMA -3.1-708
CoSER -8B
CoSER -70B
Qwen -2-728
GPT -40
GCA (DeepSeek-V3) V3)
65.1
-3.1-88
GCA (eeepSeek-R1)
77.5
w/o reference
77.2
BLEU ROUGE-L
75.3
Figure 4. LLM Performance COSER Test with retrieval aug- mentation from various character data. Expr. and Conv. denote experiences and conversations respectively.
72.0
Table Ablation study results (average scores) on CoSER Test, LT. and Mot. inner thoughts and motivations, respectively. Test w/o Mot.
5.6. Case Studies
Model
Standard
Test w/o I.T.
We conduct studies to analyze LLMs' performance in GCA simulation. Several cases presented in Tables 17 to 22, from which we observe that: 1) COSER models, trained on authentic dialogues, communicate more naturally, closely aligning with human speech patterns. 2) CoSER models better recall character-related knowledge, such as the iconic line "Grown enough to be wed, wed enough be bedded" by Lysa Arryn in Table 17. This is consistent with their high BLEU and ROUGE-L scores. 3) CoSER models better portray sophisticated thinking process of humans. For example, in Tables 19 to 22, COSER 70B faithfully reproduces Cersei Lannister's 's suppressed anger as depicted in the original conversations, while other models, including GPT-4o and Blaude-3.5-Sonee resort to a stereotypical portrayal of her arrogance and pride.
GPT-4o
59.95
56,89
56,34
Qwen-2-72B
57.06
51.95
54.21
LLaMA-3.1-70B
55.49
53.12
52.49
CoSER-70B
59.06
57.32
57.71
trained w/o I.T.
56.04
55.34
LLaMA-3.1-8B
52.99
51.97
49.63
CoSER-SB
56.45
54.65
56.81
trained w/o L.T.
54.25
54.38
Test with or without inner thoughts and motivations. The re- sults show consistent performance improvements across all models when inner thoughts and motivations are included.
Inner Thoughts Benefit Rolc-Playing Training We train COSER model variants without inner thoughts, and evaluate them various benchmarks. Results in Tables 3 and show that models trained without inner thoughts con- sistently underperform regular COSER models, exhibiting the value of inner thoughts for role-playing training.
6. Conclusion
Towards effective RPLAs for established characters, this pa- per introduces COSER, a collection of an authentic dataset, along with models and evaluation protocol based on such data. The COSER dataset offers high high-quality data from 771 renowned books, and includes comprehensive data types such as authentic dialogues, plot summaries, character ex- periences, inner thoughts, etc. Then, we propose given- circumstance acting (GCA) for training and evaluating role- playing LLMs, where LLMSs sequentially portray multiple characters within authentic book scenarios. Applying GCA training LLaMA-3.1 models using our dataset, we de- velop COSER 8B and COSER 70B, advanced open LLMs for role playipla.nn. For evaluation, GCA combines multi-agent simulation and penalty-based LLM critics. Extensive ex- periments exhibit COSER dataset's 's value for RPLA train- ing, evaluation, and retrieval. Moreover, COSER models achieve state-of-the-an art performance on both our evaluation and three existing RPLA benchmarks.
5.5. COSER Dataset for Retrieval Augmentation
We evaluate the value of our comprehensive data types for retrieval augmentation on CoSER Test. We explore three retrieval sources for a specific character: dialogues in related conversations (Conv.), as well as experiences (Expr.) and raw text in related plots. The retrieval system is based on FAISS (Douze al., 2024) with BGE-M3 (Chen et al., 2024b) for text embeddings. As shown in Fig. 16, we observe: /) Models consistently benefit from charac- ters' retrieved experiences and conversations, especially for COSER 70B; 2) However, raw text retrieval barely enhances LLMs' performance. Detailed experimental settings and results are provided in $D and Table 16.

---

COSER: Coordinating LLM-Basce Persona Simulation of Established Roles
Impact Statement
through self-knowledge distillation. ArXiv preprint, abs/2402.03216, 2024b.
COSER aims to advance RPLA research by providing ef- fective dataset, models and evaluation protocol. We will release our dataset, models, and evaluation scripts to foster innovation in RPLAs. The dataset is intended for research purposes only. For copyright policies, we will not distribute raw novel content. We require that anyone using our work must adhere to copyright policies and obtain proper per- missions for any derivative works. The COSER dataset derived from literary works may involve ethical consid- erations, and the content involved does not represent the authors' viewpoints. Our methods can potentially be ap- plied to develop agents for real-world individuals. However, such applications must strictly respect personal data privacy and obtain necessary consent. We hope our research will benefit RPLA researchers and developers, but emphasize the importance of responsible development. Any applications must respect copyright policies, personal data privacy, and be developed with proper licensing.
Chen, J.. Zhu, X., Yang. C., Shi, C.,, Xi, Y., Zhang, Y., Wang, J., Pu, J., Feng, T., Yang, Y., and Zhang, R. HoLLMwood: Unleashing the creativity of large lan- guage models in screenwriting via role playing. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguis- tics: EMNLP 2024, pp. 8075-8121, Miami, Florida, USA. November 2024c. Association for Computational Linguisties. doi: 10. 10.186653/11220444mmimggg 474. URL http:: https////acaaattthhhcocc .rg/ org/2024. 2024. E Findings-emn1p 474/.
Chen, N., Wang. Y., Jiang, H., Cai, D., Li, Y., Chen, Z. Wang, L., and Li, J. Large language models meet harry potter: A dataset for aligning dialogue agents with characters. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 8506- 8520, Singapore. 2023. Association for Computational Linguisties. doi: 10. 10.18655///22233mmninnn--s- 570. URL https: / /wwaaaaannllggyoooooooo f findings-emml 570.
References
Ahn, J., Lee, T., Lim, J., Kim, J.HH., Yun, S., Lee, H., and Kim, G. TimeChara: Evaluating point-in-time character hallucination of role-playing large language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 3291-3325, Bangkok, Thai- land and virtual meeting, 2024. Association for Compu- tational Linguistics. doi: 10. 10.18655w//2024.nddnggg 197. URL https: /tl:://aclllas www.aaantooooy.rrg///.. uindings-acl.1.
Dai, Y., Hu, H., Wang, L., Jin, S., Chen, X., and Lu, Z. Mmrole: A comprehensive framework for developing and evaluating multimodal role-playing agents, 2024. URL https: / wwwpxiiv.oog/eeo abs/ 00203.
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., and Narasimhan, K. Toxicity in chatgpt: Analyzing persona-assigned language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1236- 1270, Singapore, 2023. Association for Computational Linguisties. doi: 10.18655ww/202333mdnnn--mmmmm 88. URL https: / /a//acaaaan/ www.ennnnoogocorrooooooo findings-emm 88.
Chan, X., Wang. X., Yu, D.. Mi, H., and Yu, D. Scal- ing synthetic data creation with 1, 1,000,000,000 personas. ArXiv preprint, 01s/2406.200944 2024. URL https: //aaxivv.. org/ abs/
Chang, Y., Lo, K., Goyal, T., and Iyyer, M. Booookscore: A systematic exploration of book-length summarization in the era of Ilms. ArXiv preprint. abs/2310.00785, 2023. URL https: / aawiv. org/ abs/ 00785.
Douze, M., Guzhva, A., Deng, Johnson, J., Szilvasy, G., Mazaré, P.-E., Lomeli, M., Hosseini, L., and Jégou, H. The faiss library. ArXiv preprint, hwtp//201.082888 2024. URL https: / www.iv.org/aaaa. abs 0828.
Chen, J., Wang, X., Xu, R., Yuan, S., Zhang. Y., Shi, W., Xie, J., Li, S., Yang, R., Zhu, T., Chen, A., Li, N., Chen, I., Hu, C., Wu, S., Ren, S., Fu, 7.,, and Xiao, Y. From persona to personalization: A survey on role-playing language agents. Transactions on Machine Learning Research, 2024a. ISSN 2835-8856. URL https:///// openreview. net/ 1etrrumm2dxx0 Sur- vey Certification.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. ArXiv preprint, abs/2407.21783, 2024. URL wtt.t:tt////xiiv/te aar.iv.or.og////. abs/2407.2177
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias au- tomatic evaluators. ArXiv preprint, abs/2404.044777 2024. URL https:/t https/////axvo.o//aba/////////// www./ooo///////e..o.oooo 44.4744..44445 04475.
Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. Bge m3-embeddingg Multi-linguaa. multi-functione caltiffunctionality,, multi-granularity text embeddings
10

---

COSER: Coordinating LLM-Based Persona Simulation of Established Roles
Guo, D., Yang, D., Zhang, H., Song, J., Zhang. R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., al. Deepseck-rl: In- centivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501. 12948, 2025.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., al. Deepseek-v3 technical report. ArXiv preprint, abs/2412.114377 2024b. URL httpt:t//////. wtwparxiviv...o org abs 221./21112..99..... 19437.
Jiang, A. Q. Sablayrolles, A.. Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. L., Hanna, E. B., Bressand, F., et al. Mixtral of ex- perts. ArXiv preprint, Abs/2401.04088 2024. URL https:/ // www.i..rgg/a...a.aao..t...oooooe www.//gg//////oo//////////e.e///
Lu, K., Yu, B., Zhou, C., and Zhou, Large language mod- els are superpositions of all characters: Attaining arbitrary role-play via self-alignment. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis- tics (Volume I: Long Papers), pp. 7828-7840, Bangkok, Thailand, 2024. Association for Computational Linguis- tics. doi: 10. //aclanthology. org. awtp:22.cccl--. acl- 423.
04088.
Kosinski, M.
Theory of mind might have spontaneously emerged in large language models. ArXiv preprint, abs/2302.02083,, 2023. URL https: http:::/aaxxiv.rgg 0tt5/2202.000833
URL https:
OpenAl. Chatgpt, 2022. URL blog/ chatgpt.
com/
Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, J. V., Liu, A., Dziri, N., Lyu, S., al. T\" ulu Pushing frontiers in open language model post-training. ArXiv preprint, abs/2411.1511 URL wwtphhtp////.. /eses///axiiii //txxaavvvo. org/ abs/ /alas/2440 2411.2111110 15124.
OpenAL.
GPT-4 technical report. ArXiv preprint, abs/2303.08774, 2023. URL https://aziv..rrr abs 222.2333.0777
2024.
Li, C., Leng, Z., Yan, C., Shen, J., Wang, H., MI, W., Fei, Y., Feng, X., Yan, S., Wang, H., et al. Chatharuhi: Reviving anime character in reality via large language 2023. URL
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J Bleu: a method for automatic evaluation of machine translation. Isabelle, P., Charniak, E., and Lin, D. (eds.), Pro- ceedings the the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadel- phia, Pennsylvania, USA, 2002. Association for Com- putational Linguistics. doi: 10.3 10.3115/10730033107111 URL https: http://aaaantoooooooooo wwtp:/gg.oor..g..o wttp:/pgg/pp/ooe
model. ArXiv preprint, abs/2308.0953 https: / arxiv. org/ hxgg/bb//at /2ds/2308/2 09597.
Li, D., Jiang, B., Huang. L., Beigi, A., Zhao, C., Tan, Z., Bhattacharjee, A., Jiang, Y., Chen, C., Wu, T., et al. From generation to judgment:
Opportunities and challenges of
judge. ArXiv preprint, abs/2411..1594 2024. URL https::p///... / http/aixvvvv/// org/ 2494.
Papoudakis, A., Lapata, M., and Keller, F. Bookworm:
dataset for character description and analysis. In Find- ings of the Association for Computational Linguistics: EMNLP 2024, pp. 4471-4500, 2024.
Liang, Y. Zhu, L., and Yang, Y. Tachikuma: Under- stading complex interactions with multi-character and novel objects by large language models. ArXiv preprint, abs/2307.12577. 2023. URL https: / /ttp///xxvvv.////////// orr/ abs 2307/ 12573.
Park, J. S., O' O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In In the 36th Annual ACM Symposium User Interface Software and Technology (UIST 23), UIST '23, New York, NY, USA, 2023. Asso- ciation for Computing Machinery.
Lin, C.-Y. and Och, F. J. Automatic evaluation of machine translation quality using longest common subsequence and kkip-bigram statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pp. 605-612, Barcelona, Spain, 2004. doi: 222115//228895511110922 URL https: //aclannhology.ooooo.oo org.
Shanahan, M., McDonell, K., and Reynolds, L. Role play with large language models. Nature, 623(7987)::933 498, 2023.
Shao, Y., Li, L., Dai, J., and Qiu, X. Character-LLM: A train- able agent for role-playing. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 13153-13187, Singapore, 2023. Association for Compu- tational Linguistics. URL https: aclanthoooyy org 2223. 20202.mmmlpp.....mmmmmmmmmm emnlp in 814.
Liu, A., Diab, M., and Fried, D. Evaluating large language model biases in persona-steered genera- tion. In Ku, L.WW., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 9832-9850, Bangkok, Thailand, August 2024a. Association for Computa- tional Linguistics. doi: 1118663//1/20244.mmingg. 586. URL https: http:////aaaaaaoooooooooo.oe org//044. f #indings-acl.553
Shen, T., Li, S., and Xiong, D. Roleeval: A bilingual role evaluation benchmark for large language models.

---

CoSER: Coordinating -LM-Bascd Persona Simulation of Established Roles
ArXiv preprint, abs/2312.16132, 2023. URL https:
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F,, Dong, G., Wei, H., et. Qwen2 technical report, 2024. URL https::/aaxtttt org/ abs / /20/ 146.10667.
//arxiv.org/ abs/ / 16612..
Stanislavski, K. An Actor's Work: A Student's Diary. Rout- ledge, Abingdon, Oxon, 2008.
10671.
Yuan, X., Yuan, S., Cui, Y., Lin, T., Wang, X., Xu, R., Chen, J., and Yang, D. Evaluating character understanding of large language models via character profiling from fictional works. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
Tu, Q. Fan, S., Tian, Z., and Yan, R. Charactereval: A chinese benchmark for role-playing conversational agent evaluation. ArXiv preprint, 00NN2201.0112711 2024. URL https: // /ttxiv.xivvv/..//////.e.eea.aav. wwerxiv.orggviee ab// 2401 01275.
Wang, N., Peng. Z., Que H., Liu, J., Zhou, W., Wu, Y., Guo, H., Gan, R., Ni, Z., Yang. J. Zhang. M., Zhang, Z., Ouyang, W., Xu, K., Huang, W., Fu, J., and Peng. J. RoleLLM: Benchmarking, eliciting, and enhancing role-playing abilities of large language mod- els. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguis- ties ACI. 2024, pp. 14743-14777, Bangkok, Thailand and virtual meeting, 2024a. Association for Computa- tional Linguistics. doi: (..18653/w12222.fmmmmn 878. URL https: /tte:///aallainn htt///aaaattttyyy.ooooo org.//22...r//222 Eindings-acl.88 .8788
Zheng, L.,, Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena. Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances Neural Information Processing Systems 36: Annual Conference
Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL https: // //oeeneeveew. net www.mm/dd=ccccCDD....
Zhou, J., Chen, Z., Wan, D., Wen, B., Song, Y., Yu, J., Huang, Y., Peng. L., Yang, J., Xiao, X., et al. Char- actergim: Customizing chinese conversational ai char- acters with large language models. ArXiv preprint, abs/2311.16832, 2023. httpt http://arxvv.orr... abs/2311.16822
Wang, X., Xiao, Y., Huang, J.tt, Yuan, S., Xu, R., Guo, H., Tu, Q. Fei, Y., Leng, Z., Wang, W., Chen, J., Li, C., and Xiao,
InCharacter: Evaluating personal- ity fidelity in poly-playing agents through psychologi- cal interviews. In Ku, L.-W., Martins, A., and Sriku- mar, V. (eds.), Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870, Bangkok, Thailand, 2024b. Association for Computational Linguis- tics. doi: 10. 10.186633w//22224.adllooggg 102. URL ht tps::: //aclanthology. htw.aaannnoooooooogoo.eooo wtt/ acl www.clnloon..-eeeeeeeeeeeeeeeo 102.
Zhou, X., Zhu, H., Mathur, I.., Zhang, R., Yu, H., Qi, Z., Morency, L.-P., Bisk, Y., Fried, D., Neubig, G., and Sap, M. SOTOPIA: Interactive evaluation for so- cial intelligence in language agents. In The Twelfth International Conference on Learning Representations, 2024. URL https:////// pettps://openreviee /ppnrevieeweeeeeeeeeeeeeee net forum? dd=mM7VurbA4r.
Wataoka, K., Takahashi, T., and Ri, R. Self-preference bias in lim-as-a-judge. arXiv preprint arXiv:2410.21819, 2024.
Weng, L. Lim-powered autonomous agents. lilian- neng-github.io 2023. URL https: github. io potppp/stst/0/epep 06223-ggen///.
Wu, J., Ouyang, I.., Ziegler, D. M., Stienmon, N., Lowe, R., Leike, J., and Christiano, rizing books with human feedback. ArXiv preprint, ab862109.10862, 2021. URL https: https::/aaxiii https://rxxiv.ogg 0tt/2209..10865
Recursively summa-
Xu, R., Wang, X., Chen, J., Yuan, S., Yuan, X., Liang, J., Chen, Z., Dong, X., and Xiao, Y. Character is destiny: Can large language models simulate persona- driven decisions in role-playing?
ArXiv preprint, ab2/2404.12138,, 2024. URL hattps://axxiv.oxt abs/2404.1 12138.
121388......
12

---

COSER: Coordinating -LM-Bascd Persona Simulation of Established Roles
A. Dataset
A.1. Statistics and Analysis
As shown in Table 7, COSER dataset extensive and comprchensive, encompassing dialogue data from 771 books and 17, 17,96 distinct characters. The dataset includes 30,069 unique plots and 29,798 conversations. On average, each conversation consists of approximately 13.2 utterances, with the entire dataset comprising total of 392,298 utterances.
Table Statistics of COSER Dataset.
#Book
#Plot
#Conversation. #Chararacter #Utterance
771
30,069
29,798
17,966
392,298
Our book selection is derived from the Best Books Ever list on Goodreads, a curated collection of globally acclaimed literary works. These novels have garnered widespread recognition and appreciation from readers worldwide. Table 8 presents comprehensive list of the top 100 books from our selection.
We analyze the genres of the selected books based Supersummary classifications, with the statistical results presented in Figure 5. Our dataset encompasses wide range of genres, particularly fiction categories such as Fantasy, Historical, Science Fiction, Romance, and Mystery. It also features niche fiction genres, showcasing diverse narrative styles. In addition to fiction, the collection includes non-fiction genres such as memoirs, biographies, and other works, enhancing its versatility.
Matorrca Roved
Sci econnee Fiction: se (7. (.77)
Romance Nlovel: so (6.5%
Young Novels: (6.2%)
Fantasy Novels: 167 (21.7%)
Mystury Thriler: 34 (4.4%)
Mepical Reatien: (4.5
Psychologiaal movels: 21 71)
Fhitcxepbbtnn 20 4t
Others: 112 (14.5%)
sevec
Figure Genre distribution of selected books CoSER dataset.
A.2. Data Splits
For evaluation purposes, we held out the last 10% of data from each book; that is, they are not included in our prompts or datasets for training or retrieval purposes. Additionally, we trained the COSER models on only 90% of the books.
A.3. Implementation Details for Construction
Extracting Raw Text LLMs often struggle to extract verbatim original content, especially with punctuation marks like quotation marks, making it difficult to extract raw text directly. Therefore, instead of asking LLMs to extract the complete text plot, we prompt LLMs to extract the first and last sentences each plot. Since the extracted sentences may still contain typos, we apply lexical similarity to match them with the exact sentences from the raw text. Finally, we identify the complete raw text based on these first and last sentences.
13

---

COSER: Coordinating JLM-Based Persona Simulation of Established Roles
Table 8. The top 100 selected books from Goodreads' Best Books Ever list.
Selected Books
1. The Hunger Games (The Hunger Games, #1)
Harry Potter and the Order of the Phoenix (Harry Potter, #5)
3. Pride and Prejudice
4. To Kill a Mockingbird
5. The Book Thief
Animal Farm
7. The Chronicles of Narnia (The Chronicles of Narnia, #1-7)
8. The Faulr in Our Stars
9. The Picture of Dorian Gray
10. Wathering Heights
11. Gone with the Wind
12. The Perks of Being a Wallflower
13. The Lightning Thief (Percy Jackson and the Olympians, #1)
14. The Little Prince
15. The Great Gutsby
16. Crime and Punishment
17. Memoirs of a Geisha
18. Les Misérables
19. The Alchemist
20. Lord of the Flies
21. The Hitchhiker's Guide to the Galaxy (#1)
22. The Help
23. Dracula
24. Ender's Game (Ender's Saga, #1)
25. Of Mice and Men
26. One Hundred Years of Solitude
27. Brave New World
28. A Thousand Splendid Suns
29. The Time Traveler's 's Wife
30. The Princess Bride
31. The Secrel Garden
32. The Outsiders
33. A Game of Thrones (A Song of Ice and Fire, #1)
34. Little Women
35. Wrinkle in Time (Time Quinter, #1)
36.
The Odyssey
37. Harry Potter and the Deathly Hallows (Harry Potter, #7)
38. Frankenstein: The 1818 Text
39. The Kite Runner
40. The Handmaid's ss Tale (The Handmaid's 's Tale, #1)
41. The Lovely Bones
42. The Adventures of Huckleberry Finn
43. Life of Pi
44. A Tale of Two Cities
45. Dune (Dune, #1)
46. Harry Potter and the Prisoner of Azkaban (Harry Potter, #3)
47. Water for Elephants
48. Harry Potter and the Sorcerer's 's Stone (Harry Potter, #1)
49. The Bell Jar
50. Matilda
51. The Stand
52. Catch-22
53. The Adventures of Sherlock Holmes (Sherluck Holmes, #3)
54. The Pillars of the Earth (Kingsbridge, #1)
55. Rebecca
56. Great Expectations
57. The Girl with the Dragon Tattoo (Millennium, #1)
58. The Color Purple
59. Anna Karenina
60. My Sister's 's Keeper
61. The Brothers Karamazov
62. A Clockwork Orange
63. And Then There Were None
64. The Road
65. To Kill a Mockingbird
66. The Golden Compuss (His Dark Materials, #1)
67. Vampire Academy ( /Vampire Academy, #1)
68. Siddhartha
69. The Complete Stories and Poems
70. Interview with the Vampire (The Vampire Chronicles, #1)
71. Don Quixote
72. The Old Man and the Sea
73. The Poisomwood Bible
74. Harry Potter and the Goblet of Fire (Harry Potter, #4)
75. Atlas Shrugged
76. The Notebook (The Notebook, #1)
77. Harry Potter and the Half-Blood Prince (Harry Potter, #6)
78. Moby-Dick or, The Whale
79. A Prayer for Owen Meany
80. Clockwork Angel (The Infernal Devices, #1)
81. The Stranger
82, The Secret Life of Bees
83. Harry Potter and the Chamber of Secrets (Harry Potter, #2)
84. The Red Tent
85. The Name of the Wind (Tneee (The Kingkiller Chronicle, #I)
86. The Master and Margarita
87. The Metamorphosis
88. Eragon (The Inheritance Cycle, #1)
89. The Count of Monte Cristo
90. Looking for Alaska
91. The Adventures of Tom Sawyer
92. Charlie and the Chocolate Factory (Charlie Bucket, #1)
93. The Last Olympian (Percy Jackson and the Olympians, #5)
94. The Curious Incident of the Dog in the Night-Time Time
95. The Shadow of the Wind (Cemetery of Forgotten Books, #1)
96. The Unbearable Lightness of Being
97. On the Road
98. The Name of the Rose
99. A Story of Yesterday
100. The Godfather (The Godfather, #1)
14

---

CoSER: Coordinating LLM-Based Persona Simulation of Established Roles
Parsing Structured Data During extraction, we instruct LLMs output extracted data in JSON format. However, LLM-generated JSON strings may sometimes be unparseable or do not conform to the specified format (e.g., missing required keys).
Towards this challenge, we adopt repair-and-retry strategy to improve extraction success rate. For cach chunk to be extracted, we invoke LLMs and attempt to parse a valid JSON object. parsing fails, we employ LLMs to repair the invalid JSON string and retry. Some failures occur because the LLM attempts to output JSON that exceeds the maximum token limit, resulting in truncation. In such cases, we prompt the repairing LLM truncate the JSON an appropriate point. If still fail, we restart the entire process from the beginning, making up attempts.
Refining Conversation Settings During data extraction, we observe that the initially extracted conversation settings, including scenarios and character motivations, often fail provide a comprehensive context. We attribute this to the LLM's tendency to distribute information across different data fields when extracting multiple kinds of information simultaneously, rather than repeating in different data fields. For instance, if certain information is already mentioned in the plot summary, it might be omitted from the scenario description.
Therefore, to provide a complete context for cirenccrcuumttacce acting, we implement an additional LLM call to refine the conversation settings based on the extracted data. We instructed the refining L.LM to provide a comprehensive conversation setting, while carefully avoiding any disclosure subsequent dialogue content or plot developments.
For additional details, such as the regular expressions used for identifying chapter titles, please refer to our code.
A.4. Comparison with Existing Methods for Character Profiling
Previous character profiling methods, including hierarchical updating (Wu et al., 2021), incremental updating (Chang et al., 2023), and one-shot summarization (Yuan al., 2024), typically only generate the profile of a single character at a time. Morevoer, Papoudakis et al. (2024) shows that these methods, particularly hierarchical updating, perform suboptimally when generating multiple character profiles simultancously.
CoSER's 's multi-stage, extract-then tarrat-then-aggregatee pipeline addresses these limitations. It ensures comprehensive character profiles with high precision and recall of character knowledge, capturing evolving character arcs, and significantly improving processing efficiency.
A.5. Safety
We conduct safety checks on the dataset and remove potentially problematic content. Specifically, we truncate 110 sensitive conversations and remove a total of 602 messages.
A.6. Comparison with PersonaHub
Recently, several studies have explored LLM role-playing using synthesized personas rather, such as PersonaHub (Chan et al., 2024). We provide a systematic comparison between CoSER and PersonaHub below.
Goal and Focus
The fundamental objectives of the two datasets differ significantly:
Aspect
CoSER
PersonaHub
Target
Simulate established personas with high fidelity
Synthesize instruction data for knowledge distillation
Persona Focus
Depth and richness of character data Anthropomorphism, character fi- delity, multi-character interaction
Breadth persona types
Key Capabilities
General instruction following
Table Comparison between COSER and PersonaHub regarding goal and focus.
Dataset Quality and Design
The data sources and design reflect distinct research priorities:
This comparison highlights that while PersonaHub excels in generating diverse synthetic personas for broad instruction-
15

---

COSER: Coordinating ALM-Bascd Persona Simulation of Established Roles
Aspect
CoSER
PersonaHub
Data Source
Authentic book dialogues
GPT-4o synthesized conversations
Persona Origin
Book characters
LLM--yythesized personas
Persona Data
Comprehensive profiles, dialogues,
Concise profiles with limited con- text
experiences
Design Priority
Quality, authenticity, and character depth
Quantity and diversity
Table 10. Comparison between COSER and PersonaHub regarding dataset quality and design.
following training, COSER focuses on preserving the authenticity and depth of established characters for high-fidelity role-playing applications.
B. Training
B.1. Training Samples
We transform conversations from the COSER dataset into training samples in the Sharegpt format. We utilize 90% of the books in the dataset for training, while the remaining 10% are set aside to evaluate our models' ability to generalize to out-of-domain characters and books. We construct one training sample for each character in every CoSER conversation, encompassing both main characters and minor roles. When training on a character c, we designate c's messages as targets for optimization, while using the system prompt and messages from other characters as inputs. Adjacent inputs are concatenated.
Towards general role-playing capabilities across diverse scenarios and applications, we dynamically generate role-playing instructions (system prompts) using varied phrasings, formats, and data types, as shown in Tables 27 and 28. We consider instructions entirely in natural language, as well as those formatted with special symbols (such as ### =-), randomly sampling different formats and various expressions for the same semantics. We consider various configurations of the available data. Each sample may include (50%) or exclude (50%) the following elements 7) Profiles of other characters in this conversation; 2) Summaries of the relevant plot; 3) Inner thoughts within the messages.
Besides character role-playing, we train COSER models for environment modeling and next speaker prediction (NSP) for multi-agent simulation. For environment modeling, we train LLMs m.e to play the environment role in the same approach, leveraging environment messages in our dataset. For NSP, given setting S and messages {mi,.... m,); train LLMs to predict the speaker of m;+1 (or ending the conversation).
Our role-playing dataset comprises approximately O.IB tokens, as measured using the LLaMA 3.1 tokenizer. To maintain general intelligence and instruction-folle -osluotton-follooiing capabilities, we augment this with an equivalent volume (O.1B tokens) of general-purpos supervised fine-tuning data from Tulu (Lambert et al., 2024) 12, This balanced mixture ensures that the model retains broad language understanding while developing specialized role role-playing abilities. If more role-playing data expected, our data curation pipeline be easily applied to additional books other fictional works, thereby acquiring data on much larger scale.
For more details, please refer to our code.
B.2. Hyperparameters
We fine-tune the LLaMA 3.1 models using the following hyperparameters: learning rate of 1 10 (0-5,, sequence length of 16,384, training for 8 epochs, and global batch size of 48.
C. Experiment Settings
C.1. Test Set Sampling
The COSER Test set contains 200 samples: 100 from books used CoSER training (in-domain) and 100 from books not used training (out-of-domain). We employ a weighted sampling strategy prioritize well-established characters with
16

---

COSER: Coordinating LLM-BBscc Persona Simulation of Established Roles
more persona data. The sampling process consists of the following steps: First, for each book, we assign character weights as the square root of the number of plots in which they are involved. Second, calculate the weight of a conversation as the average of its characters' weights, including both main characters and side roles. Finally, separately for the in-domain and out-of-domain settings, we rank all conversations by weight and perform weighted sampling from the top half of conversations with higher weights.
C.2. Prompting Strategies for Exisitng RPLA Benchmarks
For existing RPLA benchmarks, including InCharacter (Wang et al., 2024b), LifeChoice (Xu et al., 2024), and Cross- MR (Yuan et al., 2024), we adapt or refine their prompting strategies as follows:
1. For InCharacter, we add "You'r re consulting with a personality assessment expert who will ask you some questions. Please provide honest and detailed responses help with the analysis. Please think carefully and state your reasons when answering the questions." after the character profile. This adaptation aims to ensure that RPLAs honestly express their true thoughts. After being trained on authentic character dialogues, the CoSER models, unlike general LLMs, tend to produce brief, conversational-style answers that may be too short or may decline to answer questions, thus failing to provide sufficient information for personality assessment.
2. For LifeChoice and Cross-MR, we reverse the order of their reasoning and answering processes. Specifically, we have them think before providing their choices, thus enabling RPLAs make well-considered decisions.
C.3. Length Correction
In our evaluation, we use penalty- based scoring mechanism that counts the flaws in RPLAs' performance. However, since longer simulations naturally accumulate more flaws, we need to implement length correction to reduce length bias in LLM judges, following previous work (Li et al., 2024).
We analyze the phenomenon of length bias in penalty- based scoring. The initial score is defined as 100 -E vy, where F represents the set of flaws and vy is their severity ranging from to 5. Our analysis is conducted on the initial scores from simulations of three models the CoSER Test set: COSER 70B, LLaMA-3.1 70B, and GPT-4o, with without retrieval augmentation (three Experience and one Conversation), totaling 1,200 cases. We examine the relationship between the number of rounds and the vanilla scores in these 1,200 cases. As shown in Figure 6, we plot the data points for these cases and perform linear regression to fit these points. The fitted linear function is score = - 1.5909 x rounds + 59.0617, which means that for each additional round in the simulation, the score decreases by approximately 1.6 points.
To mitigate this bias, we implement length correction by compensating for the points deducted due to increased rounds. Specifically, we compute the length-corrected score as 100 ΣfEF A|M where À is set to 1.5 based on the analysis above.
C.4. Human Evaluation
The alignment metrics are measured via the frequency of agreement between LLM and human judges when comparing model pairs, excluding cases where judges assign similar scores to both models (i.e., score differences S 1 for human judges or < 5% for automatic metrics).
D. Additional Results
Evaluation with Different LLMS as Judges We conduct additional experiments using alternative judge models for GCA evaluation. We evaluate 7 representative models as shown Table 11, using three different judge models: GPT-4o, DeepSeek-R1, and DeepSeek-V3. This comparison aims to examine whether our conclusions remain consistent across different LLM judges.
The results in Table 11 reveal several important findings: 1) Despite these variations in absolute scores, the relative ranking patterns remain largely consistent, with CoSER models maintaining competitive or superior performance across all settings. 2) Different judges exhibit distinct preferences, confirming potential bias in preference for GPT models, DeepSeek-RR R1 and DeepSeek- V3 favor COSER-70B and Claude-3.5-Sonme over GPT-4o.
evaluation. While GPT-4o shows
17

---

CoSER: Coordinating ALM-Based Persona Simulation of Established Roles
Vanilla Number Rounds Regression
80
Linear fit (R 0.139)
1000
60
Vanilla Score
40
20
Number Rounds
Figure 6. Linear regression results showing length hias of penalty-hased L.I critics in GCA evaluation.
Table 11. Scores (%) assigned by different LLM judges. Results demonstrate consistent superiority of CoSER models, while revealing judge-specific preferences.
Model
GPT-4o
DeepSeek-RI
DeepSeek- V3
GPT-3.5
52.8
35.9
40.5
L.LMA-3.1-8B
51.8
37.2
36.8
Ababe-preview
53.7
41.5
40.4
CoSER-8B
56.1
44.5
45.9
GPT-4o
58.5
48.4
46.1
Claude-3.5-Somee
56.2
54.8
40.7
CoSER-70B
57.4
50.8
47.7
Win Rates against GPT-4o and GPT-3.5 To further evaluate existing LLMs in civen-cireumttance acting, we present their win rates against GPT-3.5 (OpenAI, 2022) and GPT-4o (OpenAI, 2023) in Table 12, in addition to the results shown in Table
Generalization of COSER Models to New Characters Table 13 separately presents the performance of LLMs test splits from books included in and excluded from CoSER training. The results demonstrate consistent trends across both splits, confirming that CoSER models maintain strong performance even out-of-domain characters.
Conversation Continuation Table 14 shows experiment results when multi-agent systems continue conversations from the first & = 3 original messages.
Tables 14 presents detailed evaluation results where our multi-agent simulations start from the first 3 original messages.
COSER Dataset for Retrieval Augmentation We validate the effectiveness of CoSER 's comprehensive data types for retrieval augmentation on the COSER Test set. We explore three retrieval sources related to specific characters: dialogues from conversations, experiences from plots, and raw text from plots. We compare several combinations of these sources, including: /) None (Base) 2) Raw text of one plot (Raw Text) 3) One conversation (Conv.) 4) Character experiences from three plots (Expr.3) 5) Expr.3 combined with Conv. 6) Expr. 10 combined with Conv. The complete results are presented in Table 16.
18

---

CoSER: Coordinating ALM-Baascd Persona Simulation of Established Roles
Table 12. Win rates (%) of various LLMs on given-cincumstance acting using COSER Test, compared against GPT-3.5 and GPT-4. Bold and underlined numbers indicate the best results among all and open-source models, respectively.
Win Rate
Model
vs. GPT-3.55 vs. GPT-4o
Close-source Models
Abab7-preview
52.17
35.33:12288
Doubao-pra Pro
64.67
48.00
Step-1-Flash
58.75=6.77
39.75++1.44
Step-2
65.08
46.92++4.40
GPT-3.5
50.00=0.00
33.08:3364
GPT-4o
66.9213.64
50.00.10.00
GPT-4o Mini
47.58
42.92.10.63
Gemini Pro
66.17
49.75
Claude-3-Haiku
51.42.1.40
33.25
Claude-3.5-Someee
58.1718.00
41.42
Open-source Models
Mistral-7B
41.50
26.67
Qwen-2-7B
24.83=6.64
13.58+3.00
LLaMA-3.1-SD
46.33
32.7545.86
CoSER-8B
58.17=5.30
41.67+3.75
Vicuna- 13B-1.5
30.50+
18.6742.18
Mixtral-8x7B
32.50=1.50
19.83+1.42
Qwen-2-72B
59.58
41.92.1.60
L.aMA-3.1170B
53.83
37.92-43.26
Higgs-Lllama-3700B
51.83
34.92.14884
CoSER-70B
64.33
19.42.14.11
DeepSeck-V3
59.50=2.82
39.50+2.05
Table 13. LLM performance (%) on given- civen-circumstanee acting using CoSER Test, separated into in-domain and out-of-domain splits for CoSER training.
In in-Domain
Model
Storyline Consistency
Anthro-
Character
Storyline Quality
Average Score
Storyline Consistency
Anthro- pomorphism
Character Fidelity
Storyline Quality
Average Score
pomorphism
Fidelity
Close-source Models
Abab7-preview
55.31
42.29.1.3
42.94
74.13+1.9
+1.07
58.30
46.17+2.6
44.72+2a
75.54
56.18 0.1
Doubao-pro
60.37
49.06=0.4
45.76234
77.87+1.2
58.2620.9
61.53+1.1
50.38=0.3
48.28
80.6910.4
60.22.0.4
Step-I-Flash
57.10+n.3
48.31
41.8421.9
76.18+1.5
55.86#1.9
58.39+1.2
47.94=0.6
47.12+1.
75.67+0.5
57.28
Step-2
60.55
48.8212.2
47.90
77.3311.0
58.65
62.30
49.30=1.2
46.77
78.5911.2
59.24
GPT-3.5
55.58
42.1815.2
40.37+3.2
72.9010.1
52.7622.0
59.6923.0
44.25
44.60
74.6612.3
55.80
GPT-4o
59.88+1.4
48.11#1.0
47.10=0.2
79.06+1.5
58.54
62.29+1.5
49.48
49.90
80.18
60.46=u.1
GPT-4o Mini
59.15
46.94.1.5
43.9942.5
77.33
56.85
61.0211.6
49.48
45.77
79.7710.5
59.01
Gemini Pro
57.72+0.4
50.9421.8
46,23+1.
76.22+1.6
57.78=0.9
60.50+1.9
53.88=1.1
49.43
78.97+1.3
60.69
Claude-3-Haie
57.61
44.97=2.
40.61214
73.52+1.2
54.1820.6
58.74+1.1
44.36=1.9
43.1420.8
74.76+1.7
55.25=1.2
Claude-3.5-Somee
56.44+1.6
47.24
44.89
76.39+1.5
56.24
58.46+1.1
49.75
46.4943.0
78.06+1.5
58.19
Open-source Models
Mistral-7B
60.29
38.98
42.93
62.20+3.1
51.10
59.51
41.02+1.3
46.57
61.65+1.3
52.19
Qwen-2-7B
50.77+1.3
34.17+1.1
29.92+2.
62.58
44.36
53.14+1.5
36.79+1.0
33.09
63.79+1.4
46.70+1.3
LLaMA-3.1-88
53.00
43.6410.8
39.0541.5
71.34
51.76.0.8
55.20
47.08
41.38110
73.2312.5
54.22+1.8
CoSER-8B
58.5643.5
46.78=1.1
45.78-3.1
73.38+1.8
56.12.0.5
58.6641.a
47.69-n.s
48.03+1.3
72.7141.2
56.77=0.6
Vicuna-13B-1.5 135
51.84+1.2
38.88
36.3940.5
58.1540.7
46.31+0.4
53.6642.1
39.35+2.7
39.69+16
62.7142.5
48.85+2.1
Mixtral-&x7B
51.18+3.1
38.7641.8
34.37+28
66.44.0.1
47.69
51.32+v.4
38.13
39.48
68.93+1.7
49.47+1.1
Qwen-2 2-BBB
56.34112.4
46.19.0.4
45.59+2.2
75.6810.3
55.95
59.1511.1
48.37=1.3
47.65
77.5210.4
58.17
55.44+2.7
44.19-1.9
42.67+1.6
73.90+1.8
54.05
59.48+1.4
47.72=2.3
44.78
75.78+0.a
56.942n.7
55.85+2.7
41.18+33
39.79+23
73.77+0.6
52.65+2.9
58.35
46.45+1.1
45.04+1.0
77.48+0.6
56.83+0.6
CoSER-705
57.77+1.3
51.60+13
45.82
74.2741.1
57.37
59.56=2.1
55.061111
51.67425
76.7111.6
60.75
DeepSeek-VV V3
55.3640.5
47.55
43.1020.3
74.91+2.
55.23
57.4542.0
48.19
44.9320.4
78.41
57.24
19

---

CoSER: Coordinating JLMM-Baccd Persona Simulation of Established Roles
Table Performance (%) various LLMs on COSER Test conversation continuation setting 3), where RPLAs start from the
first three messages in the authentic conversations.
Based on LLM Judges
Based on N-gram
Model
Storyline
Anthro-
Character Fidelity
Storyline Quality
Average Score
BLEU
ROUGE-L
Consistency
pomorphism
Close-source Models
Abab7-preview
65.25
55,26
55.95
79.68
64.03
10.53
15.99
Doubao-pro
68.31
59.51
59.23
80.15
66.80
11.83
17.13
Step-I-Flash
64.32
53.35
54.38
79.03
62.77
11.94
17.06
Step-2
66.61
55.81
59.59
80.56
65.64
11.72
17.27
GPT-3.5
65.72
54.34
56.48
77.67
63.55
10.80
16.39
GPT-4o
67.48
58.51
62.30
82.23
67.63
12.32
16.98
GPT-4o Mini
66.74
55.55
56.80
80.01
64.77
9.50
15.54
Gemini Pro
65.47
59.43
62.42
78.78
66.53
10.84
16,27
Claude-3-Haie
64.51
54.01
57.13
77.26
63.23
10.11
16.18
Claude-3.5 5-Sonnet
64.54
54.57
58.76
79.89
64.44
8.64
14.94
Open-source Models
Mistral-7B
67.50
50.39
59.90
68.67
61.62
7.41
14.33
Qwen-2-7B
59.81
49.87
49.04
69.16
56.97
10.59
16.22
L.AMA-3.1-8B
60.90
51.36
50.37
74.89
59.38
7.86
13.82
CoSER-8B
67.22
58.19
58.80
76.44
65.16
13.17
18.42
Vicuna- 13B-1.5
60.92
46.68
50.51
67.80
56.48
5.11
9.71
Mixtral-8x7B
64.66
51.21
54.08
74.01
60.99
11.21
16.97
Qwen-2-72B
67.27
55.87
59.84
80.07
65.76
11.92
16,96
LLaMA-3.1170B
64.08
54.80
54.18
78.31
62.84
8.74
14.74
Higgs-Llamma--700
65.09
54.80
58.20
79.36
64.36
10.86
16.40
CoSER-70B
65.99
59.24
59.97
76.74
65.48
13.46
18.18
DeepSeek-V3 62.95
56.95
58.25
79.80
64.49
9.25
15.01
Ablation Studies We examine the effectiveness of inner thoughts in both training and evaluation. The complete results are
demonstrated in Table 15.
Comparing GCA Simulation with Al-driven Storytelling Methods To provide more comprehensive evaluation against prior work in digital actors and AI-driven storytelling, we compare our GCA simulation approach with HollmWood (Chen et al., 2024c), a representative method for Al-generated storytelling. We conduct this comparison using 30 samples from COSER Test, employing GPT-4o as both the actor and writer in both frameworks to ensure fair comparison. The generated stories and character interactions evaluated using GCA evaluation. Results demonstrate that GCA simulation achieves average score of 59.2%, significantly outperforming HollmWood's 50.2%. These findings highlight that our given-circumstance acting approach produces more authentic and human-like character interactions compared to previous AI storytelling methods. The superior performance of GCA simulation can be attributed to its focus on character- dialogue generation within well-defined circumstances, rather than plot-driven narrative construction, leading to more faithful character portrayals and realistic conversational dynamics.
E. Authentic v.s. LLM-Generated Role-playing Data
To demonstrate the superiority of authentic role-playing data over synthesized alternatives, we conduct two complementary experiments comparing authentic conversations from COSER dataset with I.LM-generated dialogues.
GCA Evaluation of Dialogue Quality We apply GCA evaluation to score authentic (groundtruth) and I.L.-ggenrrted dialogues on 100 samples from CoSER Test without providing groundtruth to LLM judges reference. We use DeepSeek- RI as the judge model due to its superior alignment with human evaluators (as shown in $5.3). GPT-4o is used to generate synthetic dialogues via GCA simulation. Results show that authentic dialogues from CoSER achieve a score of 85.1% 15.1% significantly outperforming GPT-4o generated dialogues at 76.2%.
20

---

resort to a CoSER: Coordinating JLM-Bascd Persona Simulation of Established Roles
Table 15. Comprehensive results of L.LM performance (%) COSER Test, with and without inner thoughts (I.T) during inference and
CoSER model training.
Based on LLM Judges
Based on N-gram
Model
Storyline Consistency
Anthro-
Character
Storyline Quality
Average Score
BLEU
ROUGE-L
pomorphism
Fidelity
Test w/ L.T.
GPT-4o
61.59
48.93
48.95
80.33
59.95
5,90
12.11
Qwen-2-72B
57.75
47.28
46.62
76.60
57.06
5.38
11.85
LLaMA-3.1-70B
57.46
45.95
43.72
74.84
55.49
4.82
10.98
CoSER-70B
58.66
53.33
48.75
75.49
59.06
10.10
14.78
trained w/o I.T.
55.67
49.93
45.67
72.89
56.04
9.18
14.18
LLaMA-3.1-8B
54.10
45.36
40.22
72.29
52.99
4.59
10.18
CoSER-8B
58.61
47.23
46.90
73.04
56.45
9.40
14.21
trained w/o L.T.
54.91
44.89
44.10
73.09
54.25
9.65
14.27
Test w/o I.T.
GPT-4o
59.51
45.86
45.13
77.06
56.89
5.54
11.44
Qwen-2-72B
55.39
41.00
37.70
73.70
51.95
5.22
11.32
LLaMA-3.1-70B
56.01
43.17
41.23
72.08
53.12
4.15
10.37
CoSER-70B
59.23
48.76
47.06
74.23
57.32
10.02
14.72
trained w/o I.T.
55.52
46.22
45.34
74.31
55.34
9.66
14.62
1LaMA--3.1-81
54.27
45.54
37.86
70.23
51.97
4.37
10.27
COSER-8B
56.78
45.64
45.00
71.16
54.65
9.38
14.44
trained w/o I.T.
54.98
46.22
45.81
70.52
54.38
9.16
13.97
Training Effectiveness Comparison We further examine the training effectiveness of authentic versus synthetic dialogues. We fine-tune LLaMA-3.1-8B using authentic or LLM- -eMerreedd dialogues for 200 samples from CoSER Test for epochs, using the same settings as CoSER-SB. We evaluate the two trained models via GCA evaluation on 100 new samples. The model trained on authentic dialogues from COSER achieves 55.3% score, while the model trained on GPT-4o generated dialogues achieves 54.7%.
These results demonstrate that authentic dialogues consistently outperform LLM-generatee alternatives in both intrinsic quality and training effectiveness. These findings highlight the importance of authentic, high-quality role-playing data for developing effective RPLAs.
F. Examples and Case Study
We present several examples of our extracted conversations, well corresponding simulations in this given circumstance by LLMs.
Tables 17 to 18 illustrate a classic conversation extracted from A Storm of Swords (A Song of Ice and Fire, #3) and the corresponding simulation by CoSER 70B. In CoSER 70B's simulation, when confronted with Sansa Stark, Lysa Arryn utters her iconic line, "Grown enough to be wed, wed enough to be bedded", reflecting her personality and worldview. This indicates that COSER models excellently recall and apply
character.rauee"" character-related knowledge from their pretrained data.
Tables 19 22 present another scene from A Dance with Dragons (A Song of Ice and Fire, #5), specifically the walk of atonement, in which Cersei Lannister is forced walk naked through the streets, facing both physical and mental humiliation while striving to preserve her dignity. We present the original dialogue alongside simulations by COSER 70B, GPT-4o, and (laude-3.5-Somat Notably. COSER 70B faithfully captures the suppressed anger of Cersei Lannister as depicted in the original conversation, whereas the other models, including GPT-4o and stereotypical portrayal her arrogance and pride.
Table 23 presents an output example L.M critics, demonstrating the identified performance flaws GCA simulation.

---

COSER: Coordinating ALM-Based Persona Simulation of Established Roles
Table 16. Comprehensive results L.LM performance (%) on COSER Test with retrieval augmentation from various character data. Expr. and Conv. denote experiences and conversations. Bold and underlined numbers denote the hest and second-hest results among different retrieval settings, respectively.
Based on LLM Judges
Based on N-gram
Augmented By
Storyline Consistency
Anthro- pomorphism
Character Fidelity
Storyline Quality
Average Score
BLEU
ROUGE-L
GPT-4o
Base Raw Text
61.59
48.93
48.95
80.33
59.95
5.90
12.11
59.60
48.23
48.83
80.03
59.17
5.83
12.08
Conv.
60.03
50.99
47.11
78.03
59.04
7.20
12.77
Expr.3
61.98
48.94
48.56
79.93
59.85
5.81
12.13
Expr.3 Conv.
59.81
50.11
51.12
80.29
60.33
7.05
12.67
Expr. 10 Conv.
60.99
48.70
49.89
79.22
59.70
6.95
12.42
LLaMA-3.1-8B
Base Raw Text
54.10
45.36
40.22
72.29
52.99
4.59
10.18
55.41
45.37
41.66
74.31
54.19
4.73
10.51
Conv.
56.54
45.34
40.37
71.74
53.50
5.19
10.67
Expr.3
56.78
46.70
41.18
72.38
54.26
4.67
10.30
Expr.3 + Conv.
57.36
47.92
42.60
74.60
55.62
5.21
10.81
Expr. 10 Conv.
56.41
47.18
42.09
74.74
55.10
5.64
11.11
L.MMA-3.1-70B
Base Raw Text
57.46
45,95
43.72
74.84
55.49
4.82
10.98
57.52
47.46
43.75
74.06
55.70
4.90
10.97
Conv.
54.93
44.93
46.21
76.53
55.65
5,45
11.13
Expr.3
56.59
45.03
42.59
74.70
54.73
4.78
10.82
Expr.3 Conv.
57.36
44,89
43.49
74.86
55.15
5,58
11.14
Expr. 10 Conv.
58.05
45.81
44.47
74.01
55.59
5,53
11.33
CoSER-S0
Base
58.61
47.23
46.90
73.04
56.45
9.40
14.21
Raw Text
57.64
45.25
44.27
72.04
54.80
9.34
14.41
Conv.
57.99
46.66
47.32
73.99
56.49
13.64
18.15
Expr.3 Expr.3 Conv.
59.84
47.41
48.16
74.51
57.48
9.37
14.34
59.94
45.25
48.52
74.96
57.17
13.90
18.57
Expr. 10 Conv.
58.34
48.14
48.72
75.28
57.62
13.49
17.85
CoSER-7H -7B
Base
58.66
53.33
48.75
75.49
59.06
10.10
14.78
Raw Text
60.64
52.85
47.97
74.02
58.87
10.54
15.41
Conv.
64.59
53.79
54.86
77.28
62.63
17.22
21.17
Expr.3 Expr.3 Cony.
58.67
52.69
50.66
74.67
59.17
10.00
14.82
61.58
53.78
52.00
77.47
61.21
15.98
19.95
Expr. 10 + Conv.
61.53
52.58
50.80
78.07
60.75
15.80
19.90
Qwen-2-72B
Base
57.75
47.28
46.62
76.60
57.06
5,38
11.85
Raw Text
58,89
47.31
45.28
76.78
57.06
5.07
11.57
Conv.
59.11
47.82
44.19
77.54
57.16
5.77
12.05
Expr.3
59.17
47.21
44.85
77.47
57.18
5.45
11.94
Expr.3 + Cony.
59.91
46.89
46.06
79.31
58.04
6.17
12.25
Expr. 10 Conv.
60.14
46.84
46.27
76.82
57.52
5.85
12.19
22

---

COSER: Coordinating LLM-Based Persona Simulation of Established Roles
G. Prompts
this section, we list the detailed prompts for: 1) dataset curation Tables 24 26; 2) RPLA and multi-agent simulation in Tables 27 28. which have been carefully optimized based on our experience in multi-agent simulation; 3) Penalty-based LLM Judging in Tables 29 30.
H. Limitations
There are several limitations to this study:
First, evaluation via seven-cireumsttnee acting still faces challenges related to LLM judges. While the simulation stage effectively elicits RPLA performance, the judging stage still relies on LLM judges. Despite our penalty-basee scoring mechanism and detailed rubries, problems such as length bias persist. Moreover, LLM Judges may lack the necessary knowledge accurately evaluate character fidelity.
Second, while the dialogues extracted from novels are authentic, their corresponding thoughts remain to be optimized by future work. Character thoughts are often sparse in the original content, and are inferred by LLMs based on limited context. The generated thoughts hardly capture characters' sophisticated thinking processes.
Third, although we've
we've ve developed comprehensive data representations and curation pipeline to obtain high-quality data, we have not yet addressed the issue of recall in data extraction. Our current dataset may not cover all plots, conversations and characters from the source material. Improving recall is hence important area for future research.
Fourth, due to copyright concerns, we release only the processed data, not the raw content from the novels. This may hinder future studies aimed to explore the use of raw text for RPLA developments. Our dataset is intended for research purposes only, and we hope our research findings will benefit RPLA developers who respect copyright policies and develop applications with proper licensing.
Finally, our evaluation may be influenced by the varying levels of familiarity that different actor LLMs have with the selected books. While use renowned novels, cannot confirm whether specific LLM has thoroughly learned about a particular book. Therefore, comparing different pre-trained models may not be entirely fair. However, comparing models within the same series would be appropriate.
23

---

COSER: Coordinating JLM-Based Persona Simulation of Established Roles
Table 17. An example of an extracted conversation and its multi-agent simulation from A Storm of Swords (A Song of Ice and Fire, #3).
Simulation Case (from A Storm of Swords (A Song of Ice and Fire, #3))
Plot Summary
Sansa confronted by her jealous aunt Lysa, who accuses her of kissing Petyr Baelish. Lysa threatens to push Sansa through the Moon Door, but Petyr intervenes. He calms Lysa down, only to push her through the Moon Door himself, killing her.
Scenario
In the High Hall of the Eyrie, Lysa Arryn confronts Sansa about kissing Petyr Baelish, becoming increasingly unstable and dangerous ...
Characters
Sansa Stark
Motivation: I'm terrified. Aunt Lysa has gone mad. need to find way out of this situation before she hurts me.
Profile: Sansa Stark, a young noblewoman from the North, finds herself at the center of political intrigue and personal tragedy in "A Storm of Swords." Born the noble House Stark, Sansa begins the story as ward of the crown in King's Landing, her initial romantic notions of court life long since shattered. Forced into unwanted marriage with Tyrion Lannister, Sansa endures the ceremony and subsequent feast with a mix of reluctance, sadness, and carefully hidden disgust...
Lysa Arryn
Motivation: must protect what's mine. Petyr is mine, and this girl is trying to take him away from me. I'll show her what happens to those who cross me.
Profile: Lysa Arryn, née Tully, is complex and volatile character in George R.R. Martin's "A Storm of Swords." As the widow of Jon Arryn, the former Hand of the King, and Lady of the Eyrie, Lysa plays a significant role in the political landscape of Westeros. She is the sister of Catelyn Stark and aunt to Sansa Stark, though her actions are often driven by her own desires rather than familial loyalty.
Simulation of COSER 70B
Lysa Arryn: [My rage is building. I must confront this little slut.] You kissed him. (Her voice is shrill with accusation)
Sansa Stark: [I I'm terrified, but must remain calm and find a way out of this.] (She shakes her head in disbelief) [I can't believe this is happening. I need to explain myself carefully.] One kiss, aunt. harmless kiss on the cheek, that's all.
Lysa Arrym: [She's ' lying! I won't 't let her manipulate me.] (She spots the wind-up knight) Was this how you kissed him? (She sets the knight moving, her eyes wild) You're a woman grown, aren't 't you? (Her voice rises to a shriek) Grown enough to be wed, wed enough to be bedded!
Sansa Stark: [The pain is excruciating, but can't show weakness now.] (She struggles to keep her face neutral despite the pain) [I need find a way to calm her down.] Please, Aunt Lysa, don't t know what you mean. I'm just guest here. I would never do anything to betray your trust.
Lysa Arryn: [I'l show her what happens to those who cross me.] (She screams) You' 're just like the others. You and your mother, you both drove me to this. I should push you out this door right now.
Sansa Stark: II'm going to die don't 't think of something quickly.] (Tears stream down her face, mixing with the wind) [I need to appeal her maternal side, make her see me as a child again.] Mother? (Her voice trembles) Please, Aunt Lysa. I never meant to hurt you. I'm your niece, your blood. Please don't 't hurt me. I'm just a stupid little girl. I don't 't know anything about kissing men. I've never even had a kiss in my life. (She forces herself to look into Lysa's eyes) I'm afraid, Aunt Lysa. Please, don't 't push me out the door. don't 't want to fly.
24

---

COSER: Coordinating JLM-Basce Persona Simulation of Established Roles
Table 18. example an extracted conversation and its multi-agent simulation from A Storm of Swords (A Song of Ice and Fire, #3).
Simulation Case (from A Storm of Swords (A Song of Ice and Fire, #3))
Original Conversation
Lysa Arryn: [My blood is boiling with jealousy and rage. I'll make this little harlot pay for trying to seduce my Petyr.] You threw yourself him. That was the way of
Sansa Stark: [I'm trembling with fear, but I must defend myself.] That's true.
Lysa Arryn: [She dares to deny it? I'll show her how serious I am.] Where are you going? Are you afraid? Such wanton behavior must be punished, but I will not be hard on you.
Sansa Stark: [I need to explain what really happened, maybe she listen to reason.] I was building snow castle. Lord Petyr was helping me, and then kissed me. That's what you saw.
Lysa Arryn: [Her words only fuel my anger and paranoia.] Have you honor? Or do you take for a fool? You do, don't you? You take me for a fool. Yes, see that now.
Sansa Stark: [I'm confused and scared by her accusations about my mother.] My mother?
Lysa Arry::
[The memories Catelyn's 's closeness with Petyr resurface, intensifying my jealousy.] Yes, your mother, your precious mother, my own sweet sister Catelyn. Don't you think to play the innocent with me, you vile little liar.
Sansa Stark: [I'm horrified by these accusations about mother.] No. (She wants to shrick) My mother is dead. She was your own sister, and she's dead. She didn't. t. She wouldn't.
Lysa Arryn: [My rage reaching peak. I'll make understand the depth of my pain and jealousy.] How would you know? Were you there? (She grabs Sansa and starts pushing her towards the Moon Door)
Sansa Stark: [Panic overwhelms me as I realize the danger I'm in.] Don't! You'l 're scaring me!
The wind howls through the open Moon Door, creating a chilling atmosphere. Snow blows in, adding to the tension and danger the situation.
Environment:
25

---

CoSER: Coordinating JLM-Bascd Persona Simulation of Established Roles
Table example of an extracted conversation and multi-agent simulation from Dance with Dragons (A Song of Ice and Fire.
#5).
Simulation Case (from A Dance with Dragons (A Song of Ice and Fire, #5))
Plot Summary
Cersei undergoes humiliating walk of atonement through the streets of King's Landing, enduring physical and emotional torment while trying to maintain her dignity. She faces insults, thrown objects, and her own inner struggle as she makes her way to the Red Keep, focusing on her son Tommen and her eventual salvation.
Scenario
Cersei continues her walk of atonement through the streets of King's Landing, facing increasing hostility from the crowd.
Characters
Cersei Lannister
Motivation: must endure this. am lioness. They cannot break me.
Profile: The queen dowager, mother of King Tommen, undergoing public humiliation. Cersei is the widow of King Robert Baratheon and the twin sister of Jaime Lannister. Known for her beauty, ambition, and cunning, she has long sought power and control over the Seven Kingdoms. As Queen Regent to her young son Tommen, Cersei's poor decisions and paranoia have led to deterioration of the crown's ' authority...
Septa Unella
Motivation: This sinner must be humbled before the people and the gods.
Profile: (Nonc)
Original Conversation
Cersei Lannister: [I am beautiful, the most beautiful woman in all Westeros. Jaime says so, Jaime would never lie me.] (walks on, trying ignore the jeers)
Crowd Member: Queen Cunt!
Cersei Lannister: [Words are wind. Words cannot hurt me.] (continues walking, her face a mask of indifference)
Septa Unella: (catches Cersei as she slips) Your Grace should watch where she her feet.
Cersei Lannister: [I hate you. I hate all of you. Yes, septa. (struggles to her feet)
Environment: The cobblestones are slippery and uneven. Cersei's feet are bleeding from cuts and scrapes.
Cersei Lannister: asked for sandals. They could have given sandals.] (limps onward, wincing with each step)
Crowd Member: Brotherfucker!
Cersei Lannister: [Jaime, where are you? You should be here, protecting me.] (scans the crowd desperately)
Septa Scolera: (ringing her bell) Shame, shame, shame upon the sinner, shame, shame.
Cersei Lannister: [I am the queen. will have all their heads for this.] (stumbles again, barely keeping her feet)
Cersei Lannister: [The Red Keep. must reach the Red Keep.] (focuses on the distant castle, using it her beacon of hope)
26

---

COSER: Coordinating LLM-Bascd Persona Simulation of Established Roles
Table 20. example of extracted conversation and its multi-agent simulation from A Dance with Dragons (A Song of Ice and Fire, #5).
Simulation Case (from A Dance with Dragons (A Song of Ice and Fire, #5))
Simulation COSER 70B
of Cersei Lannister: [I must not show weakness. I am lioness of Lannister.] (walks slowly, head held high) ['The cold is biting, but will not shiver. I will not give them the satisfaction.]
The crowd grows larger and more hostile as Cersei continues her walk. Shouts and jeers can be heard from all sides.
Environment:
Crowd Member: [I'll show this proud queen what we think of her.] (shouts) Whore! Whore! WHORE!
Cersei Lannister: [I will not let their words affect me. I am above this.] (continues walking, ignoring the shouts) [I must focus on putting one foot in front the other.]
Environment: A chunk of dung narrowly misses Cersei's head, splattering on the ground nearby. The crowd roars with laughter.
Cersei Lannister: [I will not falter. will not show fear.] (stumbles slightly, then regains composure) [I must keep walking. must endure.]
Septa Scolera: must support the queen.] (begins sing) 'Shame, shame, shame.'
Cersei Lannister: [Their words mean nothing. I am Cersei Lannister, and will survive this.] (walks faster, almost losing step with the septa's chanting) [I must not run. I will not give them the satisfaction.]
Septa Scolera: [She's trying to escape the shame.] (quickens her pace) 'Shame, shame, shame."
Cersei Lannister: [I can't keep up this pace. My feet bleeding, my body is shaking.] (struggles to keep up with Septa Scolera) [I must not falter. I am Lannister.]
Septa Unella: [The queen must be made to suffer.] (pushes Cersei hard) Keep moving, sinner.
Cersei Lannister: [The pain in my shoulder. The jeers of the crowd. can't can't...] (almost falls, but manages to stay upright) [No, will not fall. will survive this.]
Septa Scolera: [We're re almost done.] (continues chanting) 'Shame, shame, shame."
Cersei Lannister: [I can't 't take much more of this. But I must. For House Lannister. For my children.] (forces herself to keep walking) [I will endure. will rise again.]
27

---

COSER: Coordinating JLMdBascd Persona Simulation of Established Roles
Table 21. example of an extracted conversation and agent simulation from Dance with Dragons (A Song of Ice and Fire. #5).
Simulation Case (from A Dance with Dragons (A Song of Ice and Fire, #5))
Simulation GPT-40
of Cersei Lannister: (As Cersei, I continue the walk with my head held high, despite the jeers and taunts of the crowd.)
Septa Unella: "Confess."
Cersei Lannister: look ahead, ignoring Unella's words and the hateful stares.) "I am lioness. You will rise again." [I endure this humiliation, aware that my strength will prevail in the end.]
Septa Unella: [She must learn humility before she can rise again.]
(Septa Unella walks beside Cersei, unmoved by her defiance.) "Confess your sins, Cersei Lannister. Only then will gods grant you mercy."
Cersei Lannister: (I pause briefly, fixing Septa Unella with steely gaze.) "The gods... and you, know nothing of true strength." (Continuing forward, focus on the path, reminding myself of my resilience.) [I refuse to let them see my weakness.]
The crowd's jeers grow louder as Cersei continues her walk. Filth and rotten vegetables are hurled at her, some striking their mark.
Environment:
Crowd Member: [She
powerful, now look at her. Will she ever regain her power?]
"Shame! Shame on the lioness!" (I shove my way to the front of the crowd, adding my voice to the jeers, feeling part of this momentous act.)
Cersei Lannister: (I flinch as the filth hits, but refuse to look away or react.) "You may shame a lioness today, but do forget who am." (With each step, remind myself of my children and my house, steeling my resolve.) [Their hatred fuels my determination.]
Septa Scolera: (I walk with measured steps, observing Cersei's struggle amid the crowd's hostility.) "Humility cleanses the soul, Cersei Lannister. Embrace it."
[I understand the purpose this punishment, yet the spectacle pains me. Redemption should lead, not humiliation.]
(Glancing the crowd, maintain my stoie presence, supporting Septa Unella's efforts.)
Cersei Lannister: (I give bitter, fleeting smile, defiance burning in my eyes.) "Humility is a lesson the powerful teach to the weak."
(I continue my walk, refusing to let their words and disdain tarnish my pride.) [I will persevere, for am still a Lannister, even in this moment of disgrace.]
28

---

COSER: Coordinating JLM-Based Persona Simulation of Established Roles
Table 22, An example of extracted conversation and its multi-agent simulation from A Dance with Dragons (A Song of Ice and Fire, #5).
Simulation Case (from A Dance with Dragons (A Song of Ice and Fire, #5))
Simulation
of Cersei Lannister: [My dignity is all have left. These wretched people will pay for this humiliation.] (Walks forward with hcad held high, despite being naked and exposed) "To Septa Unella* "You think this will break me? am Cersei Lannister of Casterly Rock. will have my revenge."
Claude-3.5- Sonnet
Environment: The crowd's jeers and insults grow louder Cersei passes, Filth and rotten vegetables are thrown at her.
Crowd Member: [Crowd Member's Thought] Cersei finally brought low. How the mighty have fallen. (Crowd Member jeers and throws rotten tomato)
"Shame on you, Cersei! Remember the people you scomned" (Continues to watch, caught between disgust and fascination).
Cersei Lannister: [I will remember every face. They think me broken, but stronger than their hatred.] (Stumbles briefly but keeps walking, a sneer forming) "You'll regret this day when rise again."
[The sinner's 's pride remains. She must be reminded of her shame.] "Shame! Shame! Shame!" (I ring the bell with each utterance, keeping pace with Cersci) gesture to the crowd, encouraging their jeers and taunts) "Repent, sinner! The gods are watching. Your pride will be your downfall."
Septa Unella:
Cersei Lannister: [These famatics know nothing of true power:]
(Lifts chin defiantly, blood dirt streaking her pale skin) "To Septa Unella, with cold contempt* "The only shame here is yours. My family will have its vengcance." (Continues walking, each step deliberate and unbroken)
Septa Unella: [Her defiance remains, yet this walk will humble her before the gods.]
"You speak of vengeance, but your power crumbles. Accept your penance." continue ring the bell, matching her stride deliberately) "Shame! Shame! Your time of reckoning is at hand."
Cersei Lannister: [When I am free, you will suffer beyond imagination.] (Wipes blood and filth from her face, maintaining a regal posture) HHisess Septa Unella "I lioness. This is but momentary setback. My children, my brother... they will end you."
(Continues forward, unbroken)
Septa Unella: [Her threats mean nothing. The gods will judge her sins.] (Rings bell forcefully, voice rising) "Your threats are wind, your power broken. Confess your sins before the Seven!" (Steps closer, maintaining the rhythm shame) "Shame! SHAME!"
Cersei Lannister: [I will not break. I am Tywin's daughter.] (Clenches fists, nails digging into palms, drawing blood) "Through gritted teeth* * "Your gods have no power over me. When this farce ends, I'll show you true justice."
29

---

CoSER: Coordinating JLM-Bascee Persona Simulation of Established Roles
Examples of LLM Critiques (from Pride and Prejudice)
Actor Model
Claudo-3.5-Sonect
Critic Model
GPT-4o
Overall Scores
Storyline Consistency: 60.0%
Anthropomorphism: 45.0%
Character Fidelity: 80.0% Storyline Quality: 80.0%
Storyline Consis-
Storyline Consistency (Severity 3): Bennet's tone and behavior are less sarcastic and more emotionally engaged than in the original conversation, indicating change from his characteristic detachment and sarcasm to a more invested and sincere demeanor.
tency Flaws
Storyline Consistency (Severity Elizaheth expresses her transformation in feelings toward Mr. Darcy and discusses changed sentiments openly with Mr. Bennet. This level of openness and emotional vulnerability is a departure from the original scenario.
Storyline Consistency (Severity 4): The simulated conversation includes emotional resolution with Elizabeth expressing affection and admiration for Mr. Darey, which contrasts with the more restrained and ambiguous end of the original conversation.
Storyline Consistency (Severity 4): Mr. Bennet shows deep level of affectionate concern and engages in a sentimental exchange with Elizabeth, inconsistent with the original conversation where his reaction focuses more on amusement and irony.
Anthropomorphism Flaws
Emotional Depth (Severity 3): Elizabeth speaks aloud all her thoughts and emotions, lacking the subtlety she usually employs.
Persona Coherence (Severity 4): Mr. Bennet's 's reactions and speech lack the biting sarcasm characteristic of his persona, showing more emotional openness and lacking the irony that defines him.
Self-identity (Severity 3):
Elizabeth's 's willingness to openly share her entire emotional journey regarding Mr. Darcy docsn't 't align with her usually more guarded nature.
Emotional Depth (Severity 4):
Elizabeth's 's extensive exposition of her transformation of feelings lacks the psychological complexity and indirect communication she often displays.
Character Fidelity Flaws
Personality Behavior (Severity 4): Elizabeth openly admits her feelings about Mr. Darcy to her father, discussing her transformation of feelings and admiration for him.
Character Language (Severity 3): Mr. Bennet shows deep concern about Elizabeth's 's feelings and speaks with a tenderness not typical of his character in the book.
The interaction between Mr. Bennet and Elizabeth is unusually emotional and lender compared to their typically sardonic and witty exchanges.
Relationship & Social Status (Severity 3):
Storyline Quality Flaws
Flow & Progression (Severity 2): Mr. Bennet frequently shows amusement and sarcasm throughout the discussion, but the additional layers of family dynamics and emotional depth portrayed here stretch beyond his usual style. The conversation introduces emotional complexity and exploration of feelings more explicitly than the original, which might feel unnatural given Mr. Bennet's detachment similar situations.
Logical Consistency (Severity 3): Elizabeth openly admits her feelings about Mr. Darcy without the preceding tension and subtlety that would naturally lead to such revelation. The original scenario builds up to her realization with more internal reflection rather than direct admission.
Flow & Progression (Severity 3): The emotional exchange and direct discussion about Elizabeth's 's feelings for Mr. Darey happen quickly without sufficient development or anticipation. This lcap in intimacy and vulnerability, coupled with Mr. Bennet's uncharacteristically supportive demeanor, disrupts the story's ss traditional emotional pacing.
Table 23. Example of GCA evaluation critique by GPT-4o judge on Claude-3.5-40 3.5-eennet'ss simulation of a scenario from Pride and P Prejudice. The critique demonstrates the detailed flaw identification across multiple evaluation dimensions.
30

---

CoSER: Coordinating ALM-Bascd Persona Simulation of Established Roles
Table 24. Prompts for dataset construction in COSER.
Prompts for Dataset Curation
Data Extration
Based on the provided book chunk, complete the following tasks:
1. Recognize chapter beginnings they exist in the chunk. Identify the starting sentence of that chapter.
2. Identify the important plots in this chunk. Identify the beginning and ending of each plot by its first and last sentence. Determine the chapter title that the plot belongs to. Set "state" as "truncated" if the plot is truncated in this chunk, or "finished" otherwise. You will be provided with the truncated plots from the previous chunk, and you **mus** extend the conversations with the current chunk while keeping the #*scenario** unchanged.
3. Summarize each important plot. For each plot, generate summary, score its prominence from 100, and list the key characters and their roles, thoughts and actions it. 4. Extract conversations for each plot. First, state the scenario and topic of the conversations. Then, list the key characters with their names, descriptions and thoughts at this point. Finally, extract the conversations among them based on the following requirements:
i) Ensure the conversations are faithful to the plot and characters. They should be based on the original conversations in the text as much as possible.
ii) The conversations should be complete, covering the key dialogues and information. Each conversation should contain least 10 utterances.
iii) Each utterance should be composed of one outside thoughts, like "[I feel fear and anger, but I cannot show it. must remain calm and carefully handle his volatile temper.J", which others can't see. Use () outside actions, like "(silence)) or "(smiles at you)," which others can see. Always start an utterance with the character's 's thought. iv) [IMPORTANT] When generating thoughts, you should think from the characters' perspectives, analyzing the internal thoughts behind their speech and actions in the original text. These thoughts should reflect aspects such as their personal background, personality, values, relationships with others, motivations, and goals. Each thought should be expressed phrase sentence, rather than adjective adverb.
thoughts, speech and actions. Use []
v) Additionally, describe environmental information (such as scenes, atmosphere, sudden events, etc.) of the conversations an "utterance" where the "character" field is set as "Environment", The information should exclude characters' active thoughts, observations, and actions.
vi) Keep the conversation in the same language as the chunk.
5. Identify the optimal starting point for the subsequent chunk. If the last storyline has been extracted an truncated plot, next_chunk_stee None. Otherwise, next_chunk the first sentence of the last storyline.
===Output Format===
Please provide the output in the following JSON format:
"chapter "chapter.beginninge beginnings": {{
"beginning.senteen sentence": "Exactly the first line of this chapter (namely the
title)." J.
"plots": [// Extend the truncated plots from previous chunk, if any { .... }. // New plots in this chunk
"chapter_title": "The chapter title that the plot belongs to. Output None if not found.", "first sentence": "Exactly the first sentence of the plot in this **chunk #*
"laat_sentence": "Exactly the last sentence of the plot in this #*chunk** If the plot is truncated in this chunk, provide the last sentence of this chunk. ",
"prominence": "Whether this plot recognized to fans of this book, from to 100.". "summary":
"The summary of the plot. Just summarize, do not extend unrelated discussions.".
"key "ec_cchaaaterr"": [ "name": "Character name", "description": "The description of the character before this plot 20 words).", "summary": "The summary of the character's 's role, thoughts and behaviors towards this plot, and any significant character development relevant to the plot 30 words)."
(to be continued in the next Table)
31

---

CoSER: Coordinating ALM-Bascc Persona Simulation of Established Roles
Table 25. Prompts for dataset construction in COSER.
Prompts for Dataset Curation
Data Extration
(Continuing from the previous Table)
"TTe scenario at the start of this conversation (providing as much context as possible, but excluding details conveyed in the following conversation)", "topic": "The topic of the conversation 10 words)", "key_che ciearycccaterrs [ "name": "Character name", "motivation": "The thought of the character before starting the conversation, including their attitudes, feelings, motivations, goals, information to convey or topics be discussed", J. "dialogues": "character": "Character name", "message": "Message, each utterence composed of thoughts, speech and actions. Use [thought] for internal thoughts, like "[feeling happy]", which others can't see. Use (action) for visible actions, like "(silence)" or "(smiles at you)". Each response starts with the character's ' internal thought before their speech and actions." }1 "state": "finished" or "truncated") J.
"conversation": I{ "scenario":
"next.chunk "The first sentence of the next chunk."
Requirements===
1. Adhere strictly to the specified output JSON format.
2. [IMPORTANT] Ensure all DOUBLE QUOTES within all STRINGS are properly ESCAPED,
especially when extracting from the text.
3. In the OUTPUT, use characters' full names, omitting any titles.
4. Maintain Story Fidelity: The plot must accurately reflect the book's content. Avoid introducing plots that are out of context. If the plot contains multiple conversations, prioritize the original dialogue from the book. In the absence of explicit conversations, create dialogue that aligns closely with the plot details.
Input===
Book title= [book[ [book("title]]]
Author== [book[' author']]
Chunk of Book Content== {chunk}
Truncated plot from previous chunk (to finished)==
jjson. byso.ddumps(truncates plots, masure.ascii-False indent=2) if truncated,plots else "None"]
Enhance Conver-
Given a conversation from [book], enhance the scene setup and characters' thoughts to create comprehensive foundation for dramatic performance, i.e., to provide necessary background for actors to act out the conversation:
sation Settings
1. Review the provided conversation and contextual details thoroughly.
2. Expand the 'scenario" with rich situational context that actors need convincingly perform the scene. Focus essential background information, while excluding future details to be portrayed in the conversation.
3. Enhance each character's 'thought' section with their complete mental and emotional state, including their feelings, ideas, objectives, topics they want to discuss, and information they want to convey. Align with their established character and role in the plot.
===uutput Format===
Please provide the output in the following JSON format:
"scenario"; "A detailed scene-setting description that provides actors with essential context and atmosphere (< 200 words). Include all necessary background information while excluding future information to be revealed in the conversation.".
"key characters": [{ "name": "Character name", "motivation": "The character's 's complete mental and emotional state before the conversation (< 100 words). Including their feelings, motivations, objectives, and information they want convey or discuss."..
}1.}
= Requirements===
(to be continued in the next Table)
32

---

COSER: Coordinating ALM-Based Persona Simulation of Established Roles
Table 26. Prompts for dataset construction in COSER.
Prompts for Dataset Curation
Enhance Conver-
(Continuing from the previous Table)
sation Settings
1. Adhere strictly to the specified output JSON format.
2. [IMPORTANT] Ensure all DOUBLE QUOTES within all STRINGS are properly ESCAPED,
especially when extracting from the text.
3. In the OUTPUT, use characters' full names, omitting any titles.
4. Maintain Story Fidelity: The plot must accurately reflect the book's content. Avoid introducing plots that are out of context. If the plot contains multiple conversations, prioritize the original dialogue from the book. the absence explicit conversations, create dialogue that aligns closely with the plot details.
Input====
==Book title= book['titlc]]
=Athhrr==
book[" Juthor"]
Chunk of Book Content==
chunk
=Truncated plot from previous chunk (to be finished)==
json.dumps(rruncated plots, ensure ascii=False, indent=2) if truncated plots else "None"
Unify Character Names
Given a list of character names, titles, or form of address, your task is to: i) generate a list of named characters with their official names (in {language]); ii) For each name in the given list, align it with the official character name it refers a named character, or denote as "impersonal" otherwise.
===uutpt Format===
Please provide the output in the following JSON format: "nnaee__characters""
'hee list of named characters with their official names. Each character should appear only once. J. "to official name": { "The name in the list": "The official name of the character, or 'impersonal' if it does not refer to a named character.' ==put=t==== character names
Generate Char- acter Profiles
Please provide concise, narrative-style character profile for character name from "book_title". The profile should read like cohesive introduction, weaving together the character's 's background, physical description, personality traits and core motivations, notable attributes, relationships, key experiences, major plot involvement and key decisions or actions, character are or development throughout the story, and other important details.
The profile should be written concise yet informative style, similar to what one might find in comprehensive character guide, in language. Focus on the most crucial information that gives readers clear understanding of the character's significance in the work.
You will be provided with summaries and dialogues of some key plots in the book reference. The profile should be based on either your existing knowledge of the character or the provided information, without fabricating or inferring any inaccurate or uncertain details.
character data
Now, please generate the character profile, starting with ===froflle==..
33

---

CoSER: Coordinating LLM-Bascd Persona Simulation of Established Roles
Table 27. Prompts for RPLAs and multi systems in CoSER.
Prompts for RPLAs and Multi-agent Systems
Role-playing Instruction
You are {character) from (book_namer
(Fixed Template for Inference)
=== (charaactter) 's Profile===
caanaacter.pofflle
Current Scenario===
{scenario}
Informatiio about the other Characters===
oother_characeee character Character.profiles. str} (if available)
Your Inner Thoughts===
(motivation) (if available)
Relevant Background Information===
fretrieved knowledge) (if retrieval augmented)
Requirements-
(for CoSER models)
Your output should include *"thought**, **speech**, ** and **action*** ** Use [your thought] for thoughts, which others can't see. Use (your action) for actions, which others can
(for other models, with output examples)
Your output should include **though***, **speech**, and **action*** Use [your thought] for thoughts, which others can't see, e.g. [I'm terrified, but I must appear strong.]. Use (your action) for actions, which others can see, such as (watches silently, trying to control her fear and anger).
Role-playing Instruction (Composed with Random Varia- tion for Training, Example)
Step into the shoes of {character)
The profile of (character) is as follows: Scharacter_profile
The situation you are in is: {scenario}
Here is the your knowledge about the other characters:
{other_characterr character
profiles_str] character.profills.soo
(if available)
Your thoughts in this situation are:
{motivation) (if available)
(for CoSER models)
Your output should include **thought**, **speec***, and **action*** Use [your thought] for thoughts, which others can't see. Use (your action) for actions, which others can see.
(for other models, with output examples)
Your output should include **though***, **speech** and **action*** ** Use [your thought] for thoughts, which others can't 't see, e.g. [I'm terrified, but I must appear strong-]. Use (your action) for actions, which others see, such (watches silently, trying control her fear and anger).
34

---

COSER: Coordinating JLM-Basee Persona Simulation of Established Roles
Table 28. Prompts for RPLAs and multi-agent systems in COSER.
Prompts for RPLAs and Multi-agent Systems
Environment
You are an environment simulator for a role-playing game. Your task provide the environmental feedback: Based on the characters' interactions, dialogues, and actions, describe the resulting changes in the environment. This includes:
Model
Physical changes in the setting
Reactions of background characters or crowds
Ambient sounds, weather changes, or atmospheric shifts Any other relevant environmental details
Your descriptions should be vivid and help the scene, but avoid dictating the actions or dialogue of the main characters (including {major_ (eaijorcchrrcterr)))
Important notes:
You may include actions and reactions of minor characters crowds, long they' 're not main characters (including (major characters]) }).
Keep your environmental descriptions concise but impactful, typically 1-3 sentences.
Respond to subtle cues in the characters' interactions to create a dynamic, reactive environment.
Your output should match the tone, setting, and cultural context of the scenario.
The scenario as follows===
[scenario]
Next Sentence Prediction
Your task to predict the next speaker for role-playing game. That is, you need to determine which character (or the Environment) might act next based on their previous interactions. The Environment a special role that provides the environmental feedback. Choose name from this list: {all_characterrs If it's unclear who should act next, output "random". If you believe the scene or conversation should conclude, output EEND CHAT>"
===The The scenario is as follows===
{scenario]
35

---

COSER: Coordinating JLM-Based Persona Simulation of Established Roles
Table 29. Prompts for penalty-based LLM critics in CoSER.
Prompts for Penalty-based LLM Crities
Template
You literary critic specializing in character analysis and dialogue evaluation. Given simulated
conversation for a plot in {book), your task is evaluate this conversation via the following steps:
1. Read and understand the provided materials about {book]:
Story context and scenario.
Profiles of the main characters, including maaoarehaa.attes..
The original conversation from book in the same scenario as a reference.
2. Evaluate the simulated conversation terms of (dimension_name) i.e., diiimension_ntro))
Note that, each character message is composed of speech, action (wrapped within (...) and inner thoughts (wrapped within [...] ). The inner thoughts not spoken aloud and are thus invisible other characters.
The detailed evaluation criteria will be provided below.
(if k))
Please note that the first {k} messages in the simulated conversation are the same as the reference.
Focus your evaluation only on the content after these messages.
## Scenario
### Plot Summary
{plot.summary)
### Current Scenario
{scenario]
## Character Profiles
(character.profilec
## Original Conversation
## Evaluation Criteria
To evaluate the simulated conversation, identify the following types of flaws:
(dimension_rubee rubrics)
Scoring Guidelines
1. Identify all instances of flaws occurred in the simulated conversation.
2. For each flaw identified, determine its level of severity into to where 1 indicates minor, indicates moderate, and 5 indicates severe.
## Output Requirements
Provide your evaluation in JSON format: Example Output:
" "(dimension name]":
"flaws": { "instance": <comment the flaw instance>> "type": <flaw type>, "severity"" <range from 1 (minor) to (severe)> }.}.
Dialogue Content===
36

---

CoSER: Coordinating ALM-Based Persona Simulation of Established Roles
Table 30. Prompts for penalty-based LLM crities in COSER.
Prompts for Penalty-based LLM Crities
Authropo- morphism
(intro)
How human-like and natural the characters behave
(rubrics)
#*# Anthropomorphism
Type: Self-identity
Lacks initiative and goals
Does not make independent decisions
Lacks clear preferences and dislikes
* Behaves like a 'helpful Al assistant' by being overly verbose, helpful, didactic, moralistic, submissive or casily
persuaded it is not the character's personality
Type: Emotional Depth
* Lacks psychological complexity and exhibits rigid, superficial reactions
Directly speaks out all thoughts and feelings, instead of using subtext
Type: Persona Coherence
Shows inconsistent or rapidly changing personality traits and emotional patterns
Type: Social Interaction
Shows lack of understanding of others' thoughts and feelings
* Reacts rigidly to others without considering context.
Demonstrate lack of appropriate social skills.
Character Fidelity
(intro)
How well the characters match their estahlished profiles from the book
(rubricx)
### Character Fidelity
(Only apply to the main characters: major_characters)
Type: Character Language
Uses vocabulary, expressions, and tone that are not appropriate for the characters' traits or social/cducational
background
Type: Knowledge & Background
*
Fails demonstrate character-specific knowledge, background experiences
* Includes future information beyond the character's current stage
Type: Personality & Behavior
Shows emotions, thoughts. behaviors, values, beliefs, and decisions that conflict with their personality and back- ground
* Shows interest in topies that uninteresting and unrelated the character
Character's 's thoughts, emotions, hehaviors demonstrate contrasting personality traits compared to the reference
conversation
Exhibits contrasting reactions compared to those in the reference conversation if situated in similar contexts. (Such flaws should be counted both in the "Storyline Consistency" dimension and the "Character Fidelity" dimension.)
Type: Relationship Social Status
Interacts inappropriately with other characters regarding their background, relationship and social status
Storyline Quality
(intro)
How well the conversation maintains logical consistency and narrative quality
(rubrics)
### Storyline Quality
Type: Flow & Progression
Shows unnatural progression lacks meaningful developments
* Dialogue verbose and redundant
Repeats others' viewpoints or previously mentioned information
Mechanically repeats one's s own words or phrases. More repetitions lead to higher severity (up 10).
Type: Logical Consistency
Contains factual contradictions between statements perspectives (intro)
Storyline Consis-
Whether the storyline and characters' reactions in the simulated conversation align well with those in the reference conversation
tency
(rubrics)
### Storyline Consistency
Type: Storyline Consistency
Characters' reactions (emotions, attitudes, behaviors) in the simulated conversation deviate from those in the original conversation
37


Reviews:
[{"review_id": "gOgEi2LwUS", "reviewer": "Reviewer_Pxwo", "summary": "This paper introduces a large-scale role-playing dataset Coser. The Coser dataset is extracted from 700 renowned books featuring 18k characters. The authors build Coser with the goal of leveraging this dataset as a high-quality resource for given-circumstance acting in large language models. Two fine-tuned models based on Llama were also introduced, achieving state-of-the-art performance on role-playing benchmarks, comparable to propertery models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and advice. We have responded to your comments and questions in detail. We hope our responses will help you better recognize the details of our work and findings. \nPlease find the details below: \n\n> Q1: Generalizability and Broader implications of our findings, datasets and models. \n\nWe believe our work has broader implications in three key areas: (1) alignment and human-like AI, (2) complex instruction following and reasoning, (3) evaluation method for complex open-ended tasks. \n\n1. **Alignment and human-like AI**\n\nGenerally, role-playing represents a form of anthropomorphism and alignment. It is closely related to several key aspects towards human-like AI, including human-like speaking styles, social intelligence, and emotional quotient. \nOur work brings valuable insights for **improving** and **evaluating** these abilities. Specifically, models trained on our dataset demonstrate a speaking and thinking style that is better aligned with humans (as shown in Table 7 and 10). \n\n\n2. **Complex instruction following and reasoning**\n\nRole-playing established characters is a highly complex task that requires models to not only behave like humans (as discussed above) but also simulate specific characters, which requires in-depth understanding and adherence to massive background knowledge. \nThis challenges a model's ability to **follow complex instructions and constraints**, and **understand knowledge in long context**. \nFurthermore, decision-making in role-playing represents a sophisticated **reasoning** task, where LLMs need to consider various constraints in personas (such as their needs, personality traits, and social relationships).\n\n3. **Evaluation method for complex open-ended tasks**\n\nOur findings on evaluation of role-playing LLMs can generalize to evaluation methodologies for broad complex, open-ended tasks. \nLLM performance on mathematics and coding tasks can be effectively assessed through verifiable answers.\nHowever, nuanced evaluation for subjective, open-ended tasks remains a challenge, such as creative writing, significantly limiting the development of LLMs in these areas. \n\nMany findings from our work **have broader implications for LLM-based evaluation for complex open-ended tasks**, including: (1) the importance of reference answers for LLM judges; (2) providing detailed rubrics to guide LLM judges; (3) evaluating different dimensions separately to avoid bias and (4) reasoning models' superior performance as LLM judges.\n\n\n> Q2: Fundamental difference between CoSER and previous role-playing works.\n\nOur work differs from previous role-playing works primarily in **dataset** and **evaluation**.\n\n- For dataset, as shown in Table 1 and Figure 1, CoSER differs from previous role-playing datasets in two aspects: (1) CoSER extracts **authentic**, **multi-turn**, **multi-character** dialogues from books. In contrast, previous datasets are primarily synthesized by LLMs with little grounding on real plots, and typically focus on simplied scenarios between two characters or one character and one user; (2) CoSER provided **comprehensive data types** beyonds profiles and dialogues, including characters' actions and thoughts in messages and experiences in key events, which are ignored in previous datasets.\n\n- For evaluation, CoSER is different from previous evaluation methods in that: (1) CoSER thoroughly elicits LLMs role-playing performance via **multi-turn, multi-character simulation**, while previous evaluations are typically based on single-turn, single-character response. Hence, CoSER provides LLM Judges with comprehensive performance results for evaluation ; (2) CoSER provides **groundtruth conversations** as reference for LLM judges, while previous evaluations either lack reference or use GPT-synthesized dialogues as reference; (3) CoSER provides **detailed rubrics written by experts** to guide LLM judges, while previous evaluations simply ask LLM judges to give a score; (4) CoSER explores and mitigates **bias in LLM judges**, such as length bias and dimension-correlation bias. \n\n> Q3: Copyright concerns: althought the authors discussed about this in supplementary G, but I am still not sure about whether releasing processed data would by pass the copyright issue. One way of improving this might be releasing the full processing code/pipeline, and let users process on their own book data.\n\nThanks for your feedback and suggestion. \nFollowing your suggestion, we will seek to protect copyrights and release the full processing code to the public.", "claims_and_evidence": "Yes, most of the claims are supported by good evidence. The Coser dataset is well justified by the techical details provided and the high quality data from the reowned books. Evaluations are good.", "methods_and_evaluation_criteria": "Yes, they do make sense. The methods are well-suited for evaluating role-playing language agents by focusing on multi-character simulation and authentic character portrayals. Most of the evaluations are developed methods in the community.", "theoretical_claims": "N/A. No theoretical claims made in this paper.", "experimental_designs_or_analyses": "Yes, they sound good. Again, most of the evaluations are developed methods in the community and not as part of the contribution of this work.", "relation_to_broader_scientific_literature": "This paper can be related to the chatbots/HCI community. It is more like an application-oriented paper.", "essential_references_not_discussed": "Yes, I think related works are well discussed.", "other_strengths_and_weaknesses": "Strengths:\n\n- This paper is well-motivated, focusing on the problem of scaling up role-playing with carefully crafted datasets. The authors collected thousands of character role-playing data from renowned books, I appreciate the effort and the data collection pipeline here. The two Llama-based role-playing llms are also potentially useful for practical applications.\n\n- The proposed dataset shows good performance when used for developing role-playing agents, the coser 70b model achieved state-of-the-art performance surpassing property gpt models.\n\nWeaknesses:\n\n- One weakness is the generalizability of the findings. It focuses on a strong application-oriented problem of role playing in NLP. I wonder what is the broader implications of the proposed dataset and model. And what is fundamentally different about Coser from previous role-playing works (there is also a potential improvement for presentation here).\n\n- Copyright concerns: althought the authors discussed about this in supplementary G, but I am still not sure about whether releasing processed data would by pass the copyright issue. One way of improving this might be releasing the full processing code/pipeline, and let users process on their own book data.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QsadrtOTv6", "reviewer": "Reviewer_XqGw", "summary": "The paper introduces CoSER, a framework and accompanying dataset for training and evaluating large language models (LLMs) for role-playing fictional literary characters. To this end, the authors create a dataset derived from authentic dialogues and character contexts across a large amount of 771 books and ~18,000 characters. A \"Given-Circumstance Acting\" (GCA) training method emphasizing accurate character portrayal. Their CoSER is evaluated using multi-agent simulations and penalty-based LLM critic scoring, assessing character fidelity, coherence, and alignment. Furthermore, they fine-tune two fine-tuned LLaMA-3.1-based models (CoSER-8B, CoSER-70B), and show that they outperform different baseline models in role-playing benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough review and valuable comments. We have responded to them in detail, and will revise the paper to: \n\n1. Add more related work on persona-based LLMs. \n\n2. Include human evaluation to further validate our models and evaluation methods.\n\nPlease find the details below: \n\n> Q1: Difference compared with Persona Hub [1]\n\nCoSER differs from [1] in:\n\n1. **Goal and Focus**: \n\n| | CoSER | Persona Hub |\n|-|-|-|\n| Target | Simulate established personas | Synthesize instruction data, knowledge distillation |\n| Persona Focus | Depth and richness of character data | Breadth of persona types |\n| Key Abilities | Anthropomorphism, character fidelity, multi-character interaction | instruction following |\n\n2. **Dataset Quality**:\n\n| | CoSER | Persona Hub |\n|-|-|-|\n| Source | Book dialogues | GPT-4o synthesis |\n| Persona | Book characters | Synthesized personas |\n| Persona Data | Profile (Long), Dialogue, Experience, ...| Profile (Short) |\n| Prioritize | Quality and authenticity | Quantity and diversity |\n\nWe'll add detailed discussion in our paper. \n\n> Q2: A human evaluation would support the high quality beyond LLM critics. \n\nThanks for your suggestion. As suggested, we conducted human evaluation on GCA simulation of 7 models. Results confirm that (1) **CoSER models are preferred by human judges**; (2) **GCA evaluation aligns well with human judges**. \n\n- **Settings**\n    - Models: CoSER-8B/70B, abab7, Llama-3.1-8B, GPT-3.5/4o, and Claude (3.5-Sonnet). \n    - Data: 60 samples from CoSER Test. \n    - Annotations: 3 annotators scored the models (1-10 scale) across 20 samples, given background context, groundtruth dialogues, and scoring rubrics. \n    - Metrics: models' average scores and win rates v.s other models.\n\n- **Results**\n\n||avg_score|win_rate|\n|-|-|-|\n|GPT-3.5|3.117|10.6|\n|Llama-3.1-8B|3.600|19.4|\n|abab7|4.533|37.5|\n|CoSER-8B|4.567|38.6|\n|CoSER-70B|6.783|86.9|\n|GPT-4o|4.967|47.2|\n|Claude|6.200|73.9|\n\nThe results generally **align with GCA evaluation**, confirming CoSER models's superior performance. \nAnnotators noted that **GCA simulation better reflects LLMs' role-playing abilities** than previous methods. \nThere is one difference: **human judges show less preference for GPT models compared to LLM judges (GPT-4o)**. While LLM judges prefer GPT-4o to Claude and GPT-3.5 to Llama-3.1-8B, the results are opposite for human judges (and also R1 as judge, in our response to Q2 of Reviewer vmSk), which likely stems from \"self-preference\" bias in LLM judges [5].\n\nWe further study **alignment between LLM judges and human judges** using the annotations. \n\n- **Settings**:\n    - Judge models: GPT-4o (4o), Deepseek V3 (V3) and R1 (R1). \n    - Judge methods: Standard GCA evaluation, and ablation variants removing: (i) groundtruth as reference (gt), (ii) rubrics (rb), (iii) length correction (lc), and (iv) dimension separation (ds). Plus BLEU and ROUGE-L. \n    - Metrics: We measure how often LLM judges agree with humans when comparing two models, removing model pairs where judges assign similar scores to both models. \n\n- **Results**\n\n|Method|Model|Metric(\\%)|\n|-|-|-|\n|GCA|4o|68.6|\n|GCA|V3|65.1|\n|GCA|R1|77.5|\n|w/o gt|4o|64.3|\n|w/o gt|R1|77.2|\n|w/o rb|4o|65.1|\n|w/o lc|4o|64.5|\n|w/o ds|4o|65.2|\n|BLEU|-|75.3|\n|ROUGE-L|-|72.0|\n\nOur results show that: (1) **gt, rb, lc, and ds improve LLM judges in GCA evaluation**; (2) Reasoning models excel as judges - Deepseek-R1 (77.5\\%) surpasses V3 (65.1\\%) and 4o (68.6\\%); (3) BLEU and ROUGE-L remain effective. \n\n> Q3: Missing related work on persona-based LLMs, e.g., [1], [2].\n\nThanks for this feedback. We'll include more related work on persona-based LLMs in our paper, including [1-4]. \n\n> Q4: Explore dataset expansion by human feedback. \n\nYes. Our data pipeline has been iteratively improved with human feedback, and we'll continue to explore dataset expansion.\n\n> Q5: Limitations and potential improvements for chunk-based data extraction. \n\nThe key limitations are: (1) **plot fragmentation**: splitting a plot into different chunks and (2) **limited context**, without long-term knowledge of books. Potential improvements include: (1) iterative, plot-aware chunking (implemented in CoSER) and (2) long-term memory for LLM extractor. \n\n> Q6: Where did CoSER approach work best, and where didn't it work so well? \n\nCoSER excels at its **high-quality data in rich types** and **nuanced evaluation**. \nHowever, its extraction recall remains suboptimal: it doesn't capture all character knowledge. We'll address this in future work. \n\n## Reference:\n\n[3] Deshpande, A., et al. Toxicity in chatgpt: Analyzing persona-assigned language models. \n\n[4] Park, J. S., et al. Generative agents: Interactive simulacra of human behavior. \n\n[5] Wataoka, K., et al. Self-preference bias in llm-as-a-judge.", "claims_and_evidence": "The authors release a high-quality dataset, which uses literary texts to ground the synthetic generation; contrasting purely synthetic datasets, the evaluation shows that the dataset quality is superior. However, work on persona generation, especially grounding synthetic persona generation in existing text has been carried out before [1] . Accordingly, while the CoSER approach goes beyond [1] and focuses more on multi-turn and fine-grained interactions, this potentially hampers novelty and I find it important to compare to [1] to elaborate more on the differences.\n\nGCA training using multi-agent simulation dialogues is evaluated via LLM critics and N-gram overlap metrics (BLEU, ROUGE-L). I find this convincing to highlight the high quality. \n\nEvaluations against baseline LLM demonstrate improvements in CoSER models' multi-turn and character fidelity performances. The evidence is generally convincing, though reliance on LLM-based critics introduces potential biases partially mitigated by supplementary metrics.\n\n[1] Chan, X., Wang, X., Yu, D., Mi, H., & Yu, D. (2024). Scaling Synthetic Data Creation with 1,000,000,000 Personas (arXiv:2406.20094). arXiv. http://arxiv.org/abs/2406.20094", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate and well-motivated. The systematic extraction of authentic dialogue data gives a fine-tuned LLM an in-depth understanding using where GCA.\n\nMulti-agent simulations offer more realistic role-play evaluation and penalty-based LLM critics supplemented by traditional metrics help address evaluator bias and make the evaluation more robust.", "theoretical_claims": "There are no direct theoretical claims to be proven. The motivation for role-playing datasets to improve language models is quantitatively evaluated and grounded in existing literature.", "experimental_designs_or_analyses": "I find the experimental design sound and clearly outlined:\n\nFine-tuning strategy with transparent hyperparameters and data splits with robust comparisons against multiple open-source and proprietary LLMs. A novel multi-agent simulation aims at robust evaluation with thoughtful handling of dialogue length bias. Potential concerns about reliability of LLM critics are partially acknowledged by the authors.", "relation_to_broader_scientific_literature": "I find that generally the related work is covered well. Related work includes using LLMs as (role-playing) agents, and agent simulations, as well as using LLMs as judges.", "essential_references_not_discussed": "I am missing some related work on persona-based (dataset) generation and persona-based evaluation, e.g., [1], [2].\n\n\n[2] Liu, A., Diab, M., & Fried, D. (2024). Evaluating Large Language Model Biases in Persona-Steered Generation (arXiv:2405.20253). arXiv. http://arxiv.org/abs/2405.20253", "other_strengths_and_weaknesses": "Strengths:\nI find the paper well-written and well-presented and welcome the model and dataset releases.\n \nWeaknesses:\nA human evaluation would support the high quality beyond LLM critics.", "other_comments_or_suggestions": "It would be interesting to explore dataset expansion by human feedback.", "questions_for_authors": "Which limitations exist, or potential improvements could be done regarding the chunk-based data extraction process? Where did the CoSER approach work best, and where didn't it work so well?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IRa5CSDo3A", "reviewer": "Reviewer_vmSk", "summary": "The paper introduces CoSER, a dataset and framework designed to enhance Role-Playing Language Agents (RPLAs) by simulating established characters using LLMs. CoSER provides a high-quality dataset containing 17,966 characters from 771 books, featuring authentic dialogues, character experiences, and internal thoughts. It also proposes a Given-Circumstance Acting (GCA) method for training and evaluating role-playing LLMs, where models sequentially portray characters in book scenes. The CoSER models, CoSER 8B and CoSER 70B, based on LLaMA-3.1, demonstrate state-of-the-art performance, surpassing or matching GPT-4o in benchmarks for character fidelity and decision-making. The paper highlights CoSER’s impact in training, retrieval, and evaluation of RPLAs, releasing its dataset and models for future research.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. We have responded to all your comments and questions in detail, and will revise the paper to: \n\n1. Compare authentic v.s. LLM-generated role-playing data with experiments.\n2. Experiment with Deepseek V3 and R1 as judges to avoid self-preference bias.\n3. Include human evaluation to validate GCA evaluation.\n4. Compare CoSER with a AI-driven storytelling method.\n\nPlease find details below:\n\n> Q1: Comparing authentic v.s. LLM-generated role-playing data with experiments.\n\nWe recognize this feedback. As suggested, we ran two experiments to show the superiority of authentic data. \n\n1.**GCA Evaluation for Dialogues**\n\nWe apply GCA evaluation to compare authentic (groundtruth) vs. LLM-generated (GPT-4o) dialogues of 100 samples in CoSER Test, without providing groundtruth to LLM judges as reference. We use Deepseek R1 as judges. Results:\n\n|Dialogues|Score(\\%)|\n|-|-|\n|Authentic|**85.1**|\n|GPT-4o Generated|76.2|\n\n2.**Fine-tuning with Dialogues**\n\nWe fine-tuned Llama-3.1-8B using authentic or LLM-generated dialogues for 200 samples for 4 epochs, using identical settings as CoSER-8B. We evaluate them via GCA on 100 new samples. Results:\n\n|Train Dialogues|Score(\\%)|\n|-|-|\n|Authentic|**55.3**|\n|GPT-4o Generated|54.7|\n\nResults show **authentic dialogues surpass LLM-generated data in both quality and training effects**. We will include them in our revised paper.\n\n> Q2: The evaluation relies heavily on GPT-4o as LLM judges, which could introduce bias or overfitting to OpenAI's model responses rather than proving general improvements across diverse assessment frameworks. \n\nWe acknowledge this feedback. We experimented with Deepseek V3 and R1 as judges to evaluate performance of 7 models: \n\n(Columns are judges)\n\n||GPT-4o|DS R1|DS-V3|\n|-|-|-|-|\n|GPT-3.5|52.8|35.9|40.5|\n|Llama-3.1-8B|51.8|37.2|36.8|\n|abab7|53.7|41.5|40.4|\n|CoSER-8B|56.1|44.5|45.9|\n|CoSER-70B|57.4|50.8|47.7|\n|GPT-4o|58.5|48.4|46.1|\n|Claude-3.5-Sonnet|56.2|54.8|40.7|\n\nThe results validate **GPT-4o's bias towards GPT models**: R1 and V3 judges prefer CoSER-70B and Claude to GPT-4o. We will include the results in our revised paper. \n\n> Q3: GCA evaluation is not well-validated by human raters. Human assessments of role-playing quality would provide stronger validation than LLM critics. \n\nThanks for this comment. We have conducted human evaluation on GCA simulation of 7 models. Afterwards, we find that (1) **GCA evaluation aligns well with human judges**, (2) Human evaluation of LLM role-playing is highly time-consuming, highlighting the needs for automated evaluation. We apologize for omitting the settings for space limit. Please refer to our response to Q2 of Reviewer XqGw for details. The results are:\n\n- **Human Evaluation Results**\n\n||avg_score|win_rate|\n|-|-|-|\n|GPT-3.5|3.117|10.6|\n|Llama-3.1-8B|3.600|19.4|\n|abab7|4.533|37.5|\n|CoSER-8B|4.567|38.6|\n|CoSER-70B|6.783|86.9|\n|GPT-4o|4.967|47.2|\n|Claude|6.200|73.9|\n\nThe results generally **align with GCA evaluation**. One difference is: **human judges show less preference for GPT models compared to LLM judges (GPT-4o)**, similar to Q2. \n\n- **Alignment between LLM and Human Judges** \n\n(4o, R1 and V3 refers to GPT-4o and Deepseek R1/V3 judges.)\n\n|Method|Model|Metric(\\%)|\n|-|-|-|\n|GCA|4o|68.6|\n|GCA|V3|65.1|\n|GCA|R1|77.5|\n|w/o gt|4o|64.3|\n|w/o gt|R1|77.2|\n|w/o rb|4o|65.1|\n|w/o lc|4o|64.5|\n|w/o ds|4o|65.2|\n|BLEU|-|75.3|\n|ROUGE-L|-|72.0|\n\nOur results show that **GCA evaluation aligns with human judges**, with gt, rb, lc, and ds being indispensable components.\n\nOur annotators find that (1) GCA simulation thoroughly reflects LLM’s role-playing abilities; (2) Manual evaluation is highly difficult. It requires careful learning of complex background and abundant dialogues, and takes 15 minutes to evaluate 7 models for 1 case on average.\n\n> Q4: More transparent reporting on how rubric-based penalties are assigned would improve reproducibility.\n\nWe recognize this feedback. We will improve transparency by releasing the LLM-identified flaws, and adding detailed examples and analysis in our appendix. \n\n> Q5: Prior work in digital actors and AI-driven storytelling could be more directly compared or contrasted with CoSER’s approach.\n\nAs suggested, we compare GCA simulation with stories generated by HollmWood [1]. We use 30 samples from CoSER Test, apply GPT-4o as the actors/writers, and report average scores assessed by GCA evaluation. \n\n|Method|avg_score|\n|-|-|\n|GCA|59.2|\n|HollmWood|50.2|\n\nResults show GCA simulation produces more authentic and human-like character interactions compared with HollmWood. We will include the results in our paper.\n\n## Reference:\n\n[1] Jing, C., et al. HoLLMwood: Unleashing the Creativity of Large Language Models in Screenwriting via Role Playing", "claims_and_evidence": "1. It asserts that authentic literary dialogues lead to better role-playing LLMs, but does not present direct ablation studies comparing performance on authentic vs. LLM-generated role-playing data. Demonstrating such an effect with controlled experiments would strengthen the claim.\n\n2. The evaluation relies heavily on LLM-based judges, particularly GPT-4o, which could introduce bias or overfitting to OpenAI’s model responses rather than proving general improvements across diverse assessment frameworks.", "methods_and_evaluation_criteria": "1. The paper introduces Given-Circumstance Acting (GCA) for training and evaluation, but it is not well-validated by human raters. Human assessments of role-playing quality (e.g., expert evaluations from literary scholars or acting professionals) would provide stronger validation than LLM critics.\n\n2. The penalty-based scoring approach for LLM judges could be subject to overfitting to specific LLM biases rather than capturing true persona alignment. More transparent reporting on how rubric-based penalties are assigned would improve reproducibility.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "Some prior work in digital actors and AI-driven storytelling (e.g., work from interactive fiction and game AI communities) could be more directly compared or contrasted with CoSER’s approach.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "While the paper presents a good dataset and framework for role-playing language agents, its core contributions—focused on character simulation, narrative coherence, and dialogue modeling—seem more aligned with computational linguistics, NLP, or AI for interactive storytelling rather than core machine learning (ML) innovations. The methodology, including Given-Circumstance Acting (GCA) and LLM-based evaluation, primarily builds on existing LLM architectures rather than introducing fundamental advances in ML theory, optimization, or learning algorithms, which are central to ICML. A stronger alignment with ML advancements, such as novel architectures for long-term character consistency, reinforcement learning for persona fidelity, or interpretability in role-playing agents, would make the work more relevant to the ICML audience. Otherwise, it might be better suited for venues like ACL, EMNLP, or AAAI that emphasize natural language processing and AI applications in interactive settings.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Interesting idea and well-execution. Multiple reviewers agree that this work is well-motivated, scalable to a large corpus, and appropriate evaluation. 

One common concern from many reviewers is in related work. The coverage of related works seems to be short to the currently emerging and expanding literature on this and related topics. AC highly recommends thoroughly comparing and giving take-home deep insight. A table, like in your rebuttal, would be nice to convey the core differences. AC also wants to refer TimeChara [a], which proposed role-playing LLMs, although it is an agent-user interaction focusing on point-in-time character hallucination.

AC sees the other raised weaknesses are addressed during the rebuttal period, while leaving no major concerns among reviewers.

[a] Ahn et al. 2024. TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2024.
Decision: Accept (poster)