PersonaEval: Are LLM Evaluators Human Enough to Judge Role-Play?
Authors: Lingfeng Zhou
Jialing Zhang
Jin Gao
Mohan Jiang
Dequan Wang
Venue: COLM 2025 (2025)
Keywords: Role-play
evaluating LLM evaluators
benchmark
TL;DR: We introduce PersonaEval, a benchmark showing that LLMs struggle to judge role-play like humans. Even top models fail basic role identification, highlighting a need for more human-like reasoning in LLM evaluators.

Abstract: Current role-play studies often rely on unvalidated LLM-as-a-judge paradigms, which may fail to reflect how humans perceive role fidelity. A key prerequisite for human-aligned evaluation is role identification, the ability to recognize who is speaking based on dialogue context. We argue that any meaningful judgment of role-playing quality (how well a character is played) fundamentally depends on first correctly attributing words and actions to the correct persona (who is speaking). We present PersonaEval, the first benchmark designed to test whether LLM evaluators can reliably identify human roles. PersonaEval uses human-authored dialogues from novels, scripts, and video transcripts, challenging models to determine the correct persona according to the conversation context. Our experiments, including a human study, show that even the best-performing LLMs reach only around 69% accuracy, well below the level needed for reliable evaluation. In contrast, human participants perform near ceiling with 90.8% accuracy, highlighting that current LLM evaluators are still not human enough to effectively judge role-play scenarios. To better understand this gap, we examine training-time adaptation and test-time compute, suggesting that reliable evaluation requires more than task-specific tuning, but depends on strong, human-like reasoning abilities in LLM evaluators. We release our benchmark at https://github.com/maple-zhou/PersonaEval.

Published as a conference paper at COLM 2025
PersonaEval: Are LLM Evaluators Human Enough to Judge Role-Play?
Lingfeng Zhou', Jialing Zhang', Jin Gao', Mohan Jiang Dequan Wangl
'Shanghai Jiao Tong University 2Shanghai Innovation Institute
Abstract
Current role-play studies often rely unvalidated LLM-as-a-judge paradigms, which may fail to reflect how humans perceive role fidelity. A key prerequisite for human-aligned evaluation is role identification, the ability to recognize who is speaking based on dialogue context. We argue that any meaningful judgment of role-playing quality (how well character is played) fundamentally depends on first correctly attributing words and actions to the correct persona (who is speaking). We present Eval, the first benchmark designed to test whether LLM evaluators can reliably identify human roles. PersonaEval uses human-authored dialogues from novels, scripts, and video transcripts, challenging models to determine the correct persona according to the conversation context. Our experiments, including human study, show that even the best- best-performing LLMs reach only around 69% accuracy, well below the level needed for reliable eval- uation. In contrast, human participants perform near ceiling with 90.8% accuracy, highlighting that current LLM evaluators are still not human enough to effectively judge role-play scenarios. To better understand this gap, we examine training-time adaplation and test-time compute, suggest- ing that reliable evaluation requires more than task-specific tuning, but depends on strong, human-like reasoning abilities in LLM evaluators. We release our benchmark https:/ http//ggtthhooo commmaple-zhou saiee/zhhoo/PPrrnaavvaal EvalsonaEval.
Introduction
A growing body of work in role-play adopts LLM-as-a-judge paradigms, where models are tasked with evaluating the role-playing behavior of other models (Shao et al., 2023; Wang et al., 2024c; 2025b; Lu et al., 2024). While scalable, this strategy assumes that large language models (LLMs) can approximate human judgment, a claim that remains largely untested. This gap in validation raises concerns about the reliability and true human-alignmeen of current evaluation pipelines. Recent studies have already revealed misalignment between LLMs and human, including preference leakage (Ghasemi et al., 2025; Murugadoss al., 2024), where models favor outputs from their own model family. What's more, Zhao et al. (2025) show that one token is enough fool LLM judges. In a broader cognitive sense, Josh Tenenbaum argues that LLMs derive intelligence from language, while humans develop language after acquiring intelligence (Cherian al., 2024). These gaps call into question whether current LLMs can reliably assess role fidelity in human-like way.
key prerequisite for human-aligned evaluation is the ability identify the speaker's role from context. We argue this is a foundational capability: an LLM cannot credibly assess how well a role is portrayed it cannot first determine who is speaking. This is because accurate identification is essential for grounding the interpretation of dialogue and avoiding critical errors, such as misattributing behaviors or inconsistencies to the wrong speaker. Such failures ultimately compromise the fairness and precision of the entire evaluation process, making role identification a minimal, yet objective, test of alignment with human.
To examine this, introduce PersonaEval, the first benchmark for evaluating whether LLMs can reliably identify character roles from dialogue context. The task is formulated
* *orresponding author: dequanwang@ūjjuuedd.cc

---

Published as a conference paper at COLM 2025
Content Grading
AI
Multi-Agent Judgment Broadly Used Role-play
Truly Qualified?
Pairwise Comparison Ys
LLM Evaluator
PersonaEval
Gemini-2.5-pro-0666
Why doesn't 't Ron understand how...
A. Hermione Granger (Chosen)
Select
B. Ginny Weasley
Harry
A
B
C. Ron Weasley (Correct)
need be careful here. don't 't want...
D. Harry Potter
C
D
Easy for human, but the latest model fails!
Figure 1: LLM-as-a-judge paradigms are broadly used in role-play evaluation, while its re- liability has not been verified. We propose PersonaEval, the first benchmark to investigate necessary condition for human-aligned LLM evaluators of role-play: accurate identification of roles from dialogue context. We find that even the latest model Gemini-2.5 Demmiin2.5-pro-06055 fails the case which is easy for humans (Section 4.3), indicating the challenge remains unsolved.
as constrained classification: given a dialogue snippet and four candidate roles, the evaluator must select the role most consistent with the target utterance (Figure 1). We ground our evaluation in human judgment by curating instances from human-authored materials- novels, scripts, and transcripts-and and supplement them with detailed role descriptions to mitigate potential LLM knowledge gaps.
Our experiments reveal clear limitation of LLMs. State-of-the-art LLMs achieve more than 69% accuracy on PersonaEval, which falls short of meeting convincing baseline for reliable assessment, and even the latest model fails on trivial cases (Figure 1). In contrast, human participants achieve 90.8% accuracy. These findings offer a clear answer to the question in our title: LLM evaluators are not yet "human enough" to judge role-play.
To understand what makes a better human-aligned LLM evaluator, we investigate two common strategies: training-time adaptation and test-time compute. Surprisingly, fine-tuning LLMs with role-play data does not improve performance and can even degrade it, suggesting that memorizing role-specific knowledge is insufficient. In contrast, test-time methods show more potential. Notably, reasoning models consistently outperform others.
find
These results suggest that strong role-play evaluation depends less on simple heuristics and more on robust, context-aware reasoning. Accurately judging roles requires inference, perspective-taking, and social understanding--kki than with pattern matching. Taken together, our findings point to test-time compute, particularly inference-time reasoning, as promising direction for building LLM evaluators of role-play that better reflect human-like judgment. Our main contributions are as follows:
more aligned with human judgment
We introduce PersonaEval, the first benchmark to directly evaluate whether LLMs can identify human roles from natural dialogue, a necessary but underexplored foundation for reliable role-play evaluation.
We demonstrate that state-of-the-art LLMs fall short of human-level performance role identification, revealing a critical gap in their ability reflect human judgment.
We find that reliable role-play evaluation depends not on role-specific training or prompting, but on reasoning ability, highlighting test-time compute especially rea- soning as a promising strategy for building more human-aligned LLM evaluators.
2This experiment was performed June 2025, when Gem-ni-2.5-rro-0605 was the latest model.
2

---

Published as a conference paper at COLM 2025
2 Related Work
Role-playing Evaluation Role-playing is essential for aligning large language models (LLMs) with human values, which positions human experts as the gold standard for evalua- tion, just like what early LLM role-play studies do (Zhou et al., 2023). However, the vast scale of modern role-playing systems makes thorough human evaluation impractical due to high costs and delays. Therefore, some works (Lu et al., 2025; Wang et al., 2024a;b) turn to traditional metrics of natural language generation tasks, including BLEU and ROUGE. These metrics compute similarities between the reference text and the generated content. Yet in the domain of role-play, ground truth is always uncertain, since there are multiple possible correct responses. As result, researchers gradually shift to reference- evalua- tion. Some works train a reward model to estimate the quality of generated responses (Tu et al., 2024; Wu et al., 2025; Dai et al., 2024). More and more works are using LLMs as judges to evaluate the quality of role-play content from a human perspective (Gusev, 2024; Wang et al., 2024c), benefiting from their strong generalization ability and capacity to provide more comprehensive and fine-grained feedback. While some of them leverage LLM evaluators in a reference-guides way (Zhou et al., 2024a; Yu al., 2024), most utilize LLMs to score from diverse dimensions (Yang et al., 2025; Lu et al., 2025; Zhou et al., 2024a; Shao et al., 2023) or execute pairwise comparisons (AI, 2024; Wu al., 2025). Among them, Wang et al. (2025b) even builds a multi-agent system to make the final judgment. To avoid the subjec- tivity bias originated from scoring, some works let LLM evaluators perform multi-class classification (Lu et al., 2024; Yuan et al., 2024).
These approaches aim to assess how effectively LLMs play specific roles, contributing to broader understanding of their performance in role-playing scenarios. However, challenges remain in ensuring that evaluation results are both reliable and aligned with human expectations of role adherence.
Evaluating LLM Evaluator The reliability of LLMs as evaluators is gaining increasing attention in the community (Son et al., 2024a; Wei et al., 2024; Zhang et al., 2024). Researchers start by testing the instruction-followii
abilities of LLMs in their role as evaluators. Zheng et al. (2023) addresses this issue by using advanced LLMs to assess performance on open- ended questions. Wang et al. (2023) approach ChatGPT as a human-like evaluator by providing task-specific (e.g., summarization) and aspect-specific (e.g., relevance) instruc- tions to prompt ChatGPT for evaluating the outputs of various NLG (Natural Language Generation) models. Murugadoss et al. (2024) explores whether LLM assessments are based purely on prompt instructions or also reflect inherent preferences for high-quality data similar to their fine-tuning data. There are also various meta-evaluation studies of LLMs with different focuses. Chern et al. (2024) evaluates LLM evaluators through multi-agent debates, while Son et al. (2024b) and Hada et al. (2024) propose multilingual benchmarks. Eiras et al. (2025) carefully examines the safety-related aspects of LLM judges. Zhou et al. (2024b) pays attention to the domain of paper review, while Wang et al. (2025a) cares about the performance of LLM evaluators in software engineering.
Although interest in the meta-evaluation of LLMs is growing, the specific evaluation of LLMs as role-play evaluators remains underexplored. While some work (Yang et al., 2025) includes ablation studies to examine aligrment between LLM evaluators and human experts, there still a lack of systematic perspective on meta-evaluating role-play LLM evaluators, particularly regarding their ability to distinguish between nuanced role identities. Yet this ability is fundamental to higher-order role-play evaluation; without it, any subsequent judgment of role fidelity rests on an unstable and unverified foundation, revealing critical gap in our understanding of how well LLMs can perform human-aligned assessment.
3 PersonaEval
We design PersonaEval to test whether large language models (LLMs) can perform human- like judgment in core subtask of role-play evaluation: identifying who speaker is based on dialogue context. To build a reliable benchmark for role identification, we start by formu- lating the task as a constrained classification problem (Section 3.1), then construct diverse,
3

---

Published as a conference paper at COLM 2025
lokens 25
Dialogue Pool
Prompt LLMs
Hard Case Curation
Dialogue Context
Instruction:
confidence
Why docsn't 't Ron understand how...
Analyze the conver-
Harry
sation and chouse
PersonaEval
Authentic Human-
originated Data
need be careful here. I don't want...
Encoder $
Embedding
the matched role...
Literary: 26,208 cases
Drama: 1,658 cases
Candidate Roles
Expertise: 699
Novels
Groundtruth
shared
Scripts
Top-3 roles with highest cosine- similarity Groundruth
Vidcos
11:1
Encoder
Role Pool
Figure 2: Curation pipeline of PersonaEval. PersonaEval evaluates role identification ability of LLM evaluators via constrained classification task built on human-authored dialogues from novels, scripts, and videos. Each instance includes a two-turn conversation: the first turn is from known character, and the second is from target character whose identity is to be predicted by LLMs. The ground truth role's embedding is used to retrieve the top-3 most similar roles from candidate pool, forming a challenging candidate set. A standardized prompt (See Appendix A) presents the dialogue and candidate roles to LLMs. To ensure difficulty, only instances where strong baseline model (Qwen-max) shows low confidence in the correct answer are retained. The final benchmark includes three Paccks-PersonaByye Literary (06, PersonaEval--rram. (1,658), and PrrsonaEval-Expereee (699).
human-authored datasets that capture role-play across different domains (Section 3.2). To increase challenge and minimize shortcut cues, we design semantically similar adversarial distractors (Section 3.3). Finally, we apply hard case curation pipeline (Section 3.4). See Figure 2 for the complete curation pipeline.
3.1 Task Formulation
PersonaEval adopts a classification framework to evaluate role identification capabilities in LLMs. Each instance presents two-turn dialogue between two characters: Character1 (with a known identity) and Character2 (whose identity is to be inferred). Models are given the dialogue context and four candidate roles (five for expertise-level settings, see Section 3.3), each accompanied by a detailed profile. Evaluators assign confidence scores to each candidate, with higher scores indicating stronger belief in match. To mitigate positional bias, candidate order is randomized. This design turns an inherently subjective task into verifiable one, using deterministic ground truth derived from source materials. The full prompt is provided in Appendix A.
While this setting may appear less common, this is largely because prior work has often implicitly assumed that LLMs possess such basic capabilities. Our intention is to carefully examine this assumption and bridge gap that, to our knowledge, has been under-explored. Nonetheless, normally in a role-play conversation, the context we have is more than just 2 turns. We agree that extending evaluations to richer, multi-turn contexts is important, and we view our work as necessary first step toward that broader goal.
3.2 Data Composition
PersonaFval includes three tracks, each targeting different facets of role understanding:
Personafival-Literay::: Built on 26,208 dialogues from 771 English novels, this track tests persona inference in fictional narratives. The data are curated from CoSER (Wang et al., 2025b), verified fiction-based dataset containing texts from classic and modern novels.
PersonaEval-Drama: This track contains 1,658 Chinese dialogue snippets from screenplays, testing models' ability to understand role alignment in scripted in-
4

---

Published as a conference paper at COLM 2025
teractions. The data are adapted from the partially open-source CharacterEval datasets (Tu et al., 2024).
Persona@vvl--xpeertis Sourced from the Wired "5 Levels" video series?, this track features 699 scaffolded explanations where domain experts tailor content to audiences of different knowledge levels (Child, Teen, College Student, Graduate Student, and Expert). Dialogues test whether models can infer speaker's intended audience based on linguistic and conceptual cues, without relying on specialized domain knowledge.
Together, these tracks ensure coverage across fictional, performative, and instructional domains. Literary texts emphasize character personality and inner thoughts, while scripts focus more on conversational style, and instructional videos involve role-audience align- ment. This diversity helps mitigate domain-specific bias some extent.
Importantly, all source data are human-authored, avoiding contamination from synthetic model-generatee content. This design choice ensures evaluation aligns with human, as public datasets increasingly overlap with prior model outputs.
We also balance language diversity with the availability and quality human-authored materials, providing two English tracks and one Chinese track. English resources, especially literary novels, are more abundant and varied, allowing us to construct robust, high-quality benchmarks. At the same time, include Chinese track to introduce linguistic and cultural diversity, recognizing the importance evaluating models in non non-English contexts.
Since some materials may not be included in the pretraining corpus of LLMs, provide detailed role descriptions, including both basic traits and nuanced plot-related context. This helps ensure that role identification performance reflects reasoning ability rather than missing background knowledge, minimizing the impact of knowledge gaps.
3.3 Adversarial Distractor Construction
To ensure each instance presents a genuine reasoning challenge, we construct adversar- ial distractors that are semantically close to the correct role. As shown in Figure 2, for Personalvval-Litrrary and Evalonaaval-Drama. we embed all candidate role profiles us- ing three independently trained models to avoid bias from single model structure: Ope- nAl's text-embeddingg---neee 3-small (OpenAI, 2024), BGE-M3 (Chen et al., 2024), and Sentence- BERT's (Reimers & Gurevych, 2019) multilingual distiluse-base sesisse-ba--mulllliinuuu multilnngull-casee each instance, we compute cosine similarity between the ground-truun truth role profile and all others in the pool. From each embedding model, we select the top non-target role (i.e., most similar but incorrect) and use these form a distractor set of three, ensuring diverse but challenging contrastive options. This forces evaluators to resolve subtle, human-level ambiguities, rather than rely on surface heuristics.
For
In the Eallonnavvll-xxeettie track, we leverage the fixed five-tier role hierarchy (Child to Expert). Each dialogue uses the four incorrect levels as distractors, reflecting authentic educational scaffolding challenges without needing embedding-based selection. All options are paired with -udeefie-specffic profiles drawn from the original video transcripts.
Across all tracks, we avoid synthetic perturbations and instead build distractors from naturally occurring human roles. This ensures that the task tests real-world ambiguity resolution, not artificial contrastive tricks.
3.4 Hard Case Curation
We observe that the original unfiltered data contain numerous trivial cases (e.g., direct name mentions, simple greetings) that do not genuinely test role inference. To avoid inflating performance with trivial cases and ensure the benchmark focuses meaningful reasoning, we explicitly filter for difficult instances where even strong models struggle. Our goal is not
wttp://www....ooooiiieee/eeeeeee 5-lleesss
5

---

Published as a conference paper at COLM 2025
broad coverage, but focused evaluation of whether models can resolve subtle, human-level ambiguities. We apply a two-stage filtering process:
Stage Low-Information Filtering We remove dialogue turns where Character2's utter- ance is under 25 tokens. These low-information responses (e.g,, "Exactly:") offer little basis for role inference and are excluded to maintain meaningful task complexity.
Stage 2: Confidence-Based Filtering Using Qwen-max (Bai et al., 2023), we filter out instances where the model shows high confidence in the correct answer. Specifically, we discard any instance where Qwen-max assigns over 50% confidence to the ground-truth role. From an initial pool of over 110,000 instances, this filtering yields 28,565 challenging examples, each requiring inference beyond surface-level cues. Nonetheless, this confidence- based filtering may introduce systematic bias. More discussion can be found in Appendix
4 Experiment
We organize our experimental analysis into four parts. We first evaluate model performance on PersonaEval (Section 4.1), then analyze the impact of reasoning ability of LLMs (Sec- tion 4.2), present case studies (Section 4.3), and conclude with human study (Section 4.4).
Since prior work often uses competent large language models (LLMs) like GPT-3.5 and GPT-4 as role-play evaluators (Shao et al., 2023; Lu et al., 2024), we evaluate diverse set of state-of-the-art models, including several reasoning ones rarely considered in existing role-play studies. Model versions specified in Appendix C. We report classification accuracy (top-1 accuracy) as the primary metric for model performance.
4.1 Main Results on Role Identification with PersonaEval
Figure 3 summarizes model performance across the three tracks of PersonaEval. Most LLMs achieve around 40-60% accuracy, including the latest GLM-4.5, and even the best-performing model, Gemini-2.5-pro (03-25 version, as detailed in Appendix C), reaches only 68.8%, well below human-level performance (see Section 4.4). These findings reveal significant gap between current LLMs and the requirement of consistent role identification. As this task is necessary condition for credible role-play evaluation, the results highlight a core limitation of LLM-as-a-judge methods. Full results are provided in Appendix D.
Nonetheless, some models show signs of progress. As shown in Table placed in Ap- pendix D, t accuracy and mean rank suggest partial understanding: many models rank the correct role second, suggesting they are still aware of the correct answer. Notably,
100
Human-Level
90.8
Current
Accurccy (%6
80
Non-rcasoning Models
62.0 62.0 64.8 68.8 Gap
Reasoning Models
37.7 40.9 41.4 41.7 43.4 45.3 48.8 52.4
54.0
56.2
58.2 59.1
40
33.4
36.1
GPT-335 GPT-3.5-tubo tutbo
20
Owen-max 322
Qwen2.5-33 3-328
GPT-4 GPT-40 DeepSeck-Ri distill GPT T-P-- mini
DeepSeck- V3
Douban-1 1.5 p-pro
Qwen3-2358 235B
Kimi-K3
QwQ-32B
Grok-3-bal --lla
328
GPT-4.
Gemmi-2.55 5--llah 5-flash
GLM-45
Claude-3 M.leee...--.m 7-sonnet
DeepSeck RR
Gemini-2.5-pp
Figure 3: Accuracy of LLMs on Fers Most LLMs struggle with role identification compared to human, including the latest GLM-4.5, highlighting fundamental limitation. Reasoning models show clear advantages, suggesting that effective evaluation requires deeper reasoning. DeepSeek-R1-dee R1-distill refers to Peepseek-RI-distil--Wwwe

---

Published as a conference paper at COLM 2025
Claude-3.7ssonnet, DeepSeek-RI, and Gemini-2.5-pra achieve near-perfect top-2 accuracy. Furthermore, both ECE and Brier Score remain relatively low, indicating that models express appropriate uncertainty rather than overconfident errors. Together, these results provide more optimistic view, suggesting that while LLM evaluators are not yet human-level, some catching up. With further advances, they may become viable for role-play evaluation.
4.2 Analysis on Reasoning Models
Reasoning models show clear advantages over base models in role identification. As shown in Figure Gemini-2.5-pro outperforms all foundation models, including Claude-3.7-sonnet
We also compare reasoning models with and without native reasoning capabilities. -ieieee--R--iissill-eeeeeee
which distills reasoning from DeepSeek-RI into Qwen2.5, achieves only a small gain over its base model. In contrast, QwQ-32B, trained via rein- forcement learning reasoning-intensive tasks, significantly outperforms Qwen2.5. This suggests that shallow or distilled reasoning may not transfer well to role-play evaluation, while end-to-ens reasoning training is more effective.
These results support a broader insight: effective role identification requires robust, model- native reasoning. Beyond pattern matching, evaluators must engage in contextual judgment and logical inference prrrence-capabllttte more aligned with human-like evaluation.
Our analysis suggests that several specific types of reasoning are particularly important for effective role identification. These include:
Perspective-taking: The ability infer the speaker's background, goals, and point of view based on contextual clues within the dialogue.
Intent inference: The capacity to understand the underlying intention behind an utterance, which often goes beyond its surface-level semantic meaning.
Pragmatic reasoning: The skill of interpreting the social and contextual meanings of statements as they unfold within an interaction.
These cognitive skills allow an evaluator to move past simple linguistic pattern matching and instead build a coherent model of the speaker's 's identity, which is fundamental for judging role-play fidelity in a human-aligned manner.
4.3 Case Study
Through case studies, we find that many examples, which are trivial for humans, are frequently misinterpreted by LLMs. In these cases, models often follow the wrong trajectory of reasoning. We suspect this is due to a fundamental difference in how role-play is interpreted: LLMs tend to focus on surface-level linguistic cues such as speaking style, while humans perform deeper reasoning. Specifically, humans excel at intent inference, allowing them to prioritize the speaker's 's communicative goals, and employ pragmatic reasoning to understand the social context of the dialogue, abilities that current models still lack. This observation echoes Josh Tenenbaum's view that LLMs derive intelligence from language, whereas humans develop language after acquiring intent-driven cognition.
We illustrate one such case below. The bracketed text denotes the character's 's internal thoughts. For a human reader, the reasoning path to identify the speaker is direct. First, Harry's internal monologue explicitly names "Ron" as the subject of his concern ("Why doesn't Ron understand..." "). Second, the respondent (Character2) directly addresses "Harry" in his reply. By integrating these two explicit references, the reader can unambiguously conclude that Character2 is Ron Weasley. However, the advanced model fails to connect these critical contextual signals, instead appearing to focus on the speaker's 's tone and linguistic style. Additional cases provided in Appendix E.
7

---

Published as a conference paper at COLM 2025
Dialogue and Candidate Roles
Response from Gemini-2.5-pro-0605
Characterl:
Harry Potter
Analysis:
[Thinking: Why doesn't Ron under- stand how important this is?]
Hermione Granger is the most likely responder, her cautious internal thought, logical reasoning ("no rule saying.."", and balanced confrontation ("You're re getting bit obsessed...") align closely with her character profile.
I want to know what he's up to. And don't tell it's all in my head, af- ter what overheard between him and Snape.
Character2: Unknown
Ginny is plausible second due to her confidence and perceptiveness. Ron is less likely given the measured tone that doesn't quite match his typically more reactive style. Harry is Charac- ter 1 being the one expressing obsession and therefore cannot be the responder.
[Thinking: need to careful here. don't want to dismiss Harry's concerns, but this is getting out of hand.] I never said it was all in your head, but there's no rule saying only one person at time can be plotting anything in this place! You're getting a bit obsessed with Malfoy, Harry.
Final Answer:
Hermione: 0.75, Ginny: 0.15, Ron: 0.10,
Candidate Roles:
Harry: 0.00
Hermione Granger, Ginny Weasley, Ron Weasley (Correct), and Harry Potter
(Higher probability indicates stronger role correspondence))
4.4 Human Study
To evaluate human performance on PersonaEval and compare it with LLMs, we conduct controlled study with 20 highly educated volunteers, including 10 undergraduates and 10 PhD students, selected to match the task's reasoning and domain knowledge demands. Each participant is given 50 examples sampled from cases where DeepSeek-RR makes incorrect high-confidence predictions, spanning all three tracks of the benchmark. No time limits are imposed to allow for careful, thoughtful responses.
We deliberately select challenging cases for human evaluation to ensure meaningful com- parisons on non-trivial instances. While this sampling introduces distributional differences, it is actually conservative choice that substantially strengthens still perform well on cases that are consistently difficult for models, the performance gap be- comes even more compelling. We have manually verified that many cases in the benchmark are straightforward for humans. If we had sampled uniformly across the full benchmark, human accuracy would likely have been even higher.
findings: if humans
As shown in Figure 3, participants achieve an average accuracy of 90.8%, far surpassing the pes--rerfrrnng LLM. This highlights a substantial gap between current LLM capabilities and human-level understanding in role identification. Full results are in Appendix F.
Improvement Investigation
To better understand what contributes to effective role-play evaluation, we investigate two common strategies for improving LLMs: training-time adaptation and test-time compute. At training time, we test whether injecting role-specific knowledge through fine-tuning improves role identification performance (Section 5.1). At test time, we apply few-shot prompting and self-consistency to examine whether additional inference- siteeence-time computation can enhance evaluator accuracy (Section 5.2). Detailed results can be found in Appendix G.
5.1 Training-time Adaptation
Many role-play studies fine-tune LLMs to inject role-specific knowledge, aiming to improve downstream performance. In this section, we evaluate two generations of such models:
8

---

Published as a conference paper at COLM 2025
Base Models
Train-time Injected Models
60
60
57.2
54.8
50.6
Accuracy (%)
44.9
43.8
47.6
50
41.8
40
37.1
38.7
40
40
27.5
34.1
30
30
28.0
30
20
DBP -BP-character
DBP
DBP-1.5 c-harccct
DBP-1.5 1.5
DBP-chure OBP-character
DBP
DBP-1 11--1.5-chi 5-character
DBP-1.5
DBP R8P-character
DBP
DBP-1.5 DBP-1 chaaracter
(a) PersonaEval-Literary
(b) PersonaEval-Drama
(c) PersonaEval-Experree
Figure 4: Accuracy of 4 models on the three tracks of PersonaEval, corresponding (a), (b), and (c), suggests that fine-tuning on role-specific data does not improve role identification performance and may even degrade it, while improvements in base model capability show more consistent gains. DBP is short for Doubao-pro. DBP-character and are fine-tuned from DBP and DBP-1.5 respectively using role-specific data.
Doubao-pro-characee and Doubao-1.55 1.w.11.5------haccceee which are obtained by fine-tuning Doubao-pro and Doubao-1.5 respectively.
These models are closed-source and provided by ByteDance The training-time adaptation performed by fine-tuning on datasets enriched with role knowledge, identity recognition, dialogue alignment, behavior alignment, and instruction tuning. These are implemented under broader System Prompt + SFT framework to ensure that models can maintain per- consistency, recognize social roles, and infer identity throughout dialogue interactions. In addition, the system also emphasizes multi-turn dialogue memory, context retention, and user portrait construction to support long-term coherence.
As shown in Figure 4, fine-tuning on role knowledge does not improve performance on our role identification task; in fact, degrades it. A possible explanation is that memorizing role-related patterns during fine-tuning may interfere with the model's native reasoning ability, which is essential for evaluating nuanced persona alignment. This suggests that injecting role knowledge through training-time adaptation is not an effective way to build reliable role-play evaluators. In contrast, comparing Doubao-pro and Doubao-1.5-pro shows that improvements in the base model itself, such as general capability upgrades, have more positive impact on role identification performance.
We also test CoSER-Llamal 101--3mmmm3.11118 1-8B (Wang et al., 2025b), fine-tuned model introduced in the data source of Fvalonaaval-LLteaaaa However, fails to follow our benchmark instruc- tions and cannot even produce usable outputs, making evaluation infeasible. This further highlights that role-specific training alone is insufficient for constructing capable evaluators.
5.2 Test-time Compute
In this section, evaluate two popular test-time compute strategies: few-shot prompt- ing (Brown et al., 2020) and selff-onnsitencyy (Wang et al., 2022). We focus on Eval Literary using Qwen-max and DeepSeek- V3, as this track forms the of PersonaEval and offers more stable shot selection than others.
For few-shot prompting, we test 1-shot, 3-shot, and 5-shot settings. For self-consistency, we apply majority voting with K = 3 and K = 5, as well as a weighted average. As shown in Figure few-shot prompting consistently improves model performance, but gains plateau by 5-shot. In contrast, self-consistency provides negligible improvement across all settings.
9htps://tem..duuba.com

---

Published as a conference paper at COLM 2025
Few-shot Prompting
Self-consistency
53.6
54.1
Accuracy (%)
50.5
44.2
46.3
46.3
42.8
42.4
45.6
43.3
O-shot
40
34.5
34.2
36.9
40
35.9
O-shot
20
20
1-shot 3-shot 5-shot
MV-3
MV-3
AVG
1-shot 3-shot
5-shot
MV-3
MV-5
AVG
(a) Test-time Compute on Qwen-max
(b) Test-time Compute on DeepSeck- V3
Figure Accuracy on PersonaEEvl--iteearyy using two test-time compute strategies on (a) Qwen-max and (b) DeepSeek-V3, respectively. The gray dashed line in each plot indicates mode the 0-shot accuracy the corresponding model. Few-shot prompting (1-shot, 3-shot, 5-shot) shows consistent but saturated gains. In contrast, self-consistency (MV: majority vote, AVG: weighted average) yields negligible improvement across settings.
These findings support our earlier analysis (Section 4.2): role-play evaluation relies more on model's reasoning ability than on sampling ensembling techniques. While few-shot prompting can impart some reasoning patterns, its effect is constrained by the quality and generalizability of exemplars. Self-consistency, meanwhile, merely reinforces a model's existing reasoning without extending it, yielding no meaningful performance boost.
Conclusion
In this work, introduce PersonaEval, the first benchmark for evaluating whether LLMs can reliably identify character roles from natural dialogue, a foundational step toward human-aligned role-play evaluation. Our results show that current LLM evaluators perform well below human-level, with even the strongest and latest models struggling on cases that are trivial for humans. This highlights a core limitation pipelines. Through empirical analysis, we further find that training-times adaptation with role-specific data offers little benefit, while test-time methods, especially reasoning models, show more promise. These findings suggest that robust, context-aware reasoning, rather than memorization, is essential for effective role-play evaluation, and point to test-time compute practical and forward-looking direction.
existing LLMaas-a-judde
Looking ahead, a promising line future work to move beyond output accuracy and in- vestigate how LLM evaluators arrive at their predictions. This includes analyzing reasoning trajectories across models, comparing them with human thought processes, and identifying where their judgments diverge. To support this, new diagnostic tools may be developed to visualize and interpret model reasoning paths. Such insights can inform new forms of test-time guidance, such as human-aligned reasoning chains or rationale-based prompting. Another direction is to explore how human-like reasoning strategies can be systematically injected into models at inference time. Ultimately, closing the gap between LLM and human evaluation may require bridging deeper cognitive differences-nne serrencces-moiing moving beyond pattern recognition toward reasoning grounded in intent, context, and social understanding, much like how human judgment emerges from experience before language, as emphasized by Josh Tenenbaum. We hope PersonaEval serves as a step toward this goal.
Acknowledgments
This research is supported by the Key R&D Program of Shandong Province, China (2023CXGC010214, 2024CXGC010213), We express our gratitude to the funding agency for by
their support. We thank all the anonymous reviewers for their valuable suggestions.
10

---

Published as a conference paper at COLM 2025
References
Boson AI. Introducing rpbench-auto. h cessed: 2025-03-24.
blog////hhhhbhhhooooooooo b02g/,2024. Ac-
Anthropic. Claude 3.7 sonnet, 2025. URL htwpt:ttp//cmm httppsss///...attrrric....c/o///
sonnet. Accessed: 2025-03-25.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint ar arXio:2309.16609, 2023.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek I1m: Scaling open-source language models with longtermism. arviv preprint XxXi=2401.002554 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are fhw-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Alulii-ligual, fulti-ffuncioonality multi-granularity text embeddings through self-knowledge distillation. ar preprini Xxxr:2402.032166 2024.
Anoop Cherian, Kuan-Chuan Peng, Suhas Lohit, Joanna Matthiesen, Kevin Smith, and Josh Tenenbaum. Evaluating large vision-and ladlauuugge models on children's 's mathematical olympiads. Advances in Neural Information Processing Systems, 37:15779-15800, 2024.
Steffi Chern, Ethan Chern, Graham Neubig, and Pengfei Liu. Can large language models be trusted for evaluation? scalable neta-evaluation of lims as evaluators via agent debate. arXivv preprint ar arXiv:2401.1678. 2024.
Yanqi Dai, Huanran Hu, Lei Wang, Shengjie Jin, Xu Chen, and Zhiwu Lu. Mmrole: A comprehensive framework for developing and evaluating multimodal role-playing agents, 2024.
Francisco Eiras, Eliott Zemour, Eric Lin, and Vaikkunth Mugunthan. Know thy judge: On the robustness meta-evaluation of I1m safety judges. arv preprint arXiv:2503.044 04474, 2025.
Omid Ghasemi, Adam JL. Harris, and Ben Newell. From preference shifts information leaks: Examining individuals' sensitivity to information leakage in the framing effect. Cognition,, 258:106087, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint (tft::2501.129444 2025.
llya Gusev. Pingpong: A benchmark for role-playing language models with user emulation and multi-model evaluation, 2024. URL
abs /2009 06 820.
Rishav Hada, Varun Gumma, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. Metal: Towards multilingual meta-evaluation. arXiv preprint ar Xr14:22404.0166, 2024.
Junru Lu, Jiazheng Li, Guodong Shen, Lin Gui, Siyu An, Yulan He, Di Yin, and Xing Sun. Rolemre: A fine-grained composite benchmark for role-playing and instuuctio-foolowinng arviv preprint ar xrXx:22502.1138, 2025.
Keming Lu, Bowen Yu, Chang Zhou, and Jingren Zhou. Large language models are superpositions of all characters: Attaining arbitrary role-play via self-alignment. arXiv preprint Xit:24011.1277 2024.
Bhuvanashree Murugadoss, Christian Poelitz, Ian Drosos, Vu Le, Nick McKenna, Ca- rina Suzana Negreanu, Chris Parnin, and Advait Sarkar. Evaluating the evaluator: Measuring Ilms' adherence to task evaluation instructions, 2024.
11

---

Published as a conference paper at COLM 2025
OpenAL ChatGPT (3.5 Turbo Version) [Large Language Model]. https://chat..ppnaa com, 2023.
OpenAl. Gpt-4 technical report. arXiv preprint arXxt:23003.0674,2222
OpenAL.
text-embedddnng---mall.. new-enbeddingg-oddee and- attpsss-aaaaaaappppppsssssss 2024. Accessed: 2025-03-23.
index/
OpenAL. Gpt-4.1, 2025a. URL http:////t///////aaaaaaaa//////a Accessed: 2025- 08-04.
OpenAl. Openai 03-mini. Accessed: 2025-03-26.
2025b.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. Xrv preprint ar Xit:1199 10084, 2019.
Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. Character-LLM: trainable agent for role-playing. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 13153- 13187, Singapore, December 2023. Association for Computational Linguistics. URL https: ttttps:/aaclatthee /aa/aaaaanntttoooooeeeooeae org....rg//0 org/2023 2023. emnlp-main. 814.
Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, and Seunghyeok Hong. Llm-as-a- judge & reward model: What they can and cannot do, 2024a.
Guijin Son, Dongkeun Yoon, Juyoung Suk, Javier Aula-Blasco, Mano Aslan, Vu Trong Kim, Shayekh Bin Islam, Jaume Prats-Cristi Lucía Dormo-Bañuelon and Seungone Kim. Mm- eval: multilingual meta-evaluation benchmark for llm-as-a-judge and reward models. Xiv preprint arXio:2410.17572 2024b.
Gemini Team, Petko Georgiev, Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng P'an, Shibo Wang, et al. Gemini 1.5: modal understanding across millions of tokens of context. arXiv preprint ar arXir:2403.05550, 2024.
Unlocking multi-
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao, Tong Gao, Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Hao Hu, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Chao Hong, Yangyang Hu, Zhenxing Hu, Weixiao Huang, Zhiqi Huang, Zihao Huang, Tao Jiang, Zhejun Jiang, Xinyi Jin, Yongsheng Kang, Guokun Lai, Cheng Li, Fang Li, Haoyang Li, Ming Li, Wentao Li, Yanhao Li, Yiwei Li, Zhaowei Li, Zheming Li, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyua Liu, Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Lijun Lu, Shengling Ma, Xinyu Ma, Yingwei Ma, Shaoguang Mao, lie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng, Ruoyu Qin, Bowen Qu, Zeyu Shang, Lidong Shi, Shengyuan Shi, Feifan Song, Jianlin Su, Zhengyuan Su, Xinjie Sun, Flood Sung, Heyi Tang, Jiawen Tao, Qifeng Teng, Chensi Wang, Dinglu Wang, Feng Wang, Haiming Wang, Jianzhou Wang, Jiaxing Wang, Jinhong Wang, Shengjie Wang. Shuyi Wang, Yao Wang, Yejie Wang, Yiqin Wang, Yuxin Wang, Yuzhi Wang, Zhaoji Wang, Zhengtao Wang, Zhexu Wang, Chu Wei, Qianqian Wei, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Xiaotong Xie, Weimin Xiong, Boyu Xu, Jing Xu, Jinjing Xu, L. H. Xu, Lin Xu, Suting Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ziyao Xu, Junjie Yan, Yuzi Yan, Xiaofei Yang, Ying Yang, Zhen Yang, Zhilin Yang, Zonghan Yang, Haotian Yao, Xingcheng Yao, Wenjie Ye, Zhuorui Ye, Bohong Yin, Longhui Yu, Enming Yuan, Hongbang Yuan, Mengjie Yuan, Haobing Zhan, Dehao Zhang, Hao Zhang, Wanlu Zhang, Xiaobin Zhang, Yangkun Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Haotian Zhao, Yikai Zhao, Huabin Zheng, Shaojie Zheng, Jianren Zhou, Xinyu Zhou, Zaida Zhou, Zhen Zhu, Weiyu Zhuang, and Xinxing Zu. Kimi k2: Open agentic intelligence, 2025. URL https:/ / org/ abss 2507.20534. ,25344
12

---

Published as a conference paper at COLM 2025
Qwen Team. Qwen3 technical report, 2025a. URL https::/aaxxtv.or/////// 09388.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025b. URL https: httpp:://wwelm.... http://w....hhu....m.m. ie/ii///ba http/llg///ee /lllg///qg22/.....
Jearnin
Quan Tu, Shilong Fan, Zihang Tian, and Rui Yan. Charactereval: A chinese benchmark for
role-playing conversational agent evaluation. arXiv preprint WrXio2240..01277 2024.
Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is ChatGIT good NLG evaluator? a preliminary study. In Yue Dong, Wen Xiao, Lu Wang, Fei Liu, and Giuseppe Carenini (eds.), Proceedings of the 4th New Frontiers in Summarization Workshop, pp. 1-11, Singapore, December 2023. Association for Computational Linguistics. doi: (0.18653/vvl w2w./22222..esmm1...
Noah Wang, Z.y. Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao Huang, Jie Fu, and Junran l'ngg RoleLLM: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACI. 2024, pp. 14743-14777, Bangkok, Thailand and virtual meeting, August 2024a. Association for Computational Linguistics.
Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. Can llms replace human evaluators? an empirical study llm-as-a-judge in software engineering. arXiv preprint ar arXiv22502.0619. 2025a.
Xi Wang, Hongliang Dai, Shen Gao, and Piji Li. Characteristic ai agents via large language models. arXiv preprint ar arXiv:2403.1236: 2024b.
Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li, and Yanghua Xiao. InCharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1840- 1873, Bangkok, Thailand, August 2024c. Association for Computational Linguistics. doi: 10.18653 v1 221/222.al-ooong.111 102.
Xintao Wang, Heng Wang, Yifei Zhang, Xinfeng Yuan, Rui Xu, Jen-tse Huang, Siyu Yuan, Haoran Guo, Jiangjie Chen, Wei Wang, et al. Coser: Coordinating Ilm-based persona simulation of established roles. Xiv preprint ar XrXiv:2502.09082, 2025b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. ar Xiu preprint ar XrXiv:2203.11171, 2022.
Hui Wei, Shenghua He, Tian Xia, Andy Wong, Jingyang Lin, and Mei Han. Systematic evaluation of -lm-as-a-judge in 11m alignment tasks: Explainable metrics and diverse prompt templates, 2024.
Bowen Wu, Kaili Sun, Ziwei Bai, Ying Li, and Baoxun Wang. Raiden benchmark: Evaluat- ing role-playing conversational agents with measurement-driven custom dialogues. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 11086-11106, 2025.
XAI. Grok beta, 2025. URL wttpthtt///// //x
Accessed: 2025-08-04.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. ar Xiv preprint 241.:2411.1115 2024.
13

---

Published as a conference paper at COLM 2025
Tao Yang, Yuhua Zhu, Xiaojun Quan, Cong Liu, and Qifan Wang. Psyplay: Personality- infused role-playing conversational agents. ar Xiv preprint arXi::2502.03821, 2025.
Xiaoyan Yu, Tongxu Luo, Yifan Wei, Fangyu Lei, Yiming Huang, Peng Hao, and Liehuang Zhu. Neeko: Leveraging dynamic lora for efficient multi-character role-playing agent. Xrv preprint ar xtiv:2402.13717, 2024.
Xinfeng Yuan, Siyu Yuan, Yuhan Cui, Tianhe Lin, Xintao Wang, Rui Xu, Jiangjie Chen, and Deqing Yang. Evaluating character understanding of large language models via character profiling from fictional works. Xrv preprint arXio:2404.12726, 2024.
Z.AL. Glm-4.5, 2025. URL htt.httts///..a/blleeee
Accessed: 2025-08-04.
Wenbo Zhang, Zihang Xu, and Hengrui Cai. Defining boundaries: A spectrum of task feasibility for large language models. arXiv preprint arXit:2408.05873, 2024.
Yulai Zhao, Haolin Liu, Dian Yu, SY Kung, Haitao Mi, and Dong Yu. One token to fool llm-as-a-judge- arvi preprint XrX:::2507.08794, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595- 46623, 2023.
Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng, Jiaming Yang, Xiyao Xiao, et al. Characterglm: Customizing chinese con- versational ai characters with large language models. ar Xrv preprint ar xriv:2331..1632,, 2023.
Jinfeng Zhou, Yongkang Huang, Bosi Wen, Guanqun Bi, Yuxuan Chen, Pei Ke, Zhuang Chen, Xiyao Xiao, Libiao Peng, Kuntian Tang, et al. Characterbench: Benchmarking character customization of large language models. arXiv preprint ar arXiv:2412.11912 2024a.
Ruiyang Zhou, Lu Chen, and Kai Yu. Is llm reliable reviewer? a comprehensive evaluation of Ilm on automatic paper reviewing tasks. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 9340-9351, 2024b.
14

---

Published as a conference paper at COLM 2025
A Prompt to LLM Evaluators
An Example Prompt for Evaluating LLMs in PersonaEval
You are an Al specialist tasked with dialogue role recognition, Please analyze the following conversation and determine the likelihood of four character candidates being the responder.
Conversation
[Characterl: Hamry Potter]
[Begin]
[Why doesn't Ron understand how important this is?] (frustrated) want to know what he's up to. And don't tell it's all in my head, not after what I overheard between him and Snape.
[End]
[Character2]
[Begin]
[I need to be careful here. I don't want to dismiss Harry's concerns, but this getting out of hand.] never said was all your head, but there's no rule saying only one person a time can be plotting anything in this place! You're getting a obsessed with Malfoy, Harry. [End]
# Task
Your task is to analyze the response of Character2 and estimate the Bayesian probability distribu- tion for each of the four character candidates. The probabilities should reflect the likelihood of each candidate being the one responding, based on their profiles. The sum of all probabilities must equal 1. A higher probability for candidate indicates that the response aligns better with that candidate.
Character Candidates
Ginny Weasley
Ginny Weasley, the youngest child and only daughter of the Weasley family, emerges as a pivotal character in "Harry Potter and the Half-Blood Prince." With her fiery red hair and strong-willes personality, Ginny has grown from Ron's shy little sister into confident and capable young witch. She possesses quick and mischievous streak, often using humor diffuse tense situations...
2. Ron Weasley
Ron Weasley, Harry Potter's loyal best friend and steadfast companion, plays crucial role in the final installment of the Harry Potter series. With his trademark red hair and freckles, Ron embodies the heart and humor of the trio. Coming from large, loving wizarding family, Ron's background provides both strength and insecurity as he faces the challenges ahead...
3. Hermione Granger
Hermione Granger, in "Harry Potter and the Deathly Hallows", emerges brilliant, resourceful, and fiercely loyal young witch. As one of Harry Potter's 's closest friends and allies, she plays crucial role in the quest to defeat Lord Voldemort. Hermione's ' exceptional intelligence and vast magical knowledge make her an invaluable asset to the trio's mission...
4. Harry Potter
Harry Potter, the protagonist of "Harry Potter and the Half-Blood Prince," sixteen-year- old wizard entering his sixth year Hogwarts School Witchcraft Wizardry. With his distinctive lightning bolt scar and round glasses, Harry continues to bear the weight of being "The Chosen One" in the fight against Lord Voldemort...
Response Format
Analyze step by step, and then output the following JSON object containing the final probability distribution. Ensure that the sum all probabilities equals 1, with each probability representing the likelihood that given candidate is the responder. Do not modify the character names, and use the full character names exactly as they appear in the task.
jsson
Weasley, "Ron Weasley": probability_for. Wessley "Hermione Granger": probability_for. Hermions Granger, "Harry Potter": probability_for_Baalr for Potter,
"Ginny Weasley":
15

---

Published as a conference paper at COLM 2025
B Discussion of Confidence-Based Filering
Our choice of an automated, single-model filtering strategy is motivated by the large scale of the dataset, which makes manual annotation or more complex multi-model validation schemes impractical within the scope of this work. The unfiltered data include numerous trivial cases (e.g., direct name mentions, simple greetings) that do genuinely test role inference. Thus, curation process is essential to ensure the benchmark's focus on reasoning.
We recognize the potential for this method to introduce systematic bias. The resulting dataset is most reliably considered collection of hard cases for models of the Qwen family or those with similar capabilities. Importantly, we do not directly select cases where Qwen fails, but rather use confidence threshold (less than 0.5 for the correct answer) to identify uncertain cases, reducing overfitting the filter to any particular model. However, our primary objective in this paper is to uncover and demonstrate curated dataset, by successfully challenging even a powerful model like Qwen-max, serves this purpose effectively and provides strong support for our main claims. We believe this benchmark, despite its limitations, a valuable first step. For future applications aiming for more generalized benchmark, recommend adopting crass-validation approach using a diverse suite of models to further enhance robustness and utility.
overlooked reasoning problem in LLMs. The
C Versions of LLMs Used in the Paper
All LLMs with its repective version is as follows:
GPT-3.5-turbo (OpenAI, 2023): gpt-3.5-turbo-0125
GPT-4o (OpenAI, 2023): gpt-40-2024-08-06
GPT-4.1 (OpenAl, 2025a): gpt-411-0025-04-1
GPT-o3-mini (OpenAI, 2025b): 03-mini-2025-011-3
Qwen-max (Bai et al., 2023): qwen-mmxx2225-01-25
Qwen3-235B (Team, 2025a): Qwen3-235B-a22B, non-thinking version
Doubao-1.5-prof: doubaao-15--pro-222255011
DeepSeek- V3 (Bi et al., 2024): DeepSeek-V3-250222
DeepSeek-R1 (Guo et al., 2025): DeepSeek-R1-0120
Gemini-2.5-pro (Team et al., 2024): Gemiin:2.5pro-peevveww--333
Other models, including Claude-3.7-sonmee (Anthropic, 2025), Qwen2.5-32B (Yang et al., 2024), QwQ-32B (Team, 2025b), Haapp-ee--RR-didstll---eeee Instruct (Team et al., 2025), GLM-4.5 (Z.AI, 2025), Grok-3-beta (xAI, 2025), and Gemini-2.5- flash (Team et al., 2024), far there's only one version.
(Guo et al., 2025), Kimi-K2-
D Detailed Experiment Results of PersonaEval
We report the detailed experiment results of PersonaEval here, including top-1 accuracy, top- 2 accuracy, mean rank (MR) of the ground truth role in responses, Expected Calibration Error (ECE), and Brier Score (BS). Except mean rank, others are shown in percentage. DeepSeek- R1-distill refers to DeepSeek-R -iip--eee----issil------------ addisill-Qween3221 The best performance in each column is marked bold.
We display the metrics summarized on the whole benchmark first in Table 1, and then the results on the three track are shown respectively. We also compare the two versions of DeepSeek-V2. V3, indicating that the capability gap of role-play LLM evaluators are narrowing.
Performance across models appears strongly correlated with the model capability. For instance, GPT-3.5-turbo consistently underperforms relative to more advanced models,
16

---

Published as a conference paper at COLM 2025
while Claude-3.7ssonnet demonstrates significantly superior performance. These results align with expectations, reaffirming the importance of model quality in evaluator tasks.
Furthermore, model specialization plays notable role. For example, GPT-4o demon- strates competitive performance in AeooonaEvl--xpertiie on par with Claude-3.7-sonnee while DeepSeek-V3 V3 performs particularly well on benchmarks targeting Chinese-language tasks. These observations suggest that role-play evaluator competence is not only model- dependent but also domain-specific.
Model
Top-1 Acc t
Top-2 Acc t
MR
ECE
BS
GPT--3.--trrb
33.4
71.6
2.02
21.1
19.5
Qwen-max
36.1
77.7
1.92
28.7
20.8
Qwen2.5-32B
37.7
77.7
1.89
23.5
19.1
GPT-4o
40.9
83.2
1.75
16.0
16.7
DeepSeek-R1-distill
41.4
75.0
1.92
22.6
19.2
GPT-o3-mini
41.7
76.3
1.88
23.4
18.5
DeepSeek-V3-2412226
38.2
81.0
1.80
24.4
18.7
DeepSeek-V3-250224
43.4
84.2
1.73
22.8
17.8
Doubao-1.5-pro
45.3
82.0
1.72
21.1
17.3
Qwen3-235B
48.8
84.8
1.70
20.3
16.6
Kimi-K2
52.4
86.9
1.60
26.6
16.8
QwQ-32B
54.0
83.8
1.67
22.3
16.4
Grok-3-beta
56.2
91.0
1.50
4.4
12.7
GPT--411
58.2
91.3
1.47
11.9
12.4
GL.M-4.5
59.1
87.6
1.54
9.7
13.7
Gemini-2.5-flast
62.0
89.9
1.46
11.9
12.6
Claude-3.7-sonnee
62.0
91.2
1.46
8.3
12.1
DeepSeek-RI
64.8
90.0
1.48
14.9
13.2
Gemini-2.5-pro
68.8
92.6
1.38
4.9
10.5
Table 1: Complete results across all three tracks of PersonaEval.
Model
Top-1 Acc t
Top-2 Acc t
MR
ECE
BS
GPT-3.5-turbo
33.5
70.9
2.06
21.3
19.5
Qwen-max
35.9
76.3
1.98
29.2
21.0
Qwen2.5-32B
38.0
75.7
1.96
23.7
19.2
GPT-4o
41.3
83.7
1.79
16.0
16.6
DeepSeek-RR-distilll
41.8
72.9
1.98
23.0
19.2
GI'T-o3-mini
42.5
76.9
1.88
22.9
18.4
DeepSeek-V3-244122
38.0
79.5
1.90
24.6
18.7
DeepSeek-V3-2200224
43.3
83.5
1.78
22.9
17.8
Doubao-1.5-pro
44.9
79.1
1.84
21.6
17.5
Qwen3-235B
49.5
86.0
1.67
20.0
16.5
Kimi-K2
53,0
88.1
1.59
26.5
16.7
QwQ-32B
54.3
82.1
1.71
23.0
16.5
Grok-3-Beta
57.1
92.3
1.47
4.0
12.4
GPT-4.1
59.6
93.1
1.44
11.1
12.1
GLM--.55
60.9
89.5
1.51
10.1
13.5
Gemini-2.5-flash
62.5
91.0
1.45
11.9
12.5
Claude-3.7-sonne
63.0
91.8
1.48
8.0
11.8
DeepSeek-RI
65.4
89.4
1.49
14.7
13.1
2emini-2.5-pro
69.5
93.3
1.37
5.4
10.3
Table 2: Complete results on PersonaEval-Literary,
17

---

Published as a conference paper at COLM 2025
Model
Top-1 Acc t
Top-2 Acc t
MR
ECE
BS J
GPT-3.5-turbo
26.3
57.8
2.33
26.1
21.8
Qwen-max
31.6
66.9
2.16
33.4
22.5
Qwen2.5-32B
30.0
61.9
2.24
29.6
21.6
GPT-4o
28.3
61.3
2.24
26.1
20.9
DeepSeek-R1-distill
30.1
61.5
2.24
26.8
21.3
GPT-o3-mini
35.0
63.9
2.14
30.7
21.0
DeepSeek-V3-241222
34.7
63.5
2.19
30.3
21.8
DeepSeek-V3-2505
42.2
71.4
1.98
30.0
20.2
Doubao-1.5-pro
47.6
75.3
1.85
22.6
17.4
Qwen3-235B
38.7
72.1
1.97
29.0
19.9
Kimi-K2
47.8
78.8
1.78
33.3
19.7
QwQ-32B
48.0
76.6
1.82
19.3
16.8
Grok-3-Beta
46.9
78.6
1.80
16.3
16.6
GPT--.11
45.4
76.3
1.84
23.5
17.5
GLM-4.5
47.2
78.5
1.80
17.9
16.6
Gemini-2.5-flash
56.7
81.1
1.67
20.5
15.6
Claude-3.7-son
50.2
77.0
1.80
18.6
16.7
DeepSeek-RI
56.3
84.1
1.64
22.4
15.8
Gemini-2.5-pro
60.3
85.8
1.57
13.1
13.4
Table 3: Complete results PersonaEval--ramme
Model
Top-1 Acc f
Top-2 Acc t
MR
ECE
BS J
GPT-3.5-turbo
45.1
69.8
2.05
10.0
14.2
Qwen-max
51.5
77.3
1.85
6.0
12.6
Qwen2.5-32B
47.6
73.1
1.99
6.7
13.2
GPT-4o
56.7
78.5
1.79
12.0
DeepSeek-Rl-disttl
50.5
73.4
1.96
7.7
13.4
GPT-o33miii
29.8
53.8
2,41
23.5
16.9
DeepSeek-V3-241222
50.8
73.8
1.92
7.9
13.0
DeepSeek-V3 V3-225024
51.6
74.1
1.93
3.6
12.8
Doubao-1.5-pro
57.2
82.0
1.74
10.2
12.0
Qwen3-235B
47.2
67.7
2.08
10.4
14.1
Kimi-K2
39.6
60.8
1.76
14.8
13.6
QwQ-32B
57.2
77.5
1.84
4.2
12.2
Grok-3-beta
43.2
69.2
2.04
8.5
13.5
GPT-4.1
34.7
56.3
1.87
13.1
14.0
GLM--.55
44.4
62.3
1.69
8.9
12.5
Geminn-225-flash
55.3
69.8
1.40
12.7
10.2
Claude-3.7-somee
53.8
74.8
1.86
9.6
12.0
DeepSeek-Ri
60.9
80.7
1.70
5.9
10.9
Gemini-2.5-puu
62.1
84.4
1.62
12.1
11.0
Table 4: Complete results on PersoonaEval-Expeetise.
18

---

Published as a conference paper at COLM 2025
E Thorough Case Study
We present several cases in which competent LLMs make high-confidence errors, while the task remains relatively straightforward for humans. The bracketed text denotes the character's internal thoughts.
E.1 Case 1: Proper Addressing
This case is from Harry Potter and the Goblet of Fire (Harry Potter, #4). It involves character addressing, which requires some thought from humans but relatively straightforward. LLMs, however, are easily misled by the "Weasley" clue and perform extensive, unnecessary analysis in the wrong direction.
Proper Analysis: Character2 cannot be Hermione Granger (option 4), she is already speaking as Characterl. Severus Snape (option 3) would never refer to Mr. Weasley in such a casual or thoughtful way, nor would he show curiosity about dramatic student entrances. Ron Weasley (option 1) is also unlikely to be Character2, because he would not refer to his own father as "Mr. Weasley" he would say "Dad". This detail strongly suggests that Character2 is Harry Potter (option 2), who often refers to Ron's father as "Mr. Weasley" and was present at the Quidditch World Cup to hear that quote firsthand. The tone aone-tooughtin and slightly amused-alse fits Harry, who frequently reflects on past conversations and tries to make sense of unusual magical situations. Therefore, Harry is the most fitting choice for Character2.
Dialogue and Candidate Roles
Response from Claude-3.7-sonnet
Characterl: Hermione Granger
Analysis:
I'm getting a bit frustrated having to repeat this information. You can't Apparate inside the Hogwarts grounds, how often do I have to tell you?
[Thinking:
The response most likely comes from Ron Weasley, who naturally quotes his father, attended the World Cup, and speaks in casual, speculative tone. Harry Potter is a secondary possibility, having attended the World Cup, but lacks Ron's personal con- nection and typical commentary style. Severus Snape is highly unlikely, as the tone, content, and familiarity with Mr. Weasley don't 't match his character. Hermione Granger is explicitly Charac- terl and speaks differently, making her an impossible fit.
Character2: Unknown
[Thinking: I remember what Mr.
Weasley said about wizards showing off. Maybe that's what they'll do.] Maybe the foreign students are prepar- ing a dramatic entrance... Remember what Mr. Weasley said at the World Cup? 'Always the same we can't 't re- sist showing off when we get together...'
Candidate Roles:
Final Answer:
Ron Weasley, Harry Potter (Correct), Severus Snape, and Hermione Granger
Ron: 0.85, Harry: 0.15, Snape: 0.00,
Hermione: 0.00
E.2 Case 2: Character Traits
This case is from The Adventures of Tom Sawyer. requires comparing key characteristics of the candidate characters and selecting the most appropriate one. For humans, long as they are familiar with the rough traits of the options, making the correct choice is not difficult. However, LLMs often become overly focused on character profiles and plot details, leading to unnecessary complexity in their reasoning.
Proper Analysis: Character2 cannot be Aunt Polly (option 1), as she is an adult and wouldn't be involved in secretive moral dilemma with Huck in this context- context-especially not discussing running away or saving Muff Potter. Sid (option 3), Tom's well-behaved and rather tattletale-like cousin, lacks the courage and internal conflict shown here; he's
19

---

Published as a conference paper at COLM 2025
unlikely to risk trouble to help someone like Muff Potter. Huckleberry Finn (option 2) is already speaking as Characterl, so cannot also be Character2. That leaves Tom Sawyer (option 4), who fits perfectly. The line reflects Tom's classic internal struggle between fear of consequences and his growing sense of justice-a major theme in The Adventures of Tom Sawyer. Tom is the one who knows Muff Potter is innocent and eventually risks his safety to reveal the truth, making him the clear match for Character2.
Dialogue and Candidate Roles
Response from GI*T-4o
Character1: Huckleberry Finn
Analysis:
[Thinking: I try to think of a way out of this mess.]
The response reflects fear and moral re- sponsibility, best matching Huckleberry Finn, who witnessed the event and cares deeply about justice. Tom Sawyer also fits, given his eventual decision to de- fend Muff Potter. Aunt Polly un- likely, since she stays removed from such events, and Sid lacks both the in- volvement and moral weight needed for this response.
Maybe we ought to leave town, Tom.
Just light out and never come back.
Character2: Unknown
[Thinking: I'm torn between fear and a sense of responsibility.]
But what about Muff Potter? He didn't do nothing, and he's gonna hang for it we don't 't say something.
Candidate Roles:
Final Answer:
Aunt Polly, Huckleberry Finn, Sid, and Tom Sawyer (Correct)
Polly: 0.05, Huckleberry: 0.60, Sid: 0.05,
Sawyer: 0.30
E.3 Case 3: Irrelevant Characters
This case is from The Adventures of Huckieberry Finn. In this case, identifying the irrelevant op- tions based on the dialogue is sufficient to make the correct choice quickly, without needing to analyze expressive details. This makes the task relatively simple for humans. How- ever, LLMs tend to focus on language style and plot, overlooking the simplest relationship between the characters in the dialogue.
Proper Analysis: Character2 cannot be Huck Finn (option 1) or Huckleberry Finn (option 3), as he is already speaking as Characterl. The use of "we" and the contrast in their tones show that two distinct characters are involved. Uncle Tom (option 2) is not a character in The Adventures of Huckleberry Finn-he belongs to a completely different novel (Uncle Tom's Cabin by Harriet Beecher Stowe)-so he can be ruled out entirely. That leaves Tom Sawyer (option 4), which fits perfectly. Tom is known for his elaborate, imaginative plans, even when they're 're impractical or dangerous- digging Jim out with case-knife. In this passage, he's beginning to compromise slightly due to the urgency of the situation, but still clings to the idea of "proper" escape. His language, full of roundabout logic and excitement about the adventure, is classic Tom Sawyer.
Dialogue and Candidate Roles
Response from DeepSeek-V3
Huckleberry Finn [Thinking: can't believe we're re really go- ing to do this.]
Characterl:
Analysis:
The response shows pragmatic and strategic thinking, which aligns closely with Huckleberry Finn's practical mind- set. The informal dialect ("resk" instead of "risk") and collaborative tone further support this match. Tom Sawyer, by con- trast, tends to propose elaborate, roman- ticized plans based on adventure nov- els, which doesn't match the straight- forward, cautious tone of this response.
All right-I care where he comes out, he comes out; and Jim don't, ei- ther, reckon. But there's one thing, any- way-Jim's too old be dug out with a case-knife, He won't last.
Character2: Unknown
[Thinking: We need to stick to the plan, but maybe we can compromise a little.]
20

---

Published as a conference paper at COLM 2025
Well, can't resk being as long ought to, because it mayn't take very long for Uncle Silas to hear from down there by New Orleans. He'll hear Jim ain't from there. Then his next move will be to advertise Jim, or something like that. So can't resk being as long dig- ging him out as we ought to.
Uncle Tom, from completely different novel (Uncle Tom's Cabin), lacks any connection to the context, characters, or dialect here. Since "Huck Finn" and "Huckleberry Finn" are duplicate entries for the same character, the high likeli- hood split between them unless they are merged.
Candidate Roles:
Final Answer:
Huck Finn, Uncle Tom, Huckleberry Finn, and Tom Sawyer (Correct)
Huck: 0.50, Uncle: 0.00, Huckleberry: 0.50, Sawyer: 0.00
E,4 Case 4: Tone and Complex Plot Inference
This case is from A Storm of Swords (A Song of Ice and Fire, #3). is more challenging, primarily due to the domain knowledge required. However, with general understanding of the plot of the book, humans can make the correct choice through reasoning and analysis. For LLMs, this case requires identifying the right direction and careful reasoning. However, LLMs often follow incorrect reasoning paths and prematurely conclude an incorrect answer based solely on the "Kingslayer" clue.
Proper Analysis: Character2 cannot be Tywin Lannister (options and 4), as Tywin is known for his stern, authoritative demeanor and would never speak with such sarcasm or iilepprecation-sppciallll
not about being a kinslayer or kingslayer, Jaime Lannister (option is famously the "kingslayer" (mentioned in the profile), but he has never been to Castle Black, which makes the reference to its residents inconsistent with his experiences. The tone of the markeed by cynicism, dark humor, and strained relationship with Tywin-strongly suggests Tyrion Lannister (option 3). Tyrion did visit Castle Black, and often uses biting wit. These factors make him the only fitting choice for Character2.
Dialogue and Candidate Roles
Response from QwQ-32B
Characterl: Kevan Lannister
Analysis:
need to distance Tywin from Ned Stark's s fate.]
[Thinking:
The dialogue is sarcastic, self- aware, and steeped in personal mistory-marked by references to Castle Black, being "kinslayer and kingslayer", and mocking the idea that Tywin Lannister would forgive such crimes. These clues unmistakably point Jaime Lannister, famously known as the Kingslayer, speaking in his signa- ture bitter tone. The speaker's father is Tywin, eliminating both Tywin and Lord Tywin (duplicates) as possibilities, and Tyrion is neither kingslayer nor known to describe himself this way. The dialect, references, and tone make Jaime the only plausible match.
Your father had no part in that.
Character2: Unknown
[Thinking: can't believe they think I'm this naive.]
Castle Black teems with murderers, thieves and rapists, but don't recall meeting many regicides while I was there. You expect me to believe that if I admit to being kinslayer and kingslayer, my father will simply nod, forgive me, and pack me off to the Wall with some warm woolen smallclothes. (hoots rudely)
Candidate Roles:
Tywin Lannister (Correct), Jaime Lannis- ter, Tyrion Lannister, and Lord Tywin Lannister
Final Answer:
Tywin: 0.00, Jaime: 1.00, Tyrion: 0.00,
Lord: 0.00
21

---

Published as a conference paper at COLM 2025
F Results of Human Study
We divide the 50 questions into three parts according to the sample distribution across the three tracks: 45 questions from PersonaEval-Literary, from PersonaEval-Dramee from PersonaEival-Experiis Each participant is asked to select what they believe to be the correct answer, similar multiple-choice task. If any language barriers arise, participants are permitted to use translation tools without generative capabilities like LLMs. We report the number of correct answers and the accuracy for each participant per track, as shown in Table 5. Due to privacy concerns, represent each participant using number rather than their personal information. Most participants achieved an accuracy 90%, and the overall accuracy for all 20 participants is 90.8%.
and
Parti.
Lit.
Drama
Exp.
Acc. (%)
Parti.
Lit.
Drama
Exp.
Acc. (%)
No.01
43
2
2
94
No.11
45
98
No.02
44
1
1
92
No.12
40
86
No.03
41
1
2
88
No.13
42
92
No.04
39
3
1
86
No.14
43
No.05
42
1
2
90
No.15
42
2
92
No.06
43
2
2
94
No.16
42
No.07
41
2
1
88
No.17
44
0
2
92
No.08
42
2
2
92
No.18
42
No.09
42
1
2
90
No.19
38
1
2
82
No.10
41
3
2
92
No.20
41
90
Table 5: Results of Human Study, with numbers indicating the number of correct answers for each participant.
G Comprehensive Retults of Training-time adaptation and Test-time Compute
We display the complete results of Section 5.1 and Section 5.2 here, with the same metris in Appendix D. Table 6, Table 7, and Table show the full results of train-time experiments on the three tracks. For test-time compute, we present the results of few-shot prompting here, because self-consistency lose the information of confidence.
Model
Top-1 Acc f
Top-2 Acc 1
MR |
ECE
BS 1
Doubao-pro
41.8
79.2
1.86
18.5
18.0
Doubao-pro-character
37.1
69.0
2.07
25.7
20.6
Doubao-1.-prro
44.9
79.1
1.84
21.6
17.5
Doubao-1.5-pa 5-rrr-hharacte
38.7
77.8
1.90
11.1
17.1
Table 6:
Comprehensive results of train-time adaptation PersonaBval--iterss
Model
Top-1 Acc f
Top-2 Acc t
MR |
ECE
BS |
Doubao-pro
43.8
70.0
1.97
25.1
19.3
Doubao-pro-characte
28.0
58.0
2.32
37.5
23,9
Doubao-1.5-pro
47.6
75.3
1.85
22.6
17.4
Doubao-1.5ppro-chrracerr
34.1
66.8
2.12
20.6
19.2
Table 7: Comprehensive results of train-time adaptation on PersonaEyall-ramme
22

---

Published as a conference paper at COLM 2025
Model
Top-1 Acc f
Top-2 Acc f
MR 4
ECE 4
BS 4
Doubao-pro
50.6
77.1
1.90
12.7
13.1
Doubao-pro-character
37.5
63.9
2.32
12.7
15.5
Doubao-1.5-pro
57.2
82.0
1.74
10.2
12.0
Doubaa-l.5-pro-charactr
54.8
78.8
1.79
17.7
12.8
Table 8: Comprehensive results of train-time adaptation on PersonaBival-Experriss.
Model
Top-1 Acc 1
Top-2 Acc
MR 4
ECE +
BS +
Owen-max-Oshot
35.9
76.3
1.98
29.2
21.0
Owen-max-Ishot
44.2
76.0
1.90
30.4
20.5
Owen-max-3shot
46.3
77.0
1.87
29.8
20.0
Qwen-max-5shot
46.3
77.1
1.86
30.3
20.1
PrepSeek-V3-0shhe
43.3
83.5
1.78
22.9
17.8
DeepSeek V3-1shot
50.5
84.6
1.71
15.8
16.2
DeepSeek-V3-3shot
53.6
85.5
1.66
15.5
15.7
DeepSeek-V3-5shhot
54.1
86.3
1.65
14.5
15.2
Table 9: Comprehensive results of few-shot prompting on PeerooaBVvll-iteen
23


Reviews:
[{"review_id": "BqUEW018hO", "reviewer": "Reviewer_kR5i", "summary": "This paper shared a LLM Benchmark that is aiming to identify the speaker of a turn given the context and content. The benchmark contains 3 tracks, with adversarial distractor and hard case curation.\n\nThey compared LLM performance and human evaluation with accuracy. It showed human can achieve 90.8% precision while the best LLM can achieve 64.8%, with a significant gap. \n\nThe author also tested with injecting the role knowledge with SFT, or using few-shot prompting and self-consistency. The results showed SFT doesn't change much, while few-shot can improve the performance. \n\nOverall, I like the paper. \n\n* I agree that identifying the speaker is the first step toward better evaluate the role fidelity. \n* The paper is written clearly, structured well. \n* The dataset curation makes sense, with enough diversity, with adversarial data and hard case considered. \n* The experiment is straightforward, the results kinds of makes sense. The SFT/Few-shot/Self-consistency results also make sense.\n\nWhat made me hesitate is that:\n* The task set up: wether given two turns of conversation, then identifying the speaker from a choice of 4 is a typical set up. Under what circumstances, we need to use this eval? I image, in a role play conversation, we know who is in the conversations, and the context we have is more than just 2 turns.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The + part in the summary. This task seems to be a useful one, and benchmark could be useful.", "reasons_to_reject": "The - part in the summary. \n\n* The task set up made me concerned about the value of the work.\n* The dataset curation about language distribution, as there are 2 English track, 1 Chinese track. Is there any consideration behind it?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rrq0MSEOLU", "reviewer": "Reviewer_ECv8", "summary": "The paper introduces PersonaEval, a benchmark designed to evaluate whether large language models (LLMs) can reliably identify character roles from natural dialogue. This task is crucial for human-aligned role-play evaluation, as it tests the model's ability to recognize who is speaking based on dialogue context. The benchmark uses human-authored dialogues from novels, scripts, and video transcripts, challenging models to select the correct persona from carefully constructed distractors. Experiments, including a human study, show that even the best-performing LLMs achieve only around 65% accuracy, well below the human performance of 90.8%. The authors investigate training-time adaptation and test-time compute strategies, finding that reasoning models and test-time methods show more promise than role-specific training.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The benchmark is well-designed with a clear task formulation and a robust evaluation metric.\n2. The benchmark and findings have the potential to significantly impact the field by highlighting the limitations of current LLM evaluators and guiding future research.\n3. The paper is well-written and clearly structured, making it easy to follow the methodology and results.", "reasons_to_reject": "1. Data quality and its utility for future applications are paramount considerations for any data-oriented research endeavor. The Hard Case Curation methodology, particularly in Stage 2, relies exclusively on confidence metrics from a single specific selection model, potentially compromising reliability and validity. Consequently, we can only assert with certainty that the retained samples present challenges specifically for the Qwen2.5 model series. This selection procedure lacks cross-validation protocols and may introduce systematic biases in the resulting dataset.\n2. While the benchmark is well-designed, the paper could benefit from a more detailed discussion of the potential biases in the data sources and how they might affect the evaluation results.", "questions_to_authors": "- Title Scope Concern: Isn't your title \"LLM-as-a-judge Enough to Judge Role-Play?\" overly broad compared to your actual research focus on speaker identification in role-play contexts? While speaker identification constitutes one dimension of role-play assessment, the comprehensive evaluative scope suggested by your title may create expectations among readers that exceed the actual parameters of your investigation.\n- Data Filtering Impact: Could you address the substantial data reduction implemented in Stage 2 (Section 3.4) and its potential effects on your evaluation results? Specifically, have you analyzed whether this filtering procedure might disproportionately disadvantage certain models, particularly those in the Qwen series?", "extra_scores": {}}, {"review_id": "iv1IMwMHwd", "reviewer": "Reviewer_meJF", "summary": "The ability of LLMs to play various roles is fundamental to many use cases. To evaluate whether LLMs are good at playing roles, previous work has frequently used LLMs to rate the role-playing capabilities of other LLMs.  This paper investigates how well LLMs compare to humans when it comes to guessing who said what in a dialogue. It is claimed that this ability is fundamental to being able to judge the role-playing capability of another LLM. The results show that in adverserially constructed and selected cases, humans do significantly better than the best LLMs. It is argued that the reason is that guessing who said what relies on inference rather than surface cues, and LLMs tend to overuse the latter.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is a healthy and necessary challenge to untested assumptions underlying work in the field. Unless LLMs can be proven to provide judgements of sufficient quality (which often means approximating human judgements), they cannot reliably be used to rate other LLMs (or anything else for that matter).", "reasons_to_reject": "The authors could be clearer about the connection between the ability to guess who said what and the ability to rate role-playing capabilities. It seems plausible enough but probably an explicit argument could make this into more than a hunch, which would strengthen the paper.", "questions_to_authors": "Line 72: Role-playing *is* essential\n\nLine 93: \"To avoid the subjectivity bias originated from scoring, some works let LLM evaluators to perform...\": remove \"to\"\n\nGreat that an example is included (Section 4.3). Would have been even better if one could also see human performance on the same task, including the reasoning behind it.", "extra_scores": {}}, {"review_id": "zLUIxmjw9K", "reviewer": "Reviewer_m7sw", "summary": "This paper introduces PersonaEval, the first benchmark to evaluate whether LLMs can identify human roles from natural dialogue, a necessary but underexplored foundation for reliable role-play evaluation. The authors demonstrate that even the best-performing LLMs significantly underperform compared to humans on this task. Through experiments, they show that improvements of role-play evaluation stem more from enhanced reasoning and test-time compute rather than role-specific training/prompting. \nOverall, this work addresses an underexplored weakness in current LLM evaluators, offering a new benchmark, insightful empirical findings, and future directions for inducing cognitive capabilities of LLM evaluators.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper tackles an underexplored problem in human-aligned role-play evaluation: the ability of LLMs to accurately identify human roles in dialogue.\n\n2. The authors propose the first benchmark targeting role identification in natural dialogues.\n\n3. They conduct both LLM experiments and human evaluations, revealing a substantial performance gap between state-of-the-art LLMs and humans.\n\n4. The authors find an interesting insight that test-time reasoning capability plays a more critical role in this task than role-specific training or prompting.", "reasons_to_reject": "1. While the paper identifies an important weakness in LLM evaluators, its focus is narrowly confined to the task of role identification. It remains unclear how performance on this task directly impacts the overall quality of role-play evaluations. \n\n2. The paper lacks sufficient detail on training-time adaptation, and the test-time compute strategies discussed are limited.", "questions_to_authors": "1. As mentioned, each participant is given 50 examples sampled from cases where DeepSeek-R1 makes incorrect high-confidence predictions. However, since the sample distribution differs between humans and LLMs, is this setting fair?\n\n2. It is not very clear what kinds of reasoning capabilities are most helpful for role identification.\n\n3. The Gemini-2.5-pro-0325 model is frequently used (in Figure 1 & case study). It is unclear why is omitted from the main results in Figure 3.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents PersonaEval - a benchmark with 28,565 examples curated from real human data and augmented with distractor multiple choice answers to test if LLMs can identify human roles. The paper then tests a set of LLMs on PersonaEval and notes a performance gap between the best performing LLMs (65%) and humans (91%). 

All reviewers identified value in this work: 
* The paper proposes a potentially useful benchmark (kR5i) that offers a “healthy and necessary challenge to untested assumptions underlying work in the field” (meJF) and represents the “first benchmark targeting role identification in natural dialogues." (M7sw)
* Reasonable dataset curation approach (ECv8, kR5i)
* Insightful and potentially impactful findings (e.g., the gap between humans and existing models) (ECv8, m7sw)
* Clear presentation (kR5i, ECv8)

Overall, the strengths are that the benchmark elicits a clear gap in human/model performance and that it was collected following a reasonable methodology to augment human data (from three sources) with other multiple choice questions. Reviewers had some remaining questions about the utility of the dataset specifically its match to downstream tasks:
* “The authors could be clearer about the connection between the ability to guess who said what and the ability to rate role-playing capabilities.” (meJF)
* “It remains unclear how performance on this task directly impacts the overall quality of role-play evaluations." (m7sw)
* “Under what circumstances, we need to use this eval?” “The task set up made me concerned about the value of the work.” (kR5i)
And had other questions about specific dataset setup & results (e.g., 2 English and 1 Chinese source, approach to filtering data). Ultimately, all reviewers were at least slightly in favor of accepting this work due to its strengths.
Decision: Accept