Role-Playing Evaluation for Large Language Models
Authors: name: Yassine El Boudouri; name: Walter Nuninger; name: Julian Alvarez; name: Yvan Peter
Published: 2025-05-19 14:18:16

Abstract: Large Language Models (LLMs) demonstrate a notable capacity for adopting
personas and engaging in role-playing. However, evaluating this ability
presents significant challenges, as human assessments are resource-intensive
and automated evaluations can be biased. To address this, we introduce
Role-Playing Eval (RPEval), a novel benchmark designed to assess LLM
role-playing capabilities across four key dimensions: emotional understanding,
decision-making, moral alignment, and in-character consistency. This article
details the construction of RPEval and presents baseline evaluations. Our code
and dataset are available at https://github.com/yelboudouri/RPEval