PingPong: A Benchmark for Role-Playing Language Models with User
  Emulation and Multi-Model Evaluation
Authors: name: Ilya Gusev
Published: 2024-09-10 19:00:44

Abstract: We introduce a novel benchmark for evaluating the role-playing capabilities
of language models. Our approach leverages language models themselves to
emulate users in dynamic, multi-turn conversations and to assess the resulting
dialogues. The framework consists of three main components: a player model
assuming a specific character role, an interrogator model simulating user
behavior, and a judge model evaluating conversation quality. We conducted
experiments comparing automated evaluations with human annotations to validate
our approach, demonstrating strong correlations across multiple criteria. This
work provides a foundation for a robust and dynamic evaluation of model
capabilities in interactive scenarios.