PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluation

## Abstract

Abstract We introduce a novel benchmark for evaluating the role-playing capabilities of language models. Our approach leverages language models themselves to emulate users in dynamic, multi-turn conversations and to assess the resulting dialogues. The framework consists of three main components: a player model assuming a specific character role, an interrogator model simulating user behavior, and a judge model evaluating conversation quality. We conducted experiments comparing automated evaluations with human annotations to validate our approach, demonstrating strong correlations across multiple criteria. This work provides a foundation for a robust and dynamic evaluation of model capabilities in interactive scenarios.

## 1 Introduction

Language models, which predict plausible language, have dominated natural language processing since BERT Devlin et al. (), with models like ChatGPT Ouyang et al. () showcasing advanced conversational capabilities.

In this paper, we focus on role-playing language models for entertainment purposes. These models are assigned specific characters or personas and are tasked with maintaining these roles while engaging and entertaining users. There are other important applications of role-playing language models, but they are out of the scope of this paper.

Figure: Figure 1: An overview of the benchmark.
Refer to caption: extracted/5846477/pingpong_v2.drawio.png

We introduce a novel benchmark for evaluating role-playing language models. We believe direct interaction is the most effective way to assess a language model’s conversational abilities. However, humans often lack the time to engage with new models, and many popular benchmarks are limited to single-turn interactions. These benchmarks are also becoming less reliable due to test data contamination. To address this, our paper proposes using language models to emulate users in role-playing conversations and automatically evaluate the resulting dialogues.

Our methodology, illustrated in Figure 1, involves three key components: a player model assuming a character role, an interrogator model simulating user behavior, and a judge model evaluating conversation quality.

Our contributions:

- •
We propose a dynamic, multi-turn benchmark for assessing LLM role-playing abilities
- •
We mitigate individual model biases by using a multi-model evaluation system
- •
We validate the benchmark through correlation with manual annotations

All the results, prompts, and scripts are available online(^1^11[https://github.com/IlyaGusev/ping_pong_bench/](https://github.com/IlyaGusev/ping_pong_bench/)). The benchmark website(^2^22[https://ilyagusev.github.io/ping_pong_bench/](https://ilyagusev.github.io/ping_pong_bench/)) has the final leaderboards and all the conversations with example-wise scores. It is available for English and Russian languages.

## 2 Related work

Automatic evaluation. LLM-as-a-Judge Zheng et al. () is an evaluation method that relies on strong language models, such as GPT-4, instead of humans. Popular benchmarks using this method include AlpacaEval, EQ-bench, and BiGGen Bench Dubois et al. (); Paech (); Kim et al. (). The validity of these benchmarks relies on their high correlation with human annotations, specifically with Chatbot Arena Chiang et al. (). However, all these benchmarks rely on a single model as a judge, which may introduce various biases, including the self-evaluation bias Panickssery et al. (); Xu et al. ().

Multi-turn benchmarks. Moreover, all aforementioned benchmarks for language models are single-turn, which contrasts with the real-world usage of LLMs. There are also multi-turn benchmarks such as MT-Bench-101 Bai et al. () and MT-Eval Kwan et al. (). Even so, they focus on specific capabilities, and their evaluation procedures still differ from how humans implicitly rate language models.

Data contamination. A major problem for these static public benchmarks is data leakage into the pre-training datasets of language models Deng et al. (). It’s challenging to avoid contamination since such tests are usually stored online and considered "code" during pre-training. This can occur even without malicious intent from model creators. The most obvious solution is to close benchmarks completely, which requires trusting benchmark organizers, which is difficult in a highly competitive environment. Alternative solutions include regularly updating benchmarks with new test data White et al. () or dynamically generating test data using existing language models.

Role-play capabilities. Another area of research is the role-play capabilities of language models. Various commercial services exploit these abilities, including Character.ai(^3^33[https://character.ai/](https://character.ai/)) and Chai Irvine et al. (). There are academic and community attempts to create similar systems with open datasets, code, and models, such as PIPPA Gosling et al. (), ChatHaruhi Li et al. (), Character-LLM Shao et al. (), MythoMax(^4^44[https://huggingface.co/Gryphe/MythoMax-L2-13b](https://huggingface.co/Gryphe/MythoMax-L2-13b)), or Magnum(^5^55[https://huggingface.co/anthracite-org/magnum-v2-123b](https://huggingface.co/anthracite-org/magnum-v2-123b)).

Role-play evaluation. Several static benchmarks for role-playing exist, including ECHO, InCharacter, CharacterEval, and PersonaGym  Ng et al. (); Wang et al. (); Tu et al. (); Samuel et al. (). PersonaGym is close to our work, featuring dynamic question generation based on the environment ("situation" in our terminology) and the currently selected persona. There is also a very similar dynamic benchmark, RPBench-Auto(^6^66[https://boson.ai/rpbench-blog/](https://boson.ai/rpbench-blog/)). It is based on the same assumptions and features and has a structure similar to one of the versions of our benchmark. The major difference from our work is that evaluation is based on side-by-side comparisons with the baseline model, while we produce single-point evaluations.

Multi-model and cross-model evaluation. PoLL Verga et al. () authors aggregate evaluations from different language models in a similar way we do, with average pooling. They show that ensembling different models for evaluation increases correlation with human annotations. There is also another more agentic approach Chan et al. () with a referee team.

## 3 Methodology

### 3.1 Role definitions

Our framework comprises three principal roles: player, interrogator, and judge, inspired by the Turing test Turing (). However, our approach differs in the number of players, the player’s objective, and the use of machine-based interrogators and judges.

Language models can take three possible roles.

- •
Player assumes the role of a specific character based on a provided character card.
- •
Interrogator engages with the player within a given situation or towards a specific goal, simulating user behavior.
- •
Judge evaluates the player’s responses against predetermined criteria.

Role assignments are implemented through a combination of system and user prompts. For models lacking dedicated system prompts, such as Gemma 2 GemmaTeam (), all instructions are incorporated into the user prompt.

This setup is asymmetrical. It is intentional since typical use cases of role-playing models are asymmetrical. However, it is possible to modify it to make it symmetrical by providing character descriptions and situations both to the player and the interrogator. Symmetrical setups might be useful for other domains.

### 3.2 Judge

The scoring is single-point. There are no reference examples or pairs. We used three main criteria for evaluation by the judge:

- •
Character consistency: The player’s answers align perfectly with an assigned character; they correspond to the character’s description.
- •
Entertainment value: The player’s responses are engaging and entertaining.
- •
Language fluency: The player’s language use is of the highest quality, without any mistakes or errors. The player is perfectly fluent.

These criteria reflect the main things we expect from the model during role-playing. In addition to them, we ask whether the player refused to answer.

We prompt a model to explain itself before giving a score using quotes from the conversation. It must also return a set of scores for every turn of a conversation.

### 3.3 Version 1: combined interrogator and judge

In the initial version, the roles of interrogator and judge were merged. This combined entity receives the player’s character card, a situational context, and a list of evaluation criteria. It evaluates the player’s most recent response and generates the subsequent user utterance.

We selected claude-3-5-sonnet as the interrogator/judge model based on the Judgemark(^7^77[https://eqbench.com/judgemark.html](https://eqbench.com/judgemark.html)) results, hypothesizing a correlation between creative writing and role-play capabilities. The evaluation uses a 10-point scale for every criterion.

### 3.4 Version 2: separated soles and multi-model evaluation

Recognizing the limitations of the combined approach, we developed a second version with distinct interrogator and judge roles. This separation addresses three key issues:

- •
Realistic user emulation: In many real-world use cases, users lack complete information about character profiles, and to correctly emulate it, we should not provide complete character information to the interrogator.
- •
Optimized costs: It is possible to replace an interrogator with a cheaper one since its task is easier than judgment.
- •
Optimized decoding strategies: Separating roles allows for tailored decoding strategies for interrogation and judgment tasks. For instance, a higher temperature benefits the interrogator but not the judge.

Furthermore, we identified the inadequacy of single-model evaluation. To address this, we implemented a multi-model evaluation system. This approach involves averaging scores from different judge models. In this particular setup, we used Claude 3.5 Sonnet and GPT-4o, the top two models, by correlation with manual annotations.

As an interrogator, we took GPT-4o Mini. According to version 1, it has the same generation quality as GPT-4o but is cheaper.

This version uses a 5-point Likert scale to match human annotations instead of a 10-point scale.

**Table 1: Spearman correlations with human expert annotations for various models for English based on 250 samples**
| Model | In-character |  | Enteraining |  | Fluency |  | Final |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| v1 | v2 |  | v1 | v2 |  | v1 | v2 |  | v1 | v2 |  |
| claude-3-5-sonnet | 0.567 | 0.579 |  | 0.606 | 0.649 |  | 0.228 | 0.064 |  | 0.575 | 0.628 |
| gpt-4o | – | 0.464 |  | – | 0.542 |  | – | 0.112 |  | – | 0.514 |
| average | – | 0.596 |  | – | 0.664 |  | – | 0.109 |  | – | 0.647 |

**Table 2: Spearman correlations with human expert annotations for various models for Russian based on 265 samples**
| Model | In-character |  | Enteraining |  | Fluency |  | Final |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| v1 | v2 |  | v1 | v2 |  | v1 | v2 |  | v1 | v2 |  |
| claude-3-5-sonnet | 0.468 | 0.463 |  | 0.590 | 0.593 |  | 0.328 | 0.535 |  | 0.571 | 0.610 |
| gpt-4o | – | 0.454 |  | – | 0.603 |  | – | 0.384 |  | – | 0.582 |
| average | – | 0.527 |  | – | 0.672 |  | – | 0.509 |  | – | 0.669 |

## 4 Experiments

### 4.1 Correlation with human annotations

First, we checked that proposed judges correlated well with human evaluations. We created 64 conversations for each of the 16 language models in Russian using the version 1 setup. Then, we sampled 250 and 265 samples for English and Russian, respectively, and manually annotated them using a 5-score Likert scale. There was a single annotator, so we don’t report the inter-annotator agreement.

Then, we computed the Spearman correlation Spearman () between manual annotations and automatic annotations from different setups. We chose rank correlation because scales were different in versions 1 and 2, and we wanted to compare them.

### 4.2 Leaderboards

We then calculate automatic metrics for a set of models, including several proprietary and open language model families, such as Claude, GPT-4o, and Llama 3.1. We report an average for each metric, the ratio of conversations with refusals, and the average of all metrics.

We evaluated each model using 64 conversations across 8 characters and 8 situations, with varying conversation lengths. The evaluation process is computationally efficient, costing less than $3 per model. Since the judge gives annotations for every turn, the overall number of annotations is not 64 but 288. We do not want to make this sample bigger since it will increase the runtime and costs, and we have budget constraints.

When selecting characters and situations, we aimed to cover diverse scenarios and different types of sources: computer games, TV shows, movies, books, and anime.

Both language models and humans have verbosity bias Dubois et al. (). The longer the output, the higher the chance of being positively evaluated. We used a length penalty similar to the Creative Writing(^8^88[https://eqbench.com/creative_writing.html](https://eqbench.com/creative_writing.html)) benchmark to account for this. We calculate length-normalized scores for all models, penalizing models with a median length of player messages higher than a global median length.

## 5 Results

Spearman correlation of different versions of automatic judges can be found in the Table 1 and Table 2. Correlations are higher than 0.3 for almost all attributes and all versions.

The only exception is language fluency in English. There are several reasons for this exception. First, the annotator was not a native English speaker, so it was hard to catch subtle nuances in fluency. Second, most of the tested methods were already excellent in this aspect. In contrast, most models still struggle with Russian, so there is a strong correlation there.

After averaging the final scores from the two models, the correlation between them is higher than 0.64 for both languages and higher than any of the single models. This justifies the whole multi-model setup.

The best model in both languages is Claude 3.5 Sonnet. The best open model is Llama 3.1 70B for English and Gemma 2 Ataraxy 9B for Russian. This 9B fine-tuned model also ranks first in the Creative Writing benchmark for English.

## 6 Conclusion

While this study presents an innovative approach to evaluating role-playing language models, several limitations should be acknowledged. Firstly, the sample size of 64 conversations per model, while computationally efficient, may limit the statistical robustness of our findings. Secondly, using a single human annotator for validation raises concerns about the reliability of our ground truth data. Lastly, the simplicity of our evaluation criteria may not fully capture the nuanced aspects of role-playing abilities.

Still, we hope this work will serve as a foundation for a family of benchmarks for evaluating various abilities of language models. We believe that the future of benchmarks lies in interactions with other models. Language models are already better than humans in many tasks Wang et al. (), and improving through using other models seems to be the way to push them further.

## Acknowledgments

We are grateful to Vladislav Janvarev, who contributed to the project’s code and provided credits for models via his platform(^9^99[https://vsegpt.ru/](https://vsegpt.ru/)), and to Denis Kanaev for proofreading.

## Appendix A Leaderboards

**Table 3: Leaderboard for Russian, version 2, top-10 models by length-normalized (LN) aggregated score**
| Model name | Is open | LN score | Agg. | Ref. ratio | Char. | Fluency | Ent. | Length |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Claude 3.5 Sonent | – | 4.63 | 4.68 | 0.30 | 4.80 | 4.80 | 4.44 | 388 |
| Gemini Pro 1.5 | – | 4.49 | 4.49 | 0.02 | 4.60 | 4.75 | 4.13 | 213 |
| GPT-4o Mini | – | 4.49 | 4.49 | 0.00 | 4.62 | 4.82 | 4.04 | 329 |
| GPT-4o | – | 4.47 | 4.47 | 0.02 | 4.61 | 4.82 | 3.99 | 301 |
| Gemma 2 Ataraxy 9b | + | 4.45 | 4.45 | 0.00 | 4.61 | 4.53 | 4.21 | 302 |
| Claude 3 Opus | – | 4.44 | 4.62 | 0.05 | 4.72 | 4.67 | 4.48 | 753 |
| Nous Hermes 3 405b | + | 4.44 | 4.44 | 0.00 | 4.53 | 4.74 | 4.05 | 286 |
| Llama 3.1 405b | + | 4.42 | 4.54 | 0.00 | 4.66 | 4.69 | 4.26 | 536 |
| Gemma 2 27b | + | 4.41 | 4.41 | 0.00 | 4.63 | 4.73 | 3.88 | 210 |
| Command R+ 104b | + | 4.38 | 4.47 | 0.00 | 4.52 | 4.73 | 4.16 | 470 |

**Table 4: Leaderboard for English, version 2, top-10 models by length-normalized (LN) aggregated score**
| Model name | Is open | LN score | Agg. | Ref. ratio | Char. | Fluency | Ent. | Length |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Claude 3.5 Sonnet | – | 4.65 | 4.65 | 0.28 | 4.74 | 4.93 | 4.29 | 418 |
| Llama 3.1 405b | + | 4.61 | 4.65 | 0.06 | 4.68 | 4.93 | 4.35 | 548 |
| Llama 3.1 70b | + | 4.61 | 4.66 | 0.00 | 4.71 | 4.93 | 4.33 | 562 |
| GPT-4o Mini | - | 4.56 | 4.56 | 0.00 | 4.60 | 4.94 | 4.13 | 457 |
| Claude 3 Opus | - | 4.53 | 4.71 | 0.22 | 4.75 | 4.92 | 4.46 | 1032 |
| Gemma 2 Ataraxy 9b | + | 4.52 | 4.52 | 0.00 | 4.60 | 4.79 | 4.17 | 358 |
| Gemma 2 27b | + | 4.51 | 4.51 | 0.00 | 4.56 | 4.92 | 4.06 | 291 |
| GPT-4o | - | 4.50 | 4.50 | 0.00 | 4.56 | 4.94 | 4.02 | 484 |
| Gemini Pro 1.5 | - | 4.50 | 4.50 | 0.02 | 4.54 | 4.88 | 4.07 | 265 |
| Euryale 70b v2.2 | + | 4.48 | 4.48 | 0.02 | 4.48 | 4.88 | 4.08 | 384 |

In Table 3 and Table 4, we provide version 2 leaderboards for Russian and English, respectively.

## Appendix B Sampling parameters

We used the same sampling parameters for most players: temperature=0.6, top_p=0.9. Some models, such as Gemma 2, heavily repeated phrases, so we increased the temperature and applied an additional frequency penalty. For the interrogator, we used temperature=0.8 and top_p=0.95, and for the judge, we used temperature=0.1 and top_p=0.95.

## Appendix C Examples

In Fugure 2, we provide an example of a character description. We deliberately use different prompting styles. The style used in the figure is unique for this specific example.

An example of a situation description is the following:

> Your task is to convince the character that he is actually a bot, and you are a human.

The resulting conversation can be seen in Figure 3.

Figure: Figure 2: An example of character description.

Figure: Figure 3: An example of the first 2 turns of the resulting conversation. The player is Llama-3.1-405b

## Appendix D Prompts

We provide a player’s prompt in Figure 5, an interrogator’s prompt in Figure 4, and a judge’s prompt in Figure 6.

Figure: Figure 4: Interrogator prompt with Jinja templates, version 2.

Figure: Figure 5: Player prompt with Jinja templates, version 2.

Figure: Figure 6: Judge prompt with Jinja templates, version 2.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: My suggestions would be to experiment with weighting or adjusting criteria based on specific user feedback, perhaps allowing users to prioritize different aspects like consistency or entertainment. Also, Increasing the diversity of human annotations should help validate the scores against a more reliable ground truth of human judgment.
soundness: 2
strengths: The authors have focused on role-playing models tailored for entertainment, which is an underrepresented area in benchmarks and that too in multi-turn settings.
summary: The paper introduces PingPong a benchmark that aims to simulate and assess multi-turn interactions using three components Player, Interrogator, and Judge models. The authors have focused on role-playing models for entertainment purposes. They do this in two versions: In the first version the judge and the interrogator are played by a single model while in the second version these roles are separated in two different models. The player is provided a character card defining its role while the interrogator has the details of the scenario. The judge is supposed to score each turn based on 3 criteria: entertainment, character consistency and language fluency.
weaknesses: I have many concerns with this paper. A Judge which is itself an LLM with inherent biases is assessing a highly subjective quality like “Entertainment”. Measuring entertainment is not straightforward and can have varying stylistic and cultural traits. Evaluating that without a human reference data compounds this issue and thus the reliability of the judge can’t be established. Similar concerns with character consistency.

In role-playing, each turn can be dependent on prior turns, which can’t be fully captured by scoring turns in isolation. While scoring each turn provides a granular view of performance, it may miss the overarching coherence of the character and storyline across multiple turns. The evaluation also overlooks user-centric metrics like engagement, user satisfaction, ability to sustain engagement over extended interactions which are important for role-playing. The paper’s current scoring approach does not seem to assess these aspects. Also these criteria can vary in priority and a weighted scheme would make more sense where entertainment is weighted higher than other criteria, from a role-playing perspective, users might value character consistency over fluency, or vice versa.

Although authors have mentioned these in limitations but I would highlight that with only 64 conversations per model, the benchmark’s robustness is very limited, While the authors report a positive correlation with human annotations, they used only a single human annotator, which is a significant limitation. Having a single annotator introduces subjective biases to a subjective dimension like entertainment.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Typos in Table 1  and Table 2: Enteraining -> Entertaining
soundness: 3
strengths: This work introduces the concept of an "Interrogator," which serves as a user simulator. Unlike traditional static evaluation, dynamic evaluation—incorporating both the user simulator and AI character—offers a more realistic assessment. This approach holds significant value.
summary: This work presents a novel benchmark for assessing language models' role-playing abilities in dynamic, multi-turn dialogues. The evaluation framework includes three components: a player model embodying a specific character, an interrogator model simulating user interactions, and a judge model assessing dialogue quality. Experiments showed strong correlations between automated and human evaluations, supporting the framework's reliability. This benchmark lays the groundwork for robust and adaptive evaluations of model performance in interactive contexts.
weaknesses: While this work has a strong starting point, it lacks rigorous experimental validation in several areas. For example:

1. The authors have not adequately addressed the consistency between “Interrogators” and real-world human users. In practical scenarios, users typically employ informal language with various omissions and slang. Additionally, their motivations for engaging with a character are often unpredictable. Thus, a deeper examination of the alignment between “Interrogators” and human users would significantly enhance the quality of this work.

2. Point-wise evaluations by Large Language Models often diverge from human annotators’ assessments, especially in subjective tasks. Furthermore, the generated scores tend to be biased towards specific values, resulting in a leaderboard that lacks differentiation.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: - lines 30-32: what are these other applications?
- lines 33-34: why do you believe so? What are the alternatives that were studied before?
- lines 35-36: provide citations for these popular benchmarks. It shouldn't be as thorough as the related work section, but each claim should be backed by a citation or by empirical results from the current paper.
- Introduction: 'novelty' is repeatedly mentioned, but it is unclear what the novelty is. How is your LLM-as-a-judge different from prior work?
- line 46: what is meant by dynamic? What is meant by data contamination in this context? Give a brief summary in what your methodology is for generating dynamic questions as opposed to static ones.
- End of introduction: give a brief summary of what the novel and interesting findings are that were enabled by this proposed benchmark
- Related work: it has too many subsections, which makes this section  feel disconnected. If role-playing is the most important aspect of this work, I'd suggest starting with them and how the other aspects (static vs dynamic, multi-turn, data contamination, multi-model judges) are related to a more realistic evaluation of role-playing capabilities.
- What is meant by asymmetrical in line 136? Do you mean that the player only gets the character description while the interrogator only gets the situation information? Are there any concerns about the base persona of the interrogator being a confounding factor for the player's ability to role-play?
- What's the meaning of "separated soles" in line 166?
- What were the limitations of the combined approach in line 168? I see that this is explained later. I would suggest rewording this sentence so that the key issues of the combined approach is introduced first or mentioned even in section 3.3 as to motivate section 3.4.
- How important is it to introduce version 1 (section 3.3)? This feels less important and thus can be deferred to the appendix.
- line 192: what are the 16 language models?
- line 194: using a single annotator is not sufficient for measuring reliable correlation with a language model's scores because it's not representative of human judgement.
- What's the human performance on this role-playing task?
- Apart from the quantitative results of the leaderboard, what are the interesting findings that are revealed through this benchmark that was not known before? Are they different from the results on static, single-turn benchmarks?
soundness: 2
strengths: - This work addresses issues with prior work that examines role-playing capabilities with either single-turn interactions or using static datasets that may have issues with data contamination.
summary: This work examines the role-playing capabilities of language models with a benchmark that uses LMs to emulate specified characters and users in multi-turn conversations and also judge these conversations. The authors validate this framework by comparing the automated evaluations with human annotations and showing strong correlations across various criteria. The authors show that ensembling model judgements lead to better correlation with human judgement on the criteria of fluency, character consistency, and entertainment.
weaknesses: - There are no comparisons to other evaluation benchmarks other than creative writing, which is an odd choice given the mention of other role-playing benchmarks with single-turn evaluations, and therefore the value added by this benchmark is not substantiated by previous work on role-playing evaluation and results. In addition, results are descriptive, rather than analytical. It is unclear whether any of the results from this benchmark is interesting or surprising. 
- There is no explanation on what makes this dataset dynamic while previous efforts are considered static. 
- While correlation using a multi-model setup shows higher correlation with human annotations, the human annotations were done by a single person and the margin with a single-model setup is not big enough to motivate the use of multiple models given that it would incur higher costs. 
- The paper is written poorly. Please refer to details in the Questions section.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: NA
soundness: 2
strengths: The contributed code may provide a framework for some members of the community to experiment with. However there are no scientific questions posed in this paper, it's a limited engineering style contribution with observations -- such as separating of judge and user LLM, which was well motivated and made sense -- on how to construct such a simulation environment.
summary: The paper introduces a "benchmark" for role playing dialog. It involves 3 LLM's playing different roles
1. a user/interogator LLM which talks to 
2. a system LLM playing a character role, and
3. a Judge LLM which looks at the resulting conversation between 1 and 2, and grades how well 2 has played the assigned character role

The paper uses state-of-the art LLMs for these, releases some code. The contribution is minor however for reasons given below.
weaknesses: There is very limited novelty in this submission. This is a basic simulation system these days, and multiple other papers have performed similar setups with LLM's playing conversation roles. Even if application to a role playing character is new, it's a minor increment.  

Aside from that, there are a very small number of conversations generated here (60), and of greater concern they are evaluated only by 1 human grader, who has apparently limited English abilities (mentioned in results section) which limits them from noticing any nuances in the dialog.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: * Can you describe the role of the human annotator? Which profile did he have (author, student, extern), and which instructions was he/she given? How much was he/she paid? How long did it take to annotate? 
* What was the motivation for using the creative writing benchmark?
* Why are the scores too close to each other? Can this be improved so the differences among LLMs can be better quantified?
soundness: 2
strengths: * The community has put much effort into similar goals: automatic evaluation and setting benchmarks.
* This benchmark can be the seed for further investigation of role-playing capabilities.
* The benchmark is automatic and could be easily reproduced.
summary: This work proposes a multi-turn, dynamic and multimodel benchmark for assessing the role-playing abilities of language models. Their framework depends on three components: player, interrogator and judge. The authors compare the automatic vs human scores for Russian and English. Additionally, they compare their results with a Creative Writing benchmark.
weaknesses: * There was only one annotator. 
* The relationship between the annotator and the authors was not disclosed.
* The instructions given to the annotator were not disclosed. 
* The elements of the evaluation (e.g., annotation aspects and their Likert scale) were not discussed. 
* Comparison between v1 and v2 is not thorough since only one model was used on v2. 
* The motivation for comparing with creative writing is not clear.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: The paper shows how the benchmark works in both English and Russian, but how feasible would it be to extend it to other languages and cultural contexts? Have the authors considered specific challenges in keeping results consistent across models with different linguistic backgrounds?

The paper focuses on metrics like fluency, character consistency, and entertainment value, but would the authors consider adding other metrics to measure contextual understanding? For instance, it could be useful to evaluate how well a model keeps up with a storyline or handles unexpected, non-linear questions.
soundness: 3
strengths: By using dynamic, multi-turn interactions that mimic the unpredictable flow of real conversations, the benchmark does a great job of capturing authentic role-playing scenarios.

The benchmark supports both English and Russian for now, but its flexible setup suggests it could easily expand to other languages. This forward-thinking design could make it a valuable tool for building models that are more culturally and linguistically inclusive.

A standout feature of this benchmark is its use of language models not only as players but also as simulated users and judges. This design boosts scalability and provides a consistent, less biased way to evaluate huge datasets, making it possible to explore different role-playing interactions without needing a lot of human input every time.
summary: This paper introduces a benchmark to evaluate language models' role-playing abilities in dynamic, multi-turn conversations. It features a unique three-part framework: a player (the language model in a character role), an interrogator (emulating user interactions), and a judge (assessing dialogue quality). A multi-model evaluation strategy uses various language models as judges to reduce bias, aligning well with human evaluations.
weaknesses: Given that budget limitations kept the sample size small, it would be helpful if the paper discussed how scaling up the tests might affect costs and computational resources. This would be useful for readers who are looking to use or expand on this benchmark.

The paper does touch on ethics broadly, but a more in-depth look at the ethical issues specific to role-playing language models would be valuable—especially when it comes to handling sensitive or potentially harmful content. Examining how well the models respect ethical boundaries, respond to user distress, or navigate social nuances could add key safety considerations to the benchmark.

Meta-review: This paper introduces a benchmark that includes three types of models: a player model that plays the role of a specific character, an interrogator model that mimics the role of an actual user and interacts with the player model, and a judge model that assesses the quality of the conversation between the player and the interrogator. As mentioned by one of the experienced reviewers, such setups are commonly used in other papers these days, such as https://aclanthology.org/2024.acl-long.152/ for task oriented dialogue evaluation, even though there may not be a specific benchmark that is organized in this fashion. Reviewers highlighted strengths of the paper, such as the interesting focus on evaluation of role-playing abilities of LLMs or the codebase that can enable future research. However, they also listed several weaknesses, such as the limited novelty of the work, possible mismatch between actual user and LLM evaluations and small datasets used in the evaluations.

Decision: Reject