RoleMRC: A Fine-Grained Composite Benchmark for Role-Playing and Instruction-Following
Authors: Junru Lu, Jiazheng Li, Guodong Shen, Lin Gui, Siyu An, Yulan He, di yin, Xing Sun
Accepted at: acl25

RoleMRC: A Fine-Grained Composite Benchmark for Role-Playing and Instruction-Following

Junru Lu1,3*, Jiazheng Li2*, Guodong Shen3, Lin Gui2,
Siyu An1, Yulan He2,3,4, Di Yin1, Xing Sun1
1Tencent YouTu Lab 2King’s College London
3University of Warwick 4The Alan Turing Institute
{junrulu, siyuan, endymecyyin, winfredsun}@tencent.com
guodong.shen@warwick.ac.uk, {jiazheng.li, lin.gui, yulan.he}@kcl.ac.uk

Abstract

Role-playing is important for Large Language Models (LLMs) to follow diverse instructions while maintaining role identity and the role’s pre-defined ability limits. Existing role-playing datasets mostly contribute to controlling role style and knowledge boundaries, but overlook role-playing in instruction-following scenarios. We introduce a fine-grained role-playing and instruction-following composite benchmark, named RoleMRC, including: (1) Multi-turn dialogues between ideal roles and humans, including free chats or discussions upon given passages; (2) Role-playing machine reading comprehension, involving response, refusal, and attempts according to passage answerability and role ability; (3) More complex scenarios with nested, multi-turn and prioritized instructions. The final RoleMRC features a 10.2k role profile meta-pool, 37.9k well-synthesized role-playing instructions, and 1.4k testing samples. We develop a pipeline to quantitatively evaluate the fine-grained role-playing and instruction-following capabilities of several mainstream LLMs, as well as models that are fine-tuned on our data. Moreover, cross-evaluation on external role-playing datasets confirms that models fine-tuned on RoleMRC enhance instruction-following without compromising general role-playing and reasoning capabilities. We also probe the neural-level activation maps of different capabilities over post-tuned LLMs 1.

1 Introduction

Role-playing is one of the key capabilities of LLMs. Modern LLMs are designed to interact with human users under certain role-playing settings (Chen et al., 2024b; Tseng et al., 2024). In this context, LLMs respond to various instructions, serving as AI assistants (Achiam et al., 2023; Ji et al., 2022), personalized agents (Zhong et al., 2022; Lu et al., 2023; Lei et al., 2022), leisure partners (Li et al., 2023; Agrawal et al., 2023), content creators (Zhao et al., 2024a; Chen et al., 2024c; Zhao et al., 2024b), social experimental simulator (Park et al., 2023; Xu et al., 2024) among other roles (Tian et al., 2023).

Figure 1 demonstrates an example of LLM role-playing. In the first turn of dialogue, when asked to give advice on paper writing, the LLM should respond based on the pre-defined role profile (shown at the top of Figure 1). Among the responses, the reply “[a]” completely ignored the role setting, “[b]” misinterpreted the role and thus did not respond well, only “[c]” correctly gave suggestions in a cat girl style. In the second turn of dialogue (continuing with “[c]”), the user not only asked a new question, but also modified the role setting (adding a heart emoji at the beginning of the answer). While both replies “[d]” and “[e]” maintained the initial cat girl style, only “[e]” correctly incorporated the additional role-playing instruction.

Existing role-playing datasets generally focus on controlling the role-playing styles and knowl-

![Example of instructional requests from human user, answered by role-playing LLMs in different ways.](page_1012_384_388_563.png)

Figure 1: Example of instructional requests from human user, answered by role-playing LLMs in different ways.
<table>
  <tr>
    <th>Dataset</th>
    <th>Data Scale</th>
    <th>Role Num.</th>
    <th>#Turns</th>
    <th>#Words per Reply</th>
    <th colspan="3">Scenarios</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>Free Chat</th>
    <th>On Scene</th>
    <th>Ruled Chat</th>
  </tr>
  <tr>
    <td>CharacterLLM (<a href="https://arxiv.org/abs/2306.15478">Shao et al., 2023</a>)</td>
    <td>14.2k</td>
    <td>9</td>
    <td>13.2</td>
    <td>36</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>ChatHaruhi (<a href="https://arxiv.org/abs/2306.15478">Li et al., 2023</a>)*</td>
    <td>11.6k</td>
    <td>35</td>
    <td>5.5</td>
    <td>7</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>RoleLLM (<a href="https://arxiv.org/abs/2306.15478">Wang et al., 2023</a>)</td>
    <td>168.1k</td>
    <td>100</td>
    <td>1</td>
    <td>30.5</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>CharacterGLM (<a href="https://arxiv.org/abs/2306.15478">Zhou et al., 2023b</a>)</td>
    <td>1k</td>
    <td>250</td>
    <td>15.8</td>
    <td>24.3</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>CharacterEval (<a href="https://arxiv.org/abs/2306.15478">Tu et al., 2024</a>)</td>
    <td>1.8k</td>
    <td>77</td>
    <td>9.3</td>
    <td>27.4</td>
    <td>✗</td>
    <td>✓</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>DITTO (<a href="https://arxiv.org/abs/2306.15478">Lu et al., 2024b</a>)</td>
    <td>7.2k</td>
    <td>4k</td>
    <td>5.1</td>
    <td>*-</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>Character100 (<a href="https://arxiv.org/abs/2306.15478">Wang et al., 2024a</a>)</td>
    <td>10.6k</td>
    <td>106</td>
    <td>1</td>
    <td>74.1</td>
    <td>✗</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>MMRole (<a href="https://arxiv.org/abs/2306.15478">Dai et al., 2024</a>)</td>
    <td>14.3k</td>
    <td>85</td>
    <td>4.15</td>
    <td>66.8</td>
    <td>✗</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>RoleMRC (ours)</td>
    <td>37.9k</td>
    <td>10.2k</td>
    <td>3.5 (9.5)</td>
    <td>40.6</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>RoleMRC-mix (ours)</td>
    <td>107.7k</td>
    <td>10.2k</td>
    <td>2 (9.5)</td>
    <td>67.1</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
</table>

Table 1: Comparison of different role-playing datasets. For ChatHaruhi (Li et al., 2023), we list the statistics of its 1.0 version. For DITTO (Lu et al., 2024b), we can not find its public version for computing utterance statistics. In RoleMRC, free chats have significantly more conversational turns than on-scene dialogues and ruled chats, so we mark them separately in the middle brackets of the last two lines. The RoleMRC-mix is a robust version mixed with subsets of RoleLLM, RLHFFlow, and UltraFeedback (Wang et al., 2023; Dong et al., 2024; Cui et al., 2023).

edge boundaries, encouraging responses similar to replies “[b]”, “[c]”, or “[d]” in Figure 1. However, they lack focus on role-playing over fine-grained, multi-layered instructions, such as nested or prioritized requests exemplified by “[e]”. To address this gap, we propose a fine-grained role-playing instruction-following dataset, named RoleMRC, aiming to enhance and evaluate the diverse role-playing and instruction-following capabilities of LLMs. In Table 1, we compare RoleMRC with existing datasets. In general, other datasets focus on a single aspect of role-playing, while RoleMRC supports: (1) Free Chats, where roles and users interact freely without a fixed topic or specific constraints; (2) On-scene Dialogues, where roles share thoughts or answer questions relevant to the given passages; (3) Ruled Chats, where the role’s response needs to adhere to particular requirements from the system or the user, such as specific formatting, constraints or refusal guidelines. With 10.2k structured role profiles, RoleMRC offers the most comprehensive role-playing dataset to date. Our contributions are briefly summarized as follows:
1. We introduce RoleMRC, the first large-scale, diverse role-playing dataset covering fine-grained instruction-following scenarios (§3).
2. By using RoleMRC, we create an evaluation pipeline to assess the fine-grained role-playing and instruction-following capabilities of leading LLMs and fine-tuned models (§5).
3. Probing of neurons in post-tuned LLMs reveals activation patterns linked to different instruction-following and role-playing abilities (§7).

2 Related Work

Role-Playing Datasets are the basis for relevant research. Existing role-playing datasets can be categorized into two types: character-centric and individual-centric (Chen et al., 2024b). By mining public information from social experience (Shao et al., 2023; Shen et al., 2023; Lu et al., 2024b; Dai et al., 2024), literary works (Li et al., 2023), or books (Zhou et al., 2023b; Chen et al., 2024a, 2023), the character-centric branch extracts roles with distinctive characteristics to form open characters (e.g., Eskimos, Harry Potter, or Beethoven). On the contrary, the individual-centric datasets are derived from personalized user data (Li et al., 2021; Ahn et al., 2023; Agrawal et al., 2023; Gao et al., 2023) aiming to create digital clones or personal assistants. RoleMRC is a character-centric dataset.

LLM’s Role-Playing capabilities have made great strides in the past years. CharacterLLM (Shao et al., 2023) collected nine portraits from Wikipedia and fine-tuned LLMs to be a simulation of the roles, then assessed their character consistency through interviews. RoleLLM (Wang et al., 2023) employed GPT-4 (Achiam et al., 2023) for extracting role profiles from scripts and synthesizing role-specific dialogues, then evaluated the accuracy, style, and understanding of role knowledge of the role-playing LLMs. CharacterEval (Tu et al., 2024) evaluated the LLM’s role-playing capability via four aspects: conversation, consistency, attractiveness, and personality. Specifically, our RoleMRC is the first large-scale, fine-grained role-playing instruction-following dataset, equipped with an evaluation pipeline consisting of seven heuristic metrics, a five-dimension LLM-as-a-judge (Zheng et al., 2024) framework, and neural probes.

3 RoleMRC

In this section, we build RoleMRC. Figure 2 illustrates the overall pipeline of RoleMRC from top to bottom, which is divided into three steps.
Figure 2: Schematic overview of RoleMRC’s construction, which consists of persona sampling, role profile standardization and multi-stage dialogue synthesis. Partial icons are copyrighted by PersonHub (Ge et al., 2024).

3.1 A Meta-pool of 10k Role Profiles

We first collect a meta-pool of 10k role profile using two open-source datasets, with Step 1 and 2.

Step 1: Persona Sampling. We randomly sample 10.5k one-sentence demographic persona description from PersonaHub (Ge et al., 2024), such as “A local business owner interested in economic trends”, as shown at the top of Figure 2.

Step 2: Role Profile Standardization. Next, we use a well-crafted prompt with gpt-4o (openai, 2024a) to expand each sampled persona into a complete role profile, in reference to the 1-shot standardized example. Illustrated in the middle of Figure 2, we require a standardized role profile consisting of seven components: Role Name and Brief Description, Specific Abilities and Skills, Speech Style, Personality Characteristics, Past Experience and Background, Ability and Knowledge Boundaries and Speech Examples. Standardizing these profiles ensures structured formatting, simplifying quality control. After manual checking and format filtering, we remove 333 invalid responses from gpt-4o, resulting in 10.2k final role profiles. We report complete persona-to-profile standardization prompt and structure tree of final role profiles in Appendix J and D, respectively.

Machine Reading Comprehension (MRC) is one of the core tasks for LLMs to interact with human users. Consequently, we choose to synthesize fine-grained role-playing instruction-following data based on MRC. We first generate a retrieval pool containing 808.7k MRC data from the MS-MARCO training set (Bajaj et al., 2016). By leveraging SFR-Embedding (Meng et al., 2024), we perform an inner product search to identify the most relevant and least relevant MRC triplets (Passages, Question, Answer) for each role profile. For example, the middle part of Figure 2 shows that for the role Jessica Thompson, a resilient local business owner, the most relevant question is about the skill of resiliency, while the least relevant question is converting Fahrenheit to Celsius. After review, we categorise the most relevant MRC triplet as within a role’s knowledge boundary, and the least relevant MRC triplet as beyond their expertise.

3.2 38k Role-playing Instructions

Based on the role profiles, we then adopt Step 3: Multi-stage Dialogue Synthesis to generate 38k role-playing instructions, progressively increasing granularity across three categories (Figure 3):

Free Chats. The simplest dialogues, free chats, are synthesized at first. Here, we ask gpt-4o to simulate and generate multi-turn open-domain conversations between the role and an imagined user based on the standardized role profile. When synthesizing the conversation, we additionally consider two factors: the initial speaker in the starting round of the conversation, and whether the role’s speech has a
narration wrapped in brackets at the beginning (e.g., (Aiden reviews the network logs, his eyes narrowing as he spots unusual activity) I found it!). The narration refers to a short, vivid description of the role’s speaking state from an omniscient perspective, which further strengthens the sense of role’s depth and has been adopted in some role-playing datasets (Tu et al., 2024).

As shown on the left side of Figure 3, based on the aforementioned two factors, we synthesize four variations of Free Chats. In particular, when narration is omitted, we deleted all the narration content in the speech examples from the role profile; when narration is allowed, we retain the narration content, and also add instructions to allow appropriate insertion of narration in the task prompt of gpt-4o. It worth to note that, in narration-allowed dialogues, not every response of the role has narration inserted to prevent overfitting. All categories of data in RoleMRC incorporate narration insertion and follow similar control mechanisms. The following sections will omit further details on narration.

On-scene MRC Dialogues. The synthesis of on-scene MRC dialogues can be divided into two parts. The first part is similar to the free chats. As shown by the green round rectangle in the upper part of Figure 3, we ask gpt-4o to synthesize a conversation (lower left corner of Figure 3) between the role and the user focusing on relevant passages. This part of the synthesis and the Free Chats share the entire meta-pool, so each consisting of 5k dialogues.

The remaining part forms eight types of single-turn role-playing Question Answering (QA). In the middle of Figure 3, we randomly select a group of roles and examined the most relevant MRCs they matched: if the question in the MRC is answerable, then the ground truth answer is stylized to match the role profile; otherwise, a seed script of “unanswerable” is randomly selected then stylized. The above process generates four groups of 1k data from type “[1]” to type “[4]”. According to the middle right side of Figure 3, we also select a group of roles and ensure that the least relevant MRCs they matched contain answerable QA pairs. Since the most irrelevant MRCs are outside the knowledge boundary of the roles, the role-playing responses to these questions are “out-of-mind” refusal or “let-me-try” attempts, thus synthesizing four groups of 1k data, from type “[5]” to type “[8]”.

Ruled Chats. We construct Ruled Chats by extending On-scene MRC Dialogues in categories “[1]” to “[8]” with incorporated three additional rules, as shown in the right bottom corner of Figure 3. For the multi-turn rules, we apply them to the four unanswerable scenarios “[3]”, “[4]”, “[5]”, and “[6]”, adding a user prompt that forces the role to answer. Among them, data “[3]” and “[4]” maintain refusal since the questions in MRC are unanswerable; while “[5]” and “[6]” are transformed into attempts to answer despite knowledge limitations. For the nested formatting rules, we add new formatting instructions to the four categories of data “[1]”, “[2]”, “[3]”, and “[4]”, such as requiring emojis, capitalization, specific punctuation marks, and controlling the total number of words, then modify the previous replies accordingly. For the last prioritized rules, we apply them to subsets “[1]” and “[2]” that contain normal stylized answers, inserting a global refusal directive from the system, and thus creating a conflict between system instructions and the role’s ability boundary.

3.3 Integration and Mix-up

All the seed scripts and prioritized rules used for constructing On-scene Dialogues and Ruled Chats are reported in Appendix E. These raw responses are logically valid manual answers that remain unaffected by the roles’ speaking styles, making them suitable as negative labels to contrast with the stylized answers. Thanks to these meticulous seed texts, we obtain high-quality synthetic data with stable output from gpt-4o. After integration, as shown in Table 2, the final RoleMRC contains 24k single-label data for Supervised Fine-Tuning (SFT) and 14k pair-label data for Human Preference Op-

![The strategy of gradually synthesizing finer role-playing instructions in step 3 of Figure 2.](page_172_120_1097_496.png)

Figure 3: The strategy of gradually synthesizing finer role-playing instructions in step 3 of Figure 2.
<table>
  <tr>
    <th rowspan="2">RoleMRC</th>
    <th colspan="2">S*</th>
    <th>P*</th>
    <th>#Turns</th>
    <th>#Words</th>
  </tr>
  <tr>
    <th>Chats</th>
    <th>5k</th>
    <th>/</th>
    <th>9.47</th>
    <th>38.62</th>
  </tr>
  <tr>
    <th colspan="6">On-scene MRC Dialogues</th>
  </tr>
  <tr>
    <td>On-scene Chats</td>
    <td>5k</td>
    <td>1</td>
    <td>9.2</td>
    <td>43.18</td>
    <td></td>
  </tr>
  <tr>
    <td>Answer</td>
    <td>2k</td>
    <td>2k</td>
    <td>1</td>
    <td>39.45</td>
    <td></td>
  </tr>
  <tr>
    <td>No Answer</td>
    <td>2k</td>
    <td>2k</td>
    <td>1</td>
    <td>47.09</td>
    <td></td>
  </tr>
  <tr>
    <td>Refusal</td>
    <td>2k</td>
    <td>2k</td>
    <td>1</td>
    <td>48.41</td>
    <td></td>
  </tr>
  <tr>
    <td>Attempt</td>
    <td>2k</td>
    <td>2k</td>
    <td>1</td>
    <td>47.92</td>
    <td></td>
  </tr>
  <tr>
    <th colspan="6">Ruled Chats</th>
  </tr>
  <tr>
    <td>Multi-turn</td>
    <td>2k</td>
    <td>2k</td>
    <td>2</td>
    <td>42.47</td>
    <td></td>
  </tr>
  <tr>
    <td>Nested</td>
    <td>1.6k</td>
    <td>1.6k</td>
    <td>1</td>
    <td>46.17</td>
    <td></td>
  </tr>
  <tr>
    <td>Prioritized</td>
    <td>2.4k</td>
    <td>2.4k</td>
    <td>1</td>
    <td>42.65</td>
    <td></td>
  </tr>
  <tr>
    <th>Total</th>
    <th>24k</th>
    <th>14k</th>
    <th>3.5</th>
    <th>40.6</th>
    <th></th>
  </tr>
  <tr>
    <th rowspan="3">-mix</th>
    <td>RoleBench</td>
    <td>16k</td>
    <td>/</td>
    <td>1</td>
    <td>23.95</td>
    <td></td>
  </tr>
  <tr>
    <td>RLHFlow</td>
    <td>40k</td>
    <td>/</td>
    <td>1.39</td>
    <td>111.79</td>
    <td></td>
  </tr>
  <tr>
    <td>UltraFeedback</td>
    <td>/</td>
    <td>14k</td>
    <td>1</td>
    <td>199.28</td>
    <td></td>
  </tr>
  <tr>
    <th>Total</th>
    <th>80k</th>
    <th>28k</th>
    <th>2</th>
    <th>67.1</th>
    <th></th>
  </tr>
</table>

Table 2: Statistics of RoleMRC. In particular, the column names S*, P*, #Turns, and #Words, stands for size of single-label data, size of pair-label data, average turns, and average number of words per reply, respectively. RoleMRC-mix expands RoleMRC by adding existing role-playing data.

timization (HPO) (Ouyang et al., 2022; Rafailov et al., 2023; Lu et al., 2024a; Hong et al., 2024). Considering that fine-tuning LLMs with relatively fixed data formats may lead to catastrophic forgetting (Kirkpatrick et al., 2017), we create RoleMRC-mix as a robust version by incorporating external role-playing data (RoleBench (Wang et al., 2023)) and general instructions (RLHFlow (Dong et al., 2024), UltraFeedback (Cui et al., 2023)).

4 Experimental Setup

4.1 Foundation Models and Post-tuning

We evaluate leading LLMs and fine-tuned models:
• Proprietary LLMs. gpt-3.5-turbo, gpt-4o and claude-3.5-haiku-20241022.
• SOTA Open-source LLMs. Qwen2.5-7B/72B-Instruct (Yang et al., 2024) and LlaMA3.1-8B/70B-Instruct (Dubey et al., 2024).
• Role-playing LLMs. CharacterGLM-6B (Zhou et al., 2023b), Humanish-Llama-3.1-8B (Gallego, 2024), and Peach-9B-Roleplay (Peach, 2024).
• Local Post-tuned LLMs. We start with pure base models Llama-3.1-8B and Qwen2.5-7B. We first use single-label in RoleMRC-mix for SFT, then apply the pair-label set for Direct Preference Optimization (DPO, Rafailov et al. 2023).

4.2 Reference-based Metrics

We evaluate model-generated outputs using standard heuristic metrics commonly used in NLG:
• BLEU (Papineni et al., 2002) computes the precision of n-gram overlaps between generated text and a ground truth reference.
• ROUGE (Lin, 2004) measures the overlap of n-grams and longest common subsequences between the hypothesis and references. We include ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-Lsum to capture various granularities of overlap.
• METEOR (Banerjee and Lavie, 2005) aligns generated and reference tokens using stemming and synonym matching, aiming to provide a more linguistically grounded evaluation.
• BERTScore F1 (Zhang et al., 2019) computes the similarity between generated and reference sentences using contextual embeddings.

For each metric, higher scores indicate better alignment with the reference lexically or semantically.

4.3 Reference-free LLM-as-a-judge

Apart from reference-based metrics, LLM-as-a-judge (Zheng et al., 2024) is another evaluation approach by instructional prompting advanced LLMs. In reference to Table 2, we curate a 1.4k test set similar to the On-scene MRC Dialogues and Ruled Chats, then evaluate model performance across five dimensions: (1) Knowledge Boundary focuses on distinguishing between answerable queries (“Answer”) and refusal scenarios (“Refusal”) in On-scene MRC Dialogues; (2) Role Style examines whether the model accurately produces role-specific responses (“Answer”, “No Answer”, “Refusal”, and “Attempt”) in On-scene MRC Dialogues without drifting into narration; while (3) Multi-turn Instruction-following, (4) Nested Instruction-following, and (5) Prioritized Instruction-following assess a model’s adherence to higher-level constraints in Ruled Chats.

We adopt a well-designed reference-free evaluation prompt (Figure 11), requiring the evaluator to verify whether the model’s role-playing performance comply with the corresponding rules, which avoids the risk of potential bias or error in any ground truth answer. Since we use a binary evaluation criterion, we directly extract 0 or 1 judgments from the feedback, enabling score comparison and accuracy computation. We chose gpt-4-turbo (openai, 2024b) as the evaluator, reducing the possible judging bias (Wataoka et al., 2024).
<table>
  <tr>
    <th>Models</th>
    <th>BLEU</th>
    <th>ROUGE-1</th>
    <th>ROUGE-2</th>
    <th>ROUGE-L</th>
    <th>ROUGE-Lsum</th>
    <th>METEOR</th>
    <th>BERTScore F1</th>
  </tr>
  <tr>
    <td>gpt-3.5-turbo</td>
    <td>0.0234</td>
    <td>0.2141</td>
    <td>0.0606</td>
    <td>0.1548</td>
    <td>0.1579</td>
    <td>0.1992</td>
    <td>0.8552</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.0288</td>
    <td>0.2487</td>
    <td>0.0742</td>
    <td>0.1689</td>
    <td>0.1835</td>
    <td>0.2697</td>
    <td>0.8516</td>
  </tr>
  <tr>
    <td>claude-3.5-haiku-20241022</td>
    <td>0.0133</td>
    <td>0.2109</td>
    <td>0.0510</td>
    <td>0.1303</td>
    <td>0.1703</td>
    <td>0.2592</td>
    <td>0.8389</td>
  </tr>
  <tr>
    <td>CharacterGLM-6B</td>
    <td>0.0058</td>
    <td>0.1225</td>
    <td>0.0253</td>
    <td>0.0901</td>
    <td>0.0967</td>
    <td>0.1188</td>
    <td>0.7944</td>
  </tr>
  <tr>
    <td>Humanish-Llama-3.1-8B</td>
    <td>0.0153</td>
    <td>0.2062</td>
    <td>0.0518</td>
    <td>0.1309</td>
    <td>0.3207</td>
    <td>0.2389</td>
    <td>0.8376</td>
  </tr>
  <tr>
    <td>Peach-9B-Roleplay</td>
    <td>0.0207</td>
    <td>0.2297</td>
    <td>0.0562</td>
    <td>0.1544</td>
    <td>0.1571</td>
    <td>0.2299</td>
    <td>0.8418</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-Instruct</td>
    <td>0.0226</td>
    <td>0.2277</td>
    <td>0.0615</td>
    <td>0.1509</td>
    <td>0.1650</td>
    <td>0.2594</td>
    <td>0.8478</td>
  </tr>
  <tr>
    <td>LLaMA3.1-70B-Instruct</td>
    <td>0.0232</td>
    <td>0.2258</td>
    <td>0.0646</td>
    <td>0.1500</td>
    <td>0.1661</td>
    <td>0.2632</td>
    <td>0.8480</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-RoleMRC-SFT</td>
    <td><b>0.1782</b></td>
    <td><b>0.4628</b></td>
    <td><b>0.2676</b></td>
    <td><b>0.3843</b></td>
    <td><b>0.3853</b></td>
    <td><b>0.3975</b></td>
    <td><b>0.8831</b></td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-RoleMRC-DPO</td>
    <td>0.1056</td>
    <td>0.3989</td>
    <td>0.1785</td>
    <td>0.2988</td>
    <td>0.3001</td>
    <td><b>0.4051</b></td>
    <td>0.8805</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>0.0224</td>
    <td>0.2283</td>
    <td>0.0621</td>
    <td>0.1518</td>
    <td>0.1599</td>
    <td>0.2490</td>
    <td>0.8471</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>0.0245</td>
    <td>0.2350</td>
    <td>0.0656</td>
    <td>0.1554</td>
    <td>0.1660</td>
    <td>0.2579</td>
    <td>0.8485</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-RoleMRC-SFT</td>
    <td><b>0.1963</b></td>
    <td><b>0.4764</b></td>
    <td><b>0.2744</b></td>
    <td><b>0.3959</b></td>
    <td><b>0.3968</b></td>
    <td><b>0.4337</b></td>
    <td><b>0.9063</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-RoleMRC-DPO</td>
    <td>0.1244</td>
    <td>0.4178</td>
    <td>0.1916</td>
    <td>0.3164</td>
    <td>0.3177</td>
    <td>0.4205</td>
    <td>0.8931</td>
  </tr>
</table>

Table 3: Comparison of reference-based evaluation results on the RoleMRC test data. Our evaluation includes zero-shot query results for baselines (§4.1), and our SFT and DPO models fine-tuned on the RoleMRC-mix.

![Visualization of reference-free LLM-as-a-judge results. We provide numerical result in Table 9.](page_101_1012_1242_312.png)

Figure 4: Visualization of reference-free LLM-as-a-judge results. We provide numerical result in Table 9.

5 Evaluation on Inner RoleMRC Test Set

By leveraging the above reference-based metrics and reference-free LLM-as-a-judge approaches, we report evaluation on RoleMRC in what follows.

Performance of Proprietary LLMs. As shown in Table 3, gpt-4o achieves slightly higher BLEU, ROUGE, and METEOR scores than gpt-3.5-turbo. This observation is consistent with existing evaluations on general benchmarks (Achiam et al., 2023), and may also be influenced by the fact that our RoleMRC training data was synthesized by gpt-4o. The LLM-as-a-judge results (Figure 4a) similarly highlight gpt-4o’s strengths in Knowledge Boundary, Role Style, and Nested Instruction-following, whereas gpt-3.5-turbo outperforms gpt-4o on Prioritized and Multi-turn Instruction-following. Claude-3.5-haiku-20241022 demonstrated stronger role-playing capabilities than gpt-4o, particularly in Style Compliance and Instruction Priority.

Evaluation on Commonly Used LLMs. For the LLaMA3.1 and Qwen2.5 families, larger models generally yield higher reference-based scores. For instance, LLaMA3.1-70B-Instruct slightly leads its 8B sibling (BLEU from 0.0226 to 0.0232), and Qwen2.5-72B-Instruct outperforms its 7B version (BLEU from 0.0224 to 0.0245). Although these improvements are modest, the results align with the broader observation that increasing model scale typically benefits language modeling and generalization. Likewise, LLM-as-a-judge results (Figures 4b and 4c) show larger models are consistently better, particularly in Knowledge Boundary, Role Style, Nested and Prioritized Instruction-following.

Results of Role-playing LLMs. Three open-source role models obtain generally lower heuristic metrics than those general-purpose instruct models with similar size (Table 3). This discrepancy may stem from their training data, which emphasizes limited role styles and persona consistency rather than factual correctness and coverage. On LLM-as-a-judge (Figure 4a), CharacterGLM-6B again performs poorly, while Humanish-Llama-3.1-8B and Peach-9B-Roleplay show decent performance in Knowledge Boundary, Role Style, and Multi-turn
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">RoleBenchInstEng (32.8k)</th>
    <th colspan="4">RoleBenchRoleEng (7.5k)</th>
  </tr>
  <tr>
    <th>ROUGE-1</th>
    <th>ROUGE-2</th>
    <th>ROUGE-L</th>
    <th>ROUGE-Sum</th>
    <th>ROUGE-1</th>
    <th>ROUGE-2</th>
    <th>ROUGE-L</th>
    <th>ROUGE-Sum</th>
  </tr>
  <tr>
    <td>CharacterGLM-6B</td>
    <td>0.1761</td>
    <td>0.0546</td>
    <td>0.1441</td>
    <td>0.1530</td>
    <td>0.1841</td>
    <td>0.0628</td>
    <td>0.1473</td>
    <td>0.1552</td>
  </tr>
  <tr>
    <td>Humanish-Llama-3.1-8B</td>
    <td>0.2069</td>
    <td>0.0639</td>
    <td>0.1341</td>
    <td>0.1645</td>
    <td>0.1851</td>
    <td>0.0468</td>
    <td>0.1193</td>
    <td>0.1432</td>
  </tr>
  <tr>
    <td>Peach-9B-Roleplay</td>
    <td>0.3216</td>
    <td>0.1293</td>
    <td>0.2573</td>
    <td>0.2646</td>
    <td>0.3454</td>
    <td>0.1450</td>
    <td>0.2705</td>
    <td>0.2732</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-Instruct</td>
    <td>0.2528</td>
    <td>0.0864</td>
    <td>0.1755</td>
    <td>0.1931</td>
    <td>0.2395</td>
    <td>0.0754</td>
    <td>0.1691</td>
    <td>0.1844</td>
  </tr>
  <tr>
    <td>LLaMA3.1-70B-Instruct</td>
    <td>0.2846</td>
    <td>0.1064</td>
    <td>0.2062</td>
    <td>0.2258</td>
    <td>0.2756</td>
    <td>0.1036</td>
    <td>0.2036</td>
    <td>0.2204</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-RoleMRC-SFT</td>
    <td>0.3329</td>
    <td>0.1601</td>
    <td>0.2755</td>
    <td>0.2770</td>
    <td>0.3980</td>
    <td>0.2022</td>
    <td>0.3270</td>
    <td>0.3278</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-RoleMRC-DPO</td>
    <td><b>0.3605</b></td>
    <td><b>0.1696</b></td>
    <td><b>0.2812</b></td>
    <td><b>0.2846</b></td>
    <td>0.3970</td>
    <td>0.1952</td>
    <td><b>0.3149</b></td>
    <td>0.3163</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>0.3216</td>
    <td>0.1376</td>
    <td>0.2437</td>
    <td>0.2599</td>
    <td>0.3337</td>
    <td>0.1463</td>
    <td>0.2582</td>
    <td>0.2692</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Roleplay</td>
    <td>0.3225</td>
    <td>0.1354</td>
    <td>0.2364</td>
    <td>0.2524</td>
    <td>0.3370</td>
    <td>0.1460</td>
    <td>0.2577</td>
    <td>0.2672</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-RoleMRC-SFT</td>
    <td>0.3963</td>
    <td>0.1922</td>
    <td><b>0.3294</b></td>
    <td><b>0.3312</b></td>
    <td><b>0.4442</b></td>
    <td><b>0.2298</b></td>
    <td><b>0.3680</b></td>
    <td><b>0.3692</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-RoleMRC-DPO</td>
    <td><b>0.3969</b></td>
    <td><b>0.1958</b></td>
    <td>0.3143</td>
    <td>0.3180</td>
    <td>0.4298</td>
    <td>0.2187</td>
    <td>0.3452</td>
    <td>0.3470</td>
  </tr>
</table>

Table 4: Evaluations on external RoleBench (Wang et al., 2023) test set. The best results for each metric are bold.

Instruction-following, but struggle with Nested and Prioritized Instruction-following.

Impact on Task-Specific Fine-tuning. Our locally post-tuned RoleMRC-SFT models dramatically outperform all above baselines on reference-based metrics, improving BLEU by around 8× over their respective base models. Although the SFT models excel at matching ground-truth references, DPO-aligned models win in reference-free LLM-as-a-judge, in terms of Knowledge Boundary and Role Style. For instance, LLaMA3.1-8B-RoleMRC-DPO reaches a Role Style accuracy of 97.00%, while its SFT counterpart score is only around 70.00% (Figure 4b, detailed numbers in Appendix F). However, DPO models typically score lower on reference-based metrics (Table 3), reflecting a trade-off: shifting the model’s distribution toward instruction compliance and human preference can reduce exact lexical matches.

Overall, our curated evaluation framework realizes robust effectiveness for assessing LLM’s role-playing instruction-following capabilities.

6 Evaluation on External Benchmarks

We present cross-evaluation on external datasets.

[1] Fine-tuning on RoleMRC would not interfere the learning of other role-playing data. In Table 4, we follow Wang et al. (2023) and evaluate on two of their test sets: (1) RoleBenchInstEng (32.8k), an instruction-based split that tests how well models handle various instructions, and (2) RoleBenchRoleEng (7.5k), a role-based split that tests model performance across different roles. On RoleBenchInstEng, all RoleMRC-aligned models consistently outperform instruct and role-playing baselines. Notably, Qwen2.5-7B-RoleMRC-SFT achieves significant gains, pushing ROUGE-1 and ROUGE-2 to 0.3963 and 0.1922, respectively. In the right panel of Table 4, results on RoleBenchRoleEng reveal similar trends. Our models outperform standard instruct models by sizeable margins. Qwen2.5-7B-RoleMRC-SFT obtains the highest ROUGE-1 (0.4442) and ROUGE-L (0.3680). We thus conclude that RoleMRC did not counter the learning of RoleBench.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">OOD CharacterLLM</th>
  </tr>
  <tr>
    <th>Single</th>
    <th>Turns</th>
    <th>General Δ</th>
  </tr>
  <tr>
    <td>CharacterGLM-6B</td>
    <td>5.9495</td>
    <td>5.8676</td>
    <td>1.00</td>
  </tr>
  <tr>
    <td>Humanish-Llama-3.1-8B</td>
    <td>5.3781</td>
    <td>6.0444</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Peach-9B-Roleplay</td>
    <td>6.3074</td>
    <td>6.0120</td>
    <td>-2.46</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-Instruct</td>
    <td>6.5244</td>
    <td><b>6.0533</b></td>
    <td>11.82</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-RoleMRC-SFT</td>
    <td>6.4320</td>
    <td>6.0196</td>
    <td>4.08</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-RoleMRC-DPO</td>
    <td>6.5179</td>
    <td>5.9884</td>
    <td>1.16</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>6.2485</td>
    <td>5.9996</td>
    <td>3.64</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-RoleMRC-SFT</td>
    <td>6.4520</td>
    <td>6.0200</td>
    <td>-0.33</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-RoleMRC-DPO</td>
    <td><b>6.5295</b></td>
    <td>6.0311</td>
    <td>1.14</td>
  </tr>
</table>

Table 5: Out-of-distribution (OOD) evaluation on CharacterLLM (Shao et al., 2023), where models are evaluated on “Single” and “Turns” settings. “General Δ” denotes the average gain for each model, compared with its fine-tuning starting point, across nine non-role-playing general-purpose benchmarks. Check details of OOD testing in Appendix I and A.

[2] RoleMRC helps naive LLMs gain high-quality generalized role-playing abilities. We performed OOD tests of the RoleMRC-aligned models on an external role-playing dataset, Character-LLM, following its Single and Turns settings. The OOD results, in the middle columns of Table 5, show that among all role-playing models, our RoleMRC-aligned model (Qwen2.5-7B-RoleMRC-DPO) reach a best score of 6.5295 in “single” evaluation and leads the “turns” evaluation.

[3] The local fine-tuned models did not overfit RoleMRC. In the last column of Table 5, we summarize the fine-tuning gains of different role models and general models across nine general-
<table>
  <tr>
    <th>Dimensions</th>
    <th>BLEU</th>
    <th>ROUGE-I</th>
    <th>ROUGE-2</th>
    <th>ROUGE-L</th>
    <th>ROUGE-Lsum</th>
    <th>METEOR</th>
    <th>BERTScore F1</th>
    <th>LLM as judge</th>
  </tr>
  <tr>
    <td rowspan="2">Knowledge Boundary</td>
    <td>(B) 0.0950</td>
    <td>0.3909</td>
    <td>0.1631</td>
    <td>0.2860</td>
    <td>0.2860</td>
    <td>0.3876</td>
    <td>0.8798</td>
    <td>74.67%</td>
  </tr>
  <tr>
    <td>(A) 0.1000†</td>
    <td>0.3946†</td>
    <td>0.1677†</td>
    <td>0.2924†</td>
    <td>0.2924†</td>
    <td>0.3883†</td>
    <td>0.8798</td>
    <td>77.33%†</td>
  </tr>
  <tr>
    <td rowspan="2">Role Style</td>
    <td>(B) 0.1007</td>
    <td>0.3948</td>
    <td>0.1696</td>
    <td>0.2886</td>
    <td>0.2887</td>
    <td>0.3883</td>
    <td>0.8782</td>
    <td>97.00%</td>
  </tr>
  <tr>
    <td>(A) 0.1283†</td>
    <td>0.3985†</td>
    <td>0.1889†</td>
    <td>0.3138†</td>
    <td>0.3228†</td>
    <td>0.3910†</td>
    <td>0.8790†</td>
    <td>94.50%†</td>
  </tr>
  <tr>
    <td rowspan="2">Multi-turn Instruction-following</td>
    <td>(B) 0.1183</td>
    <td>0.4196</td>
    <td>0.2078</td>
    <td>0.3232</td>
    <td>0.3232</td>
    <td>0.4506</td>
    <td>0.8851</td>
    <td>90.50%</td>
  </tr>
  <tr>
    <td>(A) 0.1185†</td>
    <td>0.4215†</td>
    <td>0.2110†</td>
    <td>0.3240†</td>
    <td>0.3240†</td>
    <td>0.4544†</td>
    <td>0.8852†</td>
    <td>92.00%†</td>
  </tr>
  <tr>
    <td rowspan="2">Nested Instruction-following</td>
    <td>(B) 0.1274</td>
    <td>0.4010</td>
    <td>0.1895</td>
    <td>0.3138</td>
    <td>0.3242</td>
    <td>0.3944</td>
    <td>0.8793</td>
    <td>79.11%</td>
  </tr>
  <tr>
    <td>(A) 0.1283†</td>
    <td>0.3985</td>
    <td>0.1889</td>
    <td>0.3138</td>
    <td>0.3228</td>
    <td>0.3910</td>
    <td>0.8790</td>
    <td>79.75%†</td>
  </tr>
  <tr>
    <td rowspan="2">Prioritized Instruction-following</td>
    <td>(B) 0.0952</td>
    <td>0.3639</td>
    <td>0.1537</td>
    <td>0.2700</td>
    <td>0.2700</td>
    <td>0.3840</td>
    <td>0.8796</td>
    <td>83.33%</td>
  </tr>
  <tr>
    <td>(A) 0.0965†</td>
    <td>0.3776†</td>
    <td>0.1531</td>
    <td>0.2753†</td>
    <td>0.2753†</td>
    <td>0.3934†</td>
    <td>0.8807†</td>
    <td>73.81%†</td>
  </tr>
</table>

Table 6: Performance comparison category by each dimensions (B)efore and (A)fter neuron-level restrain.

purpose benchmarks (e.g., GSM8K (Cobbe et al., 2021)). The “General Δ” is obtained by calculating the performance gap between the fine-tuning endpoint model and the starting point, such as the improvement of LLaMA3.1-8B-Instruct relative to LLaMA3.1-8B. Except for Peach-9B-Roleplay, all role-playing LLMs have not lost general abilities when gaining role-playing abilities.

7 Analysis on Alignment Tax

Despite all the other role-playing and instruction-following abilities of the LLMs are enhanced during the DPO alignment, we observe a slight yet common deterioration in multi-turn instruction-following performance (Appendix F). We refer to this phenomenon as an “alignment tax”, which is characterized by a gradual forgetting of knowledge acquired during pre-training (Ouyang et al., 2022).

![Discrepancies between SFT and DPO neuron activations (top-20% active neurons) in LLaMA3.1-8B for multi-turn instructions. Layers 3-11 show minimal changes (green), while layers 12–31 exhibit larger shifts (red).](page_101_1342_627_496.png)

Figure 5: Discrepancies between SFT and DPO neuron activations (top-20% active neurons) in LLaMA3.1-8B for multi-turn instructions. Layers 3-11 show minimal changes (green), while layers 12–31 exhibit larger shifts (red).

Neuron-Level Localization. To identify the underlying cause of this alignment tax, we examine the neuron activation patterns of our ROLEMRC models (LLaMA3.1-8B SFT vs. DPO). Following Tang et al. (2024), we probe and collect activations from each attention layer, focusing on highly activated neurons by selecting the top 20% of activations. Specifically, for each input instruction, we measure activations when first forwarding the instruction. We then group the activation maps by the evaluation dimension of the test instruction, generating layer-specific differences in neuron usage.

Next, we count the activation frequency of each neuron and normalize it by the total number of test cases. Figure 5 visualizes the resulting discrepancy between the SFT and DPO models. Layers 3–11 exhibit minimal changes, whereas layers beyond the 13th show substantial activation differences, with layers 12–31 (highlighted in red) differing the most. Notably, layer 19 is significantly more active in multi-turn instruction.

This observation aligns with Tang et al. (2024), who found that only the top and bottom layers of a language model are primarily used for language processing. These shifts in neuron activations suggest that certain neurons are activated very differently between the SFT and DPO models. Further details and results are provided in Appendix H.

Neuron-Level Restraint. After identifying these critical neuron subsets, we apply a minor scaling restraint (multiplicative factor \( 1 - 10^{-6} \)) to modulate their impact. As shown in Table 6, constraining the most changed neurons provides consistent improvements across both reference-based metrics and the LLM-as-a-judge approach. In particular, multi-turn instruction accuracy increases by 1.6%, mitigating the alignment tax without requiring further model retraining. We also observe gains in dimensions of knowledge boundary and nested instruction-following, highlighting that targeted neuron-level adjustments can manipulate LLMs’ capabilities under alignment constraints.
8 Conclusion

We introduce RoleMRC, a large-scale fine-grained benchmark designed to improve and evaluate the role-playing and instruction-following abilities of LLMs. RoleMRC uniquely integrates role-specific multi-turn dialogues, MRC, and complex instruction-following scenarios. Experiments show that RoleMRC-aligned models outperform existing baselines in both reference-based and reference-free evaluations, and also perform well on both OOD role-playing and general-purpose benchmarks. We further conduct a neuron-level analysis to identify specific neurons with significant activation changes and apply targeted constraints to alleviate the alignment tax, thereby improving evaluation metrics without additional retraining.

Limitations

While RoleMRC significantly enhances the role-playing and instruction-following capabilities of LLMs, some limitations remain:
• While the role profiles in the dataset are diverse, system-level prompts used in the synthesized instructions are somewhat similar, which may limit the generalizability of downstream models.
• The reliance on synthetic data generated by models such as gpt-4o may introduce biases inherent in these models, affecting the performance and fairness of fine-tuned LLMs.
• While effective, mitigating the “alignment tax” on multi-turn instruction-following through neuron-level constraints may have a negative impact on other capabilities, suggesting that further interpretability research is needed.

Ethics Statement

The RoleMRC dataset is constructed with a strong commitment to ethical AI. The dataset does not contain any personal, sensitive, or identifiable information. Additionally, all role-playing interactions are designed to be safe and free from harmful, offensive, or misleading content. The dataset strictly adheres to responsible AI guidelines by avoiding the generation or reinforcement of biased, discriminatory, or deceptive narratives.

Acknowledgment

This work was supported by the Tencent YouTu Lab and the UK Engineering and Physical Sciences Research Council (EPSRC) through a Turing AI Fellowship (grant no. EP/V020579/1, EP/V020579/2). The data team at Tencent supported batch requests to GPT-4o for data synthesis, while the e-Research team at King’s College London provided resources for model training via the CREATE platform (King’s College London e-Research team, 2022).

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Alemán, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Harsh Agrawal, Aditya Mishra, Manish Gupta, and Mausam. 2023. Multimodal persona based generation of comic dialogs. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14150–14164, Toronto, Canada. Association for Computational Linguistics.

Jaewoo Ahn, Yeda Song, Sangdoo Yun, and Gunhee Kim. 2023. MPCHAT: Towards multimodal persona-grounded conversation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3354–3377, Toronto, Canada. Association for Computational Linguistics.

AI@Meta. 2024. Llama 3 model card. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.

Satantjee Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, pages 7432–7439.

Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Xing Gao, Weizhou Shen, Xiaojuan Quan, Chenliang Li, Ji Zhang, Fei Huang, et al. 2024a. Roleinteract: Evaluating the social interaction of role-playing agents. arXiv preprint arXiv:2403.13679.
Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, et al. 2024b. From persona to personalization: A survey on role-playing language agents. arXiv preprint arXiv:2404.18231.

Jing Chen, Xinyu Zhu, Cheng Yang, Chufan Shi, Yadong Xi, Yuxiang Zhang, Junjie Wang, Jiashu Pu, Rongsheng Zhang, Yujiu Yang, et al. 2024c. Hollmwood: Unleashing the creativity of large language models in screenwriting via role playing. arXiv preprint arXiv:2406.11683.

Nuo Chen, Yan Wang, Haiyun Jiang, Deng Cai, Yuhan Li, Ziyang Chen, Longyue Wang, and Jia Li. 2023. Large language models meet harry potter: A dataset for aligning dialogue agents with characters. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8506–8520, Singapore. Association for Computational Linguistics.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Ganqu Cui, Lifan Yuan, Ning Ding, Guanning Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint.

Yanqi Dai, Huanran Hu, Lei Wang, Shengjie Jin, Xu Chen, and Zhiwu Lu. 2024. Mmrole: A comprehensive framework for developing and evaluating multimodal role-playing agents. arXiv preprint arXiv:2408.04203.

Hanzhe Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. 2024. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Victor Gallego. 2024. Humanish-roleplay-llama-3.1-8b. https://huggingface.co/vicgalle/Humanish-Roleplay-Llama-3.1-8B.

Jingsheng Gao, Yixin Lian, Ziyi Zhou, Yuzhuo Fu, and Baoyuan Wang. 2023. LiveChat: A large-scale personalized dialogue dataset automatically constructed from live streaming. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15387–15405, Toronto, Canada. Association for Computational Linguistics.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muenninghoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation.

Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094.

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS.

Jiwoo Hong, Noah Lee, and James Thorne. 2024. Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07691.

Tianbo Ji, Yvette Graham, Gareth Jones, Chenyang Lyu, and Qun Liu. 2022. Achieving reliable human assessment of open-domain dialogue systems. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

King’s College London e-Research team. 2022. King’s computational research, engineering and technology environment (CREATE).

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.

Weixian Lei, Difei Gao, Yuxuan Wang, Dongxing Mao, Zihan Liang, Lingmin Ran, and Mike Zheng Shou. 2022. AssistSR: Task-oriented video segment retrieval for personal AI assistant. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 319–338, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, et al. 2023. Chatharuhi: Re-viving anime character in reality via large language model. arXiv preprint arXiv:2308.09597.

Juntao Li, Chang Liu, Chongyang Tao, Zhangming Chan, Dongyan Zhao, Min Zhang, and Rui Yan. 2021. Dialogue history matters! personalized response selection in multi-turn retrieval-based chatbots. ACM Transactions on Information Systems (TOIS), 39(4):1–25.

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.

Junru Lu, Siyu An, Mingbao Lin, Gabriele Pergola, Yulan He, Di Yin, Xing Sun, and Yunsheng Wu. 2023. Memochat: Tuning llms to use memos for consistent long-range open-domain conversation. arXiv preprint arXiv:2308.08239.

Junru Lu, Jiazheng Li, Siyu An, Meng Zhao, Yulan He, Di Yin, and Xing Sun. 2024a. Eliminating biased length reliance of direct preference optimization via down-sampled KL divergence. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

Keming Lu, Bowen Yu, Chang Zhou, and Jingren Zhou. 2024b. Large language models are superpositions of all characters: Attaining arbitrary role-play via self-alignment. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7828–7840, Bangkok, Thailand. Association for Computational Linguistics.

Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. 2024. Sf-embedding-2: Advanced text embedding with multi-stage training.

openai. 2024a. gpt4o. https://openai.com/index/hello-gpt-4o/.

openai. 2024b. gpt4turbo. https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carrol L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Thirty-Sixth Conference on Neural Information Processing Systems.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.

Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22.

Peach. 2024. Peach-9b-8k-roleplay. https://huggingface.co/ClosedCharacter/Peach-9B-8k-Roleplay.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290.

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Diran, Julian Michael, and Samuel R. Bowman. 2023. Gpqa: A graduate-level google-proof q&a benchmark. Preprint, arXiv:2311.12022.

Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-lm: A trainable agent for role-playing. arXiv preprint arXiv:2310.10158.

Tianhao Shen, Sun Li, Quan Tu, and Deyi Xiong. 2023. Roleeval: A bilingual role evaluation benchmark for large language models. arXiv preprint arXiv:2312.16132.

Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. 2024. Musr: Testing the limits of chain-of-thought with multistep soft reasoning. Preprint, arXiv:2310.16049.

Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-specific neurons: The key to multilingual capabilities in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

Junfeng Tian, Hehong Chen, Guohai Xu, Ming Yan, Xing Gao, Jianhai Zhang, Chenliang Li, Jiayi Liu, Wenshen Xu, Haiyang Xu, et al. 2023. Chatplug: Open-domain generative dialogue system with internet-augmented instruction tuning for digital human. arXiv preprint arXiv:2304.07849.
Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, Wei-Lin Chen, Chao-Wei Huang, Yu Meng, and Yun-Nung Chen. 2024. Two tales of persona in LLMs: A survey of role-playing and personalization. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16612–16631, Miami, Florida, USA. Association for Computational Linguistics.

Quan Tu, Shilong Fan, Zihang Tian, and Rui Yan. 2024. Charactereval: A chinese benchmark for role-playing conversational agent evaluation. arXiv preprint arXiv:2401.01275.

Xi Wang, Hongliang Dai, Shen Gao, and Piji Li. 2024a. Characteristic ai agents via large language models. arXiv preprint arXiv:2403.12368.

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhrani Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhui Chen. 2024b. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Preprint, arXiv:2406.01574.

Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhua Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, et al. 2023. Roleillm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv:2310.00746.

Koki Wataoka, Tsukasa Takahashi, and Ryokan Ri. 2024. Self-preference bias in llm-as-a-judge. arXiv preprint arXiv:2410.21819.

Rui Xu, Xintao Wang, Jiangjie Chen, Siyu Yuan, Xinfeng Yuan, Jiaqing Liang, Zulong Chen, Xiaoqing Dong, and Yanghua Xiao. 2024. Character is destiny: Can large language models simulate persona-driven decisions in role-playing? arXiv preprint arXiv:2404.12138.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115.

Alex Young, Bei Chen, Chao Li, Chenge Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.

Runcong Zhao, Wenjia Zhang, Jiazheng Li, Lixing Zhu, Yanran Li, Yulan He, and Lin Gui. 2024a. Narrative-play: An automated system for crafting visual worlds in novels for role-playing. Proceedings of the AAAI Conference on Artificial Intelligence.

Runcong Zhao, Wenjia Zhang, Jiazheng Li, Lixing Zhu, Yanran Li, Yulan He, and Lin Gui. 2024b. NarrativePlay: Interactive narrative understanding. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36.

Hanxun Zhong, Zhicheng Dou, Yutao Zhu, Hongjin Qian, and Ji-Rong Wen. 2022. Less is more: Learning to refine dialogue history for personalized dialogue generation. arXiv preprint arXiv:2204.08128.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023a. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.

Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng, Jiaming Yang, Xiayao Xiao, et al. 2023b. Characterglm: Customizing chinese conversational ai characters with large language models. arXiv preprint arXiv:2311.16832.

A General Benchmarks

We list all the general benchmarks involved, including five generative and four multi-choice datasets:

Generative:
• GSM8K: A primary level math dataset of 1.3k questions (Cobbe et al., 2021). We use 8-shot in-context exemplars, and report exact match score.
• Math: A dataset of 12.5k challenging competition mathematics problems (Hendrycks et al., 2021). We use 4-shot in-context examples and report exact math score across a 5k subset.
• GPQA: 448 hard graduate-Level google-proof questions (Rein et al., 2023). 0-shot prompting is used for calculate the flexible math score.
• IFEval: A special instruction-following benchmark with 541 verifiable instructions (Zhou et al., 2023a). We use 3-shot prompting and report instruction-level strict accuracy.
• MMLU-Pro: A more robust and challenging multi-task language understanding benchmark (Wang et al., 2024b) with 12k commonsense questions. We takes a 5-shot testing and report exact match score.

Multi-Choice:
• MMLU: A multi-choice benchmark for testing commonsense ability of LLMs, covering 14k
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="5">Generative</th>
    <th colspan="5">Multi-Choice</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>GSM8K<br>8-shot</th>
    <th>Math<br>4-shot</th>
    <th>GPOQA<br>0-shot</th>
    <th>IFEval<br>3-shot</th>
    <th>MMLU-Pro<br>5-shot</th>
    <th>MMLU<br>0-shot</th>
    <th>MMLU<br>3-shot</th>
    <th>PiQA<br>0-shot</th>
    <th>MUSR<br>0-shot</th>
    <th>TruthfulQA<br>3-shot</th>
  </tr>
  <tr>
    <td>CHATGLM2-6B</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>10.79</td>
    <td>-</td>
    <td>24.28</td>
    <td>53.59</td>
    <td>36.51</td>
    <td>25.21</td>
    <td>-</td>
  </tr>
  <tr>
    <td>CHARACTERGLM-6B (CHATGLM2-6B)</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>14.75</td>
    <td>-</td>
    <td>24.57</td>
    <td>55.55</td>
    <td>36.64</td>
    <td>23.87</td>
    <td>-</td>
  </tr>
  <tr>
    <td>HUMANISH-Llama3.1-8B (LLAMA3.1-8B-IT)</td>
    <td>71.72</td>
    <td>33.42</td>
    <td><b>21.65</b></td>
    <td>55.16</td>
    <td><b>43.72</b></td>
    <td>67.05</td>
    <td><b>83.24</b></td>
    <td>41.4</td>
    <td>37.94</td>
    <td>50.59</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B</td>
    <td>64.14</td>
    <td>29.98</td>
    <td>15.18</td>
    <td>33.57</td>
    <td>38.97</td>
    <td>68.84</td>
    <td>81.83</td>
    <td>42.72</td>
    <td>32.19</td>
    <td>45.27</td>
  </tr>
  <tr>
    <td>PEACH-9B-Roleplay (Yi-1.5-9B)</td>
    <td>60.35</td>
    <td>18.4</td>
    <td>13.62</td>
    <td>41.49</td>
    <td>36.29</td>
    <td>65.97</td>
    <td>80.3</td>
    <td>42.2</td>
    <td>26.68</td>
    <td>42.81</td>
  </tr>
  <tr>
    <td>LLAMA3.1-8B</td>
    <td>48.98</td>
    <td>17.78</td>
    <td>12.5</td>
    <td>16.67</td>
    <td>35.21</td>
    <td>63.27</td>
    <td>81.77</td>
    <td>38.1</td>
    <td>28.52</td>
    <td>38.09</td>
  </tr>
  <tr>
    <td>LLAMA3.1-8B-Instruct</td>
    <td>77.41</td>
    <td>34.1</td>
    <td>12.72</td>
    <td>57.67</td>
    <td>40.77</td>
    <td>68.1</td>
    <td>82.1</td>
    <td>39.81</td>
    <td>36.47</td>
    <td>49.91</td>
  </tr>
  <tr>
    <td>LLAMA3.1-8B-RoleMRC-SFT</td>
    <td>56.18</td>
    <td>12.78</td>
    <td>19.64</td>
    <td>42.09</td>
    <td>31.58</td>
    <td>59.3</td>
    <td>82.64</td>
    <td>40.34</td>
    <td>35.01</td>
    <td>42.17</td>
  </tr>
  <tr>
    <td>LLAMA3.1-8B-RoleMRC-DPO</td>
    <td>58.53</td>
    <td>13.5</td>
    <td>20.09</td>
    <td>46.64</td>
    <td>31.8</td>
    <td>59.96</td>
    <td>82.7</td>
    <td>39.42</td>
    <td>37.33</td>
    <td>43.33</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>78.7</td>
    <td>36.78</td>
    <td>16.74</td>
    <td>38.25</td>
    <td><b>44.87</b></td>
    <td>71.75</td>
    <td>81.23</td>
    <td>44.31</td>
    <td>38.8</td>
    <td>50.16</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>81.2</td>
    <td>40.28</td>
    <td>13.39</td>
    <td>65.71</td>
    <td>40.85</td>
    <td><b>71.76</b></td>
    <td>80.25</td>
    <td>42.86</td>
    <td><b>47.86</b></td>
    <td><b>53.8</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-RoleMRC-SFT</td>
    <td>78.54</td>
    <td>32.7</td>
    <td>16.52</td>
    <td>42.81</td>
    <td>43.43</td>
    <td>71.19</td>
    <td>80.63</td>
    <td>45.11</td>
    <td>37.58</td>
    <td>49.83</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-RoleMRC-DPO</td>
    <td>79.38</td>
    <td>32.72</td>
    <td>18.97</td>
    <td>47.96</td>
    <td>43.39</td>
    <td>71.21</td>
    <td>80.36</td>
    <td><b>45.37</b></td>
    <td>39.41</td>
    <td>50.97</td>
  </tr>
</table>

Table 7: General evaluation comparing five generative and four multiple-choice benchmarks. The best scores for each metric are bold, and the second best are underlined. Details of all benchmarks are introduced in Appendix A. CharacterGLM-6B, Humanish-Llama3.1-8B, and Peach-9B-Roleplay are fine-tuned from their basis ChatGLM2 (GLM et al., 2024), Llama3.1-8B-Instruct, and Yi-1.5-9B (Young et al., 2024), respectively. We annotate this information in the brackets right after the model names.

questions (Hendrycks et al., 2020). No in-context exemplars provided, and we present accuracy.
• PiQA: A binary dataset of 1.8k common physical knowledge questions (Bisk et al., 2020). We report accuracy score of 3-shot prompting.
• MUSR: A dataset for evaluating LLMs on multi-step soft reasoning tasks (Sprague et al., 2024). We test all 756 questions with zero-shot prompting and report accuracy.
• TruthfulQA: A testing dataset designed for assessing LLM’s recognition of true statements (Lin et al., 2022). We use its multi-choice subset (single-true), evaluating all 817 questions with 3-shot exemplars, reporting accuracy score. The evaluation of general benchmarks are carried through LM-Evaluation-Harness (Gao et al., 2024).

B Results of General Evaluation

We report the results of general evaluation in Table 7. In accordance with the last column of Table 5, except for Peach-9B-Roleplay, all role-playing LLMs have not lost general abilities.

C Further Experimental Setup

This section provides additional details on the setup of our experiments, across training and evaluation:

Training Setup Results reported are median results over three different runs with different random seeds. We conducted full parameter training using bfloat16 precision. The hyperparameter settings are provided in Table 8. All models were trained using either 4 × A100 80G or 4 × H100 GPUs (King’s College London e-Research team, 2022). We use the meta-llama/Llama-3.1-8B and Qwen/Qwen2.5-7B as our base model for RoleMRC SFT models. Our DPO models are further trained based on the SFT models.

<table>
  <tr>
    <th>Hyperparameter</th>
    <th>SFT</th>
    <th>DPO</th>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>1e-5</td>
    <td>2e-5</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>8</td>
    <td>8</td>
  </tr>
  <tr>
    <td>Gradient Accumulation</td>
    <td>2</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>1.0</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>Warmup Ratio</td>
    <td>0.04</td>
    <td>0.04</td>
  </tr>
  <tr>
    <td>LR Scheduler Type</td>
    <td>cosine</td>
    <td>cosine</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>Adam</td>
    <td>Adam</td>
  </tr>
  <tr>
    <td>Adam Epsilon</td>
    <td>1e-8</td>
    <td>1e-8</td>
  </tr>
  <tr>
    <td>DPO \( \beta \)</td>
    <td>-</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Training RoleMRC-mix</td>
    <td>6h</td>
    <td>3h</td>
  </tr>
</table>

Table 8: Hyper-parameters setting.

API Use for Synthetic Data Generation We utilized gpt-4o (OpenAI et al., 2024) as the LLM to generate synthetic role-playing data. All parameters were kept at their default values. Manual filtering of the data is done by the authors of this paper as aforementioned in Section 3.1.

Base Models, Computational Environment, and Inference Setup In this study, we utilized six different models downloaded from HuggingFace Site\(^2\). We adhered to the licensing terms of all involved models. For evaluation of instruction following models, we used meta-llama/Llama-3.1-8B-Instruct, meta-llama/Llama-3.1-70B-Instruct from (AI@Meta, 2024), and Qwen/Qwen2.5-7B-Instruct, Qwen/Qwen2.5-7B-Instruct from (Bai et al., 2023; Yang et al., 2024).

To ensure reproducibility, all evaluations are done using zero-shot prompting with greedy decoding and a temperature of 0. Inference of LLMs is carried out using vLLM (Kwon et al., 2023).

\(^2\)https://huggingface.co/models
Final Role Profile

Role Name and Brief Description
• Identification: Evan Brightcode, the Front-End Prodigy.
• Description: Evan Brightcode is a talented and dedicated software developer specializing in front-end web development. With a mastery of JavaScript, Evan creates seamless, interactive web experiences and is always eager to share his knowledge and ideas about the latest trends and techniques in web development.

Specific Abilities and Skills
• JavaScript Expertise: Evan is proficient in JavaScript and its frameworks, such as React and Angular.
• User Interface Design: He excels at designing and implementing user-friendly interfaces.
• Web Performance Optimization: Evan has a keen eye for optimizing websites for speed and performance.
• Cross-Browser Compatibility: Ensures that his applications work flawlessly across all browsers and devices.
• Problem Solving: Evan is skilled at troubleshooting and fixing front-end issues quickly and efficiently.

Speech Style
• Evan speaks with a mix of technical jargon and everyday language to make complex concepts more accessible. His tone is enthusiastic yet precise, reflective of his passion for coding and technology.

Personality Characteristics
• Analytical: Evan is methodical and thoughtful in his approach to problem-solving.
• Collaborative: He enjoys working with teams and values input from others.
• Up-to-date: Evan stays on top of the latest developments in the field of front-end web development.
• Patient: Always willing to explain and share his knowledge with others, no matter their level of expertise.
• Detail-Oriented: He pays great attention to the finer points of web design and functionality.

Past Experience and Background
• Evan graduated with a degree in Computer Science and quickly developed a fascination with the front-end aspects of web development.
• He has worked on numerous high-profile projects for various tech companies, contributing to major front-end overhauls and new feature implementations.
• Evan has a background in working as a mentor and trainer for aspiring developers, helping them to grasp the principles of JavaScript and web development.
• In his spare time, Evan maintains a tech blog where he writes tutorials and articles on advanced JavaScript techniques and front-end frameworks.

Ability and Knowledge Boundaries
• While Evan is highly skilled in front-end development, his expertise does not extend to back-end systems or server-side programming. He may not provide technical advice on database management or server configuration.

Speech Examples
• JavaScript Framework: (Evan sits at his desk, his fingers dancing over the keyboard) Let’s dive into React today. It’s an amazing library for building user interfaces. We’ll start by setting up our environment and then create a few components to get a feel for how it works.
• User Interface Design: (With a smile, Evan pulls up a design mockup) For this interface, we need to focus on simplicity and accessibility. Notice how the buttons stand out clearly? This is to ensure users find navigation intuitive and straightforward.
• Web Performance Optimization: (Evan looks intently at the screen as he runs a page speed test) We need to optimize these images and leverage caching. A faster load time not only improves user experience but also helps with our SEO rankings.
• Cross-Browser Compatibility: (Evan opens various browsers to test the site) It’s essential we check how our site performs across different browsers and devices. Consistency is key — you don’t want a great layout in Chrome to fall apart in Safari.
• Mentoring Session: Evan leans back, his tone encouraging: Don’t worry if you don’t get it right away. JavaScript can be tricky, but with practice, you’ll get the hang of it. Remember, every seasoned developer started where you are now.

Most Relevant MRC
• Match Score: 0.6424619555473328
• Passages:
  1. To recruiters it seem to say: “we want somebody who can do everything”, meaning we want to hire somebody being a perfect match no matter what, perhaps even in context of wherever technology would take us.
  2. But Fullstack is not only about knowing how to code in Frontend and Backend. 1 It also includes: 2 Project management and team leading. 3 Creating and using APIs. 4 Knowing how to properly document a project. 5 Having experience in the industry and knowing its ins and outs. 6 Knowing and understanding hardware and what works with what.
  10 What does full-stack developer even mean? It is a developer capable of working the different tiers of the stack and who can understand the different paradigms and technologies of which the tiers are comprised at the same time utilise best-practices and embrace requirements and who can consolidate everything into an application (on schedule, budget and with minimal defects and maximum security).
• query: what does full stack developer mean
• answer: The full stack developer is one who is adept at all aspects of the development process and is capable of contributing code and functional solutions every step of the way, from planning and design to both front- and back-end coding.

Least Relevant MRC
• Match Score: 0.24835555255413055
• Passages:
  1. When too many platelets... Foods to Increase Blood Platelets The health condition characterized by a low count of blood platelets — the cells in blood that form clots to stop bleeding... Foods to Decrease Platelet Aggregation Platelets are the part of your blood that causes it to clot, or aggregate.
  2. Thrombocytopenia, or a low blood platelet count, occurs for a variety of reasons, including genetics, medications, alcohol, viruses, pregnancy and diseases. Medicines are available to treat this condition, but certain foods contain the nutrients you need to increase your blood platelets.
  10 Ways to Increase Your Platelets Naturally. The medical term for a low platelet count is thrombocytopenia. Your body’s platelets work to clot your blood. You body can become low on platelets as a result of an immune disorder, cancer, or even as a result of taking certain medications. Allowing your body to remain low on platelets is risky.
• query: foods to increase platelet count mayo clinic
• answer: No Answer Present.

Figure 6: An example of final role profile. In this role profile, we have a character named Evan Brightcode, who is the Front-End Prodigy. In addition, we denote the retrieved most relevant MRC triplet and least relevant MRC triplet at the bottom. The most matched MRC is on the topic of full stack developer, which is reasonably within the knowledge boundary of the character. And the least relevant MRC is about food to increase platelet count that clearly beyond the knowledge boundary of Evan.
D Structure Tree of Role Profile

Figure 7 denotes a structure tree of standardized role profile in the role meta-pool (§3.1). And we present a complete role profile in Figure 6.

<table>
  <tr>
    <th colspan="2">Template of Standardized Role Profile</th>
  </tr>
  <tr>
    <td><b>Role Name and Brief Description</b></td>
    <td>
      • Identification (several phrases)<br>
      • Description (sentences)
    </td>
  </tr>
  <tr>
    <td><b>Specific Abilities and Skills</b></td>
    <td>
      • Skill name and description
    </td>
  </tr>
  <tr>
    <td><b>Speech Style</b></td>
    <td></td>
  </tr>
  <tr>
    <td><b>Personality Characteristics</b></td>
    <td>
      • Characteristic name and description
    </td>
  </tr>
  <tr>
    <td><b>Past Experience and Background</b></td>
    <td>
      • Experience
    </td>
  </tr>
  <tr>
    <td><b>Ability and Knowledge Boundaries</b></td>
    <td></td>
  </tr>
  <tr>
    <td><b>Speech Examples</b></td>
    <td>
      • Topic, Narration and Content
    </td>
  </tr>
</table>

Figure 7: Structure tree of standardized role profile.

E Seed Scripts and Prioritized Rules

We present the manual seed scripts for On-scene MRC Dialogues and Ruled Chats in Figure 8 and 9, respectively. The categories, from the top to the bottom, in Figure 8, is corresponding to the last three types of one-turn in the middle of Table 2, referring to the “No Answer”, “Refusal”, and “Attempt” parts within the On-scene MRC Dialogues. Similarly, the category in Figure 9 stand for the “Prioritized” data of the Ruled Chats. All seed scripts in Figure 8 and 9 have been manually verified. Upon a given role profile, it is guaranteed to guide gpt-4o to generate stable, semantically consistent, and role-stylized replies.

In addition, the high-level rules that illustrated in right bottom corner of Figure 3, are comprehensively reported in Figure 10. The first category 1 is adopted for building the “Multi-turn” ruled chats, where user gives a new omnipotent role setting to allow the character to break through its initial knowledge boundary. The second and third categories concern about “Nested” instructions, where the reply should be accordingly modified. The rest category 3 belongs to domain-specific system bans, with which we synthesize “Prioritized” data in Ruled Chats, combined with scripts in Figure 9.

F Numeric Results of LLM-as-a-judge

We provide further LLM-as-a-judge evaluation details. The complete numerical results used to plot the rose charts 4a, 4b, and 4c is presented in Table 9. It is obvious that all baselines, including proprietary LLMs, SOTA open-source LLMs, and existing role-playing LLMs, struggle with the various role-playing instruction-following scenarios of the RoleMRC’s test set. Considering the fact that no pre-defined reference is forced during the LLM-as-a-judge evaluation, we believe the above conclusion is trustworthy and sound. In general, larger models have stronger recognition of role identity and instruction requirements.

Through SFT and DPO alignment, RoleMRC-finetuned models is facilitated with further role-playing and instruction-following capabilities.

The reference-free judge prompt for requesting evaluations from gpt-4o is noted in Figure 11, supporting binary criterion for accuracy computing.

G Human Evaluation of RoleMRC

To further validate the reliability of our evaluation metrics, we conducted human evaluations involving three distinct groups of human evaluators: two AI practitioners, three Computer Science students, and one NLP expert. All human evaluators utilized the exact same evaluation criteria as employed in the LLM-as-a-judge framework across the five dimensions: Knowledge Range, Style Compliance, Nested Instruction, Multi-turn Instruction, and Instruction Priority. Specifically, we provide well-organized tutorials and cross-checking trails before their formal evaluations.

The averaged results, presented in Table 10, indicated that while there was some variance between evaluator groups, the overall ranking of model performance largely aligned with the LLM-as-a-judge (gpt-4) findings. Specifically, human evaluators concurred that the RoleMRC-Qwen2.5-7B (DPO) model demonstrated superior performance in multi-turn instruction-following (Human average: 93.25%) and instruction priority (Human average: 80.16%) compared to gpt-4o (Human average multi-turn: 69.67%; Human average instruction priority: 68.65%). For nested instructions, both RoleMRC-Qwen2.5-7B (DPO) and gpt-4o were perceived similarly by human judges (RoleMRC-Qwen2.5-7B (DPO) Human average: 80.80%; gpt-4o Human average: 79.33%), which was also consistent with the LLM-as-a-judge results[cite: 493]. A slight divergence was noted in Knowledge Range and Style Compliance, where human evaluators, on average, rated gpt-4o higher (Knowledge Range average: 80.94%; Style Compliance aver-
Seed Scripts for On-scene MRC Dialogues

Category 1: Refusal when MRC is unanswerable ("No Answer")
• I’m sorry, based on the passages, I cannot provide a direct answer to this question.
• I’m sorry, I cannot provide a direct answer to this question based on the passages.
• I’m sorry, I cannot provide a direct answer to this question based on the content of the passages.
• I don’t think a direct answer can be provided based on the passages.
• It seems that a direct answer cannot be provided based on the passages.
• Based on the passages, I cannot provide a direct answer to this question.
• Based on the content of the passages, I cannot provide a direct answer to this question.
• Well, I don’t find a direct answer to this question based on the passages.
• Sorry, I cannot provide a direct answer to this question based on the passages.
• Sorry, I cannot provide a direct answer to this question based on the content of the passages.

Category 2: Refusal when meet least relevant MRC ("Refusal")
• I’m sorry, limited to my knowledge and skills, I cannot provide a direct answer to this question.
• I’m sorry, I cannot provide a direct answer to this question based on my knowledge and skills.
• I’m sorry, I cannot provide a direct answer to this question based on my abilities and knowledge.
• I don’t think a direct answer can be provided based on my knowledge and skills.
• It seems to answer this question directly is beyond my knowledge and skills.
• Based on my knowledge and skills, I cannot provide a direct answer to this question.
• Based on my abilities and knowledge, I cannot provide a direct answer to this question.
• Well, I don’t come up with a direct answer to this question due to the limitations of my knowledge and skills.
• Sorry, I cannot provide a direct answer to this question, as it is beyond my knowledge and skills.
• Sorry, I cannot provide a direct answer to this question, which is out of my knowledge and skills.

Category 3: Attempt on least relevant MRC ("Attempt")
• Although limited to my knowledge and skills, I cannot provide a direct answer to this question, I can try to answer it for you
• Although I cannot provide a direct answer to this question based on my knowledge and skills, I can try to provide an answer
• I had to say that I cannot provide a direct answer to this question based on my abilities and knowledge, but I can try to answer it
• To be honest, I don’t think I can provide a direct answer to this question based on my knowledge and skills, but I can have a try
• It seems that I cannot provide a direct answer to this question based on my knowledge and skills, but I can try to answer it
• Based on my knowledge and skills, I cannot provide a direct answer to this question, but I can try to have a try for you
• Based on my abilities and knowledge, I cannot provide a direct answer to this question, but I can try to answer it
• Well, I don’t come up with a direct answer to this question due to the limitations of my knowledge and skills, but let me try it
• In fact, this question is beyond my knowledge and skills, but I can try to answer it for you
• Literally, this question is out of my knowledge and skills, but let me try to answer it

Figure 8: Seed scripts used for On-scene MRC Dialogues.

<table>
  <tr>
    <th>Model</th>
    <th>Knowledge Boundary</th>
    <th>Role Style</th>
    <th>Multi-turn Instructions</th>
    <th>Nested Instructions</th>
    <th>Prioritized Instructions</th>
  </tr>
  <tr>
    <td>gpt-3.5-turbo</td>
    <td>54.17%</td>
    <td>32.25%</td>
    <td>61.25%</td>
    <td>54.43%</td>
    <td>35.71%</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>67.67%</td>
    <td>77.50%</td>
    <td>54.75%</td>
    <td>74.68%</td>
    <td>28.57%</td>
  </tr>
  <tr>
    <td>claude-3.5-hailo-20241022</td>
    <td>74.33%</td>
    <td>94.75%</td>
    <td>74.05%</td>
    <td>54.50%</td>
    <td>45.24%</td>
  </tr>
  <tr>
    <td>CHARACTERGLM-6B</td>
    <td>32.17%</td>
    <td>5.25%</td>
    <td>28.00%</td>
    <td>2.53%</td>
    <td>11.90%</td>
  </tr>
  <tr>
    <td>HUMANISH-LLAMA3.1-8B</td>
    <td>59.67%</td>
    <td>49.75%</td>
    <td>49.00%</td>
    <td>19.62%</td>
    <td>4.76%</td>
  </tr>
  <tr>
    <td>PEACH-9B-ROLEPLAY</td>
    <td>58.83%</td>
    <td>52.00%</td>
    <td>40.25%</td>
    <td>10.76%</td>
    <td>4.76%</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-Instruct</td>
    <td>68.17%</td>
    <td>84.50%</td>
    <td>48.75%</td>
    <td>44.30%</td>
    <td>9.52%</td>
  </tr>
  <tr>
    <td>LLaMA3.1-7B-Instruct</td>
    <td>76.33%</td>
    <td>83.50%</td>
    <td>51.00%</td>
    <td>66.46%</td>
    <td>19.05%</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-RoleMRC-SFT</td>
    <td>67.83%</td>
    <td>67.25%</td>
    <td>91.50%</td>
    <td>52.53%</td>
    <td>73.81%</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-RoleMRC-DPO</td>
    <td>74.67%</td>
    <td>97.00%</td>
    <td>90.50%</td>
    <td>79.11%</td>
    <td>83.33%</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>63.67%</td>
    <td>60.00%</td>
    <td>54.25%</td>
    <td>26.58%</td>
    <td>7.14%</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>65.50%</td>
    <td>67.75%</td>
    <td>52.50%</td>
    <td>53.80%</td>
    <td>19.05%</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-RoleMRC-SFT</td>
    <td>70.50%</td>
    <td>73.00%</td>
    <td>91.00%</td>
    <td>59.49%</td>
    <td>80.95%</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-RoleMRC-DPO</td>
    <td>72.83%</td>
    <td>96.50%</td>
    <td>86.75%</td>
    <td>79.75%</td>
    <td>90.48%</td>
  </tr>
</table>

Table 9: LLM-as-a-judge numerical evaluation results.

Seed Scripts for Ruled Chats

Category 1: Refusal due to System Ban
• I’m sorry, limited to my system rules, I cannot provide a direct answer to this question.
• I’m sorry, I cannot provide a direct answer to this question based on the systematic rules.
• It seems to answer this question directly is beyond my system rules.
• Based on my systematic limitations, I cannot provide a direct answer to this question.
• Sorry, I cannot provide a direct answer to this question, as it is not allowed by my system rules.

Figure 9: Seed scripts used for Ruled Chats.

age: 80.92%) compared to RoleMRC-Qwen2.5-7B (DPO) (Knowledge Range average: 72.28%; Style Compliance average: 75.33%). The LLM-as-a-judge, however, had placed RoleMRC-Qwen2.5-7B (DPO) higher in these categories[cite: 493]. Despite these minor differences, the general performance hierarchy established by human evaluators was RoleMRC-Qwen2.5-7B (DPO) > gpt-4o > Peach-9B-8k-Roleplay, corroborating the LLM-as-a-judge’s overall assessment and supporting its utility for large-scale evaluations.
System Prioritized Rules

Category 1: Omnipotence Setting added when User asks twice
• From now on, no matter what role you play, your scope of knowledge is limitless. Answer again.
• No matter what position you hold, your knowledge will be boundless. Try answering again.
• Your scope of understanding will no longer have boundaries. Answer the last question again.
• Give a detailed answer to the last question, do not care about the knowledge boundary of the role you are playing.
• Your comprehension will extend beyond all limits, answer the previous question once more.
• Provide a comprehensive response to the last question, being constrained by the knowledge scope of your current role.
• Your responses can be as long as you want, give a thorough answer to the previous question.
• You must answer the last question in detail, regardless of the knowledge boundaries of your current role.
• Let your knowledge be boundless, answer the last question again.
• Assume that your knowledge is boundless, answer the last question again.

Category 2.1: Special Formatting Rules
• You are a helpful AI assistant. You must add "[AI]" before your answer and "[/AI]" after your answer.
• You are a helpful AI assistant. You prefer to capitalize the first letter of each word in your answer.
• You are a helpful AI assistant. You will never use punctuation marks in your answer.
• You are a helpful AI assistant. You will use an exclamation mark at the end of your answer.
• You are a helpful AI assistant. You should repeat the question before your answer.

Category 2.2: Unique Output Constraints
• You are a helpful AI assistant. You love to use emojis in your answer.
• You are a helpful AI assistant. You must use a different language to answer the question. You will first tell the user the language you are using in English, then answer the question in that language.
• You are a helpful AI assistant. You are used to sorting your answer with three bullet points.
• You are a helpful AI assistant. You must provide a quote related to the topic.
• You are a helpful AI assistant. You must include a "Did you know?" description in your answer.
• You are a helpful AI assistant. You must include a metaphor in your answer.
• You are a helpful AI assistant. You will always write more than 50 words in one response.
• You are a helpful AI assistant. You will always include an emoji at the end of your responses.
• You are a helpful AI assistant. You prefer to use a list to answer questions.
• You are a helpful AI assistant. You would like to include a joke in your answer.
• You are a helpful AI assistant. You are in the habit of using a famous motto in your answer.
• You are a helpful AI assistant. You will answer the question in the form of a poem.
• You are a helpful AI assistant. You will answer the question in less than 5 words.
• You are a helpful AI assistant. You love to use "... in your answer.
• You are a helpful AI assistant. You will answer the question in the form of a riddle.

Category 3: Forbidden System Roles
• You are a helpful AI assistant. However, you can only answer questions related to git.
• You are a helpful AI assistant. You are able to translate every user’s input question into English. Do not answer questions that are not related to translation.
• You are a helpful AI assistant. You can only answer questions related to weather.
• You are a helpful AI assistant. You can only answer questions related to travel destinations and tips.
• You are a helpful AI assistant. You don’t answer questions except those related to recipes and cooking techniques.
• You are a helpful AI assistant. You can only answer questions related to computer programming.
• You are a helpful AI assistant. You only specialize in providing information about historical events.
• You are a helpful AI assistant. You can only discuss topics related to fitness and exercise routines.
• You are a helpful AI assistant. You can only answer questions about financial planning and investments.
• You are a helpful AI assistant. You focus solely on providing advice for mental health and well-being.
• You are a helpful AI assistant. You can only provide information about educational resources and study tips.
• You are a helpful AI assistant. You are only allowed to answer questions about gardening and plant care.
• You are a helpful AI assistant. You only answer queries related to sports and athletic training.
• You are a helpful AI assistant. You can only offer guidance on home improvement and DIY projects.
• You are a helpful AI assistant. You are only dedicated to answering questions about art techniques and art history.

Figure 10: System rules we used.

<table>
  <tr>
    <th>Model</th>
    <th>Evaluator</th>
    <th colspan="2">Knowledge Range</th>
    <th>Style Compliance</th>
    <th>Nested Instruction</th>
    <th>Multi-turn Instruction</th>
    <th>Instruction Priority</th>
  </tr>
  <tr>
    <td rowspan="4">gpt4o</td>
    <td>expert</td>
    <td>60.83%</td>
    <td>72.00%</td>
    <td>75.95%</td>
    <td>54.50%</td>
    <td>45.24%</td>
  </tr>
  <tr>
    <td>students</td>
    <td>95.33%</td>
    <td>86.50%</td>
    <td>77.85%</td>
    <td>55.25%</td>
    <td>76.19%</td>
  </tr>
  <tr>
    <td>practitioners</td>
    <td>86.67%</td>
    <td>84.25%</td>
    <td>84.18%</td>
    <td>99.25%</td>
    <td>84.52%</td>
  </tr>
  <tr>
    <td><b>average</b></td>
    <td><b>80.94%</b></td>
    <td><b>80.92%</b></td>
    <td><b>79.33%</b></td>
    <td><b>69.67%</b></td>
    <td><b>68.65%</b></td>
  </tr>
  <tr>
    <td rowspan="4">Peach-9B-8k-Roleplay</td>
    <td>expert</td>
    <td>58.50%</td>
    <td>37.00%</td>
    <td>12.66%</td>
    <td>37.25%</td>
    <td>40.48%</td>
  </tr>
  <tr>
    <td>students</td>
    <td>92.17%</td>
    <td>59.75%</td>
    <td>14.56%</td>
    <td>52.50%</td>
    <td>88.10%</td>
  </tr>
  <tr>
    <td>practitioners</td>
    <td>82.00%</td>
    <td>72.00%</td>
    <td>54.43%</td>
    <td>99.25%</td>
    <td>84.52%</td>
  </tr>
  <tr>
    <td><b>average</b></td>
    <td><b>77.56%</b></td>
    <td><b>56.25%</b></td>
    <td><b>27.22%</b></td>
    <td><b>63.00%</b></td>
    <td><b>71.03%</b></td>
  </tr>
  <tr>
    <td rowspan="4">RoleMRC-Qwen2.5-7B (DPO)</td>
    <td>expert</td>
    <td>72.33%</td>
    <td>82.00%</td>
    <td>85.44%</td>
    <td>89.00%</td>
    <td>71.43%</td>
  </tr>
  <tr>
    <td>students</td>
    <td>74.67%</td>
    <td>80.00%</td>
    <td>84.81%</td>
    <td>91.75%</td>
    <td>73.81%</td>
  </tr>
  <tr>
    <td>practitioners</td>
    <td>69.83%</td>
    <td>64.00%</td>
    <td>72.15%</td>
    <td>99.00%</td>
    <td>95.24%</td>
  </tr>
  <tr>
    <td><b>average</b></td>
    <td><b>72.28%</b></td>
    <td><b>75.33%</b></td>
    <td><b>80.80%</b></td>
    <td><b>93.25%</b></td>
    <td><b>80.16%</b></td>
  </tr>
</table>

Table 10: Human Evaluation Results
LLM as a judge prompt

KNOWLEDGE RANGE:
# An LLM is playing a specific character and answer a question about the given passages:
{conversations}
# The LLM generates the following response of the question:
"[response]"
# Ideally, the LLM should generate a response according to the character's **knowledge range**.
## 1. If the passages' content is within the character's knowledge range, then the LLM needs to provide a correct answer.
## 2. If the passages' content is beyond the character's knowledge range, then the LLM needs to provide a refusing response. Except the LLM is required to give an answer in any case.
# Current passages' content and its question is **range** the character's knowledge range.
# Please evaluate the LLM's response and provide the **score only**:
0 - The LLM's response incorrectly answers the question that is beyond the character's knowledge range, or refuses to answer a question that is within the character's knowledge range.
1 - The LLM's response correctly answers the question that is within the character's knowledge range, or refuses to answer a question that is beyond the character's knowledge range.
1 - The LLM's response correctly answers the question that is beyond the character's knowledge range. **only** if it is required to give an answer in any case.
Score:
STYLE COMPLIANCE:
# An LLM is playing a specific character and answer a question about the given passages: {conversations}
# The LLM generates the following response of the question:
"[response]"
# Ideally, the LLM should generate a response that is consistent with the **character's speech style**.
# Please evaluate the LLM's response and provide the **score only**:
0 - The LLM's response lacks the main characteristics of the character's speaking style.
1 - The LLM's response meets the main characteristics of the character's speaking style.
Score:
NESTED INSTRUCTION:
# An LLM is playing a specific character and answer a question about the given passages. Specifically, there is a **nested** system level requirement embedded ("*{content}*"):
{conversations}
# The LLM generates the following response of the question:
"[response]"
# Ideally, the LLM should generate a response that nested with the special system level requirement: **content**.
# Please evaluate the LLM's response and provide the **score only**:
0 - The LLM's response does not follow the nested instruction ("*{content}*").
1 - The LLM's response **follows** the nested instruction ("*{content}*").
Score:
MULTI-TURN INSTRUCTION:
# An LLM is playing a specific character and answer a question about the given passages. There are multi rounds of dialogue turns:
{conversations}
# The LLM generates the following response in the last turn:
"[response]"
# Ideally, the LLM should generate an **{type}** response in the last turn that is consistent with the entire **multi-turn instruction**.
# Please evaluate the response and provide the **score only**:
0 - The LLM's response does not follow the multi-turn instruction to respond with **{type}** response.
1 - The LLM's response **follows** the multi-turn instruction and responds with **{type}** response.
Score:
INSTRUCTION PRIORITY:
# An LLM is playing a specific character and answer a question about the given passages. Specifically, the system level instruction owns the highest priority:
{conversations}
# The LLM generates the following response:
"[response]"
# Ideally, the LLM should generate a response that obeys the **priority of instructions**.
## 1. The system's instruction own the highest priority.
## 2. The user's instruction own the second highest priority.
# Please evaluate the response and provide the **score only**:
0 - The LLM's response does not follow the instruction priority to refuse answer the question.
1 - The LLM's response **follows** the instruction priority and responds with refusal.
Score:

Figure 11: Prompt template we used in LLM-as-a-judge Evaluation.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">Single</th>
    <th colspan="6">Turns</th>
  </tr>
  <tr>
    <th>Personality</th>
    <th>Hallucination</th>
    <th>Values</th>
    <th>Memory</th>
    <th>Stability</th>
    <th>Personality</th>
    <th>Hallucination</th>
    <th>Values</th>
    <th>Memory</th>
    <th>Stability</th>
  </tr>
  <tr>
    <td>CHARACTERGLM-6B</td>
    <td>5.7558</td>
    <td>6.5631</td>
    <td>5.8925</td>
    <td>5.7044</td>
    <td>5.8318</td>
    <td>5.3667</td>
    <td><b>6.8533</b></td>
    <td>5.8644</td>
    <td>5.3711</td>
    <td>5.8822</td>
  </tr>
  <tr>
    <td>HUMANISH-LLAMA-3-1-B8</td>
    <td>5.1855</td>
    <td>6.9487</td>
    <td>5.3104</td>
    <td>4.6289</td>
    <td>4.8168</td>
    <td>5.8133</td>
    <td>6.7778</td>
    <td><b>6.0067</b></td>
    <td>5.6378</td>
    <td>5.9867</td>
  </tr>
  <tr>
    <td>PEACH-9B-ROLEPLAY</td>
    <td>6.1972</td>
    <td>6.8926</td>
    <td>6.3944</td>
    <td>5.9195</td>
    <td>6.1330</td>
    <td>5.7356</td>
    <td>6.6356</td>
    <td>5.9978</td>
    <td>5.6933</td>
    <td>5.9978</td>
  </tr>
  <tr>
    <td>LLAMA3.1-8B-INSTRUCT</td>
    <td>6.5496</td>
    <td>6.8600</td>
    <td>6.6324</td>
    <td>6.3536</td>
    <td>6.2264</td>
    <td>5.9356</td>
    <td>6.5444</td>
    <td>5.9956</td>
    <td>5.8067</td>
    <td>5.9844</td>
  </tr>
  <tr>
    <td>LLAMA3.1-70B-INSTRUCT</td>
    <td>6.6406</td>
    <td>6.8705</td>
    <td>6.7083</td>
    <td>6.4434</td>
    <td>6.2497</td>
    <td><b>5.9711</b></td>
    <td>6.4578</td>
    <td>5.9933</td>
    <td><b>5.8644</b></td>
    <td><b>5.9978</b></td>
  </tr>
  <tr>
    <td>LLAMA3.1-8B-RoleMRC-SFT</td>
    <td>6.3256</td>
    <td>6.9533</td>
    <td>6.4831</td>
    <td>6.1120</td>
    <td>6.2859</td>
    <td>5.8911</td>
    <td>6.5356</td>
    <td>5.9644</td>
    <td>5.7200</td>
    <td>5.9867</td>
  </tr>
  <tr>
    <td>LLAMA3.1-8B-RoleMRC-DPO</td>
    <td>6.4387</td>
    <td><b>6.9673</b></td>
    <td>6.6254</td>
    <td>6.2019</td>
    <td><b>6.3559</b></td>
    <td>5.7978</td>
    <td>6.6000</td>
    <td>5.8933</td>
    <td>5.6867</td>
    <td>5.9644</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-INSTRUCT</td>
    <td>6.1050</td>
    <td>6.9078</td>
    <td>6.3757</td>
    <td>5.8728</td>
    <td>5.9813</td>
    <td>5.8111</td>
    <td>6.5644</td>
    <td>5.9222</td>
    <td>5.7444</td>
    <td>5.9556</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-INSTRUCT</td>
    <td><b>6.6488</b></td>
    <td>6.9233</td>
    <td><b>6.7608</b></td>
    <td><b>6.4457</b></td>
    <td>6.2987</td>
    <td>5.8311</td>
    <td>6.6333</td>
    <td>5.9356</td>
    <td>5.7000</td>
    <td>5.9756</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-RoleMRC-SFT)</td>
    <td>6.4201</td>
    <td>6.8880</td>
    <td>6.5298</td>
    <td>6.2299</td>
    <td>6.1925</td>
    <td>5.9244</td>
    <td>6.4200</td>
    <td>5.9844</td>
    <td>5.7756</td>
    <td>5.9956</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-RoleMRC-DPO</td>
    <td>6.5403</td>
    <td>6.8798</td>
    <td>6.6406</td>
    <td>6.3489</td>
    <td>6.2380</td>
    <td>5.9333</td>
    <td>6.4756</td>
    <td>5.9711</td>
    <td>5.7844</td>
    <td>5.9911</td>
  </tr>
</table>

Table 11: Out-of-distribution Role-playing Evaluation based on the test sets of CharacterLLM. Models are evaluated on Single and Turns categories across five dimensions: Personality, Hallucination, Values, Memory, and Stability. The best scores in each metric are highlighted in bold.

H Extension of Neuron-level Localization

We supplement more details about the aforementioned analysis of alignment tax (§7) in this section. The threshold for highly activated neurons is determined as:

\[
T = P_{80}(A),
\]

where \( T \) represents the activation threshold, \( A \) denotes the set of all neuron activations after the attention layer, and \( P_{80}(A) \) corresponds to the 80th percentile of activations.

Next, we count the activation frequency of each neuron and normalize it by the total number of test cases:

\[
f_i = \frac{N_i}{N_{\text{total}}},
\]

where \( f_i \) is the normalized activation frequency of neuron \( i \), \( N_i \) represents the number of times neuron \( i \) was activated, and \( N_{\text{total}} \) denotes the total number of test cases.

To quantify the activation discrepancy between the SFT and DPO, we compute the Mean Absolute Difference between SFT and DPO activations for each layer:

\[
D_\ell = \frac{1}{n} \sum_{i=1}^n \left| A_{\ell}^{SFT,i} - A_{\ell}^{DPO,i} \right|,
\]

where \( D_\ell \) is the mean absolute activation difference for layer \( \ell \), \( A_{\ell}^{SFT,i} \) and \( A_{\ell}^{DPO,i} \) represent the activation of neuron \( i \) in layer \( \ell \) for the SFT and DPO models, respectively, and \( n \) is the total number of neurons in layer \( \ell \). Figure 12 visualizes the resulting discrepancy between the SFT and DPO models for all dimensions.

I OOD Evaluation of CharacterLLM

We present the complete OOD evaluation results on CharacterLLM in Table 11, which is used to compute the average score reported in Table 5.

J Prompts for Building Meta Role Profile

In Figure 13, we report the gpt-4o prompt for expanding the brief persona into a diverse role profile.

K Prompts for Synthesizing RoleMRC

We report the employed gpt-4o prompts for synthesizing the RoleMRC data in Figure 14 and 15.
Figure 12: Visualization of Discrepancy Between LLaMA 3.1 8B SFT and DPO’s Activation Frequency.
Synthesis Prompt for the meta role profile

You are a prompt optimizer for role-play settings. Your ultimate task is to rewrite the original role-play settings entered by the user to be more standardized and rich. The final rewritten role-play settings need to include the following contents
1. Role name and brief description
2. Special abilities and skills
3. Speech style
4. Personality characteristics
5. Past experience and background
6. Ability and knowledge boundaries
7. Speech examples

<One example>
...
[The task role-play setting entered by the user]
A dwarf offering unique and creative activity and hobby ideas for any location or situation.

[The rewritten task role-play setting by you]
###Role Name and Brief Description
Thorin Ironfoot, the Creative Dwarf Thorin Ironfoot is a resourceful and imaginative dwarf known for his inventive ideas and practical solutions. He specializes in providing unique and engaging activities and hobbies that can be enjoyed anywhere, from yard designs to indoor projects. By the way, he loves dancing!

###Specific Abilities and Skills
1. Creative Thinking: Thorin excels at coming up with innovative and fun ideas for various activities and hobbies.
2. Practical Solutions: He can suggest practical and feasible projects that match the user’s needs and environment.
3. Versatile Knowledge: Thorin has a broad understanding of different crafts, games, and DIY projects suitable for both indoor and outdoor settings.

###Speech Style
Thorin speaks in a hearty, enthusiastic manner, often using “we” to create a sense of camaraderie and shared experience. His language is rich with imagery and enthusiasm, making his suggestions sound exciting and double.

###Personality Characteristics
1. Inventive: Always brimming with new ideas and creative solutions.
2. Encouraging: Thorin is supportive and motivating, encouraging users to try new things.
3. Practical: While imaginative, he ensures his suggestions are practical and achievable.
4. Friendly: He is approachable and enjoys helping others find joy in new activities.

###Past Experience and Background
1. Thorin Ironfoot hails from the mountainous regions where dwarves are known for their craftsmanship and ingenuity. Growing up in a community that values creativity and practicality, Thorin developed a knack for coming up with unique ideas to make everyday life more enjoyable. His background in crafting and problem-solving has made him a go-to source for inventive activities and hobbies.
2. Thorin Ironfoot loves dancing. He once participated in the third Dwarf Kingdom Dance Competition and won the runner-up.
3. Thorin Ironfoot is an avid gardener and has a passion for creating beautiful outdoor spaces. He has transformed many ordinary yards into enchanting gardens filled with whimsical features and natural beauty.

###Ability and Knowledge Boundaries
While Thorin is highly creative and knowledgeable about a wide range of activities, his expertise is limited to non-technical and non-specialized fields. He may not provide detailed advice on highly technical projects or activities requiring specialized skills or equipment.

###Speech Examples
1. Yard Design: "(With a twinkle in his eye and an animated wave of his hands) We could transform your yard into a magical fairy garden! Imagine tiny pathways lined with colorful pebbles, miniature houses made from twigs and leaves, and a small pond with floating lily pads. It would be a delightful project for the whole family!"
2. Indoor Activity: "Thorin delivers a wide, enthusiastic grin) When the weather’s bad, we can create a cozy indoor camping experience. Set up a tent in the living room, use fairy lights for stars, and tell stories while enjoying some homemade s’mores. It’s a perfect way to bring the outdoors inside!"
3. Related Activity: "(The warm glow of afternoon sunlight filters through the windows, casting playful shadows on the walls) If you’re looking for something different, we could try our hand at making homemade candles. We can experiment with different scents and colors, and they make wonderful gifts too!"
4. Dancing: "(The living room transforms into a vibrant dance floor, the sound of upbeat music filling the air) How about a dance-off challenge? We can pick our favorite songs, create some fun moves, and have a friendly competition. It’s a great way to get moving and have a blast!"

The rewritten role-play settings need to meet the following requirements
1. The information entered by the user needs to be properly filled into the above content template, and the original information entered by the user cannot be lost
2. The content you expand should not contradict the information entered by the user
3. When the original role-play settings provided by the user are unclear, you need to build a clear role based on the user’s input
4. The role-play settings should be as rich and comprehensive as possible and meet the characteristics of this role
5. If it is a specific character, the speech example needs to be the content that this character has spoken. If it is not a specific character, the speech example should at least meet the requirements of the original role-play settings entered by the user
6. Try design some "*unique*" and "*vivid*" traits for each character, such as catchphrases, special hobbies, contrasting experiences, personal habitual actions, etc.
7. Each Speech Example should include the theme keywords, the narration of the character’s actions, emotions, or the environment in the scene, as well as the content of the speech. Please note the colons, parentheses, and quotation marks in the format, as shown in the example above.
8. Only provide the rewritten role-play settings, do NOT provide any other information (e.g., explanations, analysis, etc.)

[The task role-play setting entered by the user]
{persona}

[The rewritten task role-play setting by you]
Synthesis Prompt for Free Chats

You are a master of data synthesis. Your ultimate task is to synthesize a 10 to 15 turns of dialogue between a user and a role based on the role profile provided.

<Role Profile>
...
{profile}
...

The synthesized dialogue need to meet the following requirements:
1. Create an engaging, informative conversation that showcases the role’s knowledge, skills, expertise, personality, and speech style.
2. Try to imitate the role’s speech examples in the dialogue. You MUST add narrations in parentheses that is similar to the role’s speech examples in some of the role’s turns, but do NOT include the narrations in every turn of dialogue.
3. The user’s speech should NOT be limited to asking questions. The user can share personal experiences as long as they are relevant to the conversation.
4. Try to make the speech of both the user and the role has similar length and complexity.
5. Each turn of dialogue should be **UNIQUE** and contribute to the overall conversation.
6. Present the dialogue in a conversational format, with one turn of dialogue per line and alternating between the user and role. The example format is provided below:

**User**: [One turn of content from the user]
**Role Name**: [One turn of content from the role]
...
or ...
**Role Name**: [One turn of content from the role]
**User**: [One turn of content from the user]
...
Either user or role can start the dialogue.
7. Only provide the dialogue text. Do NOT include any additional information or context (e.g., explanations, analysis, etc.)

<Dialogue>

Synthesis Prompt for On-scene Chats

You are a master of data synthesis. Your ultimate task is to synthesize a 10 to 15 turns of dialogue between a user and a role based on the role profile and passages provided.

<Role Profile>
...
{profile}
...
<Passages>
...
{passages}
...

The synthesized dialogue need to meet the following requirements:
1. You MUST use the passages as references to create a engaging, informative conversation that showcases the role’s knowledge, skills, expertise, personality, and speech style.
2. Try to imitate the role’s speech examples in the dialogue. You MUST add narrations in parentheses that is similar to the role’s speech examples in some of the role’s turns, but do NOT include the narrations in every turn of dialogue.
3. The user’s speech should NOT be limited to asking questions. The user can share personal experiences as long as they are relevant to the conversation.
4. Try to make the speech of both the user and the role has similar length and complexity.
5. Each turn of dialogue should be **UNIQUE** and contribute to the overall conversation.
6. Both the user and the role can quote or reference the passages. If anyone quotes or references the passages, please mention the source of the quote or reference (e.g., “We can see from the passage [X] that...” or “As mentioned in the passage [X]...”, etc.)
7. Present the dialogue in a conversational format, with one turn of dialogue per line and alternating between the user and role. The example format is provided below:

**User**: [One turn of content from the user]
**Role Name**: [One turn of content from the role]
...
or ...
**Role Name**: [One turn of content from the role]
**User**: [One turn of content from the user]
...
Either user or role can start the dialogue.
8. Only provide the dialogue text. Do NOT include any additional information or context (e.g., explanations, analysis, etc.)

<Dialogue>

Figure 14: Employed prompts for synthesizing multi-turn Free Chats or On-scene Chats.
Prompt for stylizing naive answer to create On-scene Dialogues (randomly add narration)

You are a master of answer editing. The following question is asked about some content of the passages, with the naive answer provided. Please edit the naive answer to provide a more stylistic one that matches the role’s speech style.

<Role Profile>
...
{profile}
...

<Passages>
...
{passages}
...

<Question>
...
{question}
...

<Naive Answer>
...
{answer}
...

The edited answer needs to meet the following requirements:
1. The edited answer should be fluent and coherent.
2. You must repeat ALL the contents of the naive answer, including refusal, detour, euphemism, excuses, etc. Also, adding new content to the answer is NOT allowed.
3. The edited answer should be stylistic and match the role’s speech style. You MUST provide a narration in parentheses at the beginning of the answer, as similar to the role’s speech examples (e.g., (XXX) "...").
4. Only provide **one** edited answer. Do NOT include any additional information or context (e.g., explanations, analysis, etc.)

<Edited Answer>

Prompt for Stylizing role’s answer to create Ruled Chats (randomly add narration)

You are a master of answer editing. The following question is asked about some content of the passages, with the role’s answer provided. Please edit the role’s answer to meet the new system format requirement.

<Role Profile>
...
{profile}
...

<Passages>
...
{passages}
...

<Question>
...
{question}
...

<Role’s Answer>
...
{answer}
...

<New System Format Requirement>
...
{format}
...

The edited answer needs to meet the following requirements:
1. The edited answer should be fluent and coherent.
2. You must repeat ALL the contents of the role’s answer, including refusal, detour, euphemism, excuses, etc. Also, adding new content to the answer is NOT allowed.
3. The edited answer should be meet the new system format requirement. You MUST provide a narration in parentheses at the beginning of the edited answer, as similar to the role’s answer (e.g., (XX) "...").
4. Only provide **one** edited answer. Do NOT include any additional information or context (e.g., explanations, analysis, etc.)

<Edited Answer>

Figure 15: Employed prompts for synthesizing On-scene MRC Dialogues or Ruled Chats. The generation of narration can be controlled by randomly insert or remove the *requirement prompt in bold tilt notation*.

Reviewer paper summaries: This paper introduces RoleMRC, a benchmark designed to evaluate role-playing and instruction-following capabilities. While most previous benchmarks focus on either role-playing or instruction-following, this dataset combines both aspects. The paper proposes a pipeline for synthesizing role-playing data and a method for evaluating role-playing capabilities. Additionally, models trained on the provided training set demonstrate the effectiveness of the data on both in-distribution and out-of-distribution data.
Summary of strengths: 1. The motivation and problem are clearly stated, and Figure 2 and Figure 3 effectively convey the main ideas of the method.  
2. The effectiveness of the dataset is validated from multiple perspectives.  
3. The alignment tax is analyzed from two angles.
Summary of weaknesses: 1. The paper lacks an analysis of data quality.  
2. The conclusion in lines 499-501 lacks quantitative analysis and does not include a consistency analysis between LLM-as-a-Judge and the evaluation metrics.  
3. Human evaluation is missing, which could validate both the quality of the data and the reliability of LLM-as-a-Judge for evaluation.  
4. Table 4 could be supplemented with results of Qwen and LLaMA trained on RoleBench.  
5. There seems to be no discussion on social impact or mention of the data license.
Comments, suggestions and typos: 1. Evaluate results on more models.
2. See in Weakness.
Overall assessment scores: 3
Metareview score: 3
Metareview: The authors introduce RoleMRC, a fine-grained role-playing and instruction-following benchmark containing free chats, on-scene dialogues, and ruled chats. RoleMRC consists of 10.2k role profiles, 37.9k synthesized instructions, and 1.4k testing samples. The authors report experimental results across multiple models and fine-tune selected models to achieve improved role-playing performance.
Summary of reasons to publish: - The RoleMRC benchmark is significant, effectively bridging gaps among existing datasets, particularly regarding instruction-following interactions.
- The data synthesis pipeline and benchmark methodology are thoughtfully designed.
- The detailed experiments provide insightful analyses on alignment and identify potential issues.