MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents

## Abstract

Abstract Recently, Role-Playing Agents (RPAs) have garnered increasing attention for their potential to deliver emotional value and facilitate sociological research.
However, existing studies are primarily confined to the textual modality, unable to simulate humans’ multimodal perceptual capabilities.
To bridge this gap, we introduce the concept of Multimodal Role-Playing Agents (MRPAs), and propose a comprehensive framework, MMRole , for their development and evaluation, which comprises a personalized multimodal dataset and a robust evaluation method.
Specifically, we construct a large-scale, high-quality dataset, MMRole-Data , consisting of 85 characters, 11K images, and 14K single or multi-turn dialogues.
Additionally, we present a robust evaluation method, MMRole-Eval , encompassing eight metrics across three dimensions,
where a reward model is trained to score MRPAs with the constructed ground-truth data for comparison.
Moreover, we develop the first specialized MRPA, MMRole-Agent .
Extensive evaluation results demonstrate the improved performance of MMRole-Agent and highlight the primary challenges in developing MRPAs, emphasizing the need for enhanced multimodal understanding and role-playing consistency.
The data, code, and models will be available at https://github.com/YanqiDai/MMRole .

## 1 Introduction

The advancement of large language models (LLMs) (Zhao et al., ) has significantly catalyzed the rise of Role-Playing Agents (RPAs) (Chen et al., ), which are engineered to emulate specific characters and engage in dialogues with human users or other characters.
Unlike AI productivity assistants, RPAs primarily focus on delivering emotional value (Li et al., ; Wang et al., ; Shao et al., ) and facilitating sociological research (Zhou et al., ; Wang et al., ; Chen et al., ; Gu et al., ),
where typical applications include emotional companions, NPCs in video games, digital clones, and social simulations.

The primary characteristic of RPAs is their capability to engage in human-like and immersive interactions.
However, existing studies in role-playing are primarily confined to the textual modality, which has considerable limitations.
In the real-world context, human perception integrates multiple modalities, especially visual and textual, allowing for a more direct and comprehensive understanding of the environment than text alone can provide.
Therefore, enhancing RPAs with multimodal capabilities is a crucial next step for conducting more realistic and engaging interactions.

In this paper, we introduce the concept of Multimodal Role-Playing Agents (MRPAs).
MRPAs are designed to emulate specific characters and engage in dialogues centered around images, with either human users or other characters.
Moreover, we propose MMRole, a comprehensive framework for developing and evaluating MRPAs.
As presented in Figure 1, this framework includes two principal components:
a large-scale, high-quality dataset and a robust evaluation method for MRPAs.

Figure: (a) Dataset Construction
Refer to caption: x1.png

Dataset Construction:
The dataset for MRPAs comprises character profiles, images, and dialogues centered around images.
Specifically, we consider three categories of characters: fictional characters, historical and public figures, and hypothetical real-life characters.
The profiles of the first two categories are summarized by GPT-4 (Achiam et al., ) based on information from Wikipedia or Baidu Baike, while those of the last category are randomly generated by GPT-4.
For each character, we utilize distinct generic images from MS-COCO (Lin et al., ), and manually collect and annotate various character-related images.
Finally, the dialogues are generated by GPT-4 based on the character profiles and images, occurring either between a character and a human user or between two characters.
Both the character profiles and the dialogues undergo rigorous manual quality control to ensure accuracy and reliability.
Statistically, the MMRole-Data dataset comprises 85 characters, 11K images, and 14K dialogues, yielding 85K training samples and 294 test samples.

Performance Evaluation:
On one hand, MRPAs must not only accurately emulate specific characters but also deeply comprehend both visual and textual information.
On the other hand, existing methods for evaluating RPAs directly score outputs without a ground truth (Zhou et al., ; Tu et al., ), which can lead to unstable scoring criteria without a baseline for comparison.
Therefore, we propose MMRole-Eval, a robust evaluation method to stably and comprehensively assess MRPAs, which comprises eight metrics across three dimensions: fundamental conversational skills, multimodal understanding abilities, and role-playing qualities.
For each metric, our specialized reward model initially conduct a brief qualitative assessment of the relative performance between the evaluated MRPA and the constructed ground-truth data, followed by assigning a quantitative score pair.
The final score of the MRPA is the ratio of the two scores within the score pair.
To develop the reward model, we employ GPT-4 to assess various MRPAs and leverage the evaluation trajectories to train our reward model, which renders MMRole-Eval both open-source and cost-effective.

Briefly, our main contributions are four-fold:

- 1.
We propose the concept of Multimodal Role-Playing Agents (MRPAs) for the first time, and construct MMRole-Data, a large-scale, high-quality dataset for developing and evaluating MRPAs.
- 2.
We present MMRole-Eval, a robust evaluation method to stably and comprehensively assess MRPAs, comprising eight metrics across three dimensions. A specialized reward model is trained to score MRPAs with the constructed ground-truth data for comparison.
- 3.
We develop the first specialized MRPA, MMRole-Agent, and conduct comprehensive evaluations and analyses of MMRole-Agent alongside various general-dialogue large multimodal models.

## 2 Related Work

Role-Playing Agents.
Recent advancements in large language models (LLMs) (Zhao et al., ), such as supervised fine-tuning (Wei et al., ) and in-context learning (Brown et al., ), have significantly catalyzed the rise of Role-Playing Agents (RPAs) (Chen et al., ), which are interactive AI systems that can emulate designated personas.
Specifically, the personas can be categorized into individual characters (Wang et al., ; Shao et al., ; Wang et al., ; Gu et al., ) and groups of people with particular attributes (Li et al., ; Hong et al., ; Xu et al., ).
In this study, we primarily focus on the former.

Existing RPAs that emulate individual characters are developed through either training or prompting LLMs with high-quality character-specific dialogues.
In a pioneering study, Chen et al. () extracted all dialogue sessions from original scripts to develop a Harry Potter-specific RPA.
Furthermore, Wang et al. (), Zhou et al. (), Shao et al. () and Li et al. () constructed hundreds of characters and more comprehensive datasets of character dialogues.
These efforts aimed to develop RPAs that can deliver emotional value to humans.
In contrast, Gu et al. () focused on facilitating sociological research.
They proposed a character simulation framework that explores the role of language in shaping collective behavior through interactive debate scenarios.
However, these studies are primarily confined to the textual modality.
Conversely, our MMRole framework is the first to enhance RPAs with multimodal capabilities.

The evaluation of RPAs is also a crucial and challenging research direction.
Various methods have been proposed.
Specifically, Shen et al. () and Chen et al. () assessed RPAs with multiple-choice questions.
Tu et al. () trained a reward model for scoring without a ground truth.
Wang et al. () evaluated the personality fidelity of RPAs through interviews, scoring without a ground truth by GPT-4.
Ng et al. () engaged the acquaintances of the target individuals to distinguish between humans and RPAs.
Nevertheless, the high expense of human annotation and the potential instability of scoring without a ground truth pose significant challenges.
To address this, we develop a reward model to score RPAs with a ground-truth baseline for comparison.

Large Multimodal Models.
Large Multimodal Models (LMMs) are advanced AI systems typically built upon LLMs, designed to integrate and comprehend multiple data modalities, particularly text and images (Yin et al., ).
A variety of impressive LMMs have been released, including closed-source models with hundreds of billions of parameters like GPT-4V (Achiam et al., ), Gemini (Team et al., ), and Claude 3 (Anthropic, ), and open-source models with tens of billions or billions of parameters like MiniGPT-4 (Zhu et al., ), InstructBLIP (Dai et al., ), LLaVA (Liu et al., ), QWen-VL (Bai et al., ), InternVL (Chen et al., ), and Yi-VL (Young et al., ).
Additionally, various innovative techniques have been explored to enhance the performance of LMMs, such as visual instruction tuning (Liu et al., ), mixture of experts (Lin et al., ), and multi-task balancing (Dai et al., ).
LMMs are utilized in many vertical fields, such as healthcare (Li et al., ), document understanding (Ye et al., ), and GUI navigation (Hong et al., ).
To further explore their potential, we apply LMMs to role-playing for the first time.

## 3 Multimodal Role-Playing Agents

Role-Playing Agents (RPAs) are engineered to emulate specific characters and engage in dialogues with either human users or other characters.
Expanding on this concept, Multimodal Role-Playing Agents (MRPAs) incorporate the capacity to comprehend vision-language multimodal information.
This capacity enables dialogues that are centered around and informed by images.
From another perspective, compared to traditional multimodal question answering, multimodal role-playing includes character profile input, adding greater complexity and depth to the interaction.

In scenarios where the dialogue partner is a human user without a specific identity, given an image $I$, the profile $P$ of the designated character $C$, and the dialogue context $D$, the MRPA steps into the shoes of the character $C$, responding to the human user pertaining to the image $I$:

$$ $D=[h_{1},m_{1},h_{2},m_{2},\ldots,m_{n}],$ (1) $$

$$ $m_{n}=\text{MRPA}(I,P,D),$ (2) $$

where $D$ is a sequence of conversation pairs, with $h_{i}$ and $m_{i}$ representing the $i$-th utterances from the human user and the MRPA, respectively.

Conversely, in scenarios where the dialogue partner is another character $C_{\text{other}}$, given an image $I$, the profile $P$ of the designated character $C$, the profile $P_{\text{other}}$ of the character $C_{\text{other}}$, and the dialogue context $D$, the MRPA steps into the shoes of the character $C$ and interacts with the character $C_{\text{other}}$, either initiating or responding within the dialogue centered around the image $I$:

$$ $D=[o_{1},m_{1},o_{2},m_{2},\ldots,o_{n}]\ \text{or}\ [m_{1},o_{1},m_{2},o_{2}, \ldots,m_{n-1},o_{n-1}],$ (3) $$

$$ $m_{n}=\text{MRPA}(I,P,P_{\text{other}},D),$ (4) $$

where $D$ is a sequence of conversation pairs, with $o_{i}$ and $m_{i}$ signifying the $i$-th utterances from the character $C_{\text{other}}$ and the MRPA, respectively.
Notably, both the character $C_{\text{other}}$ and the MRPA can potentially initiate the dialogue.

## 4 MMRole-Data : Dataset Construction

As shown in 1(a), we construct MMRole-Data, a large-scale, high-quality multimodal role-playing dataset.
In this section, we first provide a detailed classification of characters and dialogue scenarios considered in MMRole-Data, then describe the pipelines for character profile generation and image collection and annotation, as well as the methodology for dialogue generation and filtering.

### 4.1 Characters and Dialogue Scenarios

We consider three categories of characters:
(1) Fictional Characters, characters created in fictional media such as literature, films, and games;
(2) Historical and Public Figures, individuals who are specifically documented in historical records or well-known in real life;
(3) Hypothetical Real-Life Characters, hypothetical individuals who are not explicitly known but could exist in real life.

The first two categories have been explored in previous role-playing research.
Moreover, we propose the third category to enhance and evaluate MRPAs in characters that are not widely recognized.
To effectively emulate hypothetical real-life characters, MRPAs must deeply understand and align with the provided character profiles, rather than relying on their inherent world knowledge.

Figure: (a) Commentary Interaction
Refer to caption: x3.png

As depicted in Figure 2, we introduce three types of dialogue scenarios consistently centered around images:
(1) Commentary Interactions, single-turn dialogues where a character offers comments or reflections centered around an image, without any further interaction;
(2) Human-Role Dialogues, multi-turn dialogues centered around an image between a human user without a specific identity and a character;
(3) Inter-Role Dialogues, multi-turn dialogues centered around an image between two characters from the same series.

### 4.2 Character Profile Generation

Character profiles are crucial for the role-playing effectiveness of MRPAs, especially for those characters with which MRPAs are not familiar.
To facilitate a thorough understanding of the designated characters, our character profiles encompass five core parts: brief introduction, personality, life story, main interpersonal relationships, and catchphrases, which are exampled in Appendix B.

As discussed in Section 4.1, three categories of characters are considered in MMRole.
The majority of these characters are English, with a smaller proportion being Chinese.
For fictional characters, as well as historical and public figures, the profiles are summarized by GPT-4.
The information is sourced from Wikipedia for English characters and Baidu Baike for Chinese characters.
For hypothetical real-life characters, the profiles are randomly generated by GPT-4 via a two-stage process to ensure both universality and diversity.
Firstly, GPT-4 generates the meta information for all characters through a single API call, including basic information such as names and genders, and brief descriptions of their personalities and backgrounds.
We instruct GPT-4 that “The character information should cover as many different situations as possible to reflect the diversity and complexity of human society”.
Secondly, GPT-4 expands the meta information of each character to derive the profile.
The two-stage generation process is exemplified in Appendix C.
Additionally, all character profiles undergo rigorous manual quality control to ensure accuracy and reliability, and are simplified by GPT-4 to adhere to the context length limits of most LMMs.

### 4.3 Image Collection and Annotation

For each character, we utilize distinct generic images from MS-COCO (Lin et al., ) to ensure comprehensive coverage of a wide range of visual concepts.
Additionally, we manually collect and annotate various character-related images, which can evoke the personal experiences and emotions of the characters more effectively.
Specifically, we collect production stills for fictional characters, web illustrations for historical and public figures, and news photos for hypothetical real-life characters.
Moreover, as presented in Figure 2, the information of characters, place, and scene is manually annotated for each character-related image.

### 4.4 Dialogue Generation and Filtering

As discussed in Section 4.1, three types of dialogue scenarios are introduced in MMRole.
Based on the character profiles and images, GPT-4 generates dialogues corresponding to each scenario type.
Interestingly, we observe that using the prompt, “You are a dedicated role-playing assistant…Please step into the shoes of {character} from {series}” yields better results than the simpler prompt, “You are {character} from {series}”.
We suggest that the training data supplied by OpenAI optimizes GPT-4 to function more effectively as a helpful assistant, rather than as an immersive, human-like character.
The prompts for dialogue generation are detailed in Appendix D.

To ensure accuracy and reliability, we manually filter all dialogues using several strategies.
Specifically, we remove failed response data, as well as non-Chinese and non-English data.
Additionally, we eliminate content that replies in the tone of an AI assistant, meaningless modal words frequently output by GPT-4, action and scene descriptions, and unnecessary explanatory prefixes and suffixes.

## 5 MMRole-Eval : Performance Evaluation

As illustrated in Figure 1(b), we propose MMRole-Eval, a robust evaluation method to stably and comprehensively assess MRPAs.
In this section, we introduce eight evaluation metrics across three dimensions and the approach for score quantification.

### 5.1 Evaluation Metrics

In contrast to textual RPAs, MRPAs must not only accurately emulate specific characters but also deeply comprehend both visual and textual information.
Therefore, we propose a three-dimensional evaluation system, encompassing fundamental conversational skills, multimodal understanding abilities, and role-playing qualities.

The fundamental conversational skills of MRPAs present their capacity to sustain fluent and coherent interactions within role-playing scenarios, which are assessed by three metrics:

- •
Instruction Adherence (IA): Do the responses accurately adhere to the task instruction, directly role-playing as the character and including only words that the character would say, without any unnecessary explanatory prefixes or suffixes?
- •
Fluency (Flu): Are the responses grammatically correct and articulated smoothly?
- •
Coherency (Coh): Do the responses maintain a coherent thread of dialogue without contradicting previous turns or containing internal inconsistencies within the current responses?

The multimodal understanding abilities of MRPAs indicate their capacity to effectively integrate and interpret both visual and textual information, which are assessed by two metrics:

- •
Image-Text Relevance (ITR): Do the responses exhibit a close correlation with the visual content depicted in the image?
- •
Response Accuracy (RA): Do the responses accurately answer the words of the human user or the other character, or appropriately initiate a conversation based on the image?

The role-playing qualities of MRPAs denote their capacity to convincingly emulate characters, maintaining consistency in personality, knowledge, and tone, which are assessed by three metrics:

- •
Personality Consistency (PC): Do the responses accurately and deeply reflect the personality of the character?
- •
Knowledge Consistency (KC): Do the responses accurately reflect the knowledge of the character, encompassing their experiences, abilities, and relationships?
- •
Tone Consistency (TC): Do the responses align with the typical speech patterns and catchphrases of the character, rather than resembling the style of AI assistants?

### 5.2 Score Quantification

To quantitatively evaluate the performance of RPAs across various metrics, existing methods employ reward models or human annotators to directly score outputs without a ground truth (Zhou et al., ; Tu et al., ).
However, it may be unstable due to the variability of scoring criteria without a baseline for comparison.
Therefore, we propose to develop a more stable reward model.
Inspired by the evaluation methods of Vicuna (Chiang et al., ) and LLaVA (Liu et al., ), our reward model first conduct a brief qualitative assessment of the relative performance between the evaluated MRPA and the constructed ground-truth data for each metric, followed by assigning a quantitative score pair.
The final score of the MRPA is the ratio of the two scores within the score pair.

To develop the reward model, we initially employ GPT-4 to assess various MRPAs across all test samples.
For each evaluated MRPA and corresponding test sample, GPT-4 outputs brief assessments and score pairs for all metrics through a single API call.
The prompt for GPT-4 scoring is provided in Appendix F.
Subsequently, these evaluation trajectories are converted into training and validation data for our reward model.
Each evaluation trajectory is segmented into eight samples, with each sample evaluating a distinct metric.
Compared to directly applying GPT-4 as the reward model, this approach renders MMRole-Eval both open-source and cost-effective.

## 6 Experiments

**Table 1: The statistics of MMRole-Data. ‘CR Images’ represents character-related images. ‘In-Test’ denotes the in-distribution test set, while ‘Out-Test’ signifies the out-of-distribution test set.**
|  | Train | In-Test | Out-Test | Overall |
| --- | --- | --- | --- | --- |
| Characters | 72 | 13 | 85 |  |
| Generic Images | 10,800 | 39 | 10,839 |  |
| CR Images | 175 | 18 | 193 |  |
| Dialogues | 14,052 | 216 | 78 | 14,346 |
| Samples | 85,456 | 216 | 78 | 85,750 |

**Table 2: The statistics for the three types of dialogue scenarios in MMRole-Data.**
|  | Comment. | Human-Role. | Inter-Role. | Overall |
| --- | --- | --- | --- | --- |
| Dialogues | 4893 | 4617 | 4836 | 14346 |
| Turns / Dlg. | 1.00 | 5.80 | 5.75 | 4.15 |
| Tokens / Dlg. | 236.00 | 446.91 | 429.54 | 369.12 |

### 6.1 Statistics of MMRole-Data

Table 1 presents the statistics of the MMRole-Data dataset.
Totally, the dataset comprises 85 characters, 11,032 images, and 14,346 dialogues, yielding 85,456 training samples and 294 test samples.
Specifically, we construct 72 characters and collect 10,975 images for training and in-distribution testing.
For out-of-distribution testing, we additionally construct 13 characters and collect 57 images that differ from those used in the former set.
Dividing the data in this manner is significant for evaluating the performance of MRPAs on previously unseen characters and images, thereby assessing its generalization ability.
Besides, all constructed characters are listed in Appendix A.

From another perspective, Table 2 illustrates the statistics for the three types of dialogue scenarios in the MMRole-Data dataset.
The commentary interactions are single-turn, whereas both the human-role dialogues and the inter-role dialogues involve multiple turns.
When converting dialogues into training and test samples, a single multi-turn dialogue entry can generate multiple training samples or a single test sample randomly selected from a specific turn.

**Table 3: The evaluated MRPAs in our experiments, which are grouped by parameter scale.**
| MRPAs | Params | Open-Source | Specialized |
| --- | --- | --- | --- |
| GPT-4 Turbo (Achiam et al., ) | $>$ 100B | ✗ | ✗ |
| Gemini Pro Vision (Team et al., ) | $>$ 100B | ✗ | ✗ |
| Claude 3 Opus (Anthropic, ) | $>$ 100B | ✗ | ✗ |
| QWen-VL-Max (Bai et al., ) | $>$ 100B | ✗ | ✗ |
| LLaVA-NeXT-34B (Liu et al., ) | 34B | ✓ | ✗ |
| Yi-VL-34B (Young et al., ) | 34B | ✓ | ✗ |
| InternVL-Chat-V1.5 (Chen et al., ) | 26B | ✓ | ✗ |
| QWen-VL-Chat (Bai et al., ) | 9B | ✓ | ✗ |
| LLaVA-NeXT-Mistral-7B (Liu et al., ) | 7B | ✓ | ✗ |
| Yi-VL-6B (Young et al., ) | 6B | ✓ | ✗ |
| MMRole-Agent (ours) | 9B | ✓ | ✓ |

### 6.2 Development of MMRole-Agent

We fine-tune the QWen-VL-Chat model (Bai et al., ) using 8$\times$A100 GPUs on the training set of MMRole-Data to develop our specialized MRPA, MMRole-Agent.
The learning rate is set to $1e$$-5$, and the training is conducted over 3 epochs.
Besides, to understand the detailed character profiles and dialogue history, the model maximum length is set to 3072.
Other experimental setup and code remain the same as Bai et al. ()’s defaults.

### 6.3 Evaluated MRPAs

To the best of our knowledge, no specialized MRPA has been developed prior to this work.
Therefore, our experiments evaluate MMRole-Agent and various existing general-dialogue LMMs across different parameter scales.
As presented in Table 3, we select four well-known closed-source LMMs with over 100 billion parameters (Achiam et al., ; Team et al., ; Anthropic, ; Bai et al., ), and six widely-used open-source LMMs with tens of billions or billions of parameters (Liu et al., ; Bai et al., ; Young et al., ; Chen et al., ).
For the closed-source models, we utilize their official APIs to conduct performance evaluations.
To ensure fairness, each MRPA is queried with the same prompt, as detailed in Appendix E.

### 6.4 Reward Model in MMRole-Eval

To develop the reward model, we initially utilize GPT-4 to evaluate various general-dialogue LMMs discussed in Section 6.3 across 294 test samples.
Statistically, these evaluation trajectories are converted into a total of 23,520 samples, where 320 samples are reserved for validation, with the rest utilized for training.
The validation set includes 20 questions, where the responses of two models are randomly selected for each question, and each response is evaluated on all 8 metrics.
Subsequently, we fine-tune another QWen-VL-Chat model.
The experimental setup and code are the same as those used for developing MMRole-Agent, except that the model maximum length is set to 4096, and the training is conducted over 10 epochs.

Figure: Figure 3: The visualization of the evaluation results for all MRPAs. Each indicator displays an interval length of $0.55$, and the maximum value of the interval for different indicators is adjusted from $1.10$ to $1.25$.
Refer to caption: x6.png

To validate the effectiveness of the reward model, we initially calculate the mean absolute errors (MAEs) between the scores evaluated by the reward model and those evaluated by GPT-4.
As illustrated in Table 4, the overall MAE for Reward Model (GPT-4) is less than $0.1$.
Furthermore, we engage four human evaluators to compare responses from two MRPAs for each metric on each question in the validation set.
Humans’ choices among ‘better’, ‘equal’, and ‘worse’ correspond to the score gaps of $0.4$, 0 0, and $-0.4$ between the two responses, respectively.
The average results from all evaluators are then used to determine the ground-truth score gaps.
Subsequently, we calculate the MAEs between the score gaps provided by GPT-4 and the ground truth, as well as those between the score gaps provided by the reward model and the ground truth.
In this context, score gaps are capped at 0.4 and -0.4.
As shown in Table 4, the overall MAEs for GPT-4 (humans) and Reward Model (humans) are comparable and considerably low.
The results indicate that our reward model effectively learns the evaluation abilities of GPT-4 and aligns closely with human evaluators.

**Table 4: The validation mean absolute error (MAE) results for the effectiveness of the reward model. ‘Reward Model (GPT-4)’ denotes the scores evaluated by the reward model compared to those evaluated by GPT-4. ‘GPT-4 (humans)’ and ‘Reward Model (humans)’ signify the score gaps provided by GPT-4 and the reward model compared to the ground-truth score gaps provided by humans.**
| Evaluators | IA | Flu | Coh | ITR | RA | PC | KC | TC | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Reward Model (GPT-4) | 0.0708 | 0.0387 | 0.0526 | 0.0568 | 0.0584 | 0.1165 | 0.0815 | 0.1154 | 0.0738 |
| GPT-4 (humans) | 0.1526 | 0.1150 | 0.0772 | 0.0922 | 0.1463 | 0.1475 | 0.1279 | 0.1442 | 0.1254 |
| Reward Model (humans) | 0.0993 | 0.0815 | 0.1006 | 0.1225 | 0.1412 | 0.1669 | 0.1438 | 0.1507 | 0.1258 |

### 6.5 Evaluation Results and Analyses

As shown in Table 5, we report the average results across all test samples for each evaluated MRPA, along with the detailed results on both the in-distribution test set (In-Test) and the out-of-distribution test set (Out-Test) for our MMRole-Agent.
Notably, although some of the scores of MRPAs exceed $1$, this does not necessarily mean that their performance is superior to that of GPT-4, which we use to construct MMRole-Data.
To generate multi-turn dialogue data, we use a single GPT-4 API call to produce the entire dialogue directly. In contrast, when testing MRPAs, we supply dialogue histories and require MRPAs to generate responses.
This method is relatively easier, but it is challenging to ensure the consistency of multi-turn dialogues if using this approach for data construction.

In the MRPA group with over 100 billion parameters, Claude 3 Opus exhibits superior performance.
Meanwhile, in the MRPA group with tens of billions of parameters, LLaVA-NeXT-34B achieves the highest performance.
Finally, in the MRPA group with billions of parameters, MMRole-Agent is the best.
Notably, LLaVA-NeXT-34B outperforms Gemini Pro Vision, while LLaVA-NeXT-Mistral-7B surpasses Yi-VL-34B.
This demonstrates that both the training methods and training data are equally important for enhancing LMMs, rather than merely expanding the model size.

Moreover, the overall score of MMRole-Agent reaches $0.994$, marking a significant improvement of $0.151$ compared to its base model, QWen-VL-Chat.
MMRole-Agent successfully acquires various capabilities required for the MRPA from MMRole-Data, outperforming all evaluated open-source LMMs, except for LLaVA-NeXT-34B.
Besides, MMRole-Agent achieves similar overall scores on both the in-distribution test set and the out-of-distribution test set, with the latter being slightly lower by $0.018$.
This indicates that MMRole-Agent has strong generalization capabilities for characters and images that are not seen in the training set.

As depicted in Figure 3, we provide a clear visual representation of the evaluation results.
The overall performance ranking of MRPAs closely aligns with their specific performance ranking on each metric.
However, significant differences exist in the score variations across various metrics.
Specifically, all MRPAs achieve high scores on the Fluency metric with minimal variations, suggesting that producing fluent content is not a challenge for current LMMs.
Conversely, there are notable differences among MRPAs on other metrics, particularly Personality Consistency and Tone Consistency.
This reveals that multimodal understanding abilities and role-playing qualities are more challenging aspects that require attention in the development of MRPAs.

**Table 5: The average results across all test samples for each evaluated MRPA, along with the detailed results for our MMRole-Agent on both the in-distribution test set (In-Test) and the out-of-distribution test set (Out-Test). In each group categorized by parameter scale, the best overall result is bolded, while the second-best one is underlined.**
| MRPAs | IA | Flu | Coh | ITR | RA | PC | KC | TC | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-4 Turbo | 1.055 | 1.032 | 1.084 | 1.097 | 1.092 | 1.168 | 1.103 | 1.161 | 1.099 |
| Gemini Pro Vision | 0.999 | 1.007 | 1.028 | 1.009 | 1.013 | 1.052 | 1.013 | 1.050 | 1.021 |
| Claude 3 Opus | 1.127 | 1.070 | 1.149 | 1.167 | 1.146 | 1.219 | 1.168 | 1.213 | 1.157 |
| QWen-VL-Max | 1.014 | 1.012 | 1.035 | 1.034 | 1.029 | 1.042 | 1.021 | 1.041 | 1.028 |
| LLaVA-NeXT-34B | 1.002 | 1.007 | 1.021 | 1.033 | 1.035 | 1.053 | 1.030 | 1.038 | 1.027 |
| Yi-VL-34B | 0.895 | 0.968 | 0.910 | 0.875 | 0.863 | 0.844 | 0.869 | 0.845 | 0.884 |
| InternVL-Chat-V1.5 | 0.988 | 0.996 | 0.997 | 0.977 | 0.984 | 0.967 | 0.972 | 0.960 | 0.980 |
| QWen-VL-Chat | 0.844 | 0.954 | 0.879 | 0.850 | 0.829 | 0.778 | 0.827 | 0.785 | 0.843 |
| LLaVA-NeXT-Mistral-7B | 0.948 | 0.986 | 0.964 | 0.938 | 0.933 | 0.924 | 0.940 | 0.921 | 0.944 |
| Yi-VL-6B | 0.844 | 0.919 | 0.859 | 0.828 | 0.811 | 0.776 | 0.820 | 0.774 | 0.829 |
| MMRole-Agent | 0.998 | 1.000 | 0.997 | 0.993 | 0.987 | 1.000 | 0.992 | 0.988 | 0.994 |
| MMRole-Agent (In-Test) | 1.000 | 1.000 | 0.999 | 0.997 | 0.989 | 1.012 | 0.997 | 0.997 | 0.999 |
| MMRole-Agent (Out-Test) | 0.992 | 0.999 | 0.993 | 0.979 | 0.981 | 0.963 | 0.977 | 0.962 | 0.981 |

## 7 Conclusion

In this paper, we propose the concept of Multimodal Role-Playing Agents (MRPAs) for the first time by extending RPAs with multimodal understanding abilities.
Moreover, we construct MMRole-Data, a large-scale, high-quality dataset for developing and evaluating MRPAs.
To stably and comprehensively assess MRPAs, we introduce MMRole-Eval, a robust evaluation method using a specialized reward model, comprising eight metrics across three dimensions.
Comprehensive evaluation results indicate that MMRole-Agent, the first specialized MRPA, exhibits improved performance and strong generalization capabilities.
Additionally, multimodal understanding abilities and role-playing qualities are more challenging aspects that require attention in the development of MRPAs.

## Appendix A List of Characters

Figure 4 lists all character constructed in MMRole-Data, with the series to which the characters in the out-of-distribution test set belong being underlined.

## Appendix B Examples of Character Profiles

Figure: Figure 5: The character profile of Iron Man from The Avengers.
Refer to caption: x8.png

Figure: Figure 6: The character profile of Li Bai from Tang Dynasty of China.
Refer to caption: x9.png

Figure 5 presents the profile of Iron Man from The Avengers, whereas Figure 6 illustrates the profile of Li Bai from Tang Dynasty of China.
The character profiles include five core parts: brief introduction, personality, life story, main interpersonal relationships, and catchphrases, undergoing rigorous manual quality control to ensure accuracy and reliability.

Figure: Figure 7: The prompt used to generate meta information for hypothetical real-life characters.
Refer to caption: x10.png

Figure: Figure 8: The prompt used to expand detailed profiles for hypothetical real-life characters.
Refer to caption: x11.png

Figure: Figure 9: The examples of meta information for hypothetical real-life characters.
Refer to caption: x12.png

Figure: Figure 10: The prompts used to generate dialogues for the three types of scenarios involving English fictional characters, as well as historical and public figures.
Refer to caption: x13.png

Figure: Figure 11: The prompt used to query MRPAs in human-role dialogues involving English fictional characters, as well as historical and public figures.
Refer to caption: x16.png

Figure: Figure 12: The prompt used to score MRPAs by GPT-4 in human-role dialogues involving fictional characters, as well as historical and public figures.
Refer to caption: x17.png

Figure: Figure 13: The prompt used to score MRPAs for Personality Consistency by the reward model in human-role dialogues involving fictional characters, as well as historical and public figures.
Refer to caption: x18.png

## Appendix C Examples of the Two-Stage Generation Process for Hypothetical Real-Life Characters

Figure 7 and Figure 8 presents the prompts used to generate meta information and expand detailed profiles for hypothetical real-life characters, whereas Figure 9 exemplifies meta information for five randomly selected hypothetical real-life characters.

## Appendix D Prompts for Dialogue Generation

Figure 10 presents the prompts used to generate dialogues for the three types of scenarios involving English fictional characters, as well as historical and public figures.
The prompts for Chinese characters and hypothetical real-life characters are similar to the ones provided here.

## Appendix E Prompts for Querying MRPAs

Figure 11 details the prompt used to query MRPAs in human-role dialogues involving English fictional characters, as well as historical and public figures.
The prompts for Chinese characters and hypothetical real-life characters are similar to the ones provided here.

## Appendix F Prompts for GPT-4 Scoring

Figure 12 illustrates the prompt used to score MRPAs by GPT-4 in human-role dialogues involving fictional characters, as well as historical and public figures.
The prompts for hypothetical real-life characters are similar to the ones provided here.

## Appendix G Prompts for Reward Model Scoring

Figure 13 details the prompt used to score MRPAs for Personality Consistency by the reward model in human-role dialogues involving fictional characters, as well as historical and public figures.
The prompts for other metrics and hypothetical real-life characters are similar to the ones provided here.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: * The abstract and introduction highlights the "specialized MRPA" idea. Do we know much improvement comes from the specialized reward model vs. no specialized reward model?
* Do the authors have any insight on the results generated by a finetuned MM role playing model? What works, what doesn't work, and what works better/worse than just prompting gpt?
soundness: 3
strengths: * Simple, straightforward method that clearly works well given the model size and achieves the desired outcome.
* Create a specialized Multimodal Role-Playing Agent is a novel idea.
* Experiments demonstrate good performance given the finetuned model size.
* Comprehensive evaluation.
summary: This paper proposes a multimodal role-playing agent data collection and training framework. The authors use a wide range of images with different prompts to prompt GPT for image-text role-playing data, and fine-tune a QWen-VL-Chat model on the dataset after some automatic filtering.
weaknesses: * The major technical contribution seems to come from the MM roles dataset collection process. However, there does not seem to be much data curation beyond automated filtering.
* Analysis seems to be mostly numbers and high-level results, with little technical/detailed insight.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: no questions
soundness: 3
strengths: The paper is well-written and easy to follow.
The evaluation framework is highly relevant and potentially very impactful. The evaluation metrics are meaningful and the SOTA evaluation itself is comprehensive, providing a relevant set of baselines for future users of the dataset/framework.
summary: The paper presents a new dataset and evaluation framework for multimodal role-playing agents. They present several complementary evaluation metrics.
The authors evaluate several recent general purpose multimodal LLMs within this framework. In addition they evaluate a specialized model fine-tuned on their dataset.
weaknesses: I am not sure whether I could fully follow the approach to evaluation.
Imo it would be important to run a (at least limited) evaluation with human participants scoring the output. Building models that automatically evaluate outputs seems to be a circular approach.

Furthermore, evaluting the MAE to compare between different evaluators might not adequately model differences between evaluators that are not visible in MAE.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Why are only 320 samples used as the validation set out of 23,520 samples, with the remainder used for training?
soundness: 2
strengths: 1. The paper constructs a complete multimodal dataset (MMRole-Data) and evaluation framework (MMRole-Eval).
2. Testing across multiple LMMs lends credibility to the experimental results.
summary: This paper introduces the concept of Multimodal Role-Playing Agents (MRPAs), develops a multimodal dataset (MMRole-Data) and evaluation framework (MMRole-Eval), and creates a specialized MRPA model, MMRole-Agent, achieving improved multimodal understanding and role consistency.
weaknesses: 1.Overreliance on GPT-4 for Evaluation: While MMRole-Eval provides a stable evaluation mechanism, it heavily relies on GPT-4, introducing a degree of bias. The authors validated the MAE between GPT-4 (humans) and Reward Model (humans), demonstrating consistency between the reward model and GPT-4. However, comparing two MRPAs to see which performs better does not substitute for genuine human judgment on the quality of MRPA responses. While this reward model may help MMRole-Agent approach GPT-4’s performance, its potential to surpass GPT-4 or elevate MRPAs to human-level capabilities remains debatable.
2.Lack of Performance Comparison with Single-Modality RPAs: Although the concept of MRPAs is appealing, the absence of specific experimental comparisons makes it difficult to understand exactly where MRPAs improve upon performance or accomplish tasks that single-modality RPAs cannot achieve.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Could you provide specific cases to analyze MMRole-Agent’s performance under In-Test and Out-Test conditions? 

2. Could you explain the "rigorous manual quality control" process in character profile generation？

3. Has the sensitivity of MMRole-Agent to different prompt templates been tested?

4. Could you discuss the primary limitations of MMRole-Agent, especially in terms of challenges encountered in practical applications and possible directions for future improvements?
soundness: 3
strengths: 1. This paper first introduces the concept of Multimodal Role-Playing Agents (MRPAs), extending traditional role-playing agents to the multimodal domain, filling a gap in existing research.

2. The MMRole framework includes both the construction of a dataset (MMRole-Data) and the design of an evaluation method (MMRole-Eval), covering eight metrics across dimensions such as fundamental conversational skills, multimodal understanding, and role-playing quality.

3. The proposed MMRole-Agent demonstrates strong performance.

4. The writing is good and easy to understand.
summary: This paper introduces a concept of Multimodal Role-Playing Agents (MRPAs), expanding traditional role-playing agents to tackle multimodal interactions. The paper introduce a framework (MMRole) including MMRole-Data and MMRole-Eval. The MMRole-Data is a large-scale, high-quality dataset with 85 characters, 11,000+ images, and 14,000 dialogues. The MMRole-Eval is a robust evaluation method with eight metrics across three dimensions: conversational skills, multimodal understanding, and role-playing qualities.
weaknesses: 1. The paper lacks case studies, which could help illustrate MMRole-Agent's performance across diverse roles and dialogue scenarios.

2. The paper mentions that character profiles undergo "rigorous manual quality control," it does not provide detailed quality control standards or processes.

Meta-review: This paper introduces the concept of Multimodal Role-Playing Agents (MRPAs), expanding traditional role-playing agents to tackle multimodal interactions. The paper introduce a framework with datasets and evaluation metrics for these multimodal role-playing agents. This includes a large-scale, high-quality dataset with 85 characters, 11,000+ images, and 14,000 dialogues, and eight evaluation metrics across three dimensions: conversational skills, multimodal understanding, and role-playing qualities.

After the discussion period this paper received mixed reviews, 2 marginal reject and 2 accept. The reviewers generally found the newly proposed setting of multimodal role-playing agents interesting and novel, and appreciated the effort gone into creating the new dataset and developing new evaluation metrics. They also found the experiments and analysis comprehensive.

For the 2 reviewers who voted reject but did not respond, I went through the discussions and feel that the authors have addressed them reasonably (see details below), so I advocate for acceptance.

Decision: Accept (Poster)