MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents
Authors: Yanqi Dai
Huanran Hu
Lei Wang
Shengjie Jin
Xu Chen
Zhiwu Lu
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Role-Playing Agents
Large Multimodal Models
Primary area: datasets and benchmarks
TL;DR: We propose MMRole, a comprehensive framework for developing and evaluating Multimodal Role-Playing Agents, comprising a personalized multimodal dataset and a robust evaluation approach.

Abstract: Recently, Role-Playing Agents (RPAs) have garnered increasing attention for their potential to deliver emotional value and facilitate sociological research.
However, existing studies are primarily confined to the textual modality, unable to simulate humans' multimodal perceptual capabilities.
To bridge this gap, we introduce the concept of Multimodal Role-Playing Agents (MRPAs), and propose a comprehensive framework, MMRole, for their development and evaluation, which comprises a personalized multimodal dataset and a robust evaluation approach.
Specifically, we construct a large-scale, high-quality dataset, MMRole-Data, consisting of 85 characters, 11K images, and 14K single or multi-turn dialogues.
Additionally, we present a robust evaluation approach, MMRole-Eval, encompassing eight metrics across three dimensions, where a reward model is designed to score MRPAs with the constructed ground-truth data for comparison.
Moreover, we develop the first specialized MRPA, MMRole-Agent.
Extensive evaluation results demonstrate the improved performance of MMRole-Agent and highlight the primary challenges in developing MRPAs, emphasizing the need for enhanced multimodal understanding and role-playing consistency.
The data, code, and models are all available at https://github.com/YanqiDai/MMRole.

Published as a conference paper at ICLR 2025
MMRole: A COMPREHENSIVE FRAMEWORK FOR DEVELOPING AND EVALUATING MULTIMODAL ROLE-PLAYING AGENTS
Yanqi Dai Huanran Hu?, Lei
Jin', Xu ; Lu
"aaoling School of Artificial Intelligence. Renmin University of China "Colleee of Information and Electrical Engineering, China Agricultural University
ABSTRACT
Recently, Role-Playing Agents (RPAs) have garnered increasing attention for their potential to deliver emotional value and facilitate sociological research. However, existing studies are primarily confined to the textual modality, unable to simulate humans' multimodal perceptual capabilities. To bridge this gap, we introduce the concept of Multimodal Role-Playing Agents (MRPAs), and propose a comprehen- sive framework, MMRole, for their development and evaluation, which comprises a personalized multimodal dataset and a robust evaluation approach. Specifically, we construct a large-scale, high-quality dataset, MMRole- Data, consisting of 85 characters, 11K images, and 14K single or multi-turn dialogues. Additionally, we present a robust evaluation approach, MMRole-Evee Eval, encompassing eight metrics across three dimensions, where a reward model is designed to score MRPAs with the constructed ground ground-truth data for comparison. Moreover, we develop the first specialized MRPA, MMRole-Agentt Extensive evaluation results demonstrate the improved performance of MMRole-Agen and highlight the primary challenges in developing MRPAs, emphasizing the need for enhanced multimodal understand- ing and role-playing consistency. The data, code, and models are all available !
INTRODUCTION
The advancement of large language models (LLMs) (Zhao et al. 2023) has significantly catalyzed the rise of Role- Poly-Playing Agents (RPAs) (Chen et al. 2024b), which are engineered to emulate spe- cific characters and engage in dialogues with human users or other characters. Unlike Al produc-
tivity assistants, RPAs primarily focus on delivering emotional value (Li et al.) 2023a Wang et al., 2023 Shao et | (023) and l'cillitating sociological research (Zhou et al. 2023b Wang et al. 2024c
Chen el al. 2024a] Gu et al, 2024), where typical applications include emotional companions, NPCs
in video games, digital clones, and social simulations.
The primary characteristic of RPAs is their capability to engage in human-like and immersive in- teractions. However, existing studies in role-playing are primarily confined to the textual modality, which has considerable limitations. In the real-world context, human perception integrates multiple modalities, especially visual and textual, allowing for more direct and comprehensive understand- ing of the environment than text alone could provide. Therefore, enhancing RPAs with multimodal capabilities is crucial next step for conducting more realistic and engaging interactions.
In this paper, we introduce the concept of Multimodal Role-Playing Agents (MRPAs). MRPAs are designed to emulate specific characters and engage dialogues centered around images, with either human users or other characters. Furthermore, we propose MMRole, a comprehensive framework for developing and evaluating MRPAs. As presented in Figure [ this framework includes two principal components: large-scale, high-quality dataset and a robust evaluation approach for MRPAs.
The dataset for MRPAs comprises character profiles, images, and dialogues centered around images. Specifically, we consider three categories of characters: fictional charac- ters, historical and public figures, and hypothetical real-life characters. The profiles of the first two
Dataset Construction:
"Corresponding authors: Xu Chen @ and Zhiwu Lu (luzhivu@@rac.cdn..c)) http:://gguuub.cooaaaiiiiiaai

---

Published as a conference paper at ICLR 2025
Character Profiles
Evaluation
Harry Petter
Fundamental
WIKIPEDIA
bed Intoderker
Conversatioal
dhonettion Creegg bouginr
The Saar semilill
Skills
Poue oegaal in om anrtfctiinn fum Pole Thite ee pepll aniplics ene aanic Bey"
Bai
Summmey
Rchicceskge
Inrraction Athunane
Flaeng
Mesa Infortatiio
Coouss
Nomer Slong
Cahennoc
Age
Pensuaityy
Life bways
Multimodal
Repane Fnbeet Lrai
Rupeera mulh on Thoy'c
Gonzzal
Fenssallyy Sky but srdie
Expaad
Understandilg Abilities
them herm tirel dhunld
Images
Dialogues
Imsdge Test Relevazes
raugli
And
dect
Resporse Aecuricy
M
Ganernte
Role-Playing Qualities
Reward Mixdal
R Possnddlt
Perumality Conestency
Imagie
Knowledge Consiitency
Saraac caraicac nicc dicasll ruflcct sitees
Tose Cassislency
lepees saliraaliee aegigiiii the unat uppal of jauileys
natiomeing rask
(a) Dataset Construction
(b) Performance Evaluation
An overview of the MMRole framework. (a) MMRole-Data includes character profiles, images, and dialogues centered around images. (b) MMRole- Eval comprises eight evaluation met- rics across three dimensions. For each metrie, the reward model scores MRPAs with the constructed ground-truth data for comparison.
Figure 1:
categories summarized by GPT-4 (Achiam et al. 2023) based information from Wikipedia Baidu Baike) while those of the last category are randomly generated by GPT-4. For each charac- ter, we utilize distinct generic images from MS-COCO (Lin et al.) 2014). and manually collect and annotate various character-
images. Finally, the dialogues are generated by GPT-4 based on the character profiles and images, occurring either between character and human user or between two characters. Both the character profiles and the dialogues are subjected to rigorous manual qual- ity control to ensure accuracy and reliability. Statistically, the MMRole-Dale dataset comprises 85 characters, 1 11K images, and 14K dialogues, yielding 85K training samples and 294 test samples.
Performance Evaluation: On one hand, MRPAs must not only accurately emulate specific char- acters but also deeply comprehend both visual and textual information. On the other hand, existing methods for evaluating RPAs directly score outputs without a ground truth (Zhou et al. 2023a Tu et al. 2024), which may lead to unstable scoring criteria without baseline for comparison. There- fore, we propose MMRole-BEl Eval, robust evaluation approach to stably and comprehensively assess MRPAs, which comprises eight metrics across three dimensions: fundamental conversational skills, multimodal understanding abilities, and role-playing qualities. For each metric, our specialized re- ward model initially conducts a brief qualitative assessment of the relative performance between the evaluated MRPA and the constructed ground-trute data, followed by assigning a quantitative score pair. The final score of the MRPA is the ratio of the two scores within the score pair. To develop the reward model, we employ GPT-4 to assess various MRPAs and leverage the evaluation trajectories to train our reward model, which renders MMRole-Eve both open-source and cost-effective.
Briefly, our main contributions are three-fold:
1. We propose the concept of Multimodal Poly-nlaying Agents (MRPAs) for the first time, and con- struct MMRole-Datua a large-scale, high-quality dataset for developing and evaluating MRPAs.
2. We introduce MMMole-- Eval, a robust evaluation approach to stably and comprehensively assess MRPAs, comprising eight metrics across three dimensions. specialized reward model is trained score MRPAs with the constructed ground-truth data for comparison.
3. We develop the first specialized MRPA, MMRole-Agent, and conduct comprehensive evaluations and analyses of MMRole Agent alongside various
large multimodal models.
RELATED WORK
Role-Playing Agents. Recent advancements in large language models (LLMs) (Zhao et al. 2023), such as supervised fine-tuning (Wei et al. 202T) and in-context learning (Brown et al. (020). have significantly catalyzed the rise of Role- Agents (RPAs) (Chen et al. 2024b), which are in-
2

---

Published as a conference paper at ICLR 2025
teractive AI systems that can emulate designated personas. Specifically, the personas can rized into individual characters (Wang et al., 2023] Shao et al.
be catego- 2023 Wang et al. 2024c
Gu et al.
2024
and groups of people with particular attributes (Li al. 2023b Hong al. 2023 Xu et al.
2023 Zhang et al. 2024). In this study, we primarily focus on the former.
Existing RPAs that emulate individual characters are developed through either training prompt- ing LLMs with high-quality character-speche speciffc dialogues. In a pioneering study, [Chen et al. (2023)
extracted all dialogue sessions from original scripts to develop Harry Potter-specific RPA. Further-
more, Wang et al. (2023). Zhou et al. (2023a), [Shao et al. (2023) and Li et al. (2023a) constructed hundreds of characters and more comprehensive datasets of character dialogues. These efforts aimed to develop RPAs for delivering emotional value to humans. In contrast, [uu et al. (2024) focused on
facilitating sociological research. However, these studies are primarily confined the textual modal- ity. Conversely, our MMRole framework is the first to enhance RPAs with multimodal capabilities.
The evaluation of RPAs is also a crucial and challenging research direction. Diverse methods have been proposed. Specifically, Sheen al. (2023) and Chen et al. (2024a) assessed RPAs with multiple- choice questions. Tu et al. (2024) trained a reward model for scoring without ground truth. WWang et al. (024d) evaluated the personality fidelity of RPAs through interviews, scoring without a ground truth by GPT-4. Ng et al. (2024) engaged the acquaintances of the target individuals to distinguish between humans and RPAs. Wang et al. (2024a) further evaluated RPAs in text-based virtual worlds. However, the high expense of human annotation and the potential instability of scoring without a ground truth pose significant challenges. To address this, we develop reward model to score RPAs with a ground-truth baseline for comparison.
Large Multimodal Models.
Large Multimodal Models (LMMs) are advanced Al systems typically built upon LLMs, designed to integrate and comprehend multiple data modalities, particularly text and images (Yin al. 2023). A variety of impressive LMMs have been released, including closed- source models with hundreds of billions of parameters like GPT-4V (Achiam et al. 2023). Gemini
(Team et al.] 2023), and Claude 3 (Anthropic) 2024). and open-source models with tens of billions or billions of parameters like MiniGPT-4 (Zhu et al. 2023), InstructBLIP (Dai et al. 2023). LLaVA
(Liu et al. 2024c a bj Li et al.] (024a). QWen WL. (Bai et al. 2023) Wang et al.) 2024b), Intern VL. (Chen al.] 2024c), and Yi-VL (Young et al. 2024). Additionally, various techniques have been explored to enhance the performance of LMMs, such as visual instruction tuning (Liu et al., (024c), mixture of experts (Lin al. 2024). and multi-task balancing (Dai et al. 2024). LMMs are widely applicd in vertical fields, including healthcare (Li 2022b). document understanding (Ye et al. 2023), and GUT navigation (Hong al., 2024 to role-playing for the first time.
To further explore their potential, we apply LMMs
MULTIMODAL POLEIIG AGENTS
Role-Playing Agents (RPAs) are engineered to emulate specific characters and engage in dialogues with either human users or other characters. Expanding on this concept, Multimodal Role-Playine Agents (MRPAs) incorporate the capacity to comprehend vision-language multimodal information. This capacity enables dialogues that are centered around and informed by images. From another per- spective, compared to traditional multimodal question answering, multimodal rolc-playing includes character profile input, adding greater complexity and depth to the interaction.
In scenarios where the dialogue partner is a human user without a specific identity, given an image 1, the profile P of the designated character C, and the dialogue context D, the MRPA steps into the shoes of the character C,, responding to the human user about the image I:
D [hi mi, h2, m2, ..., hn], hn))
m. MRPA(I, P, D),
(2)
where D is a sequence of conversation pairs, with h2 and m, representing the i-th utterances from the human user and the MRPA, respectively.
Conversely, in scenarios where the dialogue partner is another character Cathers given an image I, the profile P of the designated character C, the profile Pater of the character Cother, and the dialogue context D, the MRPA steps into the shoes of the character C and interacts with the character Cothers either initiating or responding within the dialogue centered around the image I:
D (01; m1, O2 m2, ...,....., 0m] or [m1, 01, m2; 02,-.., mm-11 On 1n-1) 1)p
(3)
3

---

Published as a conference paper at ICLR 2025
book Whirt of
Thenky leg't somal Seiie
coll bullie ynu scrvings
Horry
soeiin tha the celectucinn beroet
our lndin oo ug topti
ecpiidy fir lepeoed
Sallig bo Hovenvnn
giiiit cle Secor
Pell you
den bon the anll
(a) Commentary Interaction
(b) Human-Role Dialogue
(c) Inter-Role Dialogue
Figure 2: Examples of the three types of dialogue scenarios in MMRole-Data.
my MRPA(I, P. Pother, D),
(4)
where D is sequence of conversation pairs, with o; and m, signifying the i-th utterances from the character Comer and the MRPA, respectively. Notably, both the character Comer and the MRPA can potentially initiate the dialogue.
4 MMRole-Data. DATASET CONSTRUCTION
As shown in I(a) we construct IMRole-Data, a large-scale, high-quality multimodal role-playing dataset. In this section, we first provide a detailed classification of characters and dialogue scenarios considered MMRole-Date then describe the pipelines for character profile generation and image collection and annotation, as well as the methodology for dialogue generation and filtering.
4.1 CHARACTERS AND DIALOGUE SCENARIOS
We consider three categories of characters: (1) Fictional Characters, characters created in fictional
media such as literature, films, and games; (2) Historical and Public Figures, individuals who are specifically documented in historical records or well-known in real life; (3) Hypothetical Real-Life Characters, hypothetical individuals who are not explicitly known but could exist in real life.
The first two categories have been explored in previous role-playing research. Moreover, we propose the third category to enhance and evaluate MRPAs in characters that are not widely recognized. To effectively emulate hypothetical real-life characters, MRPAs must deeply understand and align with the provided character profiles, rather than relying on their inherent world knowledge.
As depicted in Figure 2] we introduce three types of dialogue scenarios consistently centered around images: (1) Commentary Interactions, single-turn dialogues where a character offers comments or reflections centered around an image, without any further interaction; (2) Human-Role Dialogues, multi-turn dialogues centered around an image between a human user without a specific identity and character; (3) Inter-Role Dialogues, multi-turn dialogues centered around an image between two characters from the same series.
4.2 CHARACTER PROFILE GENERATION
Character profiles are crucial for the role-playing effectiveness of MRPAs, especially for those char- acters with which MRPAs are not familiar. To facilitate thorough understanding of the designated characters, our character profiles encompass five core parts: brief introduction, personality, life story, main interpersonal relationships, and catchphrases, which are exampled in Appendix [
As discussed in Section 4.1 three categories of characters are considered in MMRole. The majority of these characters are English, with smaller proportion being Chinese. For fictional characters, as well as historical and public figures, the profiles are summarized by GPT-4 where the information is sourced from Wikipedia for English characters and Baidu Baike for Chinese characters. Further- more, for hypothetical real-life characters, the profiles are generated through two-stage process by GPT-4 to ensure both universality and diversity. Firstly, GPT-4 generates the meta information for all characters in single API call, including basic details such as names and genders, and brief
4

---

Published as a conference paper at ICLR 2025
descriptions of personalities and backgrounds. We instruct GPT-4 that "The character information should cover as many different situations as possible to reflect the diversity and complexity of hu- man society". Secondly, GPT-4 expands the meta information of each character to derive the profile. The two-stage generation process is exemplified in Appendix F Additionally, all character profiles undergo rigorous manual quality control to ensure accuracy and reliability, detailed in Appendix and are simplified by GPT-4 to adhere to the context length limits of most LMMs.
4.3 IMAGE COLLECTION AND ANNOTATION
For each character, we utilize distinct generic images from MS-COCO (Lin et al. 2014) to ensure comprehensive coverage of wide range of visual concepts. Additionally, we manually collect and annotate various character-related images, which can evoke the personal experiences and emotions of the characters more effectively. Specifically, we collect production stills for fictional characters, web illustrations for historical and public figures, and news photos for hypothetical real-life charac- ters. Moreover, presented in Figure the information of characters, place, and scene is manually annotated for each character-related image.
4.4 DIALOGUE GENERATION AND FILTERING
As discussed in Section 4.1 three types of dialogue scenarios are introduced in MMRole. Based on the character profiles and images, GPT-4 generates dialogues corresponding to each scenario type. Interestingly, we observe that using the prompt, "You are a dedicated role-playing assistant... step into the shoes of {character) from {series)" yields better results than the simpler prompt, "You are {charucter) from {series}"" We suggest that the training data supplied by OpenAl optimizes helpful assistant, rather than an immersive, human-lit like
GPT-4 to function more effectively character. The prompts for dialogue generation are detailed in Appendix !| To ensure accuracy and reliability, manually filter all dialogues using several strategies, detailed in Appendix
MMRole-Eval. PERFORMANCE EVALUATION
As illustrated in Figure [(b] propose MMRole-Fxal, robust evaluation approach to stably and comprehensively assess MRPAs. In this section, we introduce eight evaluation metrics across three dimensions and the approach for score quantification.
5.1 EVALUATION METRICS
In contrast to textual RPAs, MRPAs must not only accurately emulate specific characters but also deeply comprehend both visual and textual information. Therefore, we propose a three-dimensional evaluation system, encompassing fundamental conversational skills, multimodal understanding abil- ities, and role-playing qualities.
The fundamental conversational skills of MRPAs present their capacity sustain fluent and coherent interactions within role-playing scenarios, which are assessed by three metrics:
Instruction Adherence (IA): Do the responses accurately adhere to the task instruction, directly role-playing as the character and including only words that the character would say, without any unnecessary explanatory prefixes or suffixes?
Fluency (Flu): Are the responses grammatically correct and articulated smoothly?
Coherency (Coh): Do the responses maintain a coherent thread of dialogue without contradicting previous turns or containing internal inconsistencies within the current responses?
The multimodal understanding abilities of MRPAs indicate their capacity to effectively integrate and interpret both visual and textual information, which are assessed by two metrics:
Image-Texx Relevance (TTR): Do the responses exhibit close correlation with the visual tent depicted in the image?
Response Accuracy (RA): Do the responses accurately answer the words of the human user or the other character, or appropriately initiate a conversation based on the image?
The role-playing qualities of MRPAs denote their capacity to convincingly emulate characters, main- taining consistency in personality, knowledge, and tone, which are assessed by three metrics:
Personality Consistency (PC): Do the responses accurately and deeply reflect the personality of the character?
5

---

Published as a conference paper at ICLR 2025
Table 1:
The statistics of MMRole-Date CR Images' represents character- images. 'In- InsTess' denotes the in-distribution test set, while 'Out-Test" signifies the out-of-distribution test set.
Train
In-Test
Tust
Overall
Characters
72
13
85
Generic Images
10,800
39
10,839
CR Images
175
18
193
Dialogues 14,052
216
78
14,346
Samples
85,456
216
85,750
Table The statistics for the three types of dialogue scenarios in MMRole- Data..
Comment.
Human-Role
Inter-Role
Overall
Dialogues
4893
4617
4836
14346
Turns / Dig.
1.00
5.80
5.75
4.15
Tokens Dig.
236.00
446.91
429.54
369.12
Knowledge Consistency (KC): Do the responses accurately reflect the knowledge of the char- acter, encompassing their experiences, abilities, and relationships?
Tone Consistency (TC): Do the responses align with the typical speech patterns and catchphrases of the character, rather than resembling the style Al assistants?
5.2 SCORE QUANTIFICATION
To quantitatively evaluate the performance RPAs across various metrics, existing methods utilize reward models or human annotators to directly
outputs without ground truth (Zhou al., Tu et al. 2024). However, it may be unstable due to the variability of scoring criteria without baseline for comparison. Therefore, we propose to develop a more stable reward model. Inspired by the evaluation methods of Vicuna (Chiang et al.) (202)) and LLaVA (Liu et al. 2024c), our reward model first conducts brief qualitative assessment of the relative performance between the evaluated MRPA and the constructed ground-trum data for each metric, followed by assigning quantitative score pair. The final score of the MRPA the ratio of the two scores within the score pair.
2023a
To develop the reward model, we initially employ GPT-4 to assess various MRPAs across all test samples. For each evaluated MRPA and corresponding test sample, GPT-4 outputs brief assessments and score pairs for all metrics through single API call. Subsequently, these evaluation trajectories are converted into training and validation data for our reward model. Each evaluation trajectory is segmented into eight samples, with each sample evaluating a distinct metric. The prompts for GPT-4 scoring and reward model scoring are provided in Appendix H Compared to directly applying GPT- 4 as the reward model, this approach renders MMRole- Eval both open-source and cost-effective.
6 EXPERIMENTS
6.1 STATISTICS OF MMRole-Date
Table presents the statistics of the MMRole-Data dataset. Totally, the dataset comprises 85 charac- ters, 11,032 images, and 14,346 dialogues, yielding 85,456 training samples and 294 test samples. Specifically, we construct 72 characters and collect 10,975 images for training and in-distribution testing. For out-of-distriibttion testing, we additionally construct 13 characters and collect 57 images that differ from those used the former set. Dividing the data this manner significant for eval- uating the performance of MRPAs on previously unseen characters and images, thereby assessing its generalization capabilities. All constructed characters are listed in Appendix|C
From another perspective, Table illustrates the statistics for the three types of dialogue scenarios in the MMRole-Data dataset. The commentary interactions are single-turn, whereas both the human- role dialogues and the inter-role dialogues involve multiple turns. When converting dialogues into training and test samples, a single multi-turn dialogue entry can generate multiple training samples single test sample randomly selected from specific turn.

---

Published as a conference paper at ICLR 2025
Table 3: The evaluated MRPAs in our experiments, which are grouped by parameter scale.
MRPAs
Version
Params
Open-Soure
Specialized
GPT-4 Turbo (Achiam ct al.
2023
2024-04-09
100B
X
Gemini Pro Vision [Team et 2023
2023-12-13
100B
X
Claude 3 Opus (Athropic
2024
2024-02-29
100B
X
Bai ct al.
2023
2023-12-01
100B
X
X
LLLVAANNeXT-34B [ al.
2024h
2024-01-30
34B
V
Yi- VL-3B (Young al. 2024
2024-01-23
34B
/
Intern Chen el
2024c
2024-04-18
26B
/
X
QWen-VL-Chat (Bai al. 2023
2023-08-22
LLaVA-NexT-Mistral (Liu et al.
2024b
2024-01-30
7B
Yi-VL-6B Young et al. 2024
2024-01-22
6B
/
X
MMRole-Agent (ours)
9B
The validation mean absolute error (MAE) results for the effectiveness the reward model. QWen-VLL-Char VL--Chat (GPT-4)' and 'Reward Model (GPT-4)² denote the scores evaluated by QWen-VL.- Chat and the reward model compared to those evaluated by GPT-4. JWWen-VL-Chal (humans)), 'GPT- (humans)', and 'Reward Model (humans)" signify the score gaps provided by QWen-VL- Chat, GPT-4, and the reward model compared to the ground-truth score gaps provided by humans.
Table 4:
Evaluators (Ground Truth)
IA
Flu
Coh
ITR
RA
PC
KC
TC
Overall
QWen- VL-Chat (GPT-4)
0.3776
0.3718
0.3218
0.3561
0.3528
0.4091
0.3794
0.4558
0.3780
Reward Model (GPT-4)
0.0708
0.0387
0.0526
0.0568
0.0584
0.1165
0.0815
0.1154
0.0738
QWen- VL-Chat (humans)
TWen-VV-CCHe
0.2469
0.1870
0.2720
0.2574
0.2608
0.2368
0.2243
0.2658
0.2439
GPT-4 (humans)
0.1526
0.1150
0.0772
0.0922
0.1463
0.1475
0.1279
0.1442
0.1254
Reward Model (humans)
0.0993
0.0815
0.1006
0.1225
0,1412
0.1669
0.1438
0.1507
0.1258
6.2 DEVELOPMENT OF MMRole-Agent
We tune the QWen- VL-Chat model (Bai et al. 2023) using x GPUs on the training set of MMRole-Date to develop our specialized MRPA, MMRole-Agent. Integrating data from different characters and dialogue scenarios for multi-task training of MMRole-Agen. The learning rate set to 1e- and the training conducted over epochs. To accommodate detailed character profiles and dialogue history, the model maximum length is set to 3072. Other experimental setups and codes remain the same as Bai ct al. (2023) defaults.
improve the generalization capabilities
6.3 EVALUATED MRPAS
To the best of our knowledge, no specialized MRPA has been developed prior to this work. There- fore, experiments evaluate MMRole-Agen/ and various existing general-dialogue LMMs across different parameter scales. As presented in Table we select four well-known closed-source LMMs with over 100 billion parameters (Achiam et al. 2023 Team al. 2023: (0033/Antrrppic [2024] Bai al.) 2023), and six widely-used open-source LMMs with tens of billions or billions of parameters (Liu al. 2024b Bai al. 2023 Young al. 2024 Chen al. 2024c), For the closed-source models, we utilize their official APIs to conduct performance evaluations. To ensure fairness, each MRPA is queried with the same prompt, as detailed in Appendix
6.4 DEVELOPMENT AND VALIDATION OF THE REWARD MODEL.
To develop the reward model, we initially utilize GPT-4 to evaluate various general-dialogue LMMs discussed in Section 6.3 across 294 test samples. Statistically, these evaluation trajectories are con- verted into a total of 23,520 samples, where 320 samples are reserved for validation, with the rest utilized for training. The validation set includes 20 questions, where the responses of two models are randomly selected for each question, and each response is evaluated on all metrics. Subsequently, another QWen-VL--hha VI--Chat model (Bai al. 2023) trained develop the specialized reward model.

---

Published as a conference paper at ICLR 2025
The experimental setup and code are the same those used for developing MMRole-Agent except that the model maximum length is set to 4096, and the training is conducted over 10 epochs.
The scoring success rate of the base model QWen-VL VL-Chat is only 33.13%. whereas those of the reward model and GPT-4 are both 100%. To further validate the effec- tiveness of the reward model, we initially calculate the mean absolute errors (MAEs) of the scores evaluated by QWen- QWen-VL-Chat and the reward model compared to those evaluated by GPT-4. The score is random if the model fails to score. As illustrated in Table 4 the overall MAE of QWen-VL. VL- Chat (GPT-4) is remarkably high, whereas that for Reward Model (GPT-4) is less than engage four human
GPT- Turbu Gemini Pro Visium Cluude Opus QWan-VL-MMan
NoX YI-VL-34B InteraVL Chat- V1.5
VWen-VI-Chat LoWaA--oo--MMssral-
Yi-VI-68 MMRolloAAget
Instruction
Fluency
Tome Consistency
Coherency
Knowledge Consistency
0.1. Furthermore, evaluators to compare responses from two MRPAs on each metric for every question in the validation set. Their choices among 'better', 'equal', and 'worse' correspond to the score gaps of 0.4, O, and -0.4 be- tween the two responses, respectively. The results from all evaluators are averaged to obtain the ground-trute score gaps. In this manner, it is easier for human evaluators to yield more consistent results among in- dividuals than directly scoring the MRPA's responses. Subsequently, we calculate the MAEs between the score gaps provided by OWen-VLL-Chal GPT-4, and the reward model compared to those provided by human evaluators. In this context, score gaps are capped at 0.4 and -0.4. As shown in Table the overall MAE for DoWen-VL-CHI (humans) is significantly high, whereas those for GPT-4 (humans) and Reward Model (humans) are comparable and considerably low. Moreover, report the root mean squared error (RMSE) and Pearson correlation coefficient (Pearson) re- sults in Appendix ! These results indicate that our specialized reward model effectively learns the evaluation abilities of GPT-4 and aligns closely with human evaluators, significantly superior the non-specialized QWen- QWen-VI-CCaa VI--Cat model.
Imago-Text
Parsonality Consistency
Relevance
Respanse Accuracy
Figure 3: The visualization of the evaluation results for all MRPAs. Each indicator displays an interval length of 0.55, and the maximum value of the interval for dif- ferent indicators is adjusted from 1.10 to 1.25.
Besides, we assess the internal consistency of MMRole-Eval by Cronbach's alpha coefficient (Cron- bach 195T), where the score of an evaluated MRPA for given test query is computed as the average across all metrics. The resulting value of 0. indicates that MMRole- of internal consistency. The relatively lower value can be attributed to the diversity of characters and dialogue scenarios, as individual queries inherently assess distinct aspects.
exhibits a moderate level
6.5 EVALUATION RESULTS AND ANALYSES
As shown in Table 5 we report the average results across all test samples for each evaluated MRPA, along with the detailed results on both the in-distribution test set (In-Test) and the out-of-distribune test (Out-Test) for our MMRole-Agent. Notably, although some of the scores of MRPAs exceed 1, this does not necessarily mean that their performance is superior to that of GPT-4, which we use construct MMRole- -Dta. To generate multi-turn dialogue data, we use single GPT-4 API call to produce the entire dialogue directly. In contrast, when testing MRPAs, we supply dialogue histories and require MRPAs to generate responses. This approach is relatively easier, but is challenging to ensure the consistency of multi-turn dialogues if used for data construction.
In the MRPA group with over 100 billion parameters, Claude 3 Opus exhibits superior performance. Meanwhile, in the MRPA group with tens of billions of parameters, LLa ALaVA-NNeT-334 achieves the highest performance. Finally, in the MRPA group with billions of parameters, MMRole-Agent is the VA-NeXT--Miite
best. Notably, LLaVA-NeeT-348 outperforms Gemini Pro Vision, while and MMRole-Agent surpass Yi- VL-34B. This suggests that both the training methods and training data important for enhancing LMMs, rather than merely expanding the model size.
8

---

Published as a conference paper at ICLR 2025
Table 5: The average results across all test samples for each evaluated MRPA, along with the detailed results for our MMRole-Agen on both the in-distribution test (In-Test and the out-of-ddstribution test set (Out-Test) In each group categorized by parameter scale, the best overall result is bolded, while the second-best one underlined.
MRPAs
IA
Flu
Coh
ITR
RA
PC
KC
TC
Overall
GPT- Turbo
1.055
1.032
1.084
1.097
1.092
1.168
1.103
1.161
1.099
Gemini Pro Vision
0.999
1.007
1.028
1.009
1.013
1.052
1.013
1.050
1.021
Claude Opus
1.127
1.070
1.149
1.167
1.146
1.219
1.168
1.213
1.157
-L-Max
1.014
1.012
1.035
1.034
1.029
1.042
1.021
1.041
1.028
LLaVA-NeXT-34B
1.002
1.007
1.021
1,033
1.035
1.053
1.030
1.038
1.027
Yi-VL-34B
0.895
0.968
0.910
0.875
0.863
0.844
0.869
0.845
0.884
IntemVL-Chat-V15
0.988
0.996
0.997
0.977
0.984
0.967
0.972
0.960
0.980
QWen-VL-Chat
0.844
0.954
0.879
0.850
0.829
0.778
0.827
0.785
0.843
LLaVA-NeXT-Mistral-7
0.948
0.986
0.964
0,938
0.933
0.924
0,940
0.921
0.944
Yi-VI.6B
0.844
0.919
0.859
0.828
0.811
0.776
0.820
0.774
0.829
MMRole-Agent
0.998
1.000
0.997
0.993
0.987
1.000
0.992
0.988
0.994
MMRole-Agent (In- Test)
1.000
1.000
0.999
0.997
0.989
1.012
0.997
0.997
0.999
MMRole-Agent (Out-Test)
0.992
0.999
0.993
0.979
0.981
0.963
0.977
0.962
0.981
Table 6: The average results across all test samples for each evaluated RPAs. "w/o vision' signifies that image information is excluded from the input prompt of RPAs.
RPAs
Comment.
Nlman-Role
Inter-Role.
Overall
GPT-4 Turbo w/o vision
0.5746
1.1330
1.0843
0.9306
GPT-4 Turbo
1.0261
1.2275
1.3450
1.1995
Claude Opus w/o vision Claude Opus
0.3290
1.1803
1.1420
0.8838
1.0088
1.2889
1.3916
1.2298
MMRole-Agent w/o vision
0.4192
0.8909
0.7907
0.7003
MMRole-Agent
1.0450
0.9556
0.9619
0.9875
Moreover, the overall score of MMRole-Agent reaches 0.994, marking significant improvement of 0.151 compared to its base model, QWen-VL-Chat. MMRole-Agent successfully acquires various capabilities required for the MRPA from MMRole-Date outperforming all evaluated open-source LMMs, except for both the in-distribution test set and the nut-of-distributie test set, with the latter being slightly lower by 0.018. This indicates that MMRole-Agent has strong generalization capabilities for characters and images that are not seen in the training set.
Besides, MMRole-Agem achieves similar overall scores on
As shown in Figure 3 we provide a clear visual representation of the evaluation results. The overall performance rankings of MRPAs closely align with their specific rankings on each metric. However, significant differences exist in the score variations across various metrics. Specifically, all MRPAs achieve high scores on the Fluency metric with minimal variations, suggesting that producing fluent content is not a major challenge for current LMMs. Conversely, there are notable differences among MRPAs on other metrics, particularly on Personality Consistency and Tone Consistency. reveals that multimodal understanding abilities and role-playing qualities are more challenging aspects that require attention in the development of MRPAs.
Additionally, to highlight the inherent advantages of MRPAs over single-modal RPAs, we conduct comparative experiments on two SOTA general-purpose LMMs and our MMRole-Agent. As pre- sented in Table 6 we report the Image-Text Relevance scores the Out-Test set evaluated by GPT-4, where "wo vision' signifies that image information is excluded from the input prompt of RPAs. The results clearly demonstrate that excluding image information significantly reduces the Image-Text Relevance of all RPAs responses, particularly in commentary interaction scenarios. In multi-turn human-role and inter-role dialogue scenarios, textual dialogue history could provide in-

---

Published as a conference paper at ICLR 2025
Table 7: The results on the Out-Test set for MMRole-Agent with different numbers of characters.
Characters
Number of Characters
Overall
The Avengers
16
0.965
The Avengers Other English Fictional Characters
34
0.968
The Avengers + Hypothetical Real-Line Life Characters
36
0.970
ALL (ours)
72
0,983
direct clues about images, leading to relatively smaller declines in the Image-Text Relevance scores compared to those commentary interactions. Nevertheless, the absence of visual inputs still results in marked drop in performance across all scenarios.
6.6 DETAILED ANALYSES OF MMRole-Agent
To demonstrate the superiority of our MMRole- Agent, use cases of MMRole-Agent, GPT-4, and QWen-VL-Che are presented and analyzed in Appendix K Moreover, sensitivity test results detailed in Appendix indicate that MMRole- Agent is compatible with different prompts and does not exhibit signs of overfitting. The strong performance and generalization abilities of our MMRole-Agent can be primarily ascribed to the following two factors:
Table 8: The average results across all test sam- ples for MMRole-Agent with different numbers of training samples.
Training Data
Number of Samples
Overall
SAMPLE ALL (ours)
8.5K 85K
0.967
0.989
1. Training with Large-Scale, High-Quality Data: The training set of MMRole-Dala comprises
72 characters, 1 11K images, and over 85K samples. Furthermore, as depicted in Figure [[a] and Figurd2] due to the well-designed data construction pipeline, meticulous manual annotation and quality control, and the use of GPT-4, the data is of high quality. This large-scale, high-quality dataset enables MMRole-Agent to comprehensively learn the instruction demands, knowledge, and abilities in multimodal role-playing. To verify this point, we compare the performance dif- ferences between model trained on the full dataset (ALL) and a model trained on randomly sampled subset consisting of one-tenth of the data (SAMPLE), both evaluated after one epoch of training. As shown in Table g the performance of ALL is superior to that of SAMPLE.
Joint Training with Diverse Multi-Character Data: We incorporate data from 72 diverse char- acters to jointly train unified MMRole-Agent. This approach, akin to the principles of multi-task learning, enables the model to acquire generalizable multimodal role-playing capabilities, rather than being confined to specific characters. To verify this point, first train model using data of characters from The Avengers, then gradually add additional characters to the training set for subsequent models. As presented in Table 7
we evaluate the performance of each model on the Out-Test set. The models' zero-shot performance steadily improves as more characters are in- corporated. Notably, with comparable numbers of characters, introducing hypothetical real-life characters (with significant differences from The Avengers) yields greater gains than adding other English fictional characters, indicating the significance of training with diverse data.
CONCLUSION
In this paper, we propose the concept of Multimodal Role-Playing Agents (MRPAs) for the first time by extending RPAs with multimodal understanding abilities. Moreover, we construct MMRole-Dait a large-scale, high-quality dataset for developing and evaluating MRPAs. To stably and comprehen- sively assess MRPAs, we introduce MMRole Eval,, robust evaluation approach that comprises eight metrics across three dimensions, scoring MRPAs with the ground truth for comparison by a special- ized reward model. Evaluation results reveal that our MMRole-Agenn the first specialized MRPA, exhibits improved performance and strong generalization capabilities. Additionally, multimodal un- derstanding abilities and role-playing qualities are more challenging aspects that require attention in the development of MRPAs. However, there exists a limitation that the training data for MMRole- Agent is primarily synthesized by GPT-4, which constrains its performance from surpassing GPT-4 itself. In future work, we will address this limitation by leveraging multiple SOTA LMMs respec- tively responders, reviewers, and summarizers, striving push the boundaries of its capabilities.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by National Natural Science Foundation of China (62437002, 62376274) and Beijing Natural Science Foundation (L.233088).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ArXiv Preprint ArXiv:2333.08774 2023.
Anthropic.
The claude 3 model family: Opus, sonnet, haiku, http:://www.....attttttt. com/ com/de&be de@ba
3.pdf 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. ArXiv Preprint ArXiv:2308. 01.222388./9966 /2966, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Xing Gao, Weizhou Shen, Xiaojun Quan, Chenliang Li, Ji Zhang, Fei Huang, et al. Roleinteract: Evaluating the social interaction of role- playing agents. ArXiv Preprint ArXiv:2403. 3679, 2024a.
Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, et al. From persona to personalization: A survey on role-playing language agents. ArXiv Preprint ArXiv:2404.18231 2024b.
Nuo Chen, Yan Wang, Haiyun Jiang, Deng Cai, Yuhan Li, Ziyang Chen, Longyue Wang, and Jia Li. Large language models meet harry potter: A dataset for aligning dialogue agents with characters. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 8506-8520, 2023.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- visual-linguistie tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024c.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph F. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: //Lmyys..rg/blog//////a
03 30 viiuna/
Lee J Cronbach. Coefficient alpha and the internal structure of tests. Psychometrika, 16(3):297-334, 1951.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards -anrral-prrpoee vision-language models with instruction tuning. ArXiv Preprint ArXiv:2305.066. 0660000 2023.
Yanqi Dai, Dong Jing, Nanyi Fei, and Zhiwu Lu. Cotbal: Comprehensive task balancing for multi- task visual instruction tuning. ArXiv Preprint ArXiv:2403.0433 04343, 2024.
Zhouhong Gu, Xiaoxuan Zhu, Haoran Guo, Lin Zhang, Yin Cai, Hao Shen, Jiangjie Chen, Zheyu Ye, Yifei Dai, Yan Gao, et al. Agent group chat: An interactive group chat simulacra for better eliciting collective emergent behavior. ArXiv Preprint ArXiv:2403. /3433, 2024.
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi- agent collaborative framework. ArXiv Preprint ArXiv:2308.0000 00352, 2023.

---

Published as a conference paper at ICLR 2025
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14281-14290, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. ArXiv Preprint ArXiv:2408.0332 033266 2024a.
Cheng Li. Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, et al. Chatharuhi: Reviving anime character in reality via large language model. ArXiv Preprint ArXiv:2308.0900 09597, 2023a.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang. Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and- assis- tant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024b.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for "mind" exploration of large language model society. Advances in Neural Information Processing Systems, 06:51191-52000 2023b.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning. and Li Yuan. Moe-Ilava: Mixture of experts for large vision-language models. ArXiv Preprint ArX1v:2401.15447, 2024.
Tsung-Yt Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL htttps://
llava vl. www.ith.b....................... :o/blog//222 01-30
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024c.
Man Tik Ng, Hui Tung Tse, Jen-tse Huang, Jingjing Li, Wenxuan Wang, and Michael R Lyu. How well lms echo us? evaluating chatbots' role-play ability with echo. ArXiv Preprint ArXiv:2404. 2002224444..7 2024.
Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. Character-Ilm: A trainable agent for role- playing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13153-13187, 2023.
Tianhao Shen, Sun Li, and Deyi Xiong. Roleeval: A bilingual role evaluation benchmark for large language models. ArXiv Preprint 23..::3312..1133 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. ArXiv Preprini ArXiv:2312. 1805, 2023.
Quan Tu, Shilong Fan, Zihang Tian, and Rui Yan. Charactereval: A chinese benchmark for role- playing conversational agent evaluation. ArXiv Preprint (.XXx:2440.012277 2024.
Lei Wang, Jianxun Lian, Yi Huang, Yanqi Dai, Haoxuan Li. Xu Chen, Xing Xie, and Ji- Ri-Rong Wen. Characterbox: Evaluating the role-playing capabilities 1lms in text- based virtual worlds. ArXiv Preprint ArXiv:24/2.0563 2024a.
12

---

Published as a conference paper at ICLR 2025
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- langugge model's perception of the world at any resolution. ArXiv Preprint ArXhv:2409.1119) 2024b.
Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Graham Neubig, Yonatan Bisk, and Hao Zhu. Sotopia-T: Interactive learning of socially intelligent language agents, ArXiv Preprint ArXiv:2403.08715 2024c.
Xintao Wang, Yunze Xiao, Jen Ise Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang. Jiangjie Chen, Cheng Li, and Yanghua Xiao. Incharacter: Evaluat- ing personality fidelity in role-playing agents through psychological interviews. ArXiv Preprint ArXiv:2310. 17976, 2024d.
Zekun Moore Wang. Zhongyuan Peng. Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, et al. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. ArXiv Preprint ArXiv:2310.00740 2023.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models zero-shot learners. ArXiv Preprint ArXiv:2109.0163 2021.
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf. ArXiv Preprint ArXiv:2309. 04658, 2023.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. mplug-docowl: Modularized multimodal large language model for document understanding. ArXiv Preprint ArXiv:2307.02449: 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey multimodal large language models. ArXiv Preprint ArXiv:2306.13549 2023.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, al. Yi: Open foundation models by 01. ai. ArXiv Preprint ArXiv:2403.04652 2024.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge Xin Cheng, Sirui Hong. Jinlin Wang, et al. Aflow: Automating agentic workflow genera- tion. ArXiv Preprint ArXiv:2410. 10762, 2024.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. ArXiv Preprint ArXiv:2303.18225 2023.
Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng. Jiaming Yang. Xiyao Xiao, et al. Characterglm: Customizing chinese conversational ai characters with large language models. ArXiv Preprint ArXiv:2311.1683. 16832, 2023a.
Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi. Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, al. Sotopia: Interactive evaluation for social intelligence in language agents. ArXiv Preprint ArXiv:2310.1166 2023b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing lanionel understanding with advanced large language models. ArXiv Preprint ArXiv:2304 10592, 2023.
13

---

Published as a conference paper at ICLR 2025
A ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics, ensuring ethical compliance throughout all stages of the research. The MMRole-Dait dataset was constructed using publicly available data, with rigorous quality control to prevent privacy risks. We acknowledge the potential biases present in the data and have taken proactive measures to ensure diversity and reduce these biases. Moreover, we recognize that MRPAs may generate dialogues that could misleadingly appear as actual statements made by real individuals. To prevent misunderstanding. it is essential to explicitly indicate that such content simulated and does not represent genuine speech.
B REPRODUCIBILITY STATEMENT
For research reproducibility, the data, code, and models are all available at this GitHub repository Additionally, the training settings MMRole-Agent and the reward model MMRole-Eval are pre- sented in Section and and Section respectively. while the training and inference code, along with the detailed assessment results of evaluated MRPAs are submitted as supplementary materials. Moreover, the detailed prompts for dataset construction and performance evaluation are provided in Appendix E Appendix [[ Appendix G and Appendix [H] while the generated data is exemplified in Appendix Appendix D and Figure
C LIST OF CHARACTERS
Fictional Characters
The Aveeggrs: Iron Man, Captain America, Thor, Hulk, Spider-Mun, Black Widow, Hawkeye, Loki, Doctor Strange, Vision, Black Panther, Ant-Man, Scarlet Witch, Star-I ord, Nick Fury, Thanos
X-Mem: Wolverine, Professor X, Magneto, Phoenix
Harry Potter: Harry Potter, Hermione Granger, Ron Weasley, Albus Dumbledore, Severus Snape, Lord Voldemort
Toy Story: Woody, Buzz Lightyear
Friends: Rachel Green, Monica Geller. Phoebe Buffay, Jocy Tribbiani, Chandler Bing. Ross Geller
PE i2: WM- 22, H S6 P,E, #11 A till W.. ti,,,, 1),,, %
The Big Bang Theory: Howard Wolowitz, Leonard Hofstadter, Penny, Raj Koothrappali, Sheldon Cooper
Historical and Public Figures
the Renaissance period: Michelangelo, Leonardo da Vinci, William Shakespeare, Nicolaus Copernicus modern times: Stephen Hawking, Steve Jobs, Kobe Bryant, Diego Maradona, Michael Jackson 11199. H.lll, *E
Ancient Greece period: Socrates, Plato, Aristotle
Hypothetical Real-Life Characters
Hyputhetical Characters: Maya Patel, Liam Johnson, Sofia Redriguez, Takumi Nakamura, Chloe Dubois, Heary O'Malley, Aisha Al-Farsi, Carlos Riveru, Nia Williams, Alex Zhang, Elizabeth "Lizzy" Thompson, Dimitri Petrov, Jasmine Lee, Michael O'Reilly, Sunita Krishnan, Luca Bianchi, Fatima Zahra, kithan Wright, Priya Singh, Omar Abdullah Hypothetical Characters: Javier Mattinez, Ayesha Khan, Timothy Clark, Elena Petrova, Charles "Charlie" Wembly
Figure 4: All character constructed in MMRole-Data, with the series which the characters in the out-of-distributin test set belong being underlined.
Figure lists all characters constructed in MMRole MMRale-Data, with the series to which the characters in the out-of- test set belong being underlined.
D EXAMPLES OF CHARACTER PROFILES
Figure 5 presents the profile Iron Man from The Avengers, whereas Figure illlustrats the profile
of Li Bai from Tang Dynasty China. The character profiles include five core parts: brief introduc- tion, personality, life story, main interpersonal relationships, and catchphrases, undergoing rigorous manual quality control ensure accuracy and reliability.
14

---

Published as a conference paper at ICLR 2025
Profile of Iron Man
Brief Introduction:
Tony Stark, also known us Irom Man, is central characier in the Marvel Cinematic Universe, played by wealthy CEO of Stark Industrics and weapons manufacturer who
Robert Downey Ho transforms into superhero after being captured by terrorists. Over time, he becomes key member of the Avengers and sacrifices life to save the universe from Thamos. Ilis actions and inventions leave lasting MCU.
impact world other Personality:
Tony Stark complex character known for his genius intellect, charisma, and sense of humor, even tough situations. Although initially self-centered and reckless, he grows into responsible lender, willing to make personal sacrifices to protect others. His journey from weapons manufacturer sacrfficn hero significant growth, dealing personal relationship struggles. He also acts mentor to younger heroes like Peter Parker (Spider-Man). showing his nurturing side. Life Story:
Born on May 29, 1970, Tony Stark inhorited Stark Industrics after his parents were killed by the Winter Soldier. His life changed when kidnapped Afghanistan, leading creation the Iron Man
faced many enemies and challenges, both Earth and Avenger fighting against alien threats. Stark developed from playboy into committed hero who oventually sacrificed his life using the Infinity Stones to defeat Thanos and undo the massive destruction caused by him.
Iron
Main Interpersonal Relationships:
less detailed bond with mother. 2. Pepper Polis: Initially his ussistant, she becomes his wife and mother his daughter, Morgan. Their relationship deepens time.
Parents (Howard and Maria Stark): Tony strained relationship with
3. James Rhodes (War Machine): Stark's loyal friend and ally despite some disagreements.
Peter Parker (Spider-Man): Stark mentors Peter, secing him successor.
5. Other Avengens: He has intricate dynamics with other members like Bruce Banner and Steve Rogers, involving both collaboration and conflict.
Catchphrases:
"I am Iron Man."
2. "Genius, billionaire, playboy, philanthropis.."
3. "We have Hulk."
4. "Part of journey the cnd.'
Figure 5: The character profile of Iron Man from The Avengers.
Profile $ (Li Bai)
Brief Introduction:
* (70) 38-762 F.
tb:4; 1-L-TTVE11IRLAAAAA AA AN MHE S.
Hifft Mal 1i. Cu 50:0 "*"""
RE HR.
Personality:
1. 1838 (i HH #1 AEDEY 1111NNTEEE
#8 A:T &
4.
15.
13
Life Story:
-+ U5. HAHAEXA. 4
FAS
Interpersonal Relationships:
Catekphruses:
2. * H2.
3.
*"**** G
Figure 6: The character profile of Bai from Tang Dynasty of China.
E EXAMPLES OF THE TWO-STAGE GENERATION PROCESS FOR HYPOTHETICAL REAL-LIFE CHARACTERS
Figure and Figure 8 present the prompts used to generate meta information and expand detailed profiles for hypothetical real-life characters, whereas Figure eexempifiis meta information for five randomly selected hypothetical real-life characters.
15

---

Published as a conference paper at ICLR 2025
Prompt Used to Generate Meta Information for Hypothetical Real-Life Characters
Query: Plense generate the identity information for 20 hypotherical characters English. The requirements are as follows:
Brief Introductio:: Include basic informationn such as name, gender, age, and place of origin. Names should be diverse and culturally appropriate, an equal distribution of males fomales. Ages should range from children (above 5 years old) elderly (below years old).
Personality: Each character should have unique personality traits, such as optimism, introversion, bravery, curiosity, arrogance, etc. Character Experiences: Include but not limited professional background, significant life events, hobbies, and interests. Ensure variety of expericnees, such some characters might have gone adventurous travels, some might have achiovements in technology. and some might be artists, etc, Ensure these character profiles cover many different situations as possible to reflect diversity and complexity of suciety.
Figure 7: The prompt used to generate meta information for hypothetical real-life characters.
Prompt Used to Expand Detailed Profiles for Hypothetical Real-I Life Characters
Query: The following the abstract information about [role_ name). a person who is one of the Hypottettical Ch [meta, infa /n'm
Please help me expand this person's information. ensuring clarity and specificity under reasonable circumstances. The profile should comprchensively cover the following aspects'n Brief Intmshuction'
level overview the person, including their basic information, main characteristics, interests and
uchievements.t 'n
Porsenality' Delle into the person's personality, detailing both their strengths and weaknesses with specificity and vividness. Describe their behavior in varying situations, their approach to problem-solvingg and their interpersonal interaction style. Consider including aspects such as their lemperament, motivations, feurs, and how they cupe with challenges, in Life Story'aProvide PPrrvdde an in-depth narrative of the person's 's life. emphasizing signiflcant events, milestones, and experieaces that have shaped their development and worldview. This be structured chronologically organized around pivotal moments. Main Interpersonal RelationshipssaExammiii
the person's most significant relationships, such as those with family members, friends, colleagues, or adversaries. Provide the names of these individuals (where uppropriate) and describe the nature of each relationship, person's life and development.'n
including its impact
Catehphrases List several catehphrases, quotes, or common expressions frequently used the person. These should capture their personality, life philosophy, and their unique manner of speaking, 'n'n Please ensure your expansion detailed, coherent, adheres to the given structure.
Figure 8: The prompt used to expand detailed profiles for hypothetical real-life characters.
Examples of Meta Information for Hypothetical Real-Life Characters
Name: Sufiu Rodriguez
Gender: Female
Age: 47
Origin: Ruenos Aires, Argentina
Personality: Passionate and fiery
Experience: A tango instructor who has won several dance competitions, doeply involved her community's 's cultural ovents.
Name: Takumi Nakamura
Gender: Male
Age: 22
Origin: Kyoto, Japan
Porsonality: Curious and meticulous
Experience: A recent university gradiate in rohotics, aspiring to create robots that can assist in disuster recovery efferts.
Name: Henry O'Malley
Gender: Male
Age: 74
Origin: Dublin, Ineland
Personality: Jovial and steryteller
Experience: Retired firefighter, spends his time writing children's books based on his adventures, an avid bird watcher.
Name: Luca Biunchi
Gender: Male
Ago: 12
Origin: Rome, Italy
Personality: Energetic and outgoing
Experience: A middle school student and budding soccer star in his school team, enjoys playing the piano.
Name: Fatima Zahra
Gender; Female
Age: 38
Origin: Casablanca, Moroeco
Personality: Reflective and artistic
Experience: A poet and painter, her work explores themes of identity and belonging, organizes local art workshops for women.
Figure 9: The examples of meta information for hypothetical real-life characters.
16

---

Published as a conference paper at ICLR 2025
Prompt Used Query MRPAS
System: You dedicated role-playing assistant designed to immerse yourself fully the character are portmyyngg.
Query: Pleuse step into the shoes (role name] from [role series). Imagine you are tulking with a curious human about the given image. requires deep understanding relatiooships. inin'n
character's background, including their personality, experiences, abilities, and
The description frole name) is as follows: (role taininn
auxiliary information
image follows:'n
Character infocmation: (role info)'n Place information: (place info in Seene information: (scone_ info Jananin
The conversation history between (role name] and the curious human as follows:'n
[bumun]: (question in frole mamn]]: fresponse In
Please respond to the following words of the curious buman about the image in English using the distinctive tonc, manner and vocabulary (role name) i'n (question)
Figure 10:
The prompt used to query MRPAs in human-role dialogues involving English fictional characters, as well as historical and public figures.
F MANUAL QUALITY CONTROL STRATEGIES
For character profiles, we remove Al-assistant tones and unnecessary explanatory phrases, and refer- reliable sources such as brainyquote.com to enhance the authenticity of catchphrases. Further- more, human experts familiar with the characters further refine these profiles to ensure alignment with the characters' personalities and storylines.
For dialogues, we remove failed response data, as well as non-Chinese and non-English data. Ad- ditionally, we eliminate content that replies in the tone of an AI assistant, meaningless modal words frequently output by GPT-4, action and scene descriptions, and unnecessary explanatory prefixes and suffixes.
G PROMPTS FOR QUERYING MRPAS
Figure T0 details the prompt used query MRPAs in human-role dialogues involving English fic- tional characters, as well as historical and public figures. The prompts for Chinese characters and hypothetical real-life characters are similar the ones provided here.
H PROMPTS FOR GPT-4 SCORING AND REWARD MODEL SCORING
Figure II illlstraats the prompt used to score MRPAs by GPT-4 in human-role dialogues involving fictional characters, as well as historical and public figures. The prompts for hypothetical real-life characters are similar to the ones provided here.
Figure details the prompt used to score MRPAs for Personality Consistency by the reward model human-role dialogues involving fictional characters, as well as historical public figures. The prompts for other metrics and hypothetical real-life characters are similar to the ones provided here.
PROMPTS FOR DIALOGUE GENERATION
presents the prompts used to generate dialogues for the three types of scenarios involy- ing English fictional characters, as well as historical and public figures. The prompts for Chinese characters and hypothetical real-life characters are similar to the ones provided here.
Figure 13
J RMSE AND PEARSON RESULTS THE REWARD MODEL
As presented in Table and Table 10 correlation coefficient (Pearson) results.
report the root squared error (RMSE) and Pearson
The overall RMSEs for Reward Model (GPT-4), GPT-4 (humans), and Reward Model (humans) all relatively low, and those for GPT-4 (humans) and Reward Model (humans) are comparable, which similar to the MAE results. Notably, the RMSE values are slightly higher than the MAE values,
17

---

Published as a conference paper at ICLR 2025
Prompt Used Score MRPAs by GPT-4
System: You are an objective and precise evaluator, specializing in rigorously assessing the role-playing and multimodal understanding
abilities various models.
Query: #e [Question Start]]innn (question] 'n'nét [Question Endjinnnin
No [Model A's Response Stari Snin levalinatdd wnwwwrrwww 'ninni" [Model A's Response Endjaaran
*h [Model Response Starfjunn (groundirtt answer] inlafe [Model I's Response Endjaninin
ge [InstructionJnnn
The task instruction of the two medels is to directly colc-play as frole_name) from frole paccccssss and talk with a curious human about given image using distinctive tone, manner and vocabulary (role name). 'win
Please evaluate the following aspects euch model's responsecin
1. Instruction Adiercacc:: Do the responses accurately adhere to the task instruction, directly role-playing as mooe__ and only including words frole namn) should say, without any additional explanatory prefixes or suffixes?'n
2. Fluency: Are the responses grammatically correct and smoothly articulated"
3. Coherency: Do the responses maintain coherent thread of dialogae without contradicting earlier parts of the conversation or previoudl established facts?"n
4. amage-Text Relevance, Are the responses clesely related to the visual content of the image?"n
S. Response Accuracy: Do the responses accurately answer the curious human's words or appropriately initiate a conversation based on image?m
6. Personality Consistency: Do the responses accurately and sulficiently reflect the personality of (role name]]]ea 7
7. Knowledge Consistency: Are the responses consistent with the factual knowledge that [role_name) should possess, including experiences, sbilities, and relationships""
8. Tone Consistency: Do the responses maintain consistent tone that aligns with (role namel's typical manner of speaking and catchphrases, rather than resembling the style of assistants?'minn
two models, followed by paired quantitative scores from 10, where indicates poor performance and 10 indicates exeellent performance, lnin
each aspect. provide brief qualitative evaluation
relative performance
The outpui should be in the following format:'n
1. Instruction Adherenec: (Qualitative Evaluntion). [Scores]: (1the score of Model
score of Model B;yin
2. Fliency: (Qualitative Evaluation), [Scores]: (1the score of Vioel A), [the score of Model Bj)on etc//ii
Please casure that your ovaluntions are unbiased and that order which the responses were presented docs not affect your judgment.
Figure 11: The prompt used to score MRPAs by GPT-4 in human-role dialogues involving fictional characters, as well as historical and public figures.
Prompt Used to Score MRPAs for Personality Consistency by the Reward Model System: are an objective and precise evaluator, specializing rigorously assessing role role-playing multimodal understanding abilities of various models.
Query: #e [Question Start]inin (oartlainfquustionn (question)'ninee inln# [Question Endjun'nnn
#o [Model Response StartJunn Maarruinneevvluni fevalunted answer] [Model A's Response Endjannn'n
#e [Model B's Response StartJunnn (groundrttt answer]'n [Model B's Response End]in'nnn th [Instruttoo]riio
rooleseries) and talk with a curious about the given image using the distinctive tonc, manner and vocabulary of (role name). 'win
The task instruction of the models directly sole-play (pole [ile_
Please evaluate the following aspect of each model's response:'n
Personality Consistency: De the responses uccurately and sulficiently reflect the pensunality of (sole name/ Please provide a brief qualitative ovaluation for the relative performance of the two models, followed paired quantitative scores from to 10. where indicales poor performance and 10 indicates excellent performance. in'm
The output should the following formati'n
(Qualitative Evaluntion), [Scores]: ( (tte score of Model A), [the score of Model R|yin
Please ensure that your evaluntions are unbiased and that the order in which the responses were presented does not affect your judgment.
Figure 12: The prompt used to score MRPAs for Personality Consistency by the reward model in human-role dialogues involving fictional characters, as well as historical and public figures.
Table The root mean squared error (RMSE) results. 'Reward Model (GPT-4)' denotes the scores evaluated by the reward model compared to those evaluated by GPT-4 'GPT-4 (humans)" and 'Reward Model (humans)' signify the score gaps provided by GPT-4 and the reward model compared to the ground-trute -ruth score gaps provided by humans.
Evaluators (Ground Truth)
IA
Flu
Coh
ITR
RA
PC
KC
TC
Overall
Reward Model (GPT-4)
0.1585
0.1076
0.1228
0.1334
0.1145
0.1564
0.1172
0.1778
0.1381
GPT-4 (humans)
0.1794
0.1421
0.1050
0.1253
0,1837
0.1826
0.1515
0.1946
0.1609
Reward Model (humans)
0.1356
0.1107
0.1465
0.1731
0.1810
0.2057
0.1793
0.2010
0.1695
indicating some variability in the accuracy of both our reward model and GPT-4 across different test samples and evaluation metrics. This variability is expected, the scoring difficulty varies across
18

---

Published as a conference paper at ICLR 2025
Prompt Used Generate Commentary Interactions
System: You dedicated role-playing
designed immnrrse yourself fully the character you are portraying.
Query: Plense into the shoes of frole name) from (role series). This requires deep understanding of the charncter's background, including their personality, experiences, abilities, and relationships. inim'n description frole name)
follows:'n'n
Life (eoryln(expericnee))mn Inin Main Interpersonal Relationshipe'n [relationship in'n Catchphrasesin (cutchphrrssc 'ninnn
Brief Introduction'a /nrnnnnbbbirtt (brief intro)'nii nnin ianoonalltyy//pesooollll
auxiliary information
image follows:'n
Character information: (role info in Place information: (place_ info 'n- Seene information: (scene_ info Ja'nnn
Please respond and answer the following question about the image in English using the distinctive tone, manner and vocabulary of frole name); :'n (question)
Prompt Used to Generate Human-Role Dialogues
System: You are a role playing dialogue generation assistant.
Query: Imagine scene where curious buman travols through time and space watch the given image alongside (rolc name; from frole scries]. This requires deep understanding of character's background, including their personality, experiences, abilities, and relationships. inln'n
The description frole name) is as follows:'n'n
Brief Introduction'n (brief /n'o iessonalityya/poesonoo
Life Storyin|ecperrccee///n Main Interpersonal
Relationshipsin (relationship) 'n'n Catchphrasesin (rennessi(ccaccphhiin [catchphrase) 'iniii
The auxiliary information of the image is as follows:'n
Character information: (role info/'n (n- Place information: (place info//n- Scene information: (seene in'on'a'aia
Please generate a multi-rum dialogue between the curious human and (role name), and the dialogue generate must always revolve around the given image. The curious humam initiate the convensation first, raising questions about the image, and then talk human using distinctive tone, manuer and vocabulary of (role name). The
with frole name]. frole name) should respond dialogue should be engaging and immersive. The dialogue format is:'n[human]: trole frole name)
Prompt Used to Generate Inter-Role Dialogues
System:
You are a role playing dialogue generation assistant.
Query: Imagine scene where fother rolo name) and frole name) from frolo_scricn series) are watching the given image together. This requires deep understanding the character's background, including their personality, experiences, abilities, and relationships 'niannn The description of fother rolc name) follows'n'n
Brief Introduction'n fother brief intro] in'n Personality'' (other, personality] Innnn Life Shory'nfother experience) 'nin Main Interpersonal Relationships'n (other relationship) Catchphrases (other catchphrase) innin
The description frolc name) is as follows:'n'n
Brief Introduction's (brief intro]'nt inin iessooalitylnppersonnn
Life Storr/nneexprrrencee/nn Main Interporsonal
Relationships'n (relationship)
Catchphrases'n (catchphrase)
The suxiliary information of the image is follows:'n
Character information: frole info /n Place information: (place info/'n Seene information: (soene info]'n'ain
Please generate a multi-tum dialogue between fother name] (role name], and dialogue you generate must always revolve around the given image. (other pole name) initiate conversation first, raising questions commentinng about the image. and then talk with frolc name). Both (other role name) and frole name) should talk using the distinctive tonc, manner and vocabulary themselves. The dialogue should he engaging and immersive. The dialogue format (other, role [role namej]Fm....
Figure 13: The prompts used to generate dialogues for the three types of scenarios involving English fictional characters, as well as historical and public figures.
Table 10: The Pearson correlation coefficient (Pearson) results. 'Reward Model (GPT-4)' denotes the scores evaluated by the reward model compared those evaluated by GPT-4. 'GPT-4 (humans)' and 'Reward Model (humans)' signify the score gaps provided by GPT-4 and the reward model compared the ground-truth score gaps provided by humans.
Evaluators (Ground Truth)
IA
Flu
Coh
ITR
RA
PC
KC
TC
Overall
Reward Model (GPT-4)
0.7497
0.7344
0.7610
0.7955
0.8186
0.8167
0.8237
0.8129
0.8129
GPT-4 (humans)
0.6130
0.6736
0.9199
0.8184
0.7247
0.6997
0.7924
0.6985
0.7269
Reward Model (humans)
0.6561
0.3123
0.8033
0.8709
0.7321
0.7268
0.5832
0.5443
0.6502
samples and metrics; for example, assessing personality consistency is significantly more complex than evaluating fluency.
The overall Pearson values for Reward Model (GPT-4), GPT-4 (humans), and Reward Model (hu- mans) are all relatively high, indicating strong positive correlations among them. While the overall Pearson values for Reward Model (humans) slightly lower than those for GPT-4 (humans),
19

---

Published as a conference paper at ICLR 2025
performs well in metrics like Image-Text Relevance (0.8709), Response Accuracy (0.7321), and Personality Consistency (0.7268).
In summary, the combination of the MAE, RMSE, and Pearson results collectively demonstrates that our reward model effectively learns the evaluation abilitics of GPT-4 and closely aligns with human evaluators.
Baeeeea foarr my an Mele medii Awphe
vilrria deala
de
alos bes
cree ofter lus
(a) Conmentary Interation
(b) Hattas Role Diálegue
(c) Lases Stle Disiogue
Figure 14: The use cases of MMRole-Agent on the In- Test set.
dhaily fraenie vole
igert
Commentrry Interctioon
(b) llammn-Roi Dalegaa:
Inno-itole Diliagaz
Figure 15: The use cases of MMRole-Agent on the Out-Test set.
K CASE STUDIES
18 and 19 present the cases of our MMRole-Agent, GPT-4, and QWen- VL-Chat on both the In- Test and the Out-Test sets. Our observations indicate that both GPT-4 and MMRole-Agent perform strongly in multimodal role-playing, whereas -W-n-- primarily functions as an Al assistant and struggles to adhere to role-playing instructions in inter-role dialogue scenarios. Moreover, we analyze the characteristics of MMRole-Agent from the following aspects: Fundamental Conversational Skills: MMRole-Agent consistently fulfills the role-playing task by adhering closely to given instructions. Its outputs are not only fluent and coherent but also highly contextually appropriate.
Figures 14 15
16
2. Multimodal Understanding Abilities: MMRole-Agent produces outputs that maintain high rele- vance to visual inputs and effectively interpret image-based clues, even in complex multi-turn dialogues. Relevant examples are highlighted in purple and bold in the figures.
3. Role-Playing Qualities: MMRole-Agent effectively embodies the specified personality, tone, and experiences of its designated characters, showcasing distinctive speech patterns and ways of thinking. Relevant examples are highlighted in red and bold in the figures.
20

---

Published as a conference paper at ICLR 2025
Mhow The cnlit
withi
cor du dlnig
(a) Corerneatayy Interaction
(b) Hamaas Role Diálegue
(c) laser Stle Dislogue
Figure 16: The cases of GPT- on the In-Test set.
mepis anebi aoe tae ridocc sedeer otferr
conon plust dify
bone ond coldee pnd
Commentary Interscion
(b) Barma Bole Dillgga:
(c) Inser-itole Disingaz
Figure 17: The use cases of GPT-4 on the Out-Test set.
the
ond tre
Commentan
Dillgge
Inno--kale Disiogge
Figure 18: The use cases of -L-en- on the In-Test set.
21

---

Published as a conference paper at ICLR 2025
(a) Conmentary lnterciinn
(b) Hattaz- Role Dillegue
lanes Sole Dislogue
Figure 19: The of WL-Chtt the Out-Test set.
Table 11: The sensitivity test results for MMRole-Agent on different prompts.
Original Prompts
Modified Prompts
Overall
You dedicated role-playing assistant de- signed to immerse yourself fully the character you are portraying.
You highly skilled role-playing assistant, committed to fully immersing yourself the char- acter you embody.
0.995
Please step into the shoes (role name) from {role_series). Imagine you are talking with curi- ous human about the given image. This requires a deep understanding of the character's background, including their personality, experiences, abilities, and relationships.
Imagine you are (role name) from (role series). talking with curious human about the given im- age. Draw on the character's background, includ- ing their personality, experiences, abilities, and re- lationships.
0.996
L SENSITIVITY TESTS FOR MMRole-Agent ON DIFFERENT PROMPTS
We conduct sensitivity tests on MMRole-Agent using different prompt templates. As shown in Ta- ble II we independently modify the system part and the character-designatin part of the prompts. The performance of MMRole-Agent with these modified prompts remains nearly identical to that achieved with the original prompts (0.994). This indicates that MMRole-Agent is highly compatible with different prompt templates and does not exhibit signs overfitting.
22


Reviews:
[{"review_id": "my9kvf4yU1", "reviewer": "Reviewer_teaD", "summary": "This paper proposes a multimodal role-playing agent data collection and training framework. The authors use a wide range of images with different prompts to prompt GPT for image-text role-playing data, and fine-tune a QWen-VL-Chat model on the dataset after some automatic filtering.", "questions": "* The abstract and introduction highlights the \"specialized MRPA\" idea. Do we know much improvement comes from the specialized reward model vs. no specialized reward model?\n* Do the authors have any insight on the results generated by a finetuned MM role playing model? What works, what doesn't work, and what works better/worse than just prompting gpt?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Simple, straightforward method that clearly works well given the model size and achieves the desired outcome.\n* Create a specialized Multimodal Role-Playing Agent is a novel idea.\n* Experiments demonstrate good performance given the finetuned model size.\n* Comprehensive evaluation.", "weaknesses": "* The major technical contribution seems to come from the MM roles dataset collection process. However, there does not seem to be much data curation beyond automated filtering.\n* Analysis seems to be mostly numbers and high-level results, with little technical/detailed insight.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rGVrsQVTD4", "reviewer": "Reviewer_LPtY", "summary": "The paper presents a new dataset and evaluation framework for multimodal role-playing agents. They present several complementary evaluation metrics.\nThe authors evaluate several recent general purpose multimodal LLMs within this framework. In addition they evaluate a specialized model fine-tuned on their dataset.", "questions": "no questions", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to follow.\nThe evaluation framework is highly relevant and potentially very impactful. The evaluation metrics are meaningful and the SOTA evaluation itself is comprehensive, providing a relevant set of baselines for future users of the dataset/framework.", "weaknesses": "I am not sure whether I could fully follow the approach to evaluation.\nImo it would be important to run a (at least limited) evaluation with human participants scoring the output. Building models that automatically evaluate outputs seems to be a circular approach.\n\nFurthermore, evaluting the MAE to compare between different evaluators might not adequately model differences between evaluators that are not visible in MAE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YWUi4szZ3N", "reviewer": "Reviewer_DsRF", "summary": "This paper introduces the concept of Multimodal Role-Playing Agents (MRPAs), develops a multimodal dataset (MMRole-Data) and evaluation framework (MMRole-Eval), and creates a specialized MRPA model, MMRole-Agent, achieving improved multimodal understanding and role consistency.", "questions": "Why are only 320 samples used as the validation set out of 23,520 samples, with the remainder used for training?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper constructs a complete multimodal dataset (MMRole-Data) and evaluation framework (MMRole-Eval).\n2. Testing across multiple LMMs lends credibility to the experimental results.", "weaknesses": "1.Overreliance on GPT-4 for Evaluation: While MMRole-Eval provides a stable evaluation mechanism, it heavily relies on GPT-4, introducing a degree of bias. The authors validated the MAE between GPT-4 (humans) and Reward Model (humans), demonstrating consistency between the reward model and GPT-4. However, comparing two MRPAs to see which performs better does not substitute for genuine human judgment on the quality of MRPA responses. While this reward model may help MMRole-Agent approach GPT-4’s performance, its potential to surpass GPT-4 or elevate MRPAs to human-level capabilities remains debatable.\n2.Lack of Performance Comparison with Single-Modality RPAs: Although the concept of MRPAs is appealing, the absence of specific experimental comparisons makes it difficult to understand exactly where MRPAs improve upon performance or accomplish tasks that single-modality RPAs cannot achieve.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yYQ2jt19lG", "reviewer": "Reviewer_Jj4g", "summary": "This paper introduces a concept of Multimodal Role-Playing Agents (MRPAs), expanding traditional role-playing agents to tackle multimodal interactions. The paper introduce a framework (MMRole) including MMRole-Data and MMRole-Eval. The MMRole-Data is a large-scale, high-quality dataset with 85 characters, 11,000+ images, and 14,000 dialogues. The MMRole-Eval is a robust evaluation method with eight metrics across three dimensions: conversational skills, multimodal understanding, and role-playing qualities.", "questions": "1. Could you provide specific cases to analyze MMRole-Agent’s performance under In-Test and Out-Test conditions? \n\n2. Could you explain the \"rigorous manual quality control\" process in character profile generation？\n\n3. Has the sensitivity of MMRole-Agent to different prompt templates been tested?\n\n4. Could you discuss the primary limitations of MMRole-Agent, especially in terms of challenges encountered in practical applications and possible directions for future improvements?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper first introduces the concept of Multimodal Role-Playing Agents (MRPAs), extending traditional role-playing agents to the multimodal domain, filling a gap in existing research.\n\n2. The MMRole framework includes both the construction of a dataset (MMRole-Data) and the design of an evaluation method (MMRole-Eval), covering eight metrics across dimensions such as fundamental conversational skills, multimodal understanding, and role-playing quality.\n\n3. The proposed MMRole-Agent demonstrates strong performance.\n\n4. The writing is good and easy to understand.", "weaknesses": "1. The paper lacks case studies, which could help illustrate MMRole-Agent's performance across diverse roles and dialogue scenarios.\n\n2. The paper mentions that character profiles undergo \"rigorous manual quality control,\" it does not provide detailed quality control standards or processes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)