MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents

Authors: Yanqi Dai, Huanran Hu, Lei Wang, Shengjie Jin, Xu Chen, Zhiwu Lu
Primary area: datasets and benchmarks
Keywords: Multimodal Role-Playing Agents, Large Multimodal Models

Abstract:
Recently, Role-Playing Agents (RPAs) have garnered increasing attention for their potential to deliver emotional value and facilitate sociological research.
However, existing studies are primarily confined to the textual modality, unable to simulate humans' multimodal perceptual capabilities.
To bridge this gap, we introduce the concept of Multimodal Role-Playing Agents (MRPAs), and propose a comprehensive framework, MMRole, for their development and evaluation, which comprises a personalized multimodal dataset and a robust evaluation approach.
Specifically, we construct a large-scale, high-quality dataset, MMRole-Data, consisting of 85 characters, 11K images, and 14K single or multi-turn dialogues.
Additionally, we present a robust evaluation approach, MMRole-Eval, encompassing eight metrics across three dimensions, where a reward model is designed to score MRPAs with the constructed ground-truth data for comparison.
Moreover, we develop the first specialized MRPA, MMRole-Agent.
Extensive evaluation results demonstrate the improved performance of MMRole-Agent and highlight the primary challenges in developing MRPAs, emphasizing the need for enhanced multimodal understanding and role-playing consistency.
The data, code, and models are all available at https://github.com/YanqiDai/MMRole.

Decision: accept (poster)
Reviewer scores: 5, 8, 5, 8

Reviews and discussion:
<thread>
creation date:
1730041459202

thread_type:
official_review

<thread_object>
creator: reviewer_jj4g

<content>
summary:
This paper introduces a concept of Multimodal Role-Playing Agents (MRPAs), expanding traditional role-playing agents to tackle multimodal interactions. The paper introduce a framework (MMRole) including MMRole-Data and MMRole-Eval. The MMRole-Data is a large-scale, high-quality dataset with 85 characters, 11,000+ images, and 14,000 dialogues. The MMRole-Eval is a robust evaluation method with eight metrics across three dimensions: conversational skills, multimodal understanding, and role-playing qualities.

strengths:
1. This paper first introduces the concept of Multimodal Role-Playing Agents (MRPAs), extending traditional role-playing agents to the multimodal domain, filling a gap in existing research.

2. The MMRole framework includes both the construction of a dataset (MMRole-Data) and the design of an evaluation method (MMRole-Eval), covering eight metrics across dimensions such as fundamental conversational skills, multimodal understanding, and role-playing quality.

3. The proposed MMRole-Agent demonstrates strong performance.

4. The writing is good and easy to understand.

weaknesses:
1. The paper lacks case studies, which could help illustrate MMRole-Agent's performance across diverse roles and dialogue scenarios.

2. The paper mentions that character profiles undergo "rigorous manual quality control," it does not provide detailed quality control standards or processes.

questions:
1. Could you provide specific cases to analyze MMRole-Agent’s performance under In-Test and Out-Test conditions? 

2. Could you explain the "rigorous manual quality control" process in character profile generation？

3. Has the sensitivity of MMRole-Agent to different prompt templates been tested?

4. Could you discuss the primary limitations of MMRole-Agent, especially in terms of challenges encountered in practical applications and possible directions for future improvements?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696087144

thread_type:
official_review

<thread_object>
creator: reviewer_dsrf

<content>
summary:
This paper introduces the concept of Multimodal Role-Playing Agents (MRPAs), develops a multimodal dataset (MMRole-Data) and evaluation framework (MMRole-Eval), and creates a specialized MRPA model, MMRole-Agent, achieving improved multimodal understanding and role consistency.

strengths:
1. The paper constructs a complete multimodal dataset (MMRole-Data) and evaluation framework (MMRole-Eval).
2. Testing across multiple LMMs lends credibility to the experimental results.

weaknesses:
1.Overreliance on GPT-4 for Evaluation: While MMRole-Eval provides a stable evaluation mechanism, it heavily relies on GPT-4, introducing a degree of bias. The authors validated the MAE between GPT-4 (humans) and Reward Model (humans), demonstrating consistency between the reward model and GPT-4. However, comparing two MRPAs to see which performs better does not substitute for genuine human judgment on the quality of MRPA responses. While this reward model may help MMRole-Agent approach GPT-4’s performance, its potential to surpass GPT-4 or elevate MRPAs to human-level capabilities remains debatable.
2.Lack of Performance Comparison with Single-Modality RPAs: Although the concept of MRPAs is appealing, the absence of specific experimental comparisons makes it difficult to understand exactly where MRPAs improve upon performance or accomplish tasks that single-modality RPAs cannot achieve.

questions:
Why are only 320 samples used as the validation set out of 23,520 samples, with the remainder used for training?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721405595

thread_type:
official_review

<thread_object>
creator: reviewer_lpty

<content>
summary:
The paper presents a new dataset and evaluation framework for multimodal role-playing agents. They present several complementary evaluation metrics.
The authors evaluate several recent general purpose multimodal LLMs within this framework. In addition they evaluate a specialized model fine-tuned on their dataset.

strengths:
The paper is well-written and easy to follow.
The evaluation framework is highly relevant and potentially very impactful. The evaluation metrics are meaningful and the SOTA evaluation itself is comprehensive, providing a relevant set of baselines for future users of the dataset/framework.

weaknesses:
I am not sure whether I could fully follow the approach to evaluation.
Imo it would be important to run a (at least limited) evaluation with human participants scoring the output. Building models that automatically evaluate outputs seems to be a circular approach.

Furthermore, evaluting the MAE to compare between different evaluators might not adequately model differences between evaluators that are not visible in MAE.

questions:
no questions
</content>
</thread_object>

</thread>
<thread>
creation date:
1731191492006

thread_type:
official_review

<thread_object>
creator: reviewer_tead

<content>
summary:
This paper proposes a multimodal role-playing agent data collection and training framework. The authors use a wide range of images with different prompts to prompt GPT for image-text role-playing data, and fine-tune a QWen-VL-Chat model on the dataset after some automatic filtering.

strengths:
* Simple, straightforward method that clearly works well given the model size and achieves the desired outcome.
* Create a specialized Multimodal Role-Playing Agent is a novel idea.
* Experiments demonstrate good performance given the finetuned model size.
* Comprehensive evaluation.

weaknesses:
* The major technical contribution seems to come from the MM roles dataset collection process. However, there does not seem to be much data curation beyond automated filtering.
* Analysis seems to be mostly numbers and high-level results, with little technical/detailed insight.

questions:
* The abstract and introduction highlights the "specialized MRPA" idea. Do we know much improvement comes from the specialized reward model vs. no specialized reward model?
* Do the authors have any insight on the results generated by a finetuned MM role playing model? What works, what doesn't work, and what works better/worse than just prompting gpt?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732068155028

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response by Authors

comment:
We sincerely appreciate all reviewers for their time and effort in reviewing our paper. We are pleased that the reviewers broadly acknowledged the contributions of our work:
- **Novelty.** The introduction of Multimodal Role-Playing Agents (MRPAs) extends traditional role-playing agents to the multimodal domain, which is a novel idea and fills a gap in existing research. [teaD, Jj4g]
- **Framework.** The paper constructs a complete multimodal dataset (MMRole-Data) and evaluation framework (MMRole-Eval), which is meaningful and potentially very impactful. [teaD, LPtY, DsRF, Jj4g]
- **Experiments.** Evaluation across multiple SOTA LMMs is comprehensive, providing a relevant set of baselines for future users of the dataset/framework. [LPtY, DsRF]
- **Performance.** The proposed MMRole-Agent demonstrates strong performance. [teaD, Jj4g]
- **Writing.** The paper is well-written and easy to follow. [LPtY, Jj4g]

We also thank all reviewers' insightful and constructive feedback, which has been invaluable in further improving our paper. Below, we summarize the additional experimental results included in the rebuttal based on the reviewers' suggestions:
- Performance comparison of our specialized reward model vs. no-specialized reward model QWen-VL-Chat. [teaD]
- Additional metric results for evaluating the reward model, including root mean square error (RMSE) and Pearson correlation coefficient. [LPtY, DsRF]
- Performance evaluation of a new reward model using more data for validation. [DsRF]
- Performance comparison of MRPAs vs. single-modality RPAs. [DsRF]
- Ablation studies of MMRole-Agent on the amount of training data, the number of training characters, and the training strategies (freezing vs. finetuning ViT). [teaD]
- Sensitivity tests of MMRole-Agent under different prompt templates. [Jj4g]
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces the concept of Multimodal Role-Playing Agents (MRPAs), expanding traditional role-playing agents to tackle multimodal interactions. The paper introduce a framework with datasets and evaluation metrics for these multimodal role-playing agents. This includes a large-scale, high-quality dataset with 85 characters, 11,000+ images, and 14,000 dialogues, and eight evaluation metrics across three dimensions: conversational skills, multimodal understanding, and role-playing qualities.

After the discussion period this paper received mixed reviews, 2 marginal reject and 2 accept. The reviewers generally found the newly proposed setting of multimodal role-playing agents interesting and novel, and appreciated the effort gone into creating the new dataset and developing new evaluation metrics. They also found the experiments and analysis comprehensive.

For the 2 reviewers who voted reject but did not respond, I went through the discussions and feel that the authors have addressed them reasonably (see details below), so I advocate for acceptance.