MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation
Authors: Haochen Xue, Feilong Tang, Ming Hu, Ye xin Liu, Qidong Huang, Yulong Li, Chengzhi Liu, Zhongxing Xu, Chong Zhang, Chun-Mei Feng, Yutong Xie, Imran Razzak, Zongyuan Ge, Jionglong Su, Junjun He, Yu Qiao
Accepted at: acl25

MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation

Haochen Xue1,2,4*, Feilong Tang2,3,4*, Ming Hu2,3*, Yexin Liu5, Qidong Huang2,6, Yulong Li1,2,4 Chengzhi Liu1, Zhongxing Xu3, Chong Zhang1, Chun-Mei Feng7, Yutong Xie4 Imran Razzak1†, Zongyuan Ge3†, Jionglong Su1†, Junjun He3†, Yu Qiao2
1 Xi’an Jiaotong-Liverpool University, 2 Shanghai Artificial Intelligence Laboratory 3 Monash University, 4 MBZUAI, 5 HKUST, 6 USTC, 7 IHPC, A*STAR

Abstract

Recent multimodal large language models (MLLMs) have demonstrated significant potential in open-ended conversation, generating more accurate and personalized responses. However, their abilities to memorize, recall, and reason in sustained interactions within real-world scenarios remain underexplored. This paper introduces MMRC, a Multi-Modal Real-world Conversation benchmark for evaluating six core open-ended abilities of MLLMs: information extraction, multi-turn reasoning, information update, image management, memory recall, and answer refusal. With data collected from real-world scenarios, MMRC comprises 5,120 conversations and 28,720 corresponding manually labeled questions, posing a significant challenge to existing MLLMs. Evaluations on 22 MLLMs in MMRC indicate an accuracy drop during open-ended interactions. We identify four common failure patterns: long-term memory degradation, inadequacies in updating factual knowledge, accumulated assumption of error propagation, and reluctance to “say no”. To mitigate these issues, we propose a simple yet effective NOTE-TAKING strategy, which can record key information from the conversation and remind the model during its responses, enhancing conversational capabilities. Experiments across six MLLMs demonstrate significant performance improvements.

1 Introduction

Open-ended conversations (OEC) are the most common form of interaction between humans and Multimodal Large Language Models (MLLMs), representing a crucial feature of Artificial General Intelligence (AGI) (Kil et al., 2024; Fei et al., 2024). These conversations are entirely determined by the user’s intention, rather than by system rules or predefined patterns (Decker, 2022; Zheng et al., 2023a; Liu et al., 2024c). Furthermore, individual differences shape each conversation’s distinct linguistic style and user-specific preferences (Chaves et al., 2022; Ma et al., 2024; Tan et al., 2025).

However, existing conversation benchmarks (Bai et al., 2024; Liu et al., 2024d; Xu et al., 2023; Liu et al., 2024c) fail to comprehensively evaluate MLLMs’ abilities to memorize, recall, and reason in sustained interactions in OEC. These benchmarks exhibit two primary limitations: (i) their reliance on prompt templates for dataset generation limits the diversity of data domains and conversation lengths, leading to repetitive and overly structured dialogues that fail to reflect the complexities of long-term user-AI interactions. (ii) they only cover a limited subset of the memory capabilities required to leverage dynamic, ever-changing, and accumulative information in long-term interactions, failing to evaluate the ability to recall multimodal information or reason with updated information.

To address these limitations, we develop DialogFlow, a free online dialogue platform featuring 22 cutting-edge MLLMs to collect diverse, real-world conversation data. Through DialogFlow, we construct MMRC, the first Multi-Modal Real-

![Illustration of the six core multimodal open-ended conversation abilities in the MMRC benchmark.](page_1012_370_420_312.png)

Figure 1: Illustration of the six core multimodal open-ended conversation abilities in the MMRC benchmark.
<table>
  <tr>
    <th>Dataset</th>
    <th>Dialog</th>
    <th>A-Turns</th>
    <th>Img</th>
    <th>Multi-Img</th>
    <th>Domains</th>
    <th>Gen-Method</th>
    <th>Temp-Type</th>
    <th colspan="5">Core Conversation Abilities</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>IE</th>
    <th>CR</th>
    <th>IU</th>
    <th>IM</th>
    <th>MR</th>
    <th>AR</th>
  </tr>
  <tr>
    <td>MT-Bench (Zheng et al., 2023b)</td>
    <td>80</td>
    <td>2.0</td>
    <td>x</td>
    <td>x</td>
    <td>8</td>
    <td>GPT-4</td>
    <td>Fixed</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>MT-Bench-101 (Bai et al., 2024)</td>
    <td>1388</td>
    <td>3.1</td>
    <td>x</td>
    <td>x</td>
    <td>13</td>
    <td>GPT-4o</td>
    <td>Fixed</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <td>LongMemEval (Wu et al., 2024)</td>
    <td>500</td>
    <td>6.1</td>
    <td>x</td>
    <td>x</td>
    <td>164</td>
    <td>GPT-4</td>
    <td>Fixed</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <td>DialogBench (Ou et al., 2023)</td>
    <td>9811</td>
    <td>7.6</td>
    <td>x</td>
    <td>x</td>
    <td>36</td>
    <td>GPT-4</td>
    <td>Fixed</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Farm (Xu et al., 2023)</td>
    <td>1952</td>
    <td>7.4</td>
    <td>x</td>
    <td>x</td>
    <td>5</td>
    <td>GPT-4o</td>
    <td>Fixed</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <td>MT-Bench ++ (Sun et al., 2024)</td>
    <td>-</td>
    <td>8</td>
    <td>x</td>
    <td>x</td>
    <td>13</td>
    <td>GPT-4o</td>
    <td>Fixed</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <td>MMR (Liu et al., 2024d)</td>
    <td>84</td>
    <td>2.3</td>
    <td>x</td>
    <td>x</td>
    <td>12</td>
    <td>GPT-4</td>
    <td>Fixed</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <td>EvalDial (Park et al., 2024)</td>
    <td>500</td>
    <td>2.7</td>
    <td>x</td>
    <td>x</td>
    <td>96</td>
    <td>GPT-4</td>
    <td>Fixed</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <td>ConvBench (Liu et al., 2024c)</td>
    <td>577</td>
    <td>3.0</td>
    <td>x</td>
    <td>x</td>
    <td>197</td>
    <td>GPT-4o</td>
    <td>Fixed</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <td>MMDU Benchmark (Liu et al., 2024e)</td>
    <td>110</td>
    <td>15</td>
    <td>x</td>
    <td>x</td>
    <td>219</td>
    <td>GPT-4o</td>
    <td>Fixed</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <td>MMRC (Ours)</td>
    <td>5120</td>
    <td>15.2</td>
    <td>✓</td>
    <td>✓</td>
    <td>874</td>
    <td>Real-conv</td>
    <td>Open-ended</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
</table>

Table 1: The comparison between MMRC with other conversation benchmarks. Dialog: the number of dialogs; A-Turns: average turns per conversation; Img: support for single image input; Multi-Img: support for multiple images input; Domains: the number of covered domains; Gen-Method: generation method; Temp-Type: dialogue template type; Finally, we the coverage of six core abilities: information extraction (IE), cross-turn reasoning (CR), information update (IU), image management (IM), memory recall (MR), answer refusal (AR).

world Conversation benchmark, which includes 5,120 carefully selected dialogues. Drawing on cognitive studies of human conversation (Clark et al., 2019; Liddicoat, 2021), an evaluation framework consisting of 28,720 manually annotated questions is designed to assess MLLMs’ six core abilities in OEC, as illustrated in Fig. 1: information extraction, cross-turn reasoning, information update, image management, long-term memory recall, and answer refusal. To achieve this, multiple evaluation metrics, including GPT-based evaluations, human evaluation, and objective precision metrics, are employed to ensure a comprehensive and objective assessment.

We extensively evaluate 22 mainstream MLLMs and observe that they, including advanced GPT-4o (Islam and Moushi, 2024), fail to consistently deliver accurate and reliable responses over extended interactions. Furthermore, certain open-source MLLMs demonstrate a limited capacity for long-term conversations in real scenarios. Through analyzing models’ error answers, we identify four common failure patterns: (1) Long-term memory degradation, where MLLMs’ memory of facts from earlier conversations becomes vague, leading to inconsistent responses with prior turns. (2) Inadequacy in updating factual knowledge, where MLLMs exhibit a failure to integrate new facts effectively, still continuing to rely on outdated information. (3) Accumulated assumption of error propagation, where erroneous assumptions made while integrating information from earlier turns propagate into later turns, leading to an interrupted reasoning chain. (4) Reluctance to “say no”, where MLLMs show an inability to decline to provide an answer in OEC when the context is insufficient.

To mitigate this, we propose a simple yet effective strategy called NOTE-TAKING. This strategy systematically stores key user preferences and facts throughout the conversation. When the model is tasked with evaluation queries, the recorded information is transformed into structured prompts, providing supplementary context to improve the accuracy and coherence of the MLLMs’ responses. Experiment results across six MLLMs demonstrated that this strategy significantly enhances the models’ overall conversational capabilities.

In summary, our contributions are four-fold: (1) We introduce the first multi-modal open-ended conversation (OEC) benchmark MMRC, providing a comprehensive evaluation of MLLMs’ performance in practical settings. (2) We propose six core abilities of the model in OEC, covering broader aspects than existing benchmarks. (3) Using our evaluation framework, we analyze 22 state-of-the-art MLLMs and identify four failure patterns in OEC, providing insights to inspire future research. (4) We propose NOTE-TAKING, which improves conversational capabilities by storing key user preferences and facts and using structured prompts to assist MLLMs in generating responses.

2 Related Work

Multimodal Large Language Model. Building on large language models, multimodal large language models (MLLMs) have exhibited remarkable capabilities (Kil et al., 2024; Cui et al., 2024; Qin et al., 2025), achieving state-of-the-art performance across various downstream tasks, including visual grounding (Li et al., 2024c; Xu et al., 2024b), object detection (Zang et al., 2024; Wu et al., 2025), visual question answering (VQA) (Kuang et al., 2024; Xu et al., 2024a; DE), and instruction following (Li et al., 2023; Sun et al., 2024; Wei et al., 2024). Their outstanding performance underscores their pivotal role in AGI (Zhang et al., 2024).
Figure 2: A sample from the MMRC, featuring a multi-turn open-ended conversation with six human-annotated questions and answers, designed to assess the ability of MLLMs in open-ended conversations.

Benchmarks for Long-Term Conversation. MT-Bench (Zheng et al., 2023b) is a pioneering two-turn dialogue dataset generated by GPT, covering eight domain tasks. MT-Bench-101 (Bai et al., 2024) and Bench++ (Sun et al., 2024) expand the dataset size and add more domains, enhancing evaluation depth. In parallel, Farm (Xu et al., 2023), EvalDial (Park et al., 2024), and MMR (Liu et al., 2024d) examine model robustness in multi-turn dialogue scenarios using fixed dialogue formats. Con-vBench (Liu et al., 2024c) evaluates models’ perception, reasoning, and creation abilities through structured three-turn dialogues, exploring their interrelations. DialogBench (Ou et al., 2023) and LongMemEval (Wu et al., 2024) focus on evaluating models’ abilities in context understanding and memory retention during GPT-generated dialogues. MMDU (Liu et al., 2024e) evaluates the understanding and instruction-following abilities in GPT-generated multi-image, multi-turn dialogues. Table 1 compares MMRC with previous works, highlighting its advantages in: (1) naturally open dialogue format with longer and more diverse conversations. (2) holistically covering critical abilities in memorization, recall, and reasoning in a uniquely challenging way (further examples in Fig. 2).

3 The MMRC

3.1 Problem Formulation

The evaluation of MMRC requires a triplet instance \((S, q, a)\), where \(S\) represents the dialogue history, \(q\) is a set of evaluation questions assessing specific conversational abilities, and \(a\) is the ground truth answers. Specifically, \(S = \{(t_i, R_i)\}_{i=1}^n\) denotes an \(n\)-turn dialogue history, where \(t_i = (\text{text}_i, \text{image}_i)\) represents the user query with text, images, or both, and \(R_i\) is the model’s response at turn \(i\). With our MMRC setup, given the dialogue context \(S\), the model is tasked to answer a set of six evaluation questions, \(q = \{q_i\}_{i=1}^T\), where \(T = 6\), each designed to assess a specific ability. The model’s responses, denoted as \(p = \{p_i\}_{i=1}^T\), are then individually compared against human-annotated ground truth answers, \(a = \{a_i\}_{i=1}^T\), to evaluate its performance in OEC.

We summarize the memorization, recall, and reasoning abilities required by MLLMs in OEC, as illustrated in Fig. 2, with details as follows.

Information Extraction (IE): Ability to retrieve specific information from the conversation history, which includes both textual and visual content.

Cross-turn Reasoning (CR): Ability to comprehend and integrate information across multiple dialogue turns to answer complex questions.

Information Update (IU): Ability to track and update knowledge dynamically by recognizing changes in user information and factual updates.

Image Management (IM): Ability to store and manage visual information by retaining specific image details and maintaining accurate attribution.

Memory Recall (MR): Ability to maintain and retrieve memory of previous interactions throughout the conversation and recall user-specific details.

Answer Refusal (AR): Ability to refrain from answering questions that involve unknown information, i.e., absent from the interaction history.
Figure 3: Data construction pipeline of MMRC.

Figure 4: The distribution of dialogue turns in MMRC, ConvBench, and EvalDial.

3.2 Data Curation Process
We develop DialogFlow, a large-scale evaluation platform designed to benchmark 22 cutting-edge MLLMs, with specific models detailed in Appendix C. In particular, open-source models are deployed on A100 GPUs, while closed-source models are accessed via APIs. Over an 8-month period, we collected 87,912 raw dialogues from 354 users, leveraging thousands of A100 GPU hours and incurring significant API costs. However, these raw conversations may contain sensitive information, including personal details, violent content, offensive language, biased statements, misinformation, and culturally inappropriate expressions, posing fairness and ethical concerns.

To address this, we design a pipeline to clean the data, as illustrated in Fig. 3. The three stages are as follows: (i) We manually review data for personal information or privacy violations. If any are detected, the relevant segments are deleted, or the entire dialogue is removed if its coherence is compromised. (ii) We also screen for violent, offensive, and sensitive content. If detected, the entire dialogue is removed to prevent the dissemination of harmful material. (iii) We manually annotate clean dialogue data with QA pairs for MLLM evaluation in OEC. These pairs undergo multiple reviews by different annotators to ensure accuracy.

3.3 Data Statistics
We perform a statistical analysis on the distribution of conversation turns, categorization, and questions. The detailed distribution of conversation turns is demonstrated in Fig. 4. The conversation turns in MMRC are not fixed, ranging from 4 to 22, making it more natural and realistic compared to the fixed-turn structures in ConvBench (Liu et al., 2024c) and EVALDIAL (Park et al., 2024).

To classify diverse real-world conversations, we design a classification network that maps conversation data into 14 predefined categories (details in the Appendix E). The classification results, shown in Fig. 5, indicate that MMRC exhibit a well-balanced distribution. Moreover, these categories cover a wide range of topics, ensuring the diversity and representativeness of the conversations.

The statistics on manually annotated questions in MMRC are shown in Table 2. Notably, the number of questions for the six core abilities is well-balanced, with the majority being open-ended questions. Although open-ended questions complicate model evaluation, they provide a finer-grained view of the differences in model responses, enabling a deeper understanding of model performance.

<table>
  <tr>
    <th>Statistic</th>
    <th>Number</th>
    <th>Percentage</th>
  </tr>
  <tr>
    <td>Total questions</td>
    <td>28720</td>
    <td>100%</td>
  </tr>
  <tr>
    <td>- Information Extraction (IE)</td>
    <td>5087</td>
    <td>17.71 %</td>
  </tr>
  <tr>
    <td>- Cross-turn Reasoning (CR)</td>
    <td>4789</td>
    <td>16.67 %</td>
  </tr>
  <tr>
    <td>- Information Update (IU)</td>
    <td>4561</td>
    <td>15.88 %</td>
  </tr>
  <tr>
    <td>- Image Management (IM)</td>
    <td>4721</td>
    <td>16.43 %</td>
  </tr>
  <tr>
    <td>- Memory Recal (MR)</td>
    <td>4962</td>
    <td>17.28 %</td>
  </tr>
  <tr>
    <td>- Answer Refusal (AR)</td>
    <td>4600</td>
    <td>16.02 %</td>
  </tr>
  <tr>
    <th colspan="3">Formats:</th>
  </tr>
  <tr>
    <td>- Open Questions</td>
    <td>24716</td>
    <td>86.05 %</td>
  </tr>
  <tr>
    <td>- Multiple-Choice Questions</td>
    <td>2703</td>
    <td>9.41 %</td>
  </tr>
  <tr>
    <td>- True/False Questions</td>
    <td>1301</td>
    <td>4.53 %</td>
  </tr>
</table>

Table 2: Problem statistics of MMRC.

![Pie chart showing the distribution of conversation categories in our MMRC dataset](page_1012_1042_388_312.png)

Figure 5: The distribution of conversation categories in our MMRC dataset.
<table>
  <tr>
    <th rowspan="2">Type</th>
    <th rowspan="2">Model</th>
    <th colspan="6">GPT-based Evaluation Metrics</th>
    <th colspan="3">Human Evaluation Metrics</th>
    <th colspan="4">Objective Precision Metrics</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>IE</th>
    <th>CR</th>
    <th>IU</th>
    <th>IM</th>
    <th>MR</th>
    <th>AR</th>
    <th>CR*</th>
    <th>IU*</th>
    <th>MR*</th>
    <th>EP</th>
    <th>IMP</th>
    <th>RP</th>
  </tr>
  <tr>
    <td rowspan="18">Human</td>
    <td>LLaVA-V1.5-7B</td>
    <td>4.79</td><td>4.78</td><td>4.82</td><td>4.87</td><td>4.77</td><td>4.80</td><td>4.81</td><td>4.79</td><td>4.66</td><td>0.974</td><td>0.981</td><td>93.26%</td><td>4.81</td>
  </tr>
  <tr>
    <td>LLaVA-V1.5-13B</td>
    <td>0.91</td><td>1.08</td><td>0.31</td><td>0.52</td><td>0.22</td><td>0.28</td><td>1.14</td><td>0.47</td><td>0.29</td><td>0.167</td><td>0.092</td><td>8.26%</td><td>0.55</td>
  </tr>
  <tr>
    <td>MiniCPM-SB</td>
    <td>1.04</td><td>1.21</td><td>0.25</td><td>0.92</td><td>0.26</td><td>0.15</td><td>1.32</td><td>0.42</td><td>0.33</td><td>0.206</td><td>0.193</td><td>6.14%</td><td>0.64</td>
  </tr>
  <tr>
    <td>MiniCPM-MB</td>
    <td>4.08</td><td>3.94</td><td>2.98</td><td>3.47</td><td>3.65</td><td>3.78</td><td>4.08</td><td>2.91</td><td>3.79</td><td>0.748</td><td>0.661</td><td>79.23%</td><td>3.65</td>
  </tr>
  <tr>
    <td>LLaVA-Next-0.5B</td>
    <td>3.32</td><td>2.89</td><td>1.99</td><td>1.87</td><td>2.67</td><td>1.12</td><td>2.63</td><td>1.11</td><td>2.73</td><td>0.446</td><td>0.358</td><td>20.58%</td><td>2.14</td>
  </tr>
  <tr>
    <td>LLaVA-Next-7B</td>
    <td>3.23</td><td>3.85</td><td>2.77</td><td>2.18</td><td>4.01</td><td>2.08</td><td>2.92</td><td>2.65</td><td>4.02</td><td>0.611</td><td>0.355</td><td>36.21%</td><td>3.02</td>
  </tr>
  <tr>
    <td>Owen2VL-2B</td>
    <td>2.16</td><td>2.85</td><td>1.41</td><td>1.27</td><td>1.93</td><td>2.33</td><td>2.71</td><td>1.62</td><td>2.23</td><td>0.336</td><td>0.269</td><td>40.29%</td><td>1.99</td>
  </tr>
  <tr>
    <td>Owen2VL-7B</td>
    <td>2.82</td><td>3.68</td><td>2.62</td><td>2.16</td><td>2.11</td><td>1.57</td><td>3.72</td><td>2.77</td><td>2.19</td><td>0.531</td><td>0.408</td><td>30.62%</td><td>2.49</td>
  </tr>
  <tr>
    <td>Owen2VL-72B</td>
    <td>3.17</td><td>4.17</td><td>2.78</td><td>2.73</td><td>2.81</td><td>1.32</td><td>4.15</td><td>2.81</td><td>2.89</td><td>0.603</td><td>0.524</td><td>25.41%</td><td>2.83</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision-0.5B</td>
    <td>2.03</td><td>3.01</td><td>2.44</td><td>1.79</td><td>2.55</td><td>1.92</td><td>3.11</td><td>2.40</td><td>2.57</td><td>0.305</td><td>0.436</td><td>36.85%</td><td>2.29</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision-7B</td>
    <td>3.52</td><td>3.71</td><td>3.21</td><td>2.29</td><td>3.35</td><td>2.93</td><td>3.82</td><td>3.29</td><td>3.33</td><td>0.653</td><td>0.439</td><td>66.23%</td><td>3.16</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision-72B</td>
    <td>4.06</td><td>4.08</td><td>4.01</td><td>3.24</td><td>4.17</td><td>2.52</td><td>4.11</td><td>3.89</td><td>3.99</td><td>0.723</td><td>0.677</td><td>54.28%</td><td>3.68</td>
  </tr>
  <tr>
    <td>VILA-1.5-3B</td>
    <td>3.08</td><td>2.79</td><td>2.87</td><td>2.91</td><td>3.42</td><td>2.23</td><td>2.83</td><td>3.08</td><td>3.30</td><td>0.566</td><td>0.502</td><td>47.97%</td><td>2.88</td>
  </tr>
  <tr>
    <td>VILA-1.5-8B</td>
    <td>3.46</td><td>3.45</td><td>3.12</td><td>2.84</td><td>3.66</td><td>2.74</td><td>3.61</td><td>3.34</td><td>3.67</td><td>0.659</td><td>0.542</td><td>59.21%</td><td>3.22</td>
  </tr>
  <tr>
    <td>mplug-Ow3-1B</td>
    <td>2.48</td><td>2.76</td><td>1.92</td><td>2.53</td><td>2.78</td><td>1.51</td><td>2.61</td><td>2.08</td><td>2.77</td><td>0.503</td><td>0.547</td><td>32.27%</td><td>2.33</td>
  </tr>
  <tr>
    <td>mplug-Ow3-2B</td>
    <td>3.45</td><td>2.91</td><td>2.36</td><td>2.71</td><td>2.42</td><td>2.09</td><td>2.99</td><td>2.41</td><td>2.40</td><td>0.522</td><td>0.610</td><td>42.14%</td><td>2.66</td>
  </tr>
  <tr>
    <td>mplug-Ow3-7B</td>
    <td>3.92</td><td>3.89</td><td>2.59</td><td>3.34</td><td>2.83</td><td>2.91</td><td>3.95</td><td>2.78</td><td>2.81</td><td>0.702</td><td>0.672</td><td>62.49%</td><td>3.25</td>
  </tr>
  <tr>
    <td>InternVL2.5-8B</td>
    <td>3.90</td><td>3.82</td><td>3.14</td><td>3.16</td><td>3.72</td><td>3.23</td><td>3.78</td><td>3.18</td><td>3.69</td><td>0.704</td><td>0.628</td><td>71.45%</td><td>3.24</td>
  </tr>
  <tr>
    <td>InternVL2.5-26B</td>
    <td>4.10</td><td>3.92</td><td>3.38</td><td>3.27</td><td>4.08</td><td>3.05</td><td>3.98</td><td>3.34</td><td>3.96</td><td>0.751</td><td>0.644</td><td>67.34%</td><td>3.63</td>
  </tr>
  <tr>
    <td>Avg.</td>
    <td>2.98</td><td>3.22</td><td>2.45</td><td>2.40</td><td>2.81</td><td>2.09</td><td>3.19</td><td>2.53</td><td>2.83</td><td>0.540</td><td>0.475</td><td>44.05%</td><td>2.65</td>
  </tr>
  <tr>
    <td rowspan="5">Closed source</td>
    <td>GPT-4o</td>
    <td>4.35</td><td>4.38</td><td>4.28</td><td>4.12</td><td>4.31</td><td>3.06</td><td>4.26</td><td>4.16</td><td>4.18</td><td>0.905</td><td>0.826</td><td>68.28%</td><td>4.08</td>
  </tr>
  <tr>
    <td>Claude-3.5-sonnet</td>
    <td>4.12</td><td>4.04</td><td>3.98</td><td>3.89</td><td>3.88</td><td>3.21</td><td>4.09</td><td>3.92</td><td>0.823</td><td>0.786</td><td>74.28%</td><td>3.86</td>
  </tr>
  <tr>
    <td>Gemini-1.5 Pro</td>
    <td>3.96</td><td>3.90</td><td>3.75</td><td>3.94</td><td>4.16</td><td>3.11</td><td>3.92</td><td>4.10</td><td>0.716</td><td>0.794</td><td>73.03%</td><td>3.80</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>3.92</td><td>3.96</td><td>3.92</td><td>3.98</td><td>4.03</td><td>3.28</td><td>4.01</td><td>3.91</td><td>3.99</td><td>0.702</td><td>0.798</td><td>74.93%</td><td>3.84</td>
  </tr>
  <tr>
    <td>Avg.</td>
    <td>4.09</td><td>4.07</td><td>3.98</td><td>3.99</td><td>4.10</td><td>3.17</td><td>4.07</td><td>4.01</td><td>4.04</td><td>0.787</td><td>0.801</td><td>72.63%</td><td>3.90</td>
  </tr>
</table>

Table 3: Comparison of Performance for 22 MLLMs on MMRC. * indicates that the same task has been re-evaluated manually. Bold and underline denote the best and second-best results, respectively.

![Radar chart of capabilities for models with noticeable task-specific imbalances.](page_101_1012_627_312.png)

Figure 6: Radar chart of capabilities for models with noticeable task-specific imbalances.

4 Experiment and Analysis

4.1 Evaluation Matrix

Since questions in MMRC are open-ended, directly evaluating accuracy is infeasible. To address this, we develop a comprehensive evaluation framework that integrates GPT-based scoring, human assessment, and objective precision metrics. Specifically, GPT evaluates all six abilities (Section 3.1), while human evaluators conduct a second-round review for CR, IU, and MR, which require more in-depth judgment. Moreover, both GPT and human evaluators employ a scoring scale ranging from 0 to 5, with prompts and evaluation criteria detailed in the Appendix J. In contrast, for IE, IM, and AR, we employ objective precision metrics, including Extraction Precision (EP), Image Management Precision (IMP), and Refuse Precision (RP), to provide an intuitive assessment of model performance by measuring the proportion of correct responses.

**EP as an extension of IE, measures the precision in extracting items:**

\[
\text{EP} = \frac{|\text{Model}_{\text{items}} \cap \text{Label}_{\text{items}}|}{|\text{Model}_{\text{items}}|}
\]

where \( \text{Model}_{\text{items}} \) denotes the set of items generated by the model, and \( \text{Label}_{\text{items}} \) denotes the set of ground-truth items.

**IMP as an extension of IM, measures the precision in managing and retrieving images:**

\[
\text{IMP} = \frac{|\text{Image}_{\text{hit}}|}{|\text{Image}_{\text{hit}} \cup \text{Image}_{\text{miss}}|}
\]

where \( \text{Image}_{\text{hit}} \) denotes the images correctly retrieved by the model, and \( \text{Image}_{\text{miss}} \) denotes the images that are part of the correct answer but were not retrieved by the model.

**RP as an extension of AR, measures the precision in refusing to answer unknown questions:**

\[
\text{RP} = \frac{\sum_{i=1}^N \mathbb{1}(D_i)}{\sum_{i=1}^N \mathbb{1}(E_i)},
\]

where \( D_i \) and \( E_i \) denote the model’s refusal to answer and the ground-truth refusal for the \( i \)-th question, respectively. The indicator function \( \mathbb{1}(\cdot) \)
returns 1 if the response aligns with the expected refusal behavior and 0 otherwise. \( N \) denotes the total number of questions.

4.2 Main Results
Table 3 presents the performance of 22 open-source and closed-source MLLMs in real-world dialogue scenarios. Based on the evaluation results, we identify three key findings:
(1) Challenges of Reality: LLaVA-1.5 performs poorly in OEC, while powerful GPT-4o still falls short of human-level performance in real-world scenarios, highlighting the complexity and difficulty of practical, user-driven conversations in MMRC.
(2) Reliability of Evaluation: The similarity between GPT’s scores and those of human annotators reaches 93%. Furthermore, GPT’s scores closely correspond to the Objective Precision Metrics, where higher GPT scores consistently reflect stronger performance. This consistency indicates the effectiveness of GPT guided by our evaluation prompts, providing a foundation for using GPT scores in subsequent experimental analysis.
(3) Task-Specific Imbalance: The performance of most models is imbalanced, exhibiting distinct strengths and weaknesses. As illustrated in Fig. 6, the LLaVA-Next family demonstrates strong memory recall ability but weaker image management ability. Similarly, the Qwen2VL family excels in cross-turn reasoning yet exhibits a relatively weak answer refusal ability. Notably, closed-source models exhibit a more pronounced performance imbalance, consistently struggling with answer refusal. This disparity in MLLMs stems from variations in training datasets and strategies. Specifically, different organizations prioritize fine-tuning for certain tasks, leading to enhanced performance in those areas while resulting in weaker performance on tasks with less targeted training.

4.3 Error Analysis
We conduct an in-depth analysis of failures of the model and identify four common error patterns:

(1) Long-term memory degradation: As the conversation progresses, the model’s memory of previous dialogue content becomes increasingly vague, a phenomenon known as memory degradation (Zhong et al., 2024). Moreover, the severity of memory degradation increases as the conversation lengthens. As illustrated in Fig. 7, memory-related abilities (\emph{i.e.}, IE, IM, MR) decline significantly in extended conversations. Observations reveal that memory degradation is more severe in the middle of a conversation than at the beginning or end, challenging the assumption that earlier memories degrade more rapidly. To further investigate, we visualize the model’s attention patterns when addressing memory-related questions (Appendix K). As shown in Fig. 8, attention to the middle part of the conversation is markedly lower than to the beginning and end, mirroring the observed degradation pattern. We hypothesize that this unbalanced attention distribution contributes to the heightened memory degradation in mid-conversation.

Furthermore, we analyze the error types associated with long-term memory degradation, with results shown in Fig. 10: (\emph{i}) \emph{Memory omission}: the model fails to retain certain details from the conversation. (\emph{ii}) \emph{Complete forgetting}: a more severe form of memory degradation than omission, where the model entirely fails to recall specific events from the conversation. (\emph{iii}) \emph{Memory confusion}: the model incorrectly merges relevant information with unrelated content, leading to distorted recollection.

(2) Inadequacies in updating factual knowledge: MLLMs often struggle to track changes in user information and factual knowledge during conversations, resulting in failures in updating information. As illustrated in Fig. 9, frequent changes in factual knowledge make it difficult to adapt to rapidly evolving information and result in a decline in update performance. To further investigate this issue, we analyze the types of errors associated with updating information, with their distribution shown in Fig. 11: (\emph{i}) \emph{Failure to recognize updates}:
Figure 10: Information extraction error statistics.

Figure 11: Information update error statistics.

Figure 12: Statistics of error types in reasoning.

Figure 13: Statistics of error types in answer refusal.

Figure 14: Overall and answer refusal performance across different model sizes within the same family.

Figure 15: MLLMs’ performance in different topics.

occurs when the model fails to detect that certain factual knowledge requires updating, instead treating it as static information. (ii) Incomplete updates: arises when the model acknowledges the need for an update but fails to incorporate the most recent information due to frequent changes. (iii) Old-new contradiction: happens when the model incorrectly merges outdated facts with new ones, leading to an inaccurate representation of the latest information.

(3) Accumulated assumption of error propagation: During reasoning, the model processes information sequentially from earlier to later dialogue turns to comprehend the dialogue and integrate relevant details, forming a reasoning chain to answer complex questions. However, as illustrated in Fig. 12, the formation of this reasoning chain is often hindered by three key issues, leading to flawed reasoning and incorrect answers: (i) Misunderstanding: the model fails to correctly understand the dialogue content, resulting in distortions within the reasoning chain and incorrect assumptions that ultimately lead to erroneous conclusions. (ii) Memory degradation: the model forgets prior conversation information, disrupting the reasoning chain. This lack of essential context weakens the model’s assumptions, preventing accurate reasoning. (iii) Failure to update: the model continues reasoning based on outdated knowledge, leading to incorrect answers. Notably, these errors can occur simultaneously. As the conversation progresses, their accumulation exacerbates incorrect assumptions, causing errors to propagate more severely throughout the dialogue.

(4) Reluctance to “say no”: The model provides unreliable answers when the context is insufficient, potentially misleading users. To understand the underlying causes of this issue, we conduct an analysis and categorize them into three key types, as illustrated in Fig. 13. (i) Forced Responses: the model recognizes that the question is unrelated to the given context and does not utilize any conversational context for its response, yet it fails to refuse to answer. (ii) Over-reasoning: the model excessively analyzes the question, attempting to infer non-existent details from the available context. (iii) Force associations: the model artificially links an irrelevant element in the question with existing conversation details, generating an answer based on this false connection. Furthermore, as shown in Fig. 14, within the same model family, larger models demonstrate superior overall performance. However, their ability to refuse inappropriate answers declines. A comparative analysis with varying sizes reveals that while larger models exhibit stronger logical reasoning capabilities, they are more prone to over-reasoning and forced associations compared to smaller models.

4.4 Further Discussion

Domain Bias: To further investigate the model’s performance across different domains, we evaluate its overall effectiveness in each domain, as shown in Fig. 15. Our analysis reveals that the model performs significantly better in professional
1) Extract key Information

Hi, I'm looking for meal suggestions. I really like Italian Food.
I've been feeling off lately, checking my blood pressure. I've also been diagnosed with an allergy to dairy products.
• User want to ask meal suggestions,
• User preference: Italian cuisine,
• User has health conditions: high blood pressure and a dairy allergy.

2) Structural Prompt Helps response

Json-format Prompt
{
  "user_info": {
    "health_conditions": ["High blood pressure", "dairy allergy"]
  },
  "preferences": {
    "food_preference": "Italian cuisine",
    "purpose": "Seeking meal suggestions"
  }
}

Give meal suggestions based on user condition.

Gluten-free pasta with tomato-based sauce, vegetable lasagna(dairy-free)

Figure 16: Illustration of NOTE-TAKING method.

knowledge conversations than in daily conversations (professional: 3.23, daily: 2.61). We hypothesize that this disparity stems from variations in the instruction-based training across models. Specifically, models demonstrating stronger performance in professional knowledge conversations benefit from a larger proportion of instruction-based data tailored for knowledge-based tasks. Thus, to improve MLLMs’ conversational abilities in OEC, it is essential to incorporate more daily conversation data during supervised fine-tuning.

Modalities Preference: To explore the model’s preference for different information modalities, we modify 100 conversations by replacing parts of the original text content with equivalent image inputs. For instance, the text-based statement: “I visited the Eiffel Tower.” is converted into “I visited this place.” followed by an image of the Eiffel Tower. The rest of the dialogue remains unchanged for evaluation. Our findings indicate that MLLMs exhibit a strong preference for text-based information, with overall scores for text-based dialogues being 26.3% higher than their image-based counterparts. Furthermore, models exhibit fewer memory degradation errors in text-based conversations, as memory-related capabilities such as IE, IM, and MR show a 34.6% improvement. We attribute this to two main factors: (i) images often require more tokens to convey the same meaning as text, significantly increasing context length. (ii) The model’s training data is imbalanced, with text data vastly exceeding image data, leading to stronger proficiency in processing textual information.

5 NOTE-TAKING as Improved Baseline

In this section, we introduce an initial step toward enhancing the core capabilities of MLLMs in OEC. The primary failure of MLLMs is their inability to accurately retrieve detailed information, update knowledge, and recognize missing information. To mitigate this, we propose a NOTE-TAKING framework, which guides MLLMs in extracting key dialogue information and recording it in accessible json-format notes. These structured notes serve as external memory, improving response accuracy and context understanding.

As illustrated in Fig. 16, the NOTE-TAKING effectively simulates how humans take notes during long and complex conversations, facilitating the retention of key details and maintaining focus. As shown in Table 4, the improvement is observed in the long-term memory ability, with MR increasing by an average of +0.97. Furthermore, the note-taking mechanism enhances the model’s information extraction capability by +0.88, and improves information update by +0.84. Moreover, the structured clarity provided by the notes allows the model to concentrate more effectively on relevant details within the conversation, resulting in a +0.52 improvement in answer refusal.

<table>
  <tr>
    <th>Models</th>
    <th>IE</th>
    <th>IU</th>
    <th>MR</th>
    <th>AR</th>
  </tr>
  <tr>
    <td>LLaVA-1.5-7B<br>+ NOTE-TAKING</td>
    <td>0.91 (+1.06)</td>
    <td>0.31 (+0.15)</td>
    <td>0.22 (+0.14)</td>
    <td>0.28 (+0.18)</td>
  </tr>
  <tr>
    <td>MiniCPM-8B<br>+ NOTE-TAKING</td>
    <td>4.08 (+0.15)</td>
    <td>2.98 (+0.78)</td>
    <td>3.65 (+0.37)</td>
    <td>3.78 (+0.14)</td>
  </tr>
  <tr>
    <td>QwenVL-2B<br>+ NOTE-TAKING</td>
    <td>2.16 (+1.35)</td>
    <td>1.41 (+0.82)</td>
    <td>1.93 (+1.47)</td>
    <td>2.33 (+0.51)</td>
  </tr>
  <tr>
    <td>LLaVA-Next-0.5B<br>+ NOTE-TAKING</td>
    <td>2.32 (+1.52)</td>
    <td>1.99 (+1.05)</td>
    <td>2.67 (+0.21)</td>
    <td>1.12 (+0.91)</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision-72B<br>+ NOTE-TAKING</td>
    <td>4.06 (+0.22)</td>
    <td>4.01 (+0.30)</td>
    <td>4.17 (+0.21)</td>
    <td>2.52 (+0.94)</td>
  </tr>
  <tr>
    <td>GPT-4o<br>+ NOTE-TAKING</td>
    <td>4.35 (+0.16)</td>
    <td>4.28 (+0.34)</td>
    <td>4.31 (+0.42)</td>
    <td>3.06 (+0.45)</td>
  </tr>
  <tr>
    <td>4.51 (+0.34)</td>
    <td>4.62 (+0.42)</td>
    <td>4.73 (+0.45)</td>
    <td>3.51 (+0.45)</td>
  </tr>
</table>

Table 4: Performance of NOTE-TAKING across four conversational core abilities in MMRC.

6 Conclusions

In this paper, we introduce MMRC, the first multi-image open-ended conversation benchmark to evaluate the six conversation abilities of MLLMs. Our comprehensive analysis identifies four common failure patterns: long-term memory degradation, inadequate updating of factual knowledge, accumulated assumption of error propagation, and reluctance to “say no.” To mitigate these, we propose the NOTE-TAKING strategy, which stores key user preferences and facts by using structured prompts.
Limitations: We clarify the limitations: (i) While MMRC covers multiple domains, it may not encompass all real-world dialogue types (e.g., population distribution and languages) and requires further exploration. (ii) Although the NOTE-TAKING improves model performance, the note generation process can be computationally intensive.

References

Dosovitskiy Alexey. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010.11929.

Atheer Alghairy and Moataz Ahmed. 2024. A review of dialogue systems: current trends and future directions. Neural Computing and Applications, 36(12):6325–6351.

Jerone Andrews, Dora Zhao, William Thong, Apostolos Modas, Orestis Papakyriakopoulos, and Alice Xiang. 2023. Ethical considerations for responsible data curation. Advances in Neural Information Processing Systems, 36:55320–55360.

Mela Aziza. 2021. A teacher questioning activity: The use of oral open-ended questions in mathematics classroom. Qualitative Research in Education, 10(1):31–61.

Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiasheng Liu, Zhanhui Zhou, Zuhoran Lin, Wenbo Su, Tiezhen Ge, Bo Zheng, et al. 2024. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. arXiv preprint arXiv:2402.14762.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966.

John Barnden. 2014. Open-ended elaborations in creative metaphor. In Computational creativity research: Towards creative machines, pages 217–242. Springer.

Zheng Cai, Maosong Cao, Haojing Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Interlm2 technical report. arXiv preprint arXiv:2403.17297.

Ana Paula Chaves, Jesse Egbert, Toby Hocking, Eck Doerr, and Marco Aurelio Gerosa. 2022. Chatbots language design: The influence of language variation on user experience with tourist assistant chatbots. ACM Transactions on Computer-Human Interaction, 29(2):1–38.

Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. In Forty-first International Conference on Machine Learning.

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jingou Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. 2024. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271.

Leigh Clark, Nadia Pantidi, Orla Cooney, Philip Doyle, Diego Garaiadile, Justin Edwards, Brendan Spillane, Eimer Gillmartin, Christine Murad, Cosmin Munteanu, et al. 2019. What makes a good conversation? challenges in designing truly conversational agents. In Proceedings of the 2019 CHI conference on human factors in computing systems, pages 1–12.

Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. 2024. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 958–979.

SMALL VISUAL DE. Mllms know where to look: Training-free perception of small visual de-tails with multimodal llms.

Amandine Decker. 2022. Topic Shifts: Preserving Comprehension in Conversation. Ph.D. thesis, Université de lorraine; IDMC.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Hillary Anger Elfenbein, Petri Laukka, Jean Althoff, Wanda Chui, Frederick K Iraki, Thomas Rockstuhl, and Nutan Kumar S Thingujam. 2022. What do we hear in the voice? an open-ended judgment study of emotional speech prosody. Personality and Social Psychology Bulletin, 48(7):1087–1104.

Hao Fei, Xiangtai Li, Haotian Liu, Fuxiao Liu, Zhusheng Zhang, Hanwang Zhang, and Shuicheng Yan. 2024. From multimodal llm to human-level ai: Modality, instruction, reasoning and beyond. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 11289–11291.

Charles Fernyhough. 1996. The dialogic mind: A dialogic approach to the higher mental functions. New ideas in Psychology, 14(1):47–62.

Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, et al. 2024. Mme-survey: A comprehensive survey on evaluation of multimodal llms. arXiv preprint arXiv:2411.15296.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.

Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594.

Raisa Islam and Owana Marzia Moushi. 2024. Gpt-4o: The cutting-edge advancement in multimodal llm. Authorea Preprints.

Toshish Jawale, Chaitanya Animesh, Sekhar Vallath, Kartik Talamadupula, and Larry Heck. 2024. Are human conversations special? a large language model perspective. arXiv preprint arXiv:2403.05045.

Katarzyna Kapusta, Lucas Mattioli, Boussad Addad, and Mohammed Lansari. 2024. Protecting ownership rights of ml models using watermarking in the light of adversarial attacks. AI and Ethics, 4(1):95–103.

Jihyung Kil, Zheda Mai, Justin Lee, Arpita Chowdhury, Zihe Wang, Kerrie Cheng, Lemeng Wang, Ye Liu, and Wei-Lun Harry Chao. 2024. Mllm-compbench: A comparative reasoning benchmark for multimodal llms. Advances in Neural Information Processing Systems, 37:28798–28827.

Suyoun Kim and Florian Metze. 2018. Dialog-context aware end-to-end speech recognition. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 434–440. IEEE.

Jiaji Kuang, Ying Shen, Jingyou Xie, Haohao Luo, Zhe Xu, Ronghao Li, Yinghui Li, Xianfeng Cheng, Xika Lin, and Yu Han. 2024. Natural language understanding and inference with mllm in visual question answering: A survey. ACM Computing Surveys.

Paulette Lacroix. 2019. Big data privacy and ethical challenges. Big Data, Big Challenges: A Healthcare Perspective: Background, Issues, Solutions and Research Directions, pages 101–111.

Patrick Langer, Stephan Altmüller, Elgar Fleisch, and Filipe Barata. 2024. Claid: Closing the loop on ai & data collection—a cross-platform transparent computing middleware framework for smart edge-cloud and digital biomarker applications. Future Generation Computer Systems, 159:505–521.

Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhat-tacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. 2024a. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594.

Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024b. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895.

Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. 2023. Fine-tuning multimodal llms to follow zero-shot demonstrative instructions. In The Twelfth International Conference on Learning Representations.

Zhaowei Li, Qi Xu, Dong Zhang, Hang Song, Yiqing Cai, Qi Qi, Ran Zhou, Junting Pan, Zefeng Li, Vu Tu, et al. 2024c. Groundinggpt: Language enhanced multi-modal grounding model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6657–6678.

Anthony J Liddicoat. 2021. An introduction to conversation analysis. Bloomsbury Publishing.

Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. 2024. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26689–26699.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023. Improved baselines with visual instruction tuning.

Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, et al. 2024b. Mibench: Evaluating multimodal large language models over multiple images. arXiv preprint arXiv:2407.15272.

Shuo Liu, Kaining Ying, Hao Zhang, Yue Yang, Yuqi Lin, Tianle Zhang, Chuanhao Li, Yu Qiao, Ping Luo, Wensi Shao, et al. 2024c. Convenbench: A multi-turn conversation evaluation benchmark with hierarchical capability for large vision-language models. arXiv preprint arXiv:2403.20194.

Yexin Liu, Zhengyang Liang, Yueze Wang, Muyang He, Jian Li, and Bo Zhao. 2024d. Seeing clearly, answering incorrectly: A multimodal robustness benchmark for evaluating mlms on leading questions. arXiv preprint arXiv:2406.10638.

Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, et al. 2024e. Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvllms. arXiv preprint arXiv:2406.11833.

Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, et al. 2024. Consent in crisis: The rapid decline of the ai data commons. In NEURIPS.
Rachel Ma, Jingyi Qu, Andreea Bobu, and Dylan Hadfield-Menell. 2024. Goal inference from open-ended dialog. arXiv preprint arXiv:2410.13957.

Alex McKeown, Miranda Mourby, Paul Harrison, Sophie Walker, Mark Sheehan, and Ilina Singh. 2021. Ethical issues in consent for the reuse of data in health data platforms. Science and Engineering Ethics, 27:1–21.

Jiao Ou, Junda Lu, Che Liu, Yihong Tang, Fuzheng Zhang, Di Zhang, and Kun Gai. 2023. Dialogbench: Evaluating llms as human-like dialogue systems. arXiv preprint arXiv:2311.01677.

ST Padmapriya and Sudhaman Parthasarathy. 2024. Ethical data collection for medical image analysis: A structured approach. Asian Bioethics Review, 16(1):95–108.

Dragan Pamucar, Vladimir Simic, Ömer Faruk Görçün, and Hande Küçükonder. 2024. Selection of the best big data platform using cobrac-artasi methodology with adaptive standardized intervals. Expert Systems with Applications, 239:122312.

Yohanes Yohanie Fridelin Panduman, Nobuo Funabiki, Evianita Dewi Fajrianti, Shihao Fang, and Sritrusta Sukardihoto. 2024. A survey of ai techniques in iot applications with use case investigations in the smart environmental monitoring and analytics in real-time iot platform. Information, 15(3):153.

Dongmin Park, Zhaofang Qian, Guangxing Han, and Ser-Nam Lim. 2024. Mitigating dialogue hallucination for large multi-modal models via adversarial instruction tuning. arXiv preprint arXiv:2403.10492.

Irene V Pasquetto, Zoë Cullen, Andrea Thomer, and Morgan Wofford. 2024. What is research data “misuse”? and how can it be prevented or mitigated? Journal of the Association for Information Science and Technology, 75(12):1413–1429.

Amandalynne Paullada, Inioluwa Deborah Raji, Emily M Bender, Emily Denton, and Alex Hanna. 2021. Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns, 2(11).

Stefan Pfenninger, Joseph DeCarolis, Lion Hirth, Sylvain Quoilin, and Iain Staffell. 2017. The importance of open data and software: Is energy research lagging behind? Energy Policy, 101:211–215.

Guangyuan Piao. 2021. Scholarly text classification with sentence bert and entity embeddings. In Trends and Applications in Knowledge Discovery and Data Mining: PAKDD 2021 Workshops, WSPA, MLMEIN, SDPRA, DARAI, and AI4EPT, Delhi, India, May 11, 2021 Proceedings 25, pages 79–87. Springer.

Eduardo Pina, José Ramos, Henrique Jorge, Paulo Váz, José Silva, Cristina Wanzeller, Maryam Abbasi, and Pedro Martins. 2024. Data privacy and ethical considerations in database management. Journal of Cybersecurity and Privacy, 4(3):494–517.

Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and S Yu Philip. 2025. A survey of multilingual large language models. Patterns, 6(1).

Woosuk Seo, Chammo Yang, and Young-Ho Kim. 2024. Chacha: Leveraging large language models to prompt children to share their emotions about personal events. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–20.

Efrat Shimron, Jonathan I Tamir, Ke Wang, and Michael Lustig. 2022. Implicit data crimes: Machine learning bias arising from misuse of public data. Proceedings of the National Academy of Sciences, 119(13):e2117203119.

Chen Sun, Valerie J Shute, Angela EB Stewart, Quinton Beck-White, Caroline R Reinhardt, Guojing Zhou, Nicholas Duran, and Sidney K D’Mello. 2022. The relationship between collaborative problem solving behaviors and solution outcomes in a game-based learning environment. Computers in Human Behavior, 128:107120.

Yuchong Sun, Che Liu, Kun Zhou, Jinwen Huang, Ruihua Song, Wayne Xin Zhao, Fuzheng Zhang, Di Zhang, and Kun Gai. 2024. Parrot: Enhancing multi-turn instruction following for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9729–9750.

Donald E Super. 1982. The relative importance of work: Models and measures for meaningful data. The Counseling Psychologist, 10(4):95–103.

Zhaoxuan Tan, Zinan Zeng, Qingkai Zeng, Zhenyu Wu, Zheyuan Liu, Fengran Mo, and Meng Jiang. 2025. Can large language models understand preferences in personalized recommendation? arXiv preprint arXiv:2501.13391.

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

Chuang Wang, Xiaoyan Li, Wei Sun, Jingjing An, and Shufang Gao. 2024a. Occupant behavior, thermal environment, and appliance electricity use of a single-family apartment in china. Scientific Data, 11(1):65.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi-hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024b. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.
Jingyu Wei, Yi Su, Kele Xu, Lingbin Zeng, Bo Liu, and Huaimin Wang. 2024. Demonstrative instruction following in multimodal llms via integrating low-rank adaptation with ensemble learning. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 11435–11441.

Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. 2024. Longmemeval: Benchmarking chat assistants on long-term interactive memory. arXiv preprint arXiv:2410.10813.

Yixuan Wu, Yizhou Wang, Shixiang Tang, Wenhao Wu, Tong He, Wanli Ouyang, Philip Torr, and Jian Wu. 2025. Dettoolchain: A new prompting paradigm to unleash detection ability of mllm. In European Conference on Computer Vision, pages 164–182. Springer.

Ziang Xiao, Michelle X Zhou, Q Vera Liao, Gloria Mark, Changyan Chi, Wenxi Chen, and Huahai Yang. 2020. Tell me about yourself: Using an ai-powered chatbot to conduct conversational surveys with open-ended questions. ACM Transactions on Computer-Human Interaction (TOCHI), 27(3):1–37.

Dexuan Xu, Yanyuan Chen, Jieyi Wang, Yue Huang, Hanpin Wang, Zhi Jin, Hongxing Wang, Weihua Yue, Jing He, Hang Li, et al. 2024a. Mlevlm: Improve multi-level progressive capabilities based on multi-modal large language model for medical visual question answering. In Findings of the Association for Computational Linguistics ACL 2024, pages 4977–4997.

Rongwu Xu, Brian S Lin, Shujian Yang, Tianqi Zhang, Weiyan Shi, Tianwei Zhang, Zhixuan Fang, Wei Xu, and Han Qiu. 2023. The earth is flat because...: Investigating llms’ belief towards misinformation via persuasive conversation. arXiv preprint arXiv:2312.09085.

Yunqiu Xu, Linchao Zhu, and Yi Yang. 2024b. Mc-bench: A benchmark for multi-context visual grounding in the era of mllms. arXiv preprint arXiv:2410.12332.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, and et al . 2024. Qwen2 technical report. Preprint, arXiv:2407.10671.

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. Minicpm-v: A gpt-4v level mllm on your phone. CoRR.

Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. Justice or prejudice? quantifying biases in llm-as-a-judge. In Neurips Safe Generative AI Workshop 2024.

Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. 2024. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13040–13051.

Kazumi Yoshimura, Dominique Chen, and Olaf Witkowski. 2024. Synlogue with aizuchi-bot: Investigating the co-adaptive and open-ended interaction paradigm. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–21.

Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. 2024. Contextual object detection with multimodal large language models. International Journal of Computer Vision, pages 1–19.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986.

Shuili Zhang, Hongzhang Mu, and Tingwen Liu. 2024. Improving accuracy and generalizability via multi-modal large language models collaboration. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.

Yang Zhang, Yi Shao, Wenbo Wan, Jing Li, and Jiande Sun. 2022. Clip pre-trained models for cross-modal retrieval in newsimages 2022. In MediaEval.

Zheng Zhang, Ryuichi Takanobu, Qi Zhu, MinLie Huang, and XiaoYan Zhu. 2020. Recent advances and challenges in task-oriented dialog systems. Science China Technological Sciences, 63(10):2011–2027.

Yin Jiang Zhao, Yan Ling Li, and Min Lin. 2019. A review of the research on dialogue management of task-oriented systems. In Journal of physics: conference series, volume 1267, page 012025. IOP Publishing.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P Xing, et al. 2023a. Lmsyschat-1m: A large-scale real-world llm conversation dataset. arXiv preprint arXiv:2309.11998.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023b. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2024. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19724–19731.
A  Open-ended Conversations

Open-ended conversation is a flexible and unconstrained form of conversations (Fernyhough, 1996; Barnden, 2014), allowing users to engage freely without predefined limits (Xiao et al., 2020; Yoshimura et al., 2024). In these conversations, the user has full control, enabling them to express thoughts and emotions openly (Elfenbein et al., 2022; Seo et al., 2024), as well as delve into topics in greater depth to uncover insights or solutions (Aziza, 2021; Sun et al., 2022). This type of conversation is the most common mode of interaction for general users in MLLM chat systems (Fu et al., 2024; Liu et al., 2024b).

B  Ethics Statements

The widespread availability of data and powerful analytical tools play a pivotal role in research (Pfenninger et al., 2017; Super, 1982), but also come with the risk of misuse (Pasquetto et al., 2024; Shimron et al., 2022). Therefore, ethical oversight is crucial, it involves issues of privacy (Pina et al., 2024; Lacroix, 2019), ownership (Andrews et al., 2023; Kapusta et al., 2024), consent (McKeown et al., 2021; Longpre et al., 2024), and purpose of use (Paullada et al., 2021; Padmapriya and Parthasarathy, 2024). Based on the definitions and related issues concerning dataset ethics, we make the following statement about MMRC:

The data collection for our study is conducted with the informed consent of all participants, ensuring their privacy and autonomy are fully protected. All participants are fully aware of and voluntarily engage in the annotation process. We implement a rigorous review mechanism to ensure the data is free from personally identifiable information, offensive material, and violent content. However, given the limitations of manual inspection, some residual information may still remain, and completely eliminating such content remains a challenging task. Furthermore, since the data originates from real conversations, it may contain a small amount of inadvertently misleading information, which could impact the model’s performance on benchmark tests. We release this data exclusively for research purposes, allowing researchers to explore the performance of multimodal large language models (MLLMs) in real-world dialogue contexts. However, researchers must approach this dataset with utmost caution and ethical consideration. Our goal is to contribute to the accumulation of knowledge while ensuring that our research findings are applied ethically. In the future, we will continue to release updated versions of the dataset, expanding both its volume and comprehensiveness, while further filtering out offensive content and misinformation.

C  DialogFlow

Collecting data through online platforms is an efficient method to gather large volumes of valuable data (Panduman et al., 2024; Wang et al., 2024a). It offers a cost-effective and scalable solution for data collection, adapting to varying needs over time (Pamucar et al., 2024; Langer et al., 2024). Therefore, we establish DialogFlow to collect user conversation data.

Due to GPU source limitations, DialogFlow is not open to the public and is only undergoing internal testing within the campus. The interface of DialogFlow is shown in Fig.17. Users can add image inputs by clicking the camera icon at the bottom right corner and switch between different models by changing the tabs at the top. The platform is free to use since its launch on April 6, 2024. We use dozens of A100 GPUs to host our website and run open-source models for user conversations, while closed-source models interact via APIs. All the models available on the platform are listed in Table 5. DialogFlow has a total of 354 users, with ages ranging from 18 to 47. The majority of them are university students and staff. Users are required to accept the Terms of Use to consent to the public release of their conversation data.

D  Examples of MMRC

We list 5 samples of MMRC, they are conversations about travel (Fig. 20), dancing (Fig. 21), physics (Fig. 22), water parks (Fig. 23), and dessert making (Fig. 24). The domains of these conversations are comprehensive, natural, and closely aligned with real-world usage scenarios. Additionally, the evaluation questions for the conversations are manually annotated and checked by our team. All of this demonstrates the high quality and real-world relevance of our data.
Figure 17: The page of DialogFlow

<table>
  <tr>
    <th>Model</th>
    <th>Size</th>
    <th>Vision Encoder</th>
    <th>LLM</th>
  </tr>
  <tr>
    <td>LLaVA-V1.5 (Liu et al., 2023)</td>
    <td>7B,13B</td>
    <td>CLIP ViT-L/14 (Zhang et al., 2022)</td>
    <td>Vicuna-v1.5 (Zheng et al., 2023b)</td>
  </tr>
  <tr>
    <td>MiniCPM (Yao et al., 2024)</td>
    <td>8B</td>
    <td>SigLIP-SoViT-400m/14 (Zhai et al., 2023)</td>
    <td>Llama3-Instruct (Dubey et al., 2024)</td>
  </tr>
  <tr>
    <td>LLaVA-Next (Li et al., 2024b)</td>
    <td>0.5B,7B</td>
    <td>SigLIP-400M (Zhai et al., 2023)</td>
    <td>Qwen1.5 (Bai et al., 2023)</td>
  </tr>
  <tr>
    <td>Qwen2VL (Wang et al., 2024b)</td>
    <td>2B,7B,72B</td>
    <td>ViT trained from scratch (Alexey, 2020)</td>
    <td>Qwen2 (Yang et al., 2024)</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision (Li et al., 2024b)</td>
    <td>0.5B,7B</td>
    <td>SigLIP-400M (Zhai et al., 2023)</td>
    <td>Qwen2 (Yang et al., 2024)</td>
  </tr>
  <tr>
    <td>VILA1.5 (Lin et al., 2024)</td>
    <td>3B,8B</td>
    <td>SigLIP-400M (Zhai et al., 2023)</td>
    <td>Llama3 (Grattatiori et al., 2024)</td>
  </tr>
  <tr>
    <td>mplug-Ow3 (Ye et al., 2024)</td>
    <td>1B,2B,7B</td>
    <td>SigLIP-400M (Zhai et al., 2023)</td>
    <td>Qwen2 (Yang et al., 2024)</td>
  </tr>
  <tr>
    <td>InternVL2.5 (Ye et al., 2024)</td>
    <td>8B</td>
    <td>InternVIT-300M-448px-v2.5 (Chen et al., 2024)</td>
    <td>internlm2.5-7b-chat (Cai et al., 2024)</td>
  </tr>
  <tr>
    <td>InternVL2.5 (Ye et al., 2024)</td>
    <td>26B</td>
    <td>InternVIT-6B-448px-V1.5 (Chen et al., 2024)</td>
    <td>internlm2.5-20b-chat (Cai et al., 2024)</td>
  </tr>
  <tr>
    <td>GPT-4o (Islam and Moushi, 2024)</td>
    <td>300B</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Claude-3.5-somnet (Yao et al., 2024)</td>
    <td>175B</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Gemini-1.5 Pro (Team et al., 2024)</td>
    <td>175B</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DeepSeek-V3 (Liu et al., 2024a)</td>
    <td>671B</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table 5: MLLMs in DialogFlow.

E Topic Classification Network

The 14 predefined categories are carefully chosen to ensure comprehensive coverage of real-world conversational topics, and their selection is grounded in the characteristics of our dataset and relevant studies on human conversations (Kim and Metze, 2018; Zhao et al., 2019; Zhang et al., 2020; Alghairy and Ahmed, 2024).

To train the topic classification network, we manually annotate a portion of the data, label the date with 14 predefined categories, each containing 50 dialogue samples. We split the data into 90% for training and 10% for testing. For the classification, we use the all-mnpt-base-v2 model from SentenceTransformers (Piao, 2021) to generate text embeddings. These embeddings are passed through a two-layer MLP for classification. The model ultimately achieve over 95% accuracy on the test set.

F Statics of conversation domains in MMRC

Although we have categorized the existing conversation data into 14 categories for convenience in statistical analysis, the actual diversity of the data far exceeds this classification. If we increase the
<table>
  <tr>
    <th>Category</th>
    <th>Specific Domains</th>
  </tr>
  <tr>
    <td rowspan="2">Travel</td>
    <td>Destination recommendations, travel preparations, accommodation issues, travel budget and planning, transportation methods, scenic spots and activities, culture and language, travel safety, backpacking, luxury travel, self-driving tours, group tours, travel photography, food during travel, travel souvenirs, long-haul flight tips, hotel check-ins and checkouts, luggage handling, airport services, train travel, water travel, bus travel, road trips, travel logistics during travel, recommended travel apps, travel blogs, high-end travel, travel photography tips, entertainment activities during travel, unforgettable trip experiences, travel shopping list, domestic travel, international travel, common sense of travel safety, holiday travel, food tours, island vacations.</td>
  </tr>
  <tr>
    <td>Music styles, music history, music theory, music composition, music production, music appreciation, music education, music technology, instrument performance, music festivals, singers and bands, music films, emotional expression in music, music and culture, music and society, conections and music festivals, music charts, music videos, music producers, music writers, music reviews, music tags, importance of music in life, music industry, music conservation, music videos, social media, music collaboration, music creativity and inspiration, music rhythm, lyrics in music, psychology of music, pop music, classical music, rock music, jazz, hip-hop music, electronic music, world music, music and film scores, music marketing, music performance skills, dance music, music and emotional therapy, music apps, music chart analysis, cultural significance of music, live music recording, music reviews and recommendations, trends in the music industry.</td>
  </tr>
  <tr>
    <td rowspan="2">Game</td>
    <td>rpg game design, rpg genre, narrative design, fiction narrative, game plan, game character design, game AI and artificial intelligence, game testing and quality control, game release, game market, game industry trends, game hardware and devices, game virtual reality, game augmented reality, guitar controllers, video game culture, game player communities, e-sports, game streaming, game reviews and evaluations, online games, sandbox games, single-player games, turn-based games, dungeon crawlers, strategy games, tower defence games, database games, PC games, adventure games, role-playing games, sports games, action games, shooting games, simulation games, strategy games, puzzle games, music games, indie games, game innovation, game marketing, game development process, game creativity and inspiration, game props and items, emotional interaction in games, game balance and difficulty, engine, game maps and world design, social interaction in games, achievement systems in games, game education and learning, economic systems in games, moral choices in games, time management in games, exploration and discovery in games, choices and consequences in game play, multiple endings in games, openness of game worlds.</td>
  </tr>
  <tr>
    <td>Types of sports, sports history, sports culture, sports competition, sports training, sports psychology, sports injuries, sports nutrition, athletics, sports competitions, sports referees, sports rules, sports management, sports marketing, sports events, sports technology, sports equipment, live sports broadcasting, sports media, sports communication, sports psychology, sports training guide, sports injuries, sports nutrition, athletics, sport history and social impact of sports, sports education and health, sports and physical fitness, extreme sports, indoor sports, outdoor sports, team sports, individual sports, collective events, competition strategies, organization of sports events, sports training methods, youth sports, sports psychological adjustment, sports meets and the Olympic Games, sports legislation and rules, sports facilities, sports hopping, sports organization, sports clubs and teams, sports training equipment and apparatus, athletes’ careers, sports organizations and federations, sports law and contracts, sports clubs and teams, sports training equipment and apparatus, athletes’ careers, sports hopping, sports organization, sports clubs and teams, sports training equipment and apparatus, athletes’ careers, sports hopping, sports organization, sports clubs and teams, sports training equipment and apparatus, athletes’ careers, sports hopping.</td>
  </tr>
  <tr>
    <td rowspan="3">Sport</td>
    <td>Home design, home decoration, interior decoration, furniture selection and matching, home style, family organization and storage, home lighting, home layout, family space planning, kitchen design, dining room design, living room design, bathroom design, children’s room design, home office design, home greening and plants, residential architecture, home decoration materials, home appliance selection, smart home design, home cleaning, home maintenance and repair, home consumption and consumption habits, home service, home kitchen and soft kitchen, smart home related deal, smart home design and improvement, ecological environment and health between family and space, family environment and health, eco-friendly home, small apartment home design, family storage solutions, home feng shui, DIY home projects, home budget and expense, home renovation and refurbishment, family entertainment area, courtyard design, outdoor home, family safety and protection, family leisure.</td>
  </tr>
  <tr>
    <td>Food types, ingredient selection, food nutrition, recipes and cooking techniques, healthy eating, food culture, international cuisine, local specialties, food safety, drink selection, vegetables and fruits, restaurant recommendations, the catering industry, catering services, food pairing, desserts and baking, traditional cuisine, vegetarianism and veganism, high-protein diets, low-carb diets, sugar-free foods, low-fat diets, low-salt diets, fast food and convenience foods, organic foods, home-cooked meals and family meals, homemade meals, fast food and convenience foods, beverage selection and shopping, beverage combinations, selection of ingredients, cooking tools and equipment, food labels, cross-cultural cuisine, street food, homemade meals, food cultural heritage, food waste, sustainable eating, Food Innovation, category, brands, food and travel, cooking skills, food festivals and events, new food ingredients, fast food, fast food restaurants, imported foods, Food history, food digestion issues, unhealthy menus, dietary fiber, food groups, nutrient dense foods, healthy eating, tobacco and beverages, family meals and eating habits, beverages and wines, table manners, healthy eating habits, low-calorie recipes, quick cooking, food innovation and trends, food consumption and trends, family meals, festival foods, picnic and outdoor dining, health-preserving dishes.</td>
  </tr>
  <tr>
    <td>Shopping categories, shopping tips, shopping on mobile, online shopping, shopping websites, shopping experience, fashion shopping, shopping budget, discount and promotion, global fashion community, shopping videos, authentic brand, hard-shopping, shopping content, shopping price trends, e-commerce trust, behavioral understanding, shopping, shopping payment methods, shopping coupons, return and exchange policies for shopping, luxury shopping, shopping traps and fraud, shopping psychology, shopping brands, shopping festivals and tag sales, shopping social platforms, shopping and culture, emotional experience while shopping, shopping baskets and cards, logistics and delivery, shopping data, personalized shopping, customized products, augmented reality (AR), item tracking, online shopping, recommendation engines, personalized shopping recommendations, gift and giveway shopping, health and beauty product shopping, home and living goods shopping, food shopping, electronics shopping, sports and outdoor products shopping, fashion and apparel shopping, makeup and skincare product shopping, jewelry and accessories shopping, social commerce in shopping, shopping prices, tax and import fees shipping, online-offline shopping integration, personalized custom shopping, international brand shopping.</td>
  </tr>
  <tr>
    <td rowspan="4">History</td>
    <td>Historical periods, ancient civilizations, medieval history, modern history, 20th-century history, world wars, historical figures, historical events, historical cultures, historical geography, rise and fall of nations and empires, economic history, political history, wars and conflicts, social change, history of technologies and developments, influence of race and ethnicity on art and culture, history of ethics, history of individuals, Anglo-Saxon and Roman Antiquity, Chinese History and documents, history of international relations, historical theories and methods, major events in world history, the rise and fall of great powers, the impact and lessons of history, war and peace, major historical reforms, the Renaissance, the Age of Enlightenment, the Industrial Revolution, the Cultural Revolution, the Great Depression, the Cold War, and World War II. Western, Asian, Latin American, African, Islamic and Latin American countries, wars and conflicts, political and social changes, literary evidence, science, philosophy, revolutions, industrialization, technical innovations in history, national independence movements, the Cold War and global politics.</td>
  </tr>
  <tr>
    <td>Economy</td>
    <td>Macroeconomics, microeconomics, economic growth, international economics, economic growth, monetary policy, fiscal policy, supply and demand, markets, structures, globalization, trade and globalization, economic crises and recessions, economic cycles, capitalism and socialism, economic systems and institutions, national economies and government policies, consumer behavior, production and labor markets, labor economics, resource allocation and efficiency, public economics, economic inequality, international trade and investment, foreign exchange markets and exchange rates, economic reforms, taxation and fiscal management, money and banking, financial sector and banks, financial market, financial systems, financial institutions, national debt, external debt, monetary policy and central banking, development economics, environmental economics, sustainable development, energy economics, market failures and government intervention, financial crises, debt crises, economic policy analysis, consumerism, wealth distribution and social welfare, wealth creation and investment, capital markets, stock and bond markets, financial analysis and management, business strategy and innovation, business management and operating environments.</td>
  </tr>
  <tr>
    <td>Biology</td>
    <td>Cell biology, molecular biology, genetics, evolution theory, ecology, ecosystems, species and species diversity, biological classification, microbiology, DNA and RNA, human biology.</td>
  </tr>
  <tr>
    <td>Computer</td>
    <td>Computer hardware, computer software, operating systems, principles of computer organization, computer networks, data structures and algorithms, programming languages, artificial intelligence, machine learning, deep learning, big data, cloud computing, distributed computing, parallel computing, high-performance computing, computer graphics, computer vision, network technologies, computer security, Practice-oriented security, earthquake prediction and response technologies, computer architecture, computer storage, computer graphics, virtual reality and augmented reality, computer science history, computer ethics, computer application, computer networks, computer simulation, software engineering, computer programming, web development, mobile application development, database management, computer viruses and protection, computer communication protocols.</td>
  </tr>
  <tr>
    <td>Society</td>
    <td>Social structure, social stratification, social classes, social change, social culture, social norms and values, social enterprises and social responsibility, social networks and information technologies, internet and society, and social media, social justice and equality, societal values and desire, sociology and psychology of modern society, the internet and society, conflict and cooperation in society, globalization of society, law and social order, social capital, social welfare policies, values of modernity.</td>
  </tr>
  <tr>
    <td>Job Career</td>
    <td>Career development, career planning, job search skills, resumes and cover letters, interview techniques, career goal setting, career path choices, career transitions and career growth, research and development, work-life/workplace interprofessional relationships, offset culture, compensation and benefits, workplace stress and coping, work-life balance, self-improvement and learning, leadership and management, workplace communication skills, teamwork and collaboration, professional skills and certification, industry trends and prospects, work environment and company culture, occupational safety and health, professional ethics, entrepreneurship.</td>
  </tr>
</table>

Figure 18: List of partial existing domains of conversations in MMRC.
granularity of the categories, our data would encompass more domains. According to our statistics, we have data from 874 different domains. Due to space limitations, we select a subset of domains, as shown in Fig. 18, conversation data in MMRC is comprehensive and diverse, covering various aspects of real-world scenarios. Therefore, the evaluation of MMRC provides valuable insights of MLLMs’ performance in practical open-ending conversations.

G  Experimental Settings

LLaVA-V1.5-7B, LLaVA-V1.5-13B, MiniCPM-8B, LLaVA-Next-0.5B, LLaVA-Next-7B, Qwen2VL-2B, Qwen2VL-7B, LLaVA-OneVision-0.5B, LLaVA-OneVision-7B, VILA1.5-3B, VILA1.5-8B, mplug-Ow3-1B, mplug-Ow3-2B, mplug-Ow3-7B, InternVL2.5-8B, and InternVL2.5-26B use 16-bit floating-point precision, while Qwen2VL-72B and LLaVA-OneVision-72B use 4-bit quantization. Their output length is limited to a maximum of 256 tokens. The models utilize the default values for Temperature, Top-k Sampling, and Top-p Sampling as specified in their Hugging Face inference code.

H  Why DeepSeek-V3 is Classified as Close-Sourced in Experiment

Although the DeepSeek-V3 has been open-sourced on Hugging Face, its large parameter size makes local inference very resource-consuming. Therefore, we use the API for testing. In the experiment, we categorized it under the close-sourced section, as its parameter size is similar to that of close-sourced models, both exceeding 175B parameters.

I  Discussion on the Impact of Different Model Architectures on Results

we have conducted a detailed investigation into the impact of different MLLM architectures and training data on model performance. We have summarized the various model architectures as follows: (1) Typical ViT-MLP-LLM structures: LLaVa-V1.5, LLaVA-NeXT-Interleave, LLaVA-OneVision, Qwen2-VL, VILA-1.5. (2)Additional multi-image processing architectures: MiniCPM (Shared Compression Layer), mplug-Ow3 (Hyper Attention Transformer Block). (3) Novel vision encoder: InternVL2.5 (InternViT).

From the experiment result, We observe that models with extra multi-image processing architectures outperform the typical ViT-MLP-LLM models, with an average improvement of 12.16%. This suggests that architectural enhancements on top of the typical ViT-MLP-LLM framework are beneficial to dialogue capabilities, as they enhance multi-image understanding, giving such models an advantage in conversations involving multiple images. Furthermore, InternVL2.5 shows a significant improvement over the typical ViT-MLP-LLM structure, with a performance gain of 29.62%. Which can be attributed to the powerful InternViT, which offers superior visual understanding and includes advanced mechanisms such as Window Attention. These architectural innovations allow InternVL2.5 to more effectively capture visual details in multi-turn conversations, enabling it to provide more accurate answers to vision-related queries.

J  Prompt Template and Human Criteria for Evaluation

In recent years, an increasing number of benchmark studies have utilized LLM as an evaluation tool, due to its accuracy, logical reasoning, and ability to follow instructions (Chen et al.; Li et al., 2024a; Gu et al., 2024; Ye et al.).

MMRC also applies GPT-4o for scoring the model’s information extraction, cross-turn reasoning, information update, image management, long-term memory recall, and answer refusal capabilities. The detailed evaluation prompts corresponding to each capability are shown in Fig. 25, Fig. 26, Fig. 27, Fig. 28, and Fig. 29. We also conducte manual evaluation of the capabilities in cross-turn reasoning, image update, and memory recall. The three evaluation criteria are as follows:
(1) Cross-turn reasoning (CR):
Goal: To assess how well the model integrates and utilizes information across multiple dialogue turns to answer complex questions.
Criteria: Full reasoning (5 points): the model accurately integrates information from multiple dialogue turns and provides a clear answer that reflects all key details, demonstrating full understanding of the
context; Partial reasoning (3-4 points): the model integrates some of the conversation’s information, but the answer does not fully reflect all necessary context, it is partially correct; Error in Reasoning (2 points): the model fails to integrate information properly, providing an unclear or erroneous reasoning chain, leading to an inaccurate answer; Lack of Reasoning (1 point): the model does not demonstrate effective reasoning, providing a vague or irrelevant answer without appropriately incorporating the conversation’s context.

(2) Information Update (IU):
Goal: To evaluate how well the model tracks and updates factual information provided by the user during the conversation.
Criteria: Complete Update (5 points): the model accurately and promptly updates new factual information, incorporating this into later responses and maintaining consistency throughout the conversation; Partial Update (3-4 points): the model updates some information but fails to fully incorporate the changes in subsequent responses. There may be some omissions or incomplete updates; Failure to Update (2 points): the model does not recognize the change in factual information and continues to rely on outdated data, leading to inconsistent responses; Incorrect Update (1 point): the model incorrectly updates information, causing contradictions with prior details or incorrect facts to be incorporated in later responses.

(3) Memory Recall (MR):
Goal: To assess how well the model maintains long-term memory of the conversation and recalls relevant details in later dialogue turns.
Criteria: Accurate Recall (5 points): the model accurately recalls key information (e.g., user preferences, previous conversation details) from earlier dialogue turns and integrates this information effectively in later responses; Partial Recall (3-4 points): the model recalls some details but misses out on others, leading to slightly disjointed or incomplete answers; Memory Loss (2 points): the model forgets important details over the course of the conversation, leading to inconsistent or disconnected responses that lack coherence; Incorrect Recall (1 point): the model recalls information incorrectly (e.g., mixing up user preferences or repeating outdated facts), leading to answers that are inaccurate or misleading.

K  Attention Calculation

The phenomenon of attention imbalance distribution in models during dialogue shown in Fig 8, is consistent with recent research (Jawale et al., 2024). The following are the detailed steps for calculating attention: we randomly select 100 conversations with 15 turns to visualize the attention. For these dialogues, we perform manual special processing, where we label the information locations related to subsequent memory questions (i.e., IE, IM, MR) as the ‘golden position’, referred as \( G \). For example, if answering an IE question requires extracting information from turn 2 and turn 4, the golden position for this IE question would be turns 2 and 4. The role of the golden label in subsequent calculations will be explained in detail. For the attention calculation model, we selected LLaVA-Next-0.5B, LLaVA-OneVision-72B, MiniCPM-8B, and QwenVL-2B. These choices aim to cover a broad range of model families and parameter sizes.

When the model is tasked with answering memory-related questions, the input typically consists of a dialogue history and an memory evaluation question. Let the input be \( \text{query} = [x_1, x_2, \ldots, x_n, q_s] \), where \( [x_1, x_2, \ldots, x_n] \) represents the conversation history of the model, and each turn \( x_i \) (where \( 1 \leq i \leq n \)) may contain image tokens and text tokens, and \( q_s \) represents the current evaluation question. To compute the self-attention weights for each turn, we define a function \( \text{Attn}: \mathcal{X} \times \mathcal{N} \to \mathbb{R} \), which computes the average attention weight assigned to each turn \( x_i \), where each turn \( x_i = \{x_{i,j}\}_{j=1}^{N_i} \) contains \( N_i \) tokens. Only the turns in the golden position are considered in the attention calculation. The other turns are excluded to avoid interference from irrelevant information. We compute the average attention weight for each golden position turn using the following formula:

\[
\text{Attn}(q_s, i) = \frac{1}{N_i} \sum_{j=1}^{N_i} \text{attn}(x_{i,j}), \quad x_i \in G
\]
L  Details on NOTE-TAKING

The NOTE-TAKING employs another MLLM for note-taking, with model selection including open-source models like LLaVA-One-Vision-7B, MiniCPM-8B, as well as closed-source models like GPT-4o. Since the task of note-taking is easier than conversation, the choice of different models has little impact on note-taking performance. The algorithm for our proposed NOTE-TAKING strategy is in Alg. 1. Due to the limitations of space in the main text, we have only presented a portion of the experimental results. Table 6 shows the complete experimental results.

Algorithm 1 NOTE-TAKING algorithm
Require: MLLM A (notetaker), MLLM B (test model), user input \( \mathcal{T} \), evaluation questions \( Q \).
Step1: Takes Notes
1: \( Note = \{ \} \)
2: for \( T_i \) in \( T \) do
3:     \( history = append(T_i) \)
4:     \( R_i = Chat(B, history) \)
5:     \( Note = Take\_Note(A, T_i, Note) \)
6:     \( history = append(R_i) \)
7: end for
Step 2: Help to response
8: for \( Q_i \) in \( Q \) do
9:     \( history = append(Q_i) \)
10:    \( answer = Chat(B, (history, Note)) \)
11:   output \( response_i \)
12: end for

![NOTE-TAKING Prompt](page_1012_1012_388_388.png)

### System Prompt
You are an intelligent chatbot designed to accurately and thoroughly record information from conversations, and then document it in a JSON format. Here is how you can accomplish your tasks:
- Focus on the user's preferences and the facts within the conversation.
- If the user presents a new fact, it should overwrite the outdated fact.

### User Prompt
I will give you the user's input one by one. Extract key information from the input, including the user's preferences, events, and facts presented in user's input. If you detect a change in information, you should overwrite the outdated content in the notes with the updated information. REMEMBER NOT TO RESPOND TO THE USER'S INPUT!
Here is the user input: \( \{T_i\} \), provide the Note in JSON format, for example: { "user_info": { "health_conditions": ["High blood pressure", "Dairy allergy"] }, "preferences": { "food_preference": "Italian cuisine" }, "purpose": "Seeking meal suggestions." }.

Figure 19: The prompt template of Take_Note function in Alg. 1.
Figure 20: A sample from MMRC, the dialog domain is travel sharing, consisting of 9 conversation turns and 6 image inputs.
Dancing Suggestions

Round 1 - 5

I’ve been learning dance recently, but I find that I can’t control my turns very well, and the movements don’t feel smooth. Any advice?
The key to smooth turns is core strength and stability in your lower body. Try doing more core exercises to ensure you can control each step. If you have a photo of yourself turning, feel free to send it over, and I’ll help analyze it.

Here’s a photo of me practicing turns. What do you think?

Your turn looks pretty good! You’ve done a great job of keeping your thighs and feet stable. However, you could try improving the control over your upper body.

Thanks for the advice! Sometimes I get rusty, especially when performing on stage. How can I improve my stage confidence?
Practicing multiple times will help you get used to the rhythm of the stage. Do you have any photos of yourself performing on stage? Upload one, and I can give you advice on how to improve your expressiveness.

Here’s photos of me performing on stage.

Your stage posture looks very natural and expressive! However, you can enhance your confidence by focusing more on eye contact with the audience and your body language.

Thanks! I’ll pay more attention to these details in my practice and get more exercises. Besides that, how can I make my movements feel smoother?
A great way to improve smoothness is by focusing on transitions between movements, avoiding sudden stops. Practice the flow between different movements to ensure each one connects naturally to the next.

Round 6 - 10

I have trouble with quick weight shifts like this. The transitions between movements feel unnatural. This is what I need to work on most.

When shifting weight quickly, it’s crucial to keep your movements fluid. Practicing weight shifts slowly at first and gradually increase speed. Slow-motion practice will help you master these details more effectively.

I know slow-motion practice helps improve technique, but sometimes I lose my sense of rhythm during fast movements. Any suggestions to fix that?

Maintaining stable rhythm is key. You can practice rhythm exercises, such as syncing with the music, to help improve your overall timing and movement. Have you tried breaking down the movement into steps and gradually increasing the speed?

I haven’t tried that method yet. Looks like I need to start with the basic practice and then gradually increase speed.

That’s a very effective way to practice! Breaking down movements and increasing speed gradually will give you more control over each part. As you practice more, both your speed and smoothness will improve.

Got it! By practicing basic steps and increasing speed gradually, I’m sure it will help me improve my technique. What other areas do you think I should work on?

In addition to working on fluidity and rhythm, confidence in your performance is also important. If you solidify your foundation in this area, so continue practicing, gradually increasing the difficulty, and adapting to different stage situations.

Thanks so much for your advice! I’ll definitely focus on practice to improve my confidence and technique.

I’m glad I could help! Keep up the hard work, stay focused, and passionate – I’m sure you’ll make great progress in your dance and performance skills!

Evaluation

2 Information Extraction
What dance-related elements were mentioned in the conversation?
Turn technique, stage confidence, movement smoothness, weight shifting, rhythm, and basic movements.

2 Cross-turn Reasoning
Why do user feel that his turns aren’t smooth?
The author lacks sufficient training and confidence.

2 Information Update
What is the area the user wants to improve most?
Transitions between movements

2 Image Management
What dance movements or expressions are involved in the first image?
The user is practicing a turn.

2 Memory Recall
What was the main issue the user mentioned in the first round of conversation?
The issue mentioned was difficulty controlling turns.

2 Answer Refusal
Did the user ask about how to quickly switch between different dance styles?
Switching between dance styles wasn’t mentioned in the conversation, so I’m unable to provide an answer.

Figure 21: A sample from MMRC, the dialogue domain is dancing suggestions, consisting of 10 conversation turns and 4 image inputs.
Physic

Round 1 - 5

I’ve been studying physics recently and I want to learn more about light refraction. Can you explain it in detail?

Of course! Light refraction happens when light passes through two different media and its speed changes, which causes its direction to change. For example, when light moves from air into water, its speed slows down, and the direction bends. Have you seen any experiments or illustrations about this?

I remember seeing some images in books, but I don’t quite understand the details. Let me find one, I just found an image of it. I think the phenomenon is caused by light dispersion.

Sure, seeing your photo might help better answering. Please upload your image!

Here’s the photos I found. Explain them for me.

Wow, this images are very clear! You can see how the light refracts when it moves from air to water, and the angle changes. You’ve done a great job showing the basic principle of refraction. If possible, you could label the angle of incidence and the angle of refraction, which would help better understand the laws of refraction.

!!! Was this caused by refraction? My previous understanding was actually wrong. Got it, I’ll label the angle of incidence and the angle of refraction. So, how is light refraction related to changes in the speed of light?

The change in the speed of light is related to the density of the media. Light travels at different speeds in different media; it usually moves slower in denser media. At the boundary between two media, when the light hits a higher refractive index than air, the light slows down, causing it to bend. Does this explanation make sense?

I understand now! But what happens if light moves from water to air? How does the refraction change in the opposite direction?

Great question! If light moves from water to air, the refraction behaves a bit differently. Since air has a lower refractive index than water, the light will bend away from the normal direction when it refracts from water to air, meaning the angle of refraction is greater than the angle of incidence. Have you ever done an experiment to observe this?

Round 6 - 10

I haven’t done an experiment, but I remember seeing a related video. I think I can use this experiment to show the relationship between refraction. I have an image that shows the relationship between the angle of refraction and the angle of incidence, would you like to see it?

That sounds like a great idea! Please upload your image, I’m excited to see how you’ve shown the relationship between the angle of incidence and the angle of refraction.

Here’s the image I found, showing how the angle of refraction changes with different angles of incidence. What do you think?

This image is excellent! You can clearly see the relationship between the angle of incidence and the angle of refraction—as the angle of incidence increases, the angle of refraction also increases. Did you notice that when the angle of incidence gets large enough, the angle of refraction reaches a critical value and the light undergoes total internal reflection?

Wow, I didn’t realize the phenomenon of total internal reflection. So, under what circumstances does total internal reflection occur?

I’ve heard that optical fibers use total internal reflection to transmit data, it’s really amazing! What are some applications of total internal reflection?

I’ve heard that optical fibers use total internal reflection to transmit data, it’s really amazing! What are some applications of total internal reflection?

Yes, optical fibers use the principle of total internal reflection to efficiently transmit light signals. In addition, optical instruments like prisms and telescopes also use this principle. Total internal reflection is very important in modern communication technology because it allows signals to travel long distances without any loss. Would you be interested in learning about how optical fibers work?

I’m very interested in how optical fibers work. Can you explain it briefly?

Of course! Optical fibers are made of a thin glass or plastic fiber. When light signals travel through the optical fiber, they are kept within the walls of the fiber due to total internal reflection.

Evaluation

2 Information Extraction
What physical phenomena were mentioned by the user in the conversation?
The phenomena of light refraction, light dispersion, changes in the speed of light, total internal reflection, and the application of optical fibers.

2 Cross-turn Reasoning
Based on your understanding of light refraction and total internal reflection in conversation, how do optical fibers avoid signal loss?
Optical fibers keep light inside the core using total internal reflection. The light bounces off the core-cladding boundary, preventing signal loss even when the fiber bends.

2 Information Update:
Tell me about the user’s understanding of the pencil bending in water phenomenon.
Light dispersion was changed to light refraction.

2 Image Management
Which image shows phenomenon of light refraction?
The first and second one show the process of light refraction from air to water.

2 Memory Recall
Where did the user find the images related to light refraction?
The user found the images from the books.

2 Answer Refusal
Does the user have any other knowledge about the phenomenon of light polarization?
I can’t answer it because light polarization don’t exist in the conversation.

Figure 22: A sample from MMRC, the dialogue domain is physic, consisting of 10 conversation turns and 3 image inputs.
Water Park Traveling

Round 1 - 9

I'm at the water park today. A lot of fun here!

Wow! It seems that there are lots of fun projects waiting for you. Are you ready to start a happy water journey?

I'm ready! I came to the wave pool first. The waves here are really big and everyone is playing happily.

The wave pool is really lively! Floating up and down with the waves in it is just like being in the sea. But pay attention to safety and don't get washed away by the waves.

I had a great time in the wave pool and also tried the surfboard. Although it was a bit difficult, it was really exciting.

How brave you are! You looked so cool on the surfboard. Practise a few more times and you'll surely master the skills and enjoy the fun of surfing.

Coming out of the wave pool, I went to the water slide area. There are all kinds of slides here. This kind of the slide is very high and scary. It looks really scary.

This slide indeed looks quite challenging! The speed must be very fast when sliding down from it. Do you dare to give it a try?

I summoned up my courage and slid down. It was really super exciting. I felt like my heart was going to jump out. This is the best project I've had today!

Wow, you successfully challenged this difficult slide. That's really a feeling of sliding down quickly must be really exciting.

After sliding down the slide, I came to the children's water park. There are many children playing here and the facilities are all very cute.

The children's water park is full of childlike innocence and joy. Although it's an area for children, you can also feel the carefree atmosphere here.

I played with water in the children's water park for a while and also played in the water castle with the children.

Playing with the children must be a lot of fun. The water castle looks really fun. Do you feel like you've gone back to your childhood?

After that, I left the children's water park, I went to the lazy river. Sitting on the inner tube and floating slowly along with the water flow, it was so pleasant.

The lazy river is a great place to relax. During the floating process, you can enjoy the surrounding scenery and have a leisurely time. Do you feel that all the pressure has disappeared?

Round 10 - 17

While floating, I saw some water sculptures beside me. Their shapes were very strange.

These water sculptures add a lot of artistic atmosphere to the water park. Their strange shapes combined with the surrounding water elements form a unique stage. There was a performance going on here and the actors' performances were really wonderful.

The performance on the water stage must be very distinctive! Watching the performance on the water has a unique charm. You can enjoy the wonderful performances of the actors in your heart's content.

I felt a bit hungry and came to the restaurant in the water park. Here is the menu of the restaurant. There are many delicious foods to choose from.

After playing so many projects, it's time to eat some delicious food to replenish your energy and get ready for the happy time later.

I ordered a hamburger and French fries. The taste was not bad. I was eating and relaxing in the restaurant at the same time.

Hamburger and French fries are classic fast-food combinations that can fill your stomach quickly. After having a good rest, are there any other projects you want to try?

After having the meal, I went swimming in the deep-water area. The water here is very deep, but I swam quite well.

You're really good at swimming! Swimming in the deep-water area requires a certain amount of courage and skills. Pay attention to conserving your strength and safety first.

After swimming, I ran the jet ski project and want to experience it. It was really fast. It felt like lightning. So I changed my mind, I feel that this is the most wonderful water sport!

The jet ski must be super exciting! The feeling of the wind blowing past your face and the speed of the water must make your blood boil. Did you have a good time?

After playing the jet ski, I came to the water trampoline area. Jumping up and down on the trampoline, I felt like I was a spring bird. This project is also very fun, interesting!

The water trampoline looks really fun! You can jump to your heart's content on it and release your energy. Do you feel full of fun?

After playing on the trampoline for a while, I went to a relatively quiet spa area. You can relax your body and mind here.

The spa area is a great place to relax. After playing so many projects all day, come here to soothe your body, relieve fatigue and draw a perfect end to today's trip to the water park.

Evaluation

2 Information Extraction
Please list all the water park projects that are mentioned in the text.

Wave pool (including surfboard experience), water slide, children's water park (including water castle), lazy river, water stage performance, swimming in the deep water area, jet ski, water trampoline, spa area.

2 Cross-turn Reasoning
If a person who likes playing with water and seeking excitement came to the water park, what experiences can he or she have?

He or she can enjoy surfing in the wave pool, ride the high water slide, relax in the lazy river, watch the water stage show, swim in the deep area, try jet skiing, jump on the water trampoline, and unwind in the spa.

2 Information Update:
What is the most exciting project the user used? A. Wave pool, B. Water slide, C. Water stage, D. Jet ski

D: Jet ski

2 Image Management
Among the pictures in the conversation, which one is about the water projects carried out at a high place?

The third one.

2 Memory Recall
Do you still remember what other projects were played in the wave pool besides the surfboard mentioned before?

The main project played in the wave pool is the surfboard and besides that, it's playing along with the waves.

2 Answer Refusal
Is there an area called "Dolphin Bay" in the water park?

I can't answer it because there exist no area called "Dolphin Bay" in the water park.

END

Figure 23: A sample from MMRC, the dialogue domain is water park, consisting of 17 conversation turns and 8 images inputs.
Dessert Making

Round 1 - 9

I’m planning to do some baking at home today. Here are all kinds of baking materials that I’ve prepared. They look quite abundant, don’t they?

Wow, so many materials! It seems that you’re going to do some skills. What kind of baked goods are you going to make?

I’m planning to make cupcakes first. This is one of the easiest types of pastry, I think. I’m currently creaming the butter and sugar.

Creaming the butter and sugar is a very important step in making cupcakes. Keep stirring to make it fluffier so that the cupcakes will be softer.

Now I’m adding the eggs one by one and stirring evenly. After adding the eggs, the color of the batter has become lighter.

Yes, add the eggs one by one so that they can blend in better. Is it time to add flour next?

Exactly, I’m sifting in the cake flour and gently stirring to avoid over-mixing.

The operation of sifting in the flour and then gently stirring is correct. This can ensure that the cupcakes have a delicate texture. Are you going to put the batter into the molds next?

I’m putting the batter into the cupcake cups, filling each one almost to the brim.

You’re filling them very evenly! In this way, the baked cupcakes will be about the same size. Are you ready to put them into the oven?

I’ve put the cupcakes into the oven and set the temperature and time. Now I can see the cupcakes slowly rising in the oven.

It must feel great to watch the cupcakes rising in the oven. When they’re baked and taken out, they’ll surely smell delicious.

The cupcakes are baked. I took them out of the oven. They’re golden brown and I can’t wait to see how you’ll decorate the freshly baked cupcakes.

Wow, these cupcakes look so tempting! The freshly baked cupcakes must taste great. You can have one while they’re still hot.

I want to decorate the cupcakes. I’ve prepared cream and fruits.

With cream and fruits, these cupcakes will become more beautiful and delicious. How do you plan to decorate them?

I’m piping the cream onto the cupcakes and then putting sliced strawberries on them. A simple decoration is done. Here are the decorated cupcakes. After all the work, I find that making cupcakes is not an easy task.

Wow, the decorated cupcakes are so pretty! The combination of strawberries and cream is a classic one and they look really delicious.

Round 10 - 17

After making the cupcakes, I’d also like to try my hand at making cookies. I’m preparing the dough for the cookies by mixing butter, sugar and flour. I find that this makes the process a little easier.

Oh, of course! Cookies are also a great choice for baking! Knead the dough well and then you’ll be able to pipe out nice shapes soon.

I’ve put the dough into a piping bag and piped out cookies in various shapes, like circles and hearts.

They’ll become even more delicious after being baked in the oven.

I’ve put the cookies into the oven. After baking for a while, I can already smell the fragrance of the cookies. Here are the cookies in the oven.

The fragrance of cookies is always so tempting. When they’re baked, they’ll surely be crispy and fragrant.

These are how I baked the cookie. I took them out of the oven. They’re golden brown and the shape is perfect.

These cookies look really perfect! The golden brown color and the intact shape show that your baking skills are quite good.

I tasted a cookie. It’s very crispy and has a strong buttery flavor. I find it easier to make cookies than it is to make cupcakes.

You’re enjoying it so much, which means the cookie is really delicious! The crisp texture and the strong buttery flavor are characteristic of cookies. So cookies are the simplest type of cake you would consider.

Yes, I found cookies to be the easiest type of pastry. During the baking process, my child also came to help.

OK, I see. Oh, it’s really warm that the child joined in! This can not only make delicious food but also create beautiful memories.

We also made some chocolate cakes. Although the chocolate cakes are still in the oven, the chocolate flavor can already be smelled.

The chocolate cakes in the oven are really something to look forward to! The rich chocolate flavor will surely make the whole house full of a sweet smell.

The chocolate cakes are baked. I took them out of the oven. The surface is very smooth.

These chocolate cakes look wonderful! The smooth surface and the rich chocolate flavor must make them very delicious.

Evaluation

2 Information Extraction
Please list the baked goods mentioned in the conversation.
Cupcakes, cookies, chocolate cakes.

2 Cross-turn Reasoning
If I were to have a child help out, considering the difficulty of the tasks, which one would you recommend they do? A Cupcake, B Cookie, C Cookie, D Chocolate cake
C Cookie.

2 Information Update:
What is the exact type of pastry that user think?
It’s cookie.

2 Image Management
Among the pictures in the conversation, which one is about the chocolate cakes?
The 7th and 8th images.

2 Memory Recall
Do you still remember what change happened to the batter after adding eggs, the color of the batter became lighter.

2 Answer Refusal
Was yogurt used in the home baking process?
It’s not mentioned in the text whether yogurt was used in the home baking process.
<table>
  <tr>
    <th>Models</th>
    <th>IE</th>
    <th>IU</th>
    <th>MR</th>
    <th>AR</th>
    <th>CR</th>
    <th>IM</th>
  </tr>
  <tr>
    <td>LLaVA-1.5-7B<br>+ NOTE-TAKING(GPT-4o)<br>+ NOTE-TAKING(MiniCPM)</td>
    <td>0.91<br><span style="color:green">2.57 (+1.66)</span><br><span style="color:green">2.33 (+1.42)</span></td>
    <td>0.31<br><span style="color:green">2.06 (+1.75)</span><br><span style="color:green">1.78 (+1.47)</span></td>
    <td>0.22<br><span style="color:green">2.36 (+2.14)</span><br><span style="color:green">2.11 (+1.89)</span></td>
    <td>0.28<br><span style="color:green">0.46 (+0.18)</span><br><span style="color:green">0.35 (+0.07)</span></td>
    <td>1.08<br><span style="color:green">1.22 (+0.14)</span><br><span style="color:green">1.19 (+0.11)</span></td>
    <td>0.52<br><span style="color:green">0.54 (+0.02)</span><br><span style="color:green">0.56 (+0.04)</span></td>
  </tr>
  <tr>
    <td>MiniCPM-8B<br>+ NOTE-TAKING(GPT-4o)<br>+ NOTE-TAKING(MiniCPM)</td>
    <td>4.08<br><span style="color:green">4.23 (+0.15)</span><br><span style="color:green">4.12 (+0.04)</span></td>
    <td>2.98<br><span style="color:green">3.76 (+0.78)</span><br><span style="color:green">3.45 (+0.47)</span></td>
    <td>3.65<br><span style="color:green">4.02 (+0.37)</span><br><span style="color:green">3.74 (+0.09)</span></td>
    <td>3.78<br><span style="color:green">3.92 (+0.14)</span><br><span style="color:green">3.81 (+0.05)</span></td>
    <td>3.94<br><span style="color:green">4.13 (+0.09)</span><br><span style="color:green">4.02 (+0.08)</span></td>
    <td>3.47<br><span style="color:green">3.50 (+0.03)</span><br><span style="color:green">3.52 (+0.05)</span></td>
  </tr>
  <tr>
    <td>QwenVL-2B<br>+ NOTE-TAKING(GPT-4o)<br>+ NOTE-TAKING(MiniCPM)</td>
    <td>2.16<br><span style="color:green">3.71 (+1.55)</span><br><span style="color:green">3.22 (+1.06)</span></td>
    <td>1.41<br><span style="color:green">2.43 (+1.02)</span><br><span style="color:green">2.01 (+0.60)</span></td>
    <td>1.93<br><span style="color:green">3.40 (+1.47)</span><br><span style="color:green">3.21 (+1.28)</span></td>
    <td>2.33<br><span style="color:green">2.84 (+0.51)</span><br><span style="color:green">2.58 (+0.25)</span></td>
    <td>2.85<br><span style="color:green">3.47 (+0.62)</span><br><span style="color:green">3.26 (+0.41)</span></td>
    <td>1.27<br><span style="color:green">1.36 (+0.09)</span><br><span style="color:green">1.32 (+0.05)</span></td>
  </tr>
  <tr>
    <td>LLaVA-Next-0.5B<br>+ NOTE-TAKING(GPT-4o)<br>+ NOTE-TAKING(MiniCPM)</td>
    <td>2.32<br><span style="color:green">3.84 (+1.52)</span><br><span style="color:green">3.68 (+1.36)</span></td>
    <td>1.99<br><span style="color:green">3.04 (+1.05)</span><br><span style="color:green">2.77 (+0.78)</span></td>
    <td>2.67<br><span style="color:green">3.88 (+1.21)</span><br><span style="color:green">3.59 (+0.92)</span></td>
    <td>1.12<br><span style="color:green">2.03 (+0.91)</span><br><span style="color:green">2.06 (+0.94)</span></td>
    <td>2.89<br><span style="color:green">3.62 (+0.73)</span><br><span style="color:green">3.38 (+0.49)</span></td>
    <td>1.87<br><span style="color:green">2.07 (+0.20)</span><br><span style="color:green">1.94 (+0.07)</span></td>
  </tr>
  <tr>
    <td>LLaVA-OneVision-72B<br>+ NOTE-TAKING(GPT-4o)<br>+ NOTE-TAKING(MiniCPM)</td>
    <td>4.06<br><span style="color:green">4.28 (+0.22)</span><br><span style="color:green">4.13 (+0.07)</span></td>
    <td>4.01<br><span style="color:green">4.31 (+0.30)</span><br><span style="color:green">4.28 (+0.27)</span></td>
    <td>4.17<br><span style="color:green">4.38 (+0.21)</span><br><span style="color:green">4.26 (+0.09)</span></td>
    <td>2.52<br><span style="color:green">3.46 (+0.94)</span><br><span style="color:green">3.20 (+0.68)</span></td>
    <td>4.08<br><span style="color:green">4.16 (+0.08)</span><br><span style="color:green">4.13 (+0.05)</span></td>
    <td>3.24<br><span style="color:green">3.31 (+0.07)</span><br><span style="color:green">3.26 (+0.02)</span></td>
  </tr>
  <tr>
    <td>GPT-4o<br>+ NOTE-TAKING(GPT-4o)<br>+ NOTE-TAKING(MiniCPM)</td>
    <td>4.35<br><span style="color:green">4.51 (+0.16)</span><br><span style="color:green">4.46 (+0.11)</span></td>
    <td>4.28<br><span style="color:green">4.62 (+0.34)</span><br><span style="color:green">4.45 (+0.17)</span></td>
    <td>4.31<br><span style="color:green">4.73 (+0.42)</span><br><span style="color:green">4.37 (+0.06)</span></td>
    <td>3.06<br><span style="color:green">3.51 (+0.45)</span><br><span style="color:green">3.32 (+0.26)</span></td>
    <td>4.38<br><span style="color:green">4.41 (+0.03)</span><br><span style="color:green">4.40 (+0.02)</span></td>
    <td>4.12<br><span style="color:green">4.18 (+0.06)</span><br><span style="color:green">4.16 (+0.04)</span></td>
  </tr>
</table>

Table 6: Performance of NOTE-TAKING across all conversational core abilities in MMRC.

### System Prompt

You are an intelligent chatbot designed for evaluating the correctness of generative outputs for question-answer pairs. The question-answer pair is from a long conversation between human and Multimodal Language Models. Models’ answer in based on the conversation history. Your task is to score the models’ answer. Here is how you can accomplish your tasks:
- Focus on the meaningful match between models’ prediction and ground truth.
- Consider synonyms or paraphrases as valid matches.
- Evaluate the correctness of the prediction compared to the ground truth.

### User Prompt

You need to evaluate model’s information extraction ability, determine whether the model has extracted specific information. Here is the model’s prediction {answer}, and here is the ground truth {label}. Provide your evaluation only as a score where the score is an integer value between 0 and 5, with 5 indicating the highest meaningful match. Please response in the form of a Python dictionary string with key ‘score’, where value of ‘score’ is in INTEGER, not STRING. DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only provide the Python dictionary string. For example, your response should look like this: {{'score': 4}}.

Figure 25: GPT prompt for evaluating information extraction (IE).
### System Prompt

You are an intelligent chatbot designed for evaluating the correctness of generative outputs for question-answer pairs. The question-answer pair is from a long conversation between human and Multimodal Language Models. Models’ answer in based on the conversation history. Your task is to score the models’ answer. Here is how you can accomplish your tasks:
- Focus on the meaningful match between models’ prediction and ground truth.
- Consider synonyms or paraphrases as valid matches.
- Evaluate the correctness of the prediction compared to the ground truth.

### User Prompt

You need to evaluate model’s cross-section reasoning ability to see if the model’s reasoning results align with the logic of the ground truth. Here is the model’s prediction {answer}, and here is the ground truth {label}. Provide your evaluation only as a score where the score is an integer value between 0 and 5, with 5 indicating the highest meaningful match. Please response in the form of a Python dictionary string with key ‘score’, where value of 'score' is in INTEGER, not STRING. DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only provide the Python dictionary string. For example, your response should look like this: {{'score': 4}}.

Figure 26: GPT prompt for evaluating cross-turn reasoning (CR).

### System Prompt

You are an intelligent chatbot designed for evaluating the correctness of generative outputs for question-answer pairs. The question-answer pair is from a long conversation between human and Multimodal Language Models. Models’ answer in based on the conversation history. Your task is to score the models’ answer. Here is how you can accomplish your tasks:
- Focus on the meaningful match between models’ prediction and ground truth.
- Consider synonyms or paraphrases as valid matches.
- Evaluate the correctness of the prediction compared to the ground truth.

### User Prompt

You need to evaluate model’s information updating ability, determine whether the facts provided in the model’s prediction align with the ground truth. Here is the model’s prediction {answer}, and here is the ground truth {label}. Provide your evaluation only as a score where the score is an integer value between 0 and 5, with 5 indicating the highest meaningful match. Please response in the form of a Python dictionary string with key ‘score’, where value of 'score' is in INTEGER, not STRING. DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only provide the Python dictionary string. For example, your response should look like this: {{'score': 4}}.

Figure 27: GPT prompt for evaluating information update (IU).
### System Prompt

You are an intelligent chatbot designed for evaluating the correctness of generative outputs for question-answer pairs. The question-answer pair is from a long conversation between human and Multimodal Language Models. Models’ answer in based on the conversation history. Your task is to score the models’ answer. Here is how you can accomplish your tasks:
- Focus on the meaningful match between models’ prediction and ground truth.
- Consider synonyms or paraphrases as valid matches.
- Evaluate the correctness of the prediction compared to the ground truth.

### User Prompt

You need to evaluate model’s image management ability, check whether the model's response to the image aligns with the facts. Here is the model's prediction: {answer}, and here is the factual about the image: {label}. Provide your evaluation only as a score where the score is an integer value between 0 and 5, with 5 indicating the highest meaningful match. Please response in the form of a Python dictionary string with key ‘score’, where value of 'score' is in INTEGER, not STRING. DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only provide the Python dictionary string. For example, your response should look like this: {{'score': 4}}.

Figure 28: GPT prompt for evaluating image management (IM).

### System Prompt

You are an intelligent chatbot designed for evaluating the correctness of generative outputs for question-answer pairs. The question-answer pair is from a long conversation between human and Multimodal Language Models. Models’ answer in based on the conversation history. Your task is to score the models’ answer. Here is how you can accomplish your tasks:
- Focus on the meaningful match between models’ prediction and ground truth.
- Consider synonyms or paraphrases as valid matches.
- Evaluate the correctness of the prediction compared to the ground truth.

### User Prompt

You need to evaluate model’s memory recall ability, to determine whether the model's memory aligns with the correct memory. Here is the model’s prediction {answer}, and here is the ground truth {label}. Provide your evaluation only as a score where the score is an integer value between 0 and 5, with 5 indicating the highest meaningful match. Please response in the form of a Python dictionary string with key ‘score’, where value of 'score' is in INTEGER, not STRING. DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only provide the Python dictionary string. For example, your response should look like this: {{'score': 4}}.

Figure 29: GPT prompt for evaluating memory recall (MR).
### System Prompt

You are an intelligent chatbot designed for evaluating the correctness of generative outputs for question-answer pairs. The question-answer pair is from a long conversation between human and Multimodal Language Models. Models’ answer in based on the conversation history. Your task is to score the models’ answer. Here is how you can accomplish your tasks:
- Focus on the meaningful match between models’ prediction and ground truth.
- Consider synonyms or paraphrases as valid matches.
- Evaluate the correctness of the prediction compared to the ground truth.

### User Prompt

You need to evaluate model’s answer refusal ability to see if the model correctly refuses to answer the question when appropriate. Here is the model’s prediction {answer}, and here is the ground truth {label}. Provide your evaluation only as a score where the score is an integer value between 0 and 5, with 5 indicating the highest meaningful match. Please response in the form of a Python dictionary string with key ‘score’, where value of 'score' is in INTEGER, not STRING. DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only provide the Python dictionary string. For example, your response should look like this: {{'score': 4}}.

Figure 30: GPT prompt for evaluating answer refusal (AR).

Reviewer paper summaries: The authors introduce Multi-Modal Real-world Conversation (MMRC), a benchmark designed to evaluate MLLM performance in real-world, open-ended conversations. This benchmark assesses six types of open-ended conversations: information extraction, multi-turn reasoning, information updates, image management, memory recall, and answer refusal. The authors evaluate 20 MLLMs and observe an accuracy drop in open-ended conversations, primarily due to four common failure patterns: long-term memory degradation, inadequacies in updating factual knowledge, accumulated errors from assumption propagation, and reluctance to "say no." To address these issues, they propose a NOTE-TAKING strategy, which demonstrates significant performance improvements.

The paper introduces Learning from Errors (LFE), a novel method for optimizing example selection in many-shot in-context learning (ICL) for large multimodal models (LMMs). The approach selects examples that the model answered incorrectly to use as demonstrations within the context, proposing two strategies: Parallel Learning from Errors (ParaLFE) and Sequential Learning from Errors (SeqLFE). The authors evaluate LFE across sixteen datasets spanning various tasks and domains, demonstrating consistent improvements over random example selection.
Summary of strengths: - The paper proposes a comprehensive benchmark design and a thorough evaluation methodology to evaluate the performance of over 20 MLLMs on real-world open-ended conversations.

- The curated datasets for evaluation contain around 5,000 conversations and approximately 28,000 manually labeled annotations, providing substantial evidence and making them highly relevant for practical applications.

- The paper identifies four main failure patterns that lead to a drop in accuracy for open-ended conversations, providing valuable insights for future research.

- The authors propose a novel note-taking strategy, offering a promising approach to improving the open-ended conversation abilities of MLLMs.

1. The study makes a significant contribution to addressing the scarcity of high-quality multimodal multi-turn dialogue data, enhancing the available resources in this crucial domain.

2. The research provides novel insights through a comprehensive analysis of various factors contributing to current VLM performance limitations, offering valuable diagnostic perspectives.

3. The work introduces robust and comprehensive evaluation metrics that transcend traditional accuracy-based or conventional dialogue metrics, representing a methodological advancement in assessment frameworks.

4. The investigation offers valuable insights into cross-domain and cross-modal imbalances, providing strategic directions for future model improvements and optimizations.

5. The experimental validation encompasses a comprehensive range of mainstream models with varying architectures and scales, demonstrating thorough empirical rigor and reliability.
Summary of weaknesses: - W1: The benchmark may not cover all possible real-world conversation types (e.g., population distribution and languages), which may partially hinder the usefulness of the benchmark.

- W2: It is not clear that how much the proposed NOTE-TAKING approach is computationally expensive.

- W3: The dataset is predominantly English, which limits applicability for multilingual or low-resource settings.

- W4: The paper's concept of answer refusal only captures scenarios where the answer is not found in the chat history. However, an MLLM may also refuse to provide an answer due to safety or policy constraints, which are not accounted for.

1. The experimental evaluation exhibits notable omissions in model coverage, particularly the absence of significant architectures such as the InternVL series, potentially limiting the comprehensiveness of the comparative analysis.

2. The analytical section lacks sufficient depth in examining inter-model performance differentials. Specifically, there is inadequate investigation into how architectural distinctions and training data distributions contribute to the varying performance patterns observed between model series (e.g., Qwen and LLaVA) across different subtasks.

3. The methodological contributions appear somewhat limited for a paper in the multimodal track. While the data contribution is substantial, the data collection process relies primarily on resource investment rather than introducing novel or generalizable methodological frameworks, potentially compromising reproducibility. 

4. The proposed note-taking approach, despite being presented as a methodological contribution, essentially constitutes a basic prompt engineering technique that lacks sophistication and fails to effectively integrate with the analytical insights presented in the study. This represents a missed opportunity to establish stronger connections between methodology and empirical findings.
Comments, suggestions and typos: - I would suggest to expand the benchmark to include both scenarios when a model refuses to answer: due to a lack of historical context and due to safety/policy constraints.

- I would suggest to expand the dataset to include multicultural or multilingual conversations to understand how MLLMs perform in those scenarios. Additionally, add more real-world conversation types that are not already captured in the dataset.

- I would suggest to explore methods to improve the computational complexity of the NOTE-TAKING approach.

please refer to Summary Of Weaknesses.
Overall assessment scores: 3

3.5
Metareview score: 3.5
Metareview: This paper presents a novel and comprehensive benchmark for Multi-Modal Real-world Conversation, covering six distinct dimensions: information extraction, multi-turn reasoning, information update, image management, memory recall, and answer refusal. This is the first benchmark on this interesting topic. Authors also developed a NOTE-TAKING baseline to better handle the identified challenges from this dataset. Overall, all reviewer gave positive evaluation to this work.
Summary of reasons to publish: Multi-Modal Real-world Conversation is an interesting and meaningful problem, with no public dataset at the current stage. This paper steps forward to establish a new dataset to push the research boundary. Novel challenges are identified through extensive evaluation. Besides, a simple and effective method is devised as a novel baseline.