Title: MMChat: A Multi-turn Multi-Modal Conversational Benchmark Grounded in Real World User Behaviors
Primary area: datasets and benchmarks
Keywords: Large vision language models, Multi-turn dialogues, Benchmark

Abstract: Large Vision-Language Models (LVLMs) are the foundation of modern AI assistants, which users typically engage in multi-turn conversations involving interrelated questions about images. However, existing benchmarks predominantly evaluate LVLMs in isolated, single-turn settings, creating a significant gap with real-world use. While recent efforts have explored multi-turn evaluation, they often rely on oversimplified, synthetic dialogues that fail to capture the complex, context-dependent nature of real-world user interactions. To address this critical gap, we introduce MMChat, a multi-turn, multimodal benchmark grounded in realistic conversational dynamics. Our benchmark systematically models conversational flows derived from authentic user data, incorporating the diversity, fragmentation, and contextual dependencies observed in practice. We synthesize 1,000 high-quality dialogue turns across 100 conversations, leveraging image-question pairs from SceMQA and interaction patterns from VisionArena-Chat. To evaluate LVLM performance, we employ a comprehensive two-way framework that combines fine-grained subdimension scoring with winrate against human verified ground truth, achieving strong agreement with human judgments. Through extensive evaluation of state-of-the-art LVLMs, we uncover systematic weaknesses in both open-source and proprietary models, particularly in handling fragmented and context-dependent user queries. For example, even an advanced proprietary LVLM like GPT-4o sees its performance score drop from 3.92 to 2.92 on direct opening queries compared to descriptive ones. Furthermore, models struggle with iterative tasks, with GPT-4o scoring just 3.65 in refinement and 3.85 in augmentation. Smaller open-source models perform even worse in these categories, with scores as low as 1.70, highlighting a critical gap in conversational ability.

TL;DR: A multi-turn empirically grounded benchmark for LVLMs.