MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large
  Vision and Language Models
Authors: name: Young-Jun Lee, name: Byung-Kwan Lee, name: Jianshu Zhang, name: Yechan Hwang, name: Byungsoo Ko, name: Han-Gyu Kim, name: Dongyu Yao, name: Xuankun Rong, name: Eojin Joo, name: Seung-Ho Han, name: Bowon Ko, name: Ho-Jin Choi
Organization: KAIST

Vision-and-Language Models (VLMs) have shown impressive capabilities on
single-turn benchmarks, yet real-world applications often demand more intricate
multi-turn dialogues. Existing multi-turn datasets (e.g, MMDU, ConvBench) only
partially capture the breadth and depth of conversational scenarios encountered
by users. In this work, we introduce MultiVerse, a novel multi-turn
conversation benchmark featuring 647 dialogues - each averaging four turns -
derived from a diverse set of 12 popular VLM evaluation benchmarks. With 484
tasks and 484 interaction goals, MultiVerse covers a wide range of topics, from
factual knowledge and perception to advanced reasoning tasks such as
mathematics and coding. To facilitate robust assessment, we propose a
checklist-based evaluation method that leverages GPT-4o as the automated
evaluator, measuring performance across 37 key aspects, including perceptual
accuracy, linguistic clarity, and factual correctness. We evaluate 18 VLMs on
MultiVerse, revealing that even the strongest models (e.g., GPT-4o) achieve
only a 50% success rate in complex multi-turn conversations, highlighting the
dataset's challenging nature. Notably, we find that providing full dialogue
context significantly enhances performance for smaller or weaker models,
emphasizing the importance of in-context learning. We believe MultiVerse is a
landscape of evaluating multi-turn interaction abilities for VLMs.