Title: MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs
Benchmark: MMDU
Summary: This paper introduces MMDU, a new benchmark, and MMDU-45k, a large-scale instruction tuning dataset, to evaluate and enhance the performance of LVLMs in realistic, multi-turn, multi-image conversation scenarios. The authors use a clustering-based method to curate diverse image-text contexts from Wikipedia and construct question-answer pairs with human and GPT-4o collaboration.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: multi-turn multi-image dialog understanding
Phenomenon defined by authors: Yes
Phenomenon definition: evaluate the ability to engage in multi-turn conversations that involve multiple image inputs and comprehend long-context histories to facilitate coherent and contextually appropriate conversations
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Take multiple images and provide answers for different turns of questions
Task item definition: <multiple images> Q1: "Describe the architectural features visible in <image-1>, <image-2>, and <image-3>." A1: "<image-1>, <image-2> and <image-3> depict three distinct architectural marvels from different parts of the world, showcasing unique styles and features. <image-1> shows the Colosseum in Rome, an ancient amphitheater renowned for its grand scale and engineering prowess...... <image-2> depicts the Hall of Prayer for Good Harvests within the Temple of Heaven complex in Beijing, China. This building, completed in the early 15th century, is a masterpiece of traditional Chinese architecture...... <image-3> showcases the Sydney Opera House, a modern architectural icon located on the shores of Sydney Harbour in Australia. Designed by Danish architect Jørn Utzon and completed in 1973, the Opera House is celebrated for its innovative design and engineering......" Q2: ..., A2: ...
Task source: LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 110
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: they conduct quality control with human annotators
Taxonomy: User Interaction / 
New taxonomy: General Capability / User Interaction