Title: CRAG-MM: Multi-modal Multi-turn Comprehensive RAG Benchmark
Primary area: datasets and benchmarks
Keywords: multimodal retrieval-augmented generation (MM-RAG), factual visual question answering (VQA), large vision-language models (LVLMs), benchmarking, multi-source augmentation, egocentric, wearable AI

Abstract: Wearable devices such as smart glasses are transforming the way people interact with their surroundings, enabling users to seek information regarding entities in their view. Multi-Modal Retrieval-Augmented Generation (MM-RAG) plays a key role in supporting such questions, yet there is still no comprehensive benchmark for this task, especially regarding wearables scenarios. To fill this gap, we present CRAG-MM---a Comprehensive RAG benchmark for Multi-modal Multi-turn conversations. CRAG-MM contains a diverse set of 6.5K (image, question, answer) triplets and 2K visual-based multi-turn conversations across 13 domains, including 6.2K egocentric images designed to mimic captures from wearable devices. We carefully constructed the questions to reflect real-world scenarios and challenges, including five types of image-quality issues, six question types, varying entity popularity, differing information dynamism, and different conversation turns. We design three tasks: single-source augmentation, multi-source augmentation, and multi-turn conversations---each paired with an associated retrieval corpus and APIs for both image-KG retrieval and webpage retrieval. Our evaluation shows that straightforward RAG approaches achieve only 32% and 43% truthfulness on CRAG-MM single- and multi-turn QA, respectively, whereas state-of-the-art industry solutions reach just 32% and 45%, underscoring ample room for improvement. The benchmark has hosted a leaderboard that attracted about a thousand participants, with winning solutions improving baseline performance by 28%, highlighting its early impact on advancing the field.