CRAG-MM: Multi-modal Multi-turn Comprehensive RAG Benchmark
Authors: name: Jiaqi Wang; name: Xiao Yang; name: Kai Sun; name: Parth Suresh; name: Sanat Sharma; name: Adam Czyzewski; name: Derek Andersen; name: Surya Appini; name: Arkav Banerjee; name: Sajal Choudhary; name: Shervin Ghasemlou; name: Ziqiang Guan; name: Akil Iyer; name: Haidar Khan; name: Lingkun Kong; name: Roy Luo; name: Tiffany Ma; name: Zhen Qiao; name: David Tran; name: Wenfang Xu; name: Skyler Yeatman; name: Chen Zhou; name: Gunveer Gujral; name: Yinglong Xia; name: Shane Moon; name: Nicolas Scheffer; name: Nirav Shah; name: Eun Chang; name: Yue Liu; name: Florian Metze; name: Tammy Stark; name: Zhaleh Feizollahi; name: Andrea Jessee; name: Mangesh Pujari; name: Ahmed Aly; name: Babak Damavandi; name: Rakesh Wanga; name: Anuj Kumar; name: Rohit Patel; name: Wen-tau Yih; name: Xin Luna Dong
Published: 2025-10-30 05:50:48

Abstract: Wearable devices such as smart glasses are transforming the way people
interact with their surroundings, enabling users to seek information regarding
entities in their view. Multi-Modal Retrieval-Augmented Generation (MM-RAG)
plays a key role in supporting such questions, yet there is still no
comprehensive benchmark for this task, especially regarding wearables
scenarios. To fill this gap, we present CRAG-MM -- a Comprehensive RAG
benchmark for Multi-modal Multi-turn conversations. CRAG-MM contains a diverse
set of 6.5K (image, question, answer) triplets and 2K visual-based multi-turn
conversations across 13 domains, including 6.2K egocentric images designed to
mimic captures from wearable devices. We carefully constructed the questions to
reflect real-world scenarios and challenges, including five types of
image-quality issues, six question types, varying entity popularity, differing
information dynamism, and different conversation turns. We design three tasks:
single-source augmentation, multi-source augmentation, and multi-turn
conversations -- each paired with an associated retrieval corpus and APIs for
both image-KG retrieval and webpage retrieval. Our evaluation shows that
straightforward RAG approaches achieve only 32% and 43% truthfulness on CRAG-MM
single- and multi-turn QA, respectively, whereas state-of-the-art industry
solutions have similar quality (32%/45%), underscoring ample room for
improvement. The benchmark has hosted KDD Cup 2025, attracting about 1K
participants and 5K submissions, with winning solutions improving baseline
performance by 28%, highlighting its early impact on advancing the field.