Ask in Any Modality: A Comprehensive Survey on Multimodal
  Retrieval-Augmented Generation
Authors: name: Mohammad Mahdi Abootorabi; name: Amirhosein Zobeiri; name: Mahdi Dehghani; name: Mohammadali Mohammadkhani; name: Bardia Mohammadi; name: Omid Ghahroodi; name: Mahdieh Soleymani Baghshah; name: Ehsaneddin Asgari
Published: 2025-02-12 22:33:41

Abstract: Large Language Models (LLMs) struggle with hallucinations and outdated
knowledge due to their reliance on static training data. Retrieval-Augmented
Generation (RAG) mitigates these issues by integrating external dynamic
information enhancing factual and updated grounding. Recent advances in
multimodal learning have led to the development of Multimodal RAG,
incorporating multiple modalities such as text, images, audio, and video to
enhance the generated outputs. However, cross-modal alignment and reasoning
introduce unique challenges to Multimodal RAG, distinguishing it from
traditional unimodal RAG. This survey offers a structured and comprehensive
analysis of Multimodal RAG systems, covering datasets, metrics, benchmarks,
evaluation, methodologies, and innovations in retrieval, fusion, augmentation,
and generation. We precisely review training strategies, robustness
enhancements, and loss functions, while also exploring the diverse Multimodal
RAG scenarios. Furthermore, we discuss open challenges and future research
directions to support advancements in this evolving field. This survey lays the
foundation for developing more capable and reliable AI systems that effectively
leverage multimodal dynamic external knowledge bases. Resources are available
at https://github.com/llm-lab-org/Multimodal-RAG-Survey.