MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
Authors: name: Chaoyou Fu; name: Yi-Fan Zhang; name: Shukang Yin; name: Bo Li; name: Xinyu Fang; name: Sirui Zhao; name: Haodong Duan; name: Xing Sun; name: Ziwei Liu; name: Liang Wang; name: Caifeng Shan; name: Ran He
Published: 2024-11-22 18:59:54

Abstract: As a prominent direction of Artificial General Intelligence (AGI), Multimodal
Large Language Models (MLLMs) have garnered increased attention from both
industry and academia. Building upon pre-trained LLMs, this family of models
further develops multimodal perception and reasoning capabilities that are
impressive, such as writing code given a flow chart or creating stories based
on an image. In the development process, evaluation is critical since it
provides intuitive feedback and guidance on improving models. Distinct from the
traditional train-eval-test paradigm that only favors a single task like image
classification, the versatility of MLLMs has spurred the rise of various new
benchmarks and evaluation methods. In this paper, we aim to present a
comprehensive survey of MLLM evaluation, discussing four key aspects: 1) the
summarised benchmarks types divided by the evaluation capabilities, including
foundation capabilities, model self-analysis, and extented applications; 2) the
typical process of benchmark counstruction, consisting of data collection,
annotation, and precautions; 3) the systematic evaluation manner composed of
judge, metric, and toolkit; 4) the outlook for the next benchmark. This work
aims to offer researchers an easy grasp of how to effectively evaluate MLLMs
according to different needs and to inspire better evaluation methods, thereby
driving the progress of MLLM research.