Redundancy Principles for MLLMs Benchmarks
Authors: name: Zicheng Zhang; name: Xiangyu Zhao; name: Xinyu Fang; name: Chunyi Li; name: Xiaohong Liu; name: Xiongkuo Min; name: Haodong Duan; name: Kai Chen; name: Guangtao Zhai
Published: 2025-01-20 08:09:42

Abstract: With the rapid iteration of Multi-modality Large Language Models (MLLMs) and
the evolving demands of the field, the number of benchmarks produced annually
has surged into the hundreds. The rapid growth has inevitably led to
significant redundancy among benchmarks. Therefore, it is crucial to take a
step back and critically assess the current state of redundancy and propose
targeted principles for constructing effective MLLM benchmarks. In this paper,
we focus on redundancy from three key perspectives: 1) Redundancy of benchmark
capability dimensions, 2) Redundancy in the number of test questions, and 3)
Cross-benchmark redundancy within specific domains. Through the comprehensive
analysis over hundreds of MLLMs' performance across more than 20 benchmarks, we
aim to quantitatively measure the level of redundancy lies in existing MLLM
evaluations, provide valuable insights to guide the future development of MLLM
benchmarks, and offer strategies to refine and address redundancy issues
effectively.