UPME: An Unsupervised Peer Review Framework for Multimodal Large
  Language Model Evaluation
Authors: name: Qihui Zhang, name: Munan Ning, name: Zheyuan Liu, name: Yanbo Wang, name: Jiayi Ye, name: Yue Huang, name: Shuo Yang, name: Xiao Chen, name: Yibing Song, name: Li Yuan

Multimodal Large Language Models (MLLMs) have emerged to tackle the
challenges of Visual Question Answering (VQA), sparking a new research focus on
conducting objective evaluations of these models. Existing evaluation methods
face limitations due to the significant human workload required to design Q&A
pairs for visual images, which inherently restricts the scale and scope of
evaluations. Although automated MLLM-as-judge approaches attempt to reduce the
human workload through automatic evaluations, they often introduce biases. To
address these problems, we propose an Unsupervised Peer review MLLM Evaluation
framework. It utilizes only image data, allowing models to automatically
generate questions and conduct peer review assessments of answers from other
models, effectively alleviating the reliance on human workload. Additionally,
we introduce the vision-language scoring system to mitigate the bias issues,
which focuses on three aspects: (i) response correctness; (ii) visual
understanding and reasoning; and (iii) image-text correlation. Experimental
results demonstrate that UPME achieves a Pearson correlation of 0.944 with
human evaluations on the MMstar dataset and 0.814 on the ScienceQA dataset,
indicating that our framework closely aligns with human-designed benchmarks and
inherent human preferences.