MLLM-Bench: Evaluating Multimodal LLMs with Per-Sample Criteria
Authors: Wentao Ge, Shunian Chen, Guiming Hardy Chen, Junying Chen, Zhihong Chen, Shuo Yan, ChenghaoZhu, Linziyue, Wenya Xie, Xinyi Zhang, Chai Yichen, Xiaoyu Liu, Song Dingjie, Xidong Wang, Anningzhe Gao, Zhang Zhiyi, Jianquan Li, Xiang Wan, Benyou Wang


Reviewer paper summaries: This paper discusses the evaluation of multimodal LLM. They propose a benchmark called MLLM-bench and evaluate samples across six critical domains (perception, understanding, applying, analyzing, evaluating, and creation). Based on this benchmark, they are able to have a leaderboard of state-of-the-art multimodal models. More detailed analysis shows that GPT-4V as a judger provides >0.8 correlation with human annotations.
Summary of strengths: 1. It provides a comprehensive benchmark with data points focusing on different aspects of the multimodal models.
2. Judgment based on GPT-4V allows the fast iteration and evaluation compared with other human evaluation methods. Additionally, compared with traditional evaluation methods based on bartscore or other non-ranking-based metrics, it provides a more accurate and comprehensive evaluation for multimodal models.
3. Different from reference-based evaluation, sample-specific criteria allows the evaluation on both simple and complex multimodal situations.
4. Technical details about the model are provided in details in the paper.
Summary of weaknesses: 1. The benchmark for evaluation can be too small to provide significantly different results among all different models, especially for those worse ones that almost always lose during the comparison. The failure of one models on some cases like OCR cases do not indicate it is definitely worse at OCR compared with the other model.
2. The GPT-4V as a judgement itself lacks enough novelty and is a widely used method for multimodal evaluation.
Comments, suggestions and typos: Q1: Is there any potential correlation between model and annotator's decision with the generated length or text style of the model?
Q2: Would the judgement show preference towards the generation of its own models when two outputs are similar? Would it bring GPT-4-V to a higher ranking place?
Q3: How many cases in the benchmark of any two models' comparison when two models are actually all incorrect but one model output is more fluent and easy to understand so it wins?
Q4: Is the judgement provided by GPT-4-V for each datapoint consistent each time or it can differ from time to time is uncertain and just random guessing?
Overall assessment scores: 3