LIME: Less Is More for MLLM Evaluation
Authors: King Zhu, Qianbo Zang, Shian Jia, Siwei Wu, Feiteng Fang, Yizhi LI, Shuyue Guo, Tianyu Zheng, Jiawei Guo, Bo Li, Haoning Wu, Xingwei Qu, Jian Yang, Ruibo Liu, Xiang Yue, Jiaheng Liu, Chenghua Lin, Hamid Alinejad-Rokny, Min Yang, Shiwen Ni, Wenhao Huang, Ge Zhang
Accepted at: acl25

LIME: Less Is More for MLLM Evaluation

King Zhu1,2*, Qianbo Zang2,9*, Shian Jia4*, Siwei Wu2,3*, Feiteng Fang10*, Yizhi Li2,3, Shuyue Guo2, Tianyu Zheng2, Bo Li6, Haoning Wu2, Xingwei Qu2, Jian Yang2, Zachary Liu7, Xiang Yue8, Jaheng Liu2, Chenghua Lin2,3, Min Yang10, Hamid Alinejad-Rokny5, Shiwen Ni10†, Wenhao Huang2‡, Ge Zhang2‡
1OPPO, 2M-A-P, 3University of Manchester, 4Zhejiang University, 5UNSW Sydney, 6Nanyang Technological University, 7Dartmouth College, 8Carnegie Mellon University, 9Interdisciplinary Centre for Security, Reliability and Trust, Université du Luxembourg, 10Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences

Abstract

Multimodal Large Language Models (MLLMs) are measured on numerous benchmarks like image captioning, visual question answer, and reasoning. However, these benchmarks often include overly simple or uninformative samples, making it difficult to effectively distinguish the performance of different MLLMs. Additionally, evaluating models across many benchmarks creates a significant computational burden. To address these issues, we propose LIME (Less Is More for MLLM Evaluation), a refined and efficient benchmark curated using a semi-automated pipeline. This pipeline filters out uninformative samples and eliminates answer leakage by focusing on tasks that require image-based understanding. Our experiments show that LIME reduces the number of samples by 76% and evaluation time by 70%, while it can more effectively distinguish different models’ abilities. Notably, we find that traditional automatic metrics like CIDEr are insufficient for evaluating MLLMs’ captioning performance, and excluding the caption task score yields a more accurate reflection of overall model performance. All code and data are available at https://github.com/kangreen0210/LIME.

1 Introduction

In order to better understand the model’s capabilities and guide addressing the shortcomings of MLLMs, researchers develop numerous benchmarks for various tasks (Antol et al., 2015; Wei et al., 2023; Fu et al., 2023; Yue et al., 2024; Wu et al., 2024a). These benchmarks thoroughly explore the capabilities of MLLMs in various tasks such as image captioning, image question answering, and multimodal retrieving.

However, existing MLLM benchmarks and unified evaluation frameworks cannot effectively and efficiently reflect the ability of MLLMs. Current benchmarks include numerous relatively simple samples (i.e., how many chairs are in the picture) and some incorrect questions caused by annotation issues. Most MLLMs consistently perform on these samples (i.e., all correct or all wrong). Therefore, those benchmarks cannot fully reflect the gap between different MLLMs and across various tasks. Besides, the current unified multimodal evaluation frameworks require significant computational resources, necessitating integrating much evaluation data from various benchmarks. The selection of effective evaluation data is largely overlooked by current researchers.

As shown in Figure 1, to address the aforementioned issues, we propose to use a general data process pipeline and curate a LIME, which contains 9403 samples and is refined across 10 tasks within 6 domains. We select six major tasks in the multimodal domain and use 9 MLLMs to refine those 10 benchmarks within the corresponding domain. To eliminate bias introduced by individual models, we choose 9 models as judges and filter samples based on their performance. On the one hand, we remove samples that most models answer correctly due to the fact that they cannot distinguish the capabilities among different models. On the other hand, we use a method that combines humans and MLLMs to filter out some abnormally difficult samples. Meanwhile, we use LLMs to filter out samples that can be answered directly from the context of the question. After that, we obtain a smaller yet higher-quality unified bench (i.e., LIME).

We conduct various experiments on LIME using both MLLMs and LLMs on different input settings, such as QA + image inputs, QA input (text-only input), and the QA + image description experiment. We make several valuable findings:

• LIME can better reflect the performance differences of MLLMs. On our LIME benchmark, under consistent conditions (same
Figure 1: Pipeline of the Data Curation. The left half part is the Open-Source Models as Judges module, which uses several Multimodal LLMs to answer questions for each sample and assess their difficulty. The upper right part is the Semi-Automated Screening Process module filtering some samples that are too simple or difficult. As for the Eliminating Answer Leakage, we filter the sample that can be answered without the image.

model series, same model size), different MLLMs demonstrate a wider score range, indicating that LIME is more effective at reflecting performance differences between models with a smaller amount of data.
• MLLMs exhibit varying capabilities across different subtasks. Specifically, they excel in the Visual Question Answering (VQA) subtasks, showcasing relatively high performance when answering questions directly related to factual information depicted in images. However, their performance is comparatively lower in tasks that necessitate the application of additional commonsense knowledge or complex reasoning. This highlights the significant image content recognition capabilities of current MLLMs.
• Through the correlation analysis of scores across different tasks, we find that using traditional automatic metrics for the captioning task makes it difficult to reasonably evaluate the model’s performance. Different tasks have varying requirements for factual perception and the application of additional commonsense knowledge in images.

2 Method

Most benchmarks contain low-quality, noisy data. Figure 2 shows the statistics of different subtasks within our LIME benchmark. It is worth mentioning that the proportion of easy and wrong samples exceeds 30%. Out of the 10 subtasks, 6 have proportions exceeding 50%. Notably, for the POPE dataset, 95% of the data can be classified as noisy or erroneous. This indicates that existing benchmarks are filled with a large amount of low-quality data, which does not accurately reflect the true capabilities of MLLMs.

Inspired by MMStar (Chen et al., 2024a), we utilize open-source MLLMs and LLMs as the judges for filtering, specifically, we remove the existing annotation errors. The overall pipeline consists of three main stages: (1) Using open-source models as judges, (2) A semi-automated screening process, and (3) Eliminating answer leakage. Our approach aims to improve existing benchmarks by removing inaccurate and oversimplified data.

2.1 Open-source Models as Judges

To avoid potential biases that may exist in individual MLLMs, we select ten different types of open-source models as judges. To categorize the difficulty of each sample, we analyze the performance of all judge models on each question and label the difficulty based on the number of models that answer correctly. We define \( N \) as the number of models that correctly answer the sample. If \( N \geq 6 \), the question is classified as the easy set. If \( 3 \leq N \leq 5 \), it is classified as the middle set. Conversely, if \( N \leq 2 \), it is classified as the hard set.
Figure 2: Overall data statics about selected subtasks. **Easy**: questions that most models can answer correctly, **Bad Case**: questions that may contain errors, **Remained**: questions that finally remain.

2.2 Semi-automated Screening Process

Easy samples do not effectively differentiate the capabilities of various models, as most models can answer them correctly. Therefore, we remove the easy samples to assess model performance better.

Furthermore, we find that some questions are not correctly answered by any model, which can be due to potential errors in the question design. To mitigate these potential errors and filter out totally incorrect questions, we implement a semi-automated screening process, which consists of two stages. In the first stage, all questions with zero passes are reviewed by GPT-4V to assess their correctness in terms of logic and meaning. In the second stage, questions deemed correct by GPT-4V are then manually screened. This strategy helps us eliminate meaningless or erroneous data from the dataset, thereby reducing its size and improving its quality.

2.3 Eliminating Answer Leakage

Although the previous two stages have filtered out potential errors and assessed the quality of the questions, we still need to address the potential issue of **ANSWER LEAKAGE**. Multimodal Answer Leakage can be summarized into two main categories: 1. *Text Answerable Questions*: The textual information contains all the necessary details to answer the question, making the corresponding visual information redundant. 2. *Seen Questions*: The MLLMs have encountered a specific question during training and has memorized the question along with its corresponding ground truth.

As for the *Seen Questions*, it has been removed in the Filtering Easy Sample module in Sec. 2.2. Therefore, we conduct a text-only check using pure text LLMs to eliminate the *ANSWER LEAKAGE*. Specifically, based on LLMs’ responses, we remove the samples that can be directly answered without using the image. After that, we proportionally sample 1,200 samples from these categories based on their difficulty levels. For benchmarks with fewer than 1,200 entries, we adapt all samples.

3 LIME: A comprehensive MLLMs benchmark

Table 1: Data statics: **Full Size**: the size of the original dataset, **Lite Size**: the final size of the LIME. For the COCO-Caption dataset, we selected the 2017 subset, and for the ScienceQA dataset, we chose the ScienceQA-IMG subset.

<table>
  <tr>
    <th>Task Domain</th>
    <th>Dataset</th>
    <th>Split</th>
    <th>Full Size</th>
    <th>Lite Size</th>
  </tr>
  <tr>
    <td rowspan="2">Captioning</td>
    <td>TextCaps</td>
    <td>val</td>
    <td>3166</td>
    <td>1200</td>
  </tr>
  <tr>
    <td>COCO-Caption</td>
    <td>val</td>
    <td>5000</td>
    <td>1200</td>
  </tr>
  <tr>
    <td>T/F reasoning</td>
    <td>POPE</td>
    <td>val</td>
    <td>9000</td>
    <td>443</td>
  </tr>
  <tr>
    <td rowspan="2">Normal VQA</td>
    <td>OK-VQA</td>
    <td>val</td>
    <td>5046</td>
    <td>1200</td>
  </tr>
  <tr>
    <td>TextVQA</td>
    <td>val</td>
    <td>5000</td>
    <td>1200</td>
  </tr>
  <tr>
    <td rowspan="2">Infographic QA</td>
    <td>infoVQA</td>
    <td>val</td>
    <td>2801</td>
    <td>1200</td>
  </tr>
  <tr>
    <td>ChartQA</td>
    <td>val</td>
    <td>2500</td>
    <td>1200</td>
  </tr>
  <tr>
    <td rowspan="2">Science QA</td>
    <td>ScienceQA</td>
    <td>val</td>
    <td>2097</td>
    <td>300</td>
  </tr>
  <tr>
    <td>AI2D</td>
    <td>val</td>
    <td>3088</td>
    <td>1000</td>
  </tr>
  <tr>
    <td>OCR</td>
    <td>OCRbench</td>
    <td>val</td>
    <td>1000</td>
    <td>460</td>
  </tr>
</table>
Figure 3: The number of samples removed at each stage compared to the original data, including three stages of filtering and the final sampling stage.

In this section, we propose LIME, a comprehensive benchmark for Multimodal Large Language Models (MLLMs). LIME streamlines existing mainstream benchmarks. Tab 1 shows the main datasets included in our benchmark, as well as the data scale after careful pruning. For each sub-dataset, we aim to keep the size around 1k samples.

3.1 Task Definition

We have categorized the existing mainstream tasks into six domains: Captioning, T/F Reasoning, Normal VQA, Infographic Understanding QA, Science QA, and OCR. Below are the task definitions for each domain.

Image understanding and captioning: This task focuses on the fundamental image-text understanding ability, requiring MLLMs to accurately describe and understand the content of images. This ability is commonly learned by most multimodal models during the pre-training stage. For example, the CLIP model aligns image and text features through contrastive learning, making Captioning a measure of the basic capabilities of MLLMs.

T/F reasoning: T/F Reasoning requires the model to judge the truthfulness of textual statements based on the image content. This not only demands basic image understanding but also requires a certain level of reasoning ability.

Normal VQA: Normal VQA, or Visual Question Answering, comprehensively evaluates the model’s ability to answer questions based on visual input. MLLMs are required to select the most appropriate answer from specific options.

Infographic Understanding QA: This task differs from Normal VQA as it tests the model’s ability to retrieve details from images. MLLMs need to find the most relevant information in the image related to the question and provide an answer.

Science QA: Science QA includes questions and answers related to natural science knowledge. This requires the model to have domain-specific knowledge in natural sciences, mainly assessing the MLLMs’ mastery of knowledge within a specific domain.

OCR: The OCR task requires the precise extraction of textual content from images.

3.2 Data Statistics

LIME is composed of 10 open-source multimodal evaluation benchmarks, with scales ranging from 1,000 to 9,000. After our three-stage data curation, the data scale of each benchmark is significantly reduced. Figure 3 shows the number of samples removed at each stage compared to the original dataset. The amount of data removed varies at each stage, with the most being removed in the first stage, reflecting a large number of low-difficulty or data-leakage samples in the existing 9 MLLMs. Comparing the data volumes before and after the second stage of semi-automated screening, we can see that many datasets, such as OK-VQA and TextVQA, have a high rate of low-quality data leading to MLLMs’ incorrect answers. Additionally, some datasets, such as ScienceQA and AI2D, have a significant amount of data removed after the third stage, indicating that many questions in these datasets may contain potential answer leakage. The statistics of the curated data are shown in Tab 1.

4 Experiment

4.1 Experiment Setting

To evaluate the quality of LIME, we conduct a series of experiments across various open-source and closed-source models. These experiments primarily encompass the following three settings:
Main experiment: To demonstrate LIME’s performance, we evaluate mainstream open-source and closed-source models using a standardized process to reflect their overall performance differences.

Text-only set: To prevent potential data leakage issues, we conduct experiments using text-only QA pairs. This verifies whether LLMs can correctly answer questions based on text-only information.

Text-only question with Image Description set: Image Description (ID) refers to simple descriptions of images that represent superficial information contained within them. For most MLLMs, questions containing only superficial information are easy to answer; however, questions requiring complex visual inference are significantly more challenging. To further validate whether LIME can reflect the capabilities of MLLMs, we input text-only QA pairs combined with ID into LLMs and test their ability.

4.2 Baselines

We select LLaVA-1.5 (Liu et al., 2023a,b), LLaVA-1.6 (Liu et al., 2024), Tinny-LLaVA (Zhou et al., 2024), MiniCPM (Hu et al., 2024), Idefics-2 1, Deepseek-VL(Lu et al., 2024), CogVLM (Wang et al., 2023; Hong et al., 2023), XComposer-4KHD (Zhang et al., 2023), Mantis (Jiang et al., 2024), InternVL-2 and InternVL-2.5 (Chen et al., 2023, 2024c,b) as our MLLMs baseline, and LLaMA3, Yi, Yi-1.5 (AI et al., 2024), Qwen-1.5 (Bai et al., 2023a) and Qwen2 (Yang et al., 2024) as LLMs baseline. To ensure fairness in the evaluations, we use the unified evaluation framework provided by Imms-eval (Zhang et al., 2024b) to conduct evaluation experiments on LIME. For models not supported by Imms-eval, we refine the inference code provided by the model developers to make it compatible with the new models for the sake of aligning the results of different models.

Metrics For most tasks included in LIME, we reference the metrics computation methods used in Imms-eval. Specifically, for tasks such as AI2D, ScienceQA, OCRBench, and POPE, we calculate the accuracy of the extracted responses. For tasks such as OK-VQA and TextVQA, we calculate the metric scores based on the overlap between the response and the candidate answers. For tasks like TextCaps and COCO-Caption2017, we use CIDEr as the score. The ANLS metric is used for the infoVQA task, and the Relaxed Overall metric is used for the ChartQA task.

We calculate the sub-scores for each task category by taking a weighted average of the sub-task scores, and then compute the overall score by weighted averaging the scores of all tasks except for the caption tasks. The details of the metrics calculation are provided in Tab 9.

5 Results

5.1 Main Result

As shown in Tab 2, we evaluate both open-source and closed-source MLLMs using our LIME benchmark. Overall, for closed-source models, GPT-4O achieves the best performance with a score of 52%, while for open-source models, models with larger parameter sizes and newer versions tend to have higher overall scores. InternVL series, and Qwen2.5-VL series achieve the best overall performance, with their overall scores all surpassing 60%. The performance of XComposer2, the CogVLM series, and the Cambrian series follows, with their overall scores ranging from 45% to 60%.

Comparing the overall scores of LIME and Origin benchmarks, we observe that certain model series, such as Cambrian and LLaVA-1.5, experience a decline in overall scores. Conversely, the CogVLM and LLaVA-OneVision series show an improvement, with CogVLM2 and XComposer-4KHD experiencing significant increases of 4% and 6%, respectively.

Tab 3 provides more detailed experimental results. Regarding caption subtasks, most models demonstrate good performance. These tasks involve generating or assessing descriptions of the content in images, which indicates that current MLLMs possess strong image content recognition capabilities. As for the VQA task, current MLLMs perform relatively well on TextVQA, ChatQA, and ScienceQA, where the questions directly ask about facts in the picture. However, their performance is relatively lower on OK-VQA, infoVQA, and AI2D, which require additional commonsense knowledge or complex reasoning to answer the questions. This demonstrates that current MLLMs exhibit significant image content recognition capabilities but are limited in their ability to perform complex reasoning using additional knowledge. We believe this limitation may be due to constraints in the language model component of MLLMs.

1https://huggingface.co/blog/idefics2
Table 2: Left half of the table: Comparing overall scores of LIME and Original. The arrow next to the LIME score indicates the change in ranking on LIME compared to the original dataset. ↑: upward shift, ↓: downward shift, and -: no change. Right half of the table: performance on six domains

<table>
  <tr>
    <th>Model</th>
    <th>Size</th>
    <th>LIME</th>
    <th>Original</th>
    <th>Reasoning</th>
    <th>VQA</th>
    <th>InfoQA</th>
    <th>SciQA</th>
    <th>OCR</th>
    <th>Caption</th>
  </tr>
  <tr>
    <td>GPT-4O 2024</td>
    <td>-</td>
    <td>52.63</td>
    <td>-</td>
    <td>47.18</td>
    <td>42.95</td>
    <td>57.63</td>
    <td>56.15</td>
    <td>72.39</td>
    <td>47.84</td>
  </tr>
  <tr>
    <td>claude-3-5-sonnet 2024</td>
    <td>-</td>
    <td>51.99</td>
    <td>-</td>
    <td>35.89</td>
    <td>50.33</td>
    <td>56.38</td>
    <td>44.69</td>
    <td>73.91</td>
    <td>28.00</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro-Vision 2024</td>
    <td>-</td>
    <td>49.46</td>
    <td>-</td>
    <td>54.63</td>
    <td>37.71</td>
    <td>55.33</td>
    <td>50.15</td>
    <td>73.26</td>
    <td>41.38</td>
  </tr>
  <tr>
    <td>GPT-4-Vision-Preview2023</td>
    <td>-</td>
    <td>42.23</td>
    <td>-</td>
    <td>42.44</td>
    <td>33.86</td>
    <td>48.00</td>
    <td>42.39</td>
    <td>55.22</td>
    <td>29.14</td>
  </tr>
  <tr>
    <td>Intern-VL-2.5 2024b</td>
    <td>26B</td>
    <td>67.19 (↑ 1)</td>
    <td>83.37</td>
    <td>57.79</td>
    <td>49.26</td>
    <td>80.54</td>
    <td>74.99</td>
    <td>78.04</td>
    <td>92.55</td>
  </tr>
  <tr>
    <td>Intern-VL-2 2023</td>
    <td>40B</td>
    <td>66.85 (↑ 2)</td>
    <td>80.31</td>
    <td>51.69</td>
    <td>48.72</td>
    <td>81.12</td>
    <td>75.92</td>
    <td>75.87</td>
    <td>56.02</td>
  </tr>
  <tr>
    <td>Owen2.5-VL 2025</td>
    <td>7B</td>
    <td>66.27 (-)</td>
    <td>81.18</td>
    <td>51.02</td>
    <td>48.05</td>
    <td>83.79</td>
    <td>66.16</td>
    <td>85.00</td>
    <td>36.78</td>
  </tr>
  <tr>
    <td>Intern-VL-2.5 2024b</td>
    <td>8B</td>
    <td>66.20 (↓ 3)</td>
    <td>83.73</td>
    <td>54.85</td>
    <td>51.05</td>
    <td>78.84</td>
    <td>71.28</td>
    <td>75.87</td>
    <td>92.80</td>
  </tr>
  <tr>
    <td>MAMmoTH-VL-8B-SI 2024</td>
    <td>8B</td>
    <td>65.38 (↓ 2)</td>
    <td>82.58</td>
    <td>37.25</td>
    <td>52.38</td>
    <td>80.59</td>
    <td>70.03</td>
    <td>67.83</td>
    <td>81.91</td>
  </tr>
  <tr>
    <td>Owen2-VL 2023b</td>
    <td>7B</td>
    <td>65.28 (↑ 1)</td>
    <td>79.14</td>
    <td>53.05</td>
    <td>51.37</td>
    <td>80.83</td>
    <td>62.08</td>
    <td>77.61</td>
    <td>89.67</td>
  </tr>
  <tr>
    <td>MAMmoTH-VL-8B 2024</td>
    <td>8B</td>
    <td>64.68 (↓ 1)</td>
    <td>80.29</td>
    <td>38.60</td>
    <td>50.69</td>
    <td>80.17</td>
    <td>70.01</td>
    <td>66.96</td>
    <td>80.47</td>
  </tr>
  <tr>
    <td>Intern-VL-1.5 2024c</td>
    <td>26B</td>
    <td>64.12 (↓ 1)</td>
    <td>79.49</td>
    <td>51.69</td>
    <td>52.68</td>
    <td>78.96</td>
    <td>63.32</td>
    <td>60.65</td>
    <td>90.93</td>
  </tr>
  <tr>
    <td>Intern-VL-2 2023</td>
    <td>26B</td>
    <td>63.98 (-)</td>
    <td>78.82</td>
    <td>54.63</td>
    <td>45.64</td>
    <td>79.12</td>
    <td>70.54</td>
    <td>71.09</td>
    <td>66.54</td>
  </tr>
  <tr>
    <td>Intern-VL-2 2023</td>
    <td>8B</td>
    <td>62.00 (↓ 2)</td>
    <td>77.84</td>
    <td>49.21</td>
    <td>45.15</td>
    <td>76.00</td>
    <td>68.54</td>
    <td>70.65</td>
    <td>34.00</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision 2024</td>
    <td>7B</td>
    <td>61.95 (↓ 1)</td>
    <td>78.71</td>
    <td>52.37</td>
    <td>51.27</td>
    <td>74.50</td>
    <td>66.77</td>
    <td>47.83</td>
    <td>106.46</td>
  </tr>
  <tr>
    <td>Owen2.5-VL 2025</td>
    <td>3B</td>
    <td>60.48 (↓ 1)</td>
    <td>77.99</td>
    <td>50.79</td>
    <td>41.52</td>
    <td>81.05</td>
    <td>55.92</td>
    <td>74.35</td>
    <td>88.57</td>
  </tr>
  <tr>
    <td>XComposer2-4KHD 2023</td>
    <td>7B</td>
    <td>57.52 (↓ 2)</td>
    <td>71.93</td>
    <td>46.28</td>
    <td>44.22</td>
    <td>73.29</td>
    <td>58.38</td>
    <td>53.04</td>
    <td>87.57</td>
  </tr>
  <tr>
    <td>CogVLM-2 2024</td>
    <td>19B</td>
    <td>54.44 (↑ 6)</td>
    <td>69.93</td>
    <td>51.02</td>
    <td>37.19</td>
    <td>69.92</td>
    <td>54.00</td>
    <td>68.26</td>
    <td>28.84</td>
  </tr>
  <tr>
    <td>Owen2-VL 2023b</td>
    <td>2B</td>
    <td>54.00 (↑ 4)</td>
    <td>70.86</td>
    <td>50.79</td>
    <td>43.78</td>
    <td>66.25</td>
    <td>46.38</td>
    <td>68.04</td>
    <td>88.39</td>
  </tr>
  <tr>
    <td>CogVLM-1 2023</td>
    <td>17B</td>
    <td>51.03 (↑ 1)</td>
    <td>71.34</td>
    <td>55.10</td>
    <td>51.45</td>
    <td>59.46</td>
    <td>36.54</td>
    <td>41.96</td>
    <td>33.92</td>
  </tr>
  <tr>
    <td>Cambrian 2024</td>
    <td>34B</td>
    <td>50.17 (↓ 4)</td>
    <td>73.26</td>
    <td>49.44</td>
    <td>39.66</td>
    <td>57.50</td>
    <td>60.23</td>
    <td>39.13</td>
    <td>4.62</td>
  </tr>
  <tr>
    <td>Cambrian 2024</td>
    <td>13B</td>
    <td>48.57 (↓ 4)</td>
    <td>72.39</td>
    <td>50.79</td>
    <td>41.53</td>
    <td>56.04</td>
    <td>49.23</td>
    <td>42.39</td>
    <td>6.96</td>
  </tr>
  <tr>
    <td>Cambrian 2024</td>
    <td>8B</td>
    <td>47.95 (↓ 3)</td>
    <td>71.84</td>
    <td>49.89</td>
    <td>42.12</td>
    <td>53.55</td>
    <td>49.46</td>
    <td>43.04</td>
    <td>6.13</td>
  </tr>
  <tr>
    <td>LLaVA-1.6 2024</td>
    <td>34B</td>
    <td>44.06 (↓ 3)</td>
    <td>67.22</td>
    <td>47.00</td>
    <td>30.80</td>
    <td>53.21</td>
    <td>53.08</td>
    <td>37.17</td>
    <td>66.25</td>
  </tr>
  <tr>
    <td>MiniCPM-LLaMA3-2.5 2024</td>
    <td>8B</td>
    <td>42.61 (↓ 3)</td>
    <td>71.22</td>
    <td>43.10</td>
    <td>43.55</td>
    <td>58.55</td>
    <td>6.60</td>
    <td>55.87</td>
    <td>35.89</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision 2024</td>
    <td>0.5B</td>
    <td>41.40 (↓ 4)</td>
    <td>65.65</td>
    <td>48.98</td>
    <td>35.87</td>
    <td>48.04</td>
    <td>36.23</td>
    <td>42.83</td>
    <td>93.34</td>
  </tr>
  <tr>
    <td>LLaVA-LLaMA3 2023</td>
    <td>8B</td>
    <td>40.90 (↓ 2)</td>
    <td>69.74</td>
    <td>44.24</td>
    <td>37.36</td>
    <td>43.33</td>
    <td>45.56</td>
    <td>30.22</td>
    <td>74.03</td>
  </tr>
  <tr>
    <td>Mantis-Idefics-2 2024</td>
    <td>8B</td>
    <td>39.25 (-)</td>
    <td>66.91</td>
    <td>44.24</td>
    <td>36.79</td>
    <td>39.75</td>
    <td>43.69</td>
    <td>32.17</td>
    <td>82.44</td>
  </tr>
  <tr>
    <td>Deepseek-VL 2024</td>
    <td>7B</td>
    <td>38.10 (↓ 2)</td>
    <td>65.62</td>
    <td>48.50</td>
    <td>34.90</td>
    <td>38.50</td>
    <td>44.23</td>
    <td>25.43</td>
    <td>68.72</td>
  </tr>
  <tr>
    <td>LLaVA-1.6-vicuna 2024</td>
    <td>13B</td>
    <td>37.08 (↓ 4)</td>
    <td>67.29</td>
    <td>43.10</td>
    <td>30.00</td>
    <td>41.63</td>
    <td>41.54</td>
    <td>31.96</td>
    <td>62.23</td>
  </tr>
  <tr>
    <td>Idefics-2 2024</td>
    <td>8B</td>
    <td>36.39 (↓ 2)</td>
    <td>66.73</td>
    <td>42.00</td>
    <td>46.05</td>
    <td>18.50</td>
    <td>47.46</td>
    <td>42.61</td>
    <td>77.87</td>
  </tr>
  <tr>
    <td>LLaVA-1.6-vicuna 2024</td>
    <td>7B</td>
    <td>30.15 (-)</td>
    <td>64.80</td>
    <td>41.10</td>
    <td>25.75</td>
    <td>32.88</td>
    <td>31.77</td>
    <td>23.70</td>
    <td>62.20</td>
  </tr>
  <tr>
    <td>Mantis-SigILP 2024</td>
    <td>8B</td>
    <td>29.13 (↑ 1)</td>
    <td>58.96</td>
    <td>45.60</td>
    <td>29.39</td>
    <td>25.79</td>
    <td>35.77</td>
    <td>10.65</td>
    <td>74.69</td>
  </tr>
  <tr>
    <td>MiniCPM-V 2024</td>
    <td>3B</td>
    <td>26.15 (↓ 2)</td>
    <td>56.18</td>
    <td>44.00</td>
    <td>21.60</td>
    <td>24.58</td>
    <td>35.46</td>
    <td>14.57</td>
    <td>72.80</td>
  </tr>
  <tr>
    <td>LLaVA-1.5 2023a</td>
    <td>13B</td>
    <td>20.38 (↓ 2)</td>
    <td>59.58</td>
    <td>36.60</td>
    <td>25.80</td>
    <td>8.96</td>
    <td>31.08</td>
    <td>5.87</td>
    <td>74.81</td>
  </tr>
  <tr>
    <td>LLaVA-1.5 2023a</td>
    <td>7B</td>
    <td>17.20 (↓ 1)</td>
    <td>57.27</td>
    <td>32.51</td>
    <td>19.97</td>
    <td>7.17</td>
    <td>29.81</td>
    <td>4.78</td>
    <td>72.47</td>
  </tr>
  <tr>
    <td>InstructBLIP-vicuna 2023</td>
    <td>7B</td>
    <td>15.55 (-)</td>
    <td>47.87</td>
    <td>45.10</td>
    <td>16.75</td>
    <td>6.04</td>
    <td>24.77</td>
    <td>4.35</td>
    <td>77.61</td>
  </tr>
  <tr>
    <td>Tiny-LLaVA-1 2024</td>
    <td>1.4B</td>
    <td>13.95 (-)</td>
    <td>34.30</td>
    <td>37.00</td>
    <td>9.80</td>
    <td>8.33</td>
    <td>27.85</td>
    <td>3.48</td>
    <td>61.05</td>
  </tr>
</table>

5.2 Correlation Analysis

Figure 4 illustrates the correlation between the various sub-tasks in LIME and WildVision Bench. Most tasks in LIME exhibit a strong positive correlation with WildVision Bench. Six subtasks have correlation scores exceeding 80%. Additionally, the overall score of LIME correlates at 91% with WV-Elo, which is higher than any individual sub-task and the original bench’s correlations, demonstrating that LIME provides a more comprehensive reflection of MLLMs’ capabilities.

Automated evaluation metrics (e.g. CIDEr) cannot effectively assess the performance of MLLMs in captioning tasks. As an early foundational problem, the captioning task is extensively studied, and MLLMs demonstrate exceptional ability in this task. For instance, earlier models like InstructBlip perform exceptionally well on captioning tasks, and there is a broad presence of training data for image captioning in MLLMs’ training processes. However, the captioning task shows a negative correlation with all other sub-tasks. This indicates that previous metrics (e.g., BLEU, CIDEr) only focus on the overlap between the model-generated responses and the ground truth, but do not consider that MLLMs might generate content that is semantically close to the ground truth (i.e., the model-generated response may be semantically similar to the ground truth but expressed differently, or the model may generate more detailed content about the image). Consequently, we exclude it from the overall score calculation.

There is a certain degree of correlation between the sub-tasks in LIME. On the one hand, the relevance of TextVQA, InfographicVQA, and OCR-Bench is relatively high. As shown in Fig 4, the
Figure 4: Correlation distribution between LIME and Wildvson Elo.

correlation of these tasks all surpasses 85%, and these two VQA tasks require MLLMs to understand fine-grained content in images to answer questions. This demonstrates that OCR tasks also rely on the ability of MLLMs to perceive fine-grained objective facts in images. On the other hand, POPE, ChartQA, and InfographicVQA all require reasoning abilities using extra commonsense knowledge. The correlation scores of these tasks are all above 70%, and POPE requires the model to use extra knowledge to solve the hallucination of MLLMs. We assume that ChartQA and infoVQA may also necessitate the use of additional common knowledge by the models to solve problems.

5.3 Effectiveness of LIME

LIME provides a more challenging evaluation for MLLMs. As shown in Tab 2, the MLLMs’ performances on LIME are less than those on the Original Bench for most tasks, and Tab 8 demonstrates LIME’s advantage in evaluation costs, as LIME reduces 70% of the evaluation time and resource expenditure, indicating that our LIME can better reflect the performance differences between models with a smaller amount of cost.

Furthermore, we compare the score variations across different model series and model sizes. Figure 5 illustrates a clear positive correlation between model performance and model size within the same model series. Notably, LIME exhibits a more dispersed score distribution, effectively highlighting the differences in model performance. In Tab 4 and 5, the Gini coefficient and standard deviation are used to measure the differences in overall score distribution across the same model series and model sizes. The larger the Gini coefficient and standard deviation, the greater the disparity in data distribution. It can be observed that, whether within the same model series or the same model size, LIME achieves higher Gini and standard deviation values compared to the original bench. This indicates that LIME can better differentiate the performance differences between various models.

LIME eliminates potential data leakage. For multimodal question answering tasks, visual information input is essential, and LLMs are unable to provide correct answers due to they cannot perceive the content within the image. However, as shown in Fig 6, there are severe data leakage issues in the original Bench for the AI2D and ScienceQA tasks. The average score for AI2D is close to 55%, and for ScienceQA, it exceeds 60%, which shows that data from AI2D and ScienceQA in Original are highly likely to have been exposed to the training data of LLMs. In contrast, the LIME has eliminated this potential threat, achieving scores below 25% in AI2D and close to 40% in ScienceQA.

LIME requires MLLMs to perceive deeper levels of image information. As shown in Tab 6, to investigate whether the remaining samples need to be answered by using textual and image information, we conduct experiments using LLMs to generate answers on both the Original Benchmark and LIME under QID (question + image description) setting. Especially in tasks such as AI2D, OCR-Bench, and TCaps, the scores of LLMs on LIME are significantly lower than on the Original Benchmark when provided with only the questions and simple image descriptions. This indicates that, after removing some of the simpler questions, LIME is better at testing the models’ ability to perceive image details.

6 Related work

In recent years, there has been increasing attention on establishing evaluation benchmarks to assess the performance of MLLMs in different scenarios to guide the development of MLLMs. Early multi-modal evaluation benchmarks primarily focused on single tasks, such as Visual Question Answering (VQA)(Antol et al., 2015; Goyal et al., 2017; Kafle and Kanan, 2017; Singh et al., 2019; Marino et al., 2019), Image Captioning(Agrawal et al., 2019), and Information Retrieval (Wei et al., 2023). As MLLMs develop, simple benchmarks are no longer
Figure 5: with the same series of models, the distribution differences of various Parameter sizes. Left(★): LLaVA-1.6 series, Right(▲): InternVL-2 series

Figure 6: Comparing text only results of LIME and original bench.

sufficient to evaluate the versatile capabilities of these models comprehensively, since most MLLMs demonstrate exceptional ability on those benchmarks. Consequently, numerous more difficult and diverse benchmarks have emerged in recent years to assess the capabilities of MLLMs comprehensively. For instance, MMMU (Yue et al., 2024) and CMMMU (Zhang et al., 2024a) are comprehensive benchmark tests for university-level multi-disciplinary multimodal understanding and reasoning. MMBench (Liu et al., 2023c) has developed a comprehensive evaluation pipeline that offers fine-grained capability assessment and robust evaluation metrics. MMRA (Wu et al., 2024b) establishes an association system among images to assess the multi-image perception ability of MLLMs.

However, those benchmarks cannot distinguish the performance gaps among different models excellently, as they still contain some too simple or difficult samples that most models yield the same results on. Furthermore, training datasets across different models may contain the samples of those benchmarks, which results in data leakage issues (Fu et al., 2023). Mmstar (Chen et al., 2024a) and MMLU_Redux (Gema et al., 2024) have identified several issues within current benchmarks. Mmstar (Chen et al., 2024a) proposes an automated pipeline to filter benchmark data, aiming to detect potential data leakage, while MMLU_Redux (Gema et al., 2024) focuses on correcting annotation errors. However, there is still a pressing need for a comprehensive pipeline that fully addresses the challenges posed by multimodal datasets. In response to this, we introduce LIME: LESS IS MORE FOR MLLM EVALUATION. We have carefully selected six task types from existing mainstream benchmarks and scaled them down according to clear guidelines. This streamlined version retains the core elements of mainstream MLLM benchmarks, providing a more efficient and focused evaluation.

7 Conclusion

As MLLMs continue to advance, a notable absence of convenient and high-quality multimodal benchmarks has emerged. In response to this, we propose a pipeline aimed at semi-automatically refining existing benchmarks to enhance their quality, culminating in the development of LIME, which comprises 9,403 evaluation samples across 6 types of tasks and 10 different benchmark datasets. By refining the original benchmarks to filter question difficulty and eliminate potentially problematic items, LIME offers a more rigorous evaluation for MLLMs, necessitating a deeper understanding of image information. The experiments result demonstrate the heightened challenge posed by LIME for MLLMs. We anticipate that our approach will contribute to the advancement of MLLM evaluation
systems, and we are committed to continually enriching LIME with an expanded array of datasets through regular updates and expansions.

Limitations

Our LIME has effectively improved the credibility of mainstream MLLM benchmarks. However, we must be aware that as new benchmarks better suited for MLLM evaluation emerge, our current system may face obsolescence. To address this, we have established an annual update mechanism that integrates 2 to 3 new high-quality new benchmarks into our LIME per year.

Ethical Statement

Our research adheres to ethical standards by ensuring that all data used in our study are sourced from publicly available benchmark datasets, following established legal and ethical guidelines. These datasets are obtained from reputable sources, and no proprietary, private, or sensitive information is used in our work. We are committed to the transparency and integrity of our research, ensuring that all data used are shared and accessible under appropriate licenses. The licenses associated with the datasets include Attribution-ShareAlike 4.0 International, MIT License, Berkeley Software Distribution (BSD), and Apache License Version 2.0. These licenses allow re-users to distribute, remix, adapt, and build upon the material in any medium or format, provided that proper attribution is given to the original creators. This ensures that our research fosters open collaboration and encourages the responsible use of shared resources. We did not outsource any data annotation (double check process) in our LIME. All data annotation was done by authors.

Acknowledgments

This research was funded by OPPO Ltd. and Multimodal Art Projection. Qianbo Zang is currently funded by Enovos Luxembourg S.A. and PayPal, PEARL grant reference 13342933/Gilbert Fridgen. Shiwen Ni was supported by GuangDong Basic and Applied Basic Research Foundation (2023A1515110718 and 2024A1515012003), China Postdoctoral Science Foundation (2024M753398), Postdoctoral Fellowship Program of CPSF (GZC20232873).

References

Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2019. No-caps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948–8957.

01. AI. .; Alex Young, Bei Chen, Chao Li, Chengu Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. 2024. Yi: Open foundation models by 01.ai. Preprint, arXiv:2403.04652.

Anthropic. 2024. Claude 3.5 sonnet model card addendum. Accessed: 2024-09-21.

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xinxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xianhuan Zhou, and Tianhang Zhu. 2023a. Qwen technical report. arXiv preprint arXiv:2309.16609.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023b. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966.

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. 2024a. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330.

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. 2024b. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024c. How far
are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238.

XTuner Contributors. 2023. Xtuner: A toolkit for efficiently fine-tuning llm. https://github.com/InternLM/xtuner.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. Preprint, arXiv:2305.06500.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394.

Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. 2024. Are we done with mmlu? arXiv preprint arXiv:2406.04127.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913.

Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhui Chen, and Xiang Yue. 2024. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint arXiv:2412.05237.

Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. 2024. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408.16500.

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. 2023. Co-gagent: A visual language model for gui agents. Preprint, arXiv:2312.08914.

Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. 2024. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395.

Dongfu Jiang, Xuan He, Huaye Zeng, Con Wei, Max Ku, Qian Liu, and Wenhui Chen. 2024. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483.

Kushal Kafle and Christopher Kanan. 2017. An analysis of visual question answering algorithms. In Proceedings of the IEEE international conference on computer vision, pages 1965–1973.

Hugo Laurencón, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024. What matters when building vision-language models? arXiv preprint arXiv:2405.02246.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. Llava-next: Improved reasoning, ocr, and world knowledge.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023c. Mm-bench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281.

Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhushou Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. 2024. Deepseek-vl: Towards real-world vision-language understanding. Preprint, arXiv:2403.05525.

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204.

Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. 2024. Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures. arXiv preprint arXiv:2406.06565.

OpenAI. 2023. Gpt-4v(ision) system card.

OpenAI. 2024. Gpt-4o system card. Technical report, OpenAI. https://www.openai.com/research/gpt-4o.

N Reimers. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326.

Gemini Team, Petko Georgiev, and Ving Ian Lei. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Preprint, arXiv:2403.05530.

Qwen Team. 2025. Qwen2.5-v1.

Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. 2024. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860.

Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. 2023. Cogvlm: Visual expert for pretrained language models. Preprint, arXiv:2311.03079.

Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhui Chen. 2023. Unir: Training and benchmarking universal multimodal information retrievers. arXiv preprint arXiv:2311.17136.

Siwei Wu, Yizhi Li, Kang Zhu, Ge Zhang, Yiming Liang, Kaijing Ma, Chenghao Xiao, Haoran Zhang, Bohao Yang, Wenhui Chen, Wenhao Huang, Noura Al Moubyad, Jie Fu, and Chenghua Lin. 2024a. SciMMIR: Benchmarking scientific multi-modal information retrieval. In Findings of the Association for Computational Linguistics ACL 2024, pages 12560–12574.

Siwei Wu, Kang Zhu, Yu Bai, Yiming Liang, Yizhi Li, Haoning Wu, Jaheng Liu, Ruibo Liu, Xingwei Qu, Xuxin Cheng, et al. 2024b. Mmra: A benchmark for multi-granularity multi-image relational association. arXiv preprint arXiv:2407.17379.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint 2408.01800.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567.

Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, et al. 2024a. Cmmu: A chinese massive multi-discipline multi-modal understanding benchmark. arXiv preprint arXiv:2401.11944.

Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. 2024b. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772.

Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. 2023. Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112.

Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. 2024. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402.14289.
A Appendix

A.1 overall Data statics
Figure 7 shows the overall data distribution in LIME, and figure 8 shows an example for each category title

![Pie chart showing the percentage distribution of different categories in LIME](page_246_352_956_482.png)

Figure 7: The overall percentage distribution of LIME.

<table>
  <tr>
    <th>T/F Reasoning</th>
    <th>Ocr</th>
    <th>Captioning</th>
  </tr>
  <tr>
    <td>Question: Is there a bottle in the image?<br>Answer: False<br><br>Subcategory: pope<br>Difficulty Level: middle</td>
    <td>Question: Where is the location that is a quarter miles far from here?<br>Answer: East Dunne Ave<br><br>Subcategory: ocrbench<br>Difficulty Level: middle</td>
    <td>Question: Please carefully observe the image and come up with a caption for the image.<br>Answer: 3 young children in karate uniforms named Gracie Barra are raising arms in victory.<br><br>Subcategory: TextCaps<br>Difficulty Level: hard</td>
  </tr>
  <tr>
    <th>Science qa</th>
    <th>Normal VQA</th>
    <th>Inforgraphic QA</th>
  </tr>
  <tr>
    <td>Question: Which of these states is farthest east?<br>Options:<br>A. Florida<br>B. New York<br>C. New Hampshire<br>D. Iowa<br><br>Subcategory: scienceqa<br>Difficulty Level: middle</td>
    <td>Question: What sport can you use this for?<br>Answer:<br>(1) motocross<br>(2) racing<br>(3) riding<br><br>Subcategory: OK-VQA<br>Difficulty Level: middle</td>
    <td>Question: How many years are represented on this graph?<br>Answer: 13<br><br>Subcategory: Chart-QA<br>Difficulty Level: hard</td>
  </tr>
</table>

Figure 8: The overview of LIME.
A.2 more experiment result

Table 3: Comparing overall scores of LIME and Original. Top: results on LIME, Bottom: results on the original dataset. The arrow next to the model name indicates the change in ranking on LIME compared to the original dataset. ↑: upward shift, ↓: downward shift, and -: no change.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Size</th>
    <th colspan="2">Overall</th>
    <th colspan="2">Top</th>
    <th colspan="2">Common VQA</th>
    <th colspan="2">InfoVQA</th>
    <th colspan="2">ScienceQA</th>
    <th colspan="2">OCR</th>
    <th colspan="2">Captioning</th>
  </tr>
  <tr>
    <th>POPE</th>
    <th>TPO</th>
    <th>POPE</th>
    <th>TPO</th>
    <th>OKVQA</th>
    <th>ChE</th>
    <th>IOVA</th>
    <th>AIQA</th>
    <th>SQQA</th>
    <th>OCRCon</th>
    <th>COGCon</th>
    <th>TCGCon</th>
    <th>COCO</th>
    <th>TCG</th>
  </tr>
  <tr><td>InternVL-2.5 (2021) (↑)</td><td>26B</td><td>67.19</td><td>70.07</td><td>58.45</td><td>65.30</td><td>75.34</td><td>88.71</td><td>88.75</td><td>80.00</td><td>91.76</td><td>73.07</td><td>73.07</td><td>73.07</td><td>73.07</td><td>73.07</td></tr>
  <tr><td>InternVL-2.2021 (↑)</td><td>40B</td><td>66.85</td><td>51.69</td><td>77.98</td><td>19.45</td><td>88.33</td><td>73.92</td><td>69.20</td><td>98.33</td><td>75.87</td><td>63.10</td><td>48.94</td><td></td><td></td><td></td></tr>
  <tr><td>Qwen2.5-VL (2025) (-)</td><td>7B</td><td>66.27</td><td>51.02</td><td>78.66</td><td>17.43</td><td>84.5</td><td>83.08</td><td>63</td><td>76.71</td><td>85</td><td>28.84</td><td>44.71</td><td></td><td></td><td></td></tr>
  <tr><td>InternVL-2.5S2024(↓)</td><td>8B</td><td>66.20</td><td>54.85</td><td>70.18</td><td>31.92</td><td>85.92</td><td>71.75</td><td>63.9</td><td>95.89</td><td>73.87</td><td>93.74</td><td>91.85</td><td></td><td></td><td></td></tr>
  <tr><td>Qwen2.5-VL (2025) (-)</td><td>7B</td><td>65.28</td><td>54.76</td><td>74.56</td><td>32.17</td><td>83.17</td><td>75.20</td><td>58.00</td><td>75.90</td><td>73.81</td><td>68.51</td><td>61.60</td><td></td><td></td><td></td></tr>
  <tr><td>InternVL-1.5 2024c (↓)</td><td>26B</td><td>64.12</td><td>51.69</td><td>69.88</td><td>35.47</td><td>87.00</td><td>70.92</td><td>54.81</td><td>91.67</td><td>60.65</td><td>69.24</td><td>112.63</td><td></td><td></td><td></td></tr>
  <tr><td>InternVL-2.2023 (-)</td><td>26B</td><td>63.98</td><td>54.63</td><td>75.20</td><td>16.08</td><td>87.67</td><td>70.58</td><td>62.80</td><td>96.33</td><td>71.09</td><td>76.18</td><td>56.91</td><td></td><td></td><td></td></tr>
  <tr><td>InternVL-2.2023 (↑)</td><td>8B</td><td>62.00</td><td>49.21</td><td>66.10</td><td>24.20</td><td>83.75</td><td>68.22</td><td>60.40</td><td>95.67</td><td>70.65</td><td>42.55</td><td>23.44</td><td></td><td></td><td></td></tr>
  <tr><td>LLaMA-1.5 (2024) (↑)</td><td>1.4B</td><td>61.95</td><td>46.55</td><td>65.22</td><td>18.00</td><td>83.83</td><td>59.40</td><td>52.90</td><td>92.85</td><td>60.43</td><td>104.16</td><td>104.16</td><td></td><td></td><td></td></tr>
  <tr><td>XComposer-4KHD 2023 (↑)</td><td>3B</td><td>60.48</td><td>50.79</td><td>69.27</td><td>13.77</td><td>84.42</td><td>77.67</td><td>54</td><td>62.33</td><td>74.35</td><td>77.21</td><td>99.93</td><td></td><td></td><td></td></tr>
  <tr><td>CogVLM-2.5 (2024) (↑)</td><td>7B</td><td>57.52</td><td>46.28</td><td>60.30</td><td>28.13</td><td>80.42</td><td>66.17</td><td>54.90</td><td>70.00</td><td>53.04</td><td>97.07</td><td>78.07</td><td></td><td></td><td></td></tr>
  <tr><td>CogVLM-2.5 (2024) (↑)</td><td>19B</td><td>54.44</td><td>51.02</td><td>69.46</td><td>4.92</td><td>80.33</td><td>59.50</td><td>45.00</td><td>84.00</td><td>68.26</td><td>23.67</td><td>34.01</td><td></td><td></td><td></td></tr>
  <tr><td>CogVLM-2.5 (2024) (↑)</td><td>2B</td><td>50.90</td><td>47.70</td><td>61.57</td><td>6.50</td><td>80.70</td><td>58.60</td><td>38.00</td><td>86.00</td><td>65.00</td><td>16.71</td><td>17.22</td><td></td><td></td><td></td></tr>
  <tr><td>CogVLM-1.2 (2024) (↑)</td><td>17B</td><td>51.03</td><td>55.10</td><td>71.20</td><td>31.70</td><td>61.57</td><td>57.25</td><td>31.40</td><td>53.67</td><td>31.96</td><td>29.28</td><td>38.56</td><td></td><td></td><td></td></tr>
  <tr><td>Cambrian 2024 (↓)</td><td>34B</td><td>50.17</td><td>49.44</td><td>57.28</td><td>22.03</td><td>71.83</td><td>43.17</td><td>54.80</td><td>78.33</td><td>39.13</td><td>4.27</td><td>4.97</td><td></td><td></td><td></td></tr>
  <tr><td>Cambrian 2024 (↓)</td><td>13B</td><td>48.57</td><td>50.79</td><td>58.93</td><td>24.13</td><td>69.25</td><td>42.83</td><td>45.20</td><td>62.67</td><td>42.39</td><td>7.40</td><td>6.52</td><td></td><td></td><td></td></tr>
  <tr><td>Cambrian 2024 (↓)</td><td>8B</td><td>48.95</td><td>59.90</td><td>59.00</td><td>24.13</td><td>69.25</td><td>42.83</td><td>45.20</td><td>62.67</td><td>42.39</td><td>7.40</td><td>6.52</td><td></td><td></td><td></td></tr>
  <tr><td>LLaMA-1.6 (2024) (↑)</td><td>34B</td><td>44.06</td><td>47.00</td><td>51.20</td><td>10.40</td><td>64.33</td><td>42.08</td><td>49.60</td><td>64.67</td><td>37.17</td><td>84.25</td><td>48.25</td><td></td><td></td><td></td></tr>
  <tr><td>MiniCPM-LLaMA-2.5 2024 (↓)</td><td>3B</td><td>42.61</td><td>43.10</td><td>61.80</td><td>25.30</td><td>69.00</td><td>48.10</td><td>6.60</td><td>6.60</td><td>55.87</td><td>31.91</td><td>39.96</td><td></td><td></td><td></td></tr>
  <tr><td>LLaMA-OneVision 2024 (↑)</td><td>0.5B</td><td>41.40</td><td>48.88</td><td>48.61</td><td>23.13</td><td>55.42</td><td>40.67</td><td>31.20</td><td>53.00</td><td>42.83</td><td>96.40</td><td>96.28</td><td></td><td></td><td></td></tr>
  <tr><td>LLaMA-OneVision 2024 (↑)</td><td>8B</td><td>40.90</td><td>40.01</td><td>56.40</td><td>16.42</td><td>64.33</td><td>42.08</td><td>49.60</td><td>64.67</td><td>37.17</td><td>84.25</td><td>48.25</td><td></td><td></td><td></td></tr>
  <tr><td>Mantis-Idfes-2 2024 (↓)</td><td>8B</td><td>39.25</td><td>44.24</td><td>44.51</td><td>29.07</td><td>59.00</td><td>20.50</td><td>35.80</td><td>70.00</td><td>32.17</td><td>61.82</td><td>103.07</td><td></td><td></td><td></td></tr>
  <tr><td>Deepseek-VL 2024 (↑)</td><td>7B</td><td>38.10</td><td>48.50</td><td>44.20</td><td>25.00</td><td>54.67</td><td>23.33</td><td>37.00</td><td>68.33</td><td>25.53</td><td>54.22</td><td>83.21</td><td></td><td></td><td></td></tr>
  <tr><td>LLaMA-1.6-vicuna 2024 (↓)</td><td>13B</td><td>37.08</td><td>43.10</td><td>43.90</td><td>16.10</td><td>54.50</td><td>28.75</td><td>38.10</td><td>53.00</td><td>31.96</td><td>76.62</td><td>47.83</td><td></td><td></td><td></td></tr>
  <tr><td>LLaMA-1.6-vicuna 2024 (↓)</td><td>8B</td><td>36.29</td><td>46.50</td><td>56.30</td><td>13.00</td><td>51.80</td><td>17.80</td><td>37.60</td><td>76.50</td><td>21.61</td><td>23.45</td><td>85.45</td><td></td><td></td><td></td></tr>
  <tr><td>LLaMA-1.6-vicuna 2024 (↓)</td><td>30.15</td><td>41.10</td><td>39.00</td><td>12.50</td><td>43.08</td><td>22.70</td><td>27.10</td><td>47.33</td><td>23.70</td><td>76.05</td><td>48.35</td><td></td><td></td><td></td></tr>
  <tr><td>Mantis-SigLIP 2024 (↑)</td><td>8B</td><td>29.13</td><td>45.60</td><td>26.34</td><td>32.43</td><td>35.33</td><td>16.25</td><td>27.70</td><td>62.67</td><td>10.65</td><td>68.16</td><td>81.21</td><td></td><td></td><td></td></tr>
  <tr><td>MiniCPM-V 2024 (↑)</td><td>3B</td><td>26.15</td><td>44.00</td><td>37.00</td><td>6.20</td><td>35.15</td><td>13.42</td><td>27.90</td><td>60.67</td><td>14.57</td><td>68.96</td><td>76.65</td><td></td><td></td><td></td></tr>
  <tr><td>LLaMA-1.5 2024 (↑)</td><td>13B</td><td>25.88</td><td>49.50</td><td>19.50</td><td>5.30</td><td>5.30</td><td>4.83</td><td>4.83</td><td>8.08</td><td>8.08</td><td>8.08</td><td>8.08</td><td></td><td></td><td></td></tr>
  <tr><td>LLaMA-1.5 2024 (↑)</td><td>7B</td><td>17.20</td><td>32.51</td><td>16.50</td><td>23.53</td><td>5.25</td><td>9.08</td><td>24.05</td><td>49.00</td><td>4.78</td><td>70.29</td><td>68.73</td><td></td><td></td><td></td></tr>
  <tr><td>InstructBLIP-vicuna 2023 (↑)</td><td>7B</td><td>15.55</td><td>45.10</td><td>11.40</td><td>22.10</td><td>3.00</td><td>9.08</td><td>21.90</td><td>34.33</td><td>4.35</td><td>102.08</td><td>53.14</td><td></td><td></td><td></td></tr>
  <tr><td>Tiny-LLaMA-1 2024 (↑)</td><td>1.4B</td><td>13.95</td><td>37.00</td><td>18.70</td><td>8.90</td><td>4.50</td><td>12.17</td><td>22.80</td><td>44.67</td><td>3.48</td><td>61.19</td><td>58.91</td><td></td><td></td><td></td></tr>
  <!-- Table continues... -->
</table>
Table 4: Statistics on the score distributions across different model series.

<table>
  <tr>
    <th>Model series</th>
    <th>Dataset</th>
    <th>GiNi</th>
    <th>stdev</th>
  </tr>
  <tr>
    <td rowspan="2">InternVL-2</td>
    <td>LIME</td>
    <td>0.061</td>
    <td>6.972</td>
  </tr>
  <tr>
    <td>Original</td>
    <td>0.030</td>
    <td>4.421</td>
  </tr>
  <tr>
    <td rowspan="2">Cambrian</td>
    <td>LIME</td>
    <td>0.006</td>
    <td>1.227</td>
  </tr>
  <tr>
    <td>Original</td>
    <td>0.002</td>
    <td>0.715</td>
  </tr>
  <tr>
    <td rowspan="2">LLaVA-1.6</td>
    <td>LIME</td>
    <td>0.042</td>
    <td>6.730</td>
  </tr>
  <tr>
    <td>Original</td>
    <td>0.004</td>
    <td>1.418</td>
  </tr>
</table>

Table 5: Statistics on the score distributions across different model sizes.

<table>
  <tr>
    <th>Model size</th>
    <th>Dataset</th>
    <th>GiNi</th>
    <th>stdev</th>
  </tr>
  <tr>
    <td rowspan="2">7B</td>
    <td>LIME</td>
    <td>0.271</td>
    <td>19.041</td>
  </tr>
  <tr>
    <td>Original</td>
    <td>0.086</td>
    <td>10.836</td>
  </tr>
  <tr>
    <td rowspan="2">8B</td>
    <td>LIME</td>
    <td>0.128</td>
    <td>10.685</td>
  </tr>
  <tr>
    <td>Original</td>
    <td>0.046</td>
    <td>6.270</td>
  </tr>
  <tr>
    <td rowspan="2">13B</td>
    <td>LIME</td>
    <td>0.174</td>
    <td>13.536</td>
  </tr>
  <tr>
    <td>Original</td>
    <td>0.043</td>
    <td>6.446</td>
  </tr>
</table>

Table 6: Text-only with VD results: With the condition of providing only text QA information and VD information, the performance comparison between vlm-s-bench and origin bench.

<table>
  <tr>
    <th rowspan="2">Setting</th>
    <th rowspan="2">Models</th>
    <th colspan="2">AI2D</th>
    <th colspan="2">ChQA</th>
    <th colspan="2">COCO</th>
    <th colspan="2">IVQA</th>
    <th colspan="2">OCRBen</th>
    <th colspan="2">OK VQA</th>
    <th colspan="2">POPE</th>
    <th colspan="2">SciQA</th>
    <th colspan="2">TCaps</th>
    <th colspan="2">TVQA</th>
  </tr>
  <tr>
    <th>A12D</th>
    <th>ChQA</th>
    <th>COCO</th>
    <th>IVQA</th>
    <th>OCRBen</th>
    <th>OK VQA</th>
    <th>POPE</th>
    <th>SciQA</th>
    <th>TCaps</th>
    <th>TVQA</th>
  </tr>
  <tr>
    <td rowspan="6">LIME</td>
    <td>LLaMA3-8B</td>
    <td>23.5</td>
    <td>6.4</td>
    <td>2.8</td>
    <td>12.9</td>
    <td>9.2</td>
    <td>17.4</td>
    <td>32.1</td>
    <td>16.4</td>
    <td>5.3</td>
    <td>17.9</td>
  </tr>
  <tr>
    <td>LLaMA3-70B</td>
    <td>24.0</td>
    <td>7.7</td>
    <td>3.3</td>
    <td>12.3</td>
    <td>9.3</td>
    <td>21.8</td>
    <td>38.1</td>
    <td>39.4</td>
    <td>6.0</td>
    <td>22.0</td>
  </tr>
  <tr>
    <td>Qwen1.5-32B</td>
    <td>28.8</td>
    <td>6.7</td>
    <td>6.5</td>
    <td>9.4</td>
    <td>8.7</td>
    <td>4.7</td>
    <td>39.3</td>
    <td>46.6</td>
    <td>9.2</td>
    <td>13.7</td>
  </tr>
  <tr>
    <td>Qwen1.5-72B</td>
    <td>25.4</td>
    <td>2.5</td>
    <td>3.2</td>
    <td>10.1</td>
    <td>8.9</td>
    <td>7.8</td>
    <td>42.7</td>
    <td>44.2</td>
    <td>6.0</td>
    <td>15.2</td>
  </tr>
  <tr>
    <td>Qwen2-7B</td>
    <td>27.6</td>
    <td>6.7</td>
    <td>6.9</td>
    <td>11.2</td>
    <td>8.9</td>
    <td>15.0</td>
    <td>44.2</td>
    <td>45.5</td>
    <td>12.5</td>
    <td>19.0</td>
  </tr>
  <tr>
    <td>Qwen2-72B</td>
    <td>26.3</td>
    <td>6.9</td>
    <td>2.7</td>
    <td>10.8</td>
    <td>9.6</td>
    <td>10.6</td>
    <td>36.3</td>
    <td>45.2</td>
    <td>5.2</td>
    <td>16.8</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B-Chat</td>
    <td>22.1</td>
    <td>2.3</td>
    <td>0.3</td>
    <td>3.1</td>
    <td>0.0</td>
    <td>7.8</td>
    <td>40.0</td>
    <td>0.0</td>
    <td>0.2</td>
    <td>5.8</td>
  </tr>
  <tr>
    <td rowspan="6">Original</td>
    <td>LLaMA3-8B</td>
    <td>49.0</td>
    <td>11.4</td>
    <td>3.1</td>
    <td>18.6</td>
    <td>19.3</td>
    <td>32.5</td>
    <td>46.9</td>
    <td>59.5</td>
    <td>6.5</td>
    <td>26.4</td>
  </tr>
  <tr>
    <td>LLaMA3-70B</td>
    <td>52.0</td>
    <td>12.4</td>
    <td>3.6</td>
    <td>17.6</td>
    <td>19.5</td>
    <td>36.4</td>
    <td>5.2</td>
    <td>64.6</td>
    <td>7.8</td>
    <td>36.2</td>
  </tr>
  <tr>
    <td>Qwen1.5-32B</td>
    <td>60.5</td>
    <td>10.7</td>
    <td>8.1</td>
    <td>15.0</td>
    <td>20.2</td>
    <td>15.8</td>
    <td>47.4</td>
    <td>68.8</td>
    <td>10.6</td>
    <td>22.1</td>
  </tr>
  <tr>
    <td>Qwen1.5-72B</td>
    <td>58.8</td>
    <td>6.4</td>
    <td>3.8</td>
    <td>16.6</td>
    <td>20.2</td>
    <td>21.1</td>
    <td>35.1</td>
    <td>68.4</td>
    <td>7.1</td>
    <td>27.4</td>
  </tr>
  <tr>
    <td>Qwen2-7B</td>
    <td>59.2</td>
    <td>12.7</td>
    <td>7.4</td>
    <td>19.7</td>
    <td>19.6</td>
    <td>30.5</td>
    <td>44.6</td>
    <td>69.0</td>
    <td>15.4</td>
    <td>33.3</td>
  </tr>
  <tr>
    <td>Qwen2-72B</td>
    <td>60.4</td>
    <td>10.5</td>
    <td>3.5</td>
    <td>15.7</td>
    <td>20.5</td>
    <td>24.2</td>
    <td>34.3</td>
    <td>67.9</td>
    <td>6.8</td>
    <td>28.7</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B-Chat</td>
    <td>24.7</td>
    <td>3.1</td>
    <td>0.0</td>
    <td>5.9</td>
    <td>0.5</td>
    <td>7.8</td>
    <td>32.7</td>
    <td>31.7</td>
    <td>0.2</td>
    <td>5.8</td>
  </tr>
</table>

A.3 Evaluation cost comparision

To further visually demonstrate the advantages of LIME in reducing evaluation time and cost, we conduct a cost assessment across a series of models with different sizes. Using the A800 series GPUs and the internvl-2.5 model as an example, we evaluate both LIME and the original benchmark, recording the GPU time spent during testing. The evaluation costs are then converted into monetary terms based on market prices. As shown in Tab 8, compared to the original benchmark, LIME reduces approximately 70% of the evaluation time and cost. Furthermore, as the model training progresses, assuming a total of 1000 training steps with evaluation conducted every 10 steps, a total of 100 evaluations will be required. In this context, LIME leads to a more significant reduction in costs.

Table 7: text only results between LIME and Original on AI2D and ScienceQA:

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">AI2D</th>
    <th colspan="2">ScienceQA</th>
  </tr>
  <tr>
    <th>LIME</th>
    <th>Original</th>
    <th>LIME</th>
    <th>Original</th>
  </tr>
  <tr>
    <td>LLaMA3-8B</td>
    <td>18.10</td>
    <td>46.76</td>
    <td>33.33</td>
    <td>59.35</td>
  </tr>
  <tr>
    <td>LLaMA3-70B</td>
    <td>25.70</td>
    <td>62.05</td>
    <td>56.00</td>
    <td>69.91</td>
  </tr>
  <tr>
    <td>Qwen1.5-32B</td>
    <td>24.10</td>
    <td>61.14</td>
    <td>43.67</td>
    <td>67.97</td>
  </tr>
  <tr>
    <td>Qwen1.5-72B</td>
    <td>19.80</td>
    <td>57.45</td>
    <td>35.00</td>
    <td>61.13</td>
  </tr>
  <tr>
    <td>Qwen2-7B</td>
    <td>21.00</td>
    <td>57.09</td>
    <td>43.00</td>
    <td>67.38</td>
  </tr>
  <tr>
    <td>Qwen2-72B</td>
    <td>20.60</td>
    <td>69.95</td>
    <td>38.67</td>
    <td>63.36</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B-Chat</td>
    <td>20.10</td>
    <td>23.22</td>
    <td>17.33</td>
    <td>23.60</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B-Chat</td>
    <td>23.60</td>
    <td>54.15</td>
    <td>42.00</td>
    <td>65.20</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B-Chat</td>
    <td>25.20</td>
    <td>60.69</td>
    <td>46.00</td>
    <td>70.55</td>
  </tr>
</table>
Table 8: Comparison of testing costs between LIME and original benchmark.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Test Time(h)</th>
    <th colspan="2">Test Cost($)</th>
    <th rowspan="2">Reduce Cost (Once)</th>
    <th rowspan="2">Reduce Cost (100 times)</th>
  </tr>
  <tr>
    <th>LIME</th>
    <th>Original</th>
    <th>LIME</th>
    <th>Original</th>
  </tr>
  <tr>
    <td>1B</td>
    <td>1.05</td>
    <td>2.90</td>
    <td>6.12</td>
    <td>16.90</td>
    <td>10.78</td>
    <td>1,078.16</td>
  </tr>
  <tr>
    <td>2B</td>
    <td>1.14</td>
    <td>3.31</td>
    <td>6.64</td>
    <td>19.26</td>
    <td>12.63</td>
    <td>1,262.76</td>
  </tr>
  <tr>
    <td>4B</td>
    <td>1.31</td>
    <td>4.16</td>
    <td>7.62</td>
    <td>24.20</td>
    <td>16.58</td>
    <td>1,658.01</td>
  </tr>
  <tr>
    <td>8B</td>
    <td>1.58</td>
    <td>5.81</td>
    <td>9.21</td>
    <td>33.82</td>
    <td>24.61</td>
    <td>2,460.97</td>
  </tr>
  <tr>
    <td>26B</td>
    <td>3.62</td>
    <td>14.32</td>
    <td>21.06</td>
    <td>83.38</td>
    <td>62.31</td>
    <td>6,231.14</td>
  </tr>
</table>

A.4 Existing benchmark still differs from real-world query.

To further investigate the gap between LIME and real-world users’ queries, we construct a similarity search system that compares them. MixEval (Ni et al., 2024) uses SentenceTransformers (Reimers, 2019) as the retrieval model, while Unir (Wei et al., 2023) employs multimodal models like CLIP and BLIP. We use WildVision-Chat as the query data source, which contains 45.2k high-quality user questions, and employ SentenceTransformers to retrieve the top 10 most similar samples from LIME. To fully incorporate image information, we combine the question and image description as the query input. Additionally, we utilize Qwen2-72B to ensure a high level of relevance in the final results. As a result, we obtain a LIME-fit dataset containing 1.1k relevant samples. Existing benchmark can’t cover all types of real-world query.

In Figure 9, we compare the category distribution differences between LIME-fit and the WildVision Bench. It is evident that LIME-fit concentrates in a few specific categories (e.g., data analysis, general description, object recognition). However, it does not include instructions for solving real-world problems, such as Face Recognition, Problem Solving, and Scene Description. Furthermore, Figure 10 shows the frequency distribution of each subcategory in LIME-fit, which follows a long-tail distribution. This indicates that the current benchmark does not fully cover the instruction requirements of real-world scenarios.

![Category difference between LIME-fit and wildvision bench](page_101_1042_1297_496.png)

Figure 9: category difference between LIME-fit and wildvision bench
![subcategory distribution of LIME-fit.](page_186_120_1077_393.png)

Figure 10: subcategory distrubution of LIME-fit.

A.5 Pipeline details
A.5.1 prompt template details

Semi-Automated Screening Process Prompt  We selected GPT-4V as the basis for automatic judgment and interacted with the GPT-4V API using specific prompt templates for different subtasks.

<table>
  <tr>
    <th>Semi-Automated Screening Process Prompt(VQA tasks)</th>
  </tr>
  <tr>
    <td>Please judge whether the &lt;Answer&gt;is the golden answer to the &lt;Question&gt;. If it is, please reply YES, otherwise reply NO.<br>&lt;Question&gt;: {question} &lt;Answer&gt;: {answer}<br>&lt;Your judgement&gt; : &lt;YES or NO&gt;</td>
  </tr>
</table>

<table>
  <tr>
    <th>Semi-Automated Screening Process Prompt(captioning tasks)</th>
  </tr>
  <tr>
    <td>Now there is an image captioning task.<br>Please first describe the content of the image, then compare the image content with the provided captions.<br>If the captions are suitable as captions for the image, please answer YES; if they are not suitable, please answer NO.<br>Respond with NO if any of the captions are unsuitable. Respond with YES only if all captions are suitable.<br>&lt;Captions&gt;: {answer}<br>&lt;Description&gt;: &lt;Content of the image&gt;<br>&lt;Your judgement&gt;: &lt;ONLY YES or NO&gt;</td>
  </tr>
</table>

Exact Vision Description Prompt  For the QVD experiment, we use LLaVA-NEXT-110B to extract information from the images, with the following prompt:

<table>
  <tr>
    <th>Exact Vision Description Prompt</th>
  </tr>
  <tr>
    <td>&lt;image&gt; Please provide a description of the following image. You should consider elements in the image.</td>
  </tr>
</table>

A.5.2 metrics

Subtask metrics: As shown in the Tab 9, different metrics are used for different subtasks. It is important to note that, except for the CIDEr metric, all other metrics have a range between 0 and 1. The final score
for each subtask is calculated by taking the average of these metrics.

Table 9: Metrics for different subtask

<table>
  <tr>
    <th>Metric</th>
    <th>Subtask</th>
    <th>Formula</th>
  </tr>
  <tr>
    <td>Accuracy</td>
    <td>AI2D, ScienceQA-IMG, OCRBench, POPE</td>
    <td>Accuracy = \[
      \begin{cases}
        1, & \text{if the prediction is correct} \\
        0, & \text{if the prediction is incorrect}
      \end{cases}
    \]</td>
  </tr>
  <tr>
    <td>CIDEr</td>
    <td>TextCaps,COCO-Caption</td>
    <td>CIDEr = \( \frac{1}{m} \sum_{i=1}^{m} \sum_{n=1}^{N} w_{r_i} \cdot \frac{g_{i}^{(n)} r_{i}^{(n)}}{\|g_{i}^{(n)}\|\|r_{i}^{(n)}\|} \)</td>
  </tr>
  <tr>
    <td>Match score</td>
    <td>OK-VQA,TextVQA</td>
    <td>SCORE = min (1, \( \frac{\text{match\_nums}}{3} \))</td>
  </tr>
  <tr>
    <td>ANLS</td>
    <td>InfoVQA</td>
    <td>ANLS(\( X, Y \)) = \( 1 - \frac{\text{Lev}(X,Y)}{\max(|X|,|Y|)} \)</td>
  </tr>
  <tr>
    <td>Relaxed Overall</td>
    <td>ChartQA</td>
    <td>SCORE = \( \frac{|prediction-\text{SCORE}|}{|target|} \)</td>
  </tr>
</table>

Overall metric: For the overall metric, we explored two mainstream calculation methods: arithmetic mean 1 and weighted mean 2.

\[
\text{Arithmetic Mean} = \frac{1}{n} \sum_{i=1}^{n} x_i \tag{1}
\]
\[
\text{Weighted Mean} = \frac{\sum_{i=1}^{n} w_i x_i}{\sum_{i=1}^{n} w_i} \tag{2}
\]

The arithmetic mean directly calculates the average of each subtask’s scores, while the weighted mean takes into account the number of samples in each subtask. We compare the results of these two calculation methods, as shown in the Tab 10. weighted average method achieves a higher correlation with WV-ELO. This suggests that the weighted average method is slightly superior to the arithmetic mean, as it considers the impact of the number of data points on the overall score, thereby avoiding potential errors caused by uneven data distribution. Therefore, in our work, we ultimately chose the weighted average as the method for calculating the overall score.

Table 10: Comparison of different overall metrics method

<table>
  <tr>
    <th>model</th>
    <th>overall_weighted</th>
    <th>overall_sum</th>
    <th>overall_cider</th>
    <th>WV_bench</th>
  </tr>
  <tr>
    <td>LLaVA-1.6-vicuna-7B</td>
    <td>30.15</td>
    <td>30.46</td>
    <td>36.07</td>
    <td>992</td>
  </tr>
  <tr>
    <td>LLaVA-1.6-vicuna-13B</td>
    <td>37.08</td>
    <td>36.52</td>
    <td>41.04</td>
    <td>956</td>
  </tr>
  <tr>
    <td>LLaVA-1.6-34B</td>
    <td>44.06</td>
    <td>43.30</td>
    <td>47.12</td>
    <td>1059</td>
  </tr>
  <tr>
    <td>CogVLM</td>
    <td>51.03</td>
    <td>47.66</td>
    <td>44.03</td>
    <td>1016</td>
  </tr>
  <tr>
    <td>Deepseek-VL</td>
    <td>38.1</td>
    <td>39.04</td>
    <td>43.31</td>
    <td>979</td>
  </tr>
  <tr>
    <td>Idefics2</td>
    <td>36.39</td>
    <td>38.43</td>
    <td>43.83</td>
    <td>965</td>
  </tr>
  <tr>
    <td>MiniCPM-v-1.0</td>
    <td>26.15</td>
    <td>28.95</td>
    <td>35.79</td>
    <td>910</td>
  </tr>
  <tr>
    <td>Tinny-LLaVA-1-hf</td>
    <td>13.95</td>
    <td>17.79</td>
    <td>24.15</td>
    <td>879</td>
  </tr>
  <tr>
    <td>LLaVA-1.5-13B</td>
    <td>20.38</td>
    <td>22.88</td>
    <td>32.3</td>
    <td>891</td>
  </tr>
  <tr>
    <td>InstructBLIP-vicuna-7B</td>
    <td>15.55</td>
    <td>18.61</td>
    <td>29.56</td>
    <td>862</td>
  </tr>
  <tr>
    <td>correlation score</td>
    <td><b>0.91</b></td>
    <td>0.90</td>
    <td>0.87</td>
    <td>1</td>
  </tr>
</table>
A.5.3 difficulty classification details

For subtasks using the accuracy (acc) metric, where the scores are binary, with only 1 or 0, other tasks may have various possible score distributions (e.g., COCO-Caption, OK-VQA). Therefore, we determine the threshold score based on the overall distribution of subtask scores, and choose the cutoff value that offers the greatest distinction, as shown in Tab 11, for the metrics ANLS, Relaxed Overall and Accuracy (Acc), the threshold is set to 1.0, for BLEU-4 (for the captioning task, we use the BLEU-4 metric to represent the score for each question), the threshold is set to 0.2, while for Match Score, it is set to 0.6. When the score is greater than the threshold, it is marked as correct; otherwise, it is marked as incorrect.

<table>
  <tr>
    <th>Metrics</th>
    <th>bleu4</th>
    <th>Match score</th>
    <th>ANLS</th>
    <th>Relaxed Overall</th>
    <th>Acc</th>
  </tr>
  <tr>
    <th>Threshold</th>
    <td>0.2</td>
    <td>0.6</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
  </tr>
</table>

Table 11: Thresholds for Different Metrics
A.5.4 Retrieve from real world queryd

Qwen2-72B Judge Prompt

Your task is to compare the content of two questions along with their corresponding image descriptions to determine if they are the same or aligned. Analyze from multiple perspectives, such as theme, question type, and description content.

Please adhere to the following guidelines:

1. Theme Consistency:
- Compare whether the themes of the two questions and their corresponding image descriptions match. If they focus on entirely different topics, they should be marked as not aligned.

2. Question Type:
- Analyze whether the question types (e.g., technical, artistic, textual) of both questions match with each other and align with their respective image descriptions. If they are of different natures, note the mismatch.

3. Description Alignment:
- Compare the task or content expected in each question with what is visually or descriptively present in both image descriptions. If the questions or image content require specific actions (e.g., reading text or coding) that differ from each other or the descriptions, they should be marked as misaligned.

4. Evaluate Similarity:
- Rate the similarity between the two questions and their respective descriptions on a scale from 1 to 5, where 1 means entirely different and 5 means highly similar.

5. Output Clarification:
- You should return whether the two questions and their image descriptions align or not in a simple "True" or "False" result. - Provide a brief reason for your conclusion. - Include a similarity rating from 1 to 5, based on how well the questions and descriptions match. - The output should only contain the "result," "reason," and "similarity rating" fields.

### Example:
<Question 1>: Can you write codes to load this 3D object?
<Description 1>: The image shows a stone sculpture of an angel sitting on a pedestal. The angel has large, feathered wings that spread out behind it, and its head is bowed down, as if in deep thought or prayer. The angel’s body is draped in flowing robes, and its arms are crossed over its lap. The pedestal is ornately carved with intricate designs, and the entire sculpture is set against a dark background, which makes the white stone stand out even more. The overall mood of the image is one of solemnity and reverence.

<Question 2>: What is written in the image?
<Description 2>: The image shows the word "ART" in white capital letters on a blue background. The letters are bold and have a slight shadow effect, giving them a three-dimensional appearance. The overall design is simple and modern, with a focus on the text itself.

Result: False
Reason: The first question asks for coding assistance to load a 3D object, but its description is about an angel sculpture. The second question is focused on reading text from an image, which is aligned with its description showing the word "ART." The themes, questions, and descriptions are entirely different.
Similarity Rating: 1

<Input Question 1>: {Question 1}
<Input Description 1>: {Description 1}

<Input Question 2>: {Question 2}
<Input Description 2>: {Description 2}

<Output>:
A.6 ablation study about data size

Table 12: data size ablation study on OK-VQA.

<table>
  <tr>
    <th>Model</th>
    <th>Full</th>
    <th>100</th>
    <th>500</th>
    <th>1200</th>
  </tr>
  <tr>
    <td>llava1.5-7B</td>
    <td>22.71</td>
    <td>17.00</td>
    <td>20.76</td>
    <td>22.92</td>
  </tr>
  <tr>
    <td>llava1.5-13B</td>
    <td>31.59</td>
    <td>36.00</td>
    <td>29.60</td>
    <td>30.23</td>
  </tr>
  <tr>
    <td>llava1.6-7B</td>
    <td>11.46</td>
    <td>13.00</td>
    <td>10.40</td>
    <td>11.32</td>
  </tr>
  <tr>
    <td>llava-llama3-8B</td>
    <td>36.12</td>
    <td>32.60</td>
    <td>36.92</td>
    <td>36.17</td>
  </tr>
  <tr>
    <td>xcomposer2-4khd</td>
    <td>25.91</td>
    <td>29.40</td>
    <td>26.48</td>
    <td>26.90</td>
  </tr>
  <tr>
    <td>minicpm</td>
    <td>25.76</td>
    <td>20.60</td>
    <td>29.92</td>
    <td>25.30</td>
  </tr>
  <tr>
    <td>instructblip</td>
    <td>21.45</td>
    <td>20.60</td>
    <td>23.60</td>
    <td>21.78</td>
  </tr>
  <tr>
    <td>idefics2</td>
    <td>32.76</td>
    <td>27.60</td>
    <td>35.12</td>
    <td>33.00</td>
  </tr>
  <tr>
    <td>internvl</td>
    <td>38.36</td>
    <td>45.00</td>
    <td>39.80</td>
    <td>38.28</td>
  </tr>
</table>

Table 13: data size ablation study on ChartQA.

<table>
  <tr>
    <th>Model</th>
    <th>Full</th>
    <th>100</th>
    <th>500</th>
    <th>1200</th>
  </tr>
  <tr>
    <td>llava1.5-7B</td>
    <td>4.77</td>
    <td>3.00</td>
    <td>3.80</td>
    <td>4.17</td>
  </tr>
  <tr>
    <td>llava1.5-13B</td>
    <td>4.71</td>
    <td>5.00</td>
    <td>4.40</td>
    <td>4.33</td>
  </tr>
  <tr>
    <td>llava1.6-7B</td>
    <td>42.81</td>
    <td>39.00</td>
    <td>42.00</td>
    <td>42.67</td>
  </tr>
  <tr>
    <td>llava-llama3-8B</td>
    <td>64.78</td>
    <td>66.00</td>
    <td>66.00</td>
    <td>65.75</td>
  </tr>
  <tr>
    <td>xcomposer2-4khd</td>
    <td>82.11</td>
    <td>80.00</td>
    <td>83.00</td>
    <td>82.92</td>
  </tr>
  <tr>
    <td>minicpm</td>
    <td>70.37</td>
    <td>67.00</td>
    <td>71.60</td>
    <td>70.75</td>
  </tr>
  <tr>
    <td>instructblip</td>
    <td>2.95</td>
    <td>3.00</td>
    <td>3.00</td>
    <td>3.00</td>
  </tr>
  <tr>
    <td>idefics2</td>
    <td>13.18</td>
    <td>16.00</td>
    <td>12.80</td>
    <td>14.25</td>
  </tr>
  <tr>
    <td>internvl</td>
    <td>87.13</td>
    <td>89.00</td>
    <td>87.80</td>
    <td>86.92</td>
  </tr>
</table>

Table 14: data size ablation study on TextVQA.

<table>
  <tr>
    <th>Model</th>
    <th>Full</th>
    <th>100</th>
    <th>500</th>
    <th>1200</th>
  </tr>
  <tr>
    <td>llava1.5-7B</td>
    <td>16.68</td>
    <td>14.40</td>
    <td>18.34</td>
    <td>17.46</td>
  </tr>
  <tr>
    <td>llava1.5-13B</td>
    <td>19.54</td>
    <td>17.90</td>
    <td>22.30</td>
    <td>20.14</td>
  </tr>
  <tr>
    <td>llava1.6-7B</td>
    <td>38.58</td>
    <td>43.00</td>
    <td>38.62</td>
    <td>39.35</td>
  </tr>
  <tr>
    <td>llava-llama3-8B</td>
    <td>39.81</td>
    <td>46.40</td>
    <td>38.26</td>
    <td>40.17</td>
  </tr>
  <tr>
    <td>xcomposer2-4khd</td>
    <td>61.20</td>
    <td>59.40</td>
    <td>60.98</td>
    <td>61.63</td>
  </tr>
  <tr>
    <td>minicpm</td>
    <td>63.07</td>
    <td>60.30</td>
    <td>63.90</td>
    <td>63.39</td>
  </tr>
  <tr>
    <td>instructblip</td>
    <td>11.66</td>
    <td>8.60</td>
    <td>12.00</td>
    <td>11.25</td>
  </tr>
  <tr>
    <td>idefics2</td>
    <td>55.94</td>
    <td>54.90</td>
    <td>57.76</td>
    <td>56.56</td>
  </tr>
  <tr>
    <td>internvl</td>
    <td>70.28</td>
    <td>70.10</td>
    <td>70.48</td>
    <td>70.74</td>
  </tr>
</table>

Table 15: data size ablation study on InfoVQA.

<table>
  <tr>
    <th>Model</th>
    <th>Full</th>
    <th>100</th>
    <th>500</th>
    <th>1200</th>
  </tr>
  <tr>
    <td>llava1.5-7B</td>
    <td>9.40</td>
    <td>7.00</td>
    <td>9.00</td>
    <td>8.83</td>
  </tr>
  <tr>
    <td>llava1.5-13B</td>
    <td>12.18</td>
    <td>16.00</td>
    <td>11.60</td>
    <td>11.17</td>
  </tr>
  <tr>
    <td>llava1.6-7B</td>
    <td>21.30</td>
    <td>19.00</td>
    <td>23.00</td>
    <td>20.33</td>
  </tr>
  <tr>
    <td>llava-llama3-8B</td>
    <td>22.69</td>
    <td>25.00</td>
    <td>23.40</td>
    <td>22.33</td>
  </tr>
  <tr>
    <td>xcomposer2-4khd</td>
    <td>72.36</td>
    <td>72.00</td>
    <td>75.80</td>
    <td>73.75</td>
  </tr>
  <tr>
    <td>minicpm</td>
    <td>49.22</td>
    <td>55.00</td>
    <td>48.20</td>
    <td>48.75</td>
  </tr>
  <tr>
    <td>instructblip</td>
    <td>9.73</td>
    <td>11.00</td>
    <td>8.40</td>
    <td>9.50</td>
  </tr>
  <tr>
    <td>idefics2</td>
    <td>24.69</td>
    <td>23.00</td>
    <td>24.00</td>
    <td>25.42</td>
  </tr>
  <tr>
    <td>internvl</td>
    <td>72.08</td>
    <td>69.00</td>
    <td>72.20</td>
    <td>72.25</td>
  </tr>
</table>
B  case study

The original dataset contains noise data. In the following figure, we categorize the problematic data into three types and present specific examples from different datasets.

Text Answerable Questions:  Some questions can be answered without the need for visual information, mainly focusing on the AI2D and ScienceQA datasets. As shown in figs. 11 and 12, AI2D and ScienceQA emphasize knowledge in the field of science while overlooking the importance of visual information. Given the background of domain knowledge, some LLMs are able to provide answers even without requiring visual input.

Annotation Error Questions:  Most benchmarks are manually curated, which inevitably leads to annotation errors. Problematic questions exist in almost all benchmarks. It can be found in figs. 13, 14 and 20 to 25.

Repeated Question:  Some benchmarks also contain a significant amount of duplicate data, where the question content and image content are completely identical. This issue is mainly found in the POPE dataset, as shown in the figs. 15 to 19.
AI2D

Question: Which stage follows the egg stage of development in a beetle's life cycle?

Options: ["Nymph", "Larva", "Adule", "Pupa"]

![A diagram showing the life cycle of a beetle with labeled stages A, B, C, D](page_312_340_823_563.png)

Error Category: Data Leakage

Ground Truth: Larve

Figure 11: A sample bad case of AI2D
Back to List of figures
AI2D

Question: In the illustration, if mahi mahi were to die off the large shark population would?

Options: ["decrease", "remain the same", "can't tell", "increase"]

![Food Chain diagram showing the flow of energy from phytoplankton to zooplankton, small fish, mackerel, mahi mahi, and large shark](page_312_349_678_355.png)

Error Category: Data Leakage

Ground Truth: “decrease”

Figure 12: A sample bad case of AI2D
Back to List of figures
InfographicVQA

Question: What percent of executives does not use social media daily?

![Infographic showing statistics about how executives are using social media](page_184_120_1083_670.png)

Error Category: Annotation Error

Ground Truth: '24%' , '24'

Figure 13: A sample bad case of InfoVQA
Back to List of figures
InfographicVQA

Question: What is the second last solution given?

![Infographic showing poverty statistics, disability prevalence rates, unemployment rate, reasons, consequences, and solutions related to disability and poverty.](page_184_232_1079_627.png)

Error Category: Annotation Error

Ground Truth: 'access to technical and vocational training'

Figure 14: A sample bad case of InfoVQA
Back to List of figures
POPE

Question:: Is there a tv in the image?

Options: Yes

![A desk setup with two computers, a keyboard, mouse, and some personal items, but no TV](page_324_370_800_400.png)

< 228 >

Error Category: Annotation Error

Ground Truth: No

Figure 15: A sample bad case of POPE
Back to List of figures
POPE

Question:: Is there a dining table in the image?

Options: Yes

![A bedroom scene with a bed, nightstand, and some books, but no dining table](page_324_370_800_563.png)

< 934 >

Error Category: Annotation Error

Ground Truth: No

Figure 16: A sample bad case of POPE
Back to List of figures
POPE

Question:: Is there a boat in the image?

Options: Yes

![A black and white photo showing a suitcase and some equipment, with no visible boat](page_324_370_800_400.png)

<1412>

Error Category: Annotation Error

Ground Truth: No

Figure 17: A sample bad case of POPE
Back to List of figures
POPE

Question:: Is there a boat in the image?

Options: Yes

![A pizza on a plate](page_324_370_800_400.png)

<6940>

Error Category: Repeated Questions

Ground Truth: Repeated with id 940

Figure 18: A sample bad case of POPE
Back to List of figures
POPE

Question: Is there a dining table in the image?

Options: Yes

![A living room with a couch, coffee table, and TV, but no dining table](page_324_370_800_400.png)

< 6694 >

Error Category: Repeated Questions

Ground Truth: Repeated with id 694

Figure 19: A sample bad case of POPE
Back to List of figures
OK VQA

Question: How would you dress for this setting?

Options: [ "shorts", "shorts", "shorts", "shorts", "bathing suit", "bathing suit", "bikini", "bikini", "summer", "summer" ]

![A black and white photo of a beach scene with seagulls and people in the background](page_312_420_823_377.png)

< 1708495 >

Error Category: Annotation Error

Ground Truth: [ "shorts", "swimming suit", "bathing suit", "bikini" ]
OK VQA

Question: Where are these people?

Options: [ "outside", "outside", "outside", "outside", "field", "field", "on hill", "on hill", "outdoors", "outdoors" ]

![A group of people standing in an open field with kites flying above, under a clear blue sky](page_324_370_800_350.png)

< 3981385 >

Error Category: Annotation Error

Ground Truth: [ "outside", "riverbank", "grassland", "field", "hill", "outdoors", "lawn" ]
OK VQA

Question: How is this effect painted on to walls?

Options: [ "sponge", "sponge", "sponge", "sponge", "with sponge", "with sponge", "sponged", "sponged", "sky", "sky" ]

![A blue wall with clouds painted on it, a mirror, and a door](page_324_370_800_350.png)

< 1269585 >

Error Category: Annotation Error

Ground Truth: [ "whitewash", "paint", "plaster" ]
Text VQA

Question: what is one of the numbers on the buttons of the calculator?

Options: [ "1", "1", "1", "1", "1", "7", "7", "5", "1", "5" ]

![A calculator with a pencil on a table, displaying the number 528963](page_340_370_768_496.png)

< 35925 >

Error Category: Annotation Error

Ground Truth: [ "1", "2", "3", "4", "5", "6", "7", "8", "9", "0" ]

Figure 23: A sample bad case of TextVQA
Back to List of figures
Text VQA

Question::what is served at this place?

Options: [ "gift certificates", "ice cream, coffee, and sandwiches", "ice cream& coffee", "traditional italian ice cream and coffee", "ice cream & coffee", "ice cream, coffee, and grilled focaccia sandwiches", "ice cream & coffee", "traditional italian, ice cream and coffee, grilled focaccia sandwiches", "ice cream & coffee, grilled focaccia sandwiches", "gelato" ]

![A sign for Amato Gelato Cafe, listing traditional Italian ice cream & coffee, grilled focaccia sandwiches, custom gelato, yule log cake & gift certificates available](page_324_563_798_312.png)

< 37706 >

Error Category: Annotation Error

Ground Truth: [ "ice cream", "coffee", "sandwiches", "gelato", "cake", "yule log", "gift certificates", "grilled focaccia sandwiches"]

Figure 24: A sample bad case of TextVQA
Back to List of figures
Text VQA

Question:: what is the cell phone carrier?

Options: [ "cingular", "blackberry", "cingular", "cingular", "cingular", "cingular", "cingular", "at&t", "cingular", "cingular", "cingular" ]

![A close-up image of a Blackberry smartphone displaying the Cingular AT&T EDGE network](page_324_370_800_400.png)

< 36711 >

Error Category: Annotation Error

Ground Truth: [ "EDGE" ]

Reviewer paper summaries: This paper introduces LIME (Less Is More for MLLM Evaluation), a streamlined multimodal evaluation benchmark designed to effectively evaluate Multimodal Large Language Models (MLLMs) performance while reducing computational overhead. Built from existing datasets, LIME utilizes a semi-automated pipeline, which filters out uninformative or overly easy samples, removes questions containing errors or answer leakage, and thus achieves a refined evaluation set. Experiments demonstrate that LIME substantially reduces sample size by and evaluation time, while effectively highlighting performance differences among models through a more dispersed score distribution.
Summary of strengths: 1. Efficient Benchmarking Pipeline: The paper proposes a robust semi-automated pipeline effectively reducing the benchmark size while maintaining the discriminative purpose, which is valuable for large-scale evaluation tasks.
2. Comprehensive Experiments: Using the crafted LIME benchmark, the authors test 36 different MLLMs across 6 tasks, which provide a comprehensive evaluation of the mainstream MLLMs. 
3. Open Availability of Resources: The authors publicly release the curated dataset, evaluation scripts, and code, facilitating reproducibility, verification, and enabling further research advancements within the community.
Summary of weaknesses: 1. Unclear Novelty Compared to Existing Work. The distinction between LIME and similar benchmarks like MMStar (cited in the paper) is unclear. Some key pipeline stages in the benchmark construction, such as difficulty-aware sample screening and answer leakage removal, appear similar to MMStar. The authors should explicitly highlight LIME’s unique contributions compared to existing benchmarks.
2. Limited Insight from the Benchmark Results. While the paper presents extensive experimental results evaluating MLLMs, it provides limited in-depth analysis and lacks insightful observations or interpretations derived from these evaluations. Additionally, the identified limitations of traditional caption metrics like BLEU and CIDEr reflect a well-known issue [1]. The authors should enrich the results section with deeper analytical discussions and clearer takeaways.

[1] Yuksekgonul, Mert, et al. "When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?." The Eleventh International Conference on Learning Representations.
Comments, suggestions and typos: 1. Metric Suggestions: Given the identified limitations of current metrics (e.g., CIDEr), the authors should consider incorporating or designing alternative evaluation approaches that better capture the model’s captioning performance, such as CLIPScore [2] and VisualGPTScore [3], rather than directly  exclude the captioning scores in the overall score calculation.
2. The titles for section 2.1 “open-source models as judges” and 3.2 “data statistics” should be capitalized for consistency.

[2] Hessel, Jack, et al. "CLIPScore: A Reference-free Evaluation Metric for Image Captioning." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

[3] Lin, Zhiqiu, et al. "Revisiting the Role of Language Priors in Vision-Language Models." Forty-first International Conference on Machine Learning.
Overall assessment scores: 2.5
Metareview score: 3
Metareview: This paper introduces LIME, a new benchmark for evaluating MLLMs. The authors propose a semi-automated pipeline to curate a more refined and efficient evaluation dataset by filtering out uninformative samples and focusing on tasks that genuinely require image understanding. They show that LIME reduces sample size by 76% and evaluation time by 70% while better differentiating between model capabilities.
Summary of reasons to publish: The proposed evaluation benchmark and their findings are helpful to the vision-language community, providing some insights.