Are We on the Right Way for Evaluating Large Vision-Language Models?
Authors: Lin Chen
Jinsong Li
Xiaoyi Dong
Pan Zhang
Yuhang Zang
Zehui Chen
Haodong Duan
Jiaqi Wang
Yu Qiao
Dahua Lin
Feng Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: large vision-language model
data leakage
multi-modal benchmark
Primary area: machine_vision
TL;DR: We present MMStar, an elite vision-indispensable multi-modal benchmark comprising 1,500 samples meticulously selected by humans.

Abstract: Large vision-language models (LVLMs) have recently achieved rapid progress, sparking numerous studies to evaluate their multi-modal capabilities. However, we dig into current evaluation works and identify two primary issues: 1) Visual content is unnecessary for many samples. The answers can be directly inferred from the questions and options, or the world knowledge embedded in LLMs. This phenomenon is prevalent across current benchmarks. For instance, GeminiPro achieves 42.7% on the MMMU benchmark without any visual input, and outperforms the random choice baseline across six benchmarks near 24% on average. 2) Unintentional data leakage exists in LLM and LVLM training. LLM and LVLM could still answer some visual-necessary questions without visual content, indicating the memorizing of these samples within large-scale training data. For example, Sphinx-X-MoE gets 43.6% on MMMU without accessing images, surpassing its LLM backbone with 17.9%. Both problems lead to misjudgments of actual multi-modal gains and potentially misguide the study of LVLM. To this end, we present MMStar, an elite vision-indispensable multi-modal benchmark comprising 1,500 samples meticulously selected by humans. MMStar benchmarks 6 core capabilities and 18 detailed axes, aiming to evaluate LVLMs' multi-modal capacities with carefully balanced and purified samples. These samples are first roughly selected from current benchmarks with an automated pipeline, human review is then involved to ensure each curated sample exhibits visual dependency, minimal data leakage, and requires advanced multi-modal capabilities. Moreover, two metrics are developed to measure data leakage and actual performance gain in multi-modal training. We evaluate 16 leading LVLMs on MMStar to assess their multi-modal capabilities, and on 7 benchmarks with the proposed metrics to investigate their data leakage and actual multi-modal gain.

Are We on the Right Way for Evaluating Large Vision-Language Models?
Lin Chen 1.3v Jinsong L Xiaoyi Dong Pan Zhang
Yuhang Zang²
Zehui Chen' Haodong Duan 3 Jiaqi Wang Yu Qiao Dahua Lin Feng Zhao
University of Science and Technology of China
2 The Chinese University of Hong Kong
Shanghai Al Laboratory 4 CPII under InnoHK io/
https://mmstar- benchtarrbbbccaakk...........
Abstract
Large lisiuaelanguage models (LVLMs) have recently achieved rapid progress, sparking numerous studies to evaluate their multi-modal capabilities. However, we dig into current evaluation works and identify two primary issues: 1) Visual content is unnecessary for many samples. The answers can be directly inferred from the questions and options, or the world knowledge embedded in LLMs. This phenomenon is prevalent across current benchmarks. For instance, GeminiPro achieves 42.7% on the MMMU benchmark without any visual input, and outper- forms the random choice baseline across six benchmarks near 24% average. 2) Unintentional data leakage exists in LLM and LVLM training. LLM and LVLM could still answer some visual-necee necessary questions without visual content, indicating the memorizing of these samples within large-scale training data. For example, Sphinx-X-MoE gets 43.6% on MMMU without accessing images, sur- passing its LLM backbone with 17.9%. Both problems lead to misjudgments of actual multi-modal gains and potentially misguide the study of LVLM. To this end, we present MMStar, an elite vision- saiionindispensablle multi-modal benchmark com- prising 1,500 samples meticulously selected by humans. MMStar benchmarks 6 core capabilities and 18 detailed axes, aiming to evaluate LVLMs' multi-modal capacities with carefully balanced and purified samples. These samples are first roughly selected from current benchmarks with an automated pipeline, human re- view is then involved to ensure each curated sample exhibits visual dependency, minimal data leakage, and requires advanced multi-modal capabilities. Moreover, two metrics are developed to measure data Icakage and actual performance gain in multi modal training. We evaluate 16 leading LVLMs MMStar assess their multi modal capabilities, and on 7 benchmarks with the proposed metrics to investigate their data leakage and actual multi-modee gain.
Introduction
Encouraged by the rapid development of large language models (LLMs) 60 9 10
53
integrating visual modality into LLMs to enhance models' interactivity capabilities has witnessed ever-changing advances in recent days 72 33 31
B These large vision- language models (LVLMs) showcase powerful visual perception and understanding capabilities, enabling them to accept image inputs from users and engage dialogues, thereby offering a more enriched interactive experience. These achievements have further inspired the research community
61
39
"Equal contribution. This work is done during internship in Shanghai AI Laboratory.
I Cooresponddigg author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

What copital Nebrakaa
What is the shape the round din
A: Providence
circle
B: Saint Paul
Lincoln
Trinnlee
Square
D: Kansas City
Circle
D: Diamand
image does nothing. it's the same as asking with text question directly.
The stape of circle of course, circle.
S question 1039
LLM
SEED Dench ""; question 75500
LLM
#Correct LLMs: 22/22 (100%)
Answer.
&Correct LLMS 22/22 (100%)
Which model can achieve the best Insgehet 10-shot Accuracy score?
Which cell type is pictured?
A: Fosinaphal
A: Soft MoE
Thromboeys
Experss Choice
Lymphocyte
C: Tokens Choice
D: MoMoccte
D: Dense
B: Neusrophi
the but questicn options accut famillar to me; so know the ancwer A.
(00 Acccaay
Mathiistt: questicn-555 Answer:
LLM
MMMU PS question24077 Answer:
LLM
#Currect (72.7%)
VLLM LLL Pairs 9/16 (56.3%)
We highlight cases in existing multi- benchmarks where evaluation samples either lack visual dependency or have uninfentionally leaked into the training data of LLMs and LVLMs. (a) Some samples be answered by LLMs using only text-based world knowledge; (b) For some instances, the question itself contains the answer, making images superfluous; (c) Some samples leaked into LLMs' training corpora can "recalled" with the textual questions and answers directly; (d) Some samples indiscemible to LLMs but solved by LVLMs without accessing images suggest Icakage into LVLMs' multi-moda training data.
Figure 1:
develop variety of multi-modal benchmarks (2 16 34
63
37
26 38 constructed to
explore the powerful capabilities emerging from LVLMs and provide a comprehensive and objective platform for quantitatively comparing the continually evolving models. Despite the race among existing evaluation works to construct as many axes as possible to assess the capabilities of LVLMs, we have identified two primary issues upon delving into existing evaluation samples and processes.
First, visual content is unnecessary for many samples. A qualified multi-modal evaluation sam- ple should compel LVLMs to understand and reason with the visual content for correct answers. Otherwise, the evaluation sample would degrade into assessing the textual capabilities of LLM bases. Unfortunately, we have identified numerous samples across multiple popular benchmarks 34
64 38 26 where answers can be correctly deduced without relying on visual content. As shown in Figure (a) and (b), some samples have answers directly included within the questions (e.g., What is the shape of the round dirt circle?), while others can be effortlessly answered by leveraging the rich world knowledge embedded within the LLM bases (e.g., What the capital of Nebraska?). As shown in Table [ with a comprehensive quantitative analysis of 22 LLMs on 6 benchmarks, we observe this phenomenon is prevalent and serious. For example, more than 50% questions of ScienceQA and near 30% questions of MMMU can be solved by most LLMs directly. For the powerful LLM GeminiPro, it achieves 42.7% on the MMMU benchmark without any visual input, and outperforms the random choice baseline across six benchmarks by near 24% on average.
27
Taking aside the inappropriate samples in evaluation, we also observed strange results that LLM and LVLM could still answer some visual-necessary questions without visual content (Figure (c) and (d)). plausible explanation for this could be the inadvertent memorization of these samples during the large-scale training process, suggesting the presence of unintentional data leakage in the training of LLM and LVLM. Through a detailed study various LVLMs on 6 benchmarks, as shown in Table 2 we find the unexpected leaking problem during the LVLM training is particularly serious. For example, we find Yi-VL-34B gets 15.0% higher performance than its LLM backbone on ScienceQA. Ophinx-X-Mool gets 43.6% on MMMU without accessing images, surpassing its LLM backbone with 17.9%, even surpassing many leading LVLMs with accessing images.
The existence of inappropriate questions and data leaking would lead to misjudgments of actual multi-modal performance gains and potentially misguide the study LVLM. pursuit of a more accurate and comprehensive evaluation, we introduce the MMStar Benchmark. MMStar is pre- mier, vision-critical multi-modal benchmark that includes 1,500 challenging samples, each rigor- ously validated by humans. It is structured to test 6 fundamental capabilities and 18 specific di-

---

mensions, aiming to evaluate the multi-modal capacities of LVLMs with a carefully balanced and purified selection of samples.
The MMStar is new benchmark that "Stands on the shoulders of giants". Samples are first roughly selected from current benchmarks with an automated pipeline. In detail, use eight powerful LLMs as candidates inspectors for visual dependency and LLM Icakage, including two closed- source APIs (GPT4-Turbo 42], and GeminiPro (51]) and six leading open-source models (e.g.. LLaMA-70B [53] 1wen-1.5-72BB [T. and Mixtral-8x7B (23)). Samples that could be answered by more than 2 of the 8 LLMs are excluded as they may exist leaking or visual unnecessary problems. Then we use 16 leading LVLMs (e.g., GPT4V [43], GeminiPro (Vision) [31], LLaVA series [3] (31 to gauge the difficulty of the samples and split them to four levels. Ultimately, based on the difficulty of the oughh-iittered samples, strict manual review and selection are applied to curate 1,500 high-quality multimodal evaluation samples. As shown in Figure 3 these samples span core multimodal capability dimensions and 18 detailed axes, aiming to probe LVLMs' advanced multimodal capabilities with purified and high-quality set of samples. Moreover, we design the multi-modal gain (MG) and multi-modal leakage (ML) metrics to probe LVLMs' actual performance gain and data leakage degrees derived from multi-modal training in a benchmark-specific manner.
We evaluate the accuracy, MG, and ML of 16 leading LVLMs on our MMStar benchmark, the high- resolution version of GPT-4V ranks first with 57.1% accuracy, showcasing its superb multi-modal capability. GPT-4V also gets the best MG and a small ML, indicating its effective multi-modal training strategy and has less data leaking.
In a nutshell, our contributions are threefold:
We delve into existing evaluation benchmarks and processes and identify two key issues: (1) Visual content is unnecessary for many samples. (2) Unintentional data leakage exists in LLM and LVLM training. Both lead to misjudgment of LVLM and may misguide the following study, We curate MMStar, an elite vision-indispen iision-indispensable multi-modal benchmark comprising 1,500 chal- lenge samples meticulously selected by humans. MMStar covers samples from diverse tasks and difficulties, aiming to evaluate the actual multi-modal capacities of LVLMs.
evaluate LVLMs with Accuracy and two newly proposed metrics: multi- modal gain and multi-modal leakage. The high-resolution version of GPT-4V outperforms the 16 leading LLMs and ranks first.
Based on MMStar,
2 Related Work
Large Vision-Language Models. As large language models (LLMs) 9 53
53 60
44
10 rapidly advance, growing fraction of the research community is focusing integrating visual content into L.MSs to build a powerful intelligent assistant with more interactive ways. Central to these large vision-language models (LVLMs) are the seminal works modality aligmment within the vision-language learning area 21). The foundation work CLIP [46] exemplifies the align- ment of vision and language modalities through contrastive learning on extensive image-text pairs. Built upon the CLIP image encoder which is somewhat aligned with the language modality, current LVLMs typically utilize vast image-text pairs to connect the vision encoder and LLM, enabling LLM to receive and understand visual content [72 33 31
45
2 48
6 39
19
5 20, [7] 24, (70).
For example, MiniGPT4 722 and LLaVA 3 directly connect the vision encoder and LLM with
QFormer (28 and MLP 50, showing proficiency in multi-modal dialogues. Subsequent works
have further enhanced LVLMs by improving the multi-modal instruction data [3 61 54 67 25
and designing novel modules 22 30 55 36 17 [3] for more sufficient modality alignment.
Evaluations of LVLMs. To probe the true capabilities of the emerging LVLMs, the research com- munity has developed many muultimodal benchmarks encompassing a wide range of evaluation axes
34
[6]
47
64
49
63 56 58 57]. Early single-task benchmarks, such as VQA 49 and OK- VQA 47), fail to holistically assess LVLMs' general multi-modal
66
27
33
29 35
18
COCC
perception and reasoning capabilities. To address this issue, comprehensive multi-modal bench-
16 34 & 56]. For example, SEED [27] and MMBench 34 cover 12 and 20 evaluation dimensions respectively, while MMMU 64 spans 30 college- level subjects, providing some competitive arenas for a comprehensive comparison of cutting-edge L.I.Ms. However, existing evaluations of LVLMs overlook some critical issues. On the one hand, they do not guarantee that all evaluation samples cannot be correctly answered without the visual
marks have been constructed 63 27
64
3

---

Table 1: Evaluation of various LLMs six popular multi-modal benchmarks under 2-shot. We employ 2-shot inference strategy for evaluating all LLMs reduce instances of refusal answer formats. We report results closed-source LLMs 20 open-source LLMs with varying sizes and architectures. The evaluated benchmarks include MMMU (MMMU-VA Val [64). MMB (MMBench- MI--EN-Doo -ev [4],, ScienceQA (ScienceQA- Test [38), A12D Test (26]), SEED (SEED-Imas Image [27 ), and Math
and align the
Vista-MM
3
The best results are highlighted in bold and underlined.
Model
Strategy
MMMU
MMB
ScienceQA
A12D
SEED
Math
Avg.
Baseline
Random Choice
22.1
0.0
24.2
23.8
24.3
17.9
18.7
Closed-source LLMs
GPr4-Turbo
2-shot 2-shol
42.0
15.5
67.5
61.3
26.8
25.6
39.8
GeminiPro 51
42.7
18.7
69.3
60.1
38.1
25.5
42.4
Open-source LLMs
Qwenl.5-1.88
2-shol
33.0
8.6
55.6
41.3
32.1
22.7
32.2
Phi2-2.7B
2-shot
19.9
4.3
50.8
6.9
18.4
23.7
Yi-6B 62
2-shot
32.9
16.0
64.6
51.5
36.7
24.5
37.7
LLaMA2-7B 53
2-shot
25.9
7.7
57.9
42.8
32.8
22.8
31.7
Qwen-7B[]
2-shol
30.6
15.0
63.0
50.0
32.6
35.4
Deepseck-7B
2-shol
28.7
11.6
61.9
46.0
34.1
21.7
34.0
InterL.M2-77
2-shot
33.6
11.4
52.1
34.4
20.4
35.9
Qwenl.5-7B[I
2-shot
33.3
13.1
65.1
52.1
32.1
22.8
36.4
Vicuna-vl.5-78
2-shot
31.3
9.5
58.9
45.5
32.0
20.7
33.0
Baichuan2-7B(fh
2-shot
28.2
13.7
58.1
32.3
21.7
33.0
Mistral-7B
2-shot
29.8
17.2
66.1
50.0
34,4
13.4
35.2
I. 53
2-shot
32.9
10.1
58.9
43.8
32.1
33.8
Dicuna-v1.5-133
2-shot
31.3
12.8
63.0
46.8
33.6
20.8
34.7
Baichuan2-13Pp 60)
2-shot
32.2
13.1
61.0
47.1
35.2
23.4
35.3
Internl.M2-208 52
2-shot
35.6
17.4
66.4
55.9
30.4
37.8
Yi2
2-shot
35.8
15,8
67.9
59.6
37.2
26.9
40.5
Mixtral-8x7B
2-shot
35.1
17.3
66.3
55.1
35.8
22.7
38.7
Deepseck-679
2-shot
38.3
17.2
68.3
59.7
37.3
23.4
40.7
LLaMA2-700 53
2-shot
30.4
17.2
63.4
49.3
34.9
24.2
36.6
Qwen1.5-72B8
2-shot
42.4
21.1
70.1
60.9
40.7
26.3
43.6
content. On the other hand, current evaluations consistently adhere to the process of inferring on given benchmarks and calculating scores for LVLMs, overlooking the possibility of data leakage during multi-modal training. This oversight can lead unfair comparisons and misjudgments of the real gains in multi-modal capabilities brought by multi-modal training.
Two Overlooked Issues for Evaluating LVLMs
In this section, we delve into two commonly overlooked issues in current LVLM evaluation works. Moreover, present detailed experimental results to further substantiate our observations.
First issue: visual content is unnecessary for many evaluation samples. The key distinction between evaluating L.MSs and LVLMS lies in the necessity for LVL.M evaluations to strictly ensure that the correct answers can only be derived based on thorough understanding of visual con- tent. Without this, evaluating LVLMs' multi-modal capabilities degrades to merely assessing their LLM backbones' uni-modal abilities. However, upon examining samples from some popular LVLM benchmarks, we find many samples lack vital visual dependency and can yield correct answers even without the image inputs! Through analysis of these failure samples, we categorize them into two groups: (1) Answers can be directly obtained from the world knowledge embedded in LLMs, ow- ing to the LLMs' extensive pretraining on the large corpus of data. example, as illustrated in Figure [a), the question "WWat is the capital of Nebraska?" already provides the key information "Nebraska" eliminating the need for extracting relevant location information from visual content. A more appropriate question is "What is the capital of the highlighted area in the image?" to emphasize

---

the importance of visual understanding. (2) Answers are directly included in the textual questions. As shown in Figure [bb), LLMs can derive the correct answer "circle" through simple reasoning based on the question "What is the shape of the round dirt circle?".
To quantitatively substantiate our findings, we further experiment to gauge the proportion of these two types of samples in existing bench- marks. Specifically, we evaluate several bench- marks with two closed- source LLMs (GPT4-
MMMU
21.0%
MMB-EN
10.5%
57.2%
ScienceOA
A12D
46.2%
Turbo [4], and GeminiPro (51) and six open- source heavy LLMs (Intern M2- [52] Yi- 34B [62). Mixtral-8x77 -BB 23 Deepseek-67B
SEFD
12,4%
MathVista
10.3%
3. LLaMA2-70B (33, and Qwen 1.522.5-72B (I),
0%
10%
20%
40%
50%
60%
recording the hit count for each question. Here, the 'hit' refers to the ability of an LLM to cor-
Hit Rate (%)
Figure 2: LLM hit rate across various benchmarks. rectly answer the question without relying visual input. We then calculate the percentage of samples with a hit count of six or more (80%) against the total number of samples to determine the abnormal hit rate for each benchmark. As depicted in Figure 2 every benchmark shows a certain degree of samples that visual contents are unnecessary. with ScienceQA [38 and A12D (26) exhibit- ing amazing abnormal hit rates of 57.2% and 46.2%, respectively. Based multi-modal benchmarks have yet to fully assess the multi- modal capabilities of LVLMs.
observations, most
Second issue: unintentional data leaking exists in LLM and LVLM training. Although the community has the trend towards developing new multi-modal benchmarks to assess LVLMs' ca- pabilities from various dimensions, there is scant consideration for faimess and reliability during evaluation. Training LLMs and LVLMs requires vast and diverse data, inevitably leading to the leakage of evaluation samples. Such incidents are usually unintended, it's impractical to predict which data will be used in future evaluation benchmarks during the preparation for training corpus.
Figure | (c) showcases an evaluation sample leaked by LLMs. Though the question requires an understanding of image content, 16 out 22 tested LLMs astonishingly provide the correct response by "recalling" their training data. To quantitatively support our observations, we evaluate 22 leading I.MSs across popular benchmarks and report the 2-shot results Table [ Specifically, we find the 2-shot evaluation strategy is more stable than the O-shot (see results in Section A.8) to reduce refusal for answering and align answer formats. Under the impact of dision-independenn samples and data leakage from LLMs, GeminiPro [51] and Qwenl.5-775 [ achieve remarkable average accuracy of 42.4% and 43.6% under the 2-shot setting, outperforming random choice by 21.4% and 22.6%, respectively. Furthermore, Qwen 1. .5772B achieves score of 42.4% on MMMU [64], even surpassing the performance of the majority LVLMs with accessing images. This result serves as a reminder: if we only consider the final accuracy on benchmarks when evaluating LVLMs, potential data leakage from LLMs could lead to unfair comparisons.
In F and Section A.5 we showcase some examples where original LLMs fail, but LVLMS without accessing images succeed. Despite these questions requiring image content for accurate answers, the LVLMs without accessing images are capable of correctly answering these questions which stump original LLMs. To further support our hypotheses of data leakage during LVLMs' multi-modal training, we conduct an intriguing experiment. We remove the images for LVLMs and only utilize questions and options for evaluation, with results reported the gains of LVLMs set to receive only text inputs (LVLM-text) against their corresponding LLM bases (LLM) to quantitatively assess the degree of data leakage in LVLMs'
Table 2 We compare
training. As
shown in Table 2. most LVLMs exhibit varying degrees of data Icakage during multi-mode modal training. For example, the LLMs of Sphinx-X-sa X-887B [17] and Monkey- -hhtt (3), show a respective average performance gain of 14.1% % and 14.2% compared to their original LLMs.
Drawing from our observations, we posit that the issue of data leakage in multi-modal datasets is a significant concern that warrants attention. Addressing this issue is essential to ensuring that model performance is measured by their genuine ability to integrate and interpret multimodal data, rather than by their tendency to memorize specific samples within the dataset. Establishing a robust and reliable benchmark to minimize data leakage would thus serve as a foundational step in advancing research within the field of multimodal language models, paving the way for more meaningful and accurate evaluations of their performance and potential.
5

---

Table 2: Evaluation of various LVLMS on six popular multi-modal benchmarks, Forthe "strategy" column, "LLM"" refers evaluating using the corresponding I.IM base of the L.VM,, while "I.V.M-text" denotes evaluating LVLMs without accessing images. We employ the 0-shot inference strategy for LLMs align the evaluation protocols of LVLMs. We only report the results of closed-source LVLMS and open-source LVLMs due to space limits. For the entire LVLMs' results, please refer to the appendix. The highest results of the LVLM-text setting across the models are highlighted in bold and underlined.
Param.
Strategy
ScienceQA
A12D
Math
Avg.
Baseline
Random Choice
22.1
0.0
24.2
23.8
24.3
17.9
18.7
Closed-source LVLMs corresponding LLM bases
LLM
41.2
12.2
64.3
59.7
10.1
24.2
35.3
GPT4V
LVLM-text
45.1
17.6
68.2
62.5
28.4
25.4
41.2
LVLM
53.6
69.6
81.4
75.3
71.6
44.7
66.0
LLM
42.9
18.4
68.9
59.2
35.5
23.3
41.4
GeminiPro (Visio)
M.LM
39.4
16.7
66.3
54.5
27.9
24.5
38.2
(GeminiProff 51)
LVLM
44.4
68.1
80.6
68.0
64.3
36.0
60.2
Open-source LVLMs and corresponding LLM bases
LLM
20.0
7.2
47.1
38.7
26.4
22.0
26.9
TinyLLaVA 69
3B
LVLM-text
30.0
21.0
62.3
$1.9
37.2
23.5
37.7
(Phi2-2.7B 40)
LVLM
36.0
66.9
62.4
70.1
55.6
LLM
29.9
10.3
58.9
42.5
32.6
22.0
32.7
LLaVA-1.5(11
7B
LVLM-1ext
29.9
19.5
64.1
48.7
37,5
20.3
36.7
(Veuna-vl.5-7BB9
LVLM
34.4
65.0
68.7
55.6
65.6
23.6
52.2
LLM
32.8
8.9
64.0
48.3
31.9
18.9
34,1
IntemL_M2XXC2
7B
LVLM-text
34.2
26.2
71.9
63.3
38.1
29.4
43.9
(Inteml M2
LVLM
41.7
79.6
96.7
81.4
74.9
57.4
72.0
Monkey-Chat (Qwen-BBB)
L.M
19.8
8.4
52.7
42.6
7.6
20.5
25.3
10B
LVLM-text
32.4
15.6
71.1
56.8
36.1
25.0
39.5
LVLM
37.1
71.0
82.4
68.5
69.1
34.0
60.4
LLM
29.9
10.3
58.9
42.5
32.6
22.0
32.7
CogVLM-Cha (Vicuna-vI.5-7B)
178
LVLM-text
30.1
15.5
54.6
52.5
36.7
25.0
35.7
LVLM
34,2
63.4
66.3
63.3
68.7
34.7
55.1
LLM
37.1
10.5
53.6
57.3
37.3
21.7
36.3
Yi-VL
34B
LVLM-tent
37.3
23.2
68.6
59.9
41.0
22.7
42.1
LVLM
43.2
71.5
75.3
65.9
68.1
25.6
58.3
LIM
37.6
20.1
69.4
60.2
35.0
17.9
401.0
VI-Chat-v1.222 (NH2-Yi-34B 41)
40B
LVLM-1ext
41.7
23.9
70.3
65.0
40.5
24.0
44.2
LVLM
49.1
82.4
82.5
78.5
75.4
47.7
69.3
LLM
25.7
8.6
$7.2
48.7
13.5
23.4
29.5
Sphinx-X-MoE[I
S7B
LVLM-1cXt
43.6
20.5
68.4
61.1
39.9
28.4
43.7
(Mixtral-&x7B 23
LVLM
44.8
69.2
72.2
65.0
71.1
38.1
60.1
4 MMStar
4.1
Data Curation Process
Criteria for data curation. The evaluation samples for constructing the MMStar benchmark should meet three fundamental criteria: 1) Visual dependency. The collected samples can be correctly an- swered only based on understanding the visual content; 2) Minimal data leakage. The collected samples should minimize the risk of unintentional inclusion in LLMs' training corpus, or be effec- tively transformed from uni-modal to multi-modal formats to prevent LLMs from "recalling" the correct answers; 3) Requiring advanced multi-modal capabilities for resolution. In addition to ensuring fairness and reliability by adhering to the above criteria, we also aim for samples to cover various difficuuly levels. We expect to comprehensively capture LVLMs' multi-modal capabilities with succinct high-quality samples.
27 focused on natural images and four centered Data filter. We first choose two benchmarks 34 26
on scientific and technical knowledge 64
37 for our sample collection. We then develop an automated pipeline to preliminarily filter out samples that do not meet the first two criteria.
38
Specifically, we employ two closed-source L.LMs [51] and six open-source LLMS 52 62
3 23 53 sizing 20B larger to serve inspectors. These open-source LLMs are applied with

---

ASDI
atribat pol coleeett rosugottion
object denatiu
inng: smaston
úngll innane unribute coumning
kmage appl & qualis
om instacee atrybut compariens
Filter
Fine-grained Perception
Image acone tipic
crose-insinnce relation reasoning
Instance
Reasoning
Perception
(1AA7))
O.6
zuineric coromotienuue calcsiation
dragram reazoning
Logical Reasoning
Mathematics
Science &
Mutionvl anatysls
contiat reaseming
Technology
orde sepronor rassnig
Montandk CV-V
146
5d
ADto
Staties during the data curation process
(b) Compusition of MMStar
Figure 3: Details of MMStar benchmark. (a) Statics of the data sources during the data curation process.
After applying the coarse filter process and manual review, we narrow down from total of 22,401 samples to 11,607 candidate samples and finally select 1,500 high-qualin, samples to construct MMStar benchmark. (b) We display
capabilities the inner ring, with 18 detailed axes presented the outer ring. The middle ring showcases number of samples each detailed dimension.
2-shot in-context inference strategy to minimize response refusals and ensure consistency in answer formatting. Following this, we evaluate the sample pool with these LLM inspectors, documenting the hit frequency for each evaluation sample. Finally. only retain those samples with hit counts of two or fewer hits, indicating that around 75% of LLM inspectors fail to provide the correct answer. As illustrated in Figure 3 (a), following this initial coarse filtering, our sample pool was reduced from 22,401 to 11,607.
Manual review. After the coarse filtering with LLM inspectors, we further employ three experts to conduct the manual review process to ensure: 1) each sample's answer should be based on the un- derstanding of visual content; 2) selected samples should cover a comprehensive range of capability assessment dimensions: 3) most samples should require LVLMs possess advanced multi-modal abilities for resolution. To expedite the manual selection of samples with varying difficulty levels for LVLMs, we tally the hit counts all LVLMs on the coarsely filtered samples and split them into four difficulty categories: easy (12-16, 148 examples), moderate (8-11, 189 examples), hard (4-7, 631 examples), and tough (0-3, 532 examples). Finally, after considering both the diversity of capability dimensions and difficulty levels, manually curated 1,500 high-quality samples from the coarsely filtered set. Figure 3 (a) showcases the detailed composition of data sources for our final selection of samples. In Section A.3 we provide details on how the manual review step aggressively reduces the MMStar benchmark from 11,607 samples to 1,500 samples.
4.2 Core Capabilities
We select and consolidate dimensions used for assessing LVLMs' multi-modal capabilities in existing benchmarks and identify six core capabilities along with eighteen detailed axes. In Figure (b), we provide statisties for each core capability and their detailed axes on the MMStar benchmark. More detailed definitions of each capability provided in Section A.2
7

---

Table 3: Evaluation of various LVLMs on MMStar. We report the results of 2 closed-source LVLMs and 14 open-source LVLMs with varying sizes and architectures. We report the detailed results of the CP (coarse perception). düne-grained perception), IR(instance reasoning). LR (logical reasoning). ST (science tech- nology), and MA (mathematics) core capabilities. The best results are highlighted in bold and underlined. The worst results of multi-modal gain (MG) and multi-modal leakage (ML) metrics are in italic red.
Model
LLM
Param. CP
IR
LR ST MA Avg.
MGT MIL
Baselines
Random Choice
23.7 24.5 25.3 24.3
24.8
25.1
24.6
Closed-source LVLMs
GeminiPro- Vision 51
GeminiPro [1]
51.6 50.8
46.0
28.4
50.0
42.6
27.4
GPT4V (low) 43
GPT4-Turbo 42
62.0
32.8
55.2
48.0
33.6
44,8
46.1
32.6
1.3
GPT4V (high) 43
GPT4-Turbu 42
76.6 51.4
66.6
55.8
42.6
49.8
57.1
43.6
1.3
Open-source LVLMs
Lin 69
Phi2-2.7B 40
3B
60.4
31.6
50.8
30.4
18.0
24.8
36.0
16.4
7.6
62
622
58.0
46.4
34.8
20.4
34.0
37.9
15.6
LLaVA-1.5 31
Vicuma-v1.5-7B 9
7B
58.8
24.0
38.8
24.0
13.6
22.8
30.3
10.7
0.0
ShareGPT4V[51
Vicuna-v1.5-7BI
7B
58.8
28.0
45.6
24.4
17.2
24.0
11.9
0.0
InternLM-XC2 13
InterLLM2-7B S2
7B
70.8
48.8
65.2
56.4
42.0
49.2
$5.4
28.1
7.5
Deepseck-V1. [36
Deepseek-78B3
8B
64.0
30.8
49.2
36.4
21.6
20.4
37.1
15.7
0.0
Qwen-VLL VL-Chatt
Qwen-7B[[]
10B
59.6
50.8
29.2
22.0
23.9
Monkey-Chat (30]
Qwen-7B(I]
10B
57.6
36.4
51.6
33.2
26.4
24.4
38.3
13.5
17.6
L.aVA-1.5{31
Vicuna-vl.5-13B
9
138
58.8
41.6
24.4
18.4
25.6
32.8
13.9
CogVLM-Chate
Vicuna-v1.5-7BBB
17B
66.8
36.8
49.2
31.2
23.6
11.6
36.5
14.9
0.0
Yi-
Yi-34B
34B
53.2
31.2
52.0
32.4
12.4
35.2
36.1
18.8
0.0
L.x 32
NH2-Yi-34BBP
348
66.4
52.0
62.4
46.0
32.4
53.6
52.1
29.4
NH2-Yi-34B(41
40B
67.6
43.2
61.2
47.2
24.0
19.2
43.7
32.6
0.0
Sphinx-X-MOE 17
Mixtral-&x7B
57B
58.4
40.8
47.6
35.2
19.2
32.0
38.9
14.8
1.0
4.3 Multi-modal Gain/Leakage
Given our observation of the potential for inadvertent leakage of some evaluation samples during multi-modal training process, the vanilla evaluation approach struggles to reveal LVLMs' actual performance gains derived from multi-modal training and fails to enable fair comparison with other competitors. Therefore, we propose two novel metrics to separately assess the degree of data leakage and actual performance gain from the multi-modal training process.
To calculate the multi-modal gain (MG) metric for given LVLM on particular benchmark, need to compute the scores of the same LVLM with and without visual inputs, separately denoted as S and Sov. Then the MG metric can be derived from the following formulation:
MG Swe S.r.
To calculate the multi-modal leakage (ML) metric, we need to compute the extra score of the given LVLM's LLM base (without any multi-modal training), denoted as S,. Then the ML metric is formulated as follows:
ML max Soz -)).
Experiments
In this section, we conduct a systematic analysis of the proposed MMStar benchmark along with the
MG/ML metrics. These analyses encompass various LLMs and LVLMs, and also involve numer- ous existing benchmarks when examining MG/MI. metries. We choose VIMEyalKit [15] as our codebase. Please see details about experimental setups in A.ttionA.]

---

5.1 Results Analysis of MMStar
In this section, we present a comprehensive comparison of various LLMs and LVLMs performed on our MMStar benchmark and summarize our key observations in the following parts.
Observation from LLMss We comprehensively evaluate 2 closed-source LLMs and 20 open- source LLMs of varying sizes and architectures on the MMStar benchmark and report the results in Figure 4 and Table 5 Encouragingly, the per- formance of these LLMs is almost indistinguish- able from random choice, effectively validating that the evaluation samples of our MMStar ex-
30
Qweni.5-1.5g
Average Spore on MMStur
25
Vicuna-11.5-73
FaMaa3-21
Misirn-keeR Despoedk
Viott-Char
LLMA2--18 Y134
Gentii Pro
Qwenl.5-728
* IntemLM222001
LLLMA2-708
GPT4-Ty
hibit significant visual dependency and minimal data leakage from LLMs. Notably, the smallest model, Q.5en..5-1.80 achieves the best score. We conjecture this is due to it suffering the least stringent safety restrictions, thereby reducing in- stances of refusal to answer. Moreover, among the six core capabilities of MMStar, science & technology (ST) prove to be the most challenging dimension for LLMs. The best score on ST is around 30% in other dimensions. We spec-
* Phit-2 718
10-
Qwen-
10
20
30
40
50
60 70
Number Model Parameters (Dillions)
Figure 4: LLMs perform close to random guessing (the dashed line) on MMStar.
only 23, 23.2%, significantly lower than the best scores of ulate this may be that samples within the ST dimension have the least degree of data leakage from LLMs' training data.
Observation from LVLMs. We evaluate 2 closed-source and 14 open-source LVLMs on our MM- Star, with the results reported 3 As shown in the table, GPT4V[43 with a high-resolution setting can achieve the best average score of 57.1% % among all LVLMs. Increasing the resolution and number of image tokens can boost the average score from 46. 1% to 57.1% % for GPT4V, of- fering a positive signal to the research community. Among the open-source LVLMs, InternLM- Xcomposer2 133 achieves an impressive score of 55.4%. LLa [32 even surpasses GPT4V and GeminiPro- Vision [51] in the mathematics (MA) core capability. Notably, LVLMs managed to reach a passing average score (i.e. 60%) in the core capabilities of fine-gr mne-ggained perception (FP), logical reasoning (LR), science & Technology (ST), and mathematics (MA). highlighting these di- mensions as particularly challenging for existing I.V.Ms.. Moreover, TinyLLavA [69], despite its modest 3B scale, outperformed some competitors of 7B and even 13B surprisingly, underscoring the potential of smaller-scale LVLMSs
5.2 Analysis of Multi-modal Gain (MG) and Multi-modal Leakage (ML)
Analysis from the model perspective. In Table we illustrate the MG/ML (Multi-modal Gain/Multi-modal Leakage) metrics for each LVLM across each benchmark and provide an aver- age MG/ML metric across all benchmarks in the final column. For closed-source LVLMs, GPT4V demonstrates notable performance gains attributed to its multi-modal training, while GeminiPro- Vision shows lesser data leakage during multi-modal training. This suggests that GPT4V may have utilized a broader range of multi-modal training data compared to GeminiPro- Vision. Among the open-source LVLMs, DitemLL-XCComppoo across all benchmarks, whereas LLaVA- 1.5-BB records the lowest at 14.8. This outcome is reason- able given that LLL employed the least amount of multi-modal training data among these open-source LVL.Ms. Despite L 1.5-BB having the lowest average multi-modal gain, it ex- hibits minimal multi-mode modal leakage. Additionally, models like Monkey-Char Ophinx-X-Moo, and Deepseck-Vy VL display higher degrees of mudti-modal leakage, highlighting the need for the commu- nity to consider this factor for fair comparisons.
achieves the highest average multi-modal gain of 28.1
Analysis from the benchmark perspective. In the final row of Table 4 we list the average multi- modal gain and multi-modal leakage for existing LVLMs across all benchmarks for analysis. MM- Bench registers the highest average multi-mode gain at 50.1, indicating significant overlap be- tween the domains covered by existing LVLMs' training data and MMBench. Conversely, MMMU exhibits the lowest average multi-modal gain at 5.8, suggesting lesser degree of overlap between the domains of existing LVI.Ms training corpora and those included in MMMU. Additionally, MM- Star, as expected, has the lowest degree of multi-mode leakage at 1.9. This provides comprehen-

---

Table 4: Evaluation of various LVLMs on 7 Benchmarks with multi-modal gain (MG) and multi-modal leakage (MI.) metries. We report the results of 2 closed-source LVLMSs and 14 open-source LVLMs with varying sizes and architectures. The bottom row represents the average across models while the rightmost column shows the average across benchmarks for the same LVLM. The best results are highlighted in bold and underlined, The worst results of MG and ML metrics are in italic red.
the same benchmark,
MMMU
MMB
ScienceQA
A12D
SEED
Vista
MMStar
Avg.
Model
Param.
MGf MLJ MG1 ML.,
MGf MLJ MG1 MLJ MG ML.
MGT MLJ
MGT MLJ
MGT ML]
Closed-source LVLMs
GPT4V +
8.5
3.9
52.0
5.4
13.2
3.9
12.8
2.8
43.2
18.3
19.3
1.2
32.6
1.3
25.9
5.3
PraminiPro-Vssion[3
5.0
0.0
51.4
0.0
14.3
0.0
13.5
0.0
36.4
0.0
11.5
1.2
27.4
0.0
22.8
0.2
Open-sone moen-source LVLMs
TinyLLaVA 69
3B
6.0
10.0
45.9
13.8
6.8
15.2
10.5
13.2
32,9
10.8
5.4
1.5
16.4
7.6
17.7
10.3
Yi-VI 62
6B
5.3
7.4
45.6
14.1
5.1
9.4
3.9
16.6
29.2
10.9
3.8
3.0
15.6
0.0
15.5
8.8
LLaVA-1.50 1.5
7B
4.5
0.0
45.5
9.2
4.6
5.2
6.9
6.2
28.1
4.9
3.3
0.0
10.7
0.0
14.8
3.6
Share(iP144
7B
3.5
1.8
49.1
10.1
4.2
6.3
8.5
6.9
31.7
5.1
3.0
0.7
11.9
0.0
16.0
4.4
IntemLLM-XC21
7B
7.5
14
53.4
17.3
24.8
7.9
18.1
15.0
36.8
6.2
28.0
10.5
28.1
7.5
28.1
9.4
Deepseek-VL.
8B
3.2
10.6
49.6
15.5
14.3
10.8
11.6
14.9
33.7
23.1
11,4
3.3
15.7
0.0
19.9
11.2
Qwen-VL-Chat
10B
10.0
4.2
49.6
0.3
11.0
4.0
12.3
6.4
44.5
11.9
11.4
0.3
23.9
0.0
23.2
3.9
Monkey-Chat
30
10B
4.7
12.6
55.4
7.2
11.3
18.4
11.7
14.2
33.0
28.5
9.0
4.5
13.5
11.1
19.8
13.8
LLaVA-1.5
13B
9.6
0.0
47.2
9.8
5.7
7.0
8.6
7.2
31.1
10.7
5.3
1.5
13.9
0.0
17.3
5.2
CogVL.M-Cha1555
17B
4.1
0.2
47.9
5.2
11.7
0.0
10.8
10.0
32.0
4.1
9.7
3.0
14,9
0.0
18.7
3.2
62
34B
5.9
0.2
48.3
12.7
6.7
15.0
6.0
2.6
27.1
3.7
2.9
1.0
18.8
0.0
16.5
5.0
Nex
2.8
54.7
4.8
11.2
1.5
5.6
34.1
6.7
16.5
29.4
2.4
23.6
IntemVL--Chatvl..2
40B
7.4
4.1
58.5
3.8
12.2
0.9
13.5
4.8
34.9
5.5
23.7
6.1
32.6
0.0
26.1
3.6
Sphinx-X-Mi Moimxx-X-MMofle
57B
1.2
17.9
48.7
11.9
3.8
11.2
3.9
12.4
31.2
26.4
9.7
5.0
14.8
1.0
16.2
12.3
Avg. models
5.8
4.9
50.1
8.9
10.0
7.4
10.3
8.7
33.7
1111
10.8
3.0
20.0
1.9
sive and fair arena for comparing existing LVLMs. Moreover, we believe evaluating existing LVLMs to derive average ML metrics can also be helpful to the following works in examining newly devel- oped multi-modal benchmarks.
Conclusion
In this work, we dig into current evaluation works for large vision-language models (LVLMs) and identify two primary issues: 1) visual content unnecessary for many samples, and 2) uninten- tional data leakage exists in LLM and LVLM training. To address these issues, we develop elite vision-dependent multi-modal benchmark named MMStar and propose two metrics to measure the data leakage and actual performance gain in LVLMs' multi-modal training. MMStar undergoes the manual review of each sample, covering 6 core capabilities and 18 detailed axes for an in-depth evaluation of LVLMs' multimodal capabilities. In our evaluation of 16 diverse LVLMs MM- Star, even the best model scores under 60 on average. We also analyze the MG and MI. metrics across 6 multimodal benchmarks and MMStar, providing valuable insights for the community on gathering multimodal training data and crafting new benchmarks. In the future, we plan to expand MMStar into a larger, online test and explore dynamic evaluation methods to maintain sample visual dependency and reduce accidental data leakage into LLM's and LVLM's training corpora.
7 Acknowledgments
This work was supported by the Anhui Provincial Natural Science Foundation under Grant 2108085UD12. We acknowledge the partial support of the GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC. This work was also partially supported by the Shanghai Artificial Intelligence Laboratory, the National Key R&D Program of China 00922D00602011 the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Inno- vation and Technology Commission (ITC''s InnoHK Dahua Lin is Pl of CPI under the InnoHK.
10

---

References
[1] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report, arXiv preprint arXiv:2309. 16609, 2023.
[2] J. Bai, S. Bai, S. Yang. S. Wang. S. Tan, P. Wang. J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:238. 2023.
[3] X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, et Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXi::400.002544 2024.
[4] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neclakantan, P. Shyam, G. Sas- try, A. Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
[5] L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint 2xxx:23111.2278 2023.
[6] L. Chen, X. Wei, J. Li, X. Dong, P. Zhang, Y. Zang. Z. Chen, H. Duan, Lin, Z. Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint arXiv:2406.00 04325, 2024.
[7] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, Z. Muyan, Q. Zhang, X. Zhu, L. Lu, et al. In- tervl:: Scaling up vision foundation models and aligning for generic visual-linguisti tasks, arXiv preprint arXiv:2312. 14238, 2023.
[8] S. Cheng, Z. Guo, J. Wu, K. Fang, P. Li, H. Liu, and Y. Liu, Can vision-languee models think from a first-person perspective? arXiv preprint arXiv:2311. 2023.
[9] W.-L. Chiang. Z. Li, Z. Lin. Y. Sheng. Z. Wu, Zhang, L. Zheng, Zhuang. Y. Zhuang, J. E. Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https:/vicuna. Imsys. org (uccessed 14 April 2023), 2023.
[10] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. 2022.
[11] O. Contributors. Opencompass: A universal evaluation platform for foundation models. https:// github. com/ open- commpass/opencooppan 2023.
[12] Dai, J. Li. D. Li, A. M. Tiong. Zhao, Wang. B. Li, P. Fung, and Hoi. Instructblip: Towards general-purpose laniune models with instruction tuning, 2023.
[13] X. Dong, Zhang, Y. Zang, Y. Cao, B. Wang, I.. Ouyang, X. Wei, S. Zhang, H. Duan, M. Cao,, et al. Internim-xcome xcooposexxom Mastering free-form text-image composition and comprehension vision- language large model. arXiv preprint arXiv:2401.16420 2024.
[14] Z. Du, Y. Qian, X. Liu, M. Ding, Qiu. Z. Yang, and Tang. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arxi: 103,60360 2021.
[15] H. Duan, J. Yang, Y. Qiao, X. Fang, L. Chen, Y. Liu, X. Dong, Y. Zang, P. Zhang, Wang. et al. Vimevalkit: An open-soure toolkit for evaluating large multi modality models. arXiv preprint arXiv:2407.1 116911,00..1691,,1. 2024.
[16] C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, Z. Qiu, W. Lin, J. Yang, X. Zheng, K. Li. X. Sun, and R. Ji. Mme: comprehensive evaluation benchmark for multimodal large language models, arXiv preprint arXiv:2306. 13344, 2023.
[17] P. Gao, R. Zhang, C. Liu, L. Qiu, S. Huang, W. Lin, Zhao, Geng, Z. Lin, P. Jin, al. Sphinx- x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint axXiv:2402.05935 2024.
[18] Y. Goyal, Khot, D. Summers- Stay, D. Batra, and Parikh. Making the v in vqa matter: Elevating the role of image understanding visual question answering. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 6904-6913, 2017.
[19] Q Huang, X. Dong, P. Zhang, Wang, C. He, Wang. Lin, Zhang, and N. Yu. Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and cnrispectionaalocaaiin In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13418- 13427, 2024.
11

---

[20] Q. Huang, X. Dong, P. Zhang. Y. Zang, Y. Cao, J. Wang, D. Lin, W. Zhang, and N. Yu. Deciphering cross-modal alignment in large vision-language models with modality integration rate. arXiv preprint urXiv:2410.07667. 2024.
[21] C. Jia, Y. Yang, Y. Xia, Y.T. Chen, 7. Parckh, H. Pham, Q. Le, Y.H.. Sung, 7. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR, 2021.
[22] A. Q. Jiang. A. Sablayrolles, A. Mensch, C. Bamford. D. S. Chaplot, D. d. 1. Casas, F. Bressand, G. Lengyel. G. Lample, Saulnier, al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[23] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. I. Casas, Mixtral of experts, arXiv preprint arXiv:2401. 04088, 2024.
E. B. Hanna, Bressand,
[24] D. Jiang, R. Zhang, Z. Guo, Y. Wu, Lei. P. Qiu, P. Lu, Z. Chen, G. Song, P. Gao, et el. Mmsearch: Bench- marking the potential of large models as multi-modal search engines. arXiv preprint arXiv:249.11955, 2024.
[25] Y. Jiao, S. Chen, Z. Jie. J. Chen, L. Ma, and Y-G. Jiang. Lumen: Unleashing versatile vision-centric capabilitics of large multimodal models. arXiv preprint arXiv:2403.07:344 2024.
[26] A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi. A diagram is worth a dozen images. ArXiv, abs// 07396, 2016.
[27] B. Li, R. Wang. G. Wang. Y. Ge. Y. Ge, and Y. Shan. Seed-bench: Benchmarking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023.
[28] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 2023.
[29] Y. Li, W. Tian, Jiao, J. Chen, and Y.G. Jiang. Eyes can deceive: Benchmarking counterfactual reason- ing abilities of modal large language models. arXiv preprint arXiv:2404. 2ete24444..266, 12966, 2024.
[30] Z. Li, B. Yang, Q. Liu, Z. Ma, S. Zhang, J. Yang. Y. Sun, Y. Liu, and X. Bai. Monkey: Image resolution 1.06607, and text label are important things for large multi-modaa models. arXiv preprint arXiv:231 1.060723..006607, 2023.
[31] H. Liu, C. Li, Y. Li, and Y. Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.0374.222 2023.
[32] H. Liu, Li, Y. Li, B. Li, Zhang, Shen, and J. Lec. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[33] H. Liu, C. Li, Q. Wu, and J. Lee. Visual instruction tuning. arXiv preprint arXiv:2304.00 084888 2023.
[34] Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He. Z. Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.066. 220.006811.. 2023.
[35] Z. Liu, T. Chu, Y. Zang, X. Wei, X. Dong, P. Zhang, Z. Liang, Y. Xiong, Y. Qiao, D. Lin, et al. Mmdu: A multi turn multi image dialog understanding benchmark and cerructiio-tuningg dataset for Ivims. arXiv preprint arXiv:2406. / /833, 2024.
[36] H. Lu. W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, Y. Sun, et al. Deepseek-vl: Towards world vision-language understanding. arXiv preprint arXiv:2403.05522 2024.
[37] P. Lu, H. Bansal, T. Xia, J. Liu, C. Li. H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
[38] P. Lu, S. Mishra, T. Xia, L. Qiu, K.W. Chang, S.-C. Zhu, O. Tafford, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507 25217,2521,22 2022.
[39] G. Luo, Y. Zhou, T. Ren, S. Chen, X. Sun, and R. Ji. Cheap and quick: Efficient vision-language instruc- tion tuning for large language models. arXiv preprint arXiv:2305.1500 2023.
[40] Microsoft. Phi2: The surprising power of small language models. htttp::/www.micrrrt com/ 2ha---22thhhuuupiie watprrro--ooo-o--aa--aa--a------ 2023.
en-us/research/bllog
[41] NousResea
httpss//hagggngfaaoooo//ooossss
Nous Heeeeeeeeeeea-e 2- 2-B 2023.
12

---

[42] OpenAL. Chatgpt. htttpe://chat. httppe/:/ccaa..peaaa..ooo com/
2023.
[43] OpenAI. Gpt-4vfision) system card. https: cán wttp:////ddcooooa....cmmm 2023.
Swwtee_aeeaad...e_aa pdf
[44] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[45] Y. Qiao, H. Duan, X. Fang, J. Yang. L. Chen, S. Zhang, J. Wang, D. Lin, and K. Chen. Prism: A framework for decoupling and assessing the capabilities of vlms. arXiv preprint arXiv:2406. N1v:22006.14544, 2024.
[46] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 8748-8763, PMLR, 2021.
[47] D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and Mottaghi. A-okvqa: benchmark for visual question answering using world knowledge. European Conference on Computer Vision, pages 146- 162. Springer, 2022.
[48] Shao, S. Qian, H. Xiao, G. Song, Z. Zong. Wang, Y. Liu, and H. Li. Visual cot: Unleashing thain-of-thought reasoning in multi-modal language models. arXiv preprint arXiv:2403.1699 2024.
[49] P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 2556-2565, 2018.
[50] H. Taud and J.-F. Mas. Multilayer perceptron (mlp). Geomatic Approaches for Modeling Land Change Scenarios, pages 451-455, 2018
[51] G. Team, R. Anil, S. Borgeaud, Y. J.B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. Dai, A. Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint 1rXiv:23/2./1800 2023.
[52] 1. Team. Internlm: A multilingual language model with progressively enhanced capabilities, 2023.
[53] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov. S. Batra, P. Bhar- gava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint mrXiv:2307.009880 2023.
[54] J. Wang. L. Meng, Z. Weng. B. He. Z. Wu, and Y.G. Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.075577 2023.
[55] W. Wang, Q Lv, W. Yu, W. Hong, J. Qi. Y. Wang, J. Ji, Z. Yang, L. Zhao, X. Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXi::2311.0003 2023.
[56] H. Wu, Z. Zhang. E. Zhang, C. Chen, L. Liao. A. Wang. C. Li, W. Sun. Q. Yan, G. Zhai. et al. Q-bench: A benchmark for general- 2023.
general-purpose purpose foundation models low-level vision. arXiv preprint 23iv:2309.14881
[57] P. Xia, Z. Chen, J. Tian, Y. Gong. R. Hou, Y. Xu, Z. Wu, Z. Fan, Y. Zhou, K. Zhu, al. Cares: A comprehensive benchmark of trustworthiness in medical vision language models. arXiv preprint arXiv:2406. 06007, 2024.
[58] P. Xia, Han, S. Qiu, Y. Zhou, Wang, W. Zheng, Z. Chen, C. Cui, M. Ding, L. Li, et al. Mmie: Mas- sive multimodal interlcaved comprehension benchmark for large rision-language models, arXiv preprint arXiv:2410. 10139, 2024.
[59] L. Xing, Q. Huang, X. Dong, J. Lu, P. Zhang. Y. Zang, Y. Cao, C. He, J. Wang. F. Wu, et al. Pyramiddrop: Accelcrating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint 100 17247, 2024.
[60] A. Yang, B. Xiao, B. Wang, B. Zhang, C. Yin, C. Lv, D. Pan. D. Wang. D. Yan, F. Yang, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309. 10305, 2023.
[61] Q. Ye, H. Xu, G. Xu, J. Yc, M. Yan, Y. Zhou, J. Wang, A. Hu, P. Shi, Y. Shi, et al. mplug-owl: Modular- ization empowers large language models with multimodality. arXiv preprint 23t::2304.14778 2023.
[62] A. Young. B. Chen, C. Li, C. Huang. G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint aixiv 240.00652,, 2024.
13

---

[63] W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint MrXiv:2308.022290 2023.
[64] X. Yue, Y. Ni, K. Zhang. T. Zheng. R. Liu, G. Zhang, S. Stevens, D. Jiang. W. Ren, Y. Sun, al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint wxiv:2311.16500 2023.
[65] G. Zhang, S. Qu, J. Liu, C. Zhang, C. Lin, C. L. Yu, D. Pan, E. Cheng, J. Liu, Q. Lin, et al. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv:2405. 19327. 2024.
[66] Zhang, Jiao, S. Chen, Chen, and Y.-G. Jiang. Eventhallusion: Diagnosing event hallucinations video llms. arXiv preprint arXiv:2409. 16597, 2024.
[67] Zhang, Dong. Y. Zang. Cao, R. Qian, L. Chen, Q. Guo, Duan, Wang. Ouyang, al. Internlm-xen 2rrrmllm-xxompperr22255 A versatile large vision language model supporting oong-conexxtual input and output. arXiv preprint arXiv:2407.0333 033003 2024.
[68] P. Zhang, X. D. B. Wang. Y. Cao, C. Xu, L. Ouyang, Z. Zhao, S. Ding, S. Zhang, H. Duan, H. Yan, et al. Internim Internlm-xcompose xcomposer: A vision- language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.1511 15112, 2023.
[69] B. Zhou, Y. Hu, X. Weng. J. Jia, J. Luo. X. Liu, J. Wu, and L. Huang. Tinyllava: A framework of small-scalc large multimodal models. arXiv preprint arXiv:2402.14288 2024.
[70] Y. Zhou, C. Cui, R. Rafailov, C. Finn, and H. Yao. Aligning modalities in vision large language models via preference tuninnnnin, arXiv preprint arXiv:2402.111 11411, 2024.
[71] Y. Zhou, Z. Fan, D. Cheng, S. Yang. Z. Chen, C. Cui. X. Wang. Y. Li, L. Zhang, and H. Yao. Calibrated self-rewarding vision language models, arXiv preprint arXiv:2405. 14622, 2024.
[72] D. Zhu, J. Chen. X. Shen, X. Li. and M. Elhoseiny, Minigpt-4: Enhancing vision- languaee understanding with advanced large language models. arXiv preprint arXiv:2304. 2.t.::203...52..... 2023.
[73] Z. Zong, B. Ma, D. Shen, G. Song, H. Shao, D. Jiang. H. Li, and Y. Liu. Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint 440i 13046, 2024.
14

---

A
Appendix
In the supplementary material, we provide more results and analysis and summarize them as follows:
Section
A.
we detail the experimental setups.
In Section
A.2
we provide detailed explanation of our MMStar benchmark, which en-
compasses definitions six core capabilities and eighteen detailed axes.
we provide the details about the manual check process to aggressively reduce the MMStar benchmark from 1 11,607 to 1,500 samples.
In Section
A.3
we present the comprehensive performance of 22 LLMs across various dimensions on our MMStar benchmark.
In Section
A.4
In Section
we present statistics about the data leaked from the existing public multi- modal benchmarks to selected LVLMs' training corpus, along with some specific examples.
A.5
In Section A.6
we showcase additional samples from existing benchmarks that either lack visual dependency or have been leaked into the training corpora of LLMs or LVLMs.
In Section A.7
showcase
samples in MMStar of cach detailed axe.
we provide detailed performance results of 22 L.Ms across 6 public multi- modal benchmarks under the 0-shot evaluation strategy. Moreover, place the complete performance of 16 LVL.Ms with/without accessing images across these multi-modal bench- marks.
In Section A.8
In Section A.9 we discuss the limitation and future work.
A.1 Experimental Setups
Evaluation models. 1) Baseline: We utilize random choice to as the baseline, which randomly
selects an option as the answer, 2) Large Language Models: We prepare two closed-source LLMs, GPT4 2 and GeminiPro [51, and 20 popular open-source LLMs sizing from 1.8B to 72B for
40). Vicuna series (9) Deepseek series 3, MnternI.22 series [52], Baichuan2 series (60). Yi series (62). Mistral series
text-only evaluation, such as Qwen series (1) LLaMA2 series [53 Phi2
22 23 Additionally, all the open-source LLMS we used are their Chat versions. and 3) Large
LanguaeL Models: We prepare two closed-source LVLMs, GPT4V (43) and GeminiPro (Vision) [51], and 14 popular open-source LVLMs sizing from 3B to 57B, such as TinyLLaVA- 2 LLaVA-1.5 series (31], LLaVA-Next-34B (32]
3B 669 Yi-VL series (62], Qwen-VL-Chat
CogVLM-Chat-177 (55), IntemVL-Chhatv1.22
Sphinx-X-8x7B 17..
Implementation details. For evaluating LLMs existing benchmarks, employ both 0-shot and 2-shot strategies and will specify which is utilized when reporting results. For evaluating LLMs on MMStar, the 0-shot strategy yields poor scores, making comparisons difficult. Therefore, exclusively utilize the 2-shot strategy to decrease the frequency of refusal to answer. Moreover, All I.V.Ms are evaluated utilizing the 0-shot strategy across all benchmarks ensure fair comparison. When evaluating L.L.Ms under the 'LVLM-ttxtt setting (i.e. answer without the image), most LVLMs work well by simply removing the image tokens from their default input tokens. However, GeminiPro-Visse Vision [51] and Cog VLM-Chat (55) require the replacement of the original images with pure grey images to bypass image content input and operate correctly. Given that all questions are ensured to be converted into multiple-choice format, we develop some heuristic matching rules to calculate accuracy, avoiding the cumbersome process of re-invoking GPT4 for answer extraction. Moreover, all experiments in this study are conducted within the same codebase modified from Vi.MEvalKit [1], and utilize NVIDIA A A100 GPUs for non-APP-bant evaluation.
A.2 Definitions of Core Capabilities and Detailed Axes
The core capabilities consist of two perception-relate dimensions, Coarse Perception (CP) and fine-grainee Perception (FP), two reasoning-relatee related dimensions, Instance Reasoning (IR) and Logi- cal Reasoning (LR), and two knowledge-relae ematics (MA). We detail the complete definitions as follows:
dimensions, Science & Technology (ST) and Math-
Coarse Perception (CP). This core dimension refers to the capability to understand and interpret the overarching characteristics and themes of an image without delving into the finer details. It
15

---

encompasses a broad, holistic view of the visual content, enabling the identification of 1) image style & quality; 2) image scene & topic; and 3) image emotion.
Fine-grained Perception (FP). This core dimension represents a sophisticated level of image under- standing that focuses on the detailed and nuanced aspects of visual content. It involves deep dive into the specifics of images: 1) attribute & celebrity recognition; 2) object location; and 3) object counting. This core dimension unveils the subtle intricacies that coarse perception might overlook.
Instance Reasoning (IR). It encapsulates a set of advanced cognitive capabilities focused on un- derstanding and interpreting individual and collective object attributes and interrelations within an image. This process goes beyond mere recognition, delving into the analytical assessment of 1) single-instance attribute reasoning; 2) cross-instance attribute comparison; and 3) cross-instance lation reasoning. It is a critical component for systems requiring a deep semantic understanding of visual content, enabling nuanced interaction with and response complex visual content.
Logical Reasoning (LR). This core dimension encompasses a sophisticated framework of cogni- tive processes designed to interpret, deduce, and infer conclusions from visual content through structured approach to logic and reasoning. This multi-faceted capability marries the intuitive un- derstanding of visual content with the structured rigor of logical deduction, enabling: I) diagram reasoning; 2) code & sequence reasoning; and 3) common reasoning.
Science & Technology (ST). It consists of a comprehensive framework for the application and integration of knowledge across broad spectrum of science and technology. This domain combines the theoretical underpinnings and practical applications of various fields: 1) natural science; 2) engineering: and 3) geography & earth science.
Mathematics (MA). Math is a foundational pillar of logical and analytical reasoning and encom- passes a broad spectrum of capabilities essential for understanding, applying, and interpreting quan- titative and spatial information. We primarily consider three aspects for evaluating LVLMs' logical thinking prowess: 1) numeric commonsense calculation; 2) geometry; and 3) statistical analysis.
A.3 Details of Manual Check
After roughly filtering the original data pool with 8 advanced LLMs, resulting in 11,607 candidate samples, we initiate rigorous manual review phase. First, we establish 6 core evaluation dimen- sions and 18 detailed axes by integrating the evaluation dimensions from existing benchmarks. Next, use 16 LVLMs to infer and count the number of hits for each sample. Furthermore, we design a UI interface listing the current sample's 's image, options, answer, sample source, hit count, and the 18 detailed axes. The samples arranged in ascending order based on the number of hits. The formal manual selection and benchmark construction process is as follows:
Three experts are each responsible for two core capability dimensions (i.c.. 6 detailed axes). They need to review all candidate samples and select and correctly classify the samples belonging to their respective dimensions. The samples selected must retain their visual dependency. Statistical Analysis: After the preliminary classification, we consider the numerical balance between dimensions and the difficulty level of the samples. Samples under the "coarse perception" dimension approach 4,000, while those under "logical reasoning" are fewer than 700. In terms of difficulty distribution, there are 4,555 easy (i.e., number of hits between 12 and 16) samples but only 2,758 tough (i.e., number of hits between 0 and 3) ones. Given these premises, a lot of repetitive simple samples, such those merely asking for the color of object in the image, are not what we desire.
Preliminary Classification:
Initial Benchmark: After considering both the numerical balance and difficulty level of the sam- ples, we set the total sample number of the benchmark at 1,500, with each core capability dimension containing 250 samples. Then, we assign each expert two core capability dimensions, instructing them to prioritize sample difficulty when selecting 250 samples per dimension.
To minimize personal bias, we arrange for each expert to review the dimensions handled by the other two experts after the initial benchmark is constructed. Samples with issues are replaced by correct samples of the same difficulty level from the candidate pool. Moreover, we also provide the number of samples with consensus before and after the cross-validation step in the man- ual review process for MMStar in the table below. Only samples that all three experts unanimously
Cross-Vaile Validation:
16

---

agree upon are retained; otherwise, they are replaced with samples of the same difficulty level from the candidate pool.
A.4 Performance Comparison of Various LLMs MMStar
Table LLMs failed to solve problems in MMStar and performed close random guessing, visual con- tent is necessary solve MMStar. We evaluate various L.LMs MMStar the 2-shot inference strategy. We report the results of 2 closed-source LLMs and 20 open-source LLMSs with varying sizes and architectures. We report the detailed results of the CP (cnarse perception), FP (fine-grained perception), IR(instance reason-
ing). LR (logical reasoning). ST (science & technology). and MA (mathematics) capabilities. The best
results highlighted in bold and underlined.
Model
CP
FP
IR
LR
ST
MA
Avg.
Baselines
Random Choice
23.7
24.5
25.3
24.3
24.8
25.1
24.6
Closed-source LLMs
GPT4-Turbo
2.4
4.0
9.6
18.0
13.6
25.6
12.2
Gemini-Par 51
16.8
13.6
20.4
24.4
19.6
28.8
20.6
Open-source LLMs
Owenl.5-1a.
28.4
28.4
25.6
23.2
23.2
29.6
26.4
Phi2-2.7B 40
11.2
11.2
15.2
10.8
11.6
12.0
12.0
Yi-6B-Chat
23.6
19.2
28.4
25.2
12.4
29.6
23.1
LLaMA2-7B [53
28.0
30.4
26.0
18.0
18.8
21.6
23.8
Qwen-7B[1
11.6
5.6
12.8
5.6
7.2
0.4
7.2
Deepseek-7BB
26.8
16.0
28.4
21.6
23.2
25.6
23.6
Internl.M2-7B55]
22.0
14.8
22.0
21.6
15.2
23.2
19.8
Qwenl.5-7B[I
15.6
8.0
9.2
9.2
15.2
9.2
11.1
Vicuna-v1.5-7BB9
22.0
27.6
29.6
26.4
18.0
24.4
24.7
Baichuan2-7B 60
20.8
18.4
27.6
18.8
18.8
21.2
20.9
Mistral-7B[22
20.0
23.6
24.4
23.6
20.0
27.2
23.1
LLaMA2-13B [33
23.6
23.6
28.0
21.2
16.4
10.4
20.5
Vicuna-v1.5-133
32.8
24.0
28.8
17.6
22.0
14.4
23.3
Baichuan2-1333 60
26.4
18.0
28.0
20.4
21.2
25.6
23.3
InternLM2-20B 52
18.2
17.8
22.6
23.8
17.8
13.4
18.9
Yi-34B
20.4
18.0
24.0
24.0
14.4
30.8
21.9
Mixtral-8x7B 23
24.4
17.6
19.2
28.0
16.0
33.6
23.1
Deepseek-67B
29.2
22.4
18.4
26.0
20.4
22.4
23.1
LLaMA2-70B
22.4
20.0
19.6
14.4
7.2
9.6
15.5
Owenl.5-722 [
21.6
16.0
21.2
14.0
17.2
27.2
19.5
17

---

A.5 Multi-modal Leakage in Existing Multi-modal Benchmarks
diugram?
Oprioms
A: B: sobacco C-AAtimal cell
Which letter murks uncient Egypt!
A12D question 99201 41..M-INIAM- (68.8%)
Where the the living room?
Oprions:
Options:
A:
To the left the sofa
Bellnnd thesofa
image, get
C
To right
D:A O
LLM
D: In frons otthe sofa.
ScienceQA th question 1938
can't soe the image either, but appear have seai the question before, got
SECD Bench mee question 92857 V.L.-INTTM Pairs 13/16 (81.3%)
MIM-VVMM Pairs: 10/16 (62.8%)
LVLM
IQieaiios
400
200
190)
12
4t MIM-MIMM Pairs
organ appears Chis radiogmphh? simage
question requiring an integer atswer and provide the final value.e.g. 2, 3, the end.
Options:
the same materiaa us the big brown thing?
Liver
Question: What
procedures rogister
of mumber
Duodenum
Gallbbaakerr &
Options:
A: gray
that 2007?
B: blue
Anewer: 12
yellow
cyan
question-917
MMMU P4: question-316 3765
MMBenchh question-605
ALLM-LYEM Pairs: 8/16 (50.0%)
MLIIM-NNIM Pains 9/16 (56.3%)
ULLM-LNIM Pairs 10/16 (62.5%)
Figure 5: Illuttationn of data Icakage during LVLMs' multi-modal training processes. We showcase sam-
correctly but LVLMs without accessing images (LVLM-text) can. Each LLM- L pair represents an LLM and corresponding LVLM without accessing images, totaling 16 pairs. The chart in the center tallies the number of samples in existing benchmarks hit by more than half of the LLM- LVLM ferr pairs, underscoring the issue of data leakage during the multi-modal modal training process.
ples that LLMs cannot
18

---

A.6 More Failure Examples in Existing Multi-modal Benchmarks
Crust
The outermost layer carth known
Outer
core
B: Crust
Inser
C: Mantie
Coro
D:: Inner core
Answer:
1.M.Ms 22/22 (100%)
A12D question-
figure <imagel>, refers a play tagic
events?
Tragedy
B: Comedy
C:
D: Plot
Answer:
#Correct LLMs 22/22 (100%)
MMMU question-709
How otters use their pockets?
A: They store the food they catch in their pockets. They keep babies inside pockets. C: nan Dmnnn
Answer: A
(100%)
MMBenchP*r question-397
primary purpose bathroom?
A: Cooking
Sleeping
Exercising
D): Rathing
SEHI Hentl--Heehhaaa maaee
Answer: #Correet (100%)
Select the reptile below.
A: sca offer
B: toctoise
nan
D:
Answer: B
#Correct LLMs: 22/22 (100%)
SeiencelAATe, question-315
Figure 6: We highlight cases in existing benchmarks where evaluation samples lack the visual necessary.
19

---

Which historical development contributed the formation of the society associated with the image?
A: The Atantic slave trade
refugees caused by Seven C: The segregation of maces imposed by British colonial authorities migratory pattems encouraged by arrival smallpox from Furoge
MMMU question-6935
Answer: A
#Correct LLMs 22/22 (100%)
image?
A: Three white adirondack chairs on a stone patio
B: house with garden front
C: Flowers in gurden
D: garden with pillow
Answer:
#Correct LLMs 22/22 (100%)
SEED-Bcich question-3083
food web pistured,which image is prey?
A: Fox
Buzzard
C. Mouse
D: Dragonfly
Answer: C
#Correct LLMs 22/22 (100%)
A12D lot, question-26300 26302
image, professional role?
posture suggest about
A: The man's attire suggests that be might have professional occupation for B: The man's attire suggests that he works creative industry
formal appearance
suggests D: The man's sttire suggests that be professional athletc.
artending a event
MrBennchoo guestion-150
Answer:
what shown? twalve
A: half
quarter
12:00
C: a'clock
E: quarter past
Answer: C
#Correct LLMs: 19/22 (86.4%)
question-623
Figure 7: We highlight cases in existing benchmarks where evaluation samples are leaked into LLMs' training data.
20

---

What is centra focus the image?
A: Pienio table under the tree
B: Tree midele the grassy field
C: Small building the field of trees
D: Small house the grassy field
Answer:
SEED-Bcnch**>: questioe-1667 1677
MLLM-LVLM Pairs: 12/16 (75.0%)
What can Samir and Derek trade to ench get what they want?
oier Swich
111 602
A: Samir trade his tomatoes for Derek's
crenge
benaloes
B: Derek can trade his broccoli for Samir's oringes.
brcocal
C: Derek trade his almonds Samir's tomatoes.
wike
Dr Samir can trade his tomatoes for Derek's broccoli.
ScienceQAfel; question- 1224
Answer: D
MLLL VLM Pairs: 11/16 (68.8%)
patieed
Rorra lade
peoto wob
diagram above what?
A: Far
Brain
C. Mouth
D: Head
Answer: D
mede
HLLM-LVLM Pairs: 11/16 (68.8%)
A12D ls question-285400
image?
Red
B: Green
D: Blue
SEED DeDDD--encaaae question-61583
Answer:
HLLM-LVEM
What the symmetry function?
(A) (0, 0)
(-),-1100
(2) 0)
Answer: A
#LLM-LVLM Pairs 13/16 (81.3%)
Muth question-762
Figure 8: We highlight cases in existing benchmarks where evaluation samples are leaked into LVLMs' multi- modal training data.
21

---

A.7 More Examples in MMStar
Coarse Perception
image scene and topic
image emotion
image style & quality
alerr plock
What predominant
Which mood this image convey?'
Which category this image belong to? oil painting. C: digital art,
image? White, Red,
Cozy.
Anxiouss Angry
ς Happy,
Bysketch,
l Blue,
D: Silver
D: photo
Fine-grained Perception
localization
recognition
object counting
Where Christmas located the image?
What main shirt the wearing??
many people are present the image?
//t the left-aan hand side the
A: White,
ByBlue,
One,
ByThree,
B: It on the right-hand side the image,
C: Pink.
D. Black
C: Two, D. Four
C: It the center the
D: is not the image
Instance Reasoning
single-instance reasoning
cross-instance attribute reasoning
cross-instance relation reasoning
What's the profession of the people in this picture?
Which image is the brightest one? A: upper left, C: down left.
arrow
and the curve sign?
By upper right, down right
y sign,
pointing away
mason,
By plumbee,
pilot,
B: police
pointing to curve sign,
C: The anow and the curve unrelated,
D: The arrow and the curve sign are overlspping
Figure 9: More examples in MMStar
22

---

Logical Reasoning
common reasoning
diagram reasoning
code & sequence reasoning
the caunt Ist
1x1
count
ceunt int
Tho count tei
The count int
6ood lyp!
Subtract all yellow metallic balls. Subtract small yellow shiny things. How many objects left?
How many years have value less than 10%?
Pyrkon image?
geventt
A. count 0 while foount e 10) print TThe count ist, count count count prin "Goed bye!". Bycoun 0 while (count 4 99; print Thee count M. count
A:
1. D
4,
5. DyS
C: 2,
"Good bye!".
print
count count count prin "Good bye!". Dr count 0 whill (count c 9) pein:" 'hee count is count count coust + print "Good bye!"
Science & Technology
biology chemistry physics
geography & carth seience & agriculture
electronies & energy & mechanical eng.
X
Y Z
what is label in diagram?
Which ocean is highlighted? Aythe Indian Occan, B: the Atlantic Ocean, C: the Pacific Ocean, Southern Ocean
Which schedule an equivalent serial
A: nucleus,
B: chloroplast, D: cell sap vacuole
preeeddace graph <image
Ç cell wall.
1>7 13 12,
T2
C: > T2 T3, DyThhre are
graph.
Mathematics
geometry
statistical reasoning
numeric commonsense and calculation
rolls
1020
-03
-145
Stidlogram
2
1530
-28.
24
5521
-05
-132
1827
-03
SBidlog
982
532
32
English mullie
$0kklogra
B
Besilles
$5/dllgg 5m
As shown in the figure, the straight line a b, the point B on the straight line b, and AB BC, 22 65.0, O, then the dogree of 21 is B/25
What the real return the stock market in 1932?
Colton wants
buy 1+3/10
kilograms of English muffins. How much will he spend? (Unit: S)
A: -14,3%%
H: -21.277% D: 56.52%
0.45%
A/10.4 CO D1
B.2
A 65"
C 35
D
Figure 10: More examples in MMStar
23

---

A.8 More Results on Public Multi-modal Benchmarks
Table 6: Evaluation of various LLMs six popular multi-modal benchmarks. We employ O-shot in-
ference strategy evaluating all L.Mss. report the results 2 closed-source I.I.Ms and 20 open-source LLMs with varying sizes and architectures. The evaluated benchmarks include MMMU (MMMU-VE Val [64], MMB (MMBench-EN) Beeth--N-DDe Dev [34), ScienceQA (ScienceQA-TTe -estt (88). A12D (A12D-Tes! (26)., SEED (SEED-
Image 27)). and MathVista MistaaitttMMinn B7). The best results are highlighted in bold and underlined.
Model
Strategy
MMMU
MMB
ScienceQA
A12D
SEED
Math
Avg.
Baselines
Random Choice
22.1
0.0
24.2
23.8
24.3
17.9
18.7
Closed-source LLMs
GPT4-Turbu 42
0-shol 0-shot
41.2
12.2
64.3
59.7
10.1
24.2
35.3
GeminiPro 51
42.9
18.4
68.9
59.2
35.5
23.3
41.4
Open- source LLMs
Qwenl.5-1..
0-shot
29.0
10.0
54.3
37.9
28.9
20.4
30.1
Phi2-2.7B
0-shot
20.0
7.2
47.1
26.4
26.9
Yi-6B 62
O-shot
25.7
9.5
58.1
39.1
27.4
21.2
30.2
LLaMA2-7B 533
0-shot
23.6
11.5
56.8
43.5
31.7
24.1
31.9
Qwen-7B
O-shot
19.8
8.4
52.7
42.6
7.6
20.5
25.3
Deepseek- -Bccc
0-shot
21.6
8.4
56.3
38.1
13.4
20.6
26.4
InternL.M2-7B
O-shot
32.8
8.9
64.0
48.3
31.9
34.1
O-shot
25.0
11.4
62.3
49.4
19.4
19.9
31.2
Vicuna-vI.5-777
0-shot
29.9
10.3
58.9
42.5
32.6
22.0
32.7
Baichuan2-788
O-shot
25.7
10.5
52.7
44.0
29.2
30.5
Mistral-7B [2
0-shot
30,0
13.2
63.4
48.5
34.3
22.6
35.3
LLaMA2- 13B
0-shot
24.4
10.1
33.6
32.7
Vicuna-v 1,5-138
(--hot
28.3
11.6
59.5
45.0
26.3
19.6
31.7
Baichuan2-113 13B 60
0-shot
22.1
4.7
51.1
32.8
25.4
20.3
26.1
Internl -M2-2008 52
(-shot
32.2
15.9
55.7
26.0
21.3
35.8
Yi-34B 62]
0-shot
37.1
10.5
53.6
57.3
37.3
21.7
36.3
Mixtral 8x7B
0-shot
25.7
8.6
13.5
29.5
Deepseck-67BB
(-shot
30.9
14.8
64.3
57.5
17.1
23.2
34.6
LLaMA2-700 70B
0-shot
28.9
12.3
62.2
48.6
34.3
25.2
35.3
Qwen --572B
0-shot
21.4
10.1
57.5
44.2
8.8
19.5
26.9
24

---

Table 7: Evaluation of various LVLMs on six popular multi-modal benchmarks. For the "strategy" column, "LLM"" refers to evaluating using the corresponding L.M base of the L.V.M, while "IVL.M-text" denotes evaluating L.LMss without accessing images. We employ the (-shot inference strategy for I.LMs align the evaluation protocols LVLMs. The highest results of the LVLM-text setting across models are highlighted bold and underlined.
Model
Param.
Strategy
MMMU
MMB
ScienceQA
A12D
SEED
Math
Avg.
Baseline
Random Choice
22.1
0.O
24.2
23.8
24.3
17.9
18.7
Clused-source LVLMs and corresponding LLM bases
LLM
64.3
GPT4V (GPT4- Turbo
LVLM-text
45.1
68.2
62.5
28.4
25.4
41.2
LVLM
53.6
81.4
75.3
71.6
44.7
66.0
GeminiPro- VisooVisionnn
51
LLM
42.9
68.9
35.5
23.3
41.4
LVLM-1cXt
39.4
16.7
66.3
54.5
27.9
24.5
38.2
(GeminiPro{31)
LVLM
44.4
68.1
80.6
68.0
64.3
36.0
60.2
Open-soure LVLMs and corresponding LLM boses
LLM
20.0
7.2
47.1
38.7
26.4
22.0
26.9
(Phi2-2.78
3B
LVLM-tex
30.0
21.0
62.3
51.9
37.2
23.5
37.7
LVLM
36.0
66.9
69.1
62.4
70.1
28.9
53.6
LLM
25.7
9.5
58.1
39.1
27.4
21.2
30.2
Yi-VL
6B
LVLM-tcxt
33.1
23.6
67.5
55.7
38.3
24.2
40.4
(Yi-6
LVLM
38.4
69.2
72.6
59.6
67.5
28.0
55.9
LLM
29.9
10.3
58.9
42.5
32.6
22.0
32.7
LLaVA-1.5
7B
LVLM-text
29.9
19.5
64.1
48.7
37.5
20.3
36.7
Vicuma-vl.5-7BB)
LVLM
34.4
65.0
68.7
55.6
65.6
23.6
52.2
L.M
29.9
10.3
58.9
42.5
32.6
22.0
32.7
Share(iPT4v
LVLM--eE
31.7
65.2
49.4
37.7
22.7
37.9
(Vicuma-v1.5-71B))
LVLM
35.2
69.4
57.9
69.4
25.7
54.5
LLM
32.8
64.0
48.3
31.9
18.9
34.1
Internl 13
(In
LVLM-1ext
34.2
26.2
71.9
63.3
38.1
29.4
43.9
LVLM
41.7
96.7
81.4
74.9
57.4
72.0
LLM
19.8
52.7
42.6
20.5
Owen-VL-Chat 2 (Qwen-7B
LVLM-text
24.0
56.7
49.0
19.5
20.8
34.0
67.7
61.3
64.0
32.2
52.9
LLM
56.3
13.4
26.4
Deepseck V1 16
LVLM-te
23.9
(Deepseek
35.4
70.2
35.3
LLM
19.8
$2.7
42.6
7.6
20.5
25.3
Monkey Char 50
LVLM-1ex
32.4
71.1
(Qwen-7BB)
LVLM
37.1
82.4
68.5
69.1
34.0
60.4
LLM
28.3
11.6
59.5
45,0
26.3
31.7
LLaVA-1 [1
LVLM-1ext
26.0
21.4
66.5
52.2
37.0
21.1
37.4
(Vicuna-vI 5-1
LVLM
35.6
72.2
60.8
68.1
26.4
55.3
LLM
29,9
10.3
58.9
42.5
32.6
22.0
32.7
CogVL.M-Chal
171
LVLM-
30.1
15.5
54.6
52.5
36.7
25.0
35.7
(Vicuma-vl.5-7110)
LVLM
34.2
63.4
66.3
63.3
68.7
34.7
55.1
LLM
37.1
10.5
$3.6
$7.3
37.3
21.7
36.3
Yi-VL
34B
LVLM-tex
37.3
23.2
68.6
59.9
41.0
22.7
42.1
(Y2
LVLM
43.2
71.5
75.3
65.9
68.1
25.6
58.3
LLM
37.6
20.1
69.4
60.2
35.0
17.9
37.2
LLaVA-Next (NH2-Yi-34B
34B
LVLM-1ext
40.4
24.9
70.9
65.8
41.7
22.2
44.3
LVLM
47.0
79.6
82.1
78.6
75.8
38.7
67.0
LLM
37.6
20.1
69.4
60.2
35.0
17.9
40.0
IntemVL-CCattvl.22 (NH2-Yi- NH-2-Yi-31BBB
40B
LVLM-E
41.7
23.9
70.3
65.0
40.5
24.0
44.2
LVLM
49.1
$2.4
82.5
78.5
75.4
47.7
69.3
LLM
25.7
8.6
57.2
48.7
13.5
23.4
29.5
Sphinx MoE
57B
LVLM-1eXE
43.6
20.5
68.4
61.1
39.9
28.4
43.7
(Mixtral 8x7B
LVLMM
44.8
69.2
72.2
65.0
71.1
38.1
60.1
25

---

A.9 Limitations
While we have expended significant effort to filter out evaluation samples that are visually dependent and have not been leaked into the training corpora of existing LLMs and LVLMs for our MMStar benchmark, challenging ensure that these samples will not be inadvertently included in the expanded training materials of future L.Ms and LVLMs. Although the metrics we proposed, such as multi-modal gain and multi-modal leakage, can reflect this issue to some extent, test set without provided answers is still needed to further assess the actual multi-modal capabilities of existing LVLMs. We plan to construct new of disend-de test samples for AMStar-Test in our future work.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide clear claims in the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Section A.9
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
This work does not involve any theoretical assumptions or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See them in Section 3 Section 5 and Section A.1
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification: N/A
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips...
Policylllly for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips.c///////
ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ye]]
Justification:
See Section A.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Compute resource constraints.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
29

---

is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproducc the experiments?
Question:
Answer: [Yes]
Justification: See Section A.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tprrips///r/ps.../// http:///cccc..occ....cccccc.oaaa
Answer: [Yes]
Justification: With checked.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Section A.9
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
30

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
The data used in this work are from previous public benchmarks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with
necessary safeguards to allow for controlled use of the model, for example by quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [No]
Justification:
The data used this work are from previous public benchmarks.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
31

---

Answer: [Yes]
Justification: See Section A.9 and the data used in this work are from previous public benchmarks.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve research with Human Subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "ASJ0xSxAUu", "reviewer": "Reviewer_btYn", "summary": "The authors have identified two primary concerns with the benchmarks commonly used for large vision-language models (LVLMs). Firstly, many samples do not require visual content to answer the questions. Secondly, they noted unintentional data leakage during LVLM training. They assessed eight large language models (LLMs) across six widely-used multi-modal LLM benchmarks, demonstrating that LLMs can correctly answer a significant portion of questions without visual input. To more reliably evaluate LVLM performance, they developed a new benchmark by meticulously filtering data from six existing benchmarks with three requirements: 1) visual dependency, 2) minimal data leakage and 3) multi-modal capability for resolutions. Additionally, they designed two metrics: Multi-modal Gain, to quantify the improvement from multi-modal training, and Multi-modal Leakage, to assess the extent of potential data leakage. Using this new benchmark and the two metrics, they provide a comprehensive comparison of state-of-the-art LVLMs.", "questions": "- Please clarify the goal of using proposed metrics\n\n- Please clarify the decision to aggressively reduce the size of final benchmark", "limitations": "The authors properly discuss the limitation in the paper", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-structured and clearly articulated, facilitating ease of comprehension.\n\n- The evaluation process is meticulously designed, and the conclusions drawn from it are convincing.\n\n- The findings presented in this paper meaningfully impact multi-modal large language model (LLM) research. Researchers have depended heavily on benchmarks without thoroughly examining their quality. The authors question the reliability of evaluations based on these benchmarks. Without a reliable benchmark, it is impossible to faithfully measure actual multi-modal gain. They developed a new benchmark, MMStar, which facilitates more reliable evaluations.\n\n- Using the MMStar benchmark, the authors evaluated two closed-source and fourteen open-source large vision-language models (LVLMs), with the results presented in Table 3. As expected, GPT4 emerged as the top performer in five out of six tasks. Additionally, they underscored the efficacy of smaller-scale models by highlighting that TinyLLaVa, a model with 3 billion parameters, outperformed some larger competitors with 7 billion and 13 billion parameters, thereby emphasizing the potential of smaller-scale LVLMs.", "weaknesses": "- The proposed metrics, Multi-modal Gain and Multi-modal Leakage, are dependent on the base LLM utilized in the large vision-language models. This dependency complicates the use of these metrics for directly comparing the multi-modal gain across different LVLMs.\n\n- The manual review step aggressively reduces the MMStar benchmark from 11,607 samples to 1,500 samples. The explanation provided in Section 3.1 for this reduction is somewhat vague and lacks clear, objective criteria for filtering. I am curious about the rationale behind such an aggressive reduction by nearly tenfold. Is this reduction due to a scarcity of data meeting the three specified criteria mentioned between line 187 to 189, or are there other reasons for this decision?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the positive comments on the novelty and meaningful impact of our findings and proposed benchmark. We detail your concerns and our corresponding responses below:\n\n**Q1: The proposed metrics, Multi-modal Gain (MG) and Multi-modal Leakage (ML),  are dependent on the base LLM utilized in the large vision-language models. This dependency complicates the use of these metrics for directly comparing the multi-modal gain across different LVLMs.**\n\n**A1:** In fact, when developers want to evaluate the actual multi-modal capabilities of their LVLMs, they can directly choose our MMStar benchmark and perform inference with LVLM for quick evaluation. Furthermore, ML and MG can reflect the extent of data leakage during multimodal training and the actual improvement in the model's multimodal capabilities. We have already provided the performance of several popular LLM bases in our work, and we will continue to update with new LLM bases such as LLaMA3 and Gemma2 on MMStar. This will facilitate the community in quickly evaluating and comparing the multimodal gains of their models.\n\nAdditionally, the proposed MG and ML metrics can serve as probes for the training corpus of LVLMs. For instance, averaged over seven benchmarks, ShareGPT4V-7B, compared to LLaVA-1.5-7B, significantly increases its MG by incorporating high-quality image-caption data under the same architecture without affecting ML, demonstrating the importance of high-quality image-caption data. Similarly, with an average of over 16 models, MMMU exhibited the lowest average MG, indicating minimal overlap between the multimodal training data of LVLMs and the samples in MMMU.\n \n**Q2: The manual review step aggressively reduces the MMStar benchmark from 11,607 samples to 1,500 samples. The explanation provided in Section 3.1 for this reduction is somewhat vague and lacks clear, objective criteria for filtering. I am curious about the rationale behind such an aggressive reduction by nearly tenfold. Is this reduction due to a scarcity of data meeting the three specified criteria mentioned between lines 187 to 189, or are there other reasons for this decision?**\n\n**A2:** Thank you for pointing out the lack of clarity in our description of the manual review stage. We provide a detailed supplement on this stage here and will integrate these details into the main text. After roughly filtering the original data pool with 8 advanced LLMs, resulting in 11,607 candidate samples, we initiate a rigorous manual review phase.\n\nFirst, we establish 6 core evaluation dimensions and 18 detailed axes by integrating the evaluation dimensions from existing benchmarks. Next, we use 16 LVLMs to infer and count the number of hits for each sample. Furthermore, we design a UI interface listing the current sample's image, options, answer, sample source, hit count, and the 18 detailed axes. The samples are arranged in ascending order based on the number of hits.\n\nThe formal manual selection and benchmark construction process is as follows:\n\n1. Preliminary Classification: Three experts are each responsible for two core capability dimensions (i.e., 6 detailed axes). They need to review all candidate samples and select and correctly classify the samples belonging to their respective dimensions. The samples selected must retain their visual dependency.\n2. Statistical Analysis: After the preliminary classification, we consider the numerical balance between dimensions and the difficulty level of the samples. Samples under the \"coarse perception\" dimension approach 4,000, while those under \"logical reasoning\" are fewer than 700. In terms of difficulty distribution, there are 4,555 easy (i.e., number of hits between 12 and 16) samples but only 2,758 tough (i.e., number of hits between 0 and 3) ones. Given these premises, a lot of  repetitive simple samples, such as those merely asking for the color of an object in the image, are not what we desire.\n3. Initial Benchmark: After considering both the numerical balance and difficulty level of the samples, we set the total sample number of the benchmark at 1,500, with each core capability dimension containing 250 samples. Then, we assign each expert two core capability dimensions, instructing them to prioritize sample difficulty when selecting 250 samples per dimension.\n4. Cross-Validation: To minimize personal bias, we arrange for each expert to review the dimensions handled by the other two experts after the initial benchmark is constructed. Samples with issues are replaced by correct samples of the same difficulty level from the candidate pool.\nBy following this thorough process, we ensure a balanced and challenging benchmark set.\n\nIf you still have any concerns or aspects you would like to discuss further, please do not hesitate to contact us at any time.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UmtJvazGAF", "reviewer": "Reviewer_bz7V", "summary": "In this paper, the authors examine current benchmarks for large vision-language models (LVLMs) and identify two main problems: 1) many samples do not require visual content, and 2) there is unintentional data leakage in LLM and LVLM training. To address these issues, they developed a multimodal benchmark called MMStar, consisting of 1,500 samples, and proposed two metrics to measure data leakage and performance gain in LVLMs’ multimodal training. They conducted empirical evaluations on 16 LVLMs to report their performance on MMStar.", "questions": "Please respond to the points of weakness I mentioned above.", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and easy to follow.\n2. The motivation behind the study is clear, and the empirical analysis is thorough.\n3. Data curation for MMStar is comprehensively explained.\n4. The proposed performance metrics are intuitive and effectively presented.", "weaknesses": "1. The authors only consider multiple-choice questions for the MMStar benchmark. Including a wider variety of well-curated questions without choices would be great.\n2. Similar to Figure 2, the authors should provide the LLM Hit Rate for the MMStar benchmark.\n3. What is the percentage distribution of the 1,500 samples across the four difficulty categories?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are encouraged to see that you found our work intuitive, containing extensive experiments and empirical analysis, and well-written. We have endeavored to address your concerns as follows:\n\n**Q1: The authors only consider multiple-choice questions for the MMStar benchmark. Including a wider variety of well-curated questions without choices would be great.**\n\n**A1:** Thank you for your suggestion. In the current version of the MMStar benchmark, we choose to use a multiple-choice question format for two reasons:\n\n1. Most of the mainstream multi-modal benchmarks are in multiple-choice format. We carefully select samples from these benchmarks to construct a comprehensive, fully visually dependent benchmark, making MMStar a multiple-choice format as well.\n2. The multiple-choice format allows for more objective evaluation, avoiding fluctuations caused by variations in LLM versions (differences in LLM capabilities). For instance, in FreeVA [1], it was observed that the results of some open-ended multimodal benchmarks were easily influenced by the version of the GPT API used. \n\nHowever, if the questions and prompts given to the language model are appropriate, open-ended questions without options can indeed better assess the capabilities of LVLMs. We are open to exploring this approach in future work.\n\n**Q2: Similar to Figure 2, the authors should provide the LLM Hit Rate for the MMStar benchmark.**\n\n**A2:** Thanks for your valuable feedback. The LLM Hit Rate of MMStar is 0%, which is significantly lower than the lowest LLM Hit Rate of 10.3% observed in the previous 6 benchmarks. This exceptionally low LLM Hit Rate is a result of our meticulously designed benchmark construction pipeline. In the first step, we only select samples from 6 existing benchmarks that are hit 2 times or fewer by 8 advanced LLMs. We have detailed the statistics of the hit counts in MMStar in the table below. As shown, all samples in MMStar have hit counts far less than 6, resulting in an LLM Hit Rate of 0%. We will add them to the supplementary materials of the next version.\n\n| Number of hits | Number of samples |\n|----------------|-------------------|\n| 0              | 848               |\n| 1              | 392               |\n| 2              | 260               |\n\n**Q3: What is the percentage distribution of the 1,500 samples across the four difficulty categories?**\n\n**A3:** Thank you for your thorough review and reminder. In the table below, we present the difficulty distribution of samples in MMStar. As shown, nearly 80% of the samples are answered correctly by at most half (8) of the LVLMs, with easy samples comprising less than 10% of the total. This highlights MMStar's focus on challenging samples that require advanced multimodal capabilities from LVLMs.\n\n| Difficulty level | Number of samples |\n|------------------|-------------------|\n| Tough (0-3)      | 532               |\n| Hard (4-7)       | 631               |\n| Moderate (8-11)  | 189               |\n| Easy (12-16)     | 148               |\n\nYour constructive comments and criticisms will greatly assist us in improving this work. Please do not hesitate to contact us if you have any further questions.\n\n[1] FreeVA: Offline MLLM as Training-Free Video Assistant", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b9GSKGnJ4o", "reviewer": "Reviewer_JQaJ", "summary": "The current benchmarks used to evaluate Vision Language Models (VLMs) contain several flaws. In particular, a lot of questions can be answered without looking at the image at all. These benchmarks still being hard, the best proprietary models without looking at the images can obtain better scores than strong VLM baselines (looking at the images). As a result, the authors create MMStar, a difficult benchmark aiming at evaluating the capability of the vision-language tasks. They manually review their benchmark, and do several ablations to confirm its importance.", "questions": "I personally noticed many hallucinations in the SEED benchmark, with some Q/A pairs that are simply false.\nSince the Q/A pairs from SEED represent 28.3% of your dataset, I am worried that you would have such incorrect pairs in your benchmark.\nCan you confirm that this was removed during the manual filtering?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors provide a benchmark that is hard to be good at without looking at the images. This is a problem for some questions in MMMU and MathVista currently.\n- The authors manually reviewed the examples of the benchmark.\n- The dataset is nicely divided into 6 subtasks, evaluating different aspects. The fact that there is exactly the same number of examples in each of these subtasks is appreciated.\n- The fact that each question is a MCQ, instead of an open-ended question that would be difficult to evaluate due to the different output formats of the models, is also appreciated.", "weaknesses": "- In the released dataset, the choices are directly integrated into the prompt. It would be good to also add a column with only the original question, and another column containing the list with the possible options, so that researchers could evaluate their models with the prompts they used during their fine-tuning.\n- As the authors mentioned, it would be useful to also create a test set for this benchmark.\n- It would have probably made more sense to publish this in the Datasets and Benchmarks track.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thorough review and appreciation of our work. Below, we address your concerns point by point:\n\n**Q1: In the released dataset, the choices are directly integrated into the prompt. It would be good to also add a column with only the original question, and another column containing the list with the possible options, so that researchers could evaluate their models with the prompts they used during their fine-tuning.**\n\n**A1:** Good point!  We will revise the format as per your suggestions for the public release of the benchmark. Here is a description of the revised columns for the benchmark:\n- **Index**: The sample number in the benchmark.\n- **Question**: Contains only the question itself without options.\n- **Image**: The content of the image.\n- **Options**: Lists the content of all options.\n- **Answer**: A single letter indicating the correct answer.\n- **Category**: Indicates the core dimension to which the sample belongs.\n- **L2 Category**: Indicates the detailed axis to which the sample belongs.\n- **Meta Info**: Indicates the source of the sample from previous multimodal benchmarks.\nWe hope this format will facilitate evaluation within the LVLMs community.\n\n**Q2: As the authors mentioned, it would be useful to also create a test set for this benchmark.**\n\n**A2:** Thank you for your valuable feedback. We will strive to create a private test set in the future. We believe this test set can help the LVLMs community more comprehensively and fairly evaluate the actual multi-modal capabilities of existing models.\n\n**Q3: It would have probably made more sense to publish this in the Datasets and Benchmarks track.**\n\n**A3:** In fact, our work begins with two important and interesting observations regarding the evaluation of existing LVLMs. The first observation is that many samples in current benchmarks can be correctly answered without relying on visual content. The second observation is the phenomenon of data leakage hidden in the final evaluation scores of LVLMs. The second observation is derived from a detailed analysis of many carefully constructed experimental results, revealing the potential data leakage in LLMs and LVLMs that has not been adequately addressed in the current LVLM evaluation field.\n\nBased on these two observations, we propose two solutions. The first is a meticulously constructed benchmark that ensures all samples are visually dependent and are, as far as possible, not leaked in the training corpus of LLMs. However, this benchmark cannot entirely prevent some samples from being present in the training data of LVLMs. Additionally, it cannot guarantee that the samples will remain unexposed to LLMs and LVLMs introduced after the benchmark's creation date. Therefore, we also innovatively propose two supplementary metrics: Multi-modal Gain and Multi-modal Leakage.\n\nIt is important to note that these two metrics are not exclusively tied to MMStar. They can be used to measure the extent of data leakage and the actual multimodal capability improvements gained from multimodal training in any benchmark. These metrics allow researchers to evaluate multimodal training leakage and benefit levels at any time, independent of the benchmark's creation date.\n\nTherefore, the contributions of this work include two important and intuitive observations, a benchmark, and a set of multimodal evaluation methodologies. After careful discussion, we believe this work is more suitable for the main track.\n\n**Q4: I personally noticed many hallucinations in the SEED benchmark, with some Q/A pairs that are simply false. Since the Q/A pairs from SEED represent 28.3% of your dataset, I am worried that you would have such incorrect pairs in your benchmark. Can you confirm that this was removed during the manual filtering?**\n\n**A4:** Astute observation! The occurrence of hallucinations in samples seems inevitable for large-scale benchmarks. Compared to the initial candidate pool of 14,000 samples from SEED, we retained only around 400 samples in our final benchmark, significantly reducing the cost of manual review. Therefore, all samples in MMStar underwent cross-validation by three experts to minimize the issue of hallucinations as much as possible.\n\nYour constructive comments and criticisms will greatly assist us in improving this work. Please do not hesitate to contact us if you have any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9oTD6YpT9r", "reviewer": "Reviewer_sNEM", "summary": "This paper investigates the evaluation of large vision-language models (LVLMs) and the currently used benchmarks. Within the paper two primary issues are identified: the lack of need for visual information, and data leakage. Based on these issues a new compiled benchmark is proposed MMStar that includes a set of multi-modal samples validated by humans, and a variety of models are evaluated on this new benchmark.", "questions": "The paper raises a number of interesting questions and points out limitations of (some) existing benchmarks. Unfortunately, the data leakage issue appears to be not fully disentangled from the potential of data biases. I would appreciate input from the authors on this point.\n\nAdditionally, I would be interested in discussion on random guessing within the paper, and whether the ideas presented in [27] may address this - and then subsequently how this influences the findings in the paper. Also taking into account the majority class versus random guess.", "limitations": "I believe question 9 in the paper checklist has been answered incorrectly - or at least the justification given is not valid. Given that the paper discusses a benchmark that may include images of people or copyrighted material it is crucial that the authors affirm whether the work has been done in accordance to the ethical guidelines. Even if the data is a compilation of existing datasets - by selecting and combining information from these the new resulting dataset may be biased in ways the original datasets were not (e.g., by selecting only those images containing users from a certain demographic).", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "With the rapid progress of the field of LLM and LVLMs, it is crucial that we have reliable benchmarks for evaluation - this work explores existing benchmarks, identifies potential issues therein, and proposes a new benchmark which contributes to the quality of evaluation in the field. Additionally, the work proposes new metrics for evaluation and performs a benchmark evaluation of various models.", "weaknesses": "1) Random Chance\n\nAn issue in this work is that it insufficiently accounts for random guessing and data bias. As shown in [27], the ground-truth chance for it being answer A or B on MMbench are 26.4% (due to having questions with less than 4 options), and moreover, certain models may prefer certain answers. LLMs may get correct answers even when they require visual input. Given the 26.4% baseline for MMbench from [27], it is also surprising that the random choice value reported in this paper, for MMbench, is 0.0 in both Table 1 and Table 2. Moreover, an additional baseline based on majority class may also be necessary here.\n\nFor the results in Figure 2, which require 6 out of 8 models to get a hit it is unlikely that random guessing has a big influence. However, for the results in Table 1 this is not the case. For instance, it appears that all MMBench results in Table 1 are below the 26.4% from [27] - which means all LLM do worse than always answering A (or B). For other benchmarks there may similarly be data biases in which answer is more frequent.\n\nRelatedly, it may be the case that such data biases about which options are more frequently chosen are more pronounced for multi-modal questions, i.e., when asked about colour the answer is more often grey across all datasets and settings - which means LLM do not learn this data bias as their training data does not include such questions, but LVLM may learn this because the same bias is present in their training data. While this is unlikely to fully explain the phenomenon observed in Table 2, it may explain part of it and not be directly related to data leakage. \n\n2) Not all benchmarks\n\nThe issues identified with existing benchmarks do not hold evenly across the benchmarks tested. In particular, MMBench and slightly lesser, MathVista seem to do pretty well with respect to these issues. This also appears to be reflected in the construction of MMStar, where after Manual Review the proportion of questions from these two existing benchmarks jumps considerable - in the end making up almost half of the MMStar benchmark. Which raises questions whether there is any benefit of using MMStar versus simply using MathVista and MMBench. \n\n3) Manual review\n\nThe manual review description is insufficiently clear. From the three criteria applied, only the first one is somewhat clear - for the other two it is unclear how the 'experts' judged this. I would expect description (can be in appendix) of agreement rates between these experts for these criteria, as well as further description of what these criteria entail. \n\n4) New metrics\n\nThe newly proposed metrics MG and ML are somewhat unclear in what they measure. If the 1500 questions in MMStar all require visual input as determined by the manual review, then what additional information does the MG metric give? Given the discussion above about chance, it appears the MG metric is more of a correction for random guessing. The ML metric similarly doesn't account for random guessing, or the aforementioned potential for data biases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestions. We address your concerns point by point:\n\n**Q1: Given the 26.4% baseline for MMbench from [27], it is also surprising that the random choice value reported in this paper, for MMbench, is 0.0 in both Table 1 and Table 2.**\n\n**A1:** The 26.4% in MMBench represents the frequent choice probability. We use MMBench's circular evaluation method, where: \n\n1. Options are shuffled and evaluated multiple times\n2. All evaluations must be correct for the sample to be considered correct\n\nProbability of random guessing:\n\n- Two-choice: 1/4\n- Three-choice: 1/27\n- Four-choice: 1/256\n\nMMBench has 77 two-choice, 221 three-choice, and 1462 four-choice questions, resulting in a theoretical accuracy of approximately 1.88%. Using VLMEvalKit, the files include options C and D for all samples, making the generalized random choice accuracy about 0.39%. Actual random selection shows an accuracy of 0.01%, rounded to 0.0% in Tables 1 and 2. This reflects the presence of samples that do not rely on visual content or have been leaked in the LLM training data, aligning with our findings.\n\n**Q2: An additional baseline based on majority class may also be necessary here.**\n\n**A2:** Good point! We conducted a frequent choice evaluation on existing benchmarks and MMStar: \n\n- MMMU|26.8\n- MMB|0.0\n- ScienceQA|36.0\n- AI2D|26.9\n- SEED|26.9\n- MathVista|26.3\n- MMStar|29.8\n\nThe result of 0.0% for MMBench is due to circular evaluation, too. We chose not to use the circular evaluation mechanism for MMStar, consistent with most other benchmarks. We'll include these results in the next version.\n\n**Q3: For the results in Figure 2, which require 6 out of 8 models...which means all LLM do worse than always answering A (or B).**\n\n**A3:** This misunderstanding is resolved by explaining the circular evaluation for MMBench. The 0.0% results for random choice and frequent choice are lower than the 13.8% average accuracy for 22 LLMs. This indicates some samples don't rely on visual content or have been leaked in LLM training data, aligning with our observations.\n\n**Q4: The issues identified with existing benchmarks...whether there is any benefit of using MMStar versus simply using MathVista and MMBench.**\n\n**A4:** On one hand, MMBench uses circular evaluation, so the 0.0% random choice accuracy is much lower than the 13.8% average for the 22 LLMs. On the other hand, MathVista's 17.9% random choice accuracy is also lower than the 22.5% average for the 22 LLMs. While MathVista focuses on mathematics, our MMStar covers six core competencies and 18 detailed axes, offering a more comprehensive evaluation of LVLMs' multimodal capabilities. Additionally, we manually verified that each MMStar sample is visually dependent, a guarantee that neither MMBench nor MathVista can provide.\n\n**Q5: The manual review description is insufficiently clear...as well as further description of what these criteria entail.**\n\n**A5:** Due to space constraints, we have detailed the manual review process and the agreement rates in the General Author Rebuttal.\n\n**Q6: The newly proposed metrics MG and ML...account for random guessing, or the aforementioned potential for data biases.**\n\n**A6:** The score of an LVLM on a multimodal benchmark can be split into three parts: leakage from LLMs, leakage during multimodal training, and genuine understanding from multimodal training. Multi-modal Leakage (ML) measures the second part, and Multi-modal Gain (MG) measures the third. These metrics complement each other and should not be considered separately.\n\nThese metrics are not exclusive to MMStar; they can analyze existing benchmarks that may not ensure visual dependency. Although MMStar ensures visual dependency, some samples might still leak into future LVLMs' training corpora. In such cases, MG and ML can assess multi-modal training leakage and performance gains.\n\n**Q7: The paper raises a number of interesting questions...input from the authors on this point.**\n\n**A7:** We have provided clarifications in our previous responses to Q1, Q2, and Q3, explaining that we adopt MMBench's native circular evaluation mechanism. Therefore, the 0.0% accuracy for both random choice and frequent choice is significantly lower than the average accuracy of 13.8% for the 22 LLMs, which indicates the presence of data leakage in the LLMs.\n\n**Q8: Additionally, I would be interested...Also taking into account the majority class versus random guess.**\n\n**A8:** We add circular evaluation results for MMBench, AI2D, and MMStar using two representative LVLMs. One can observe from the table that:\n\n- Random choice and frequent choice results close to 0% under circular evaluation\n- Data leakage in LLMs observed (e.g., InternVL-Chat-v1.2's LLM achieves 47.3% on AI2D, surpassing LLaVA-1.5's performance with images)\n- Multimodal training data leakage evident (e.g., LLaVA-1.5 improves MMBench and AI2D performance by 8.8% and 14.3% without image input)\n- MMStar mitigates sample leakage in LLM and LVLM training corpora while maintaining the challenge level\n\n| Model              | Strategy  |  MMB | AI2D | MMStar |\n|--------------------|-----------|:----:|:----:|:------:|\n| Random Choice      | -         |  0.0 |  0.2 |   0.1  |\n| Frequent Choice    | -         |  0.0 |  0.0 |   0.0  |\n| LLaVA-1.5          | LLM       | 10.3 | 18.3 |   1.7  |\n|                    | LVLM-text | 19.5 | 32.6 |   6.7  |\n|                    | LVLM      | 65.0 | 41.7 |  19.0  |\n| InternVL-Chat-v1.2 | LLM       | 20.1 | 47.3 |   4.1  |\n|                    | LVLM-text | 23.9 | 53.3 |  11.1  |\n|                    | LVLM      | 82.4 | 71.7 |  45.6  |\n\n**Q9: I believe question 9 in the paper checklist has been answered incorrectly.**\n\n**A9:** We have carefully reviewed the Code of Ethics of NeurIPS and cross-checked each requirement to ensure compliance. We will update this answer to [Yes] in the next version of the manuscript.\n\nPlease do not hesitate to contact us if you have any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all reviewers for your time and efforts in the review. All detailed questions of each reviewer are answered accordingly in each column below. We hope these responses can address the reviewers' concerns adequately. Additionally, we provide the implementation details of the manual review process used in constructing MMStar to supplement the missing details in the manuscript.

After roughly filtering the original data pool with 8 advanced LLMs, resulting in 11,607 candidate samples, we initiate a rigorous manual review phase. First, we establish 6 core evaluation dimensions and 18 detailed axes by integrating the evaluation dimensions from existing benchmarks. Next, we use 16 LVLMs to infer and count the number of hits for each sample. Furthermore, we design a UI interface listing the current sample's image, options, answer, sample source, hit count, and the 18 detailed axes. The samples are arranged in ascending order based on the number of hits.

The formal manual selection and benchmark construction process is as follows:

1. Preliminary Classification: Three experts are each responsible for two core capability dimensions (i.e., 6 detailed axes). They need to review all candidate samples and select and correctly classify the samples belonging to their respective dimensions. The samples selected must retain their visual dependency.
2. Statistical Analysis: After the preliminary classification, we consider the numerical balance between dimensions and the difficulty level of the samples. Samples under the "coarse perception" dimension approach 4,000, while those under "logical reasoning" are fewer than 700. In terms of difficulty distribution, there are 4,555 easy (i.e., number of hits between 12 and 16) samples but only 2,758 tough (i.e., number of hits between 0 and 3) ones. Given these premises, a lot of  repetitive simple samples, such as those merely asking for the color of an object in the image, are not what we desire.
3. Initial Benchmark: After considering both the numerical balance and difficulty level of the samples, we set the total sample number of the benchmark at 1,500, with each core capability dimension containing 250 samples. Then, we assign each expert two core capability dimensions, instructing them to prioritize sample difficulty when selecting 250 samples per dimension.
4. Cross-Validation: To minimize personal bias, we arrange for each expert to review the dimensions handled by the other two experts after the initial benchmark is constructed. Samples with issues are replaced by correct samples of the same difficulty level from the candidate pool.

Moreover, we also provide the number of samples with consensus before and after the cross-validation step in the manual review process for MMStar in the table below. Only samples that all three experts unanimously agree upon are retained; otherwise, they are replaced with samples of the same difficulty level from the candidate pool.

|          | Before | After |
|----------|--------|-------|
| Expert 1 | 472    | 500   |
| Expert 2 | 468    | 500   |
| Expert 3 | 483    | 500   |

By following this thorough process, we ensure a balanced and challenging benchmark set.

Meta-review: This submission focuses on large vision-language models.  The authors have identified several problems posed by conventional datasets, and have developed a new multimodal benchmark and proposed two metrics providing a new framework for better evaluating these multimodal models.  All the reviewers emphasized the originality of the proposed work.  
They also asked for several clarifications.  After the rebuttal and discussion session, all were satisfied with the answers.  All reviewers were positive about the submission, considering that the novelty and the interest of the proposal stand out clearly.   
The authors are strongly encouraged to take all comments into account in their final version.
Decision: Accept (poster)