Are We on the Right Way for Evaluating Large Vision-Language Models?

Authors: lin  chen
jinsong  li
xiaoyi  dong
pan  zhang
yuhang  zang
zehui  chen
haodong  duan
jiaqi  wang
yu  qiao
dahua  lin
feng  zhao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 13:32:46

Abstract: Large vision-language models (LVLMs) have recently achieved rapid progress, sparking numerous studies to evaluate their multi-modal capabilities. However, we dig into current evaluation works and identify two primary issues: 1) Visual content is unnecessary for many samples. The answers can be directly inferred from the questions and options, or the world knowledge embedded in LLMs. This phenomenon is prevalent across current benchmarks. For instance, GeminiPro achieves 42.7% on the MMMU benchmark without any visual input, and outperforms the random choice baseline across six benchmarks near 24% on average. 2) Unintentional data leakage exists in LLM and LVLM training. LLM and LVLM could still answer some visual-necessary questions without visual content, indicating the memorizing of these samples within large-scale training data. For example, Sphinx-X-MoE gets 43.6% on MMMU without accessing images, surpassing its LLM backbone with 17.9%. Both problems lead to misjudgments of actual multi-modal gains and potentially misguide the study of LVLM. To this end, we present MMStar, an elite vision-indispensable multi-modal benchmark comprising 1,500 samples meticulously selected by humans. MMStar benchmarks 6 core capabilities and 18 detailed axes, aiming to evaluate LVLMs' multi-modal capacities with carefully balanced and purified samples. These samples are first roughly selected from current benchmarks with an automated pipeline, human review is then involved to ensure each curated sample exhibits visual dependency, minimal data leakage, and requires advanced multi-modal capabilities. Moreover, two metrics are developed to measure data leakage and actual performance gain in multi-modal training. We evaluate 16 leading LVLMs on MMStar to assess their multi-modal capabilities, and on 7 benchmarks with the proposed metrics to investigate their data leakage and actual multi-modal gain.

Main paper: 
 introduction: 
 Encouraged by the rapid development of large language models (LLMs) [60, 4, 9, 10, 14, 1, 53] , integrating visual modality into LLMs to enhance models' interactivity capabilities has witnessed ever-changing advances in recent days [72, 33, 31, 12, 68, 2, 61, 39, 5, 13] . These large visionlanguage models (LVLMs) showcase powerful visual perception and understanding capabilities, enabling them to accept image inputs from users and engage in dialogues, thereby offering a more enriched interactive experience. These achievements have further inspired the research community to develop a variety of multi-modal benchmarks [27, 16, 34, 47, 63, 64, 37, 26, 38] , constructed to explore the powerful capabilities emerging from LVLMs and provide a comprehensive and objective platform for quantitatively comparing the continually evolving models. Despite the race among existing evaluation works to construct as many axes as possible to assess the capabilities of LVLMs, we have identified two primary issues upon delving into existing evaluation samples and processes.First, visual content is unnecessary for many samples. A qualified multi-modal evaluation sample should compel LVLMs to understand and reason with the visual content for correct answers. Otherwise, the evaluation sample would degrade into assessing the textual capabilities of LLM bases. Unfortunately, we have identified numerous samples across multiple popular benchmarks [34, 27, 64, 38, 26] where answers can be correctly deduced without relying on visual content. As shown in Figure 1 (a) and (b), some samples have answers directly included within the questions (e.g., What is the shape of the round dirt circle?), while others can be effortlessly answered by leveraging the rich world knowledge embedded within the LLM bases (e.g., What is the capital of Nebraska?). As shown in Table 1 , with a comprehensive quantitative analysis of 22 LLMs on 6 benchmarks, we observe this phenomenon is prevalent and serious. For example, more than 50% questions of ScienceQA and near 30% questions of MMMU can be solved by most LLMs directly. For the powerful LLM GeminiPro, it achieves 42.7% on the MMMU benchmark without any visual input, and outperforms the random choice baseline across six benchmarks by near 24% on average.Taking aside the inappropriate samples in evaluation, we also observed strange results that LLM and LVLM could still answer some visual-necessary questions without visual content (Figure 1 (c) and (d) ). A plausible explanation for this could be the inadvertent memorization of these samples during the large-scale training process, suggesting the presence of unintentional data leakage in the training of LLM and LVLM. Through a detailed study of various LVLMs on 6 benchmarks, as shown in Table 2 , we find the unexpected leaking problem during the LVLM training is particularly serious. For example, we find Yi-VL-34B gets 15.0% higher performance than its LLM backbone on ScienceQA, Sphinx-X-MoE gets 43.6% on MMMU without accessing images, surpassing its LLM backbone with 17.9%, even surpassing many leading LVLMs with accessing images.The existence of inappropriate questions and data leaking would lead to misjudgments of actual multi-modal performance gains and potentially misguide the study of LVLM. In pursuit of a more accurate and comprehensive evaluation, we introduce the MMStar Benchmark. MMStar is a premier, vision-critical multi-modal benchmark that includes 1,500 challenging samples, each rigorously validated by humans. It is structured to test 6 fundamental capabilities and 18 specific di- 
related work: 
 Large Vision-Language Models. As large language models (LLMs) [9, 53, 53, 60, 52, 42, 65, 44, 10] rapidly advance, a growing fraction of the research community is focusing on integrating visual content into LLMs to build a powerful intelligent assistant with more interactive ways. Central to these large vision-language models (LVLMs) are the seminal works in modality alignment within the vision-language learning area [46, 21] . The foundation work CLIP [46] exemplifies the alignment of vision and language modalities through contrastive learning on extensive image-text pairs. Built upon the CLIP image encoder which is somewhat aligned with the language modality, current LVLMs typically utilize vast image-text pairs to connect the vision encoder and LLM, enabling LLM to receive and understand visual content [72, 33, 31, 12, 45, 2, 48, 6, 39, 19, 5, 20, 71, 73, 24, 59, 70] . For example, MiniGPT4 [72] and LLaVA [33] directly connect the vision encoder and LLM with QFormer [28] and MLP [50] , showing proficiency in multi-modal dialogues. Subsequent works have further enhanced LVLMs by improving the multi-modal instruction data [31, 61, 5, 54, 67, 25] and designing novel modules [2, 30, 55, 36, 17, 13] for more sufficient modality alignment.Evaluations of LVLMs. To probe the true capabilities of the emerging LVLMs, the research community has developed many multi-modal benchmarks encompassing a wide range of evaluation axes [34, 16, 47, 64, 49, 66, 27, 33, 29, 35, 63, 56, 58, 57] . Early single-task benchmarks, such as VQA [18] , MS-COCO [49] , and OK-VQA [47] , fail to holistically assess LVLMs' general multi-modal perception and reasoning capabilities. To address this issue, comprehensive multi-modal benchmarks have been constructed [33, 27, 64, 16, 34, 8, 56] . For example, SEED [27] and MMBench [34] cover 12 and 20 evaluation dimensions respectively, while MMMU [64] spans 30 collegelevel subjects, providing some competitive arenas for a comprehensive comparison of cutting-edge LVLMs. However, existing evaluations of LVLMs overlook some critical issues. On the one hand, they do not guarantee that all evaluation samples cannot be correctly answered without the visual Table 1 : Evaluation of various LLMs on six popular multi-modal benchmarks under 2-shot. We employ a 2-shot inference strategy for evaluating all LLMs to reduce instances of refusal to answer and align the answer formats. We report the results of 2 closed-source LLMs and 20 open-source LLMs with varying sizes and architectures. The evaluated benchmarks include MMMU (MMMU-Val [64] ), MMB (MMBench-EN-Dev [34] ), ScienceQA (ScienceQA-Test [38] ), AI2D (AI2D-Test [26] ), SEED (SEED-Image [27] ), and MathVista (MathVista-Mini [37] ). The best results are highlighted in bold and underlined.Strategy MMMU MMB ScienceQA AI2D SEED MathVista Avg. content. On the other hand, current evaluations consistently adhere to the process of inferring on given benchmarks and calculating scores for LVLMs, overlooking the possibility of data leakage during multi-modal training. This oversight can lead to unfair comparisons and misjudgments of the real gains in multi-modal capabilities brought by multi-modal training.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See Section A.9.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
two overlooked issues for evaluating lvlms: 
 In this section, we delve into two commonly overlooked issues in current LVLM evaluation works. Moreover, we present detailed experimental results to further substantiate our observations.First issue: visual content is unnecessary for many evaluation samples. The key distinction between evaluating LLMs and LVLMs lies in the necessity for LVLM evaluations to strictly ensure that the correct answers can only be derived based on a thorough understanding of visual content. Without this, evaluating LVLMs' multi-modal capabilities degrades to merely assessing their LLM backbones' uni-modal abilities. However, upon examining samples from some popular LVLM benchmarks, we find many samples lack vital visual dependency and can yield correct answers even without the image inputs! Through analysis of these failure samples, we categorize them into two groups: (1) Answers can be directly obtained from the world knowledge embedded in LLMs, owing to the LLMs' extensive pretraining on the large corpus of data. For example, as illustrated in Figure 1 (a), the question "What is the capital of Nebraska?" already provides the key information "Nebraska", eliminating the need for extracting relevant location information from visual content. A more appropriate question is "What is the capital of the highlighted area in the image?" to emphasize the importance of visual understanding. (2) Answers are directly included in the textual questions.As shown in Figure 1 (b), LLMs can derive the correct answer "circle" through simple reasoning based on the question "What is the shape of the round dirt circle?". To quantitatively substantiate our findings, we further experiment to gauge the proportion of these two types of samples in existing benchmarks. Specifically, we evaluate several benchmarks with two closed-source LLMs (GPT4-Turbo [42] , and GeminiPro [51] ) and six opensource heavy LLMs (InternLM2-20B [52] , Yi-34B [62] , Mixtral-8x7B [23] , Deepseek-67B [3] , LLaMA2-70B [53] , and Qwen1.5-72B [1] ), recording the hit count for each question. Here, the 'hit' refers to the ability of an LLM to correctly answer the question without relying on visual input. We then calculate the percentage of samples with a hit count of six or more (80%) against the total number of samples to determine the abnormal hit rate for each benchmark. As depicted in Figure 2 , every benchmark shows a certain degree of samples that visual contents are unnecessary, with ScienceQA [38] and AI2D [26] exhibiting amazing abnormal hit rates of 57.2% and 46.2%, respectively. Based on our observations, most multi-modal benchmarks have yet to fully assess the multi-modal capabilities of LVLMs.Second issue: unintentional data leaking exists in LLM and LVLM training. Although the community has the trend towards developing new multi-modal benchmarks to assess LVLMs' capabilities from various dimensions, there is scant consideration for fairness and reliability during evaluation. Training LLMs and LVLMs requires vast and diverse data, inevitably leading to the leakage of evaluation samples. Such incidents are usually unintended, as it's impractical to predict which data will be used in future evaluation benchmarks during the preparation for training corpus.Figure 1 (c) showcases an evaluation sample leaked by LLMs. Though the question requires an understanding of image content, 16 out of 22 tested LLMs astonishingly provide the correct response by "recalling" their training data. To quantitatively support our observations, we evaluate 22 leading LLMs across 6 popular benchmarks and report the 2-shot results in Table 1 . Specifically, we find the 2-shot evaluation strategy is more stable than the 0-shot (see results in Section A.8) to reduce refusal for answering and align answer formats. Under the impact of vision-independent samples and data leakage from LLMs, GeminiPro [51] and Qwen1.5-72B [1] achieve a remarkable average accuracy of 42.4% and 43.6% under the 2-shot setting, outperforming random choice by 21.4% and 22.6%, respectively. Furthermore, Qwen1.5-72B achieves a score of 42.4% on MMMU [64] , even surpassing the performance of the majority of LVLMs with accessing images. This result serves as a reminder: if we only consider the final accuracy on benchmarks when evaluating LVLMs, potential data leakage from LLMs could lead to unfair comparisons.In Figure 1 (d) and Section A.5, we showcase some examples where original LLMs fail, but LVLMs without accessing images succeed. Despite these questions requiring image content for accurate answers, the LVLMs without accessing images are capable of correctly answering these questions which stump original LLMs. To further support our hypotheses of data leakage during LVLMs' multi-modal training, we conduct an intriguing experiment. We remove the images for LVLMs and only utilize questions and options for evaluation, with results reported in Table 2 . We compare the gains of LVLMs set to receive only text inputs (LVLM-text) against their corresponding LLM bases (LLM) to quantitatively assess the degree of data leakage in LVLMs' multi-modal training. As shown in Table 2 , most LVLMs exhibit varying degrees of data leakage during multi-modal training. For example, the LLMs of Sphinx-X-8x7B [17] and Monkey-Chat [30] , show a respective average performance gain of 14.1% and 14.2% compared to their original LLMs.Drawing from our observations, we posit that the issue of data leakage in multi-modal datasets is a significant concern that warrants attention. Addressing this issue is essential to ensuring that model performance is measured by their genuine ability to integrate and interpret multimodal data, rather than by their tendency to memorize specific samples within the dataset. Establishing a robust and reliable benchmark to minimize data leakage would thus serve as a foundational step in advancing research within the field of multimodal language models, paving the way for more meaningful and accurate evaluations of their performance and potential. Criteria for data curation. The evaluation samples for constructing the MMStar benchmark should meet three fundamental criteria: 1) Visual dependency. The collected samples can be correctly answered only based on understanding the visual content; 2) Minimal data leakage. The collected samples should minimize the risk of unintentional inclusion in LLMs' training corpus, or be effectively transformed from uni-modal to multi-modal formats to prevent LLMs from "recalling" the correct answers; 3) Requiring advanced multi-modal capabilities for resolution. In addition to ensuring fairness and reliability by adhering to the above criteria, we also aim for samples to cover various difficulty levels. We expect to comprehensively capture LVLMs' multi-modal capabilities with succinct high-quality samples.Data filter. We first choose two benchmarks [34, 27] focused on natural images and four centered on scientific and technical knowledge [64, 38, 26, 37] for our sample collection. We then develop an automated pipeline to preliminarily filter out samples that do not meet the first two criteria. Specifically, we employ two closed-source LLMs [51, 42] and six open-source LLMs [1, 52, 62, 3, 23, 53] sizing 20B or larger to serve as inspectors. These open-source LLMs are applied with a 2-shot in-context inference strategy to minimize response refusals and ensure consistency in answer formatting. Following this, we evaluate the sample pool with these LLM inspectors, documenting the hit frequency for each evaluation sample. Finally, we only retain those samples with hit counts of two or fewer hits, indicating that around 75% of LLM inspectors fail to provide the correct answer. As illustrated in Figure 3 (a), following this initial coarse filtering, our sample pool was reduced from 22,401 to 11,607.Manual review. After the coarse filtering with LLM inspectors, we further employ three experts to conduct the manual review process to ensure: 1) each sample's answer should be based on the understanding of visual content; 2) selected samples should cover a comprehensive range of capability assessment dimensions; 3) most samples should require LVLMs to possess advanced multi-modal abilities for resolution. To expedite the manual selection of samples with varying difficulty levels for LVLMs, we tally the hit counts of all 16 LVLMs on the coarsely filtered samples and split them into four difficulty categories: easy (12-16, 148 examples), moderate (8) (9) (10) (11) , 189 examples), hard (4-7, 631 examples), and tough (0-3, 532 examples). Finally, after considering both the diversity of capability dimensions and difficulty levels, we manually curated 1,500 high-quality samples from the coarsely filtered set. Figure 3 (a) showcases the detailed composition of data sources for our final selection of samples. In Section A.3, we provide details on how the manual review step aggressively reduces the MMStar benchmark from 11,607 samples to 1,500 samples.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: This work does not involve any theoretical assumptions or proofs. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
core capabilities: 
 We select and consolidate the dimensions used for assessing LVLMs' multi-modal capabilities in existing benchmarks and identify six core capabilities along with eighteen detailed axes. In Figure 3 (b), we provide statistics for each core capability and their detailed axes on the MMStar benchmark. More detailed definitions of each capability are provided in Section A.2. Given our observation of the potential for inadvertent leakage of some evaluation samples during the multi-modal training process, the vanilla evaluation approach struggles to reveal LVLMs' actual performance gains derived from multi-modal training and fails to enable fair comparison with other competitors. Therefore, we propose two novel metrics to separately assess the degree of data leakage and actual performance gain from the multi-modal training process.To calculate the multi-modal gain (MG) metric for a given LVLM on a particular benchmark, we need to compute the scores of the same LVLM with and without visual inputs, separately denoted as S w v and S ov . Then the MG metric can be derived from the following formulation:EQUATIONTo calculate the multi-modal leakage (ML) metric, we need to compute the extra score of the given LVLM's LLM base (without any multi-modal training), denoted as S t . Then the ML metric is formulated as follows:EQUATIONQuestion: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: See them in Section 3, Section 5, and Section A.1. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiments: 
 In this section, we conduct a systematic analysis of the proposed MMStar benchmark along with the MG/ML metrics. These analyses encompass various LLMs and LVLMs, and also involve numerous existing benchmarks when examining MG/ML metrics. We choose VLMEvalKit [15] as our codebase. Please see details about experimental setups in Section A.1.In this section, we present a comprehensive comparison of various LLMs and LVLMs performed on our MMStar benchmark and summarize our key observations in the following parts.Gemini-Pro [13] achieves an impressive score of 55.4%. LLaVA-Next [32] even surpasses GPT4V and GeminiPro-Vision [51] in the mathematics (MA) core capability. Notably, no LVLMs managed to reach a passing average score (i.e. 60%) in the core capabilities of fine-grained perception (FP), logical reasoning (LR), science & Technology (ST), and mathematics (MA), highlighting these dimensions as particularly challenging for existing LVLMs. Moreover, TinyLLaVA [69] , despite its modest 3B scale, outperformed some competitors of 7B and even 13B surprisingly, underscoring the potential of smaller-scale LVLMs.EQUATIONAnalysis from the model perspective. In Table 4 , we illustrate the MG/ML (Multi-modal Gain/Multi-modal Leakage) metrics for each LVLM across each benchmark and provide an average MG/ML metric across all benchmarks in the final column. For closed-source LVLMs, GPT4V demonstrates notable performance gains attributed to its multi-modal training, while GeminiPro-Vision shows lesser data leakage during multi-modal training. This suggests that GPT4V may have utilized a broader range of multi-modal training data compared to GeminiPro-Vision. Among the open-source LVLMs, InternLM-XComposer2 achieves the highest average multi-modal gain of 28.1 across all benchmarks, whereas LLaVA-1.5-7B records the lowest at 14.8. This outcome is reasonable given that LLaVA-1.5-7B employed the least amount of multi-modal training data among these open-source LVLMs. Despite LLaVA-1.5-7B having the lowest average multi-modal gain, it exhibits minimal multi-modal leakage. Additionally, models like Monkey-Chat, Sphinx-X-MoE, and Deepseek-VL display higher degrees of multi-modal leakage, highlighting the need for the community to consider this factor for fair comparisons.Analysis from the benchmark perspective. In the final row of Table 4 , we list the average multimodal gain and multi-modal leakage for existing LVLMs across all benchmarks for analysis. MM-Bench registers the highest average multi-modal gain at 50.1, indicating a significant overlap between the domains covered by existing LVLMs' training data and MMBench. Conversely, MMMU exhibits the lowest average multi-modal gain at 5.8, suggesting a lesser degree of overlap between the domains of existing LVLMs' training corpora and those included in MMMU. Additionally, MM-Star, as expected, has the lowest degree of multi-modal leakage at 1.9. This provides a comprehen- sive and fair arena for comparing existing LVLMs. Moreover, we believe evaluating existing LVLMs to derive average ML metrics can also be helpful to the following works in examining newly developed multi-modal benchmarks. 
conclusion: 
 In this work, we dig into current evaluation works for large vision-language models (LVLMs) and identify two primary issues: 1) visual content is unnecessary for many samples, and 2) unintentional data leakage exists in LLM and LVLM training. To address these issues, we develop an elite vision-dependent multi-modal benchmark named MMStar and propose two metrics to measure the data leakage and actual performance gain in LVLMs' multi-modal training. MMStar undergoes the manual review of each sample, covering 6 core capabilities and 18 detailed axes for an in-depth evaluation of LVLMs' multimodal capabilities. In our evaluation of 16 diverse LVLMs on MM-Star, even the best model scores under 60 on average. We also analyze the MG and ML metrics across 6 multimodal benchmarks and MMStar, providing valuable insights for the community on gathering multimodal training data and crafting new benchmarks. In the future, we plan to expand MMStar into a larger, online test set and explore dynamic evaluation methods to maintain sample visual dependency and reduce accidental data leakage into LLM's and LVLM's training corpora.encompasses a broad, holistic view of the visual content, enabling the identification of 1) image style & quality; 2) image scene & topic; and 3) image emotion.Fine-grained Perception (FP). This core dimension represents a sophisticated level of image understanding that focuses on the detailed and nuanced aspects of visual content. It involves a deep dive into the specifics of images: 1) attribute & celebrity recognition; 2) object location; and 3) object counting. This core dimension unveils the subtle intricacies that coarse perception might overlook.Instance Reasoning (IR). It encapsulates a set of advanced cognitive capabilities focused on understanding and interpreting individual and collective object attributes and interrelations within an image. This process goes beyond mere recognition, delving into the analytical assessment of 1) single-instance attribute reasoning; 2) cross-instance attribute comparison; and 3) cross-instance relation reasoning. It is a critical component for systems requiring a deep semantic understanding of visual content, enabling nuanced interaction with and response to complex visual content.Logical Reasoning (LR). This core dimension encompasses a sophisticated framework of cognitive processes designed to interpret, deduce, and infer conclusions from visual content through a structured approach to logic and reasoning. This multi-faceted capability marries the intuitive understanding of visual content with the structured rigor of logical deduction, enabling: 1) diagram reasoning; 2) code & sequence reasoning; and 3) common reasoning.Science & Technology (ST). It consists of a comprehensive framework for the application and integration of knowledge across a broad spectrum of science and technology. This domain combines the theoretical underpinnings and practical applications of various fields: 1) natural science; 2) engineering; and 3) geography & earth science. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: Compute resource constraints. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean.generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [No] Justification: The data used in this work are from previous public benchmarks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [No] Justification: The data used in this work are from previous public benchmarks. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 
 
 Appendix: Math is a foundational pillar of logical and analytical reasoning and encompasses a broad spectrum of capabilities essential for understanding, applying, and interpreting quantitative and spatial information. We primarily consider three aspects for evaluating LVLMs' logical thinking prowess: 1) numeric commonsense & calculation; 2) geometry; and 3) statistical analysis.After roughly filtering the original data pool with 8 advanced LLMs, resulting in 11,607 candidate samples, we initiate a rigorous manual review phase. First, we establish 6 core evaluation dimensions and 18 detailed axes by integrating the evaluation dimensions from existing benchmarks. Next, we use 16 LVLMs to infer and count the number of hits for each sample. Furthermore, we design a UI interface listing the current sample's image, options, answer, sample source, hit count, and the 18 detailed axes. The samples are arranged in ascending order based on the number of hits. The formal manual selection and benchmark construction process is as follows:Preliminary Classification: Three experts are each responsible for two core capability dimensions (i.e., 6 detailed axes). They need to review all candidate samples and select and correctly classify the samples belonging to their respective dimensions. The samples selected must retain their visual dependency. Statistical Analysis: After the preliminary classification, we consider the numerical balance between dimensions and the difficulty level of the samples. Samples under the "coarse perception" dimension approach 4,000, while those under "logical reasoning" are fewer than 700.In terms of difficulty distribution, there are 4,555 easy (i.e., number of hits between 12 and 16) samples but only 2,758 tough (i.e., number of hits between 0 and 3) ones. Given these premises, a lot of repetitive simple samples, such as those merely asking for the color of an object in the image, are not what we desire.Initial Benchmark: After considering both the numerical balance and difficulty level of the samples, we set the total sample number of the benchmark at 1,500, with each core capability dimension containing 250 samples. Then, we assign each expert two core capability dimensions, instructing them to prioritize sample difficulty when selecting 250 samples per dimension.To minimize personal bias, we arrange for each expert to review the dimensions handled by the other two experts after the initial benchmark is constructed. Samples with issues are replaced by correct samples of the same difficulty level from the candidate pool. Moreover, we also provide the number of samples with consensus before and after the cross-validation step in the manual review process for MMStar in the table below. Only samples that all three experts unanimously agree upon are retained; otherwise, they are replaced with samples of the same difficulty level from the candidate pool. 7 : Evaluation of various LVLMs on six popular multi-modal benchmarks. For the "strategy" column, "LLM" refers to evaluating using corresponding LLM base of the LVLM, while "LVLM-text" denotes evaluating LVLMs without accessing images. We employ the 0-shot inference strategy for LLMs to align the evaluation protocols of LVLMs. The highest results of the LVLM-text setting across the models are highlighted in bold and underlined.Param. Strategy MMMU MMB ScienceQA AI2D SEED MathVista Avg. We provide clear claims in the abstract and introduction sections.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This submission focuses on large vision-language models.  The authors have identified several problems posed by conventional datasets, and have developed a new multimodal benchmark and proposed two metrics providing a new framework for better evaluating these multimodal models.  All the reviewers emphasized the originality of the proposed work.  
They also asked for several clarifications.  After the rebuttal and discussion session, all were satisfied with the answers.  All reviewers were positive about the submission, considering that the novelty and the interest of the proposal stand out clearly.   
The authors are strongly encouraged to take all comments into account in their final version.

Reviews: review: paper_summary: paper_summary: The authors have identified two primary concerns with the benchmarks commonly used for large vision-language models (LVLMs). Firstly, many samples do not require visual content to answer the questions. Secondly, they noted unintentional data leakage during LVLM training. They assessed eight large language models (LLMs) across six widely-used multi-modal LLM benchmarks, demonstrating that LLMs can correctly answer a significant portion of questions without visual input. To more reliably evaluate LVLM performance, they developed a new benchmark by meticulously filtering data from six existing benchmarks with three requirements: 1) visual dependency, 2) minimal data leakage and 3) multi-modal capability for resolutions. Additionally, they designed two metrics: Multi-modal Gain, to quantify the improvement from multi-modal training, and Multi-modal Leakage, to assess the extent of potential data leakage. Using this new benchmark and the two metrics, they provide a comprehensive comparison of state-of-the-art LVLMs.
strength_weakness: strength_weakness: - The proposed metrics, Multi-modal Gain and Multi-modal Leakage, are dependent on the base LLM utilized in the large vision-language models. This dependency complicates the use of these metrics for directly comparing the multi-modal gain across different LVLMs.

- The manual review step aggressively reduces the MMStar benchmark from 11,607 samples to 1,500 samples. The explanation provided in Section 3.1 for this reduction is somewhat vague and lacks clear, objective criteria for filtering. I am curious about the rationale behind such an aggressive reduction by nearly tenfold. Is this reduction due to a scarcity of data meeting the three specified criteria mentioned between line 187 to 189, or are there other reasons for this decision?
questions: questions: - Please clarify the goal of using proposed metrics

- Please clarify the decision to aggressively reduce the size of final benchmark
limitations: limitations: The authors properly discuss the limitation in the paper
paper_summary: paper_summary: In this paper, the authors examine current benchmarks for large vision-language models (LVLMs) and identify two main problems: 1) many samples do not require visual content, and 2) there is unintentional data leakage in LLM and LVLM training. To address these issues, they developed a multimodal benchmark called MMStar, consisting of 1,500 samples, and proposed two metrics to measure data leakage and performance gain in LVLMs’ multimodal training. They conducted empirical evaluations on 16 LVLMs to report their performance on MMStar.
strength_weakness: strength_weakness: 1. The authors only consider multiple-choice questions for the MMStar benchmark. Including a wider variety of well-curated questions without choices would be great.
2. Similar to Figure 2, the authors should provide the LLM Hit Rate for the MMStar benchmark.
3. What is the percentage distribution of the 1,500 samples across the four difficulty categories?
questions: questions: Please respond to the points of weakness I mentioned above.
limitations: limitations: Yes
paper_summary: paper_summary: The current benchmarks used to evaluate Vision Language Models (VLMs) contain several flaws. In particular, a lot of questions can be answered without looking at the image at all. These benchmarks still being hard, the best proprietary models without looking at the images can obtain better scores than strong VLM baselines (looking at the images). As a result, the authors create MMStar, a difficult benchmark aiming at evaluating the capability of the vision-language tasks. They manually review their benchmark, and do several ablations to confirm its importance.
strength_weakness: strength_weakness: - In the released dataset, the choices are directly integrated into the prompt. It would be good to also add a column with only the original question, and another column containing the list with the possible options, so that researchers could evaluate their models with the prompts they used during their fine-tuning.
- As the authors mentioned, it would be useful to also create a test set for this benchmark.
- It would have probably made more sense to publish this in the Datasets and Benchmarks track.
questions: questions: I personally noticed many hallucinations in the SEED benchmark, with some Q/A pairs that are simply false.
Since the Q/A pairs from SEED represent 28.3% of your dataset, I am worried that you would have such incorrect pairs in your benchmark.
Can you confirm that this was removed during the manual filtering?
limitations: limitations: Yes
paper_summary: paper_summary: This paper investigates the evaluation of large vision-language models (LVLMs) and the currently used benchmarks. Within the paper two primary issues are identified: the lack of need for visual information, and data leakage. Based on these issues a new compiled benchmark is proposed MMStar that includes a set of multi-modal samples validated by humans, and a variety of models are evaluated on this new benchmark.
strength_weakness: strength_weakness: 1) Random Chance

An issue in this work is that it insufficiently accounts for random guessing and data bias. As shown in [27], the ground-truth chance for it being answer A or B on MMbench are 26.4% (due to having questions with less than 4 options), and moreover, certain models may prefer certain answers. LLMs may get correct answers even when they require visual input. Given the 26.4% baseline for MMbench from [27], it is also surprising that the random choice value reported in this paper, for MMbench, is 0.0 in both Table 1 and Table 2. Moreover, an additional baseline based on majority class may also be necessary here.

For the results in Figure 2, which require 6 out of 8 models to get a hit it is unlikely that random guessing has a big influence. However, for the results in Table 1 this is not the case. For instance, it appears that all MMBench results in Table 1 are below the 26.4% from [27] - which means all LLM do worse than always answering A (or B). For other benchmarks there may similarly be data biases in which answer is more frequent.

Relatedly, it may be the case that such data biases about which options are more frequently chosen are more pronounced for multi-modal questions, i.e., when asked about colour the answer is more often grey across all datasets and settings - which means LLM do not learn this data bias as their training data does not include such questions, but LVLM may learn this because the same bias is present in their training data. While this is unlikely to fully explain the phenomenon observed in Table 2, it may explain part of it and not be directly related to data leakage. 

2) Not all benchmarks

The issues identified with existing benchmarks do not hold evenly across the benchmarks tested. In particular, MMBench and slightly lesser, MathVista seem to do pretty well with respect to these issues. This also appears to be reflected in the construction of MMStar, where after Manual Review the proportion of questions from these two existing benchmarks jumps considerable - in the end making up almost half of the MMStar benchmark. Which raises questions whether there is any benefit of using MMStar versus simply using MathVista and MMBench. 

3) Manual review

The manual review description is insufficiently clear. From the three criteria applied, only the first one is somewhat clear - for the other two it is unclear how the 'experts' judged this. I would expect description (can be in appendix) of agreement rates between these experts for these criteria, as well as further description of what these criteria entail. 

4) New metrics

The newly proposed metrics MG and ML are somewhat unclear in what they measure. If the 1500 questions in MMStar all require visual input as determined by the manual review, then what additional information does the MG metric give? Given the discussion above about chance, it appears the MG metric is more of a correction for random guessing. The ML metric similarly doesn't account for random guessing, or the aforementioned potential for data biases.
questions: questions: The paper raises a number of interesting questions and points out limitations of (some) existing benchmarks. Unfortunately, the data leakage issue appears to be not fully disentangled from the potential of data biases. I would appreciate input from the authors on this point.

Additionally, I would be interested in discussion on random guessing within the paper, and whether the ideas presented in [27] may address this - and then subsequently how this influences the findings in the paper. Also taking into account the majority class versus random guess.
limitations: limitations: I believe question 9 in the paper checklist has been answered incorrectly - or at least the justification given is not valid. Given that the paper discusses a benchmark that may include images of people or copyrighted material it is crucial that the authors affirm whether the work has been done in accordance to the ethical guidelines. Even if the data is a compilation of existing datasets - by selecting and combining information from these the new resulting dataset may be biased in ways the original datasets were not (e.g., by selecting only those images containing users from a certain demographic).

Comments: title: Official Comment by Authors
Thanks for the appreciation
Answer to authors
comment: Dear Reviewer sNEM:

Thank you for your time and patience in reviewing our submission and rebuttal. Your appreciation for our work, along with the questions and suggestions you provided, has greatly helped us improve the quality of our work.

Best regards and thanks,

Paper 2491 Authors
Thank you for the detailed reply and clarifications, with this my concerns are addressed, I will update my score to a weak accept.
Thank you for your prompt response and appreciation. Your suggestions have indeed helped us improve the quality of this work.
Thank you for answering the questions.