ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 13:47:19

Abstract: Recent years have witnessed remarkable progress in the development of large vision-language models (LVLMs). Benefiting from the strong language backbones and efficient cross-modal alignment strategies, LVLMs exhibit surprising capabilities to perceive visual signals and perform visually grounded reasoning. However, the capabilities of LVLMs have not been comprehensively and quantitatively evaluated. Most existing multi-modal benchmarks require task-oriented input-output formats, posing great challenges to automatically assess the free-form text output of LVLMs. To effectively leverage the annotations available and reduce the manual efforts required for constructing new benchmarks, we propose to re-formulate existing benchmarks into unified LVLM-compatible formats. Through systematic data collection and reformulation, we present ReForm-Eval benchmark, offering substantial data for evaluating various capabilities of LVLMs. Through extensive experiments and analysis in ReForm-Eval, we demonstrate the comprehensiveness and reliability of ReForm-Eval in assessing various LVLMs. Our benchmark and evaluation framework will be open-sourced as a cornerstone for advancing the development of LVLMs.

Main paper: 
 introduction: 
 With the trend led by ChatGPT (OpenAI, 2023a) , LLMs (Large Language Models) (OpenAI, 2023b; Touvron et al., 2023a; Chiang et al., 2023) have ushered in revolutionary advancements in Natural Language Processing (NLP). Inspired by these efforts, researchers attempt to extend the success of LLMs to the realm of vision language. By equipping LLM with visual encoders and aligning multimodal representations through generative pre-training, large vision language models (LVLMs) (Li et al., 2023b; Liu et al., 2023b; Zhu et al., 2023; Ye et al., 2023) possess the capability to comprehend visual information and engage in multi-modal conversations with users.However, the reliability of such LVLMs remains a mystery. On the one hand, these models demonstrate surprising abilities like OCR (Liu et al., 2023d) , meme understanding (Zhu et al., 2023) , and visual commonsense reasoning (Li et al., 2023b) . On the other hand, LVLMs suffer from fundamental issues, such as object hallucination (Li et al., 2023d) . Meanwhile, due to the lack of suitable benchmarks, there is a shortage of quantitative analysis and comparison of LVLMs.The main reason for this situation is the structural gap between existing task-oriented multi-modal benchmarks and LVLMs. Most existing benchmarks are designed for specific tasks and demand highly structured input-output formats (Lin et al., 2014) . For instance, VQA v2 (Goyal et al., 2017) requires concise answers, typically in the form of single words or short phrases. Previously evaluated vision-language pre-trained models (Chen et al., 2020; Zhang et al., 2021) need to be fine-tuned and learn task-specific parameters to fit the structures of such benchmarks. On the contrary, LVLMs are flexible and tend to provide detailed responses, even for yes-or-no questions. As depicted in the flowchart in the upper part of Figure 1 , such gap poses the greatest obstacle to accurate automated evaluation, particularly when assessing the desired zero-shot capabilities.To bridge the structure gap, we explore ways of re-formulating existing benchmarks into unified formats that are compatible with LVLMs. Referring to Figure 1 , we adapt the evaluation process to the unified form shown in the lower part. Multi-modal benchmark datasets are re-formulated as Figure 1 : Illustration of the unified re-formulation of existing benchmarks into multiple-choice problems. The text within square brackets indicates the evaluation methods, with red and green denoting incorrect and correct judgment, respectively. "EM" is short for exact match.multiple-choice problems or specialized text generation problems. Datasets for tasks with specific text generation requirements, like OCR and image captioning, are re-formulated as specialized text generation problems. Other datasets are restructured into multiple-choice problems.The unified formulation enables universal and comprehensive evaluation. For each formulation, we design a consistent and reliable evaluation method. As mentioned in (Fu et al., 2023) , current LVLMs may struggle to follow multiple-choice instructions, we propose both black-box and white-box approaches to assist: (1) Guiding LVLMs to output in desired formats through in-contextlearning; (2) Directly calculating the generation probability for options and selecting the one with the highest value. Considering the sensitivity of LVLMs to the input prompts (Zeng et al., 2023) , we design an instability-aware evaluation strategy and introduce a metric to characterize such instability.Based on the re-formulation framework, we present our unified multi-modal benchmark, ReForm-Eval. For a comprehensive evaluation, we re-formulate 61 benchmark datasets based on existing data resources, the evaluation dimensions range from basic visual perception to high-level visual reasoning and dialog. Compared with recent LVLM benchmarks that require manual annotation (Fu et al., 2023; Liu et al., 2023c) , ReForm-Eval fully utilizes publicly open resources and provides significantly more data, almost 100 times the size of MMBench. Meanwhile, unlike LVLM-ehub (Xu et al., 2023) , which requires designing complex and dataset-specific evaluation strategies, ReForm-Eval offers greater scalability and a more universally applicable and efficient evaluation approach.Based on ReForm-Eval, we conduct a comprehensive evaluation of 16 open-source LVLMs across various capability dimensions. We hope ReForm-Eval and the associated findings can constitute a valuable augmentation to the ongoing efforts in LVLM research and development. 
large vision language models: 
 Inspired by the advancements of LLMs and the multi-modal understanding abilities demonstrated by GPT-4 (OpenAI, 2023b) , developing open-source LVLMs currently dominates the multi-modal research. Visual signals encoded by visual encoders (Radford et al., 2021) are incorporated in LLMs through linear projection (Tsimpoukelli et al., 2021) , Q-former (Li et al., 2023b) , or cross-attention layers (Alayrac et al., 2022) . To enable multi-modal instruct tuning, MiniGPT4 (Zhu et al., 2023) bootstraps high-quality data by refining the previous output, LLaVA (Liu et al., 2023b) proposes to employ GPT-4 to generate image-involved dialogs while other works construct instruct tuning data from existing vision-language benchmarks (Xu et al., 2022; Dai et al., 2023; Li et al., 2023c) .To seamlessly adapt LLMs for multi-modal scenarios, many efforts are paid including designing strategies for parameter freezing (Ye et al., 2023) , introducing light-weight trainable modules into the backbone (Gong et al., 2023; Gao et al., 2023) , incorporating continuous output (Peng et al., 2023; Chen et al., 2023) , and enhancing the visual representations (Zeng et al., 2023; Hu et al., 2023; Li et al., 2023a) . Benefiting from the aligned representations from ImageBind (Girdhar et al., 2023) , LVLMs can be further extended to more modalities (Han et al., 2023; Su et al., 2023) .However, the capabilities of existing LVLMs are mainly demonstrated by qualitative examples (Zhu et al., 2023; Su et al., 2023; Gong et al., 2023) . To our knowledge, few benchmarks are suitable for evaluating the capabilities of LVLMs, hindering quantitative analysis and comparison of LVLMs.Task-Oriented Benchmarks Most existing multi-modal benchmarks can not be directly utilized to evaluate LVLMs since they are designed for specific tasks and rely on structured inputoutput formats for evaluation. VQA v2 (Goyal et al., 2017) requires concise answers, retrieval benchmarks (Lin et al., 2014; Young et al., 2014) demand dense scores for all image-text pairs, VCR (Zellers et al., 2019) provides coordinates to refer visual object in the question, and bounding box output is necessary for RefCOCO (Kazemzadeh et al., 2014) . This characteristic makes it challenging to utilize such benchmarks to evaluate the free-form text outputs of LVLMs unless complex post-processing and evaluation methods are designed specifically (Xu et al., 2023; Yin et al., 2023) .Benchmarks for LVLMs To facilitate reliable and efficient automated evaluation of LVLMs, efforts are paid to construct LVLM-compatible benchmarks, such as yes-or-no problems in MME (Fu et al., 2023) and multiple-choice problems in MMBench (Liu et al., 2023c) . A portion of the benchmarks are designed to assess specific capabilities (Liu et al., 2023d; Wang et al., 2023) or diagnose particular issues (Li et al., 2023d; Zhao et al., 2023) , while others aim for comprehensive evaluation (Fu et al., 2023; Liu et al., 2023c) . However, limited manual annotation (around 100 samples per dimension in MME and MMBench) could potentially introduce evaluation bias into the results. 
reform-eval benchmark: 
 In this section, we describe how to construct ReForm-Eval by re-formulating existing task-oriented multi-modal benchmarks. Section 3.1 introduces the general framework of re-formulation. Section 3.2 summarizes the capability dimensions assessed in ReForm-Eval and corresponding datasets. Section 3.3 illustrates the methods and strategies used to evaluate LVLMs based on ReForm-Eval.Existing LVLMs primarily adopt LLMs as backbones and use free-form text to interact with users. This paradigm makes the output more flexible and aligned with human needs. However, the gap between these models and existing highly structured benchmarks poses challenges for evaluation. In order to effectively reuse the annotations in existing benchmarks, these benchmarks need to be reformulated into appropriate formats. Motivated by benchmarks for LLMs (Hendrycks et al., 2020; Srivastava et al., 2022; Huang et al., 2023) , ReForm-Eval considers two formats that are compatible with LVLMs, namely multiple-choice problems and text-generation problems.Multiple-choice problem is the primary format in ReForm-Eval. By providing options for the questions, models are guided to produce responses in a constrained format. The key in multiplechoice problem construction is how to prepare meaningful negative options. Generally, for closevocabulary classification tasks, we build relationships between categories based on which hard negative options are selected. For open-ended tasks, based on the question and the correct answer, negative options can be obtained with the help of task-specific strategies or LLMs like ChatGPT.For OCR and image captioning that involves text generation, corresponding benchmarks are formulated as text-generation problems tailored to various scenarios. We curate the input prompts to describe the tasks and requirements. For OCR tasks, responses should contain the target tokens in the image. For description tasks, models should provide concise depictions of the visual content.To address the wide range of questions posed by users, LVLMs need to possess diverse capabilities. For a comprehensive evaluation, we curate 61 benchmark datasets from existing resources, summarizing the assessed capabilities into 2 major categories and 8 sub-categories which are illustrated in Figure 2 . To avoid information overload, details about the re-formulation procedures and dataset statistics are provided in Appendix A. Coarse-Grained Perception (CG) Coarse-grained perception is the ability to recognize the overall layout and main objects at the image level. We evaluate this capability through image classification using Flowers102 (Nilsback & Zisserman, 2008) , CI-FAR10 (Krizhevsky et al., 2009) , ImageNet-1K (Deng et al., 2009 ), Pets37 (Parkhi et al., 2012) , and MEDIC (Alam et al., 2023) benchmarks, and scene recognition using TDIUC (Kafle & Kanan, 2017) and VizWiz (Gurari et al., 2018) benchmarks. The samples are re-formulated as multiple-choice questions.Fine-Grained Perception (FG) Fine-grained perception requires detailed sensing at the object level. We set up the object perception task (using TDIUC (Kafle & Kanan, 2017) and MSCOCO (Lin et al., 2014) benchmarks) and the object grounding task (using MSCOCO (Lin et al., 2014) and Ref-COCO (Yu et al., 2016) benchmarks) for evaluation. Object perception measures how well a LVLM can identify local semantics, while object grounding assesses the ability to localize fine-grained objects. All tasks are formulated as multiple-choice questions.Scene Text Perception (STP) Scene text perception enables LVLMs to identify, understand, and perform inference based on text in images. This evaluation is conducted through optical character recognition (OCR) using 6 benchmarks (including CUTE80 (Risnumawan et al., 2014) , IC15 (Karatzas et al., 2015) , IIIT5K (Mishra et al., 2012) , COCO-Text (Mishra et al., 2012) , Wor-dArt (Xie et al., 2022) , TextOCR (Singh et al., 2021)) , key information extraction (KIE) using 3 benchmarks (including SROIE (Huang et al., 2019) , POIE (Kuang et al., 2023) and FUNSD (Jaume et al., 2019) ) and OCR-based VQA using 3 benchmarks (including TextVQA (Singh et al., 2019) , DocVQA (Mathew et al., 2021) and OCR-VQA (Mishra et al., 2019) ). We consider STP as a specialized text-generation problem that requires output to contain exactly matched words.Visually Grounded Reasoning (VGR) A reliable LVLM is supposed to perform reasoning based on multi-modal contextual information. In order to assess such capability, we adopt the commonly applied visual question answering (VQA) task and its variant, knowledge-based visual question answer (K-VQA), which further requires models to utilize internally stored knowledge. For vanilla VQA, we adopt VQA v2 (Goyal et al., 2017 ), GQA (Hudson & Manning, 2019 ), and Whoops (Bitton-Guetta et al., 2023) . As for KVQA, we consider 6 benchamrks including OK-VQA (Marino et al., 2019) , ScienceQA (Lu et al., 2022) , VizWiz (Gurari et al., 2018) , Vi-QuAE (Lerner et al., 2022) , A-OKVQA (Schwenk et al., 2022) and ImageNetVC (Xia et al., 2023) . The aforementioned benchmarks are re-formulated into multiple-choice questions.Spatial Understanding (Spatial) Spatial understanding is the key to the real-life application of LVLMs on robots. This task requires a comprehensive understanding of both the object-object and object-observer relationship so as to make reasonable behaviors. We access such capability through spatial relation judgment (SRJ) using VSR (Liu et al., 2023a ) and MP3D-Spatial, a benchmark designed for embodied tasks in real-world environments, constructed from Matterport3D (Chang et al., 2017) . Additionally, we employ Space-Based Reasoning (SBR) through the CLEVR (Johnson et al., 2017) benchmark. The SRJ task aims to accurately identify spatial relationships, forming a concept of where the ego is in space. The SBP task entails complex reasoning ability based on the understanding of spatial relationships. All samples are re-formulated as multiple-choice questions.Cross-Modal Inference (CMI) A thorough comprehension of both modalities is required to perform cross-modal inference on the relationship between images and texts. We consider two tasks: image-text matching (ITM) requires models to measure the cross-modal similarities and visual entailment (VE) demands models to check whether the information is entailed across modalities. MSCOCO (Lin et al., 2014) , WikiHow (Koupaee & Wang, 2018) , Winoground (Thrush et al., 2022) are adopted for ITM while VE considers SNLI-VE (Xie et al., 2019) and MOCHEG (Yao et al., 2023) . Both tasks are re-formulated as multiple-choice questions.Visual Description (Desc) Visual description is an inherent capability of LVLMs as generative models. We adopt the image captioning task on MSCOCO (Lin et al., 2014) , TextCaps (Sidorov et al., 2020) , NoCaps (Agrawal et al., 2019), and Flickr30K (Young et al., 2014) for evaluation. These datasets are formulated as text-generation problems with the requirement of concise outputs.Multi-Turn Dialogue (Dialog) Existing benchmarks primarily focus on single-turn conversation. ReForm-Eval evaluates the performance of LVLMs in multi-turn dialogues. We consider the multiturn VQA task using VisDial (Das et al., 2017) and VQA-MT, the latter is constructed by reorganizing questions in VQA v2. Both benchamrks are formulated as multiple-choice questions.With the unified problem formulation, the performance of LVLMs can be universally evaluated. For specialized text-generation problems, the evaluation method depends on the scenario. For visual description, we follow Li et al. (2023b) to use CIDEr (Vedantam et al., 2015) as the evaluation metric. Since the adopted datasets mainly provide concise references, we craft the prompt to require concise responses and restrict the maximum number of tokens a model can generate. As for STP, input prompts are well-designed to instruct models to identify the scene texts. The evaluation metric is word-level accuracy: the proportion of ground-truth words that appear complete in the output.Considering multiple-choice problems, the model performance is assessed using accuracy. We label the answer options with markers like "(A)" and then determine correctness by checking the markers in the output of models. The challenge with this approach is that current LVLMs may not always adhere well to multiple-choice instructions, i.e. the output may not include the required marker.To assist in the evaluation of multiple-choice problems, ReForm-Eval provides both a black-box method and a white-box method. The black-box method provides in-context samples to guide LVLMs to generate responses in desired formats. Here is an example of the input prompt: Assistant: The answer is (A) Yes. Human: X question Options: X options Assistant: The answer is where X SystemMessage is the system message required by most LVLMs, X question and X options are respectively the question and the answer options described in text, the text in red is the in-context sample provided to the model. Notice that the in-context sample provides no information about the image. The effectiveness of the black-box strategy is demonstrated in Section 4.3.3.The white-box approach is based on the inherent attribute of current LVLMs as generative models. Given the visual context v, the question q, and N answer optionsC = {c i } Ni=1 , the answer prediction can be determined by the generation likelihood predicted by the evaluated model:EQUATIONwhere P θ (c i t |v, q, c i <t ) is parameterized by the causal-LLM-based LVLMs and {c i 1 , ..., c i tc } is the tokenized sequence of c i . For multiple-choice problem assessment, we provide both the black-box generation evaluation results and the white-box likelihood evaluation results.As demonstrated in previous work (Xu et al., 2022; Zeng et al., 2023) , LLM-based models are sensitive to the different but equivalent instructions. In ReForm-Eval, instability-aware evaluation To directly characterize the instability of models, we further introduce a metric. For a multiplechoice problem with answer options C = {c i } N i=1 , the empirical prediction distribution of a model can be calculated from the M tests asp i = 1 M M j=1 1(ĉ j = c i )where ĉj is the prediction of the j-th test. Then the instability is measured by the entropy of the prediction distribution: e = -N i=1 p i log(p i ). Larger e indicates higher uncertainty in the predictions for that sample. For textgeneration tasks, instability is not accessible as the prediction distribution is not directly measurable. 
implementation details: 
 Based on ReForm-Eval, we evaluate 16 models with around 7B parameters that are trained with 13 different methods, including BLIP-2 (Li et al., 2023b) , InstructBLIP (Dai et al., 2023) , LLaVA (Liu et al., 2023b) , MiniGPT4 (Zhu et al., 2023) , mPLUG-Owl (Ye et al., 2023) , PandaGPT (Su et al., 2023) , ImageBind-LLM (IB-LLM) (Han et al., 2023) , LLaMA-Adapter V2 (LA-V2) (Gao et al., 2023) , multimodal-GPT (mmGPT) (Gong et al., 2023) , Shikra (Chen et al., 2023) , Lynx (Zeng et al., 2023) , Cheetor (Li et al., 2023a) , BLIVA (Hu et al., 2023) . Details of the methods are introduced in Appendix B.2. All experiments are conducted in the same software and hardware environment to ensure fairness. For specific parameter settings, please refer to Appendix B.1.Notations For models with multiple variants based on different backbones, we use subscripts to denote the backbone used: F , V , L, and L 2 represent FlanT5, Vicuna, LLaMA, and LLaMA2, respectively. For multiple-choice problems, "Generation Evaluation" and "Likelihood Evaluation" are respectively based on the black-box and white-box strategies. For each task under different strategies, the best result is marked in bold while the runner-up is underlined.Table 1 presents the comprehensive performance of each model across dimensions, from which several insights can be gleaned. (1) BLIP-2 and InstructBLIP continue to hold the top-2 positions in most dimensions, but in some individual dimensions, Lynx, BLIVA, and Shikra also take the lead.(2) It's worth noting that the effectiveness of models like BLIVA and Lynx only becomes apparent when using likelihood evaluation. We suspect this is attributed to the instruction-following ability of models, please refer to Section 4.3.4 for a detailed analysis. (3) Compared to models based on CLIP visual encoders, PandaGPT and IB-LLM, which are based on the ImageBind encoder, exhibit relatively poorer performance in image-text tasks. Meanwhile, most top-performing models utilize Vicuna and FlanT5 as the backbone. Further analysis is available in Section 4.3.1 regarding the Figure 3 : The influence of different language and visual backbones. For generation evaluation, we average the results of various models based on the backbone used. To better visualize the results, we selected heatmaps across six dimensions (dialog and desc are omitted). For likelihood evaluation, we further compute the average score across dimensions since the performance trend is consistent. Note that "ImgBD" is short for ImageBind in this figure. we compare the average performance between models pre-trained with and without MSCOCO. As shown in Figure 4 (a), MSCOCO not only helps with in-domain tasks but also enhances generalization results on out-domain tasks. Therefore, to effectively align cross-modal representations during pre-training, it is crucial to include such high-quality pre-training data.To scale up the LVLM training, it is necessary to utilize imagetext pairs crawled from the web. Figure 4 (b) compares two groups of models: the red-marked group uses data filtered based on rules or CLIP, such as CC (Sharma et al., 2018) and LAION (Schuhmann et al., 2021) , while the blue-mark utilizes relatively high-quality data including aforementioned annotated data and synthetic captions from BLIP (Li et al., 2022) . Results show that it is more effective to scale up utilizing synthetic data, resulting in a desired increasing curve. We believe the reason behind this is that synthetic captions are cleaner and more associated with images. While the diversity of data may be impaired, the generalizable backbones mitigate the negative impact.Instruct-Tuning Dataset We also explore the impact of the number of instruct-tuning samples. The fitted curve in Figure 4 (c) demonstrates that increasing the number of instruct-tuning samples leads to improved performance of LVLMs.In general, the quality of pre-training data and the scale of instruct-tuning samples are crucial factors for improving LVLMs. Appendix C.4 provides the complete data used in this section.To demonstrate the effectiveness of the black-box evaluation strategy introduced in Section 3.3.1. We assess LVLMs' ability to follow multiple-choice instructions under different strategies. The experiments are conducted in the re-formulated VQA v2, a response is considered as hitting the format if it includes the option mark like "(A)". Some results are listed in Table 3 . It is obvious that the ability is tightly related to the backbone. LVLMs based on raw LLaMA inherit the weak instruction-following ability of the backbone. At the same time, fine-tuning the full backbone results in catastrophic forgetting of the capability, while LoRA-based fine-tuning does not. However, incontext samples can effectively provide format information and guide LVLMs to respond in the desired format, facilitating automated evaluation. The complete results are in Table 22 . For generation evaluation, the results reflect the coupling of the multi-modal understanding capability and the instruction-following capability. Meanwhile, likelihood evaluation directly probes the generative models and relaxes the requirement for instruction following.As shown in Figure 5 , likelihood evaluation yields better results than generation evaluation in most cases, even when LVLMs are guided through in-context learning. This indicates that most LVLMs have limited instruction-following capability, further hindering downstream performance. We believe the primary factor behind this is the LLM backbone, as models based on FlanT5 and LLama2-Chat have the least performance gap between likelihood and generation evaluation in all the dimensions, FlanT5-based models even perform better using generation evaluation in CG, FG, VGR, and CMI. To address the issue, LVLMs should leverage stronger backbones or introduce sufficiently diverse data for instruct tuning, as done in FlanT5. Besides, the comparison between Vicuna and Vicuna+ demonstrates that multi-modal instruct tuning the backbone currently can not improve the instruction-following capability of LVLMs. To investigate the source of instability, we conduct experiments on ScienceQA by applying three types of perturbations separately to LVLMs, including random instructions, shuffling option orders, and random option marks (uppercase, lowercase, or numeric).As illustrated in Table 4 , shuffling the option order results in the highest instability, highlighting a misunderstanding of the option contents. Similar to MM-Bench (Liu et al., 2023c) , we observe that most models exhibit some degree of preference for specific options (refer to Appendix C.6 for more details). Our in-depth finding is that option preference reduces the instability from random instructions and random option marks, but increases the instability from random option orders. The randomness of instruction has the least effect, suggesting that LVLMs can reasonably comprehend the carefully crafted instructions. With likelihood evaluation, the instability is significantly lower because it is a white-box method that directly probes generative models without the need for random sampling during generation. These phenomenons are common to all models, the complete results are in Appendix C.5. In summary, current LVLMs are unstable and sensitive to subtle changes in the prompt, especially during black-box evaluations. 
conclusion: 
 In this paper, we propose to re-formulate task-oriented multi-modal benchmarks to evaluate LVLMs. By systematically collecting and efficiently re-formulating 61 benchmarks into unified formats that are compatible with LVLMs, we construct a benchmark, ReForm-Eval, which covers 8 capability dimensions. Compared with recently constructed benchmarks for LVLMs, ReForm-Eval provides more data without the need for manual annotation. Additionally, we design dependable automated evaluation methods based on the unified formats, ensuring an impartial assessment of different LVLMs. Leveraging ReForm-Eval, we conduct an exhaustive evaluation of various LVLMs and delve into the factors influencing their performance. Generally, ReForm-Eval serves as a reliable tool for quantitative analysis of LVLMs, aiding in the research and development of LVLMs. 
 
 Appendix: In this section, we introduce the collected datasets and the corresponding re-formulation procedures in detail. The statistics of the re-formulated datasets are provided in Table 5 and Table 6. A.1 COARSE-GRAINED PERCEPTION For the Flowers102 dataset, we employ the complete validation set for evaluation purposes. However, for CIFAR10, ImageNet-1K, Pets37, and VizWiz, we perform random subsampling of 10%.Concerning the TDIUC dataset, given that certain models in their training phase utilized a portion of the TDIUC dataset originating from the Visual Genome, we initially exclude this subset of data to prevent potential data leakage. Subsequently, we apply a shuffling operation to the entire TDIUC dataset and perform equidistant sampling, resulting in the selection of 2.5% of the sport recognition data (TDIUC sport ) and 1% of the scene recognition data (TDIUC scene ). In the case of MEDIC (Alam et al., 2023) , we sample an equal number of samples from each label to balance the answer distribution.For Flowers102 and Pets37, we randomly select three incorrect class labels, in addition to the correct label, from their original set of categories to form multiple-choice question options. For the TDIUC, we aggregate all answers for the same task to create an answer pool, and then utilize the same approach above to construct four answer options for multiple-choice questions.For ImageNet-1K, we calculate similarities within its own set of 1000 categories using WordNet and selected the four options with the highest similarity to the correct class as choices (the highest one must be the right answer, and we need to get them out of order).For CIFAR10, we initially employ WordNet to identify synonyms of the answers that are semantically related but not synonymous. These synonyms are then ranked based on their similarity. Subsequently, we manually adjust some of the less common candidate options. Finally, we likewise select the top four options with the highest similarity as all choices.As for VizWiz, we re-formulate it into two benchmarks: VizWiz 2 as a binary classification task to determine whether there is any quality issue with the image. VizWiz 4 as a 4-choice question, requiring the model to determine the exact reason for the quality issue. We sort the issues related to image quality based on the number of votes in the annotations, the top one is considered the true label while the second to fourth options serve as negative choices.For MEDIC (Alam et al., 2023) , it is re-formulated to MEDIC dts , a benchmark for disaster type selection (dts), we directly use all seven classification labels as choice options.For TDIUC (Kafle & Kanan, 2017) , we initially exclude the subset sourced from Visual Genome (Krishna et al., 2017) to prevent evaluation on the training data. Then, we shuffle the entire dataset and conducted an equidistant sampling strategy for task sample balance. Specifically, we sample 1% of the data for color (TDIUC color ), detection(TDIUC detection ), and counting tasks (TDIUC counting ), and 2.5% for position tasks. As for the utility task (TDIUC utility ), we retain and utilized all 171 data samples. For answer options, we uniformly count all answers within the data and randomly selected three options other than the correct answer to form all four choices.Regarding RefCOCO (Yu et al., 2016) , we re-formulate the referring expression selection (RefCOCO res ) task, in which the LVLMs are supposed to select the correct referring expression from the options based on the image region in the bounding box. We sample an equal number of samples from each object category, in order to balance the correct referring expression categories appearing in the questions. As for negative options in each question, we sample the negative referring expression from a distinct subcategory within the same category as the positive sample.For MSCOCO (Lin et al., 2014) , we re-formulate four tasks: object counting (counting), multiple class identification (MSCOCO mci ), grounded object identification (MSCOCO goi ) and missing object selection (MSCOCO mos ) for object-level evaluation. The multiple class identification task aims to evaluate the LVLM's ability of object classification. Further, the grounded object identification and missing object selection tasks concentrate on object perception within a specified region of interest. The former allows models to assess which object exists within the given bounding box of the image, while the latter asks models to judge which object disappears within all the given bounding boxes of the image.For the multiple class identification and grounded object identification tasks, we randomly sample 300 object annotations from each super-category in the valid split to ensure balance. This results in a total of 3600 evaluation data samples for each task. For the mos task, we filter out the objects with the height and width of their bounding boxes smaller than 50 and finally get 2479 samples. As for options generation, we employ a hierarchical strategy. For the multiple class identification task, we begin by randomly selecting the object class from within the super-category of the target object.If there are insufficient options, we broaden our selection to all object categories. In tasks related to region, our initial step is to randomly choose object categories present in the image but do not meet the requirement specified in the question. In cases where this is not possible, we follow the sampling procedure used in the multiple-class identification task. The examples of these grounded fine-grained tasks as shown in Table 6 . The counting task has the same setting as the counting task in the TDIUC dataset.For OCR, we use 6 original OCR benchmarks (including CUTE80 (Risnumawan et al., 2014) , IC15 (Karatzas et al., 2015) , IIIT5K (Mishra et al., 2012) , COCO-Text (Mishra et al., 2012) , Wor-dArt (Xie et al., 2022) and TextOCR (Singh et al., 2021) ) as the evaluation tasks. Current OCR benchmarks utilize cropped images containing only target text as visual input sources (Xu et al., 2023; Liu et al., 2023d) . To further assess text identification in complex visual contexts, we propose grounded OCR tasks (including gIC15, gCOCO-Text, and gTextOCR). Specifically, we filter out the bounding boxes containing target texts larger than 40x40 for better evaluation. The image, along with the bounding box annotations and the corresponding instruction, will be fed into the model for evaluation, which is similar to the grounded fine-grained tasks (i.e. MSCOCO goi ).For KIE, we utilize the test splits of 3 benchmarks (including SROIE (Huang et al., 2019) , POIE (Kuang et al., 2023) and FUNSD (Jaume et al., 2019) ) as the evaluation tasks.And for OCR-based VQA, we use 3 benchmarks (including TextVQA (Singh et al., 2019) , DocVQA (Mathew et al., 2021) and OCR-VQA (Mishra et al., 2019) ) as the evaluation tasks. We filter out the question-answer pairs that need to be inferred based on the scene texts.A.4 VISUALLY GROUNDED REASONING For VQAv2 (Goyal et al., 2017) , we sample 10% for reformulation owing to the extremely large population. Besides, since ViQuAE (Lerner et al., 2022) provides relevant knowledge information for each question, we additionally construct K-ViQuAE with knowledge as context, which assesses models' reasoning ability hierarchically with ViQuAE (Lerner et al., 2022) . For ScienceQA (Lu et al., 2022) , only 2017 questions of all the 4241 test set are paired with an image, which are selected in our benchmark. Besides, original A-OKVQA (Schwenk et al., 2022) gives rationales for answering each question, therefore we construct A-OKVQRA and A-OKVQAR for hierarchical evaluation.For VQAv2 (Goyal et al., 2017) , GQA (Hudson & Manning, 2019), OK-VQA (Marino et al., 2019) , VizWiz (Gurari et al., 2018) , ViQuAE (Lerner et al., 2022) and Whoops (Bitton-Guetta et al., 2023) , ChatGPT is employed to generate appropriate negative options, and the prompt template for querying is:You are a multiple-choice generator. Given a question and an answer, you need to generate three additional incorrect options while ensuring their plausibility and confusion. Question: {question} Answer: {correct answer} Note that for yes or no questions, the negative option is directly derived as no or yes, and ChatGPT is not employed.While ImageNetVC (Xia et al., 2023) randomly selects 3 candidate options from the correspondent answer set with the commonsense type of each question. As for ScienceQA (Lu et al., 2022) and A-OKVQA (Schwenk et al., 2022) , they adopt their original options because of the original singlechoice formulation.As for A-OKVQAR, the prompt template for querying ChatGPT to generate negative rationales is:You are a multiple-choice generator. Given a question and an answer, along with a rationale for that answer, you need to generate 3 counterfactual rationales. These counterfactual rationales should be contextually relevant while also being sufficiently distinct from the correct rationale. Question: {question} Answer: {correct answer} Rationale: {rationale} A.5 SPATIAL UNDERSTANDING For CLEVR (Johnson et al., 2017) , we filter out the question types that do not involve spatial relations and randomly select 300 samples from each question type related to spatial relations. For different question types, we randomly select false options from their corresponding answer sets. In cases where some question types have insufficient options, we add 'Not sure' and 'Unknown' as false options to maintain the number of four options.For VSR (Liu et al., 2023a) , the original dataset comprises captions that describe true or false spatial relations among objects in the corresponding images. We select image-caption pairs from the test split where the spatial descriptions are right and use them for our evaluation tasks. The false options are generated by randomly sampling different spatial relations from the test split. MP3D (Chang et al., 2017 ) also known as Matterport3D, comprises a large-scale collection of RGB-D images captured from nearly 10,800 real indoor viewpoints with 50,811 object instance annotations. Based on this dataset, we extract two types of spatial relations to re-formulate our benchmark MP3D-Spatial: object-object level relations (left, right, above, under, on top of, and next to) and object-observer level relations (far and close). For spatial relations 'on top of' and 'next to,' we use In this section, we consider two kinds of tasks, including image text matching and visual entailment.For MSCOCO (Lin et al., 2014) , we re-formulate two tasks, including COCO image text matching (COCO itm ) and COCO image text selection (COCO its ). The matching task requests LVLMs to determine whether the given image and text are matched. The selection task instructs LVLMs to select the best-matched caption for the given image. We randomly sample image and text pairs as positive samples. For each image, we first find the negative images that have distinct but similar object categories. For each negative image, we find the most similar captions with the positive caption according to the object appearing in the sentence.WikiHow (Koupaee & Wang, 2018) provides introductions to common skills. Within each skill, there are multiple crucial tasks, and each task is composed of several steps. We re-formulate the Wikihow image text selection task, in which given the task name, the LVLMs are supposed to choose the matching step description. We randomly sample visual and textual descriptions of the steps to form multiple-choice questions. To mine the hard negative options, we try to randomly take three samples from the same task as the positive sample. If the negative samples from the task level are insufficient, we then select some from the skill level and dataset level in turn.For Winoground (Thrush et al., 2022) , we re-formulate a caption selection task, which requires models to choose the correct caption. Since Winoground has captions for each image pair that have exactly the same words but in a different order, the options are the captions for every image pair.For SNLI-VE (Xie et al., 2019) , we re-formulate the visual entailment task, in which the LVLMs are required to determine whether the text can be inferred based on the image clues and should give answer of uncertainty when the evidence is insufficient. The options of multiple-choice question comprise "yes", "not sure" and "no". To balance the correct answer distribution, for each image, we sample an equal number of samples from each label.For MOCHEG (Yao et al., 2023) , we re-formulate the visual and textual entailment task, in which the LVLMs are supposed to determine whether the claim can be infered based on the visual and textual evidence and judge out whether the evidences are insufficient. The options consist of "supported", "refuted" and "not enough information".We re-formulate the image captioning task from four dataset including MSCOCO (Lin et al., 2014) , TextCaps (Sidorov et al., 2020) , NoCaps (Agrawal et al., 2019) and Flickr30K (Young et al., 2014) .In this task, the LVLMs are expected to generate a brief description for given image. Among these dataset, TextCaps additionally examines the optical character recognition capability of LVLMs by requesting models to pointing out the text in the image. We randomly sample these datasets for evaluation.To mimic a naive setup, we construct VQA-MT (VQA Multi-Turn) by considering multiple questions for the same image and gathering them into a multi-turn conversation. For VQA-MT, different images are accompanied by different amounts of questions in the re-formulated VQA v2 (Goyal et al., 2017) , only the images with more than 2 questions are kept. For images with more than 10 questions, only the first 10 questions are kept. All questions for the same image are arranged into a dialogue without inter-round dependencies. In the filtered dataset, there are 1073 image-dialogue pairs. The negative options are directly adopted from the re-formulated VQA v2.As for VisDial (Das et al., 2017) , there is a 10-turn QA dialogue for each image. the original datasets provide 100 options for each question while. The prompt template for querying GPT-3.5 to generate negative options is: I will provide a question with the correct answer, please give me 3 incorrect options to help me get a single-choice question. Question: {question} Answer: {correct answer} Different from the original VisDial to perform offline dialogue (the history contains correct answers), we perform online dialogue (the history contains the previous output of the models). To further investigate whether the performance of LVLMs changes with an increasing number of dialogue turns, we calculate the correlation coefficient between the accuracy and the number of dialogue turns.Our benchmark and the evaluation framework are PyTorch-based. All experiments are conducted on 8 Tesla V100 GPUs. During the evaluation, half precision is used to accelerate the process.To ensure fair comparisons between LVLMs, we try our best to keep the parameter setting aligned with the demo code provided by the original codebase. However, we limit the maximum number of tokens a model can generate for all LVLMs. It is set to 30 for most questions except the imagecaption task where it is set to the upper quartile (the 75th percentile) of the reference caption length in the corresponding datasets. all input texts are formulated into conversations as required by different models, using the same system messages, roles, and separators. As for the image input, we only consider single-image inputs, we use the same preprocess method mentioned in the original paper except for Lynx, which utilizes 420 × 420 input resolution and we still use 224 × 224 for a fair comparison.It is worth noting that ReForm-Eval comprises a total of over 500,000 evaluation instances across over 300,000 images, and considering the need for multiple tests for each instance, this results in significant computational cost. To this end, we further construct a subset by sub-sampling 10% data from the whole ReForm-Eval. All experiments conducted in this paper are based on the subset. We will open-source both the subset we use and the complete data for the research community.In this section, we introduce the evaluated LVLMs in detail. For each method, we identify the version assessed in this paper if multiple variants are provided by the method. Additionally, we summarize the architecture of LVLMs in BLIP-2 BLIP-2 (Li et al., 2023b ) is pre-trained in two stages: the representation learning stage and the generative learning stage, where the image encoder and the LLM are frozen and only a lightweight Q-Former is trained for bridging the modality gap. "blip2-pretrain-flant5xl" is evaluated in our experiment.InstructBLIP InstructBLIP (Dai et al., 2023) further extends BLIP-2 with task-oriented instruct tuning, pre-trained with Vicuna using the same procedure as BLIP-2. Additionally, an instructionaware Q-Former module is proposed in InsturctBLIP, which takes in the instruction text tokens as additional input to the Q-Former. During instruction tuning, only parameters of Q-Former are finetuned based on pre-trained checkpoints, while keeping both the image encoder and the LLM frozen. We take "blip2-instruct-vicuna7b" and "blip2-instruct-flant5xl" as evaluation versions.MiniGPT-4 MiniGPT4 (Zhu et al., 2023) adds a trainable single projection layer based on BLIP-2 and also adopts a two-stage training approach, where the first stage is pre-training the model on large aligned image-text pairs and the second stage is instruction tuning with a smaller but high-quality image-text dataset with a designed conversational template. During training, the image encoder, the LLM, and the Q-Former are all frozen. "pretrained-minigpt4-7b" is used in our setup.LLaVA LLaVA (Liu et al., 2023b) employs a linear layer to convert visual features into the language embedding space, with a pre-training and instruction tuning stage. During pre-training, both the visual encoder and LLM weights were frozen. Then, keeping only the visual encoder weights frozen, the weights of the projection layer and LLM in LLaVA are updated with generated instruction data. In our experiment, "liuhaotian/LLaVA-7b-delta-v0" and "liuhaotian/llava-llama-2-7bchat-lightning-lora-preview" are used for evaluation. mPLUG-Owl mPLUG-Owl (Ye et al., 2023) proposes a novel training paradigm with a two-stage fashion. During pre-training, mPLUG-Owl incorporates a trainable visual encoder and a visual abstractor, while maintaining the LLM frozen. In the stage of instruction tuning, language-only and multi-modal instruction data are used to fine-tune a LoRA module on the LLM. "MAGAer13/mplugowl-llama-7b" is used in our experiment, but LoRA is not implemented in this version.ImageBind-LLM (IB-LLM) ImageBind-LLM (Han et al., 2023) adopts a two-stage training pipeline. In the pre-training stage, a learnable bind network and a gating factor are updated. The bind network transforms image features, while the gating factor weights the transformed image features to determine the magnitude of injecting visual semantics and the result is added to each word token for each layer. In the instruction tuning stage, a mixture of language instruction data and visual instruction data is used to update partial parameters in LLaMA by LoRA and bias-norm tuning. We utilize "Cxxs/ImageBind-LLM/7B" for evaluation and call it "IB-LLM".LLaMA-Adapter V2 In LLaMA-Adapter V2 (Gao et al., 2023) , a joint training paradigm is proposed, where only the visual projection layers and early zero-initialized attention with gating are pre-trained using image-text data, while the late adaptation prompts with zero gating, the unfrozen norm, newly added bias, and scale factors are implemented for learning from the language-only instruction data. "LLaMA-Adapter-V2-BIAS-7B" is applied for evaluation.Multimodal-GPT (mmGPT) Multimodal-GPT (Gong et al., 2023) is fine-tuned from Open-Flamingo, where the whole open-flamingo model is frozen and the LoRA module is added and updated to the self-attention, cross-attention, and FFN part in the LLM, using language-only and multimodal instruction data. "mmgpt-lora-v0-release" is used for evaluation. To simplify, we refer to it as "mmGPT".PandaGPT PandaGPT (Su et al., 2023) utilizes a one-stage training method using a combination of 160k image-language instruction-following data from MiniGPT-4 and LLaVA, where only two components are trained: a linear projection matrix connecting the visual representation generated by ImageBind to Vicuna, and additional LoRA weights applied to Vicuna attention modules. "openllmplayground/pandagpt-7b-max-len-1024" is evaluated as our implemented version.Shikra Shikra (Chen et al., 2023) consists of a vision encoder, an alignment layer, and an LLM. This model is trained in two stages, where both the fully connected layer and the entire LLM are trained and the visual encoder is frozen. We select "shikras/shikra-7b-delta-v1" for our evaluation in this experiment.Cheetor Cheetor (Li et al., 2023a ) is initialized from BLIP-2 and pre-trains Q-Former that matches Vicuna and LLaMA2. A lightweight CLORI module is introduced that leverages the sophisticated reasoning ability of LLMs to control the Q-Former to conditionally extract specific visual features, and further re-inject them into the LLM. During training, only a set of query embeddings and two linear projection layers need to be updated. "cheetah-llama2-7b" and "cheetah-vicuna-7b" are specifically used for assessment.Lynx Lynx (Zeng et al., 2023) leverages the method of pre-training and instruction tuning, where lightweight trainable adapters are added after some layers in the frozen LLMs and a resampler mechanism is adapted to reduce the vision token sequence, using learnable latent queries. During pre-training, the newly added layers are trained to build connections of different modalities. "finetune-lynx" is opted for evaluation.BLIVA BLIVA (Hu et al., 2023) is initialized from a pre-trained InstructBLIP and merges learned query embeddings output by the Q-Former with projected encoded patch embeddings. Demonstrating a two-stage training paradigm, the patch embeddings projection layer is pre-trained and both the Q-Former and the project layer are fine-tuned by instruction tuning data. "mlpc-lab/BLIVA-Vicuna" is employed under evaluation in our experiment.C.1 PER-DIMENSION RESULTS AND ANALYSISIn this section, we will provide the complete results and corresponding analysis of all capability dimensions.Tabel 9 and Table 10 provide results of coarse-grained perception tasks, including image classification and scene recognition. For generation evaluation, BLIP-2 and InstructBLIP take the lead in most tasks except the image quality assessment in VizWiz. We speculate that this is because the training data primarily consists of text describing the visual content, with very little description of the image quality, which may result in models not understanding the image quality. However, this challenge is also faced by most LVLMs, some of them act worse than random guesses.For likelihood evaluation, the situation is slightly different. Most models perform better across different datasets while BLIP-2 and InstructBLIP struggle in Flowers102, ImageNet-1K, Pets37, and MEDIC. We hypothesize this is due to the fact that the category names of these datasets are not common, the calculated likelihood may not be as effective as those for common tokens. As mentioned in Section C.4, the diversity of the pre-training data used by BLIP-2 and InstructBLIP is limited, while mPLUG-Owl utilizes diverse pre-training data. This explains why mPLUG-Owl Table 9 : Evaluation results on coarse-grained perception. "Acc" and "Instby" are short for accuracy and instability, respectively. performs well on these tasks while BLIP-2 and InstructBLIP perform relatively poorly. However, BLIP-2 and InstructBLIP are able to distinguish these categories if provided in the question, enabling them to perform well under generation evaluation.Tabel 11 and Table 12 provide results of fine-grained perception tasks, including Object Perception and Object Grounding. For object perception, BLIP-2 and InstructBLIP dominate most tasks, especially when evaluated with the generation evaluation strategy. Under likelihood evaluation, the effectiveness of BLIVA is demonstrated, indicating that incorporating patch features helps the model to perceive fine-grained information. Additionally, Lynx is a strong competitor in object-perception tasks benefiting from its enhanced visual representations. Considering the results of MSCOCO goi , most models are able to solve a part of the questions, indicating that LVLMs are able to understand the bounding boxes in images and the grounded questions. Bounding box labeling can be an optional method to provide locality information without the need for understanding continuous input. As for object grounding, the task is quite difficult for most models. Only BLIP-2 and InstructBLIP perform well under generation evaluation, but they still struggle under likelihood evaluation. We speculate that this is because there are only subtle differences between options in these questions, such as "the person on the left" and "the person on the right". In generation evaluation, all options are provided in the context, helping the models with strong instruct understanding abilities to distinguish between them. As for likelihood evaluation, options are provided to the model separately, models may not be able to distinguish them effectively. also exhibit some outstanding performance. As discussed in Section 4.3.4, likelihood evaluation performances of many models are significantly better than generation.We also conducted a hierarchical evaluation of LVLMs' external knowledge incorporation and reasoning abilities. Comparing the results of ViQuAE and K-ViQuAE, as well as A-OKVQA and A-OKVQRA, it is evident that, with the provision of external knowledge, the performance of most models has significantly improved.In addition, it can be observed that, except for models based on FlanT5 and LLaMA2-Chat, the generation performance of other models on the A-OKVQAR dataset is significantly lower than the likelihood performance. This may be attributed to the fact that the rationale options are often longer, and for language models with weaker instruction-following capabilities (as mentioned in Section 4.3.4), memory lapses may occur during generation, thereby affecting performance.Table 17 provides results of the spatial understanding capability dimension, which includes relation judgment and space-based perception tasks. For generation evaluation, BLIP-2 and InstructBLIP dominate in most tasks. For likelihood evaluation, although BLIP-2 and InstructBLIP still perform well, Lynx and BLIVA start to hold leadership positions. We attribute this to the fact that BLIP-2, InstructBLIP, Lynx and BLIVA uses relatively high-quality MSCOCO data. As the images in VSR dataset are retrieved from COCO 2017, models tend to have better performance on this dataset.For the spatial relation judgment task, the performance on the MP3D-Spatial dataset is relatively poor. We believe the reason for this is that MP3D-Spatial is sampled from real navigation environments, which are inherently more complex. For space-based perception tasks, likelihood evalua- Table 13 : Evaluation results on scene text perception.tion yields better results than generation evaluation, especially for LLaVA, mPLUG-Owl, LA-V2, mmGPT, Shikra and Cheetor. This might be attributed to the high demand for spatial reasoning skills for this task, thereby placing a greater emphasis on the image comprehension abilities of visual backbones. Most of these models use ViT-L, which lacks robust spatial semantic understanding. evaluation. However, the performance is not satisfactory under the likelihood evaluation for imagetext selection, we attribute this to the same reason that is mentioned in the analysis of referring expression selection in Appendix C.1.2. Unlike MSCOCO, WikiHow considers the scenarios to match images and abstract instructions, while Winogroud uses negative options with only minor word-level modifications. These pose significant challenges for the models, resulting in a noticeable decrease in accuracy. However, InstructBLIP maintains a lead. Regarding the visual entailment task, apart from the two models based on FlanT5, the performance of the other models is not promising. In summary, we believe that current LVLMs still have relatively weak capabilities in logical reasoning and understanding fine-grained textual details. 19 and Table 20 provides image captioning results of the visual description capability dimension. We choose CIDEr metric to estimate visual description capability while providing BLEU-4, METEOR and ROUGE-L results for additional references. As mentioned in previous work (Xu et al., 2023) , these datasets require concise captions while most LVLMs tend to generate detailed descriptions. Therefore, the performance of most models is not satisfying enough. For this task, PandaGPT always generates a sentence starting with "the image features" and the performance is limited. At the same time, BLIP-2 dominates the task because BLIP-2 is able to provide short captions. To adapt to the development of LVLMs, there is a strong need for a benchmark for evaluating detailed description capabilities.Table 21 provides results of the multi-turn Dialogue task. BLIP-2 and InstructBLIP perform the best in this task while BLIVA takes second place under likelihood evaluation. For VQA-MT, there is no inter-turn dependency, and there is no obvious common relationship between the performance and the number of dialogue turns. However, for VisDial where strong dependencies exist between the proceeding and following questions, LVLMs generally perform worse in multi-turn dialogues where models face more complex contexts. Moreover, the performance of LVLMs deteriorates with the increase in dialogue turns. We hypothesize the reason is that single-turn image-text pairs dominate the training data of LVLMs. Multi-turn data should be incorporated during training to further improve existing LVLMs. Additionally, the advantages of some models like BLIVA and Shikra are only demonstrated with the likelihood evaluation method, this phenomenon has been discussed in Section 4.3.4Table 22 provides the complete results for Section 4.3.3. The models that are not previously mentioned already possess strong abilities to follow single-choice instructions, either through the introduction of language-only instruction-following data during training (ImageBind-LLM) or through the inherent abilities of the frozen backbones like Vicuna (Chiang et al., 2023) questions, however, the in-context samples still assist in addressing issues on the remaining samples.For qualitative analysis, we provide some examples in Figure 11 , the questions are from VQA v2.We also examine the effect of model parameters on different evaluations, and these model parameters are categorized into four types, including number of total parameters (#oP), number of trainable parameters (#oTP), number of trainable LLM parameters (#oLTP), and number of visual tokens (#oVT). According to evaluation results, we consider the score of each model in a single task as an individual sample. These samples are partitioned into multiple groups based on different parameter scales and we generate boxplots for each group, shown in Figure 8 and Figure 9 .For the impact of total parameters on evaluation results, we round the parameters into three groups, including "4B", "7B", and "8B". BLIP-2 and InstructBLIP achieve superior results with fewer parameters, utilizing FlanT5-XL LLM. Except that, in the range from 7B to 8B, the score exhibits an increasing tendency with growth in total parameters.For the effect of trainable parameters, it is observed that trainable parameters range of 100M to 1B may be an ideal scenario. Further, by analyzing different scales of trainable parameters from 0 to 1B, we conclude that: (1) For models with trainable parameters scales of 0-10M, primarily linear layers are trained but still achieve competitive results on these evaluation tasks.(2) For models with 10-100M trainable parameters, all of them implement LoRA or bias-tuning, and these parameterefficient fine-tuning approaches even have negative impacts on generation and likelihood tasks. (3) For models with 100M-1B trainable parameters, Q-Former or Perceiver are trained and applied as connection modules, which enhance the capacity of models across various tasks.In the context of LLM trainable parameters, fine-tuning all the parameters of LLMs or adding an additional small amount of trainable parameters to the LLMs does not contribute to the overall performance of LVLMs. However, adding a sufficient and appropriate number of trainable parameters to the LLM may have the potential to improve models' performances on visual language tasks. With regard to the number of visual tokens shown in Figure 9 , the model's performance shows a decline with a visual token count of 1 and 10, performs optimally at 32 but deteriorates when further increased to 64 or 256, which still outperforms models with only 1 or 10 visual tokens. This indicates that a marginal increase in the number of visual tokens may be beneficial for enhancing model performance. However, the observed outcomes that poor performance exhibited by models with 1-10 VT are attributed to the joint influence of two factors: the use of ImageBind visual encoder and and MiniGPT4) not trained with COCO. For fair evaluation, we choose 6 in-domain tasks (containing images from COCO) and 6 out-domain tasks (not containing images from COCO) across perception and cognition capacity. To mitigate the influence of score fluctuations across tasks, we utilize the average rank within each group as a metric to evaluate their collective performance on particular tasks. The selected tasks and the average rank for each group are listed in Table 23 .To explore the influence of high-quality pre-training data, we select two groups of models: one group (including LLaVA V , MiniGPT4, ImageBind-LLM, and mPLUG-Owl) utilizes data filtered based on rules or CLIP like CC (Sharma et al., 2018) and LAION (Schuhmann et al., 2021) , while the other group (including LA-V2, Shikra, BLIP2, and Lynx) is pre-trained using relatively high-quality data like COCO (Chen et al., 2015) and BlipCap-Filt (Li et al., 2022) as indicated in Table 8 .Instruct-Tuning Dataset Based on the data in Table 8 , we calculate the number of instructiontuning samples for each model and plot the relationship between the number of instruction-tuning samples and the average rank. Table 23 : Average rank of model groups in in-domain and out-domain tasks.Table 24 provides the complete results of models' instability caused by different perturbations. Under the generation evaluation, all models are most sensitive to the order of options, followed by the option marks, and lastly, random instructions. FlanT5 models are the most stable models under the generation evaluation, showing that FlanT5 can well comprehend the multiple-choice questions.For likelihood evaluation, all models are stable since the evaluation strategy directly utilizes the characteristics of generative models.To further perceive the influence of instruction perturbation on the answer accuracy, we analyze the above instruction perturbation results. As we employ different instructions to describe the same task, the accuracy of samples that follow each instruction can be calculated. For the accuracy of each instruction, we adopt the difference between the maximum and minimum accuracies to represent the model's instability level towards the instruction. The results are shown in Table 25 . We discover that all models exhibit some fluctuations in accuracy, illustrating that LVLMs are sensitive to designed prompts. However, the fluctuations in accuracy under generation and likelihood evaluation of most LVLMs are both within an acceptable range. There are still models exhibiting fluctuations in accuracy exceeding 10%, indicating the restricted instruction-following capabilities of LVLMs.In general, LVLMs require further improvements to enhance its ability to understand and follow diverse instructions.Option preference is a phenomenon in our benchmark that when uncertain about the answer, LVLMs prefer a particular option regardless of options' content. We verify the option preference inside the LVLMs in Figure 10 . It has been observed that ImageBind-LLM, Shikra, and BLIVA exhibit a preference for option "A" when confronted with uncertainty. MiniGPT4, mPLUG-Owl, PandaGPT, LA-V2 and mmGPT show a strong preference for option "B". Other LVLMs show no obvious preference in this task. It's worth noting that predicted choice distribution under the likelihood evaluation method has no preference, as all options are considered in an unordered state.The phenomenon of option preference contributes to the instability from random option order but reduces that from random instruction and option mark (as mentioned in Section 4.3.5). Concretely, when LVLMs are uncertain about answers, they select the ceratin option repeatedly for the essentially identical questions. As the option contents have been shuffled in random option mark mode, the LVLMs are regarded as selecting distinct answers. Regarding random instruction and option mark situations, LVLMs are firm in their answers regardless variation of question form. In addition to evaluating with the complete data in ReForm-Eval, we introduce two data selection strategies to construct sub-benchmarks from the complete data. This allows users to assess LVLMs from specific perspectives to meet different evaluation requirements.The first strategy is model-oriented, designed to assess the zero-shot capabilities of models. Specifically, given the models and their corresponding training datasets, we filter out the overlapping parts in ReForm-Eval, retaining only the commonly held-out benchmark datasets for assessment.The second strategy is user-oriented. Users can select one or more of the 61 benchmarks included in ReForm-Eval to construct a specific sub-benchmark based on the dimensions of abilities, tasks, or other requirements. For example, a biologist might select Flowers102 and Pets37 to construct a sub-benchmark for animal and plant recognition.In order to assess the zero-shot capabilities of LVLMs, we follow the first strategy introduced in Appendix D.1 to automatically construct a sub-benchmark based on ReForm-Eval, excluding held-in datasets used in the training processes of the 16 compared models. Specifically, we exclude the following benchmarks: ImageNet, TDIUC, TextVQA, SNLI-VE, GQA, VizWiz, and Flickr30K (used by Lynx); refCOCO (used by Shikra); OK-VQA, VQA v2, OK-VQA, A-OKVQA, A-OKVQRA, A-OKVQAR, VQA-MT, and TextCaps (used by InstructBLIP, Lynx, and BLIVA); COCO image captioning, COCO-ITM, and COCO-ITS that are based on the caption data in MS-COCO (used for pre-training most LVLMs). Benefiting from the richness and ample data in ReForm-Eval, the constructed sub-benchmark still provides sufficient evaluation data for each dimension.Table 26 presents the comprehensive zero-shot performance of LVLMs. compared to results provided in Table 1 , the average ranks of models show little difference and similar trends can be observed.(1) In the black-box generation evaluation, InstructBLIP and BLIP-2 show clear advantage and hold the top-2 positions in most dimensions, followed by Lynx and LLaVA L2 , which also perform well. In the white-box likelihood evaluation, BLIVA excels in many dimensions and the average rank, while InstructBLIP, BLIP-2, and Lynx emerge as strong competitors.(2) Some models exhibit performance differences under the two evaluation strategies, particularly noticeable for LLaVA L2 and BLIVA. We also investigate the phenomenon from the perspective of the models' instruction-following abilities in Appendix D.4. held-in dataset, the average scores of all models have decreased. This overall trend indicates that the commonly held-out datasets tend to be more challenging. For a relative comparison between LVLMs, we find that the average ranks of each model do not change significantly. Models like BLIVA, Lynx, and InstructBLIP have slightly higher ranks with the complete dataset, reflecting the impact of instruct tuning, but the differences are not substantial. The relative ranking trends of the compared models are not significantly affected. In conclusion, we believe that the evaluation results with the complete data and zero-shot evaluation show strong consistency. We attribute this to (1) data in ReForm-Eval has been re-formulated into different formats, weakening the correlation between evaluation data and instruct tuning data; (2) the rich and diverse datasets covered by ReForm-Eval, with a sufficiently large volume of data, preventing the results from being heavily influenced by individual parts. Hence, Reform-Eval benchmark is versatile and fair for evaluating LVLMs.Here we clarify the sampling method used in this paper. Since the Reform-Eval covers 61 benchmarks and contains more than 500,000 evaluation samples, we employ a balanced sampling strategy to ensure the sampled subsets maintain similar characteristics of original benchmarks, thereby enhancing the robustness of the evaluation. Within each benchmark, we perform a balanced sampling based on the distribution of the original data, at a rate of 10% except for three cases: (1) when the size of the original benchmark is less than 1000, we keep the whole benchmark for a stable evaluation;(2) when the original benchmark has more than 10,000 evaluation samples, we filter the data and then conduct the sampling process;(3) for benchmarks used in Multi-turn Dialogue dimension, we retain all evaluation samples as the total sample volume is moderate in this dimension (∼3000).It is worth noting that our calculation method is to first compute the scores of the models on each evaluation benchmark and then average across the benchmarks to obtain the final score, rather than mixing all the evaluation benchmarks together and then computing the overall score on the mixed dataset. Therefore, such sampling methods guarantee that the results on each evaluation benchmark are stable and reliable, leading to relative fairness and balance across all benchmarks. We derive the results of different models on test sub-benchmarks under each sampling ratio, and calculate the correlation coefficient ρ and average absolute deviation d of these results compared to the results on the complete test benchmark.We further analyze whether the shrink or expansion of the dataset size will change the evaluation results. We conduct several experiments on the VQAv2 benchmark and Flowers102 benchmark (the evaluation sample size is 21441 and 818, respectively) in the Coarse-Grained Perception and Visually Grounded Reasoning dimensions. Table 29 demonstrates that the more data sampled, the better the stability of the results, and the more consistent they are with the evaluation on the complete dataset. A 10% sampling ratio can achieve a good balance between evaluation efficiency and consistency.Moreover, for larger datasets, the sampling ratio has little impact on the results; for smaller datasets, the sampling ratio greatly affects the results (see Table 30 ). Therefore, we generally perform balanced distribution sampling for larger datasets and retain the entire dataset for smaller datasets.As we mentioned in Section 3.1, how to efficiently and effectively construct negative options is the key aspect in creating reasonable multiple-choice questions. In ReForm-Eval, different approaches are adopted for classification tasks and open-ended QA tasks. In this section, we elaborate on the rationale behind each construction method for these two types of tasks.As for classification tasks, each dataset provides a fixed set of labels, restricting the output space.In order to maintain consistency with the original formulation, we do not generate new candidate options. Instead, for each question in datasets with an excess of candidate options, we apply hard negative sampling to select N categories most semantically similar to the correct label as distractors.We take Flowers102 as an example to investigate the impact of the N value on the evaluation results.The results are illustrated in Figure 15 . It can be observed that as the number of options increases, the questions become more challenging. N = 4 is a turning point, where the impact of increasing N diminishes. Additionally, the computational cost rises with the increase of N , and the four-option question is the most common format for multiple-choice questions, facilitating comprehension by language backbones. Therefore, we set N to 4.In terms of open-ended QA tasks, it is necessary to automatically generate negative options. We considered two sources of distractors: answers from other questions in the dataset and ChatGPT. In Table 31 , we construct distractors in VisDial through 3 methods: randomly selecting from the answers in the dataset, hard negative sampling from the answers in the dataset based on text similarities, and generating from ChatGPT. We estimate the false-negative rate among the distractors based on human evaluation and measure the impact on evaluation results with the average score of all models. and "HN" are respectively short for false negative and hard negative. "*" indicates that the corresponding experiment is omitted due to the false-negative issues.The results indicate that the main issue with selecting distractors from the answer pool within the dataset is the high likelihood of generating false negatives. For instance, when the correct answer is "yes, it is," "yes" might also be selected as a distractor, especially in the case of hard negative sampling, resulting in a falsenegative rate of 37%. Therefore, we did not evaluate models with those unreliable data. In contrast, distractors generated by ChatGPT are more reasonable, as they are less likely to be false negative and more challenging than randomly selected distractors. Overall, we believe that for open-ended QA tasks, ChatGPT is the most suitable source for distractors. 


Decision: There are divergent ratings after rebuttal. jQTs did not give its final recommendation, but according to the authors' response, the concerns of the data scale and the evaluation of zero-shot capability of the benchmark should have been addressed. Yuod concerned that the reformulation process may require further study, such as the gap between the proposed output formats and the motivations behind them. AC agrees that the proposed reformulation process can be improved, but current contributions have met the bar of ACM MM. Therefore, after considering all reviews, the rebuttal, and the subsequent discussion, the consensus is to accept the paper.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel benchmark called ReForm-Eval for evaluating Large Vision Language Models (LVLMs). The authors present a systematic data collection and reformulation process to convert 61 existing benchmark datasets into unified formats compatible with LVLMs, spanning various capability dimensions from basic visual perception to advanced visual reasoning and dialogue. ReForm-Eval aims to make full use of public resources to provide a large-scale, easy-to-use, universal, and efficient evaluation approach. Additionally, the paper demonstrates the comprehensiveness and reliability of ReForm-Eval through extensive experiments and analyses across various LVLMs.
strength_weakness: strength_weakness: 1. This paper introduces an innovative benchmark, ReForm-Eval, which unifies the evaluation of LVLMs by re-structuring existing benchmarks.
2. This paper presents an automated strategy that reduces the need for manual annotation and offers a fair assessment across models.
limitations: limitations: 1. There is too much data in this benchmark, which will prolong the evaluation process. In addition, the original intention of bmk designation should be to cover as much data distribution as possible on the limited data.
2. Whether the evaluation framework can fully reflect the generalization ability of LVLMs in practical applications remains to be verified, and the indicators and somatosensory are not aligned.
paper_summary: paper_summary: This paper introduces ReForm-Eval, a unified benchmark for comprehensively evaluating Large Vision Language Models (LVLMs). ReForm-Eval re-formulates 61 existing task-oriented datasets into multiple-choice questions or text generation problems that are compatible with the free-form output of LVLMs. The benchmark covers a wide range of capabilities including visual perception, spatial understanding, visually grounded reasoning, cross-modal inference, and multi-turn dialogue. The authors also propose an evaluation framework that considers the input sensitivity and output control of LVLMs. Extensive experiments are conducted to assess various state-of-the-art LVLMs using ReForm-Eval.
strength_weakness: strength_weakness: 1. ReForm-Eval efficiently re-formulates existing datasets into LVLM-compatible formats, enabling the utilization of abundant existing resources for evaluation.

2. The benchmark covers a wide range of capability dimensions and tasks, providing a comprehensive assessment of LVLMs.

3. ReForm-Eval offers a large-scale evaluation resource with over 500,000 samples, surpassing previous benchmarks that rely on manual annotation.

4. The proposed evaluation framework is designed to be reliable and fair, considering factors such as input sensitivity and output control of LVLMs.

5. The paper conducts extensive experiments and analysis using ReForm-Eval, providing valuable insights into the performance and characteristics of various LVLMs.
limitations: limitations: 1. The re-formulation process may introduce noise or bias compared to the original datasets, potentially affecting the evaluation results.
2. The benchmark primarily focuses on vision-language tasks and may not cover all possible multimodal scenarios or capabilities of LVLMs.
3. The evaluation metrics used in the paper, such as accuracy and instability, may not fully capture the nuances and qualitative aspects of LVLM performance.
4. The paper does not provide detailed ablation studies or analysis on the impact of different re-formulation strategies or evaluation design choices.
5. The benchmark and evaluation framework may require updates and extensions in the future to keep pace with the rapid development of LVLMs and emerging task requirements.
6. MMT-Bench[1] is a new multitask benchmark which also repurposes a lot of open-source benchmarks to evaluate the LVLM in a multimodal multitask manner. The author should mention this in the submission. In addition, it is necessary to mention [2] and [3] in the paper.

[1] Ying, Kaining, et al. "MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI." arXiv preprint arXiv:2404.16006 (2024).

[2] Shao, Wenqi, et al. "Tiny lvlm-ehub: Early multimodal experiments with bard." arXiv preprint arXiv:2308.03729 (2023).

[3] Liu, Shuo, et al. "ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Capability for Large Vision-Language Models." arXiv preprint arXiv:2403.20194 (2024).
paper_summary: paper_summary: This paper proposes a unified framework of evaluation on LVLMs, aiming to bridge the gap between the structured output of task-oriented benchmarks and the free-form output by the nature of LVLMs. Multiple benchmarks are mixed and universally evaluated with ReForm-Eval on 26 LVLMs to show the effectiveness.
strength_weakness: strength_weakness: 1. The motivation on bridging the gap between structured output from task-oriented benchmarks and free-form output from LVLMs seems makes sense. It is good if there is a unified framework to 1) not limit the form of the outputs from LVLMs; 2) yet can precisely semantically capture the answers from LVLMs despite the free forms; 3) can be plugged into arbitrary task-oriented benchmarks, i.e., plug-and-play.

2. The presentation is clear and easy-to-read.

3. The examples in Fig. 1 are desired and match the contribution claim.
limitations: limitations: [The motivation is a fancy dream while the method is too naive.] The authors seem just manually classify the current tasks into two output formats: multiple-choice and text generation. And for each, the authors still conduct conventional structuring methods such as in-context learning to enforce LVLMs outputting structured output. In the other words, the only difference from previous works is that the authors mix multiple benchmarks together, arange them with different output structures, and proceed with conventional evaluation formats. I think it is much more interesting if the authors can devise a method that, given the free-form answers from LVLMs and structured labels from benchmarks, ReForm-Eval may semantically determine it is a correct answer or not, instead of changing the outputs of LVLMs depending on what the task is. The current solution is no different compared to just averaging results across multiple task-oriented benchmarks. 

[The experiments do not exhibit the advantage of ReForm-Eval.] Although the authors conduct substantial experiments evaluating more than 20 LVLMs with ReForm-Eval, I do not see any advantages of ReForm-Eval compared to other benchmarks. And the remaining studies such as the effect of in-context examples may be well-studied in some works such as MMICL[1], and not the main advantage of a benchmark paper.

[Some points to consider.] Depending on the interesting motivation proposed by the authors, i.e., bridging the gap between the structured outputs and free-form outputs, I wonder if the structured outputs of LVLMs enforced by methods such as in-context learning compromise the accuracy of LVLMs in current task-oriented benchmarks. In the other words, if the authors may devise a new method to detect the free-form texts reflect the correct answers or not, and outperforms the current conventional structured outputs, there may be a performance gap between the two and the enforced structured outputs of LVLMs may compromise their performance. For instance, enforcing LVLMs outputting the option "(A) yes" instead of some free-form texts such as "Yes, it is" may be harmful? This is an open-ended question and it is interesting to know. 

[Summary] This is an acceptable paper if the authors can add more contributions to the motivation and answers the questions as mentioned above. Overall it is promising yet not ready as an impactful work.

[1] MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning, ICLR2024