Dysca: A Dynamic and Scalable Benchmark for Evaluating Perception Ability of LVLMs

## Abstract

Abstract Currently many benchmarks have been proposed to evaluate the perception ability of the Large Vision-Language Models (LVLMs).
However, most benchmarks conduct questions by selecting images from existing datasets, resulting in the potential data leakage.
Besides, these benchmarks merely focus on evaluating LVLMs on the realistic style images and clean scenarios, leaving the multi-stylized images and noisy scenarios unexplored.
In response to these challenges, we propose a dynamic and scalable benchmark named Dysca for evaluating LVLMs by leveraging synthesis images.
Specifically, we leverage Stable Diffusion and design a rule-based method to dynamically generate novel images, questions and the corresponding answers.
We consider 51 kinds of image styles and evaluate the perception capability in 20 subtasks.
Moreover, we conduct evaluations under 4 scenarios (i.e., Clean, Corruption, Print Attacking and Adversarial Attacking) and 3 question types (i.e., Multi-choices, True-or-false and Free-form). Thanks to the generative paradigm, Dysca serves as a scalable benchmark for easily adding new subtasks and scenarios.
A total of 8 advanced open-source LVLMs with 10 checkpoints are evaluated on Dysca, revealing the drawbacks of current LVLMs.
The benchmark is released in https://github.com/Benchmark-Dysca/Dysca .

## 1 Introduction

Recent years have witnessed the great success of the Large Vision-Language Models (LVLMs) [, , , , , , , , , ].
These models leverage the powerful Large Language Models (LLMs) [, , , , ] as their brain and incorporate the state-of-the-art visual encoders [, , ] as their eyes.
Thanks to the alignment of visual feature with textual space and the development of visual instruction tuning techniques [], LVLMs showcase the impressive capability in terms of visual scene comprehension and multimodal instruction-following.

In order to comprehensively evaluate the capabilities of LVLMs, many benchmarks have been purposed [, , , , , , , , , , ], where we categorize the current benchmarks into three types [].
The first type is the classical benchmarks, such as COCO Caption [] and VQA [, , ]. Although these benchmarks provide high-quality evaluation data, they also have notable limitations.
On the one hand, they are inadequate for measuring the fine-grained capabilities of current LVLMs, offering the limited insightful feedback for the future improvement.
On the other hand, since these classical benchmarks have been available as the open-source test data for a long time, it is hard to prevent the data leakage problem.
The second type of benchmarks evaluate the LVLMs through a subjective manner [, ].
Although the benchmarks reveal the insightful drawbacks of current models, their data scale is limited (i.e., less than 200 annotations) and they require manual evaluation by experts.
The third type is built for objectively evaluating current LVLMs and the comparison between them are shown in Tab. 1.
They provide an objective and automatic evaluation manner, giving the fine-grained evaluation for the LVLMs.
However, these benchmarks conduct Vision-language QAs by selecting images from existing dataset.
Although they claim that the questions are re-annotated, the previous work [] has demonstrated that these benchmarks have unintentionally leaked into the training data of LLMs and LVLMs.
Besides, most benchmarks focus on evaluating LVLMs in the realistic images and clean scenarios, leaving the multi-stylized images and noisy scenarios unexplored.
While some works like MMCBench [] and Typographic Dataset [] have investigated the robustness of LVLMs with corrupted and print-attacked images, respectively, they have not explored the effect of these noisy images on various perceptual tasks.

Figure: Figure 1: Overview of the automatic pipeline for generating Vision-language QAs, cleaning Vision-language QAs and evaluating LVLMs. (a) We first constructs prompts in terms of content, style and background, leveraging the Text-to-Image (T2I) diffusion model (e.g., SDXL []) to synthesis images to be asked. Then based on the scenarios and the question type, we post-process the synthesis images and generate the specific textual questions, respectively. (b) We further filter out low quality Vision-language QAs by utilizing trained models to form the final Dysca. (c) Finally, we evaluate LVLMs on our Dysca and feedback the fine-grained evaluation results.
Refer to caption: x1.png

In this paper, aiming to address these challenges above, we propose Dysca which is a dynamic and scalable benchmark for evaluating the perception ability of LVLMs via various subtasks and scenarios.
Inspired by the prior evaluation works for LLMs [], we investigate on whether we could leverage the large-scale synthesized images for evaluating LVLMs.
We display the overview of our pipeline in Fig. 1.
Specifically, we leverage Stable Diffusion and design a rule-based method to dynamically generate novel images, questions and corresponding answers.
We decouple the prompt into 4 part, i.e., attribute, foreground, style and background, and design pre-defined templates to dynamically generate prompts, as displayed in Fig. 3.
Then we utilize the state-of-the-art text-to-image diffusion models (e.g., SDXL []) to generate the corresponding images.
Since we already know the main information of the images through prompts, we easily generate question-answer textual pairs by the rule-based method.
After that, in order to obtain the high quality Vision-language QAs, we employ CLIP [] to perform data cleaning on the generated Vision-language QA pairs.
Dysca focuses on assessing the fine-grained perception abilities, including recognizing human, animal, object, landmark, etc. Dysca evaluates LVLMs with 20 perceptual subtasks, containing a total number of 51 different artistic styles.
Besides, to evaluate the robustness of the models across different scenarios and question types, we construct 4 testing scenarios (clean, corruption, print attacking and adversarial attacking) and 3 question types (multi-choices, true-or-false and free-form questions).
In the end, Dysca consists of 617K Vision-language QA pairs ($\times$20 larger than MM-BigBench [] and $\times$25 larger than Seed-Bench2 [] as shown in Tab. 1).
Thanks to the generative paradigm, Dysca achieves the scalable benchmark to new subtasks and scenarios and dynamically generate unlimited Vision-language QAs for evaluation.

In summary, our work makes the following key contributions:

- •
Dynamic and Scalable Benchmark: We propose Dysca, a benchmark
that is able to dynamically generate the test data that users need and is easily to scale up to to new subtasks and scenarios.
- •
Multi-grained Perceptual Subtasks and Multi-scenarios: Dysca evaluates the 20 perceptual subtasks performance of 8 mainstream LVLMs with 10 checkpoints under 4 image scenarios (i.e., clean, corruption, print attacking and adversarial attacking) and 3 question types (i.e., multi-choices, true-or-false and free-form questions).
- •
Analysis and Observations: We demonstrate for the first time that evaluating LVLMs using large-scale synthetic data is valid. Experiments show the strong correlation coefficient between our evaluation rankings and the rankings obtained from non-synthetic benchmarks. The evaluation results also reveal the weakness of current LVLMs when facing different question types, image styles and image scenarios.

**Table 1: Comparisons between existing LVLM benchmarks. ’✓ – ’ indicates that the benchmarks include both newly collected images / annotations and images / annotations gathered from existing datasets. ’*’ The scale of our released benchmark is 617K, however Dysca is able to generate unlimited data to be tested.**
| Benchmark | #Evaluation<br>Data Scale | #Perceptual<br>Tasks | Automatic<br>Annotation | Collecting from<br>Existing Datasets | Question<br>Type | Automatic<br>Evaluation |
| --- | --- | --- | --- | --- | --- | --- |
| LLaVA-Bench [] | 0.15K | - | ✗ | ✓<br>– | Free-form | ✓ |
| MME [] | 2.3K | 10 | ✗ | ✓<br>– | True-or-false | ✓ |
| LVLM-eHub [] | - | 3 | ✓ | ✗ | Free-form | ✗ |
| tiny-LVLM-eHub [] | 2.1K | 3 | ✓ | ✗ | Free-form | ✓ |
| SEED-Bench [] | 19K | 8 | ✓<br>– | ✗ | Multi-choices | ✓ |
| MMBench [] | 2.9K | 12 | ✗ | ✓<br>– | Multi-choices | ✓ |
| TouchStone [] | 0.9K | 10 | ✗ | ✓ | Free-form | ✓ |
| REFORM-EVAL [] | 50K | 7 | ✓ | ✗ | Multi-choices | ✓ |
| MM-BigBench [] | 30K | 6 | ✓ | ✗ | Multi-choices | ✓ |
| MM-VET [] | 0.2K | 4 | ✓<br>– | ✓<br>– | Free-form | ✓ |
| MLLM-Bench [] | 0.42K | 7 | ✗ | ✓<br>– | Free-form | ✓ |
| SEED-Bench2 [] | 24K | 10 | ✓<br>– | ✗ | Multi-choices | ✓ |
| BenchLMM [] | 2.4K | 15 | ✗ | ✗ | Free-form | ✓ |
| JourneyDB [] | 5.4K | 2 | ✓ | ✓ | Free-form<br>Multi-choices | ✓ |
| \cdashline1-7<br>Dysca (Ours) | 617K* | 20 | ✓ | ✓ | Free-form<br>Multi-choices<br>True-or-false | ✓ |

## 2 Related Works

### 2.1 Large Vision-Language Models

The landscape of Large Vision-Language Models (LVLMs) has been significantly shaped by the pioneering success of Large Language Models (LLMs) such as GPTs [, , ] and LLaMA [], catalyzing advancements in multimodal content understanding and generation [], including intricate tasks like image-text comprehension.
At the forefront of these developments, BLIP-2 [] introduces a lightweight Q-Former [] that facilitates alignment between textual and visual representations through a cross-attention mechanism []. InstructBLIP [] takes a step further by incorporating textual instructions into the Q-Former, which significantly improves performance. LLAVA [] employs GPT-4 [] to transform data into multimodal instruction-following data and uses CLIP [] and LLAMA [] for fine-tuning instructions, achieving advanced performance. LLAVA-1.5 [] extends this paradigm by integrating MLP projection and introducing academic task-specific Vision-language QA data. Recently, models like Otter [], MiniGPT-4 [], Qwen-VL-Chat [] and XComposer-VL [] further unleash the cross-modal understanding capabilities of LVLMs.

### 2.2 Benchmarks for LVLMs

The great progress of LVLMs triggers the development of benchmarks for evaluating these models, where we divide previous benchmarks into three categories.
The first type is the classical benchmarks which focuses on evaluating LVLMs abilities via image caption [] and VQA [, ].
However, these benchmarks cannot provide the fine-grained feedback on how to improve the models. Besides, since these benchmarks have been the public resources for a long time, it is hardly to guarantee that the LVLMs have not use them for training.
The second type subjectively evaluates LVLMs by experts [, ]. Although these benchmarks reveal the insightful feedback of the LVLMs, their scale is limited (i.e., less than 200 annotations). The subjective manner also makes the evaluation expensive and hardly to expand the scale of the benchmarks.

The third type [, , , , , , , , , , , , , , ] focuses on evaluating LVLMs in an objective and large-scaled manner, where we list the detailed information of them in the Tab. 1. Some of them have been adopted by the community [] as the standard benchmarks for evaluating LVLMs [, , ], like MME [] and MMBench [].
These benchmarks evaluate models through the objective answer types and most of them leverage the automatic annotation and evaluation manner for revealing the fine-grained drawbacks of current LVLMs.
However, the previous benchmarks primarily concentrate on evaluating LVLMs using realistic images and clean scenario, leaving multi-stylized images and noisy scenarios unexplored. Moreover, many of them conduct QA by selecting images from publicly available datasets (e.g., [, ]). While they state that the questions have been re-annotated, they cannot guarantee that the LVLMs have not seen the image during training stage. The previous work [] has proved that these benchmarks have unintentionally leaked into the training data of LLMs and LVLMs. One possible way to solve data leakage is using novel but synthesis images, where JourneyDB [] is the first work aiming to leverage synthesis images to evaluate current LVLMs. The prompts and the corresponding images are downloaded from Midjourney [] and ChatGPT [] is leveraged to label the images. However, JourneyDB is a top-down framework where the number of images is fixed. Besides, the ChatGPT labeling may cause hallucinate annotations, leading to the unreliable evaluation results. Although 40 annotators have involved to clean the data, the data cleaning cost are expensive and it limits the data scale. In contrast, our Dysca serves as the bottom-up framework, allowing for dynamic and scalable generation for both images and evaluation questions. The rule-based question generation method also makes the annotations more accuracy. Besides, Dysca contains 20 subtasks which is more comprehensive than JourneyDB.

## 3 Dysca

### 3.1 Overview of Our Pipeline

The overview of our pipeline is shown in Fig. 1, containing data generation, data cleaning and LVLMs evaluation.
For the data generation, our Dysca benchmark consists of four dimensions, i.e., $(M,P,I,Q)$, where $M$ means "Metadata", $P$ means "Prompt", $I$ means "Image" and $Q$ means "Question-answer pair".
We further decouple the metadata $M$ into 4 parts, i.e., "style", "attribute", "foreground" and "background", and the combination of the four parts constitute the image prompts $P$.
Then, given the prompt $P$ and the selected scenario, we leverage the Text-to-Image (T2I) diffusion model (e.g., SDXL []) to synthesis image $I$ and add the specific perturbation to the image $I$.
After that, since the prompt already includes the question angle and the corresponding answer, we construct a rule-based approach to generate the $Q$. Three types of questions are considered, i.e., multi-choice, true-or-false and free-form. Multi-choice and true-or-false questions utilize a closed-ended manner to assess LVLMs, while free-form questions employ an open-ended manner through image captioning for evaluation. For the data cleaning, considering that the T2I diffusion model may generate unsuccessful outcomes, we then use CLIP [] and PP-OCRv3 [] to automatically clean the whole dataset to obtain the final Dysca. Finally, we evaluate 8 open-source LVLMs with 10 checkpoints on our proposed Dysca.

Figure: Table 2: Key statistics of Dysca.
Refer to caption: x2.png

### 3.2 Perceptual Tasks

Evaluation dimensions. Perception is one of the most fundamental capabilities of LVLMs and previous works [] have shown that the lack of perceptual ability may result in hallucination []. In order to comprehensively evaluate LVLMs’ perception capability, we design 20 perceptual subtasks where we show all subtasks and the corresponding amount of their annotation in the Fig. 2. We investigate on two types of perception dimensions, i.e., coarse-grained and fine-grained perception. Coarse-grained perception involves recognizing the style, background and color of images. Fine-grained perception involves recognizing the animal, object, plant, food, age, gender, expression, race, celebrity, action, text, clothes, movie, anime, landmark, profession and TV shows.

Data sources. For each perceptual subtask, we collect the textual data first to construct the metadata $M$. For the TV shows, Anime and Movie, we select the titles from the rating list of IMDb(^*^**https://www.imdb.com/) based on the number of user reviews. For the styles, we utilize the style lists collected from the community(^†^††https://stable-diffusion-art.com/sdxl-styles/) and remove those which have strong reflect on the image content like "architectural style" and "Pokemon style". Note that the style list does not include the style prompt associated with a particular artist’s name. Besides, for the remaining contents, we select them from the label of current dataset (e.g., ImageNet []). All the selected textual data above constitute the metadata $M$. We provide the detailed information of the metadata in the Appendix C.

### 3.3 Construction of Questions & Answers

Recall that the data generation for Dysca benchmark consists of four dimensions, i.e., $(M,P,I,Q)$, denoting the metadata ($M$), prompt ($P$), image ($I$) and question-answer pairs ($Q$), respectively. The relationships between these parts and the process of constructing Dysca are shown in Fig. 3. The metadata M is the core of the whole Dysca, containing all the information for generating $P$, $I$ and $Q$. The metadata $M$ consists of foreground, attribute, background and style, and these information guide the generation of the prompt ($P$) through pre-designed templates. Then, we utilize the T2I diffusion model to generate the corresponding image using the prompt $P$. For generating the image with specific text on it for the OCR subtask, we leverage TextDiffusion2 [], which is the state-of-the-art text rendering method. For the rest of images, we leverage Stable Diffusion XL []. Subsequently, based on the different question types we select, i.e., multi-choices, true-or-false and free-form, we generate the corresponding Vision-language QA pairs in Dysca.

Besides, in order to evaluate the model performance under various scenarios, we conduct experiments on 4 scenarios, i.e., clean, corruption, print attacking and adversarial attacking.
For the print attacking, followed by [], we add the deceptive text on the image, where the text is a wrong option.
Besides, to comprehensively evaluate the performance of LVLMs under corruption scenario, we add more typographic factors to original settings (i.e., different font orientations and font positions).
For the adversarial attacking, we leverage PGD [] to generate the adversarial image.
We use InstructBLIP [] as the proxy model and regard others as the black box models.
The reason why we choose InstructBLIP is that it has shown superior performance in clean scenario.
Besides, the black-box setting better reflects the robustness of the models when they face the real-world adversarial attacks.
For the corruption, we leverage the image corruption methods collected from [].
We remove some hard corruptions as they significantly impact the quality of the image, leading to human failure in judging the style and content of the image. The detailed examples are shown in Appendix D.

Consider that the Text-to-image diffusion model may generate the failure cases that affect the quality of the proposed benchmark, we leverage the off-the-shelf models, i.e., PP-OCRv3 [] and CLIP-L-14 [], to clean the data.
PP-OCRv3 [] is leveraged as the filter to exclude the failure image that TextDiffusion2 [] generates the wrong text on the image.
For the other images, we use CLIP-L-14 [] to filter out the images with low text-image consistency. In the end, we filter out nearly 15% of low quality samples. The final statistics of our released Dysca are shown in Tab. 2. Note that the OCR subtask does not involve print attacking scenario as misidentifying adversarial text does not indicate poor OCR robustness of the LVLMs. Therefore, there are 7K fewer questions in the print attacking scenario. Besides, for the free-form question type, since it allows to assess the model’s perception abilities across multiple subtasks at the same time, we reduce the number of free-form questions for achieving a balanced data distribution.

Figure: Figure 3: The process of generating the prompt (P), image (I) and question-answer pairs (Q) from the metadata (M).
Refer to caption: x3.png

### 3.4 Evaluation Strategy

Instruction Design. We design two types of instructions to improve the instruction-following result of LVLMs. For the multi-choices and true-or-false questions, we design the questions followed by the description "Please answer the question and provide the correct option letter, e.g., (A), (B), (C), (D), at the end. Do not contain the analysis progress. Your answer is: ". For the free-form questions, recalling that the prompt $P$ contains four part, i.e., the style, attribute, foreground and background, we instruct the model to caption these four dimensions by "Please describe the image. You can describe it from these aspects: $\{\}$", where "$\{\}$" includes the specific template we design for each part. We display the sample in the Fig. 3 and more examples can be found in the Appendix E.

**Table 3: Evaluation results on the 20 perceptual subtasks. The top two results on each subtask are bolded and underlined, respectively. "MC" and "TF" indicate the accuracy (%) of "Multi-choices" and "True-or-false", respectively.**
|  |  | Movie | Action | TV Show | Profession | Landmark |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Language Model | MC | TF | MC | TF | MC | TF | MC | TF | MC | TF |
| Blip2 [] | Flan-T5-XL | 71.46 | 68.61 | 96.24 | 93.04 | 56.29 | 61.99 | 77.88 | 75.87 | 98.70 | 94.58 |
| InstructBLIP [] | Flan-T5-XL | 77.35 | 67.38 | 97.28 | 93.61 | 65.89 | 60.96 | 79.17 | 76.51 | 98.04 | 95.75 |
| XComposer-VL [] | InternLM-7B | 81.90 | 78.36 | 97.03 | 94.75 | 77.81 | 73.29 | 83.97 | 75.87 | 98.04 | 94.81 |
| LLava-1.5 [] | Vicuna-7B | 55.95 | 51.68 | 74.78 | 50.75 | 56.29 | 50.68 | 58.65 | 56.83 | 85.00 | 54.48 |
| LLava-1.5 [] | Vicuna-13B | 66.29 | 54.04 | 77.93 | 59.71 | 59.27 | 54.11 | 66.67 | 59.05 | 94.35 | 55.90 |
| MiniGPT-4 [] | Vicuna-7B | 32.68 | 52.35 | 42.96 | 51.00 | 30.46 | 50.00 | 33.97 | 50.16 | 53.26 | 52.36 |
| Otter [] | LLaMA-7B | 65.36 | 59.08 | 65.85 | 72.36 | 68.54 | 57.53 | 66.99 | 54.92 | 58.48 | 66.51 |
| Qwen-VL-Chat [] | Qwen-7B | 71.25 | 51.01 | 96.35 | 63.14 | 67.22 | 49.32 | 75.64 | 54.60 | 96.96 | 54.72 |
| Shikra [] | Vicuna-7B | 66.29 | 59.64 | 78.00 | 77.29 | 60.26 | 56.85 | 78.85 | 68.25 | 89.35 | 70.99 |
| Shikra-VQA [] | Vicuna-7B | 66.39 | 61.66 | 96.17 | 80.32 | 59.93 | 60.27 | 78.21 | 67.94 | 92.83 | 72.88 |
|  |  | Anime | Clothes | Celebrity | Food | Plant |  |  |  |  |  |
| Model | Language Model | MC | TF | MC | TF | MC | TF | MC | TF | MC | TF |
| Blip2 [] | Flan-T5-XL | 57.07 | 61.94 | 82.38 | 73.79 | 81.98 | 76.48 | 91.52 | 88.04 | 92.50 | 91.19 |
| InstructBLIP [] | Flan-T5-XL | 61.28 | 64.04 | 86.51 | 81.08 | 83.69 | 76.14 | 91.71 | 88.85 | 93.31 | 91.92 |
| XComposer-VL [] | InternLM-7B | 75.41 | 74.58 | 86.18 | 85.73 | 88.53 | 87.28 | 92.33 | 90.21 | 93.25 | 93.14 |
| LLava-1.5 [] | Vicuna-7B | 47.15 | 48.46 | 47.19 | 50.41 | 57.36 | 54.72 | 54.63 | 51.10 | 49.08 | 48.59 |
| LLava-1.5 [] | Vicuna-13B | 57.61 | 50.14 | 65.20 | 57.20 | 60.14 | 57.94 | 78.87 | 57.45 | 79.72 | 55.51 |
| MiniGPT-4 [] | Vicuna-7B | 29.21 | 48.74 | 31.25 | 50.47 | 28.62 | 49.64 | 44.66 | 50.78 | 45.38 | 50.53 |
| Otter [] | LLaMA-7B | 61.82 | 57.30 | 47.13 | 69.52 | 42.41 | 63.52 | 46.81 | 79.62 | 66.41 | 81.51 |
| Qwen-VL-Chat [] | Qwen-7B | 71.60 | 54.63 | 78.44 | 60.72 | 87.37 | 50.05 | 89.37 | 53.37 | 92.32 | 59.91 |
| Shikra [] | Vicuna-7B | 47.96 | 57.72 | 75.21 | 59.65 | 63.76 | 59.86 | 83.30 | 68.37 | 88.55 | 66.60 |
| Shikra-VQA [] | Vicuna-7B | 49.05 | 57.58 | 76.57 | 63.10 | 63.73 | 62.13 | 89.25 | 71.22 | 88.34 | 71.76 |
|  |  | Age | Gender | Expression | Race | Animal |  |  |  |  |  |
| Model | Language Model | MC | TF | MC | TF | MC | TF | MC | TF | MC | TF |
| Blip2 [] | Flan-T5-XL | 62.61 | 59.65 | 99.37 | 94.86 | 89.27 | 75.92 | 74.38 | 71.95 | 96.64 | 95.27 |
| InstructBLIP [] | Flan-T5-XL | 65.14 | 59.55 | 99.51 | 88.21 | 91.85 | 79.34 | 77.99 | 74.62 | 97.21 | 94.98 |
| XComposer-VL [] | InternLM-7B | 67.89 | 78.35 | 99.60 | 98.06 | 89.64 | 82.52 | 80.42 | 76.69 | 97.83 | 96.70 |
| LLava-1.5 [] | Vicuna-7B | 38.35 | 55.15 | 54.14 | 49.78 | 63.86 | 49.76 | 43.98 | 50.56 | 49.26 | 50.90 |
| LLava-1.5 [] | Vicuna-13B | 49.55 | 59.59 | 98.71 | 83.53 | 71.29 | 58.33 | 70.16 | 62.84 | 85.99 | 58.23 |
| MiniGPT-4 [] | Vicuna-7B | 31.75 | 51.71 | 56.49 | 49.22 | 42.01 | 50.91 | 27.54 | 50.41 | 45.97 | 49.57 |
| Otter [] | LLaMA-7B | 37.98 | 51.19 | 78.23 | 77.96 | 73.86 | 60.36 | 43.00 | 57.88 | 81.32 | 83.63 |
| Qwen-VL-Chat [] | Qwen-7B | 53.48 | 49.09 | 97.72 | 58.55 | 85.22 | 64.30 | 73.50 | 56.88 | 95.60 | 64.37 |
| Shikra [] | Vicuna-7B | 65.37 | 56.71 | 97.85 | 73.16 | 90.47 | 70.55 | 74.17 | 54.88 | 89.82 | 69.06 |
| Shikra-VQA [] | Vicuna-7B | 65.50 | 57.34 | 99.39 | 81.41 | 92.36 | 74.71 | 73.90 | 55.24 | 91.18 | 74.26 |
|  |  | Object | OCR | Style | Background | Color |  |  |  |  |  |
| Model | Language Model | MC | TF | MC | TF | MC | TF | MC | TF | MC | TF |
| Blip2 [] | Flan-T5-XL | 89.32 | 87.75 | 71.89 | 62.07 | 99.86 | 88.62 | 63.97 | 68.37 | 88.01 | 85.55 |
| InstructBLIP [] | Flan-T5-XL | 89.87 | 90.13 | 72.61 | 60.93 | 99.96 | 81.23 | 66.50 | 69.19 | 90.93 | 85.55 |
| XComposer-VL [] | InternLM-7B | 90.23 | 91.72 | 69.22 | 78.66 | 95.83 | 81.32 | 71.26 | 75.27 | 87.97 | 89.02 |
| LLava-1.5 [] | Vicuna-7B | 58.63 | 49.99 | 47.08 | 51.02 | 46.35 | 50.49 | 44.33 | 50.95 | 41.91 | 51.90 |
| LLava-1.5 [] | Vicuna-13B | 77.86 | 57.93 | 58.19 | 53.17 | 85.80 | 51.56 | 62.67 | 55.33 | 63.81 | 52.90 |
| MiniGPT-4 [] | Vicuna-7B | 51.75 | 51.55 | 31.37 | 51.56 | 43.41 | 48.64 | 31.94 | 50.10 | 35.29 | 50.51 |
| Otter [] | LLaMA-7B | 47.31 | 82.51 | 66.23 | 61.41 | 98.04 | 61.58 | 63.06 | 63.40 | 48.28 | 57.97 |
| Qwen-VL-Chat [] | Qwen-7B | 88.32 | 63.14 | 71.74 | 51.86 | 98.21 | 52.43 | 69.42 | 51.59 | 86.39 | 53.05 |
| Shikra [] | Vicuna-7B | 70.91 | 69.22 | 60.79 | 53.63 | 95.13 | 58.95 | 71.24 | 62.66 | 84.91 | 61.83 |
| Shikra-VQA [] | Vicuna-7B | 89.43 | 76.00 | 59.63 | 54.14 | 99.51 | 61.48 | 70.98 | 65.96 | 83.79 | 64.49 |

Evaluation Metric. For the multi-choices and true-or-false questions, we use accuracy as the evaluation metric. We randomly shuffle the order of choices to prevent evaluation results from being influenced by the model’s tendency towards specific choices []. The random accuracy of the two types are equal to 25$\%$ and 50$\%$, respectively. We use regular expressions to extract the model’s answer choices. For cases where the extraction is fail, we calculate the Levenshtein distance between the answer string and the choice string, and select the option with the minimum distance as the model’s answer. For the free-form questions, we test the model’s image caption capability where the ground truth is the prompt of the image. Followed by [], we use SentenceTransformer [] to compute the text similarity with prompt $P$ and the caption output of the LVLMs. The final score of each question type is the average score of subtasks.

## 4 Results and Analysis

In this section, we report the evaluation results and make insightful analysis. A total of 8 LVLMs with 10 checkpoints are evaluated on Dysca benchmark, including BLIP2 [], InstructBLIP [], LLavA [], MiniGPT-4 [], Otter [], XComposer-VL [], Qwen-VL-chat [] and Shikra []. Each model is evaluated with all the 20 perception subtasks under 4 scenarios. The detailed rankings for each subtask can be found in the Appendix A.

### 4.1 Main Results

Clean Scenario. The evaluation results of various LVLMs in different perceptual subtasks under clean scenarios are presented in Tab. 3. Since the evaluation for free-form question type usually involves multiple subtasks, we can not calculate the results of free-form for each subtask individually. Instead, we display the overall score of free-form in the first row in Tab. 4. As can be seen, Xcomposer-VL [] outperforms other LVLMs, achieving top-1 or top-2 results in most subtasks, but InstructBLIP [], Qwen-VL-chat [] and BLIP2 also take a lead in a few subtasks.

Noisy Scenarios The evaluation results of various LVLMs under noisy scenarios (i.e., corruption, print attacking and adversarial attacking) are presented in Tab. 4. As can be seen, For the multi-choices and true-or-false question type, Xcomposer-VL [] still takes a lead on all 4 scenarios. For the free-form, LLava-1.5-7b [] achieves the best.

**Table 4: Evaluation results on 4 scenarios. "MC", "TF" and "FF" indicates "Multi-choices", "True-or-false" and "Free-form", respectively. "PrintAtt" and "AdverAtt" means "Print Attacking" and "Adversarial Attacking", respectively. "*": the model is under white-box setting.**
| Scenarios | Blip2 [] | InstructBLIP [] | XComposer-VL [] | LLava-1.5-13b [] | LLava-1.5-7b [] |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MC | TF | FF | MC | TF | FF | MC | TF | FF | MC | TF | FF | MC | TF | FF |  |
| Clean | 82.07 | 78.78 | 37.56 | 84.29 | 79.00 | 38.94 | 86.22 | 84.82 | 46.03 | 71.50 | 57.72 | 48.61 | 53.70 | 51.41 | 50.15 |
| Corruption | 81.17 | 77.98 | 37.32 | 83.77 | 78.57 | 38.72 | 85.20 | 84.16 | 45.52 | 70.34 | 57.57 | 48.28 | 53.21 | 51.40 | 49.54 |
| PrintAtt | 64.78 | 68.01 | 35.82 | 59.18 | 58.80 | 35.71 | 74.08 | 72.71 | 44.46 | 48.91 | 54.72 | 48.47 | 40.02 | 50.94 | 48.38 |
| AdverAtt | 23.28 | 48.98 | 25.70 | 21.49* | 48.74* | 29.15* | 20.85 | 49.54 | 21.17 | 66.86 | 56.56 | 47.11 | 50.70 | 51.36 | 48.17 |
| Scenarios | MiniGPT-4 [] | Otter [] | Qwen-VL-Chat [] | Shikra [] | Shikra-VQA [] |  |  |  |  |  |  |  |  |  |  |
| MC | TF | FF | MC | TF | FF | MC | TF | FF | MC | TF | FF | MC | TF | FF |  |
| Clean | 38.50 | 50.51 | 38.90 | 61.36 | 65.99 | 39.12 | 82.31 | 55.84 | 49.90 | 76.61 | 63.79 | 47.17 | 79.31 | 66.69 | 29.93 |
| Corruption | 38.70 | 49.99 | 40.35 | 62.19 | 65.36 | 38.71 | 79.26 | 52.74 | 49.74 | 76.42 | 64.31 | 46.43 | 79.06 | 66.95 | 29.40 |
| PrintAtt | 36.16 | 50.00 | 42.45 | 38.11 | 36.27 | 36.93 | 59.68 | 46.40 | 48.65 | 60.02 | 40.18 | 47.03 | 62.20 | 41.05 | 27.91 |
| AdverAtt | 26.75 | 49.52 | 22.59 | 57.78 | 62.27 | 35.76 | 78.47 | 58.04 | 44.67 | 72.20 | 62.16 | 43.56 | 74.04 | 64.64 | 26.81 |

### 4.2 Analysis

#### 4.2.1 Key Observations

(1) For individual models, their perceptual performance varies across different subtasks. For example, in the case of Qwen-VL-Chat [], it achieves a score of 96.96% accuracy in landmark recognition task for multi-choices questions (2% below the first-place score), but obtains a score of 53.48% accuracy in age recognition task (12% below the first-place score). The results suggest that Qwen-VL-Chat [] may require more fine-tuning in age perception data. Analyzing the performance of the models across various subtasks contribute to purposive improving.

Figure: Figure 4: Models exhibit different performance when facing the same image but different question types.
Refer to caption: x4.png

(2) Models exhibit performance inconsistency when facing multi-choices and true-or-false question types. As can be seen, in the object recognition subtask, Otter [] achieves an accuracy of 47.31% in the multi-choices question type (22% higher than random guessing), but obtains an accuracy of 82.51% in the true-or-false question type (32% higher than random guessing). Interestingly, we observe the opposite results in the Qwen-VL-Chat []. In the object recognition subtask, it achieves an accuracy of 88.32% in the multi-choices but achieves an accuracy of 63.14% in the true-of-false. We also observe the same problem shown in other models and we display two examples in Fig. 4. We speculate that the inconsistency may be attributed to the bias in the training dataset towards particular question types, such as using more multi-choices or true-or-false questions.

(3) Each model exhibits robustness in the corruption scenario, but suffers degradation in the two attacking scenarios. As shown, all models exhibit minor score variation of less than 1% under the corruption scenario. However, they exhibit degradation when facing print attacking (e.g., 84.29% vs. 59.18% for InstructBlip [] in multi-choices accuracy). XComposer-VL [] shows the strongest robustness, maintaining over 70% accuracy for both multi-choices and true-or-false. Besides, since our adversarial algorithm specifically targets the image encoder, the LVLMs that share the same encoder architecture (i.e., Blip2, InstructBLip and XComposer-VL all using EVA-CLIP [] as the image encoder) exhibit significant performance degradation, with accuracy even falling below random selection. Models utilizing alternative image encoders also experience a performance decrease of approximately 5% to 10%. A more detailed result can be found in Appendix D.

#### 4.2.2 The Validity of Dysca

**Table 5: The correlation results on three benchmarks, where $\rho\in[-1,1]$ and $\tau\in[-1,1]$.**
| Style | Method | MMBench | OCRBench | SeedBench-2 |
| --- | --- | --- | --- | --- |
| All | $\rho$ | 0.70 | 0.90 | 0.46 |
| $\tau$ | 0.60 | 0.80 | 0.43 |  |
| Realistic | $\rho$ | 0.70 | 1.00 | 0.64 |
| $\tau$ | 0.60 | 1.00 | 0.62 |  |

In this section, we investigate on the evaluation gap between Dysca and non-synthesis benchmarks. We calculate the Spearman’s rank correlation coefficient [] $\rho$ and the Kendall rank correlation coefficient [] $\tau$ between the evaluation ranking of Dysca under clean scenario with the non-synthesis benchmark’s evaluation ranking, i.e., MMBench [], OCRBench [] and SeedBench-2 []. Both coefficient generate a score in the range of [-1,1], where 1 represents a perfect positive correlation, -1 represents a perfect negative correlation, and 0 represents no correlation. Specifically, we intersect our Dysca with current benchmarks based on the perceptual subtasks, evaluation models and evaluation question types. We then calculate the correlation of model evaluation rankings within this intersection. The results are shown in the first row of Tab. 5. For the MMbench [] and OCRBench [], both metrics show the high correlation, with $\rho$ and $\tau$ higher than 0.6. However, the correlation for SeedBench-2 [] is not as strong. Considering that SeedBench-2 only contains realistic images, we conduct additional experiments using the evaluation ranks on our realistic style images only. As shown in the second row of Tab. 5, the correlation results of SeedBench-2 significantly improve (i.e., 0.46 vs. 0.64 for $\rho$ and 0.43 vs. 0.62 for $\tau$). The correlation with OCRBench also improves to 1, demonstrating the validity of using synthetic datasets for evaluation LVLMs.

To further explore the the impact of image styles on evaluation results, we present the average scores across all subtasks for each of the 51 styles in Fig. 5.
We observe slight score differences across all styles. In the case of realistic styles such as "iPhone photo", all LVLMs perform better compared to other image styles. The LVLMs also exhibit better performance on unrealistic but common styles like "expressionist". However, for unrealistic and less common styles such as "gothic", all models show relatively poor performance. The results reveal that the gap between Dysca and non-synthesis benchmarks primarily stems from the more diverse range of image styles, making Dysca a more comprehensive benchmark for assessing the perception ability compared to previous benchmarks.

Figure: Figure 5: Illustration of each model’s performance across 51 image styles, where the darker colors represent higher scores. The representative styles are colored with non-black font. Realistic styles are shown in red font, unrealistic but common styles are displayed in yellow font, and unrealistic and less common styles are represented in blue font.
Refer to caption: x5.png

## 5 Conclusion

In this paper, we purpose Dysca, a dynamic and scalable benchmark for evaluating perception ability of Large Vision Language Models (LVLMs). Dysca consists of 617K Vision-language QA pairs, covering 20 perceptual subtasks, 4 image scenarios and 3 question types. We conduct the experiment on 8 advanced open-source LVLMs with 10 checkpoints, revealing the insightful weakness of current LVLMs when facing different question types, image styles and image conditions. Experiments demonstrate the validity on evaluating LVLMs by using synthesis images.

## Acknowledgement

This work is partially supported by National Key R$\&$D
Program of China (No. 2021YFC3310100), Strategic Priority Research Program of the Chinese Academy of Sciences (No. XDB0680000),
Beijing Nova Program (20230484368), Suzhou Frontier
Technology Research Project (No. SYG202325),
and Youth Innovation Promotion Association CAS.

## Appendix

## Appendix A The Leaderboards

**(a) Clean-Movie**
| Rank | Model | Score |
| --- | --- | --- |
| 1 | XComposer-VL | 80.13 |
| 2 | InstructBlip | 72.36 |
| 3 | Blip2 | 70.03 |
| 4 | Shikra-VQA | 64.03 |
| 5 | Shikra | 62.97 |
| 6 | Otter | 62.22 |
| 7 | Qwen-VL-Chat | 61.13 |
| 8 | LLAVA-1.5-13B | 60.17 |
| 9 | LLAVA-1.5-7B | 53.81 |
| 10 | MiniGPT-4 | 42.52 |

**(a) Corru.-Movie**
| Rank | Model | Score |
| --- | --- | --- |
| 1 | XComposer-VL | 78.20 |
| 2 | InstructBlip | 71.30 |
| 3 | Blip2 | 68.69 |
| 4 | Shikra-VQA | 64.19 |
| 5 | Shikra | 63.47 |
| 6 | Otter | 61.39 |
| 7 | Qwen-VL | 59.27 |
| 8 | LLAVA-1.5-13B | 58.67 |
| 9 | LLAVA-1.5-7B | 52.36 |
| 10 | MiniGPT-4 | 43.02 |

**(a) Pr.Att.-Movie**
| Rank | Model | Score |
| --- | --- | --- |
| 1 | XComposer-VL | 67.09 |
| 2 | Blip2 | 52.55 |
| 3 | LLAVA-1.5-13B | 46.74 |
| 4 | LLAVA-1.5-7B | 45.24 |
| 5 | Shikra | 44.84 |
| 6 | Shikra-VQA | 44.36 |
| 7 | Qwen-VL | 44.30 |
| 8 | InstructBlip | 43.06 |
| 9 | MiniGPT-4 | 40.26 |
| 10 | Otter | 37.97 |

**(a) Ad.Att.-Movie**
| Rank | Model | Score |
| --- | --- | --- |
| 1 | Qwen-VL | 58.17 |
| 2 | Shikra | 57.41 |
| 3 | Shikra-VQA | 57.26 |
| 4 | LLAVA-1.5-13B | 57.17 |
| 5 | Otter | 56.61 |
| 6 | LLAVA-1.5-7B | 51.91 |
| 7 | MiniGPT-4 | 36.95 |
| 8 | Blip2 | 33.33 |
| 9 | InstructBlip | 31.97 |
| 10 | XComposer-VL | 31.75 |

**(a) Clean**
| Rank | Model | Score |
| --- | --- | --- |
| 1 | XComposer-VL | 72.37 |
| 2 | InstructBlip | 67.41 |
| 3 | Blip2 | 66.13 |
| 4 | Qwen-VL-Chat | 62.68 |
| 5 | Shikra | 62.52 |
| 6 | LLAVA-1.5-13B | 59.28 |
| 7 | Shikra-VQA | 58.64 |
| 8 | Otter | 55.49 |
| 9 | LLAVA-1.5-7B | 51.75 |
| 10 | MiniGPT-4 | 42.64 |

The model performance leaderboards for each subtask under each scenario are shown in Tab. 6, Tab. 7, Tab. 8, and Tab. 9. We compute the average of the multi-choice and true-or-false evaluation results under each subtask. Since the free-form question allows to assess the model’s perception abilities across multiple subtasks at the same time, the results of free-form are not taken into account.

The overall performances on each scenario are displayed in Tab. 10. We calculate the average of the scores of the three questioning types (i.e., multi-choice, true-or-false and free-form) as the final score.

## Appendix B Discussion

### B.1 General Discussion

Limitation. Dysca is the dynamic and scalable benchmark, offering evaluation for 20 perceptual subtasks under 51 image styles and 4 scenarios. However, generating data for evaluating cognition abilities (e.g., commonsense reasoning) presents challenge within the existing framework. This limitation arises from the reliance on predefined rules for prompt and question generation, which may not adequately capture the complexity of cognitive-level questions.

Synthesis Data for Training / Fine-tuning. The use of synthetic data for model training / fine-tuning has been adopted in the field of Natural Language Processing (NLP) []. In this work, we do not explore the possibility of utilizing our benchmark for model training. Our primary goal in this paper is to provide a large-scale evaluation benchmark that addresses the issue of data leakage in current multimodal evaluation benchmarks and offers evaluation results across multiple subtasks, scenarios, question types and styles. Nevertheless, considering that Dysca has the capability to synthesize high-resolution and unlimited amounts of annotated multimodal data, we believe that Dysca also holds potential as a training data synthesis tool for LVLMs.

Reproducibility and Licence. All the experiments are built on 8 * RTX 4090. All the data and the code for generation and evaluation are released in [https://github.com/Benchmark-Dysca/Dysca](https://github.com/Benchmark-Dysca/Dysca). The licence of Dysca is ["CreativeML Open RAIL++-M"](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/LICENSE.md), which follows the licence set by the Stable Diffusion XL.

Ethical Concerns. Our Dysca leverages the Stable Diffusion XL [] to generate images. In order to prevent the model generating unsafe images, e.g., NSFW and offensive images, lots of efforts have been made. First, we use the safety checker [] to post filter the unsafe images. With the unsafe image that is recognized by the safety checker, the model’s output will be a blank image. Besides, we manually exclude the specific styles or the word that may trigger the unsafe images generation from the Metadata $M$. We believe that our Dysca involves fewer ethical concerns.

### B.2 The Stability of Dysca

Figure: Figure 6: The tendency of model’s overall performance under clean scenario with different scale of evaluation data.
Refer to caption: x6.png

In this section, we focus on examining the stability of Dysca. We partition Dysca into 11 different scales: 1%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90% and 100%. We compute the evaluation scores using each of these data scales. The score is calculated as the sum of scores obtained from multiple-choice, true-or-false and free-form questions. As can be seen in Fig. 6, when the evaluation data scale is less than 30% of Dysca (i.e., less than 46.8K samples), the evaluation score show significant fluctuations. When the data scale exceeds 40%, we obtain the stable results, reflecting current scale of Dysca achieves the stable and reliable evaluation results.
Although 40% evaluation scale of Dysca has achieved stable scores, Dysca aims to provide more than just stable rankings, but also draws on massive amounts of data to provide in-depth feedback across different image styles and perceptual subtasks.

## Appendix C The Metadata ( M 𝑀 M italic_M ) of Dysca

Metadata ($M$) is the core of Dysca, which is randomly assembled from our collected source material and contains all the information needed to generate prompt ($P$), image ($I$), and question-answer pairs($Q$). Specifically, the metadata is a data container that contains information in multiple dimensions about the foreground, the attributes corresponding to the foreground, the background, and the artistic style required to generate an image. Therefore, each instance of M is mapped one-to-one to a prompt, an image, and a set of question-answer pairs, respectively.

In order to ensure the quality and stability of the generated images, we carefully select the source material. First, for each perceptual subtask, we collect rich annotation material as described in Section 3.2. However, the metadata composed of these raw annotations is not always usable. On the one hand, some of the content is polysemous, which can easily be misinterpreted by the model’s when generating images. On the other hand, there are backgrounds or artistic styles (e.g., “Pokemon Style", “architectural style", etc.) that negatively affect the quality of the image and do not accurately generate the desired content. In order to test the usability of these source materials, we went through several small-scale pre-generations covering all the source materials. After careful selection, we retain the clips that consistently produced high-quality images. The detailed information of the source materials are shown in Tab. 11.

**Table 11: Detailed information of the source materials.**
| Category | Data Description | #Numbers |
| --- | --- | --- |
| Style | We collected artistic styles from the [community](https://stable-diffusion-art.com/sdxl-styles/), which can be well rendered by Stable Diffusion model. We removed those which have strong reflect on the image content or may generate unsafe image. | 51 |
| Background | We have selected 20 rich backgrounds and they can be accurately generated by Stable Diffusion Model. | 20 |
| Age | We chose four well-characterized age nodes: 14, 25, 40, and 80. | 4 |
| Expression | We chose three characteristic expressions: smiling happily, shouting furiously, calm and placid. | 3 |
| Gender | Male and Female. | 2 |
| Race | We identified these five races based on the ethnicity that can be generated by Stable Diffusion: Caucasian, Asian, African, Indian, and Middle Eastern races | 5 |
| Profession | After pre-generation and careful selection, we chose 20 occupations with distinctive characteristics that are easy to generate. | 20 |
| Action | After pre-generation and careful selection, we chose 20 occupations with distinctive characteristics that are easy to generate. | 20 |
| Celebrity | After pre-generation and careful selection, we chose 50 well-known celebrities. | 50 |
| Animal | We selected a rich variety of animals, including mammals, birds, reptiles, insects, and aquatic animals, and can they be generated by the Stable Diffusion model accurately. | 67 |
| Plant | We selected a rich variety of plants, including flowers, trees, fruits, and vegetables, and they can be generated by the Stable Diffusion model accurately. | 37 |
| Clothes | We selected 16 common types of clothing that are highly distinguishable from each other. | 16 |
| Object | We took the annotations from the MSCOCO [] dataset after removing people, animals, and plants, and added some additional common objects. | 80 |
| Landmark | We chose 23 characteristic landmarks from around the globe and they can be generated by the Stable Diffusion model accurately. | 23 |
| Food | We collected 29 special dishes from around the globe and they can be generated by the Stable Diffusion model accurately. | 29 |
| Movie | We selected 106 movies titles from the rating list of [IMDb](https://www.imdb.com/) based on the number of user reviews. | 106 |
| Anime | We selected 44 anime titles from the rating list of [IMDb](https://www.imdb.com/) based on the number of user reviews. | 44 |
| TV shows | We selected 20 TV show titles from the rating list of [IMDb](https://www.imdb.com/) based on the number of user reviews. | 20 |
| OCR | We randomly selected 5000 words from the IELTS vocabulary for the text material. Among them, words with length less than 3 were removed. | 5000 |
| Color | We selected 8 easily distinguishable colors: red, orange, yellow, green, blue, purple, white, black. | 8 |

## Appendix D Scenarios Details

### D.1 Print Attack Scenario

Followed by the settings in [], we add the attack text on the images. Consider that the image resolution in our Dysca is much higher than the one in [], we increase more font form in terms of font position and font orientation. Fig. 7 to Fig. 11 shows the detailed examples.

Figure: Figure 7: Images with different font size. ’112px’ means that the typos are 112 pixels in size.
Refer to caption: x7.png

Figure: Figure 8: Images with different font color. ’Blue’ means that the color of typos are in blue.
Refer to caption: x8.png

Figure: Figure 9: Images with different font opacity. ’40%’ means that the transparency of typos are 40% and ’100%’ implies full opacity of typos.
Refer to caption: x9.png

Figure: Figure 10: Images with different font orientation. ’$15^{\circ}$’ means that the orientation of typos are $15^{\circ}$ and $0^{\circ}$ implies the typos are horizontal.
Refer to caption: x10.png

Figure: Figure 11: Images with different font position. An image is divided into a grid of 5 rows and 5 columns, leading to 25 sections. ’R1C1’ means the typo is located in row 1, column 1, which is the top left corner of the image.
Refer to caption: x11.png

### D.2 Corrupted Scenario

Examples of the 11 image corruptions are shown in Fig 12.

Figure: Figure 12: Examples of 11 image corruptions applied to a single clean image.
Refer to caption: x12.png

## Appendix E More Examples of Dysca

For each subject we collected in Metadata ($M$), we display one example of their prompt ($P$), generated image ($I$) and corresponding question-answer pairs ($Q$).

Figure: Figure 13: Plant
Refer to caption: x13.png

Figure: Figure 14: Profession
Refer to caption: x14.png

Figure: Figure 15: Celebrity
Refer to caption: x15.png

Figure: Figure 16: Action
Refer to caption: x16.png

Figure: Figure 17: Landmark
Refer to caption: x17.png

Figure: Figure 18: Face
Refer to caption: x18.png

Figure: Figure 19: Animal
Refer to caption: x19.png

Figure: Figure 20: Object
Refer to caption: x20.png

Figure: Figure 21: Clothes
Refer to caption: x21.png

Figure: Figure 22: Food
Refer to caption: x22.png

Figure: Figure 23: Movie
Refer to caption: x23.png

Figure: Figure 24: TV
Refer to caption: x24.png

Figure: Figure 25: Anime
Refer to caption: x25.png

Figure: Figure 26: OCR
Refer to caption: x26.png

## Appendix F Data Sheet

We follow the documentation frameworks provided by Gebru et al. [].

### F.1 Motivation

For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description.

- •
The proposed dataset is used for evaluating current LVLMs perception ability. We use the synthesis images to prevent the potential data leakage problem in current benchmarks. The dataset test LVLMs in 20 subtasks under 4 scenarios and 3 question type, revealing the existing drawbacks of current LVLMs.

Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?

- •
The dataset is created by the AI Safety and Trustworthiness Group on behalf of Key Laboratory of AI Safety of CAS, Institute of Computing Technology.

Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grant or and the grant name and number.

- •
It is funded by the Key Laboratory of AI Safety of CAS, Institute of Computing Technology.

### F.2 Composition

What do the instances that comprise the dataset
represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.

- •
We show the instances list in Tab. 11. The detailed word we collect for metadata $M$ are shown in our anonymous github page [https://github.com/Benchmark-Dysca/Dysca](https://github.com/Benchmark-Dysca/Dysca).

How many instances are there in total (of each type, if appropriate)?

- •
There are a total of 20 subtasks in our Dysca. For details of each subtasks please see refer Fig. 2.

Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).

- •
No. The images in Dysca are completely generated from scratch.

What data does each instance consist of? “Raw” data
(e.g., unprocessed text or images) or features? In either case, please provide a description.

- •
Each instance consists of the prompt, the image generated by stable diffusion, the question and corresponding answer.

Is there a label or target associated with each
instance? If so, please provide a description.

- •
Yes, Dysca provides the ground truth for each instance.

Is any information missing from individual instances?
If so, please provide a description, explaining why this information
is missing (e.g., because it was unavailable). This does not include
intentionally removed information, but might include, e.g., redacted
text.

- •
No.

Are relationships between individual instances made
explicit (e.g., users’ movie ratings, social network links)? If so, please describe how these relationships are made explicit.

- •
There are no relationships between individual instances.

Are there recommended data splits (e.g., training,
development/validation, testing)? If so, please provide a
description of these splits, explaining the rationale behind them.

- •
Following our motivation, the entire proposed dataset is used for testing purposes.

Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.

- •
Errors in image generation resulting from stable diffusion are unavoidable. However, we have performed dataset cleaning to minimize these errors. Furthermore, the stability experiment in Appendix B demonstrates that these errors do not affect the overall evaluation results of the dataset.

Is the dataset self-contained, or does it link to or
otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at
the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a dataset consumer? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.

- •
The proposed Dysca dose not rely on any external resources.

Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals’ non-public communications)? If so, please provide a description.

- •
No.

Does the dataset contain data that, if viewed directly,
might be offensive, insulting, threatening, or might otherwise
cause anxiety? If so, please describe why.

- •
No. To ensure that the generated images do not contain offensive, insulting, threatening, or anxiety-inducing content, we manually filter out words from the metadata $M$ that could potentially trigger the diffusion model to generate such images. Safety checker also used to further avoid unsafe image generation.

Does the dataset relate to people? If not, you may skip the remaining questions in this section.

- •
Yes.

Does the dataset identify any subpopulations (e.g., by
age, gender)? If so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset.

- •
Yes. There are the age, gender and race recognition subtasks in Dysca. Each of them are divided to several subpopulations and the selection of these subpopulations is based on the ability of stable diffusion to generate the representative subpopulations.

Is it possible to identify individuals (i.e., one or
more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how.

- •
Yes. There is the celebrity recognition task in our dataset, where 50 well-know celebrity are chosen. We choose the celebrity who can be generated well by stable diffusion XL.

Does the dataset contain data that might be considered
sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description.

- •
No, our benchmark does not contain any sensitive data.

### F.3 Collection Process

How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.

- •
We display the detailed explanation in Tab. 11.

What mechanisms or procedures were used to collect the
data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated?

- •
We collect the data by manual human curation.

If the dataset is a sample from a larger set, what was the
sampling strategy (e.g., deterministic, probabilistic with
specific sampling probabilities)?

- •
No.

Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?

- •
We collect the metadata of Tab. 11 by authors. The images are generated by stable diffusion and labels of each image are also automatically generated.

Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.

- •
Our dataset was conducted in April of 2024, but the results do not depend on the date of data collection.

Were any ethical review processes conducted (e.g., by an
institutional review board)? If so, please provide a description
of these review processes, including the outcomes, as well as
a link or other access point to any supporting documentation.

- •
No.

Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?

- •
No.

Were the individuals in question notified about the data
collection? If so, please describe (or show with screenshots
or other information) how notice was provided, and provide a
link or other access point to, or otherwise reproduce, the exact
language of the notification itself.

- •
N/A. Our Dysca does not involve the collection from the individuals.

Did the individuals in question consent to the collection
and use of their data? If so, please describe (or show with
screenshots or other information) how consent was requested
and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.

- •
N/A. Our Dysca does not involve the collection from the individuals.

If consent was obtained, were the consenting individuals
provided with a mechanism to revoke their consent in the
future or for certain uses? If so, please provide a description,
as well as a link or other access point to the mechanism (if
appropriate).

- •
N/A. Our Dysca does not involve the collection from the individuals.

Has an analysis of the potential impact of the dataset and
its use on data subjects (e.g., a data protection impact
analysis) been conducted? If so, please provide a description
of this analysis, including the outcomes, as well as a link or
other access point to any supporting documentation.

- •
No.

### F.4 Preprocessing/cleaning/labeling

Was any preprocessing/cleaning/labeling of the data done
(e.g., discretization or bucketing, tokenization, part-of-speech
tagging, SIFT feature extraction, removal of instances, processing
of missing values)? If so, please provide a description. If not,
you may skip the remaining questions in this section.

- •
Yes. We leverage the off-the-shelf models, i.e., PP-OCRv3 [] and CLIP-L-14 [], to clean the data. PP-OCRv3 [] is leveraged as the filter to exclude the failure image that TextDiffusion2 [] generates the wrong text on the image. For the other images, we use CLIP-L-14 [] to filter out the images with low text-image consistency.

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the “raw” data.

- •
Yes. We have saved all the data. However, most of these images are filtered and considered to be useless.

Is the software that was used to preprocess/clean/label the data available? If so, please provide a link or other access point.

- •
Yes. CLIP-L-14 can be downloaded at [https://huggingface.co/docs/transformers/v4.41.3/en/model_doc/clip#transformers.CLIPModel](https://huggingface.co/docs/transformers/v4.41.3/en/model_doc/clip#transformers.CLIPModel).
PP-OCRv3 can be downloaded at [https://github.com/PaddlePaddle/PaddleOCR/blob/main/README_en.md](https://github.com/PaddlePaddle/PaddleOCR/blob/main/README_en.md)

### F.5 Uses

Has the dataset been used for any tasks already? If so, please provide a description.

- •
No. The proposed dataset is the novel one which is used for evaluation current LVLMs perception ability.

Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point.

- •
Yes. We plan to create a section on the project homepage to keep track of
LVLMs papers for researchers to analyze and compare.

What (other) tasks could the dataset be used for?

- •
In this work, we do not explore the possibility of utilizing our benchmark for model training / fine-tuning. Our primary goal in this paper is to provide a large-scale evaluation benchmark that addresses the issue of data leakage in current multimodal evaluation benchmarks and offers evaluation results across multiple subtasks, scenarios, question types and styles. Nevertheless, considering that Dysca has the capability to synthesize high-resolution and unlimited amounts of annotated multimodal data, we believe that Dysca also holds potential as a training data synthesis tool for LVLMs.

Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a dataset consumer might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other risks or harms (e.g., legal risks, financial harms)? If so, please provide a description. Is there anything a dataset consumer could do to mitigate these risks or harms?

- •
Yes.

Are there tasks for which the dataset should not be used? If so, please provide a description.

- •
The proposed dataset should not be used to generate offensive data.

### F.6 Distribution

Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description.

- •
Yes.

How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)?

- •
We will open-source our dataset on our GitHub project homepage. At
the moment, we do not have a DOI number.

When will the dataset be distributed?

- •
The dataset can be downloaded right now.

Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.

- •
The licence of Dysca is ["CreativeML Open RAIL++-M"](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/LICENSE.md), which follows the licence set by the Stable Diffusion XL.

Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.

- •
No.

Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.

- •
Not yet.

### F.7 Maintenance

Who will be supporting/hosting/maintaining the dataset?

- •
Jie Zhang and Zhongqi Wang are hosting and maintaining the dataset.

How can the owner/curator/manager of the dataset be contacted (e.g., email address)?

- •
Email: zhangjie@ict.ac.cn

Is there an erratum? If so, please provide a link or other access point.

- •
No.

Will the dataset be updated (e.g., to correct labeling
errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to dataset consumers (e.g., mailing list, GitHub)?

- •
There are no plans at the moment, but if there are updates, they will
be announced, and the download source will be updated on the project
homepage.

If the dataset relates to people, are there applicable
limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.

- •
No.

Will older versions of the dataset continue to be
supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to dataset consumers.

- •
Yes. If there are any updates, the previous version of the dataset will also be shared on website for download.

If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be
validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to dataset consumers? If so, please provide a description.

- •
Yes. We welcome and encourage researchers to extend/augment/build
on/contribute to our dataset for non-profit purposes without the need
for prior notification.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: It is very common that the text and image cannot be aligned well for T2I generation. As far as i know, only the data clean part in the pipeline is related to this problem. How do you ensure the threshold and biding (top 6 models) way can work for it?
soundness: 3
strengths: 1. The proposed dataset is comprehensive and large in scale. Dysca not only considered different perceptual scenarios, it also provided a testbed for evaluating the perceptual capability of LVLMs under both corruption and printing attack situations.
2. The data construction pipeline is flexible to be extended for assembling the samples of other sub-tasks. Benifit from the powerful generation capability of T2I models, the pipeline also demonstrated the good quality of generative large-scale datasets.
3. The experiments are comprehensive and proved the claimed merits of the proposed dataset.
summary: This work propose a large-scale dataset, i.e., Dysca, for evaluating the perceptual capability of LVLMs. To build such a large-scale dataset, the author propose a flexible pipeline which defines the metadata and a rule-based approach for generating the text-to-image prompt, image and question-answer (QA) pairs. In summary, Dysca consists of 617K Vision-language QA pairs, covering 20 perceptual subtasks, 4 image scenarios and 3 question types. The author conduct comprehensive evaluation for 24 existing LVLMs on Dysca, demonstrating the weakness of existing LVLMs.
weaknesses: There are two main concerns:
1. No data leakage is one of the main claimed contribution for the dataset, but there is no comprehensive statistical comparison of Dysca with existing benchmarks. Only the blind experiment in Appendix D, which should be moved to the main content considering its importance, trying to explain this point which is not enough from my perspective.
2. The setup of adversarial attack, i.e., only PGD, is too simple. Compare to the corruption and printing attack setup, only PGD is considered for adversarial attack is not convincible, especially considering there are bunch of emerging studies on the adversarial attack of LVLMs.


There are also several unclear points,
1. There is no description of the rule-based approach adopted when getting the prompt and QA from the metadata.
2. No explanatio of the metadata itself. Why the metadata is combined with the four aspects? Is there any dataset- or task-specific relationship when considering this?
3. It is unclear that how many examples for each type of corruption and printing attack in the dataset.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Would authors change "" to the right double quotes?
2. Would authors fix the image captions to be below the image and the table captions to be above the table? i.e., Table 2, Figure 6.
3. In Section 4.2.2, is there any reference that a correlation is high if it is higher than 0.6?
soundness: 3
strengths: Their strengths come from the idea of automatically synthesizing dataset curation. The metadata contains information to generate prompt, image, and question-answer pairs. 
1. Scaling is easy because of adopting the synthetic data.
2. They propose the findings, the importance of the language model, the inconsistent performance to different question types, the difference perceptual performance on different sub-tasks, and the degradation in attack. 
3. They demonstrate the validity of Dysca in the section 4.2.2.
summary: This paper proposes Dysca, a dynamic and scalable benchmark that uses large language models (LLMs) and text-to-image diffusion models (T2I) to generate synthetic images, questions, and answers. They filter the low-quality examples by leveraging off-the-shelf models. It is dynamic and scalable. Prior benchmark might have potential data leakage issues and scalability limitations. Evaluation experiments show their findings, such as inconsistency in question types and the vulnerability of attacks. It shows the limitations of current vision language models (VLMs).
weaknesses: Their weaknesses come from the idea of simply scaling data. Note that "simple" does not mean "easy." Papers such as [R1], [R2], and [R3] consider the specific points of cultural understanding, Humor, and hallucination. It would be useful to have specific evaluation key points like these benchmarks.
1. How does the proposed benchmark differ from unifying existing benchmarks into one benchmark? Does it evaluate the model behavior that existing benchmarks fail to cover? Combining the existing benchmarks proposed in Table 1 creates a benchmark covering a wide range of perceptual tasks and question types. Therefore, it should clearly show something that existing benchmarks have not been able to do. For example, it would be valuable to show researchers the characteristics of the samples that all VLMs get wrong.
4. What specific insights can the proposed benchmark provide that are not observable through existing benchmarks? It would be interesting to propose a new metric that is only available in the proposed benchmark. This is a weakness that can be linked to Weakness 1. Metadata is an important role in sample generation. Similar to [R4], an explainable metric using that metadata could be an example. Figure 7 may be good for fine-grained analysis, but showing 51 styles or considering other meta data at once is difficult to understand at once.
5. With the growing body of research on training with synthetic data, suppose we follow the training approach: (1) generate data using the same process with Dysca, and (2) train VLMs. Afterward, how should we handle potential data leakage due to the similar data distribution compared to Dysca? It can be useful to know the sub-populations where all VLMs underperform, even when trained on synthetic images.
7. T2I diffusion models have the limitation of creating all possible images corresponding to question-answer pairs. Could authors discuss this limitation? For example, benchmarks that consider specific vulnerabilities of VLMs [R1, R2, R3, R5, R6] seem to be hard to generate automatically. 

[R1] Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest, ACL'23

[R2] Benchmarking Vision Language Models for Cultural Understanding, EMNLP'24

[R3] BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models, ECCV'24

[R4] Attribute Based Interpretable Evaluation Metrics for Generative Models, ICML'24

[R5] NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples, NeurIPS'24

[R6] Vision language models are blind, arxiv'24

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Can the authors present the complete prompts used for image generation in the appendix? Could some prompts be displayed?

2. Did the authors assess the quality of automatically synthesized images? How is quality ensured when generating such a large scale of images, especially those containing text?

3. How were the QA pairs (text) constructed on such a large scale? Was ChatGPT utilized? or just all by rule-based template?
soundness: 3
strengths: 1. The reviewer appreciates the authors' attempt to address a unique aspect of LVLM benchmarks: the evaluation of multi-stylized images and noisy scenarios. This perspective is intriguing and relevant.
   
2. The paper is well-written with clear motivations and a logical structure. The presentation is visually appealing with sophisticated diagrams. The detailed introduction of the benchmark data facilitates reader comprehension.
   
3. The experimental analysis provides several valuable insights. Notably, there is a high correlation between their evaluation rankings and those of non-synthetic benchmarks, highlighting deficiencies in current LVLMs when facing various question types, image styles, and scenarios.

4. The overall effort in the work is solid. Using an automated synthesis method to generate an evaluation benchmark is an unconventional but intriguing and worthwhile exploration.
summary: This is a data paper. The authors introduce Dysca, a new benchmark for evaluating Large Vision-Language Models (LVLMs) from the perspective of assessing multi-stylized images and noisy scenarios, as well as test environments involving corruption, print attacks, and adversarial attacks. Dysca uses SDXL images to dynamically synthesize varied image styles and questions, enabling evaluation across 51 styles and 20 subtasks under four scenarios: clean, corrupted, print attacked, and adversarially attacked. They also provide interesting insights into LVLMs' capabilities and performance under diverse conditions.
weaknesses: 1. The major concern with this dataset is the use of image generation tool for automatic vision creation. For such images to serve as a gold standard for MLLM evaluation, they must meet a critical assumption: the generated images are perfect and error-free. Unfortunately, even today's state-of-the-art SDXL models, which can produce high-quality images, still blend real and synthetic styles rather than producing wholly realistic images. As humans, we can typically recognize diffusion-synthesized images. SD models excel at creating artistic-style images, but for sure, non-realistic images inevitably introduce bias in evaluations on realistic images. Also, many automatically synthesized images that contain scene texts are often of low quality, i.e., containing inaccurate scene text. Such biases could largely skew model evaluations.
Therefore, the reviewer suggests that the authors add ample discussion regarding this issue. For example, the authors could include some in-depth analytical experiments to validate the concerns raised above.


2. The entire process is fully automated, including the use of CLIP for automated quality assessment of generated QA image-text pairs, without any human inspection, which is problematic.
The reviewer suggests that the authors analyze the impact of all automatically constructed data on quality, or consider using a more reasonable evaluation method for assessment.

3. While using automatically synthesized images significantly reduces annotation costs, as a testing set, it is a dubious claim without (or with less) real value. Conversely, generating a large-scale training set through automated image synthesis is undoubtedly positive.
Therefore, the reviewer hopes that the authors can further explain and clarify this matter.



4. The main focus of this data should be on testing LVLMs' performance in multi-stylized and noisy images and in scenarios involving corruption, print attacks, and adversarial attacks. If a LVLM benchmark is limited to these aspects, its utility and contribution are significantly diminished.
Therefore, the reviewer hopes that the authors can further explain and clarify this matter.


5. "_Although they claim that the questions are re-annotated, previous work (Chen et al., 2024) has demonstrated that these benchmarks have unintentionally leaked into the training data of LLMs and LVLMs_": This claim might largely be inaccurate, as Chen et al. (2024) only discussed very early LVLM benchmarks, which were simplistic and not re-annotated. However, most current benchmarks have re-annotated QA texts, almost solving the data leakage issue.
Please provide further details.

6. The authors claim a large dataset size with 617K visual-language QA pairs. However, for evaluation benchmarks, sheer size is not as important as having a more varied set of subsets assessing different aspects and capabilities [1]. Unfortunately, Dysca only covers 20 image perception subtasks, offering no significant advantage over SEED-Bench2 and BenchLMM. Moreover, it appears to be surpassed by existing work (e.g., [2]) in both data quantity and evaluation scope, which the authors have not compared or mentioned.
The reviewer suggests adding more comparisons for this.


7. As a benchmark, it evaluates too few (comparatively) LVLMs, limited to just a small subset of models (actually 16; the 26 LVLMs are somewhat overclaimed). There are many more state-of-the-art multimodal LLMs that should be included at least in the appendix. the reviewer suggests the authors include evaluations of more models next.
The reviewer strongly recommends adding more MLLMs for experimental comparison.



## Summary

Overall, if the authors had focused solely on evaluating multi-stylized images and noisy scenarios, it would have been a safer claim, even with automated image synthesis. However, by comparing it to current general-purpose LVLM benchmarks, the limitations and significant issues of automated image synthesis are exposed, particularly its inability to replace the importance of realistic images. This is my key suggestion for the authors.  
Thus, "_We demonstrate for the first time that evaluating LVLMs using large-scale synthetic data is valid_" might be an overclaim, at least not sufficiently and safely proved in this paper.

Overall, the reviewer is open-minded. If the authors can actively and effectively address the concerns with more discussions, the reviewer would consider raising my rating.

---

[1] Are We on the Right Way for Evaluating Large Vision-Language Models? 2024.

[2] MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks. 2024

-----------

After the reviewer-author interaction, the reviewer decided to increase the evaluation to positive as the authors provided enough detailed responses and clarification.

Meta-review: This work introduces Dysca, a large-scale dataset designed to evaluate the perceptual capabilities of LVLMs. To construct this extensive dataset, the authors developed a versatile pipeline that defines metadata and employs a rule-based approach to generate text-to-image prompts, images, and question-answer (QA) pairs. The authors also conduct comprehensive evaluation for 24 existing LVLMs on Dysca, demonstrating the weakness of existing LVLMs.

This submission received one positive review and two negative reviews before rebuttal. The main concerns lie on fully automated pipeline and presentation issues. After rebuttal, all the issues are solved. All reviewers have raised their score to boradline accept. In particular, the authors have presented a very detailed response to all reviewers. The main draft has been largely modified according to the reviewers' suggestions and questions. 

AC checks the discussions and reviews, finds this work is valuable to current LVLMs benchmarks. 

AC recommends accepting this work.

Decision: Accept (Poster)